Writing Kubernetes Controllers

Introduction: how a controller works

A Kubernetes controller often looks almost suspiciously small:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
{
    // read objects
    // create or update children
    // update status
    return ctrl.Result{}, nil
}

Most of the work is hidden below that function.

Kubernetes still has to answer several questions. What is being watched? Who keeps the local cache up to date? How does a Pod change cause a custom resource to be checked? Why does the queue store a name instead of the complete object? Why can a read look old just after a successful write?

This guide answers those questions one layer at a time. It also explains terms such as informer, indexer, lister, event handler, predicate, and workqueue when we first use them.

If these pieces are not clear, a controller may work in a small test and fail when the cluster gets busy, the cache falls behind, or the process restarts.

We will build the model from the bottom up. First, we will see what the Kubernetes libraries do for us. Then we will use controller-runtime and Kubebuilder, knowing what each layer does.

The main idea is simple: a controller runs the same loop again and again. A watch event is only a signal that something may have changed. The reconciler reads the current state and moves it closer to the desired state.

The controller is a convergence loop, not an event processor.

The examples use Go and the current client-go / controller-runtime model as of August 2026. API names evolve, but the architecture in this article is deliberately about the durable concepts underneath those APIs.

1. The controller mental model

A controller continuously compares two things:

• Desired state - normally represented by Kubernetes API objects, especially .spec.
• Observed state - Kubernetes objects in the local cache plus any external systems the controller manages.

It then performs actions until the two agree closely enough.

A useful equation is:

reconcile(desired, observed) -> minimal actions needed to converge

For a Deployment-like controller:

Desired: replicas = 3
Observed: 2 matching Pods
Action: create 1 Pod

For a database operator:

Desired: backupPolicy = daily
Observed: cloud backup schedule missing
Action: create cloud backup schedule

For an infrastructure CRD:

Desired: ApplicationRoute references Service payments
Observed: generated HTTPRoute still points to old Service checkout
Action: patch HTTPRoute

The controller does not need a perfect history of everything that happened. It needs an eventually accurate view of current state and repeated opportunities to reconcile.

That is the reason Kubernetes controllers can survive:

• duplicate events;
• coalesced events;
• controller restarts;
• temporary API failures;
• temporary external failures;
• receiving only the latest object state instead of every intermediate mutation.

If your logic requires receiving the exact sequence A -> B -> C and becomes incorrect if it only sees A -> C, you are building an event-processing system, not a robust Kubernetes reconciler.

2. What a controller actually promises

A good controller should usually satisfy five properties.

2.1 Idempotency

Calling reconciliation repeatedly with the same observed state should not create repeated side effects.

Bad:

// Every reconciliation creates another object.
client.Create(ctx, &Job{GenerateName: "backup-"})

Better:

// Derive a stable desired object identity.
jobName := app.Name + "-backup"
// Get it; create only if absent; patch if drifted.

When a non-idempotent external API must be called, introduce an idempotency token or persist enough state that the operation can be retried safely.

2.2 Level-triggered behavior

A robust reconciler asks:

What is true now, and what should be true?

rather than:

What event just happened?

The event decides what key to reconsider. The reconcile function determines what action is currently required.

2.3 Eventual convergence

Temporary errors should delay convergence, not permanently corrupt the state machine.

A controller may need multiple passes:

pass 1: create Deployment
pass 2: Deployment exists; create Service
pass 3: Service gets ClusterIP; publish status

There is no requirement to do every step in one invocation.

2.4 Explicit ownership

Know which fields and resources your controller owns.

Two controllers continuously overwriting the same field create an oscillating system. Resource ownership should be visible through API design, field-management conventions, owner references, status responsibilities, or separate resources.

2.5 Bounded work per reconciliation

Prefer reconciling one primary object in approximately O(1) or O(k) work, where k is the number of things related to that object.

Avoid one global object whose reconcile scans every Pod, Service, or CR in the cluster unless the problem is truly global. At large scale, indexes and reverse mappings matter as much as the reconcile algorithm itself.

3. The Kubernetes API basics: List, Watch, and resourceVersion

Before informers, understand what they are protecting you from having to implement manually.

Kubernetes exposes collection operations that support List and Watch.

A simplified sequence is:

1. LIST /apis/example.io/v1/webapps
   -> objects + collection resourceVersion=10245
2. WATCH /apis/example.io/v1/webapps?resourceVersion=10245
   -> ADDED
   -> MODIFIED
   -> DELETED
   -> ...

The list gives the initial state. The watch gives mutations after that point.

resourceVersion is an opaque version identifier used to coordinate those reads. Your controller should normally not invent ordering logic around its numeric shape. Client libraries use it to continue watches and reason about freshness.

3.1 Watches end

A watch is not an eternal connection. Network failures happen. API servers restart. Historical versions are compacted.

If a client requests a version the API server can no longer serve, the server can return 410 Gone. The correct recovery is to obtain a fresh collection state and begin watching again from the returned version. client-go encapsulates this list/watch lifecycle in the Reflector machinery (Kubernetes API concepts).

3.2 Watch events are not your queue contract

A watch may report:

• ADDED
• MODIFIED
• DELETED
• BOOKMARK

Modern Kubernetes also supports streaming the initial collection as watch events in supported configurations. But controller authors should avoid coupling business logic to transport details. The informer layer turns these mechanisms into a maintained local state view and notifications.

3.3 Why not call the API server for every reconcile?

Imagine a controller with:

• 50,000 Pods;
• 5 controllers interested in Pods;
• hundreds of reconciles per second.

If every reconciliation performs fresh list operations, the controller becomes an API-server load generator. Informers instead maintain long-lived local caches so normal reads are memory reads, while writes still go to the API server.

That read-cache/write-API pattern is fundamental to Kubernetes controller scalability.

4. The raw client-go controller pipeline

The most useful diagram in this lesson is the one below. Learn this pipeline and the controller-runtime abstractions become much easier to reason about.

The raw client-go path from kube-apiserver to workqueue and back.

At a high level:

kube-apiserver
    |
    v
ListerWatcher
    |
    v
Reflector
    |
    v
DeltaFIFO
    |
    v
Store / Indexer
    |
    v
SharedInformer event handlers
    |
    v
Workqueue of keys
    |
    v
Workers
    |
    v
sync/reconcile(key)
    |
    +---- reads local cache/lister
    |
    +---- writes kube-apiserver

The important separation is:

• Informer side: maintain a local view and cheaply notice possible changes.
• Queue side: buffer and de-duplicate reconciliation demand.
• Worker side: perform potentially slow reconciliation outside the informer delivery path.

Do not perform cloud calls, long database operations, or complex reconciliation directly inside informer event handlers. Shared informer documentation explicitly expects handlers to process notifications promptly; lengthy work belongs in a workqueue (client-go tools/cache).

5. Reflector: maintaining the local view

A Reflector connects a Kubernetes list/watch source to a local store-like consumer.

Conceptually it does this:

LIST initial objects
       |
       v
replace local known set
       |
       v
WATCH from resourceVersion
       |
       +-- add
       +-- update
       +-- delete
       |
       v
if watch breaks -> reconnect / relist as required

The reflector is intentionally not your business logic. Its job is synchronization.

In raw client-go, the source is commonly represented by a ListerWatcher. Generated informers and shared informer factories hide creation of the list/watch functions for known Kubernetes types.

For a CRD you have several choices:

1. Generated typed clients/informers/listers. Best when you control a stable API and want compile-time types.

2. Dynamic client + dynamic informer. Useful when GVRs are discovered at runtime or you intentionally operate on unstructured resources.

3. controller-runtime cache. The normal Kubebuilder/operator path; it internally creates informers for registered object types and exposes cache-backed client.Client reads.

6. DeltaFIFO: why the informer does not hand events directly to your logic

Between the reflector and the local store sits a queue-like structure called DeltaFIFO.

A delta records what happened to an object from the informer’s perspective, such as:

Added
Updated
Deleted
Replaced
Sync

The FIFO groups deltas by object key. This is useful because a hot object may change many times faster than consumers can process every intermediate change.

Suppose an object changes rapidly:

v10 -> v11 -> v12 -> v13

A convergence-oriented controller normally cares that the cache ends up at v13; it does not need to run business logic four times merely because four transport updates arrived.

DeltaFIFO is part of getting the informer store to a correct current representation while tolerating bursts, relists, and resync notifications (client-go tools/cache).

6.1 Delete tombstones

Deletes contain an important edge case.

If the watch connection was interrupted, the client might discover during relist that an object disappeared without having observed the original delete event. In that case client-go can represent the deletion using DeletedFinalStateUnknown, often called a tombstone.

The embedded object may be stale; the key is the durable piece of information. Raw informer delete handlers therefore need to handle both a normal object and a tombstone (client-go tools/cache).

With controller-runtime, common handlers abstract most of this away, but understanding tombstones explains why delete handling should not depend on having a pristine final object snapshot.

7. Store, Indexer, and SharedIndexInformer

After deltas are processed, the informer maintains an in-memory object store.

A basic store supports operations such as:

Add
Update
Delete
Get
List

An Indexer adds secondary indexes so objects can be found by something other than namespace/name.

Examples:

Pods by spec.nodeName
Jobs by owner UID
ApplicationRoutes by referenced Service
WebApps by referenced ConfigMap
Certificates by secretName

A SharedIndexInformer combines:

• informer event distribution;
• a shared local store;
• indexes over that store.

7.1 Shared means shared inside the process

Suppose three controllers in one process all care about Pods.

Without sharing:

Controller A -> LIST/WATCH all Pods -> own full cache
Controller B -> LIST/WATCH all Pods -> own full cache
Controller C -> LIST/WATCH all Pods -> own full cache

With a shared informer:

one LIST/WATCH all Pods
        |
        v
one Pod cache/indexer
   |       |       |
 handler A handler B handler C

A SharedIndexInformer lets multiple consumers share the underlying watch and cache.

Each event handler gets its own notification path. Events delivered to one handler are sequential for that handler, but there is no global coordination ordering between separate handlers (client-go tools/cache).

That is another reason not to use cross-handler event order as a correctness mechanism.

7.2 Never mutate informer-owned objects

Raw listers and informer handlers may expose objects that belong to the shared cache. Treat those objects as immutable.

If code mutates an object from the shared store in place, it can corrupt indexes and surprise every other controller using the same informer.

Controller-runtime’s cache-backed Get and List normally deep-copy on reads before returning objects, which is safer. Predicate and event-handler paths can still receive informer-owned objects; if you must mutate them, deep-copy first (controller-runtime cache deep dive).

8. Listers and cache-backed reads

A lister is a typed read facade over an informer’s local indexer.

Raw generated code often gives you APIs such as:

pod, err := podLister.Pods(namespace).Get(name)
pods, err := podLister.Pods(namespace).List(selector)

These do not normally make a fresh API call. They read the local cache.

This has two major consequences.

8.1 Reads are cheap

Controllers can reconcile at high frequency without issuing a GET for every object relationship.

8.2 Reads can be behind writes

You may successfully write an object to the API server, then immediately read through the informer cache and still see the previous version until the watch path catches up.

This is not a bug in your controller. It is the architecture.

Never write reconciliation logic like this:

if err := r.Update(ctx, child); err != nil {
    return ctrl.Result{}, err
}
// Dangerous assumption: cache must already contain our write.
if err := r.Get(ctx, key, child); err != nil {
    ...
}
if child.Spec.Value != justWrittenValue {
    return ctrl.Result{}, fmt.Errorf("write did not stick")
}

Instead, let the write produce a watch event and reconcile again from the newer cached state.

9. SharedInformerFactory: one watch, many consumers

client-go provides shared informer factories that lazily construct and share informers for resource types.

Conceptually:

factory := informers.NewSharedInformerFactory(clientset, resyncPeriod)
podInformer := factory.Core().V1().Pods()
nodeInformer := factory.Core().V1().Nodes()
podLister := podInformer.Lister()
nodeLister := nodeInformer.Lister()
factory.Start(stopCh)
factory.WaitForCacheSync(stopCh)

The factory is useful because it coordinates lifecycle and prevents each piece of code from accidentally creating its own full watch/cache for the same type.

9.1 and startup correctness

HasSynced

A controller should generally not begin making decisions from a cache until the initial informer population has completed.

Otherwise, this sequence is possible:

controller starts
cache has loaded only 20% of Pods
reconcile sees 2 Pods instead of actual 10
controller creates 8 duplicates

Informer lifecycle includes synchronization checks for this reason. Controller-runtime’s Manager waits for shared caches to synchronize before starting managed controllers (controller-runtime overview).

Startup cache synchronization is different from saying the cache will never be stale afterward. It only establishes that the initial view has caught up to the synchronization point used when the informer started.

10. Workqueue internals: de-duplication, dirty keys, retries, and delay

The workqueue is one of the most important controller primitives and one of the most commonly under- explained.

A Kubernetes controller usually enqueues keys, not object snapshots.

For namespaced resources the key is effectively:

namespace/name

With controller-runtime it is represented by reconcile.Request, whose common form contains a types.NamespacedName.

Why keys?

Because by the time a worker executes, the event object’s snapshot may already be stale. The worker should fetch the latest state from the cache.

10.1 De-duplication

Suppose 20 Pod events all map to the same owning WebApp:

pod-1 update -> enqueue default/app
pod-2 update -> enqueue default/app
pod-3 update -> enqueue default/app
...

The queue coalesces identical pending keys. The controller can reconcile default/app once and observe all 20 current Pods.

Controller-runtime explicitly relies on this batching behavior when multiple sources map to the same reconcile key (controller-runtime event handlers).

10.2 Dirty while processing

A subtle but crucial case:

T0 queue contains app-a
T1 worker gets app-a and starts reconciling
T2 another event enqueues app-a
T3 reconcile finishes

You do not want the T2 update to disappear.

The standard workqueue tracks whether an item became dirty while it was being processed. When the worker calls Done(item), if the item was marked dirty again, the queue re-adds it (client-go util/workqueue).

That gives you both:

• de-duplication while pending;
• no lost wakeup if the key changes during processing.

A controller normally uses one de-duplicating queue and multiple workers.

10.3 Done and Forget are different

In raw client-go rate-limiting queues:

• Done(item) tells the queue that current processing is complete.
• Forget(item) tells the rate limiter to forget retry history for that key.

A normal raw worker pattern looks like:

item, shutdown := queue.Get()
if shutdown {
    return
}
defer queue.Done(item)
err := sync(item)
if err != nil {
    queue.AddRateLimited(item)
    return
}
queue.Forget(item)

Forgetting retry history matters because the rate limiter otherwise keeps treating future failures as later attempts in the same failure sequence (client-go util/workqueue).

Controller-runtime handles this queue protocol around your Reconcile result.

10.4 Build the controller once with raw client-go

Even if you intend to use controller-runtime in production, writing the skeleton once with raw client-go is one of the fastest ways to understand what the framework is doing.

Assume generated code gives us:

type Controller struct {
    webAppLister platformlisters.WebAppLister
    webAppSynced cache.InformerSynced
    queue        workqueue.TypedRateLimitingInterface[string]
    kubeClient   kubernetes.Interface
    appClient    platformclient.Interface
}

Register an event handler on the informer:

func NewController(
    kubeClient kubernetes.Interface,
    appClient platformclient.Interface,
    appInformer platforminformers.WebAppInformer,
) *Controller {
    c := &Controller{
        webAppLister: appInformer.Lister(),
        webAppSynced: appInformer.Informer().HasSynced,
        queue: workqueue.NewTypedRateLimitingQueue(
            workqueue.DefaultTypedControllerRateLimiter[string](),
        ),
        kubeClient: kubeClient,
        appClient:  appClient,
    }
    appInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: func(obj interface{}) {
            c.enqueue(obj)
        },
        UpdateFunc: func(oldObj, newObj interface{}) {
            oldApp := oldObj.(*platformv1alpha1.WebApp)
            newApp := newObj.(*platformv1alpha1.WebApp)
            if oldApp.ResourceVersion == newApp.ResourceVersion {
                return
            }
            c.enqueue(newObj)
        },
        DeleteFunc: func(obj interface{}) {
            c.enqueue(obj)
        },
    })
    return c
}

The event handler does almost nothing:

func (c *Controller) enqueue(obj interface{}) {
    key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
    if err != nil {
        return
    }
    c.queue.Add(key)
}

DeletionHandlingMetaNamespaceKeyFunc is useful because a delete callback may contain a tombstone rather than a normal object.

Start the controller only after the informer cache has synchronized:

func (c *Controller) Run(ctx context.Context, workers int) error {
    defer c.queue.ShutDown()
    if ok := cache.WaitForCacheSync(ctx.Done(), c.webAppSynced); !ok {
        return fmt.Errorf("failed to sync WebApp informer")
    }
    for i := 0; i < workers; i++ {
        go wait.UntilWithContext(ctx, c.runWorker, time.Second)
    }
    <-ctx.Done()
    return nil
}

Each worker repeatedly drains one item:

func (c *Controller) runWorker(ctx context.Context) {
    for c.processNextItem(ctx) {
    }
}
func (c *Controller) processNextItem(ctx context.Context) bool {
    key, shutdown := c.queue.Get()
    if shutdown {
        return false
    }
    defer c.queue.Done(key)
    if err := c.sync(ctx, key); err != nil {
        c.queue.AddRateLimited(key)
        return true
    }
    c.queue.Forget(key)
    return true
}

And sync uses the lister for current state:

func (c *Controller) sync(ctx context.Context, key string) error {
    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        return nil
    }
    app, err := c.webAppLister.WebApps(namespace).Get(name)
    if apierrors.IsNotFound(err) {
        // Primary no longer exists. If external cleanup is needed,
        // use a finalizer before deletion instead of relying on this snapshot.
        return nil
    }
    if err != nil {
        return err
    }
    // Lister-owned object must be treated as immutable.
    app = app.DeepCopy()
    return c.reconcileCurrentState(ctx, app)
}

That is the controller architecture in plain form:

Informer handler -> key -> queue -> worker -> lister -> reconcile -> API writes

Controller-runtime mostly gives you safer, composable versions of these same pieces.

10.5 Generated listers versus dynamic informers

When you own a CRD API and compile against it, code generation can produce:

typed clientset
informers
listers
DeepCopy methods
apply configurations

This gives strong typing and a familiar built-in-controller style.

When a controller must operate on resources whose schema is not compiled into the binary, client-go also offers dynamic clients and dynamic shared informer factories. Objects are typically represented as unstructured.Unstructured, and the controller identifies the resource with a GroupVersionResource.

Use dynamic machinery when the API really is dynamic, for example:

• a generic policy controller discovers arbitrary resources;
• a platform watches extension CRDs installed by plugins;
• a migration tool works across versions unknown at compile time.

Do not use unstructured objects merely to avoid defining Go API types for a CRD you own. Typed APIs catch whole classes of mistakes before the controller is deployed.

11. Reconciliation: events are hints, state is truth

Now we can state the controller rule precisely:

Handlers translate events into reconciliation keys. Reconcilers translate current state into convergence actions.

That separation is what makes controllers resilient.

Imagine your WebApp controller watches the CR, Deployment, Service, and ConfigMap.

You should not write:

switch event.Type {
case DeploymentDeleted:
    recreateDeployment()
case ConfigMapChanged:
    restartPods()
case WebAppUpdated:
    updateEverything()
}

Instead, all relevant events map to:

default/webapp-a

and Reconcile(default/webapp-a) does something like:

1. Read WebApp.
2. Read referenced ConfigMap.
3. Read owned Deployment.
4. Read owned Service.
5. Compute desired Deployment and Service.
6. Create or patch drift.
7. Compute status.
8. Return.

If the Deployment vanished, step 5/6 recreates it. If the ConfigMap changed, step 5 computes a different desired Pod template. If the WebApp spec changed, the same logic converges the new desired state.

The reconciler is simpler because it does not care which path woke it up.

12. controller-runtime: what it builds on top of client-go

Kubebuilder-generated operators normally use sigs.k8s.io/controller-runtime.

It does not replace the client-go architecture; it organizes it.

The major components are:

12.1 Manager

ctrl.Manager owns process-level infrastructure:

• shared cache;
• cache-backed client;
• direct API writer behavior;
• controller lifecycle;
• leader election;
• schemes and REST mapping;
• health/readiness endpoints;
• metrics;
• webhooks;
• event recording and other runnables.

A process usually creates one Manager and registers multiple Controllers with it.

12.2 Cache

The controller-runtime cache maintains informers and object stores for resource types. It acts as a read client and drives Kubernetes object event handlers (controller-runtime cache package).

12.3 Client

The default Manager client is a split model:

Get/List -> local cache
Create/Update/Patch/Delete -> API server

Controller-runtime explicitly does not promise immediate cache invalidation after a write (controller-runtime cache deep dive).

12.4 Controller

A Controller owns:

• event sources;
• event handlers;
• predicates;
• a workqueue;
• one or more worker goroutines;
• a Reconciler.

12.5 Source

A Source produces events. The common source is a Kubernetes Kind driven by the Manager cache, but sources can also represent external/generic channels.

12.6 EventHandler

A handler turns source events into reconcile requests.

The three patterns to memorize are:

EnqueueRequestForObject
EnqueueRequestForOwner
EnqueueRequestsFromMapFunc

We will use all three.

12.7 Predicate

A predicate filters events before they are handed to the handler. This is useful for dropping irrelevant updates, but overly aggressive predicates can also remove opportunities for self-healing.

12.8 Reconciler

A Reconciler is called with a request key. It reads current state and converges it.

12.9 FieldIndexer

The Manager’s FieldIndexer adds indexes to the cache so relationship lookups do not require scanning entire object collections.

13. Your first CRD controller

Use a deliberately simple CRD:

apiVersion: platform.tejo.dev/v1alpha1
kind: WebApp
metadata:
  name: payments
  namespace: prod
spec:
  image: ghcr.io/acme/payments@sha256:abc...
  replicas: 3
  configRef: payments-config

The controller should ensure:

• one Deployment named payments exists;
• one Service named payments exists;
• Deployment replicas/image/config match the WebApp spec;
• status reflects readiness.

A stripped-down reconciler:

type WebAppReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}
func (r *WebAppReconciler) Reconcile(
    ctx context.Context,
    req ctrl.Request,
) (ctrl.Result, error) {
    var app platformv1alpha1.WebApp
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if !app.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, &app)
    }
    if err := r.ensureDeployment(ctx, &app); err != nil {
        return ctrl.Result{}, err
    }
    if err := r.ensureService(ctx, &app); err != nil {
        return ctrl.Result{}, err
    }
    if err := r.updateStatus(ctx, &app); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

Notice what is missing:

• no event type;
• no “DeploymentCreated” branch;
• no “ConfigMapChanged” branch;
• no assumption that this call is the first call;
• no assumption that previous steps completed.

That is intentional.

14. Watching the primary resource

The simplest setup is:

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&platformv1alpha1.WebApp{}).
        Complete(r)
}

Conceptually For(&WebApp{}) wires:

WebApp informer events
      |
      v
EnqueueRequestForObject
      |
      v
queue: namespace/name of WebApp
      |
      v
Reconcile(namespace/name)

For a primary resource, this is usually what you want.

15. Watching resources your controller owns

A controller frequently creates secondary Kubernetes resources.

For WebApp:

WebApp
  +-- owns Deployment
  +-- owns Service

When creating a child, set a controller owner reference:

dep := &appsv1.Deployment{...}
if err := controllerutil.SetControllerReference(app, dep, r.Scheme); err != nil {
    return err
}

Then configure the controller:

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&platformv1alpha1.WebApp{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Complete(r)
}

Owns is effectively the owner-mapping pattern:

Deployment event
      |
      v
read controller owner reference
      |
      v
WebApp namespace/name
      |
      v
enqueue WebApp

That means if someone deletes an owned Deployment, the Deployment delete event maps back to the WebApp, and the next reconciliation recreates the missing child (Kubebuilder: Watching Resources).

Primary, owned-secondary, and unrelated-resource watch patterns.

15.1 OwnerReference is not merely a watch hint

Owner references also participate in Kubernetes garbage collection.

Use them when the child’s lifecycle is genuinely subordinate to the owner. Do not add owner references merely because you want an event mapping. A cluster-scoped/namespaced mismatch or incorrect ownership model can create surprising garbage-collection behavior.

16. Watching resources you do not own

Now suppose WebApp.spec.configRef points to a ConfigMap managed by another team.

The WebApp controller does not own that ConfigMap, but a change may require recreating or rolling the Deployment.

You need this relationship:

ConfigMap event
      |
      v
Which WebApps reference this ConfigMap?
      |
      v
enqueue those WebApps

Controller-runtime uses a mapping handler:

Watches(
    &corev1.ConfigMap{},
    handler.EnqueueRequestsFromMapFunc(r.mapConfigMapToWebApps),
)

The mapping function returns zero, one, or many reconcile.Requests (controller-runtime event handlers).

A naive implementation might list every WebApp and scan them:

func (r *WebAppReconciler) mapConfigMapToWebApps(
    ctx context.Context,
    obj client.Object,
) []reconcile.Request {
    var apps platformv1alpha1.WebAppList
    if err := r.List(ctx, &apps, client.InNamespace(obj.GetNamespace())); err != nil {
        return nil
    }
    var reqs []reconcile.Request
    for i := range apps.Items {
        if apps.Items[i].Spec.ConfigRef == obj.GetName() {
            reqs = append(reqs, reconcile.Request{
                NamespacedName: client.ObjectKeyFromObject(&apps.Items[i]),
            })
        }
    }
    return reqs
}

Functionally correct. Operationally poor at scale.

We fix it with an index in Section 18.

17. Watching multiple resource types

A Controller can watch many kinds while reconciling one logical primary type.

For example:

ctrl.NewControllerManagedBy(mgr).
    For(&platformv1alpha1.WebApp{}).
    Owns(&appsv1.Deployment{}).
    Owns(&corev1.Service{}).
    Watches(
        &corev1.ConfigMap{},
        handler.EnqueueRequestsFromMapFunc(r.mapConfigMapToWebApps),
    ).
    Watches(
        &corev1.Secret{},
        handler.EnqueueRequestsFromMapFunc(r.mapSecretToWebApps),
    ).
    Complete(r)

The important architecture is:

WebApp ---------- EnqueueForObject --------+
Deployment ------ EnqueueForOwner ---------+
Service --------- EnqueueForOwner ---------+--> ONE WebApp queue --> Reconcile(WebApp)
ConfigMap ------- MapFunc -----------------+
Secret ---------- MapFunc ----------------+

Many event sources converge into a single keyspace.

This is called event multiplexing. It is powerful because all relationships are normalized to the object whose desired state you know how to reconcile (controller-runtime event handlers).

17.1 Do not make one giant “controller for everything”

Watching multiple resource types is appropriate when all those events feed the lifecycle of one primary API.

It is not an argument for one process-wide reconciler that owns unrelated APIs such as:

Database + WebApp + Certificate + Backup + Route + Tenant

If resources have different keyspaces, different failure domains, different concurrency requirements, and separate APIs, use separate Controllers. They may still share the same Manager and informer caches.

18. Field indexes: the missing skill in many controllers

Indexes are where basic controller code becomes scalable controller code.

Suppose 20,000 WebApps exist and a ConfigMap changes. Scanning all 20,000 to discover the 3 that reference the ConfigMap is O(N) work per event.

Instead, build a reverse index once and maintain it incrementally as objects change.

An informer index turns relationship discovery into a reverse lookup.

Register it during setup:

const configRefIndex = "spec.configRef"
func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    if err := mgr.GetFieldIndexer().IndexField(
        context.Background(),
        &platformv1alpha1.WebApp{},
        configRefIndex,
        func(obj client.Object) []string {
            app := obj.(*platformv1alpha1.WebApp)
            if app.Spec.ConfigRef == "" {
                return nil
            }
            return []string{app.Spec.ConfigRef}
        },
    ); err != nil {
        return err
    }
    return ctrl.NewControllerManagedBy(mgr).
        For(&platformv1alpha1.WebApp{}).
        Owns(&appsv1.Deployment{}).
        Watches(
            &corev1.ConfigMap{},
            handler.EnqueueRequestsFromMapFunc(r.mapConfigMapToWebApps),
        ).
        Complete(r)
}

Then the map function becomes:

func (r *WebAppReconciler) mapConfigMapToWebApps(
    ctx context.Context,
    obj client.Object,
) []reconcile.Request {
    var apps platformv1alpha1.WebAppList
    if err := r.List(
        ctx,
        &apps,
        client.InNamespace(obj.GetNamespace()),
        client.MatchingFields{configRefIndex: obj.GetName()},
    ); err != nil {
        return nil
    }
    reqs := make([]reconcile.Request, 0, len(apps.Items))
    for i := range apps.Items {
        reqs = append(reqs, reconcile.Request{
            NamespacedName: client.ObjectKeyFromObject(&apps.Items[i]),
        })
    }
    return reqs
}

The index name is a controller convention; controller-runtime does not interpret "spec.configRef" as JSONPath. The index function determines the actual indexed values (controller-runtime cache deep dive).

18.1 Index by stable relationship keys

Useful keys include:

owner UID
namespace/name of referenced object
node name
tenant ID
cluster ID
secret name
gateway name
service account
external resource ID

If names can collide across namespaces, encode namespace into the indexed value:

return []string{app.Namespace + "/" + app.Spec.ConfigRef}

18.2 Multi-valued indexes

An object can emit multiple index keys.

If a route references three backends:

func(obj client.Object) []string {
    route := obj.(*ApplicationRoute)
    values := make([]string, 0, len(route.Spec.Backends))
    for _, b := range route.Spec.Backends {
        values = append(values, route.Namespace+"/"+b.ServiceName)
    }
    return values
}

Then one Service change can efficiently find every route that references it.

18.3 Cache indexes are not database indexes

Indexes live in the controller’s in-memory cache. They:

• consume memory;
• are rebuilt as the informer cache initializes;
• are updated as informer objects change;
• only help reads served by that cache.

Do not expect arbitrary server-side field selectors to magically use your controller-runtime index. These are local reverse-lookup structures.

19. Predicates: reduce noise without breaking convergence

Predicates decide whether an event should proceed to its event handler.

Example: reconcile a CR only when .metadata.generation changes:

For(
    &platformv1alpha1.WebApp{},
    builder.WithPredicates(predicate.GenerationChangedPredicate{}),
)

For Custom Resources with a status subresource, generation commonly changes when spec changes, not on normal status updates. This can prevent your own status writes from immediately causing another reconciliation (controller-runtime predicates).

But predicates are not free optimization.

19.1 The danger of over-filtering

If you use GenerationChangedPredicate, then a status-only external correction may not trigger your controller.

Example:

Your controller sets Ready=True.
Another actor accidentally wipes status.
Generation does not change.
Your predicate drops the update.
Status remains wrong until some other event wakes the object.

The controller-runtime documentation calls out this tradeoff explicitly (controller-runtime predicates).

Use predicates when:

• the dropped events are truly irrelevant;
• another source still guarantees convergence;
• periodic RequeueAfter or resync is intentionally part of the design;
• load measurements prove the filter is useful.

Do not use them simply because “status updates cause extra reconciles.” A cheap no-op reconcile is often safer than a clever predicate that suppresses self-healing.

20. Single queue, multiple workers

For one Controller and one logical primary resource, the default architecture should usually be:

many watched sources
       |
       v
one keyspace
       |
       v
one de-duplicating rate-limiting queue
       |
       +--> worker 1
       +--> worker 2
       +--> worker 3
       ...

Controller-runtime exposes concurrency through controller options such as MaxConcurrentReconciles (controller-runtime controller options).

Example:

ctrl.NewControllerManagedBy(mgr).
    For(&platformv1alpha1.WebApp{}).
    WithOptions(controller.Options{
        MaxConcurrentReconciles: 8,
    }).
    Complete(r)

This does not create eight queues. It creates multiple workers consuming one controller queue.

20.1 Why one queue is usually the right abstraction

You gain:

• key de-duplication across every watched source;
• one retry history per reconciliation key;
• one place to measure queue depth and latency;
• natural batching when many secondaries map to one primary;
• simpler fairness and shutdown behavior.

20.2 Concurrency correctness

With N workers, assume two different primary objects can reconcile at the same time.

Your reconciler therefore must not depend on unprotected process-global mutable state.

The workqueue prevents the same key from being processed simultaneously by ordinary worker flow, but it does not serialize:

namespace/app-a
namespace/app-b

If both modify one shared external resource, you need a concurrency strategy at that shared resource boundary.

20.3 How many workers?

Do not set concurrency by folklore.

Measure:

reconcile CPU time
API QPS / throttling
external API latency
queue depth
queue wait duration
cache read cost
write contention / 409 conflicts
memory

A mostly-I/O reconciler may benefit from more workers. A CPU-heavy global planner can harm itself with high concurrency. An external provider with a 10-QPS limit may require a low controller concurrency even when the Kubernetes cluster could handle much more.

21. Multi-queue architectures: when they are actually justified

“Multi queue” can mean several different designs. Separate them.

Prefer multiple Controllers for genuinely distinct queueing domains.

21.1 Case A: multiple Controllers in one Manager - recommended

Example operator:

Manager
  +-- WebApp Controller        -> WebApp queue -> 16 workers
  +-- Certificate Controller  -> Cert queue   -> 2 workers
  +-- Backup Controller       -> Backup queue -> 4 workers

This is the natural multi-queue architecture.

Use it when domains differ in:

• primary resource type;
• reconciliation algorithm;
• latency requirements;
• concurrency limits;
• external dependency;
• retry behavior;
• failure containment;
• metrics/SLO ownership.

All controllers can still share the Manager’s caches and clients.

21.2 Case B: one Controller with a priority-aware queue

Sometimes all requests are the same logical key type, but some deserve earlier processing.

Examples:

• deletion/finalization should outrank routine drift repair;
• a fleet controller wants fresh watch-triggered changes ahead of periodic background scans;
• user-facing objects have higher urgency than maintenance objects within the same API.

Current controller-runtime supports a priority-queue model and custom queue construction through Controller options. This is often better than maintaining two independent queues that can process the same key concurrently (controller-runtime controller options).

The question becomes:

same reconciliation semantics, different urgency?
    -> one priority-aware queue

rather than:

create a second queue

21.3 Case C: several custom queues in one controller - rare

You might consider custom queues for:

fast lane: metadata-only changes
slow lane: expensive external provisioning

or:

region A queue
region B queue

But now you own complexity:

• Can the same key appear in both queues?
• Who de-duplicates across queues?
• Can two workers reconcile the same object simultaneously?
• What happens when work changes class while queued?
• How is retry history moved?
• How do you guarantee starvation does not occur?
• How do you drain both queues during shutdown?

Often the better design is to split the lifecycle into two resources and two Controllers, or keep one queue and persist an explicit phase in the API.

21.4 Case D: per-tenant queues - usually a warning sign

Per-tenant queues sound attractive for fairness, but they turn a controller into a scheduler.

If you genuinely need:

• weighted fair sharing;
• reservations;
• quotas;
• priorities;
• preemption;
• admission control;

model those as first-class scheduling policy, not a pile of ad-hoc Go queues inside a reconciler.

21.5 Decision table

NeedPreferred design

More throughput for same resourceOne queue + more workers

Different primary APIsSeparate Controllers / queues

Different external API limitsSeparate Controllers or explicit limiter

Same keyspace, different urgencyPriority-aware single queue

Independent failure domainsSeparate Controllers

Fair scheduling across tenantsDedicated scheduler/admission design

Long async workflow stagesPersist phase/state; often separate controllers

Horizontal shardingPartition ownership/input stream, not just create more local queues

22. Rate limiting, backoff, and scheduled reconciliation

There are three different reasons to reconcile again. Treat them differently.

22.1 A real watched event arrives

No explicit requeue is necessary. The event handler enqueues the key.

22.2 Reconciliation failed transiently

Return an error:

return ctrl.Result{}, err

Controller-runtime requeues the request using its rate-limiting behavior; current reconciliation APIs describe exponential-backoff behavior for ordinary errors (controller-runtime reconcile semantics).

Examples:

• API server timeout;
• temporary cloud API 503;
• dependency not reachable;
• transient optimistic-concurrency conflict after retry budget.

22.3 You intentionally want to check later

Use:

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

Examples:

• external operation is asynchronous and has no callback;
• certificate renewal window should be checked later;
• lease-like state needs periodic refresh;
• you need a bounded safety poll for an external system.

RequeueAfter uses delayed queueing; it does not need a sleeping worker. If a real event arrives sooner, the same key can be reconciled sooner because the queue de-duplicates demand (controller-runtime cache deep dive).

22.4 Do not use errors as polling

This:

if !providerReady {
    return ctrl.Result{}, fmt.Errorf("not ready")
}

mixes expected waiting with failure handling and causes backoff semantics to determine your poll interval.

Prefer:

if !providerReady {
    return ctrl.Result{RequeueAfter: 20 * time.Second}, nil
}

23. Cache consistency and read-after-write traps

The controller-runtime default client reads from the local cache and writes directly to the API server (controller-runtime cache deep dive).

That means:

read cached child: replicas=2
patch API server: replicas=3
read cached child immediately: may still say replicas=2
watch arrives
cache becomes replicas=3

Writes go directly to the API server while ordinary reads come from the informer cache.

23.1 Design for this intentionally

A clean pattern is:

if changed {
    if err := r.Patch(ctx, obj, patch); err != nil {
        return ctrl.Result{}, err
    }
    // Stop here. Let watch feedback trigger the next observation.
    return ctrl.Result{}, nil
}

Do not write, then repeatedly poll the cache in the same reconciliation waiting for it to catch up.

23.2 When you truly need a live read

Controller-runtime can construct clients/readers that bypass the cache, but use them deliberately.

A direct API read is justified when correctness requires current server state that cannot tolerate cache lag. It is not a general replacement for understanding the cache model.

If you bypass the cache frequently, re-evaluate whether:

• the resource should be cached;
• the index/watch configuration is wrong;
• the API contract is forcing imperative sequencing;
• an explicit status/phase handshake would be cleaner.

24. Resync is not relist

This is an important advanced distinction.

An informer resync periodically re-presents objects that are already in its local store to event handlers as synchronization notifications. It does not mean “issue a new full list to the API server.” Current controller-

runtime cache documentation and Kubernetes project explanations explicitly distinguish resync from relist (controller-runtime cache deep dive).

Why have resync at all?

Suppose your CR references an external DNS provider. Kubernetes objects do not change, but someone manually deletes the DNS record.

Without an external event source, a periodic opportunity to reconcile can detect the drift.

24.1 Resync can be filtered accidentally

A synthetic resync update can present equivalent old/new state. A predicate such as generation- changed may drop it because generation did not change (controller-runtime cache deep dive).

If periodic reconciliation is essential to correctness, make that requirement explicit. Often RequeueAfter on the primary resource is easier to understand than depending on global informer resync behavior.

24.2 Relist is different

A relist re-establishes collection state with the API server, for example when a watch can no longer continue from its previous resourceVersion.

Think:

resync = "please reconsider cached objects"
relist = "rebuild/reconcile cache contents with server collection state"

25. Idempotency, optimistic concurrency, Patch vs Update

Kubernetes objects are concurrently modified shared state.

Even if your controller is the only intended spec writer, other actors can modify:

• labels;
• annotations;
• finalizers;
• status;
• managed fields;
• child resource status;
• fields controlled by admission/defaulting.

25.1 Optimistic concurrency

Objects carry resourceVersion. If you issue a full update based on a version that is no longer current, the API server can reject the update with a conflict (Kubernetes API concepts).

The controller must tolerate this.

A conflict is often not an exceptional system failure. It means:

Someone changed the object since your read. Re-observe and calculate again.

25.2 Prefer small patches for narrow ownership

Suppose you own one annotation and status, while another actor owns labels.

A full Update can accidentally overwrite unrelated fields or create conflict pressure. Patches can express smaller mutations (Kubernetes API concepts).

Controller-runtime offers several patch styles, and server-side apply can be appropriate where declarative field ownership is intentional.

The design question is not “Patch is always better than Update.” It is:

Which fields do I own, and what write primitive makes that ownership explicit and conflict-
tolerant?

25.3 Create-or-patch pattern

For owned children, a common reconcile shape is:

var dep appsv1.Deployment
dep.Name = app.Name
dep.Namespace = app.Namespace
op, err := controllerutil.CreateOrPatch(ctx, r.Client, &dep, func() error {
    if err := controllerutil.SetControllerReference(app, &dep, r.Scheme); err != nil {
        return err
    }
    dep.Spec.Replicas = ptr.To(app.Spec.Replicas)
    dep.Spec.Template.Spec.Containers = desiredContainers(app)
    return nil
})

Whether you use CreateOrPatch, server-side apply, or explicit Get/Create/Patch, preserve the same invariant: reconstruct desired state deterministically and make the smallest safe change.

26. Status, conditions, observedGeneration, and ownership

A useful Kubernetes API separates:

spec   = user/controller desired configuration
status = controller's observation of reality

For example:

status:
  observedGeneration: 7
  readyReplicas: 3
  conditions:
  - type: Ready
    status: "True"
    reason: AllComponentsReady

26.1

observedGeneration

If the CR’s generation is 8 but status reports observedGeneration: 7, consumers know the controller has not yet finished processing the latest desired state.

This is much more meaningful than a bare Ready=True that might describe an older spec.

26.2 Conditions are an API contract

Good conditions describe states that callers care about:

Ready
Progressing
Degraded
DependenciesReady

Avoid turning conditions into a log stream:

CreatingDeployment
DeploymentCreated
CreatingService
ServiceCreated

Those are internal steps, not durable API truths.

26.3 Avoid status write loops

Before writing status, compare desired status to current status. Do not write the same status on every reconciliation.

A no-op status update still creates API traffic and can create another watch event.

27. Deletion and finalizers

A Kubernetes delete can be a multi-stage operation.

If an object has finalizers, the API server sets deletionTimestamp and retains the object until finalizers are removed (Kubernetes API concepts).

A controller finalizer is a promise:

Before this object disappears, I have cleanup that must be completed or explicitly abandoned.

Typical cleanup:

• delete cloud load balancer;
• revoke credentials;
• remove external DNS record;
• release allocated IP;
• delete external database;
• detach policy from a remote system.

A reconcile state machine should look like this:

A production reconciler has an explicit deletion/finalization path.

Simplified code:

const finalizer = "webapp.platform.tejo.dev/finalizer"
if app.DeletionTimestamp.IsZero() {
    if !controllerutil.ContainsFinalizer(&app, finalizer) {
        patch := client.MergeFrom(app.DeepCopy())
        controllerutil.AddFinalizer(&app, finalizer)
        if err := r.Patch(ctx, &app, patch); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }
} else {
    if controllerutil.ContainsFinalizer(&app, finalizer) {
        if err := r.cleanupExternalState(ctx, &app); err != nil {
            return ctrl.Result{}, err
        }
        patch := client.MergeFrom(app.DeepCopy())
        controllerutil.RemoveFinalizer(&app, finalizer)
        if err := r.Patch(ctx, &app, patch); err != nil {
            return ctrl.Result{}, err
        }
    }
    return ctrl.Result{}, nil
}

Cleanup must itself be idempotent. The controller can crash after deleting the external resource but before removing the finalizer; the next reconciliation must treat “already absent” as successful cleanup.

Finalizers are not ordered as a cross-controller workflow mechanism. Kubernetes deliberately does not enforce finalizer ordering (Kubernetes API concepts).

28. Leader election, HA, and what it does not scale

Running two replicas of an operator does not automatically mean both are doing useful reconciliation work.

The common controller-runtime deployment uses leader election:

replica A -> leader -> controllers running
replica B -> follower -> waiting
replica C -> follower -> waiting

This provides availability: if the leader dies, another replica can acquire leadership and start controllers.

It does not provide horizontal work sharing.

If you need multiple active replicas processing disjoint objects, you need explicit sharding/ownership semantics so that:

• each object belongs to one active worker shard at a time;
• watches/cache input can ideally be partitioned;
• failover can reassign ownership safely;
• shared external resources remain concurrency-safe.

Simply disabling leader election on three identical replicas usually means three controllers all trying to reconcile the same keys.

29. Scaling controllers in large clusters

Controller scalability is mostly about reducing unnecessary work in four places:

API server -> network -> deserialize -> cache -> event mapping -> queue -> reconcile -> writes

Optimize from left to right.

29.1 Restrict what you cache

If your controller only cares about one namespace, do not automatically cache every Secret in the cluster.

Controller-runtime supports cache configuration by namespace/object/selectors. Selective caches can drastically reduce memory and startup cost for high-cardinality resource types (controller-runtime cache deep dive).

Be deliberate with:

• Pods;
• Secrets;
• ConfigMaps;
• Events;
• EndpointSlices;
• large custom resources.

29.2 Use indexes instead of repeated scans

This is often the single easiest improvement.

Bad:

Service event -> list 100,000 routes -> inspect each

Better:

Service event -> index lookup -> 12 routes

29.3 Keep reconcile work bounded

If one object owns 50,000 children, one reconciliation might inherently be expensive. Consider whether the API is too coarse.

Sometimes introduce a lower-level resource:

Fleet
  -> many FleetShard resources
       -> each controller reconcile bounded subset

29.4 Control write amplification

Do not patch resources whose desired state already matches current state. Avoid status churn. Avoid timestamp fields that change on every reconcile unless they serve a real contract.

29.5 Respect client/API rate limits

More workers are not useful when the client is continuously throttled. Queue delay and API throttling should be visible in metrics.

30. Kubernetes 1.36 controller advances: staleness mitigation and sharded watches

Kubernetes 1.36 added two advanced areas worth knowing because they expose longstanding controller scaling/correctness boundaries.

30.1 Cache staleness mitigation

The normal controller model accepts that cache reads can lag writes. At very high scale, a controller can make a second decision from stale cache state before its own previous write is reflected locally.

Kubernetes 1.36 introduced client-go machinery that can track store resource versions and help selected controllers avoid acting when their informer cache has not yet caught up to resource versions they have written (Kubernetes 1.36 controller staleness mitigation).

The important lesson for most custom-controller authors is not “adopt every new internal primitive immediately.” It is:

Cache staleness is a real correctness dimension, not merely a latency metric.

If your controller performs destructive or expensive operations based on relationships across several high-churn resource types, understand what freshness assumptions it makes.

30.2 Server-side sharded list and watch

Horizontal controller replicas traditionally have a painful scaling property:

replica 0 gets all Pod events, discards 2/3
replica 1 gets all Pod events, discards 2/3
replica 2 gets all Pod events, discards 2/3

CPU work may be partitioned, but API-server network and per-replica deserialization/cache cost are multiplied.

Kubernetes 1.36 introduced an alpha server-side sharded list/watch capability. When explicitly enabled and used, the API server can filter collection data so each replica receives only its assigned hash shard. (Kubernetes 1.36 server-side sharded list and watch)

At very large scale, sharding the input stream matters as much as sharding workers.

This is advanced and alpha in 1.36, but it reinforces an architectural principle:

horizontal scaling = partition ownership + partition input + partition work

not merely:

run more identical Pods

31. Observability and controller SLOs

A controller should be observable as a queueing system and as a convergence system.

31.1 Queue metrics

Track:

queue depth
oldest item age / queue wait time
add rate
retry rate
rate-limited additions
worker utilization

A queue depth of 10 may be fine if keys wait 20 ms. A queue depth of 2 may be alarming if those two objects have waited 30 minutes.

31.2 Reconcile metrics

Track by controller and result:

reconcile count
reconcile duration p50/p95/p99
success
error
requeue
requeue_after

Controller-runtime exposes standard controller metrics, including reconciliation counts/durations and errors in current releases (controller-runtime metrics).

31.3 API interaction

Track:

client request rate
errors
409 conflicts
429 throttling
latency by verb/resource

31.4 Cache health

Useful signals:

initial sync duration
watch reconnects
resource-version progress
cache object count
memory usage

Kubernetes 1.36 also added observability related to informer/store resource versions in the staleness work (Kubernetes 1.36 controller staleness mitigation).

31.5 Domain convergence metrics

This is the part generic controller metrics cannot provide.

For a WebApp controller:

webapps_not_ready
webapp_reconcile_to_ready_seconds
webapps_degraded_by_reason
orphaned_children
external_provisioning_seconds

The actual user SLO is often:

“After a valid desired-state change, 99% of resources converge within 30 seconds.”

not:

“Reconcile() returns in under 100 ms.”

A fast reconciler that is stuck retrying forever is not healthy.

32. Testing strategy: unit tests, fake clients, envtest, and real clusters

Controller testing should reflect the layered architecture.

32.1 Pure unit tests for desired-state functions

Extract deterministic logic:

func DesiredDeployment(app *WebApp, cfg *ConfigMap) *appsv1.Deployment

Test this with plain Go.

These tests are fast and should cover most combinations of:

• defaults;
• labels;
• owner metadata;
• rollout-triggering hashes;
• resource settings;
• invalid combinations.

32.2 Reconciler unit tests with a fake client

Useful for focused CRUD flows, but know the limitation: a fake client is not a real API server, not a real watch/cache pipeline, and may not reproduce defaulting, validation, subresources, resourceVersion behavior, or indexes exactly like production.

Treat it as a unit-test tool, not proof that controller integration semantics are correct.

32.3 envtest integration tests

controller-runtime/pkg/envtest starts a local etcd and kube-apiserver for integration tests (controller-runtime envtest).

This lets you test:

• real CRD schemas;
• API validation;
• watch-driven reconciliation;
• manager cache;
• field indexes;
• status subresource behavior;
• conflicts closer to real behavior;
• finalizer flows.

Kubebuilder documentation recommends running the reconciler with the Manager cache in these tests when cache features such as indexes matter (Kubebuilder: Writing Tests with EnvTest).

Remember: envtest does not run an entire Kubernetes cluster. There is no kubelet and no normal controller-manager unless you add corresponding behavior. A Deployment you create will not magically produce Pods.

32.4 Real-cluster end-to-end tests

Use Kind or a dedicated cluster when you need:

• garbage collector behavior;
• built-in controllers;
• scheduling;
• real Pods;
• admission chain interaction;
• network policy;
• leader election/failover;
• realistic API-server load.

32.5 Useful failure tests

Test these intentionally:

controller restarts after child create but before status update
child deleted by another actor
external API succeeds but response is lost
API write returns conflict
cache has not observed previous write yet
finalization is retried 10 times
referenced object is deleted and recreated with same name
many secondary events map to same primary
controller is shut down with queued work

If your controller is correct under those cases, you probably understand controllers better than someone who has only tested the happy path.

33. Debugging controllers systematically

When a controller “is not reacting,” debug the pipeline in order.

33.1 Was the source actually watched?

Check setup code and RBAC:

get/list/watch permissions
correct GVK/GVR
correct namespace/cache scope
correct label/field selectors

33.2 Is the informer synced?

A controller can appear idle because its cache never completed initial sync, often due to authorization or API discovery problems.

33.3 Did the predicate drop the event?

Log or test predicates directly.

A common mistake is applying a global GenerationChangedPredicate and then wondering why secondary status/deletion changes do not trigger reconciliation.

33.4 Did the handler map to the expected key?

For owner mapping:

Does the child have the right controller ownerRef?
Is the owner GVK registered in the Scheme?

For map functions:

Did the field index register?
Is the index value namespaced correctly?
Does delete/recreate behavior still map correctly?

33.5 Is the key in the queue but workers are saturated?

Look at:

queue depth
queue wait
reconcile duration
external API latency
worker count
rate limiter

33.6 Is reconciliation happening but doing nothing?

Log the decision, not just “starting reconcile.”

Useful structured fields:

controller
namespace
name
uid
generation
observedGeneration
action
reason
child
externalResourceID

33.7 Are you reading stale cache state?

If logs show:

patch succeeded
next line reads old value

check whether the read came from the Manager’s cache-backed client. Often the fix is not bypassing the cache; it is ending the current reconcile and allowing watch feedback to drive the next pass.

34. Worked example: ApplicationRoute controller

Now combine the concepts in a controller closer to a real infrastructure platform.

Assume this CRD:

apiVersion: networking.tejo.dev/v1alpha1
kind: ApplicationRoute
metadata:
  name: checkout-api
  namespace: commerce
spec:
  hostnames:
  - checkout.internal.example.com
  backends:
  - service: checkout
    port: 8080
    weight: 90
  - service: checkout-canary
    port: 8080
    weight: 10
  tlsSecret: checkout-tls
  timeout: 2s

The controller renders an HTTPRoute and policy resources.

It needs to react when:

ApplicationRoute changes             -> primary watch
owned HTTPRoute changes/deletes      -> owner watch
referenced Service changes/deletes   -> reverse-map watch
referenced TLS Secret changes        -> reverse-map watch
Gateway policy changes               -> perhaps reverse-map/global mapping

A production controller often watches several inputs but converges one primary API.

34.1 Index routes by Service

const backendServiceIndex = "applicationRoute.backendService"
if err := mgr.GetFieldIndexer().IndexField(
    ctx,
    &networkingv1alpha1.ApplicationRoute{},
    backendServiceIndex,
    func(obj client.Object) []string {
        route := obj.(*networkingv1alpha1.ApplicationRoute)
        values := make([]string, 0, len(route.Spec.Backends))
        for _, backend := range route.Spec.Backends {
            values = append(values,
                route.Namespace+"/"+backend.Service,
            )
        }
        return values
    },
); err != nil {
    return err
}

34.2 Index routes by Secret

const tlsSecretIndex = "applicationRoute.tlsSecret"
if err := mgr.GetFieldIndexer().IndexField(
    ctx,
    &networkingv1alpha1.ApplicationRoute{},
    tlsSecretIndex,
    func(obj client.Object) []string {
        route := obj.(*networkingv1alpha1.ApplicationRoute)
        if route.Spec.TLSSecret == "" {
            return nil
        }
        return []string{route.Namespace + "/" + route.Spec.TLSSecret}
    },
); err != nil {
    return err
}

34.3 Map a Service event back to routes

func (r *ApplicationRouteReconciler) routesForService(
    ctx context.Context,
    obj client.Object,
) []reconcile.Request {
    key := obj.GetNamespace() + "/" + obj.GetName()
    var routes networkingv1alpha1.ApplicationRouteList
    if err := r.List(
        ctx,
        &routes,
        client.MatchingFields{backendServiceIndex: key},
    ); err != nil {
        return nil
    }
    reqs := make([]reconcile.Request, 0, len(routes.Items))
    for i := range routes.Items {
        reqs = append(reqs, reconcile.Request{
            NamespacedName: client.ObjectKeyFromObject(&routes.Items[i]),
        })
    }
    return reqs
}

34.4 Setup all watches

func (r *ApplicationRouteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // Index registration omitted here for brevity.
    return ctrl.NewControllerManagedBy(mgr).
        For(&networkingv1alpha1.ApplicationRoute{}).
        Owns(&gatewayv1.HTTPRoute{}).
        Watches(
            &corev1.Service{},
            handler.EnqueueRequestsFromMapFunc(r.routesForService),
        ).
        Watches(
            &corev1.Secret{},
            handler.EnqueueRequestsFromMapFunc(r.routesForSecret),
        ).
        WithOptions(controller.Options{
            MaxConcurrentReconciles: 16,
        }).
        Complete(r)
}

All those event streams feed one queue of ApplicationRoute keys.

34.5 Reconcile logic

func (r *ApplicationRouteReconciler) Reconcile(
    ctx context.Context,
    req ctrl.Request,
) (ctrl.Result, error) {
    var route networkingv1alpha1.ApplicationRoute
    if err := r.Get(ctx, req.NamespacedName, &route); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if !route.DeletionTimestamp.IsZero() {
        return r.finalize(ctx, &route)
    }
    if err := r.ensureFinalizer(ctx, &route); err != nil {
        return ctrl.Result{}, err
    }
    deps, err := r.loadDependencies(ctx, &route)
    if err != nil {
        // If dependency is temporarily missing, update condition and wait
        // for Service/Secret watch events instead of tight polling.
        _ = r.setDependenciesReady(ctx, &route, false, err.Error())
        return ctrl.Result{}, nil
    }
    desired := buildHTTPRoute(&route, deps)
    if err := r.applyHTTPRoute(ctx, &route, desired); err != nil {
        return ctrl.Result{}, err
    }
    if err := r.updateStatus(ctx, &route, deps); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

Notice the controller does not ask:

Was I called because Service checkout changed?

It asks:

Given ApplicationRoute checkout-api and all dependencies as they exist now,
what should the generated route state be?

That is the durable controller mindset.

34.6 Event burst example

Assume a deployment rollout causes:

Service update
EndpointSlice update x 20
Secret update
HTTPRoute status update x 3
ApplicationRoute status update

If all relevant events map to commerce/checkout-api, queue de-duplication can collapse much of the burst into a small number of actual reconcile passes.

The controller does not need to “process 25 events.” It needs to converge one route.

34.7 Failure example: HTTPRoute deleted during reconcile

Timeline:

T0 reconcile reads HTTPRoute exists
T1 another actor deletes it
T2 controller patches old object -> NotFound / conflict
T3 reconcile returns error or simply re-observes
T4 delete watch maps owner -> ApplicationRoute key
T5 reconcile reads current state: child absent
T6 controller recreates desired HTTPRoute

Correctness comes from convergence, not locking the entire cluster state while a reconciliation runs.

35. Production checklist and anti-patterns

Before calling a controller production-ready, answer these questions.

35.1 Watch topology

• What is the primary resource?
• Which owned resources should map through owner references?
• Which non-owned resources should map through reverse indexes?
• Are any watched resource types unnecessarily cluster-wide?
• Can one high-fanout event enqueue an unbounded number of primaries?

35.2 Reconciliation

• Is the reconciler idempotent?
• Does it reconstruct desired state from current truth?
• Can it tolerate missing children?
• Can it tolerate duplicate calls?
• Can it tolerate cache lag after writes?
• Is work bounded per key?
• Are external side effects idempotent?

35.3 Queueing

• Is one queue with N workers sufficient?
• If there are multiple queues, why are they separate correctness domains?
• Can the same key enter two queues?
• What is the retry policy?
• What is the maximum queue wait SLO?
• Can poison keys retry forever?

35.4 API writes

• Which fields does the controller own?
• Does it patch only what it owns?
• Does it avoid no-op updates?
• Are conflicts expected and recoverable?
• Does status include observedGeneration where appropriate?

35.5 Deletion

• Does the resource actually need a finalizer?
• Is cleanup idempotent?
• What happens if the external system is permanently unreachable?
• Can operators diagnose a stuck terminating object?

35.6 Operability

• Can you explain why an object is not converged?
• Are queue, reconcile, API, and domain metrics available?
• Can you distinguish user errors from transient infrastructure errors?
• Are controller events useful and rate-limited?
• Is there a runbook for stuck finalizers and queue growth?

35.7 Common anti-patterns

Anti-pattern: business logic in event handlers

Symptom: handlers call external APIs or mutate resources.

Why it fails: blocks informer delivery and couples correctness to event snapshots/order.

Instead: map event -> key; do work in reconcile.

Anti-pattern: list everything on every event

Symptom: ConfigMap update lists 100k CRs.

Why it fails: O(N) cache scans become CPU/memory-lock contention.

Instead: maintain an informer field index.

Anti-pattern: use GenerationChangedPredicate everywhere

Symptom: controller misses status/dependency drift.

Why it fails: dropped events can remove self-healing opportunities.

Instead: filter only proven noise and preserve another convergence trigger.

Anti-pattern: write then demand immediate cache coherence

Symptom: controller thinks its successful write failed.

Why it fails: normal reads are cache-backed and asynchronous relative to writes.

Instead: return; let the watch feed back the new state.

Anti-pattern: one giant reconciler for unrelated APIs

Symptom: one queue, one error policy, one worker pool for certificates, routes, backups, databases, and apps.

Why it fails: head-of-line blocking and failure coupling.

Instead: separate Controllers; share the Manager/cache where useful.

Anti-pattern: multiple queues as a shortcut for state machines

Symptom: “provisioning queue”, “ready queue”, “cleanup queue” for the same object.

Why it fails: duplicate ownership and hard recovery after restart.

Instead: persist phase/conditions in the API and let reconciliation derive the next action.

Anti-pattern: finalizer with unbounded impossible cleanup

Symptom: CR remains Terminating forever because a deleted external account cannot be accessed.

Why it fails: finalizer became an availability dependency for Kubernetes deletion.

Instead: define retry bounds, operator override policy, and clear status/events.

36. Further study

The following primary references are worth reading after this article.

• Kubernetes API concepts - List/Watch, resourceVersion, watch recovery, streaming lists, sharded list/watch, finalization, and update conflicts.
• client-go tools/cache - Reflector, DeltaFIFO, Store, Indexer, SharedInformer, SharedIndexInformer, and tombstones.
• client-go util/workqueue - Typed workqueues, delaying queues, rate-limiting queues, and dirty/done semantics.
• How the controller-runtime Cache Actually Works - Cache behavior, indexes, resync versus relist, sizing, and read-after-write implications.
• controller-runtime overview - Manager, cache-backed client, builder, controllers, reconciler, and metrics.
• controller-runtime event handlers - Object, owner, and map-function handlers.
• controller-runtime controller options - Concurrency and queue configuration.
• controller-runtime cache - Cache configuration and informer interfaces.
• Kubebuilder: Watching Resources - Primary, owned, and externally managed resources.
• controller-runtime predicates - Generation, label, annotation, and resource-version filters and their caveats.
• Kubebuilder CronJob controller - An end-to-end controller with field indexing and owned Job watches.
• controller-runtime reconcile semantics - Retry and requeue behavior.
• Kubernetes 1.36 controller staleness mitigation - Staleness signals and observability.
• Kubernetes 1.36 server-side sharded List/Watch - Partitioning high-cardinality resource streams.
• controller-runtime envtest - Local API-server integration testing.
• Kubebuilder: Writing Tests with EnvTest - Controller integration-test patterns.

37. Closing mental model

If you remember only one architecture, remember this:

                         KUBERNETES API SERVER
                                |
                         LIST / WATCH feedback
                                |
                                v
                    +-------------------------+
                    | Informer / Shared Cache |
                    | Store + Indexes         |
                    +-----------+-------------+
                                |
                         object notifications
                                |
                                v
                    +-------------------------+
                    | Predicate + Handler     |
                    | event -> primary key(s) |
                    +-----------+-------------+
                                |
                                v
                    +-------------------------+
                    | Workqueue               |
                    | dedupe + delay + retry  |
                    +-----------+-------------+
                                |
                         N concurrent workers
                                |
                                v
                    +-------------------------+
                    | Reconcile(primary key)  |
                    | read CURRENT state      |
                    | compute desired state   |
                    | make idempotent changes |
                    +-----------+-------------+
                                |
                             WRITES
                                |
                                +-----------> API server

From that model, the terminology falls into place:

• Watch tells the client that state may have changed.
• Reflector maintains the list/watch stream.
• DeltaFIFO organizes incoming object deltas for cache synchronization.
• Informer maintains a local store and publishes notifications.
• SharedInformer lets multiple consumers share that watch/cache.
• Indexer creates reverse lookup structures over cached objects.
• Lister reads typed objects from the informer cache.
• Handler maps an event to one or more reconciliation keys.
• Predicate filters events before mapping.
• Workqueue de-duplicates, delays, rate-limits, and schedules keys.
• Worker pulls a key.
• Reconciler reads current state and converges it.
• Controller wires sources, handlers, queue, workers, and reconciler together.
• Manager owns process-wide cache/client/lifecycle/leader-election plumbing.

And the most important design rule becomes simple:

Do not build a controller that depends on remembering every event. Build one that can read what is true now and move it closer to what should be true.

Once this model is clear, Kubernetes controllers stop feeling like magic. You can choose the right watch, build the right index, understand the queue, and trace a missed reconciliation from the API server to the worker.