Kubernetes Operators in Practice: Custom Controllers, Reconciliation Loops, and CRD Design Patterns

Most Kubernetes extensions start the same way: someone writes a shell script that polls the API, patches a resource, and calls it automation. It works until it does not. The script races against itself on restart. It misses events during downtime. It has no retry logic and no status reporting. Two weeks later there is a second script to watch the first one.

The operator pattern exists because cluster automation has specific failure modes that general scripting does not address. The API server is the source of truth, not your process memory. Level-triggered reconciliation means your controller recovers from any missed event by re-running the same logic on the next sync. Status subresources give you a typed, observable surface for reporting what the controller actually did.

This article covers the operator pattern from first principles: CRD design, the reconciliation loop, status management, finalizers, leader election, and the production pitfalls that are not in the getting-started guides.

What an Operator Actually Is

An operator is a controller that watches Custom Resources and drives the cluster toward a declared state. It is not a sidecar, not a job runner, not a cron. The defining characteristic is the control loop:

1. Observe the current state of a resource and its dependencies.
2. Compare against the desired state declared in the spec.
3. Take actions to close the gap.
4. Write the result into the status subresource.
5. Requeue if work remains.

This is identical to how built-in controllers work. The Deployment controller watches Deployment objects and manages ReplicaSets. A database operator watches a PostgresCluster object and manages StatefulSets, Services, ConfigMaps, and backup CronJobs. The mechanics are the same.

The key insight is level-triggered vs edge-triggered. An edge-triggered system responds to events (a resource was created). A level-triggered system responds to state (this resource exists and its desired state differs from actual state). The reconciliation loop is level-triggered. This means if your controller crashes and misses 50 events, it will still converge correctly when it restarts, because it re-reads the current state from the API server.

CRD Design: Get the Schema Right Early

CRDs are hard to migrate once they have users. The structural schema matters.

// api/v1alpha1/postgrescluster_types.go

// PostgresClusterSpec defines the desired state of a PostgresCluster.
type PostgresClusterSpec struct {
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=7
    Replicas int32 `json:"replicas"`

    // +kubebuilder:validation:Pattern=`^\d+(\.\d+)?[KMGT]i?$`
    StorageSize string `json:"storageSize"`

    Version string `json:"version"`

    // +optional
    Resources corev1.ResourceRequirements `json:"resources,omitempty"`

    // +optional
    Backup *BackupSpec `json:"backup,omitempty"`
}

type PostgresClusterStatus struct {
    // +optional
    Phase PostgresClusterPhase `json:"phase,omitempty"`

    // +optional
    ReadyReplicas int32 `json:"readyReplicas,omitempty"`

    // +optional
    // +listType=map
    // +listMapKey=type
    Conditions []metav1.Condition `json:"conditions,omitempty"`

    // The resource version of the StatefulSet this controller last reconciled.
    // +optional
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type PostgresCluster struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   PostgresClusterSpec   `json:"spec,omitempty"`
    Status PostgresClusterStatus `json:"status,omitempty"`
}

A few decisions worth calling out explicitly:

The +kubebuilder:subresource:status marker is critical. Without it, spec and status are updated in the same API call. With it, they are separate subresources with separate RBAC. Controllers update status without touching spec. Users update spec without touching status. This prevents the controller from accidentally overwriting user changes.

ObservedGeneration tracks which version of the spec the controller last acted on. It lets you detect whether a status is stale. If metadata.generation is 5 and status.observedGeneration is 3, someone updated the spec and the controller has not processed it yet.

For conditions, use metav1.Condition (from k8s.io/apimachinery/pkg/apis/meta/v1), not a custom struct. It has a defined LastTransitionTime, Reason, Message, and ObservedGeneration. The Kubernetes ecosystem tooling understands it.

Avoid storing derived state in the spec. If you can compute it from the cluster, compute it. Specs should be declarative intent, not a cache of what you discovered.

The Reconciliation Loop

Controller-runtime is the library that most Go operators are built on. The reconciler interface is one method:

// internal/controller/postgrescluster_controller.go

type PostgresClusterReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // Fetch the resource. If it was deleted, the watch will fire with a
    // tombstone. If it is gone, we are done.
    cluster := &v1alpha1.PostgresCluster{}
    if err := r.Get(ctx, req.NamespacedName, cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Always update status at the end of reconciliation, regardless of outcome.
    defer func() {
        if err := r.Status().Update(ctx, cluster); err != nil {
            log.Error(err, "failed to update status")
        }
    }()

    // Handle deletion before proceeding.
    if !cluster.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, cluster)
    }

    // Ensure the finalizer is present before doing any work.
    if !controllerutil.ContainsFinalizer(cluster, finalizerName) {
        controllerutil.AddFinalizer(cluster, finalizerName)
        if err := r.Update(ctx, cluster); err != nil {
            return ctrl.Result{}, err
        }
        // Returning here causes a re-queue, which will re-enter Reconcile
        // with the finalizer now present.
        return ctrl.Result{}, nil
    }

    // Core reconciliation logic.
    if err := r.reconcileStatefulSet(ctx, cluster); err != nil {
        r.setCondition(cluster, "StatefulSetReady", metav1.ConditionFalse, "ReconcileError", err.Error())
        return ctrl.Result{}, err
    }

    if err := r.reconcileServices(ctx, cluster); err != nil {
        r.setCondition(cluster, "ServiceReady", metav1.ConditionFalse, "ReconcileError", err.Error())
        return ctrl.Result{}, err
    }

    // Observe actual readiness.
    ready, err := r.checkReadiness(ctx, cluster)
    if err != nil {
        return ctrl.Result{}, err
    }

    if !ready {
        cluster.Status.Phase = v1alpha1.PhaseCreating
        // Re-check in 10 seconds if pods are not ready yet.
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }

    cluster.Status.Phase = v1alpha1.PhaseReady
    cluster.Status.ObservedGeneration = cluster.Generation
    return ctrl.Result{}, nil
}

A few non-obvious decisions here:

Return ctrl.Result{}, nil to stop requeuing. Return ctrl.Result{}, err to requeue with exponential backoff. Return ctrl.Result{RequeueAfter: duration} to requeue at a fixed interval. Return ctrl.Result{Requeue: true} to requeue immediately (use sparingly, it can cause hot loops).

Never return an error if the resource was not found. client.IgnoreNotFound handles this. The object was deleted between the watch event and your fetch, which is a normal race condition.

The defer for status update is deliberate. If reconciliation fails partway through, you still want to write the condition that describes what failed. Without the defer, a mid-loop error returns before status is updated and the resource’s status becomes stale.

Finalizers and Cleanup

Finalizers block deletion until the controller explicitly removes them. This is how you run cleanup logic before a resource disappears.

const finalizerName = "postgres.example.com/cleanup"

func (r *PostgresClusterReconciler) handleDeletion(
    ctx context.Context,
    cluster *v1alpha1.PostgresCluster,
) (ctrl.Result, error) {
    if !controllerutil.ContainsFinalizer(cluster, finalizerName) {
        return ctrl.Result{}, nil
    }

    log := log.FromContext(ctx)
    log.Info("running cleanup before deletion")

    // Your cleanup: delete backups, deregister from external systems, etc.
    if err := r.deleteExternalResources(ctx, cluster); err != nil {
        // Do not remove the finalizer. Kubernetes will retry deletion.
        return ctrl.Result{}, err
    }

    // Cleanup succeeded. Remove the finalizer. Kubernetes will proceed
    // with actual deletion once all finalizers are removed.
    controllerutil.RemoveFinalizer(cluster, finalizerName)
    if err := r.Update(ctx, cluster); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

One pitfall: if deleteExternalResources panics or the controller crashes mid-cleanup, the finalizer stays. The next reconciliation will re-run cleanup. Your cleanup logic must be idempotent. Deleting a resource that is already gone must not return an error.

Another pitfall: orphaned finalizers. If you rename or remove a finalizer in a new controller version, any resources with the old finalizer name will be stuck in Terminating forever. Either migrate finalizers in a new controller version, or write a migration job that clears old finalizers.

Status Subresource Management

The status subresource is your contract with the outside world. GitOps tools, monitoring systems, and other controllers read it. Keep it honest.

func (r *PostgresClusterReconciler) setCondition(
    cluster *v1alpha1.PostgresCluster,
    conditionType string,
    status metav1.ConditionStatus,
    reason, message string,
) {
    meta.SetStatusCondition(&cluster.Status.Conditions, metav1.Condition{
        Type:               conditionType,
        Status:             status,
        Reason:             reason,
        Message:            message,
        ObservedGeneration: cluster.Generation,
    })
}

meta.SetStatusCondition handles the LastTransitionTime update correctly: it only changes the timestamp if the status actually changed. If you set ConditionTrue when it was already ConditionTrue, the timestamp stays. This prevents misleading “flapping” in status history.

A common mistake is calling r.Update(ctx, cluster) instead of r.Status().Update(ctx, cluster). The first updates the whole object (including spec), which can overwrite user changes and will conflict with the API server’s optimistic locking. The second only updates the status subresource.

Leader Election for HA Operators

Running a single controller replica is a single point of failure. Running multiple without coordination causes split-brain: two controllers reconciling the same resource simultaneously can produce conflicting patches.

Leader election via a Kubernetes lease solves this. Controller-runtime handles it at startup:

// cmd/main.go

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme: scheme,
    LeaderElection:          true,
    LeaderElectionID:        "postgres-operator.example.com",
    LeaderElectionNamespace: "postgres-operator-system",
    // How long a leader can miss renewing the lease before it is considered lost.
    LeaseDuration: ptr.To(15 * time.Second),
    // How frequently the leader renews its lease.
    RenewDeadline: ptr.To(10 * time.Second),
    // How frequently followers try to acquire the lease.
    RetryPeriod: ptr.To(2 * time.Second),
})

With leader election, only one instance runs the reconciliation loop at a time. When the leader dies, a follower acquires the lease within LeaseDuration and takes over. The tradeoff is that during the gap, no reconciliation happens. For most operators, 15 seconds of inactivity is acceptable. For latency-sensitive operators, tune LeaseDuration down, but be aware that aggressive values cause unnecessary leader churn when the network is briefly degraded.

Watching Related Resources

A controller often needs to react when a dependent resource changes. If a StatefulSet’s pod count changes, the operator should reconcile the parent PostgresCluster.

func (r *PostgresClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&v1alpha1.PostgresCluster{}).
        Owns(&appsv1.StatefulSet{}).
        Owns(&corev1.Service{}).
        // Watch ConfigMaps not owned by this controller.
        Watches(
            &corev1.ConfigMap{},
            handler.EnqueueRequestsFromMapFunc(r.findClustersForConfigMap),
            builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}),
        ).
        Complete(r)
}

func (r *PostgresClusterReconciler) findClustersForConfigMap(
    ctx context.Context,
    obj client.Object,
) []reconcile.Request {
    // Return the PostgresCluster that references this ConfigMap.
    cm := obj.(*corev1.ConfigMap)
    clusterName, ok := cm.Labels["postgres.example.com/cluster"]
    if !ok {
        return nil
    }
    return []reconcile.Request{
        {NamespacedName: types.NamespacedName{
            Name:      clusterName,
            Namespace: cm.Namespace,
        }},
    }
}

.Owns() automatically sets up watches for resources with owner references pointing to the parent. .Watches() with a MapFunc handles the general case: any resource change can be mapped to zero or more reconcile requests.

Operator Frameworks: Kubebuilder vs Operator SDK

Both kubebuilder and Operator SDK are scaffolding tools that generate boilerplate and manage the build pipeline. Under the hood, both use controller-runtime.

Kubebuilder is the upstream project from the Kubernetes sig-controller-tools group. It generates the project structure, Makefile targets, CRD manifests from marker comments, and RBAC rules.

Operator SDK is built on kubebuilder but adds the Operator Lifecycle Manager (OLM) integration, bundle format tooling, and support for Ansible and Helm operators (for teams that do not want to write Go).

For new Go operators, kubebuilder’s scaffolding is generally simpler and closer to the upstream conventions. Use Operator SDK if you need OLM bundle support or are running on OpenShift.

Neither framework locks you in at the code level. The generated controller code uses controller-runtime directly. You can always swap tooling without rewriting business logic.

Tradeoffs in Common Design Decisions

Decision	Option A	Option B	When to prefer A
Reconcile frequency	Event-driven only	Fixed periodic re-sync	Prefer event-driven; add periodic re-sync only for external state you cannot watch
Status update	Deferred at end of Reconcile	Inline at each step	Deferred is simpler; inline is better for long-running reconciliation where intermediate progress matters
CRD versioning	Single version with +optional fields	Multiple versions with conversion webhooks	Single version until you need a breaking schema change
Watch scope	All namespaces	Specific namespace	Cluster-wide when the operator manages infrastructure; namespace-scoped when tenants own their CRs
Finalizer strategy	One finalizer per resource	Multiple finalizers per stage	One is simpler; multiple only if different subsystems need independent cleanup acknowledgment
Error handling	Return error (exponential backoff)	Return Result with RequeueAfter	Return error for transient failures; RequeueAfter for polling external state

Production Pitfalls

Thundering herd on restart. When an operator restarts, controller-runtime enqueues all resources in the watch cache for reconciliation. If you have 2,000 PostgresClusters, they all reconcile concurrently (up to the worker pool limit). The API server gets hammered and the etcd write rate spikes. Set MaxConcurrentReconciles conservatively (8-16 for most workloads) and verify that your reconcile function completes quickly. Avoid synchronous external API calls in the critical path.

Watch bookmark performance. By default, controller-runtime uses a ListWatch with ResourceVersion: "" which starts a full list on every cache resync. For large clusters with many objects, this is expensive. Enable watch bookmarks by setting WatchBookmarkEnabled: true in the cache options (controller-runtime 0.17+). This allows the API server to send a bookmark event instead of a full relist, dramatically reducing load during normal operation.

Status update loops. If your controller reads status and sets a condition unconditionally, it generates an update event, which triggers another reconciliation, which reads status, and so on. This burns API server bandwidth and makes logs unreadable. Use meta.SetStatusCondition, which only updates LastTransitionTime when the status changes, and add a generation check: only update status if cluster.Generation != cluster.Status.ObservedGeneration or a real state change occurred.

Optimistic locking conflicts. The Kubernetes API server uses resourceVersion for optimistic locking. If your reconciler fetches a resource, another actor modifies it, and then your reconciler tries to update it, you get a 409 Conflict. Controller-runtime handles this by requeuing automatically when you return the error. The mistake is swallowing the error. Always propagate 409s upward.

Owner reference cross-namespace. Owner references are namespace-scoped. You cannot set an owner reference from a resource in namespace A to a resource in namespace B. If your operator creates cluster-scoped resources (ClusterRole, CRD) owned by a namespaced CR, those owner references will be silently ignored by the garbage collector. Manage cluster-scoped resource cleanup explicitly in your finalizer.

Slow reconciliation blocking the work queue. All resources sharing a queue are processed sequentially per key but the keys are processed in parallel up to MaxConcurrentReconciles. If one PostgresCluster takes 5 minutes to reconcile (waiting for a pod to start, for example), it does not block others. However, if your reconcile function calls an external API synchronously with no timeout, it can block a worker indefinitely. Always use context.WithTimeout for external calls, and pass the context from the Reconcile function.

Observability

Expose Prometheus metrics. Controller-runtime registers a set of default metrics at the /metrics endpoint: reconcile duration, queue depth, work queue adds, work queue retries. These are the first metrics to add to your runbook.

Add custom metrics for domain-specific signals:

var (
    clustersReady = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "postgres_operator_clusters_ready",
        Help: "Number of PostgresCluster resources in Ready phase",
    }, []string{"namespace"})

    reconcileErrors = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "postgres_operator_reconcile_errors_total",
        Help: "Total reconciliation errors by error type",
    }, []string{"resource", "error_type"})
)

func init() {
    metrics.Registry.MustRegister(clustersReady, reconcileErrors)
}

Track reconcile_errors_total by error type. Transient network errors look different from persistent misconfiguration. An alert on rate(reconcile_errors_total{error_type="InternalError"}[5m]) > 0 catches real problems without being noisy.

What the Operator Pattern Actually Gives You

Running Kubernetes automation as an operator versus a script is not just an architectural preference. It gives you crash recovery for free (the reconciliation loop replays on restart), optimistic concurrency handled by the API server (no distributed locks), a standard status surface for monitoring and GitOps tools, and a natural integration point for admission webhooks when you need to validate or mutate resources before they are committed.

The cost is real: you write Go, you manage CRD schema versions, you think carefully about finalizer lifecycle and leader election. For simple tasks, a Job or CronJob is probably enough. For anything that needs to own cluster resources across its lifetime, track external state, or handle partial failures gracefully, the operator pattern is the right abstraction.