GitOps in Practice: Declarative Infrastructure, Drift Detection, and Reconciliation Loops

Most teams discover GitOps by accident. Someone reads a blog post, sets up ArgoCD, and calls it GitOps because the YAML lives in a repository. That works until the first on-call incident where the cluster state and the repo diverge in ways nobody fully understands. Then the question becomes: what is GitOps actually supposed to do, and why did it fail to prevent this?

The short answer: GitOps is not about storing config in Git. It is about making Git the authoritative source of desired state and automating the enforcement of that state continuously. The key word is “continuously.” If reconciliation only runs on commit, you do not have drift detection. You have a deployment trigger.

The Reconciliation Loop

The core of any GitOps system is the reconciliation loop. It runs on a fixed interval (typically 1-5 minutes) and does three things:

1. Observe: Read the current state of the cluster
2. Diff: Compare it against the desired state in Git
3. Act: Apply changes to close the gap

This is the control loop pattern from control theory. What makes GitOps different from a cron-driven kubectl apply is that the loop is stateful, bidirectional, and (in mature implementations) aware of health checks before and after each change.

Here is a simplified version of the reconciliation logic in TypeScript, which represents what operators like Flux and ArgoCD implement at their core:

interface ClusterState {
  resources: Map<string, KubernetesResource>;
  observedAt: Date;
}

interface DesiredState {
  resources: Map<string, KubernetesResource>;
  revision: string; // git commit SHA
}

interface ReconcileResult {
  applied: string[];
  deleted: string[];
  skipped: string[];
  errors: ReconcileError[];
}

async function reconcile(
  desired: DesiredState,
  observed: ClusterState
): Promise<ReconcileResult> {
  const result: ReconcileResult = {
    applied: [],
    deleted: [],
    skipped: [],
    errors: [],
  };

  for (const [key, desiredResource] of desired.resources) {
    const currentResource = observed.resources.get(key);

    if (!currentResource) {
      // Resource missing from cluster: create it
      try {
        await applyResource(desiredResource);
        result.applied.push(key);
      } catch (err) {
        result.errors.push({ resource: key, error: err as Error });
      }
      continue;
    }

    const patch = computePatch(currentResource, desiredResource);
    if (patch === null) {
      result.skipped.push(key);
      continue;
    }

    // Resource exists but differs: update it
    try {
      await patchResource(key, patch);
      result.applied.push(key);
    } catch (err) {
      result.errors.push({ resource: key, error: err as Error });
    }
  }

  // Resources in cluster not in desired state: prune them
  for (const [key] of observed.resources) {
    if (!desired.resources.has(key) && isManagedByGitOps(key)) {
      try {
        await deleteResource(key);
        result.deleted.push(key);
      } catch (err) {
        result.errors.push({ resource: key, error: err as Error });
      }
    }
  }

  return result;
}

The isManagedByGitOps check is important. Without it, the loop will happily delete resources that were intentionally created outside of Git (debugging tools, temporary jobs, manually applied fixes). GitOps operators use labels or annotations for this.

Pull-Based vs Push-Based Deployment

GitOps practitioners distinguish between two models:

Push-based: A CI pipeline runs, builds artifacts, and pushes changes to the cluster using kubectl apply or helm upgrade. GitHub Actions deploying to Kubernetes via kubeconfig is the common example.

Pull-based: An agent running inside the cluster watches the Git repository and pulls changes on its own reconciliation interval. Flux and ArgoCD are both pull-based.

The architectural difference has real security implications. Push-based deployment requires your CI system to have credentials that can modify the cluster. Those credentials need to be stored somewhere (usually as CI secrets) and rotated. If your CI system is compromised, the attacker has cluster access.

Pull-based eliminates outbound credentials entirely. The cluster agent needs read access to the Git repository. Nothing outside the cluster needs cluster credentials. This is a meaningful reduction in attack surface.

The tradeoff is operational complexity. Debugging a failed push deployment is straightforward: look at the CI logs. Debugging a failed pull-based reconciliation requires knowing where to look inside the cluster itself.

Flux vs ArgoCD

Both are mature, production-ready GitOps operators. The choice between them is mostly about operational model, not capability.

Dimension	Flux	ArgoCD
Architecture	Controllers per concern (source, kustomize, helm)	Monolithic application server
UI	Minimal, optional	Full-featured dashboard
Configuration surface	Everything is a CRD in Git	Mix of CRDs and ArgoCD UI state
Multi-tenancy	Namespace isolation via RBAC	Projects and AppProjects
Notification system	Pluggable (Slack, PagerDuty, etc.)	Built-in with webhook support
Bootstrapping	Flux bootstrap CLI, GitOps-native	ArgoCD install + manual App creation
Learning curve	Steeper (more primitives)	Gentler (UI drives understanding)
Secret management	Native SOPS decryption	Delegates to external solutions
Best for	Platform teams, multi-tenant, Helm-heavy	Single-cluster, UI-driven teams

Flux’s modular design is an advantage if you want to compose your own GitOps stack or manage dozens of clusters with different configurations. ArgoCD’s UI is genuinely useful for teams that want visibility without querying CRDs directly.

A Flux HelmRelease looks like this:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: podinfo
      version: ">=6.0.0"
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
  values:
    replicaCount: 2
    resources:
      limits:
        cpu: 200m
        memory: 256Mi
  upgrade:
    remediation:
      retries: 3
  rollback:
    cleanupOnFail: true

The equivalent ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: podinfo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops-repo
    targetRevision: HEAD
    path: apps/podinfo
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Note selfHeal: true in ArgoCD’s sync policy. This is the drift correction flag. Without it, ArgoCD will detect drift but not automatically remediate it.

Handling Secrets in GitOps

Secrets are the hardest part of GitOps. You cannot store raw secrets in Git. The common solutions are:

SOPS (Secrets OPerationS): Encrypts secret values in YAML files using KMS keys (AWS KMS, GCP KMS, Azure Key Vault) or age keys. The encrypted files live in Git. Flux has native SOPS decryption support. The operator decrypts at reconcile time using a key it holds in a Kubernetes secret or via IRSA/Workload Identity.

# Encrypted with: sops --encrypt --age <age-public-key> secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: default
sops:
  age:
    - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
  lastmodified: "2026-04-11T10:00:00Z"
  version: 3.7.3
data:
  password: ENC[AES256_GCM,data:abc123...,tag:xyz==,type:str]

Sealed Secrets: A Kubernetes controller that watches SealedSecret CRDs and decrypts them into Secret objects using a cluster-side certificate. The sealed form is safe to commit to Git. The limitation: you need the cluster’s public certificate to seal new secrets, which ties the encryption to a specific cluster. Key rotation requires re-sealing everything.

External Secrets Operator (ESO): Reads secrets from an external store (AWS Secrets Manager, Vault, GCP Secret Manager) and creates Kubernetes Secrets from them. The CRD in Git describes where to find the secret, not the secret itself. ESO is the cleanest model for teams that already have a secrets management platform.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/database/credentials
        property: password

ESO is the right default for most production environments. It centralizes secret rotation (update in Secrets Manager, ESO syncs within the refresh interval), and the CRDs in Git contain no sensitive material.

Multi-Environment Promotion with Kustomize Overlays

A standard pattern for managing multiple environments is a Kustomize base with per-environment overlays:

k8s/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    staging/
      kustomization.yaml
      replica-patch.yaml
      values.yaml
    production/
      kustomization.yaml
      replica-patch.yaml
      values.yaml

The base kustomization.yaml defines shared resources. Each overlay patches what differs:

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - path: replica-patch.yaml
images:
  - name: ghcr.io/org/app
    newTag: "1.4.2" # updated by CI after image build
namePrefix: prod-
namespace: production

# overlays/production/replica-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 5
  template:
    spec:
      containers:
        - name: app
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi

Promotion from staging to production is a pull request that updates the image tag in the production overlay. The CI system builds an image, pushes it to a registry, and opens a PR (or commits directly to a branch) with the new tag. Flux or ArgoCD detects the change and reconciles.

This is a concrete workflow you can implement with a TypeScript script running in CI:

import { execSync } from "child_process";
import { readFileSync, writeFileSync } from "fs";

interface PromotionConfig {
  environment: "staging" | "production";
  imageTag: string;
  kustomizationPath: string;
}

function promoteImage(config: PromotionConfig): void {
  const kustomizationPath = `${config.kustomizationPath}/kustomization.yaml`;
  const content = readFileSync(kustomizationPath, "utf-8");

  // Update image tag using kustomize edit
  execSync(
    `kustomize edit set image ghcr.io/org/app:${config.imageTag}`,
    { cwd: config.kustomizationPath }
  );

  // Stage the change
  execSync(`git add ${kustomizationPath}`);
  execSync(
    `git commit -m "chore: promote ${config.imageTag} to ${config.environment}"`,
    {
      env: {
        ...process.env,
        GIT_AUTHOR_NAME: "ci-bot",
        GIT_AUTHOR_EMAIL: "ci@org.com",
        GIT_COMMITTER_NAME: "ci-bot",
        GIT_COMMITTER_EMAIL: "ci@org.com",
      },
    }
  );
}

// Usage in CI:
promoteImage({
  environment: "production",
  imageTag: process.env.IMAGE_TAG!,
  kustomizationPath: "k8s/overlays/production",
});

Drift Detection and Alerting

Drift happens. A manual kubectl edit during an incident. An admission webhook that mutates resources. A Helm chart that applies defaults differently than expected.

The reconciliation loop handles most drift automatically if selfHeal (ArgoCD) or prune: true (Flux) are enabled. But there are cases you want to detect and alert on rather than auto-remediate:

• Drift in production that was intentionally applied during an incident and not yet captured in Git
• Drift in resources that the GitOps operator does not manage
• Repeated reconciliation failures (indicating a broken desired state in Git)

Flux surfaces this through Kubernetes events and custom metric endpoints. You can scrape the Flux metrics endpoint and alert on gotk_reconcile_condition{type="Ready",status="False"}.

A lightweight TypeScript tool to audit drift across namespaces via the Kubernetes API:

import * as k8s from "@kubernetes/client-node";

interface DriftReport {
  namespace: string;
  resource: string;
  kind: string;
  driftedFields: string[];
  detectedAt: Date;
}

async function detectDrift(
  namespaces: string[]
): Promise<DriftReport[]> {
  const kc = new k8s.KubeConfig();
  kc.loadFromDefault();
  const customObjectsApi = kc.makeApiClient(k8s.CustomObjectsApi);

  const reports: DriftReport[] = [];

  for (const ns of namespaces) {
    const kustomizations = await customObjectsApi.listNamespacedCustomObject(
      "kustomize.toolkit.fluxcd.io",
      "v1",
      ns,
      "kustomizations"
    );

    for (const ks of (kustomizations.body as any).items) {
      const conditions: any[] = ks.status?.conditions ?? [];
      const readyCondition = conditions.find(
        (c: any) => c.type === "Ready"
      );

      if (readyCondition?.status === "False") {
        reports.push({
          namespace: ns,
          resource: ks.metadata.name,
          kind: "Kustomization",
          driftedFields: [readyCondition.message],
          detectedAt: new Date(),
        });
      }
    }
  }

  return reports;
}

async function main() {
  const driftReports = await detectDrift(["default", "production", "staging"]);

  if (driftReports.length > 0) {
    console.error(`Drift detected in ${driftReports.length} resource(s):`);
    for (const report of driftReports) {
      console.error(
        `  ${report.kind}/${report.resource} in ${report.namespace}: ${report.driftedFields.join(", ")}`
      );
    }
    process.exit(1);
  }

  console.log("No drift detected.");
}

main().catch(console.error);

This script is runnable as a scheduled job or in CI as a smoke test after deployments.

Operational Reality

What actually breaks in production GitOps:

CRD version skew: The operator version and the CRD API version in your manifests drift apart after upgrades. Flux and ArgoCD upgrade their CRD schemas between minor versions. Upgrading the operator without updating the manifests (or vice versa) causes reconciliation to stop silently.

Prune deletes resources you forgot to annotate: If you enable pruning and deploy a resource outside of GitOps without the GitOps management annotation, the next reconciliation will delete it. This is surprising when it happens to a shared resource like a cluster-wide RBAC binding.

Secrets rotation lag: External Secrets Operator refreshes on a refreshInterval. If your secret rotates in AWS Secrets Manager and your ESO interval is 1 hour, you have up to 60 minutes of lag before the cluster secret updates. Applications that cache secrets at startup need a pod restart to pick up the new value.

Git repository becomes a bottleneck: With many teams pushing to the same GitOps repo, merge conflicts in auto-committed image tags become a real problem. The standard mitigation is to partition the repo: one repo per team, or a directory-per-team structure with automated PR workflows that use rebase instead of merge commits.

Helm values overrides across environments: When the same Helm chart is used across three environments with slightly different values files, the diff between environments becomes hard to reason about. A value set in values.yaml can be silently overridden in an environment-specific overlay in a non-obvious way. Document which values are allowed to differ per environment and enforce it in code review.

Reconciliation noise in alerting: Alert on reconcile condition = false for more than 5 minutes, not on every failed attempt. Transient network errors cause reconciliation failures that resolve on the next loop. Alerting on every failure generates noise that trains engineers to ignore the channel.

Tradeoffs

Concern	Auto-remediation on	Auto-remediation off
Drift recovery	Automatic, fast	Manual, requires awareness
Incident response	Risk: remediation undoes a live fix	Safer: manual edits survive until next commit
Audit trail	Git log is always authoritative	Divergence between Git and cluster state
Operator trust required	High: operator must be correct	Lower: human reviews before apply
Secret sync lag	Bounded by refreshInterval	Same
Suitable for	Stable, well-tested desired state	Environments with frequent manual intervention

Secrets approach	Security	Operational cost	Rotation UX
SOPS + age	High (key in cluster)	Medium (encrypt/decrypt tooling)	Re-encrypt on rotation
Sealed Secrets	Medium (cluster-bound cert)	Low (kubeseal CLI)	Re-seal on rotation
External Secrets Operator	High (delegated to vault)	Higher (ESO + store setup)	Automatic on next refresh
Raw secrets in Git	None	Zero	Not applicable

The operational recommendation: use External Secrets Operator if you have a secrets management platform. Use SOPS with age keys if you want everything self-contained and offline-signable. Sealed Secrets is a reasonable choice for a single cluster where cluster cert management is not a concern.

Closing

GitOps done right is continuous enforcement, not one-time deployment. The reconciliation loop is the invariant: whatever state the cluster is in, the operator will converge it toward the desired state in Git. That property is what gives you drift detection, audit trails, and confidence in your production state. The operational cost is real: secret management, multi-environment promotion, and prune configuration all require deliberate design. The teams that find GitOps frustrating are usually the ones who configured the operator but skipped that design work.