GitOps in Practice: Declarative Deployments with ArgoCD, Flux, and Reconciliation Loops

Most teams adopt GitOps because it sounds like a good idea. Pull-based deployments, audit trail in Git, no kubectl in CI. The pitch is clean. The reality is messier: two tools with different mental models, a repo structure that needs deliberate design, and a new class of operational problems around secret management and multi-environment promotion that you have to solve before the workflow is actually trustworthy.

This is a practical guide for teams running GitOps in production or evaluating whether to. It assumes you already know Kubernetes basics and want to understand the tradeoffs, not a quickstart tutorial.

The Four GitOps Principles (and What They Actually Mean)

The CNCF definition gives you four properties: declarative, versioned, automated, and self-healing. They are worth unpacking because each one has operational implications that shape how you structure everything else.

Declarative. Your desired state is expressed as configuration, not as imperative commands. You do not run kubectl set image. You update a YAML file and let the operator apply it. This means your cluster state should always be derivable from Git alone. If it is not, you have drift.

Versioned. Git is the source of truth. Every change to desired state is a commit with an author, a timestamp, and a parent. This is your audit log and your rollback mechanism. It also means your Git history must be clean and meaningful, not full of “fix fix fix” commits that obscure what changed.

Automated. The sync from Git to cluster is automated and does not require human invocation. A push to the target branch triggers the reconciliation loop. No one manually runs helm upgrade in staging before prod.

Self-healing. If something diverges from the desired state (manual kubectl patch, node restart, operator bug), the controller corrects it. This is the property that makes GitOps different from “we deploy from Git” and the one that causes the most friction when teams are not ready for it.

ArgoCD vs. Flux: Head-to-Head

Both tools implement the same loop: watch a Git repository, compare desired state to actual cluster state, apply the diff. They diverge in architecture, UX, and extensibility.

ArgoCD is a Kubernetes controller with a built-in UI, a REST API, and a CRD called Application that maps a Git source to a cluster destination. It is opinionated about having a central control plane. The UI is genuinely useful for debugging sync failures and visualizing resource trees. ArgoCD also has ApplicationSet, which generates Application resources from a template, covering multi-cluster and multi-environment cases without repetition.

Flux is a set of composable controllers: source-controller, kustomize-controller, helm-controller, notification-controller. There is no built-in UI (though Weave GitOps adds one). Flux is more modular and fits naturally into teams that prefer to compose their tooling rather than adopt a platform. It also has first-class support for OCI registries as artifact sources, which matters for teams moving away from raw Git sources.

The practical differences that shape the choice:

• ArgoCD has a central argocd namespace where all Application resources live. Flux spreads its CRDs across namespaces by default, which aligns better with multi-tenant cluster setups.
• ArgoCD sync policies are per-application. Flux sync intervals are per source and per kustomization, giving finer control over how frequently each component reconciles.
• ArgoCD does explicit sync (you define when and what gets synced). Flux is more continuous by default, reconciling on a configurable interval rather than only on Git events.
• ArgoCD’s RBAC model is well-documented and integrates with SSO out of the box. Flux relies on standard Kubernetes RBAC, which is more flexible but requires more upfront setup for multi-team scenarios.

Dimension	ArgoCD	Flux
UI	Built-in, feature-rich	External (Weave GitOps) or none
Architecture	Monolithic controller + API server	Composable micro-controllers
Multi-cluster	ApplicationSet, cluster secrets	Kubeconfig secrets per cluster
Secret management	Vault plugin, Sealed Secrets via pre-sync hook	Native SOPS integration, External Secrets Operator
OCI support	ArgoCD 2.6+	First-class in Flux v2
Sync model	Event-driven + manual	Interval-based + webhook
Multi-tenancy	App projects with RBAC	Namespace-scoped CRDs
Community	Larger, CNCF graduated	Smaller, CNCF graduated

Neither tool is strictly better. ArgoCD is easier to get started with if you want visibility first. Flux is easier to operate at scale if you prefer composability and avoid central control planes.

Repository Structure

The most common mistake is treating the GitOps repo as an afterthought. You clone the app repo, add a k8s/ directory, and start dropping YAML in. Three environments later, you have a tangled mess.

Two patterns work in practice:

App-of-apps (ArgoCD) or Kustomization-per-env (Flux). One Git repository contains all environment configurations. Each environment is a directory with a kustomization.yaml that references a base and applies overlays.

infra/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    dev/
      kustomization.yaml
      patch-replicas.yaml
      patch-resources.yaml
    staging/
      kustomization.yaml
      patch-replicas.yaml
    prod/
      kustomization.yaml
      patch-replicas.yaml
      patch-hpa.yaml

The base holds the canonical resource definitions. Overlays use patchesStrategicMerge or patches to modify specific fields per environment. Image tags are the most common overlay target.

Monorepo vs. separate config repo. Keeping application code and Kubernetes configs in the same repo is simpler for small teams. Separating them becomes necessary when you want to promote a version across environments without triggering the full CI build pipeline, or when you have multiple services sharing infrastructure config.

For the separate-repo pattern, the app CI pipeline updates the image tag in the config repo on successful build. ArgoCD or Flux detects the change and reconciles. This decoupling is important: deployment is not automatic on code push, it is automatic on config push, which gives you a deliberate promotion step.

Kustomize Overlays in Practice

A minimal ArgoCD Application pointing at a Kustomize overlay:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-service-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/infra
    targetRevision: main
    path: apps/payments-service/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The overlay’s kustomization.yaml looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: payments
resources:
  - ../../base
images:
  - name: your-registry/payments-service
    newTag: "a3f9b21"
patches:
  - path: patch-replicas.yaml
  - path: patch-resources.yaml

prune: true tells ArgoCD to delete resources that exist in the cluster but not in Git. selfHeal: true enables the self-healing loop: if someone manually changes a resource, ArgoCD reverts it on the next sync cycle. Both are off by default. You want both on in production, but enabling them mid-flight on a cluster with manual configuration will cause unpleasant surprises.

Helm Releases in GitOps

Flux has HelmRelease, a CRD that manages a Helm chart lifecycle declaratively. ArgoCD supports Helm as a source type with value overrides in the Application spec.

A Flux HelmRelease for a dependency (say, a PostgreSQL operator):

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: cnpg
  namespace: cnpg-system
spec:
  interval: 10m
  chart:
    spec:
      chart: cloudnative-pg
      version: "0.20.x"
      sourceRef:
        kind: HelmRepository
        name: cnpg-repo
        namespace: flux-system
  values:
    replicaCount: 1
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true

The interval controls how often Flux checks for chart updates within the version constraint. remediation.retries tells Flux to attempt rollback on failed upgrades before giving up. Without remediation configured, a failed upgrade leaves the release in a broken state and stops reconciliation for that resource.

Drift Detection

Drift happens. Someone applies a hotfix directly to prod at 2am. An admission webhook mutates a resource in a way the controller did not expect. A node gets replaced and the kubelet reports slightly different metadata.

ArgoCD surfaces drift in the UI as OutOfSync with a diff view showing what changed. Flux exposes it through flux get all and through the Ready condition on each resource. Both tools emit Kubernetes events and expose Prometheus metrics.

The operationally useful signal is not whether drift exists (it always will, transiently) but whether drift is persistent and unresolved. Set an alert on argocd_app_info{sync_status="OutOfSync"} staying above zero for more than five minutes. For Flux, alert on gotk_reconcile_condition{type="Ready",status="False"} persisting.

Drift in immutable fields (like spec.selector on a Deployment) cannot be corrected by patching. The controller will report an error and stop. You need to delete and recreate the resource, or use syncOptions: ["Replace=true"] in ArgoCD, which does a full replace instead of a patch. Use replace sparingly: it causes a brief outage for stateful resources.

Secret Management in GitOps

Secrets are the hardest part of GitOps to get right. You cannot commit plaintext secrets to Git. You need a workflow where the desired secret state is version-controlled but the actual values are never in the repository.

Sealed Secrets is the simplest approach. A kubeseal CLI encrypts a Kubernetes Secret using a public key tied to the cluster. The resulting SealedSecret CRD is safe to commit. The in-cluster controller decrypts it using the private key and creates the actual Secret. The encrypted form is environment-specific because the key pair is per-cluster.

kubectl create secret generic db-credentials \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml > sealed-db-credentials.yaml

The limitation: if you need to rotate the cluster key, you must re-seal every secret. If the private key is lost, the secrets are unrecoverable from Git. Back up the key.

External Secrets Operator (ESO) is the production-grade approach. It pulls secrets from external stores (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault) and syncs them into Kubernetes Secret objects. The ExternalSecret CRD is what you commit to Git: it references the secret store and the key path, not the value.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

The refreshInterval determines how quickly ESO picks up rotation. For credentials that rotate automatically (RDS IAM auth, short-lived tokens), set this to 5-15 minutes. For static credentials, 1 hour is fine. ESO also supports DataFrom for pulling all keys from a single secret path, which reduces boilerplate when you have many fields.

The tradeoff between Sealed Secrets and ESO is complexity vs. operational flexibility. Sealed Secrets requires no external dependency but ties you to a cluster-specific workflow. ESO requires running a secrets manager but gives you centralized rotation, audit logs, and cross-cluster sharing.

Multi-Environment Promotion

The promotion question is: how does a change move from dev to staging to prod without manual YAML edits?

The simplest automated pattern uses CI to update image tags. After a successful build and dev deploy, your pipeline opens a PR (or directly commits with a signed bot) that bumps the image tag in the staging overlay. After staging passes, the same pattern promotes to prod. Each promotion is a discrete Git commit, which is your audit trail.

With ArgoCD ApplicationSet and a Git generator, you can drive all environments from a single template:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-service
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/your-org/infra
        revision: main
        directories:
          - path: apps/payments-service/overlays/*
  template:
    metadata:
      name: "payments-service-{{path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/infra
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "payments-{{path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: "{{path.basename != 'prod'}}"

The last line is a common pattern: auto-sync for non-prod, manual sync for prod. In practice, this is enforced differently (sync policy per environment), but the principle holds. Prod deployments should require an explicit sync trigger, even if the desired state is already in Git.

For Flux, the equivalent is a Kustomization per environment, each with its own path and interval. Promotion is just updating the image tag in the right overlay directory.

Rollback Workflows

GitOps makes rollback a git revert or a forced image tag change. The mechanics are simple. The operational question is how fast the rollback lands.

ArgoCD with auto-sync will detect the reverted commit and apply it within seconds (assuming a webhook trigger from GitHub/GitLab). Flux’s interval-based polling means you wait up to the configured interval (often 1-5 minutes) unless you trigger a reconciliation manually with flux reconcile kustomization my-app.

For production incidents, waiting is not acceptable. Keep these commands in your runbook:

# ArgoCD: force sync immediately
argocd app sync payments-service-prod --prune

# Flux: force reconciliation immediately
flux reconcile kustomization payments-service -n flux-system

# Flux: suspend auto-reconciliation for manual intervention
flux suspend kustomization payments-service -n flux-system

The suspend command is underused. When you are debugging a reconciliation loop that keeps overwriting your manual patches, suspend buys you time without disabling GitOps globally.

Rollback to a specific Git SHA:

# Flux: point to a specific revision
flux create kustomization payments-service \
  --source=GitRepository/infra \
  --path=./apps/payments-service/overlays/prod \
  --revision=a3f9b21 \
  --prune=true \
  --export | kubectl apply -f -

Production Considerations

Health checks matter more in GitOps. If your Deployment has a misconfigured readiness probe and ArgoCD shows it as Healthy, you will not catch the issue until users do. Configure ArgoCD health checks for your custom resources and verify that Healthy means your app is actually serving traffic.

Namespace hygiene. Use separate namespaces per environment per service. This gives you clean RBAC boundaries, clear resource quotas, and makes prune: true safe (it only prunes within the namespace the application manages).

Resource limits are required. Without resource limits, a reconciliation loop will over-provision a node and cause cascading evictions. GitOps makes it easy to forget this because you are focused on the deployment workflow, not the runtime behavior.

Sync waves for ordering. If your application needs a CRD to exist before a CR, or a namespace before a deployment, use ArgoCD sync waves (argocd.argoproj.io/sync-wave: "-1" on the CRD). Flux uses dependsOn in Kustomization for the same purpose. Unordered syncs against Kubernetes admission will fail in non-obvious ways.

Observability hooks. Both ArgoCD and Flux integrate with Prometheus. Track argocd_app_sync_total, argocd_app_info, and gotk_reconcile_duration_seconds. Set alerts on sync failures and long reconciliation times. A reconciliation that takes 10 minutes for a small application means something is wrong with the API server or the controller’s backoff logic.

Tradeoffs

Dimension	GitOps	Direct deploy (CI push)
Audit trail	Full Git history per change	CI logs (less durable)
Rollback speed	Revert commit, wait for sync	Re-run previous pipeline
Drift detection	Built-in, continuous	None (point-in-time)
Secret management	Requires separate solution (ESO, Sealed Secrets)	CI secrets injected at deploy time
Operator burden	New CRDs, controllers, and reconciliation loop to operate	Fewer moving parts
Multi-cluster	Well-supported via ApplicationSet or Flux multi-tenancy	Script-based, error-prone
Onboarding time	Higher (tooling, repo conventions, RBAC)	Lower
Self-healing	Yes, continuous	No

When GitOps Is Overkill

GitOps adds real operational weight. Before adopting it, be honest about whether you need what it provides.

If you have one environment and one engineer, the audit trail is in your head and the rollback is a git push. The overhead of maintaining an ArgoCD or Flux installation, a config repo, and a promotion workflow is not justified.

If your deployments are infrequent (once a week or less) and changes are small, the self-healing loop does not save you meaningful time. You will spend more time learning the tooling than the tooling saves you.

If your team is not comfortable with Kubernetes YAML and Kustomize, GitOps will not fix that. It will amplify it. You need a baseline of Kubernetes literacy before GitOps is useful.

The cases where GitOps is clearly worth it: multiple environments that need to stay in sync, teams where more than one engineer touches deployments, clusters where manual kubectl access is a compliance risk, and situations where drift (someone changes a config in prod directly) has caused incidents.

The pattern earns its complexity when the alternative is worse. For a three-person team deploying a monolith to one cluster, the alternative is almost always fine.

Closing

GitOps is a deployment contract, not just a tool. The contract is: the cluster state at any point is a function of the Git history. The tools (ArgoCD, Flux) enforce the contract. The repository structure, sync policies, and secret management patterns are how you make the contract sustainable to operate.

If you commit to the contract, you get a lot for free: audit trail, rollback, drift detection, and promotion workflows that do not require custom scripting. If you adopt the tools without the contract (auto-sync off, manual kubectl still in use, secrets in CI env vars), you get the operational burden without the benefit.

Pick one tool, design the repo structure before you start, get secret management working in one environment before promoting the pattern, and turn on self-healing only after you trust your health checks. That sequence keeps the migration tractable.