Skip to content

Argo CD GitOps application pattern — App-of-Apps, sync waves, and drift handling

A working Argo CD pattern using App-of-Apps, sync waves, ApplicationSets, and automated sync with self-heal — the layout we deploy for clients running Kubernetes in production.

Argo CD makes Git the source of truth for what runs in a cluster, but the shape you give your Argo CD repo decides whether that promise holds at scale. A pile of one-off Application objects with hand-written manifests works for a demo and breaks the first time you onboard a fourth cluster. This article walks the layout we use for clients in production — App-of-Apps as the cluster bootstrap, ApplicationSet for fan-out, sync waves for ordering, and automated sync with self-heal so reality matches Git within minutes of any drift.

How to verify

kubectl get applications.argoproj.io -n argocd
kubectl get applicationsets.argoproj.io -n argocd
argocd app list -o wide
argocd app get root-app
argocd app history root-app
argocd app diff platform-cert-manager
argocd app sync platform-cert-manager --dry-run
kubectl get application platform-cert-manager -n argocd \
  -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'

The two status fields you check first are sync.status (Synced/OutOfSync) and health.status (Healthy/Progressing/Degraded). argocd app diff shows the live-vs-desired delta when sync says OutOfSync.

What’s happening

Argo CD is a reconciliation loop. The Application resource declares (source repo/path, target cluster/namespace, sync policy); the controller compares the rendered manifests to the live cluster every three minutes by default and, if syncPolicy.automated is set, applies the diff. The App-of-Apps pattern is one bootstrap Application whose source points at a directory of other Application (or ApplicationSet) manifests — so the cluster’s full inventory is itself reconciled from Git.

Sync waves order the apply within a single sync operation. An argocd.argoproj.io/sync-wave annotation on a resource tells Argo CD which batch it belongs to; wave -2 applies before wave -1 before 0. We use negative waves for CRDs and namespaces, 0 for platform services (cert-manager, ingress, metrics-server), and positive waves for tenant workloads that depend on them.

Drift handling is the third pillar. With selfHeal: true, a manual kubectl edit is reverted on the next reconcile. With prune: true, a resource deleted from Git is deleted from the cluster. Both default to off — turning them on is a deliberate choice that says “Git is the only writer.”

The procedure

  1. Lay out the repo. One repo, three top-level directories.

    bootstrap/
      root-app.yaml              # the one Application you kubectl apply by hand
    platform/
      applicationset.yaml        # generates Application per cluster
      cert-manager/
        Chart.yaml
        values-prod.yaml
      ingress-nginx/
        ...
    tenants/
      applicationset.yaml        # generates Application per tenant+cluster
      team-payments/
        Chart.yaml
        values-prod.yaml
  2. Define the root Application.

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: root-app
      namespace: argocd
      finalizers: ['resources-finalizer.argocd.argoproj.io']
    spec:
      project: default
      source:
        repoURL: git@github.com:stackharbor/argo-config.git
        targetRevision: main
        path: bootstrap
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
  3. Fan out platform components with ApplicationSet.

    apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: platform
      namespace: argocd
    spec:
      generators:
        - matrix:
            generators:
              - clusters: {}
              - list:
                  elements:
                    - { name: cert-manager,   wave: '-1' }
                    - { name: ingress-nginx,  wave: '0'  }
                    - { name: external-dns,   wave: '0'  }
                    - { name: kube-prometheus,wave: '1'  }
      template:
        metadata:
          name: 'platform-{{name}}-{{cluster.name}}'
          annotations:
            argocd.argoproj.io/sync-wave: '{{wave}}'
        spec:
          project: platform
          source:
            repoURL: git@github.com:stackharbor/argo-config.git
            targetRevision: main
            path: 'platform/{{name}}'
            helm:
              valueFiles: ['values-{{metadata.labels.env}}.yaml']
          destination:
            server: '{{server}}'
            namespace: '{{name}}'
          syncPolicy:
            automated: { prune: true, selfHeal: true }
            syncOptions: ['CreateNamespace=true', 'ApplyOutOfSyncOnly=true']

    The matrix generator multiplies clusters by components — one Argo CD instance, every cluster, every platform service, one config change to roll a fix everywhere.

  4. Order with sync waves on CRDs and namespaces. Add the annotation in the chart’s Chart.yaml or via a Helm extraDeploy:

    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: '-2'

    CRDs go in wave -2, namespaces in -1, workloads in 0 or later. Without this, Application resources race and you get no matches for kind on the first sync.

  5. Project boundaries. Define AppProject resources so tenants cannot escape their namespace or pull from arbitrary repos.

    apiVersion: argoproj.io/v1alpha1
    kind: AppProject
    metadata: { name: team-payments, namespace: argocd }
    spec:
      sourceRepos: ['git@github.com:stackharbor/argo-config.git']
      destinations:
        - { namespace: 'payments-*', server: 'https://kubernetes.default.svc' }
      clusterResourceWhitelist: []
      namespaceResourceBlacklist:
        - { group: '', kind: ResourceQuota }

Common pitfalls

  • Turning on automated.prune before the team understands that a deleted file means a deleted resource — set prune: false in non-prod for a sprint before flipping it.
  • Forgetting ServerSideApply=true and watching Argo CD fight kube-controller-manager over field ownership on resources that get patched (HPA replicas, Service ClusterIP).
  • Putting CRDs in the same chart as the controller without a sync wave — first install works because Helm installs CRDs first, second install fails because Argo CD does not.
  • Letting the App-of-Apps repo be writable by every developer; the blast radius is the entire fleet.
  • Running with one Argo CD per cluster when one Argo CD with cluster credentials would do — multiplies upgrade work without buying isolation that AppProject already provides.

In the engagements we run, the Argo CD config repo and the workload repos are separate — one is platform-owned and changes through pull request review, the other belongs to the application team. We wire the GitOps loop, the sync-wave conventions, the project boundaries, and the alerting on stuck applications so the cluster keeps converging on Git without a human watching it. See how this fits the broader operating model on our managed operations page.