Argo CD makes Git the source of truth for what runs in a cluster, but the shape you give your Argo CD repo decides whether that promise holds at scale. A pile of one-off Application objects with hand-written manifests works for a demo and breaks the first time you onboard a fourth cluster. This article walks the layout we use for clients in production — App-of-Apps as the cluster bootstrap, ApplicationSet for fan-out, sync waves for ordering, and automated sync with self-heal so reality matches Git within minutes of any drift.
How to verify
kubectl get applications.argoproj.io -n argocd
kubectl get applicationsets.argoproj.io -n argocd
argocd app list -o wide
argocd app get root-app
argocd app history root-app
argocd app diff platform-cert-manager
argocd app sync platform-cert-manager --dry-run
kubectl get application platform-cert-manager -n argocd \
-o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
The two status fields you check first are sync.status (Synced/OutOfSync) and health.status (Healthy/Progressing/Degraded). argocd app diff shows the live-vs-desired delta when sync says OutOfSync.
What’s happening
Argo CD is a reconciliation loop. The Application resource declares (source repo/path, target cluster/namespace, sync policy); the controller compares the rendered manifests to the live cluster every three minutes by default and, if syncPolicy.automated is set, applies the diff. The App-of-Apps pattern is one bootstrap Application whose source points at a directory of other Application (or ApplicationSet) manifests — so the cluster’s full inventory is itself reconciled from Git.
Sync waves order the apply within a single sync operation. An argocd.argoproj.io/sync-wave annotation on a resource tells Argo CD which batch it belongs to; wave -2 applies before wave -1 before 0. We use negative waves for CRDs and namespaces, 0 for platform services (cert-manager, ingress, metrics-server), and positive waves for tenant workloads that depend on them.
Drift handling is the third pillar. With selfHeal: true, a manual kubectl edit is reverted on the next reconcile. With prune: true, a resource deleted from Git is deleted from the cluster. Both default to off — turning them on is a deliberate choice that says “Git is the only writer.”
The procedure
-
Lay out the repo. One repo, three top-level directories.
bootstrap/ root-app.yaml # the one Application you kubectl apply by hand platform/ applicationset.yaml # generates Application per cluster cert-manager/ Chart.yaml values-prod.yaml ingress-nginx/ ... tenants/ applicationset.yaml # generates Application per tenant+cluster team-payments/ Chart.yaml values-prod.yaml -
Define the root Application.
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root-app namespace: argocd finalizers: ['resources-finalizer.argocd.argoproj.io'] spec: project: default source: repoURL: git@github.com:stackharbor/argo-config.git targetRevision: main path: bootstrap destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true -
Fan out platform components with
ApplicationSet.apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: platform namespace: argocd spec: generators: - matrix: generators: - clusters: {} - list: elements: - { name: cert-manager, wave: '-1' } - { name: ingress-nginx, wave: '0' } - { name: external-dns, wave: '0' } - { name: kube-prometheus,wave: '1' } template: metadata: name: 'platform-{{name}}-{{cluster.name}}' annotations: argocd.argoproj.io/sync-wave: '{{wave}}' spec: project: platform source: repoURL: git@github.com:stackharbor/argo-config.git targetRevision: main path: 'platform/{{name}}' helm: valueFiles: ['values-{{metadata.labels.env}}.yaml'] destination: server: '{{server}}' namespace: '{{name}}' syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: ['CreateNamespace=true', 'ApplyOutOfSyncOnly=true']The matrix generator multiplies clusters by components — one Argo CD instance, every cluster, every platform service, one config change to roll a fix everywhere.
-
Order with sync waves on CRDs and namespaces. Add the annotation in the chart’s
Chart.yamlor via a HelmextraDeploy:metadata: annotations: argocd.argoproj.io/sync-wave: '-2'CRDs go in wave
-2, namespaces in-1, workloads in0or later. Without this,Applicationresources race and you getno matches for kindon the first sync. -
Project boundaries. Define
AppProjectresources so tenants cannot escape their namespace or pull from arbitrary repos.apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: { name: team-payments, namespace: argocd } spec: sourceRepos: ['git@github.com:stackharbor/argo-config.git'] destinations: - { namespace: 'payments-*', server: 'https://kubernetes.default.svc' } clusterResourceWhitelist: [] namespaceResourceBlacklist: - { group: '', kind: ResourceQuota }
Common pitfalls
- Turning on
automated.prunebefore the team understands that a deleted file means a deleted resource — setprune: falsein non-prod for a sprint before flipping it. - Forgetting
ServerSideApply=trueand watching Argo CD fight kube-controller-manager over field ownership on resources that get patched (HPA replicas, Service ClusterIP). - Putting CRDs in the same chart as the controller without a sync wave — first install works because Helm installs CRDs first, second install fails because Argo CD does not.
- Letting the App-of-Apps repo be writable by every developer; the blast radius is the entire fleet.
- Running with one Argo CD per cluster when one Argo CD with cluster credentials would do — multiplies upgrade work without buying isolation that
AppProjectalready provides.
In the engagements we run, the Argo CD config repo and the workload repos are separate — one is platform-owned and changes through pull request review, the other belongs to the application team. We wire the GitOps loop, the sync-wave conventions, the project boundaries, and the alerting on stuck applications so the cluster keeps converging on Git without a human watching it. See how this fits the broader operating model on our managed operations page.