Skip to content

AWS Updated 2025-08-30

appspec.yml anatomy for EC2 — every hook, every order, every runas that bites

The full appspec.yml reference for EC2 deploys — hook ordering, runas users, the hooks people skip and regret, and the file_exists_behavior that overwrites a config you needed.

An appspec.yml is 30 lines of YAML that the team writes once and rarely revisits — until a deployment drops a file in /etc/ that should have been in /opt/, or skips ApplicationStop and leaves two copies of the service running, or runs BeforeInstall as the app user when it needed root to install a system package. The CodeDeploy agent’s hook execution order is documented, but the implications of each hook — who runs as what, what gets overwritten, what gets left behind — are the kind of thing teams figure out from incidents. Here is the reference, organized by failure mode.

What’s happening

The CodeDeploy agent reads appspec.yml from the root of the deployment bundle. The file defines three things: which files get copied where, what permissions they end up with, and which scripts run at each lifecycle event. Lifecycle events fire in a fixed order, and the agent runs each event’s scripts sequentially with the configured timeout and runas user.

The lifecycle events for an EC2 in-place deployment, in order:

  1. ApplicationStop — the previous revision is still installed. Stop the running service before files get overwritten.
  2. DownloadBundle — agent-managed, no user hook. The agent pulls the revision from S3 or GitHub.
  3. BeforeInstall — the previous files are still on disk; the new bundle is staged but not yet copied. Run pre-install steps here (package installs, directory creation).
  4. Install — agent-managed, no user hook. The agent copies the bundle’s files block to the target locations per file_exists_behavior.
  5. AfterInstall — the new files are on disk. Run post-install steps (chmod, chown, config templating, asset compilation).
  6. ApplicationStart — start the service.
  7. ValidateService — verify the service is actually serving. Failure here triggers rollback in a deployment group with auto-rollback-configuration set.

For blue/green on EC2, the order is the same but everything runs on the green instances before any traffic shift. For ECS and Lambda, the hooks are different — BeforeAllowTraffic and AfterAllowTraffic for traffic-shift events — and they run as Lambda functions, not as scripts on the instance.

The hooks people skip and regret:

  • ApplicationStop — skipped on the assumption that “the service will get killed when the new files overwrite the old”. This is sometimes true and sometimes not. A Node.js service whose node_modules is overwritten while the process is running will keep running with stale modules cached in memory and crash on the next garbage collection. Stop the service explicitly.
  • ValidateService — skipped because “the smoke test in CI is enough”. CI smoke tests run in a staging environment with different env vars, different secrets, and different network paths than production. ValidateService runs against the just-installed binary on the actual instance with the actual config. It is the only test that proves this particular instance is healthy.
  • BeforeBlockTraffic and AfterBlockTraffic (in-place deploys only) — used for connection draining and lb hooks. Skipped because most teams use ALB deregistration delay instead. Both work, but if you have long-running websocket connections, BeforeBlockTraffic is where you send the graceful-shutdown signal before draining.

How to verify

The four checks that catch the common mistakes.

Verify the agent picks up the file. A missing appspec.yml at the bundle root will fail the deployment with a clear error; a malformed YAML file fails less clearly. Validate locally before pushing:

yamllint appspec.yml

For deployment to S3, list the bundle and confirm the file is at the root:

aws s3 cp s3://deploy-bucket/builds/saas-api-build-127.zip /tmp/
unzip -l /tmp/saas-api-build-127.zip | head

appspec.yml should appear as a top-level entry, not under a subdirectory.

Verify the runas user exists on every instance. A runas: app hook on an instance where the app user does not exist will fail the hook with User app does not exist. The user creation happens in BeforeInstall, but BeforeInstall itself runs as root — so the user is reliably present by the time later hooks need it. The trap is when ApplicationStop is set to runas: app and the previous revision never created the user (a fresh blue/green green instance has no app user yet). Set ApplicationStop to runas: root to avoid the dependency.

Verify file_exists_behavior does not silently overwrite configuration. The default DISALLOW fails the deployment if the target file exists. OVERWRITE replaces it. RETAIN keeps the existing file. The trap is when the bundle includes a config template that should be templated per-environment, but the bundle ships a default config — OVERWRITE will replace the production config with the default on every deploy. The fix is either to not ship the config in the bundle (and template it in AfterInstall) or use RETAIN.

Verify permissions land correctly. The permissions block runs after the file copy. If a hook in AfterInstall modifies file ownership, the permissions block does not re-run. Order: files (copied as root), permissions (applied per spec), AfterInstall (last word). A chmod 600 /etc/app/secret in AfterInstall overrides anything in permissions.

The fix

The reference appspec.yml for a typical Node.js or Python service on EC2, with every hook present and every runas explicit:

version: 0.0
os: linux
files:
  - source: /app
    destination: /opt/saas-api/current
  - source: /config
    destination: /etc/saas-api
file_exists_behavior: OVERWRITE
permissions:
  - object: /opt/saas-api/current
    pattern: "**"
    owner: app
    group: app
    mode: 0644
    type:
      - file
  - object: /opt/saas-api/current/bin
    pattern: "**"
    owner: app
    group: app
    mode: 0755
    type:
      - file
  - object: /etc/saas-api
    pattern: "**"
    owner: root
    group: app
    mode: 0640
    type:
      - file
hooks:
  ApplicationStop:
    - location: scripts/stop.sh
      timeout: 60
      runas: root
  BeforeInstall:
    - location: scripts/install_deps.sh
      timeout: 300
      runas: root
    - location: scripts/create_user.sh
      timeout: 30
      runas: root
  AfterInstall:
    - location: scripts/migrate.sh
      timeout: 600
      runas: app
    - location: scripts/build_assets.sh
      timeout: 300
      runas: app
  ApplicationStart:
    - location: scripts/start.sh
      timeout: 60
      runas: root
  ValidateService:
    - location: scripts/validate.sh
      timeout: 120
      runas: app

The matching scripts:

scripts/stop.sh — systemctl stop is reliable; sending SIGTERM directly is fragile because the agent’s pid lookup may miss workers spawned by the main process. The hook should exit 0 even if the service is not running (a fresh blue/green instance has no service to stop):

#!/usr/bin/env bash
set -e
systemctl stop saas-api.service || true
exit 0

scripts/create_user.sh — idempotent user creation:

#!/usr/bin/env bash
set -e
if ! id -u app >/dev/null 2>&1; then
  useradd --system --no-create-home --shell /usr/sbin/nologin app
fi

scripts/start.sh — systemctl start with a final status check before declaring success:

#!/usr/bin/env bash
set -e
systemctl daemon-reload
systemctl enable saas-api.service
systemctl start saas-api.service
sleep 2
systemctl is-active saas-api.service

scripts/validate.sh — wait up to 90 seconds for the local health endpoint:

#!/usr/bin/env bash
set -euo pipefail
for i in $(seq 1 18); do
  if curl -fsS http://127.0.0.1:8080/healthz | grep -q '"status":"ok"'; then
    exit 0
  fi
  sleep 5
done
exit 1

Three rules we hold for appspec.yml:

Every hook gets an explicit runas. The agent defaults to root, but writing runas: root makes intent visible. A future reviewer should not have to read the agent docs to know who ran migrate.sh.

ApplicationStop runs as root, not as the application user. The trap is the first blue/green deploy where the app user does not exist yet. Root always exists.

Hook timeout includes the slow path. A database migration that runs in AfterInstall and takes 4 minutes on a large dataset will fail a 180-second timeout 5% of the time. Size the timeout for the 99th percentile run, not the median. Migration hooks routinely need 600+ seconds.

The appspec.yml is the smallest file in the deploy bundle and the one that decides whether a deploy succeeds. Cross-reference the blue/green tutorial for the deployment group config that calls these hooks.

Stack Harbor runs CodeDeploy + CodePipeline on production EC2 / ECS / Lambda fleets as part of Managed AWS and ongoing Managed Cloud Ops. If your deploys still take a maintenance window, start with an infrastructure consult.