A four-node m6i.large ASG sits behind an ALB serving an internal API. The team has been doing in-place CodeDeploy rolling updates for two years, and the morning of a routine deploy one of the four boxes ends up half-way through ApplicationStart with a stale node_modules cache when the request rate spikes. The ALB cycles a chunk of traffic to the half-deployed box; the error rate jumps for ninety seconds; the on-call engineer rolls back manually by re-running the previous revision through CodeDeploy. The post-mortem conclusion is correct: in-place deploys on an ASG have no atomic shift point. The fix is to switch to BLUE_GREEN with a replacement ASG. The next paragraphs are the wiring.

What’s happening

In-place CodeDeploy on an EC2 ASG deploys to the existing instances one or two at a time. The CodeDeploy agent on each instance pulls the revision, runs the hooks, and the deployment group manages how many instances are out of rotation at once. The failure mode in the scenario above is that “out of rotation” is enforced by deregistering the instance from the target group — but the deregistration delay (default 300 seconds) and the deployment group’s OverrideAlarmConfiguration interact in ways that leave a small window where requests can hit a half-deployed box.

Blue/green eliminates the window. CodeDeploy provisions a brand-new ASG (the green) using the launch template of the existing one (the blue). The new instances boot, the agent runs ApplicationStop → BeforeInstall → AfterInstall → ApplicationStart → ValidateService on each instance in isolation. Only after every green instance passes ValidateService and registers as healthy in the ALB target group does CodeDeploy shift listener traffic from the blue target group to the green target group. The shift is a target-group swap at the ALB listener — atomic from the load balancer’s point of view.

After the shift, CodeDeploy holds the blue ASG online for a configurable duration (the “original instance termination wait time”). If you wired automatic rollback to a CloudWatch alarm — five-minute average 5xx rate, p95 latency, or a custom metric — and the alarm fires inside the wait window, the listener swaps back to blue and the green ASG is terminated. If no alarm fires, the blue ASG terminates at the end of the wait window and the deployment is done.

The three knobs that matter:

Deployment configuration — CodeDeployDefault.AllAtOnce shifts 100% in one swap; CodeDeployDefault.HalfAtATime shifts 50% then 50% after a hold; custom configs let you do 10/30/60/100. For an internal API where the team monitors error rate at five-minute granularity, AllAtOnce plus a 15-minute termination wait is the right starting point.
Termination wait — the window where the old ASG is still alive and rollback is one CloudWatch alarm away. Long enough to catch alarms based on five-minute metrics with at least two evaluation periods. 15 minutes is the floor; 30 is comfortable.
Provisioning method — COPY_AUTO_SCALING_GROUP (CodeDeploy creates a brand-new ASG cloned from the existing) is the safe default. DISCOVER_EXISTING is for cases where the green ASG is pre-provisioned by Terraform and CodeDeploy just deploys to it.

How to verify

Three things to verify before declaring the deployment group production-ready: the launch template renders a bootable instance with the agent installed, the validation hook actually validates something, and the alarms are wired correctly.

For the launch template, the failure mode is an AMI that boots fine in-place (because the agent is already installed from the previous deploy) but fails in blue/green (because the new instances need the agent baked into the AMI or installed via user-data). Verify by launching a single instance from the launch template, SSHing in, and checking the agent:

sudo service codedeploy-agent status
sudo cat /var/log/aws/codedeploy-agent/codedeploy-agent.log | tail -50

For the validation hook, the failure mode is a script that exits 0 unconditionally — which makes ValidateService a no-op and lets a broken deployment pass. The hook should hit the local application port, check a meaningful endpoint, and exit non-zero on failure:

#!/usr/bin/env bash
set -euo pipefail
for i in $(seq 1 12); do
  if curl -fsS http://127.0.0.1:8080/healthz | grep -q '"status":"ok"'; then
    exit 0
  fi
  sleep 5
done
exit 1

That hook waits up to 60 seconds for the service to come up, hits /healthz, and fails the deployment if the endpoint is missing or returning an unhealthy payload. A surface-level “is port 8080 listening” check is not enough — systemd will keep restarting a crash-looping process and the port will appear listening between restarts.

For alarms, list the deployment group’s alarmConfiguration:

aws deploy get-deployment-group \
  --application-name internal-api \
  --deployment-group-name internal-api-prod \
  --query 'deploymentGroupInfo.alarmConfiguration'

If the response is {"enabled": false, "ignorePollAlarmFailure": false, "alarms": []}, automatic rollback is wired to nothing. The alarms field needs at least one CloudWatch alarm name — typically a 5xx rate alarm on the target group and a p95 latency alarm on the listener.

The fix

The full deployment group config for BLUE_GREEN with COPY_AUTO_SCALING_GROUP, AllAtOnce traffic shift, and a 20-minute termination wait:

aws deploy create-deployment-group \
  --application-name internal-api \
  --deployment-group-name internal-api-prod \
  --service-role-arn arn:aws:iam::111122223333:role/CodeDeployServiceRole \
  --auto-scaling-groups internal-api-prod-asg \
  --deployment-style deploymentType=BLUE_GREEN,deploymentOption=WITH_TRAFFIC_CONTROL \
  --blue-green-deployment-configuration '{
    "terminateBlueInstancesOnDeploymentSuccess": {
      "action": "TERMINATE",
      "terminationWaitTimeInMinutes": 20
    },
    "deploymentReadyOption": {
      "actionOnTimeout": "STOP_DEPLOYMENT",
      "waitTimeInMinutes": 30
    },
    "greenFleetProvisioningOption": {
      "action": "COPY_AUTO_SCALING_GROUP"
    }
  }' \
  --load-balancer-info '{
    "targetGroupInfoList": [{"name": "internal-api-tg"}]
  }' \
  --auto-rollback-configuration '{
    "enabled": true,
    "events": ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]
  }' \
  --alarm-configuration '{
    "enabled": true,
    "alarms": [
      {"name": "internal-api-5xx-rate"},
      {"name": "internal-api-p95-latency"}
    ]
  }'

The matching appspec.yml lives at the root of the revision and points at the build artifact and hooks:

version: 0.0
os: linux
files:
  - source: /
    destination: /opt/internal-api
file_exists_behavior: OVERWRITE
permissions:
  - object: /opt/internal-api
    owner: app
    group: app
    mode: 0755
hooks:
  ApplicationStop:
    - location: scripts/stop.sh
      timeout: 60
      runas: root
  BeforeInstall:
    - location: scripts/before_install.sh
      timeout: 120
      runas: root
  AfterInstall:
    - location: scripts/after_install.sh
      timeout: 180
      runas: app
  ApplicationStart:
    - location: scripts/start.sh
      timeout: 60
      runas: root
  ValidateService:
    - location: scripts/validate.sh
      timeout: 90
      runas: app

Three rules we hold for blue/green on EC2 + ASG:

Never set termination wait to zero. The wait is the entire safety net — without it, blue is gone the moment green is healthy, and rollback means a fresh provisioning cycle instead of a listener flip. Twenty minutes is a reasonable default for an internal API; thirty for anything customer-facing.

Validate against a real endpoint, not a TCP check. A ValidateService hook that proves the application is serving correct responses catches more deployment regressions than one that proves the port is open.

Pre-warm the green ASG if the application has a slow start. A JVM service that takes 90 seconds to warm caches will fail ValidateService if the hook timeout is 60 seconds, even when the deployment would otherwise succeed. Either raise the timeout or have the hook block until the warm-up signal lands.

The blue/green pattern is more moving parts than in-place — a second ASG, a target group swap, the termination wait — but it is the only EC2 deployment shape that lets you roll back from a regression in under a minute. Cross-reference the appspec anatomy reference for the hook ordering details and the rollback automation runbook for the alarms.

Stack Harbor runs CodeDeploy + CodePipeline on production EC2 / ECS / Lambda fleets as part of Managed AWS and ongoing Managed Cloud Ops. If your deploys still take a maintenance window, start with an infrastructure consult.