Skip to content

AWS Updated 2025-12-25

Automatic rollback on CloudWatch alarm — wiring it once and stopping the nightly rollback loop

Wiring CodeDeploy automaticRollbackConfiguration to CloudWatch alarms is mechanical. Keeping the alarms tight enough to catch regressions but loose enough not to rollback every Tuesday at 03:00 — that is where teams go wrong.

A team wires CodeDeploy automatic rollback to a 5xx-rate CloudWatch alarm on a Friday. The deploy succeeds, the alarms stay green, the team goes to the long weekend. On Tuesday morning the on-call sees three rollbacks during the night — none of which were responding to a real deploy regression. One was a transient ALB target failure during instance refresh. One was a downstream timeout that propagated as 5xx for 90 seconds. One was a load test from the QA environment that hit production by mistake. The rollback feature is the right design; the alarm wiring needs work. Here is the runbook for getting it right the first time and the post-rollback investigation flow that keeps it tight.

What’s happening

CodeDeploy’s automaticRollbackConfiguration and alarmConfiguration are two separate fields that both need to be set for automatic rollback to work. The first defines which events trigger rollback; the second defines which CloudWatch alarms count as a trigger. Either one missing breaks the feature.

The events field accepts three values:

  • DEPLOYMENT_FAILURE — a hook script exits non-zero or ValidateService fails. Always include this.
  • DEPLOYMENT_STOP_ON_ALARM — a configured CloudWatch alarm enters ALARM state during the deployment. Include this if any alarms are wired.
  • DEPLOYMENT_STOP_ON_REQUEST — a manual aws deploy stop-deployment triggers rollback rather than just stopping. Include for operator-friendly behavior.

The alarms themselves are standard CloudWatch alarms. CodeDeploy polls their state during the deployment window — and for blue/green, also during the termination wait window if terminateBlueInstancesOnDeploymentSuccess is configured. An alarm that fires while blue is still alive triggers a listener swap back to blue.

The trap that turns the feature into a nightly rollback loop is alarm sensitivity. Three patterns cause false-positive rollbacks:

  • Single-evaluation alarms. An alarm that fires on a single one-minute datapoint exceeding the threshold is going to fire on a transient blip every other day. Two evaluation periods is the floor; three is comfortable.
  • Percentage-based alarms at low traffic. A “5xx rate > 1%” alarm trips at one error in a hundred. A service that does 30 RPS sees 1800 requests/minute — fine. A service that does 0.5 RPS sees 30 requests/minute, and a single 5xx is a 3.3% rate. Use absolute counts at low traffic.
  • Alarms shared with downstream noise. An alarm on overall ALB 5xx rate will trip on any upstream timeout — a downstream database hiccup that has nothing to do with the new deploy will still trigger rollback. Scope alarms to target-group level or the specific new task set.

How to verify

Three checks before declaring rollback automation production-ready: the alarms exist and are in OK state, the deployment group references them by name, and a forced-alarm test actually rolls back.

For alarm state:

aws cloudwatch describe-alarms \
  --alarm-names saas-api-5xx-rate saas-api-p95-latency \
  --query 'MetricAlarms[].{Name:AlarmName,State:StateValue,Reason:StateReason}'

All alarms should be OK. An alarm in INSUFFICIENT_DATA will not block a deploy, but it also will not catch regressions — fix the underlying metric collection before relying on it.

For the deployment group reference:

aws deploy get-deployment-group \
  --application-name saas-api \
  --deployment-group-name saas-api-prod \
  --query 'deploymentGroupInfo.{Alarms:alarmConfiguration,Rollback:autoRollbackConfiguration}'

The alarms field should list the alarms by name. The rollback events should include DEPLOYMENT_STOP_ON_ALARM.

For the forced-alarm test, the only way to know automatic rollback works is to trigger it deliberately. The cleanest test is to set an alarm threshold to a level the production service is already crossing, run a deployment, and watch it roll back. Alternatively, use set-alarm-state:

aws cloudwatch set-alarm-state \
  --alarm-name saas-api-5xx-rate \
  --state-value ALARM \
  --state-reason "rollback test"

This forces the alarm into ALARM state for a few seconds. If a deployment is in progress during the forced state, CodeDeploy will register it and trigger rollback. Do this once in a non-production environment before relying on the feature.

The fix

The full configuration for a service-level rollback setup. Three alarms: 5xx rate as absolute count, p95 latency as a percentile, custom application metric for a workload-specific failure mode.

The alarms:

aws cloudwatch put-metric-alarm \
  --alarm-name saas-api-5xx-rate \
  --alarm-description "Target group 5xx > 5 in 2 of last 3 minutes" \
  --metric-name HTTPCode_Target_5XX_Count \
  --namespace AWS/ApplicationELB \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --dimensions Name=TargetGroup,Value=targetgroup/saas-api-prod/abc \
               Name=LoadBalancer,Value=app/saas-alb/abc

aws cloudwatch put-metric-alarm \
  --alarm-name saas-api-p95-latency \
  --alarm-description "Target p95 > 800ms in 2 of last 3 minutes" \
  --metric-name TargetResponseTime \
  --namespace AWS/ApplicationELB \
  --extended-statistic p95 \
  --period 60 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --threshold 0.8 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --dimensions Name=TargetGroup,Value=targetgroup/saas-api-prod/abc \
               Name=LoadBalancer,Value=app/saas-alb/abc

aws cloudwatch put-metric-alarm \
  --alarm-name saas-api-webhook-failures \
  --alarm-description "Application webhook failures > 10 in 2 of last 3 minutes" \
  --metric-name WebhookDeliveryFailures \
  --namespace SaasApi \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching

Three alarms, all with evaluation-periods 3, datapoints-to-alarm 2 — meaning the alarm fires when 2 of the last 3 one-minute datapoints exceed the threshold. This catches sustained regressions while filtering single-minute blips.

The deployment group wires them all:

aws deploy update-deployment-group \
  --application-name saas-api \
  --current-deployment-group-name saas-api-prod \
  --auto-rollback-configuration '{
    "enabled": true,
    "events": ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM", "DEPLOYMENT_STOP_ON_REQUEST"]
  }' \
  --alarm-configuration '{
    "enabled": true,
    "ignorePollAlarmFailure": false,
    "alarms": [
      {"name": "saas-api-5xx-rate"},
      {"name": "saas-api-p95-latency"},
      {"name": "saas-api-webhook-failures"}
    ]
  }'

The ignorePollAlarmFailure: false matters — if CodeDeploy cannot read the alarm state (IAM permission missing, alarm deleted), the deployment is marked failed. With true, the agent shrugs and proceeds — defeating the safety net.

Post-rollback investigation runbook. A rollback is data. The runbook starts within an hour of the rollback to keep context fresh:

  1. Pull the rollback event timeline. aws deploy get-deployment --deployment-id d-XXX returns the lifecycle events and which alarm tripped.
  2. For the tripping alarm, query CloudWatch for the minute-by-minute datapoints during the deploy window. Was the alarm a sustained regression or a single spike? If single spike, the alarm needs more evaluation periods.
  3. Check the application logs on the green instances during the rollback window. The CodeDeploy agent log on each instance is at /var/log/aws/codedeploy-agent/codedeploy-agent.log; the application logs are wherever the service writes them. The signal is: were the green instances actually serving badly, or did the alarm trip on noise unrelated to the deploy?
  4. If the rollback was a true positive (the new code regressed), the diff between the rolled-back revision and the previous one is the root cause. The fix is a code change before re-deploying, not a tighter alarm.
  5. If the rollback was a false positive (the alarm tripped on noise), the alarm is the problem. Tune evaluation periods, switch to absolute counts, or scope the alarm to a narrower dimension.

Three rules we hold for rollback automation:

Tune alarms over a week of normal traffic before wiring them to rollback. A reasonable alarm threshold is the steady-state value plus a margin — pick the margin from observed traffic, not from a guess. An alarm that fires once per week on real noise will trip ~14% of deployments; an alarm that fires once per month will trip ~3%.

Distinguish service-level alarms from infrastructure alarms. A spike in EC2 NetworkOut is infrastructure noise; a spike in target-group HTTPCode_Target_5XX_Count is service signal. Only service-level alarms belong in the rollback config.

A repeated rollback on the same alarm without a regression in the diff is the alarm’s fault, not the deploy’s. Three rollbacks in a month with no code-change root cause means the alarm is too tight. Loosen, do not disable.

Automatic rollback is the difference between a 90-second recovery and a 25-minute one — but only if the alarms are wired to fire on real regressions, not on the noise that lives in every production service. Cross-reference the blue/green tutorial for the termination-wait window where alarms get their chance to fire.

Stack Harbor runs CodeDeploy + CodePipeline on production EC2 / ECS / Lambda fleets as part of Managed AWS and ongoing Managed Cloud Ops. If your deploys still take a maintenance window, start with an infrastructure consult.