Skip to content

AWS Updated 2026-05-05

Canary deploys on Lambda — the pre/post-traffic hooks that actually catch regressions

CodeDeployDefault.LambdaCanary10Percent5Minutes only works if the validation hook does something real. Here is the hook that runs a smoke probe and checks CloudWatch error rate before promoting.

A Lambda function fronted by API Gateway handles webhook deliveries from a third-party SaaS — about 80 requests per second peak. The team ships a refactor on a Friday afternoon. The deployment uses CodeDeployDefault.LambdaCanary10Percent5Minutes and the pre/post-traffic hooks both exit 0 with no checks. Five minutes later the 100% shift completes, the alarm bell rings forty minutes after that, and an entire afternoon of webhooks have been dropped because the new code is returning 502 on a payload shape the test suite never exercised. Canary deploys on Lambda are only as good as the hook that validates the canary. Here is the pattern that catches the regression instead of waiting for the alarm.

What’s happening

Lambda canary via CodeDeploy works by alias-based traffic shifting. The alias (typically live) points at version N. The deployment publishes version N+1 and shifts a fraction of invocations to it according to the deployment configuration. CodeDeployDefault.LambdaCanary10Percent5Minutes shifts 10% of traffic for 5 minutes, then 100%. LambdaCanary10Percent10Minutes holds longer. LambdaLinear10PercentEvery1Minute ramps gradually. LambdaAllAtOnce is the no-canary option.

Two hooks bracket the shift:

  • BeforeAllowTraffic runs once, before any production traffic hits the new version. The new version is published but the alias still routes 100% to the old one. This is the synthetic-test window — invoke the new version directly by ARN, exercise the critical paths, exit non-zero to abort.
  • AfterAllowTraffic runs once, after the 100% shift completes. The new version is in production. This is the validation window — check CloudWatch metrics over the canary period, exit non-zero to trigger rollback.

The default hooks do nothing. CodeDeploy ships templates with return {'hookStatus': 'Succeeded'} and developers leave them as-is. The hooks must do real work, and the rollback configuration must respect the hook’s exit code.

The other failure mode is the alarm wiring. If automaticRollbackConfiguration does not include DEPLOYMENT_STOP_ON_ALARM, a CloudWatch alarm firing during the canary will surface in the console but not actually roll back. The events list needs DEPLOYMENT_FAILURE, DEPLOYMENT_STOP_ON_ALARM, and DEPLOYMENT_STOP_ON_REQUEST for full coverage.

How to verify

Three checks before promoting a Lambda canary deployment group to production: the alias is real, the hooks have permissions, and the alarms exist.

The alias check matters because aws lambda publish-version increments the version number every deploy, and the alias must point at a specific version (not $LATEST). Verify:

aws lambda get-alias \
  --function-name webhook-processor \
  --name live

Output should include "FunctionVersion": "47" (or whatever the current production version is) and not "FunctionVersion": "$LATEST". An alias pointing at $LATEST cannot be canary-deployed — CodeDeploy refuses to create a deployment.

The hook permission check matters because the pre/post-traffic hooks are themselves Lambda functions, and they need codedeploy:PutLifecycleEventHookExecutionStatus to report their result. The hook function’s execution role needs this policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "codedeploy:PutLifecycleEventHookExecutionStatus",
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": ["lambda:InvokeFunction", "cloudwatch:GetMetricStatistics"],
    "Resource": "*"
  }]
}

Without the first action, the hook will run, do its work, and then fail to report — and CodeDeploy will time out the lifecycle event after an hour.

The alarm check is the same as for EC2 deploys. List the deployment group’s alarm config:

aws deploy get-deployment-group \
  --application-name webhook-processor \
  --deployment-group-name webhook-prod \
  --query 'deploymentGroupInfo.alarmConfiguration'

For Lambda, the alarms that matter are function-level: Errors over the canary period, Duration p95 or p99, and Throttles. The error rate alarm should fire if errors exceed a fixed count (not a percentage) over a one-minute window with two evaluation periods — a percentage-based alarm is noisy at low traffic.

The fix

The AfterAllowTraffic hook that actually validates. This is a Python Lambda that runs a smoke probe against the new version, then checks the CloudWatch error rate over the canary window:

import boto3
import json
import os
import time
from datetime import datetime, timedelta, timezone

cloudwatch = boto3.client('cloudwatch')
lambda_client = boto3.client('lambda')
codedeploy = boto3.client('codedeploy')

FUNCTION_NAME = 'webhook-processor'
CANARY_WINDOW_MINUTES = 5
MAX_ERRORS = 3

def handler(event, context):
    deployment_id = event['DeploymentId']
    hook_execution_id = event['LifecycleEventHookExecutionId']
    status = 'Succeeded'

    try:
        smoke_test()
        check_error_rate()
    except Exception as e:
        print(f'Validation failed: {e}')
        status = 'Failed'

    codedeploy.put_lifecycle_event_hook_execution_status(
        deploymentId=deployment_id,
        lifecycleEventHookExecutionId=hook_execution_id,
        status=status
    )
    return {'hookStatus': status}

def smoke_test():
    payload = {'webhook_id': 'smoke-test', 'event': 'ping'}
    response = lambda_client.invoke(
        FunctionName=FUNCTION_NAME,
        Qualifier='live',
        Payload=json.dumps(payload).encode()
    )
    body = json.loads(response['Payload'].read())
    if response.get('FunctionError') or body.get('statusCode') != 200:
        raise RuntimeError(f'smoke test failed: {body}')

def check_error_rate():
    end = datetime.now(timezone.utc)
    start = end - timedelta(minutes=CANARY_WINDOW_MINUTES)
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/Lambda',
        MetricName='Errors',
        Dimensions=[{'Name': 'FunctionName', 'Value': FUNCTION_NAME}],
        StartTime=start,
        EndTime=end,
        Period=60,
        Statistics=['Sum']
    )
    total = sum(d['Sum'] for d in response['Datapoints'])
    if total > MAX_ERRORS:
        raise RuntimeError(f'{total} errors in {CANARY_WINDOW_MINUTES}m, threshold {MAX_ERRORS}')

The BeforeAllowTraffic hook is a thinner version of the same — it invokes the new version directly by version-qualified ARN (not the live alias) and runs the smoke test only. The error-rate check has no data yet at that point.

The full deployment group config with both hooks and alarm wiring:

aws deploy create-deployment-group \
  --application-name webhook-processor \
  --deployment-group-name webhook-prod \
  --service-role-arn arn:aws:iam::111122223333:role/CodeDeployServiceRoleLambda \
  --deployment-config-name CodeDeployDefault.LambdaCanary10Percent5Minutes \
  --auto-rollback-configuration '{
    "enabled": true,
    "events": ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM", "DEPLOYMENT_STOP_ON_REQUEST"]
  }' \
  --alarm-configuration '{
    "enabled": true,
    "alarms": [
      {"name": "webhook-processor-errors"},
      {"name": "webhook-processor-duration-p99"}
    ]
  }'

The appspec.yaml for the Lambda revision references the hook ARNs:

version: 0.0
Resources:
  - webhook-processor:
      Type: AWS::Lambda::Function
      Properties:
        Name: webhook-processor
        Alias: live
        CurrentVersion: 47
        TargetVersion: 48
Hooks:
  - BeforeAllowTraffic: arn:aws:lambda:ca-central-1:111122223333:function:webhook-before-traffic
  - AfterAllowTraffic: arn:aws:lambda:ca-central-1:111122223333:function:webhook-after-traffic

Three rules we hold for Lambda canary deploys:

The validation hook must invoke the target version directly, not via the alias. During the canary, only 10% of alias-routed traffic reaches the new version — a smoke test via the alias has a 90% chance of testing the old code. Invoke by version ARN.

Use a fixed error count, not a percentage. A function doing 80 RPS will see hundreds of invocations during a 5-minute canary; a function doing 0.5 RPS will see 150. A 1% threshold means 1.5 errors in the second case — round to 2 and you accept 1.3% real failure rate. Fixed thresholds are reasonable.

Time the canary window for at least two CloudWatch evaluation periods. A 5-minute canary against a 1-minute, 2-period error alarm has exactly enough room for the alarm to fire during the canary. A 5-minute canary against a 5-minute, 2-period alarm cannot possibly fire in time — the alarm needs 10 minutes of bad data to trip.

The canary pattern is the safest Lambda deploy shape. Cross-reference the rollback automation runbook for the alarm wiring details.

Stack Harbor runs CodeDeploy + CodePipeline on production EC2 / ECS / Lambda fleets as part of Managed AWS and ongoing Managed Cloud Ops. If your deploys still take a maintenance window, start with an infrastructure consult.