Skip to content

AWS Updated 2026-05-11

ASG lifecycle hooks done right — draining, deregistering, and the heartbeat pattern

The default ASG terminates a spot instance two minutes after the warning. Two minutes is fine if your hook drains the LB, flushes logs, and deregisters from service discovery. Here is the pattern that does all three reliably.

The fleet runs Consul for service discovery, ships logs through Fluent Bit to a CloudWatch destination, and serves traffic behind an ALB. When the ASG terminates an instance, the default behavior is “stop and go” — AWS kills the instance, the ALB notices the target is unhealthy thirty seconds later and stops sending traffic, Consul times out the node check after a minute and removes the entry, and the last few seconds of logs never make it out of the Fluent Bit buffer. Most of the time it does not matter. The 1% of the time it does, you are looking at orphaned Consul entries that take six hours to clean up, a 30-second window where the ALB sends to a dead box and returns 502s, and a missing log fragment that happens to contain the line you need for a debug. Lifecycle hooks are how you make termination orderly.

What’s happening

A lifecycle hook is a pause in the ASG state machine. When the ASG decides to launch or terminate an instance, it can pause the transition and ask the outside world to do work before continuing. There are two transitions you can hook:

  • autoscaling:EC2_INSTANCE_LAUNCHING — fires after the instance boots but before it is marked InService. Your hook can pull configuration from a config server, register with service discovery, prewarm caches, run a sanity check. The instance is not in the load balancer yet, so no traffic concerns.
  • autoscaling:EC2_INSTANCE_TERMINATING — fires after the ASG decides to terminate but before AWS actually stops the instance. Your hook can drain connections, flush logs, deregister from service discovery, signal external systems to stop sending work.

The hook works by message-passing. When the ASG enters the paused state, it publishes a message to an SQS queue or EventBridge bus you specified. Something on the instance (or off it) reads the message, does the work, and calls complete-lifecycle-action to release the pause. If the work is not completed before the heartbeat-timeout (default 1 hour, max 1 hour), the ASG resumes the transition with the default-result you configured — CONTINUE (proceed) or ABANDON (mark the instance as failed and try again).

The pattern for terminate hooks specifically:

  1. The instance receives the hook message — usually via a small daemon polling SQS, or via an SSM document triggered by EventBridge.
  2. The daemon deregisters from service discovery first. Consul agent leave, removing the instance from any DNS round-robin, deregistering from any internal load balancer. This is the longest-cooling step because clients with cached endpoints need time to expire.
  3. Drain the load balancer. For an ALB, this is automatic once the target group’s deregistration_delay.timeout_seconds fires — but the hook should call deregister-targets explicitly to start the timer immediately rather than waiting for the ASG’s default detach behavior.
  4. Wait for in-flight requests to complete. A health endpoint on the instance can serve “draining” while still returning 200 to keep the existing connections alive. New connections from the ALB stop because the target is deregistered.
  5. Flush logs. pkill -USR1 fluent-bit or equivalent forces a buffer flush. Wait long enough for the buffered records to be acknowledged by the destination.
  6. Complete the hook. The daemon calls complete-lifecycle-action with CONTINUE. The ASG resumes and AWS terminates the instance.

The one-hour timeout gotcha bites teams that try to fit slow work into the hook. Hooks have a fixed maximum heartbeat timeout. If your work needs longer, you must call record-lifecycle-action-heartbeat periodically during the work, each call resetting the timer for another heartbeat-timeout seconds. Forget the heartbeats during a slow drain, the hook times out, the ASG executes the default-result (usually CONTINUE), and the instance is killed mid-drain.

Spot interruptions are different. AWS gives you two minutes before terminating a spot instance, and the lifecycle hook fires inside that window. The hook has at most two minutes, not one hour. Whatever drain logic you have on a spot fleet must complete in well under that — typically 90 seconds, leaving margin for the complete-lifecycle-action call to land before AWS forcibly terminates.

How to verify

List the hooks on an ASG:

aws autoscaling describe-lifecycle-hooks \
  --auto-scaling-group-name my-app-prod-asg \
  --output table

The output shows the hook name, the transition (LAUNCHING or TERMINATING), the heartbeat timeout, and the notification target. Each hook should have a sensible timeout — under 5 minutes for terminate hooks (anything longer is a smell), longer is fine for launch hooks.

Check whether the hook is actually firing and completing:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name my-app-prod-asg \
  --max-items 50 \
  --query 'Activities[].[StartTime,StatusCode,Description,Cause]' \
  --output table

A clean hook lifecycle shows WaitingForInstanceWarmup or WaitingForInstanceTermination, then Successful. A hook that timed out shows the description ending in “due to lifecycle action result”. When you see ABANDON results without an obvious cause, the daemon is the suspect — usually it crashed, lost the SQS message, or completed against the wrong instance.

Time a real termination end-to-end:

# Pick a healthy instance and trigger termination through the ASG
aws autoscaling terminate-instance-in-auto-scaling-group \
  --instance-id i-0123456789abcdef0 \
  --no-should-decrement-desired-capacity

Watch the scaling activities log and the application’s own logs in parallel. Healthy termination shows: hook fires → ALB receives deregister-targets → connections drain over the deregistration delay → logs flush → hook completes → AWS stops the instance. Time from terminate-instance-in-auto-scaling-group to “instance stopped” should be the hook’s actual drain time (typically 60–120 seconds for an LB-backed instance with logs).

If the hook fires but does not complete in a useful time, attach to the daemon and check whether the SQS message is being picked up:

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/asg-lifecycle-events \
  --max-number-of-messages 1

A message stuck in flight for tens of seconds means the daemon has it but is not completing the action. Lost messages mean the daemon never picked them up — usually a permissions problem or a crashed daemon.

The fix

The terminate-hook pattern in full, for an ALB-backed fleet using Consul for service discovery:

  1. Create the hook. This is the one-time setup at ASG provisioning:

    aws autoscaling put-lifecycle-hook \
  --auto-scaling-group-name my-app-prod-asg \
  --lifecycle-hook-name terminate-drain \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --heartbeat-timeout 300 \
  --default-result CONTINUE \
  --notification-target-arn arn:aws:sqs:us-east-1:123456789012:asg-lifecycle-events \
  --role-arn arn:aws:iam::123456789012:role/asg-lifecycle-role

    Five-minute heartbeat is enough margin for a clean drain on a busy instance. Default-result CONTINUE means “if the hook times out, terminate anyway” — the alternative ABANDON leaves the instance in Terminating:Wait indefinitely, which is rarely what you want.

  2. The daemon on each instance. A small systemd service running awscli and reading SQS:

    while true; do
  msg=$(aws sqs receive-message \
    --queue-url "${QUEUE_URL}" \
    --wait-time-seconds 20 \
    --max-number-of-messages 1)

  # Parse the message, check whether this instance is the target
  # If yes, run the drain steps and call complete-lifecycle-action
  # Always call delete-message after complete, regardless of result
done

    In practice this is more often a Lambda triggered by the SQS queue and using SSM to run drain commands on the instance, because that path survives the instance being unable to reach AWS APIs during drain.

  3. The drain steps, in order, with a heartbeat between each:

    # 1. Mark draining in the health endpoint
echo "draining" > /var/run/app/health-state

aws autoscaling record-lifecycle-action-heartbeat \
  --auto-scaling-group-name my-app-prod-asg \
  --lifecycle-hook-name terminate-drain \
  --instance-id "${INSTANCE_ID}" \
  --lifecycle-action-token "${TOKEN}"

# 2. Deregister from Consul
consul leave
sleep 5

# 3. Deregister from the ALB explicitly
aws elbv2 deregister-targets \
  --target-group-arn "${TG_ARN}" \
  --targets Id="${INSTANCE_ID}"

# 4. Wait for in-flight connections to drain (target group deregistration delay)
sleep 30

# 5. Flush logs
systemctl reload fluent-bit
sleep 10  # give Fluent Bit time to confirm flush

# 6. Complete the hook
aws autoscaling complete-lifecycle-action \
  --auto-scaling-group-name my-app-prod-asg \
  --lifecycle-hook-name terminate-drain \
  --instance-id "${INSTANCE_ID}" \
  --lifecycle-action-token "${TOKEN}" \
  --lifecycle-action-result CONTINUE

  4. The launch-hook counterpart, for completeness, has the inverse shape: instance boots → daemon receives EC2_INSTANCE_LAUNCHING message → pulls latest config from a config server → registers with Consul → completes hook → ASG marks instance InService → ALB starts health checks. The launch hook protects against the instance going InService before its dependencies are ready, which is the root cause of half the deploy failures teams see on ASG-backed services.

Three rules across every hook design:

Hook timeouts should match the worst-case drain, with margin. A 60-second drain on a P95 instance should have a 180-second hook timeout, not a 60-second one. If P99 is 90 seconds, set the timeout to 300. Margin protects against the slow case without padding the common case (you complete when ready, not when the timeout hits).

Default-result CONTINUE for terminate hooks, ABANDON for launch hooks. A terminate hook that times out should still let the instance terminate — leaving it in Terminating:Wait blocks the ASG from doing useful work. A launch hook that times out should not let a half-bootstrapped instance into the LB — ABANDON retries the launch, which is the safer default.

Test the failure mode. Kill the daemon mid-drain. Drop the SQS queue’s permissions. Make sure the hook timeout fires when it should, and that the default-result behavior is what you expected. Hook bugs are silent until the day the workload actually hits them in production.

The lifecycle hook pattern is forty lines of script and an SQS queue. Done right, terminations are unobservable from outside the cluster. Done wrong, you find out about it when the alert about 502s during deploys becomes someone’s morning.

Stack Harbor operates ASG-backed application tiers and cost-capped auto-scaling policies as part of Managed AWS and the Auto-Scaling operating layer. If your scale events are still surprises, start with an infrastructure consult.