Skip to content

AWS Updated 2026-01-20

EC2 vs ELB health checks — why InService instances still serve 500s

The ASG says every instance is InService. The ALB target group says half are Unhealthy. Both are right, and the gap is one configuration flag away. Here is how the two health-check models interact and the grace-period gotcha that breaks deploys.

The team rolled a new application version through an instance refresh. The ASG console shows every instance in the fleet as InService. The ALB target group console shows half of them as Unhealthy, returning 5xx on the health endpoint. The pager goes off. Twenty minutes of debugging later: the application started, the EC2 instance is reachable, but the application crashes on first request because of a missing environment variable. The EC2 health check sees a running instance and reports healthy. The ELB health check sees the application returning 503 and reports unhealthy. The ASG, configured with the default EC2 health check type, sees its own check passing and leaves the instance in service indefinitely. The ALB stops sending traffic to it. The fleet is half its real capacity. Both health checks are doing exactly what they were told to do. Nobody told the ASG to listen to the ALB.

What’s happening

There are two health-check models in play on an ALB-backed ASG, and they answer different questions.

EC2 health checks answer “is the instance running?” — the ASG asks AWS’s instance-status API. The check passes if the system status is ok (the EC2 host is reachable) and the instance status is ok (the OS is reachable and not in an impaired state). The check fails when AWS detects host-level failure, kernel panic, or the instance going into a stopped/terminated state. EC2 health checks are the ASG default. They are fast, cheap, and useful — but they do not answer the question that matters: “is the application working?”

ELB health checks answer “is the application responding?” — the load balancer makes an HTTP, HTTPS, TCP, or gRPC request to a configured endpoint on each target and judges based on the response. For an ALB, this is typically GET /health returning 200. For an NLB, it can be a TCP connection probe or an HTTP/HTTPS check. The check passes when the target meets the healthy threshold (consecutive successful responses) and fails when it crosses the unhealthy threshold.

The ASG can be configured to listen to either or both. The HealthCheckType field on the ASG is EC2 (default) or ELB. Setting it to ELB does not disable EC2 health checks — it adds the ELB’s verdict to the equation. The ASG considers an instance unhealthy if any of its configured health checks fail. So HealthCheckType=ELB means “consider the instance unhealthy if either EC2 or ELB says so” — which is what most teams actually want.

The interaction:

  1. ASG launches an instance. It is Pending in the ASG state machine.
  2. The instance starts. EC2 health checks start passing once the system status is ok.
  3. The ALB starts health-checking the new target. The target is initial while the first check completes.
  4. The ASG marks the instance InService when EC2 health checks pass.
  5. The ALB marks the target healthy when ELB health checks pass.
  6. The ALB sends traffic to the target only when it is healthy.

Two timing problems result from this dance:

  • The instance is InService (ASG) before the application is ready (ALB). This is normal — the ASG’s job is to track whether the instance exists; the ALB’s job is to track whether it responds. The two converge once the application boots.
  • The instance stays InService (ASG) while the ALB has marked it Unhealthy. This is the broken state. The ASG sees a running box, the ALB sees a broken application, and without HealthCheckType=ELB the ASG never replaces the box.

The health-check-grace-period gotcha applies to step 4 specifically. The grace period (HealthCheckGracePeriod on the ASG) is the number of seconds the ASG waits after launching an instance before it starts evaluating health checks. The default is 300 seconds, which is fine for most applications. The mistake is leaving it at 300 when the application takes 8 minutes to start — the ASG hits the 5-minute mark, the ELB is still reporting unhealthy because the app is still booting, the ASG sees “ELB unhealthy past grace period” and terminates the instance. Next instance launches, same thing. The ASG burns through instances in a loop. The fleet never grows.

The inverse mistake is setting the grace period too high. With a 30-minute grace period, a genuinely broken instance — one that crashes immediately on start — stays InService for 30 minutes before the ASG notices. The fleet runs at reduced capacity through that whole window.

How to verify

Inspect the health-check configuration on an ASG:

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names my-app-prod-asg \
  --query 'AutoScalingGroups[0].[HealthCheckType,HealthCheckGracePeriod]' \
  --output table

EC2 with a 300-second grace period is the default. Look for two flags of misconfiguration:

  • HealthCheckType=EC2 on an ALB-backed ASG. Almost always wrong. The team intended ELB and forgot.
  • HealthCheckGracePeriod=300 on an application that takes longer than 300 seconds to boot. Common after migrating from a thin AMI to a thicker one and forgetting to bump the grace period.

Verify the ALB target group’s view of the same fleet:

aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-app/abc \
  --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]' \
  --output table

The states you might see, and what each means:

  • healthy — the target is passing health checks and receiving traffic.
  • unhealthy — the target is failing health checks. The reason field clarifies: Target.FailedHealthChecks, Target.Timeout, Target.ResponseCodeMismatch, etc.
  • initial — the target was just registered and the ALB has not completed the first health check round.
  • draining — the target was deregistered and is finishing in-flight requests.
  • unused — the target is not in any availability zone the ALB is configured for. Rare; usually a subnet mismatch.

The combination that triggers the opener — ASG says InService, ALB says unhealthy — is the smoking gun for HealthCheckType=EC2 on an ALB-backed ASG.

To time how long the application actually takes to be ready:

# Launch a fresh instance into the ASG and watch
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name my-app-prod-asg \
  --desired-capacity $(($(aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-names my-app-prod-asg \
    --query 'AutoScalingGroups[0].DesiredCapacity' \
    --output text) + 1))

# Then poll target health until the new instance shows healthy

The delta between “instance shows in target group as initial” and “instance shows as healthy” is the real application-readiness time. The ASG HealthCheckGracePeriod should be that number plus margin (typically 30–60 seconds), not the default 300.

The fix

The configuration that actually works on an ALB-backed ASG, in order of decreasing importance:

  1. Set HealthCheckType=ELB. This is the single most important flag. The ASG listens to both checks, replaces instances that the ALB says are broken, and stays useful through grace-period-related edge cases:

    aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name my-app-prod-asg \
  --health-check-type ELB \
  --health-check-grace-period 360

  2. Set HealthCheckGracePeriod to the application’s real boot time plus 30–60 seconds margin. Measure it. Do not guess. An application that takes 4 minutes to boot needs 270–300 seconds grace; an application that takes 90 seconds needs 120–150.

  3. Configure the target group health check to match the application’s actual readiness signal. The default check (GET / returning 200) is rarely the right one. A proper health endpoint:

    • Returns 200 only when the application is actually able to serve real traffic — including database connectivity, downstream service reachability, and any in-process initialization.
    • Returns 503 (not 500, not 404) when the application is starting, draining, or otherwise not ready. 503 has the explicit “service unavailable, try again” semantic that load balancers understand.
    • Is cheap. Health checks fire every 30 seconds or so, per target, per LB instance. A health endpoint that does a real database query on every check is a DDoS against the database.

    Set the target group’s health check config:

    aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-app/abc \
  --health-check-path /health \
  --health-check-interval-seconds 30 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 3 \
  --health-check-timeout-seconds 5

    The defaults are mostly fine — interval=30, healthy_threshold=5, unhealthy_threshold=2, timeout=5. The tweak above lowers healthy_threshold to 2 (faster recovery) and raises unhealthy_threshold to 3 (tolerates a single flap during peak load).

  4. Match the grace period to the deployment pattern. During an instance refresh, the ASG launches new instances and waits the grace period before evaluating their health. If your grace period is 600 seconds and you have 20 instances refreshing one at a time, the refresh takes at least 200 minutes just on the grace periods. Faster boots = shorter grace periods = faster deploys.

  5. Use HealthCheckGracePeriod overrides for special cases. Lifecycle hooks during the launch phase can extend the effective readiness time without bumping the grace period for everyone. The default_instance_warmup field on the ASG is the modern way to control this for scaling policies specifically — independent of the health-check grace.

The deploy gotcha: when you push a new AMI and start an instance refresh, the new instances boot from a new AMI but the ASG keeps the same HealthCheckGracePeriod. If the new AMI takes longer to bootstrap than the old one — common during a transition from instance-baked to user-data-pulled artifacts — the grace period that was fine for the old AMI is too short for the new one. The refresh “succeeds” for the first few instances (they boot inside the grace period, get a chance to come healthy, all good), then fails for instances that have noisy-neighbor IO contention and boot slowly. The fix is to bump the grace period as part of the new-AMI rollout, not as a separate change.

Three rules across every ALB-backed ASG:

HealthCheckType=ELB is the default you want. Anything else means the ASG cannot replace broken instances that look running from EC2’s perspective. The savings from running EC2-only are not real — the ALB still does its own checks; the question is whether the ASG hears the answer.

Health endpoints are application code, not infrastructure config. A health endpoint that returns 200 when the database is down is worse than no health endpoint at all. Make the endpoint actually verify the things that fail.

Measure the boot time. Set the grace period from measurement. “300 seconds because it is the default” is a future incident report waiting to happen.

The model is simple once you see it: EC2 health checks for “the box is alive,” ELB health checks for “the application works.” The ASG listens to whichever set you configure. Most teams want both, and most teams find that out the hard way during a deploy gone wrong. See the related lifecycle hooks article for the launch-hook pattern that protects against the inverse problem — instances going InService before they are actually ready.

Stack Harbor operates ASG-backed application tiers and cost-capped auto-scaling policies as part of Managed AWS and the Auto-Scaling operating layer. If your scale events are still surprises, start with an infrastructure consult.