Skip to content

Health check tuning for SaaS targets — what / actually means

Default health check is /. The default catches a dead process, not a broken application. The path that matters talks to the database. The interval and threshold that matter prevent a slow target from killing the whole fleet.

A SaaS application starts losing requests at 14:00 every day. The ALB CloudWatch metrics show targets going unhealthy and healthy in rapid succession. The team checks the application logs and finds nothing — the app is running, the process is alive, requests are being served. The health check path is /, the default. / returns the SPA’s index HTML in 5ms because the request never touches the database or the cache. The actual problem is that the application’s database connection pool is exhausted; real user requests time out, but the health check passes because it does not exercise the dependency that is failing. ALB sees healthy targets, routes traffic to them, requests fail, users see errors. The fix is to make the health check actually exercise the dependencies the application needs — and to tune the interval and threshold so a slow target is taken out of rotation before it cascades, but not so aggressively that a momentary blip flaps the entire fleet.

What’s happening

ALB health checks are independent HTTP requests the ALB sends to each target on a schedule. The target is “healthy” or “unhealthy” based on whether the response matches the configured success criteria. ALB only routes traffic to healthy targets.

The parameters that matter:

  • HealthCheckPath — The URL path the check hits. Default is /. For most modern applications, / returns the SPA bundle from a CDN-cached asset or a static index file — it tells you nothing about whether the application can actually serve requests.

  • HealthCheckIntervalSeconds — How often ALB checks each target. Default is 30 seconds. Range is 5 to 300.

  • HealthCheckTimeoutSeconds — How long ALB waits for a response. Default is 5 seconds. Must be less than the interval.

  • HealthyThresholdCount — How many consecutive successes flip a target from unhealthy to healthy. Default is 5.

  • UnhealthyThresholdCount — How many consecutive failures flip a target from healthy to unhealthy. Default is 2.

  • Matcher — Which HTTP status codes count as success. Default for HTTP targets is 200. Can be a range like 200-299 or a list like 200,201,302.

  • Health check grace period (target group attribute health_check.grace_period_seconds) — On a freshly registered target, how long ALB waits before its first health check counts. Default is 30 seconds. For a target that takes 90 seconds to warm up (JVM, large image pull, dependency resolution), the default means the target is killed before it ever serves a request.

The two distinct health check paths every application should expose:

  1. Liveness path (/healthz). Returns 200 if the process is up and the runtime is responsive. Does not touch the database, cache, or external services. Cheap, fast, never fails for an upstream reason. Used by the orchestrator (ASG, ECS) to decide if the process should be killed and restarted.

  2. Readiness path (/ready). Returns 200 if the application can actually serve real requests. Touches the database with a SELECT 1, pings the cache, confirms the external API client is configured. Slower, more expensive, can fail for upstream reasons. Used by the load balancer to decide if the target should receive traffic.

ALB should health-check the readiness path. If the database is down and /ready returns 503, ALB takes the target out of rotation and stops sending it user traffic. The target stays alive (the process is fine); when the database recovers, /ready starts returning 200 again and ALB puts the target back into rotation. The user sees a brief blip during the database outage but not cascading errors.

Pointing ALB at /healthz instead is the wrong choice for SaaS. The process is up; the application cannot serve real requests; ALB sends traffic anyway; users get errors.

How to verify

Inspecting the current health check configuration on a target group:

aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
  --query 'TargetGroups[].{
    Path: HealthCheckPath,
    Interval: HealthCheckIntervalSeconds,
    Timeout: HealthCheckTimeoutSeconds,
    Healthy: HealthyThresholdCount,
    Unhealthy: UnhealthyThresholdCount,
    Matcher: Matcher.HttpCode
  }'

Checking the current health state of every target:

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].{
    Target: Target.Id,
    Port: Target.Port,
    State: TargetHealth.State,
    Reason: TargetHealth.Reason,
    Description: TargetHealth.Description
  }'

The Reason field tells you why a target is unhealthy. Target.FailedHealthChecks means the path returned a non-success status. Elb.RegistrationInProgress means the target is still in the grace period. Target.Timeout means the response did not arrive within HealthCheckTimeoutSeconds.

Tuning the timeout for a SaaS app that does a database round-trip in the readiness check:

  • A /ready that hits the database typically responds in 50-200ms when healthy. Setting HealthCheckTimeoutSeconds=2 is reasonable — long enough for a healthy response, short enough to catch a slow one.
  • The interval of 10 seconds means ALB sees an unhealthy target within 10-30 seconds.
  • UnhealthyThresholdCount=2 means two consecutive failures take the target out. With a 10-second interval, that is 20 seconds from “first failure” to “out of rotation.”
  • HealthyThresholdCount=2 means two consecutive successes put it back. Faster recovery.

A typical SaaS configuration:

aws elbv2 modify-target-group \
  --target-group-arn "$TG_ARN" \
  --health-check-path /ready \
  --health-check-interval-seconds 10 \
  --health-check-timeout-seconds 2 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 2 \
  --matcher HttpCode=200

For the grace period on a target that takes time to warm up:

aws elbv2 modify-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --attributes Key=health_check.grace_period_seconds,Value=120

For a Java app with JVM warmup, 120 seconds is reasonable. For a Node app, 30 seconds is plenty. For a Lambda-backed target group, the grace period is moot — the Lambda cold start is the warm-up.

Confirming that the readiness path actually exercises the dependencies. The cheapest way is to check on the application side that /ready calls the database:

# minimal Flask handler
@app.route('/ready')
def ready():
    try:
        db.session.execute('SELECT 1')
        cache.get('healthcheck')  # cache ping
        return 'OK', 200
    except Exception as e:
        return str(e), 503

For the operations side, the test is to break the dependency on purpose. Stop the database in a staging environment. The target should go unhealthy within (interval × unhealthy threshold) seconds. The application should keep running. Restart the database. The target should recover within (interval × healthy threshold) seconds.

The fix

The runbook for tuning health checks on an existing target group:

  1. Audit the health check path. If it is /, change it. The path needs to be a route the application owns, not a static asset.

  2. Add or fix the /ready endpoint in the application. The endpoint exercises the dependencies the application uses for real requests. Database ping, cache ping, downstream API client status.

  3. Set the path to /ready and the matcher to 200 only. Do not match a range; a 200 says “I can serve traffic,” a 503 says “I cannot.” Anything else is a coding mistake to surface, not to absorb.

  4. Tune the interval and threshold for the workload. Fast detection (small interval, low threshold) for high-traffic SaaS where one minute of dead targets is many failed requests. Slower detection for low-traffic internal tools where false positives matter more than absolute speed.

  5. Set the grace period to the actual warm-up time. Time the application from start to first successful request in a representative environment. Set the grace period to that number plus 30 seconds margin.

  6. Wire up a CloudWatch alarm on UnHealthyHostCount. Goes off when any host is unhealthy for more than a minute. The alarm tells you which target group, you check the TargetHealth.Reason to know why.

  7. Test by breaking a dependency in staging. Verify that the target goes unhealthy as expected, that user traffic stops landing on it, and that recovery is automatic. The first time you discover the health check does not behave as expected should not be in production.

Three rules we hold:

Liveness and readiness are different endpoints. The orchestrator wants to know “is the process up.” The load balancer wants to know “can the process serve traffic.” Conflating them means either the orchestrator kills targets it should not, or the load balancer routes to targets that cannot serve.

Cap the cost of the readiness check. A /ready that runs a full integration test every 10 seconds is its own load on the database. The path should do the cheapest queries that prove the dependency is reachable — SELECT 1 is enough.

The default is wrong. ALB’s default health check (path /, interval 30s, threshold 5, matcher 200) is calibrated for “static site with no dependencies.” For a SaaS, the defaults will take 2.5 minutes to detect a broken target and may never detect a dependency failure.

For more on the ALB rule context where health checks live, see ALB listener rules — host and path routing. For sticky-session interactions with health checks (an unhealthy target evicts its sticky clients), see Sticky sessions are a smell.


Stack Harbor designs ALB + NLB topologies and the operating model around them as part of Managed AWS onboarding and ongoing Managed Cloud Ops. If your fleet flaps every afternoon and the application logs are silent, start with an infrastructure consult.