Skip to content

AWS Updated 2026-05-31

Scaling on a custom metric — queue depth, not CPU

CPU is a lagging proxy on workloads where the bottleneck is the queue. The right metric is the thing you actually want to control. Here is how to publish it, alarm on it, and do the math for what 100 requests per instance means after the scale event lands.

The worker fleet processes jobs from SQS. CPU on the workers averages 40% — comfortable. The queue depth is at 50,000 jobs and climbing. SLAs are starting to slip. The team has a target-tracking policy on CPU set to 60%, and it is doing nothing because the CPU is not stressed. The bottleneck is not CPU at all — it is the I/O wait on the downstream API the workers are calling. The CPU is idle because the workers are blocked on network. Scaling on CPU here is scaling on a proxy that does not reflect the actual load. The right thing to scale on is the queue itself: how many jobs are waiting per worker. Here is how to publish that metric, alarm on it, and pick a target that gets the math right.

What’s happening

Target tracking and step scaling both accept any CloudWatch metric as input. The default metrics — CPUUtilization, NetworkIn, NetworkOut, ASGAverageCPUUtilization, ALBRequestCountPerTarget — are convenient because they require no setup. But they are the right metric only when CPU or request count is actually the bottleneck. For queue-driven workers, p95 latency-bound services, in-flight-request-limited workloads, or any case where the limiting resource is something other than CPU, the right metric is a custom CloudWatch metric.

The standard custom-metric scaling pattern:

  1. Publish the metric. Either as a CloudWatch agent push from the workload itself, or via aws cloudwatch put-metric-data on a small interval. The metric should be a normalized “per-instance” or “per-target” value, not a global aggregate — target tracking does the math correctly when the metric is already in the right shape.
  2. Configure the scaling policy to target the published metric. Target tracking creates the alarms; step scaling requires you to create the alarms first.
  3. Define what “right” means. For queue depth, this is jobs-per-instance. For latency, this is the p95 you commit to. For in-flight requests, this is the concurrency limit per instance.

The pattern for queue-driven workers, specifically:

  • The metric is BacklogPerInstance, computed as queue_depth / current_instance_count. Publishing this rather than raw queue depth means target tracking does not double-correct when scale-out lands.
  • The target is the number of jobs each worker can finish in one acceptable-latency window. If workers process a job in 5 seconds and you want the oldest job in the queue to be no more than 2 minutes old, target backlog is 24 jobs per instance (120 seconds / 5 seconds per job).
  • The publishing interval is shorter than the alarm period. If you publish every minute and the alarm evaluates over three 60-second periods, that is fine. Publishing every 5 minutes against a 1-minute alarm causes “metric missing” gaps and unpredictable scaling.

The math that catches teams. A target-tracking policy with target=24 backlog-per-instance, queue at 240, current fleet at 5 instances:

  1. Current backlog-per-instance = 240 / 5 = 48. Above target. Scale out.
  2. AWS estimates: to hit target=24, need 240 / 24 = 10 instances. Add 5 instances.
  3. The new instances launch and start consuming the queue.
  4. Five minutes later: queue is at 100 (workers are draining it faster), fleet is at 10, backlog-per-instance = 10. Below target.
  5. The cooldown protects against immediate scale-in, but eventually backlog-per-instance settles to 24 once the queue stabilizes — if the publication rate of jobs to the queue is steady. If the publication rate dropped, the fleet will scale in.

The trap is when publication is bursty. Queue depth spikes to 5,000 during a marketing email send, then drops back to baseline within 10 minutes. Backlog-per-instance hits 5000/5 = 1000, alarm fires, scale-out adds many instances. By the time they are warm, the burst is over, backlog drops, scale-in fires, instances terminate. This is the same flapping pattern as the target-tracking-vs-step article, and the answer there applies: target tracking is the wrong tool for bursty metrics; step scaling on an alarm with multiple evaluation periods is what you want for short surges.

How to verify

Check what metric the current scaling policy is using:

aws autoscaling describe-policies \
  --auto-scaling-group-name my-worker-asg \
  --query 'ScalingPolicies[].[PolicyName,PolicyType,TargetTrackingConfiguration.CustomizedMetricSpecification.MetricName]' \
  --output table

If the policy is TargetTrackingScaling and CustomizedMetricSpecification is null, the policy is using a predefined metric (CPU or ALB request count). If CustomizedMetricSpecification has a metric name, that is the custom metric driving scaling — read its details:

aws autoscaling describe-policies \
  --auto-scaling-group-name my-worker-asg \
  --query 'ScalingPolicies[0].TargetTrackingConfiguration' \
  --output json

Verify the metric is actually being published with the expected dimensions and at the expected rate:

aws cloudwatch list-metrics \
  --namespace MyApp/Workers \
  --metric-name BacklogPerInstance \
  --output table

aws cloudwatch get-metric-statistics \
  --namespace MyApp/Workers \
  --metric-name BacklogPerInstance \
  --dimensions Name=AutoScalingGroupName,Value=my-worker-asg \
  --start-time $(date -u -v-1h +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Average \
  --output text

Gaps in the time series mean the publisher is unreliable. Target tracking treats missing data as “treat as breaching threshold” by default, which causes scale-out on a metric outage — not what you want. If you see gaps, fix the publisher before tuning the policy.

For the queue-depth case specifically, watch the SQS metric and the derived BacklogPerInstance side by side:

aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=my-app-jobs \
  --start-time $(date -u -v-1h +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Average

The two should track — BacklogPerInstance is the SQS metric divided by the fleet size. A divergence means the publisher is using a stale fleet count, which usually points to caching in the publishing process.

The fix

Publishing the metric, the lambda pattern that is cheapest and most reliable:

  1. Lambda fires every minute on an EventBridge schedule.
  2. Lambda reads ApproximateNumberOfMessagesVisible from the SQS queue.
  3. Lambda reads DesiredCapacity from the ASG (not InService — DesiredCapacity is the target, which is what target tracking should normalize against).
  4. Lambda computes backlog_per_instance = max(queue_depth, 1) / max(desired, 1) — the floor prevents division-by-zero and weird values when scaling from zero.
  5. Lambda publishes the metric with aws cloudwatch put-metric-data, dimensioned by AutoScalingGroupName.

The IAM permissions for the lambda are minimal: read SQS, read ASG, write CloudWatch. The lambda code is twenty lines. Cost is one invocation per minute, well under the free tier.

# Example put-metric-data, the line the lambda runs
aws cloudwatch put-metric-data \
  --namespace MyApp/Workers \
  --metric-name BacklogPerInstance \
  --value "${BACKLOG_PER_INSTANCE}" \
  --dimensions AutoScalingGroupName=my-worker-asg \
  --unit Count

Configure target tracking on the custom metric:

aws autoscaling put-scaling-policy \
  --auto-scaling-group-name my-worker-asg \
  --policy-name backlog-target-tracking \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "TargetValue": 24.0,
    "CustomizedMetricSpecification": {
      "MetricName": "BacklogPerInstance",
      "Namespace": "MyApp/Workers",
      "Dimensions": [{"Name": "AutoScalingGroupName", "Value": "my-worker-asg"}],
      "Statistic": "Average"
    }
  }'

The math, made explicit. Picking the target value is the part teams overthink:

  1. Pick the SLA on the oldest job in the queue. “No job sits in the queue longer than N seconds before a worker picks it up.”
  2. Measure the worker’s per-job processing time. Pull ApproximateAgeOfOldestMessage from CloudWatch and the worker’s own processing-duration metric.
  3. Compute target = SLA / per_job_time. SLA=120s, per_job=5s → target=24 jobs per instance.
  4. Add headroom for variance. If per-job is bimodal (5s for most, 30s for some), the average is misleading. Use p95 of per-job time for the math: target=120/p95. Otherwise the queue will routinely violate SLA on the long-tail jobs.
  5. Round the target value to something a human will recognize in a year. 24 is fine. 23.7 looks like a derivation that nobody remembers. The exactness is false precision against the actual variance in the workload.

For workloads where the right metric is latency rather than queue depth:

  • ALBTargetResponseTime p95 is a built-in metric on ALB target groups. Target tracking accepts it natively as a predefined metric specification, no custom publishing required.
  • The math: pick a p95 latency SLA, multiply by some confidence factor (1.2 is common — target tracking tends to overshoot a touch, and you want margin), set that as the target.
  • The trap: a fleet that is barely meeting p95 with current load and just barely below target will scale in if latency drops slightly. You end up oscillating around the target.

For workloads limited by in-flight requests (concurrency-bound, common for AI inference and external-API-bound services):

  • Publish InFlightRequestsPerInstance from the workload — incremented on request start, decremented on response.
  • Target = concurrency_limit_per_instance × 0.7. The 0.7 is headroom; 100% target tracking on a hard limit will queue requests behind the limit, increasing latency.

Three rules across every custom-metric scaling policy:

Publish a per-instance metric, not a global aggregate. Target tracking does the right math when the metric is in the per-instance shape. If you publish global queue depth and target “1000,” the policy has to do its own divide-by-fleet-size, which it does — but the second-order effects (what counts as “current capacity” at the moment the alarm fires) are subtle and prone to overshooting.

Set the target from the SLA and the worker’s actual performance. “60% CPU” is a tradition. “24 jobs per instance because SLA is 120 seconds and p95 processing is 5 seconds” is a derivation. Tradition does not survive workload changes.

Watch the metric for publication gaps. Target tracking will scale on missing data unless explicitly configured otherwise. A flaky publisher causes more incidents than a wrong target value.

The publishing-and-scaling loop is forty lines of Lambda code and a single put-scaling-policy call. Done right, the fleet sizes itself to meet the SLA you committed to, not the SLA the metric happens to be a proxy for.

Stack Harbor operates ASG-backed application tiers and cost-capped auto-scaling policies as part of Managed AWS and the Auto-Scaling operating layer. If your scale events are still surprises, start with an infrastructure consult.