Skip to content

Setting actionable HTTP latency alerts

HTTP latency alerts that page on a single slow request are noise. Here is how to write alerts that wake someone up only when the user experience is actually degrading.

The default HTTP latency alert most teams ship with looks like “alert if any request takes longer than 2 seconds.” That alert pages on a single slow request from a bot, a long-tail admin endpoint, or a one-off cold start. It does not page on the thing the team actually cares about: a sustained degradation in user experience.

What’s happening

HTTP latency is a distribution, not a single number. A site can serve 99% of requests in 300 ms and still produce a handful of multi-second responses every minute — those are usually expected (slow admin queries, large file uploads, scheduled exports). Alerting on the maximum or on a single high sample produces a constant trickle of pages with no corresponding user impact.

The alert that matters is when a meaningful percentile crosses a meaningful threshold for a meaningful duration. “p95 of public-facing GET requests over 1.5 seconds for 10 minutes” describes a real degradation. The other failure mode is alerting on a global p95 across all routes — the checkout page is allowed to be slower than the home page, and a global aggregate hides where the problem is.

How to verify

Pull the histogram for the last 7 days and break it out by route:

histogram_quantile(0.95,
  sum by (route, le) (
    rate(http_request_duration_seconds_bucket{job="frontend"}[5m])
  )
)

If p95 for /api/checkout sits at 800 ms and p95 for /admin/reports sits at 4 seconds, a single global threshold cannot serve both. The admin route either needs its own rule with a higher threshold or needs to be excluded from the alert entirely.

Check the alert history against the SLO. If alerts fire while the SLO error budget is healthy and the user-facing dashboards are green, the alert is sampling something the SLO already says is acceptable.

The fix

Three rules, in order of priority:

  1. Alert on a percentile, not a single sample. histogram_quantile(0.95, ...) > 1.5 over a 10-minute window is the floor. 5-minute windows produce false positives on routine traffic spikes.
  2. Separate routes by latency budget. Public read paths, public write paths, authenticated admin paths, and background API routes each get their own rule, with thresholds derived from the 30-day baseline plus headroom.
  3. Anchor to the SLO. If the SLO is “99% of requests under 1 second”, the page-worthy condition is burning that error budget faster than the budget refills. That keeps the alert tied to user-perceived degradation, not to outlier requests.

After these changes, the alert fires when the experience is degrading. Single slow requests show up on dashboards for trend analysis, not on the pager.


Stack Harbor’s Monitoring & Support practice writes latency alerts against real route-level baselines as part of the managed operating model. If your latency pages are louder than your incidents, we can take a look.