Skip to content

AWS Updated 2026-10-23

AWS 24/7 monitoring: what a real overnight rotation takes (and why a dashboard is not one)

AWS 24/7 monitoring works only when each alarm wakes a trained person who has a runbook, the access to act, and a backup if they miss the page. Use this checklist to judge what a provider really offers before you sign.

Buying 24/7 monitoring for an AWS account is mostly a question about people, and this article helps you check that the people are really there. Picture an alarm firing at 03:14. Someone qualified has to be woken within minutes, open a runbook (the written steps for that alarm), and have the access to fix the problem. If they do not answer, a second person must be paged. A dashboard does none of that on its own, so when a provider offers “24/7 monitoring”, ask about the rotation, the escalation chain and their authority to act before you look at the graphs.

What’s happening

Monitoring splits into three layers, and vendors love to sell the first while implying the third:

  • Telemetry — metrics, logs, and traces being collected. CloudWatch does this out of the box; it is table stakes, not a service.
  • Alerting — thresholds and anomaly detection that turn telemetry into pages. Good alerting is real engineering work (tuning, deduplication, severity mapping), but it still ends at a notification.
  • Response — a staffed rotation that acknowledges the page, follows a runbook, mitigates, and escalates. This is the only layer that shortens outages.

A genuine overnight rotation has hard requirements that are easy to verify and expensive to fake. You need at least two people available in any given hour (primary and escalation — one person is a single point of failure who also sleeps), and a paging tool with acknowledgment timeouts, not email or Slack. You need per-alarm runbooks so the 03:00 responder is executing, not researching. You also need write access sufficient to mitigate (restart a service, fail over a database, scale a group); and a handoff process so daytime knows what happened overnight. Every failed alert should also raise the question CloudWatch cannot answer alone: is it us or is AWS? — which is what https://health.aws.amazon.com/ is for.

The economics follow from the staffing. Covering one seat around the clock takes several people once you account for nights, weekends, vacations, and turnover — that is the actual cost driver behind 24/7 pricing, and why credible offers cost real money while “24/7 monitoring included free” reliably means telemetry only.

What to check first

  1. Where do your alarms notify right now? An SNS topic with zero subscribers, a muted Slack channel, or one engineer’s personal phone are the three classic dead ends.
  2. What is your alarm-to-acknowledgment time overnight? If you cannot measure it, it has never been tested.
  3. Do runbooks exist for your top five alarms? Ask for the replica-lag runbook; watch what happens.
  4. Who is the escalation if the primary sleeps through the page? “Nobody” is the honest answer in most shops.
  5. Can the overnight responder actually do anything? Read-only access at night means the outage waits for morning regardless.

How to verify

Test the pipeline end to end, not the pieces:

# Every alarm and where it notifies
aws cloudwatch describe-alarms \
  --query 'MetricAlarms[].[AlarmName,StateValue,AlarmActions[0]]' --output table

# Does the SNS topic have live subscribers?
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ca-central-1:123456789012:prod-alerts

# Fire a controlled drill — the alarm returns to its real state
# at the next metric evaluation
aws cloudwatch set-alarm-state \
  --alarm-name prod-alb-5xx-rate \
  --state-value ALARM \
  --state-reason "overnight drill"

Run the drill at 22:00, unannounced to the rotation (announced to management). Measure minutes to acknowledgment and minutes to a human reading the correct runbook. In the console, CloudWatch > Alarms > History shows the state transitions with timestamps, which gives you the objective half of the measurement. The CloudWatch documentation at https://docs.aws.amazon.com/AmazonCloudWatch/ covers alarm states and actions if you are building the alerting layer yourself.

The runbook

  1. Fix the alarm set before staffing anything. A rotation paged for noise burns out and starts ignoring pages; that is how real incidents get slept through. Start from a deliberate baseline like the CloudWatch alarms a production SaaS needs and delete every alarm nobody would act on at night.
  2. Route pages through a tool with escalation, acknowledgment, and timeouts. SNS to email is where pages go to die overnight. SNS should feed a paging system that calls the primary, waits, then calls the escalation.
  3. Write the runbook for each pageable alarm. Symptom, first checks, mitigation steps with exact commands, rollback, and the stop-line where the responder must escalate rather than improvise. One page per alarm; written by whoever knows the system best, executed by whoever is awake.
  4. Give the rotation mitigation access, scoped. Restart, scale, fail over — through roles with least privilege, so the 03:00 blast radius is bounded. The same discipline as instance roles done properly, applied to humans.
  5. Staff it honestly or buy it. If you have fewer than roughly six engineers willing to carry a pager, a sustainable in-house 24/7 rotation is arithmetic you cannot make work — buy the overnight layer from a provider that runs a real rotation, and verify it with the same drill.
  6. Drill quarterly and review every page monthly. Each page is either actionable (keep), noise (tune or delete), or a miss (add coverage). The alarm set is a living artifact.

When this is the wrong move

If your customers are all in one timezone and your revenue stops at 18:00, paying for 24/7 response is buying insurance against an empty night — business-hours coverage with a best-effort overnight page may be the honest fit, priced accordingly. If your application falls over nightly from memory leaks, a rotation just documents the suffering; fix the application first. And if alarms currently fire dozens of times a day, staffing a rotation before tuning is paying people to be tortured.

Common mistakes

  • Equating a monitoring stack with monitoring coverage. Prometheus, Grafana, and CloudWatch dashboards are telemetry; nobody is on the hook.
  • One hero on permanent informal call. It works until they burn out, take a vacation, or quit — usually in that order.
  • Alarms tuned once at launch, never revisited. Traffic doubles, thresholds do not; pages become noise; noise becomes ignored.
  • No authority at night. A responder who must wake the CTO for permission to restart a service adds a hop to every incident.

Keeping it under control

  • Track two numbers monthly: median time-to-acknowledge and pages-per-week. The first should be minutes; the second should trend down.
  • Keep runbooks in version control next to the infrastructure code, reviewed when the architecture changes.
  • Rotate the drill: different alarm, different hour, each quarter.
  • Re-verify subscriber lists after every personnel change — dead phone numbers in the escalation chain are silent failures.

The dashboard is the easiest part of 24/7 monitoring and the only part most vendors show you; the rotation behind it is the product. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.