Skip to content

AWS Updated 2026-07-04

AWS CloudWatch monitoring setup that catches real incidents (not 400 unread alarms)

A working AWS CloudWatch monitoring setup is a dozen symptom-based alarms routed to someone who will act — not hundreds of unread notifications. Here is how to build one that detects incidents before customers do.

A CloudWatch monitoring setup that catches real incidents is smaller than most teams expect: roughly a dozen alarms on customer-visible symptoms — error rate, latency, saturation, failed backups — routed through SNS to a path that actually wakes someone, with everything nobody acts on deleted. Four hundred alarms feeding an unread inbox is not monitoring, it is decoration, and the customers become the detection layer. The setup is a focused day of work; the hard part is the discipline of keeping every alarm actionable.

What’s happening

Accounts do not reach 400 alarms on purpose. They get there through console defaults accepted during setup, Terraform modules copied with their alarm blocks intact, one-off alarms created during an old incident and never removed, and every new service adding alarms while nobody deletes any. The result is alarm fatigue, and it has a well-documented mechanism: once a meaningful fraction of notifications are noise, humans stop reading all of them, including the one that mattered.

CloudWatch itself gives you three building blocks. Metric alarms watch one threshold on one metric or math expression. Composite alarms combine several alarms with boolean logic, useful for suppressing downstream noise when an upstream dependency is already alarming. Anomaly detection alarms on deviation from a learned band, which is the right tool for “traffic dropped 40% below normal” — a failure no static threshold catches. The full reference is in the CloudWatch documentation.

The delivery chain is where most setups silently break: alarm, then SNS topic, then subscription, then a human. A topic with zero subscribers, a Slack channel someone muted, or an email address of a person who left — every one of these produces an account that looks monitored and is not. The other structural failure is alarming on causes instead of symptoms. CPU at 90% is a cause and might be perfectly fine; a 5xx rate at 2% is a symptom and never is.

What to check first

  • How many alarms sit in ALARM or INSUFFICIENT_DATA right now and have for weeks — that is your noise floor, and everyone has already learned to ignore it.
  • Where alarm actions point. SNS topics with no subscribers are the most common finding in account reviews.
  • Whether anything alarms on user-visible symptoms (ALB 5xx percentage, p95 latency, queue age) rather than only host metrics.
  • Whether an after-hours delivery path exists at all, or whether everything terminates in email.
  • When an alarm last caused a human to do something. If nobody remembers, the setup has already failed once, quietly.

How to verify

Count the noise and trace the delivery path:

# How many alarms are firing right now?
aws cloudwatch describe-alarms --state-value ALARM \
  --query 'length(MetricAlarms)'

# Every alarm, its action, and its state
aws cloudwatch describe-alarms \
  --query 'MetricAlarms[].[AlarmName,AlarmActions[0],StateValue]' \
  --output table

# Does the topic those actions point at have live subscribers?
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ca-central-1:123456789012:ops-alerts

In the console, CloudWatch > Alarms > All alarms, filtered by state, gives the same picture. Then run a live drill:

aws cloudwatch set-alarm-state --alarm-name prod-alb-5xx-rate \
  --state-value ALARM --state-reason "paging drill"

If nothing reaches a phone within a couple of minutes, the monitoring setup does not exist in any way that matters, whatever the alarm count says.

The runbook

  1. Inventory and cut. Export the alarm list and delete anything with no action, no owner, or no state change acted on in 90 days. Deleting an alarm nobody reads loses nothing and buys back attention.
  2. Alarm on symptoms first. ALB 5xx percentage (rate, not absolute count), target p95 latency, unhealthy host count, queue age, replica lag, backup job failures, certificate expiry, disk fill. The eleven-alarm CloudWatch baseline is a copy-pasteable starting set with defended thresholds.
  3. Split severities into two paths. Two SNS topics: one that pages (PagerDuty, Opsgenie, or an SMS/call path), one that files tickets. An alarm that does not justify waking someone at 2 a.m. does not belong on the paging topic.
  4. Tune out flapping. Use DatapointsToAlarm of 3 out of 5 evaluation periods for anything paging, so a single scrape blip does not fire. Set TreatMissingData deliberately per alarm instead of accepting the default.
  5. Cover the silent failures. For heartbeat-style metrics (cron results, backup counts), treat missing data as breaching — a dead pipeline should page, not vanish. Add one anomaly-detection alarm on request count to catch “traffic fell off a cliff.”
  6. Drill the delivery path quarterly. set-alarm-state during a maintenance window, and time how long until a human acknowledges.
  7. Review weekly, fifteen minutes. For every alarm that fired: was it actionable? If not, retune it or delete it that day. This single habit is what separates accounts that stay clean from accounts that drift back to 400.

When this is the wrong move

If nobody will answer the page — no rotation, no agreed response expectation, no after-hours coverage — better alarms only document outages in real time. Fix ownership before tuning thresholds. At the other end of the scale, a single low-stakes site does not need this machinery; an external uptime check may honestly be enough. And CloudWatch alarms tell you that something broke, not which downstream call is slow — if that is the recurring question, you are shopping for tracing (X-Ray or an APM), not more alarms.

Common mistakes

  • Alarming on causes (CPU, memory) while nothing watches the symptoms customers actually feel.
  • One SNS topic for everything, so a cosmetic warning and a production outage get identical treatment.
  • TreatMissingData left at its default, so a metric pipeline that dies looks like an account with no problems.
  • Email as the only after-hours path — median time-to-read measured in hours, not minutes.

Keeping it under control

  • Every alarm has a named owner and one line of runbook: what to do when it fires.
  • Adding a service to production includes adding its alarms in the same change, via the same infrastructure-as-code.
  • Quarterly paging drills, with the acknowledgment time recorded.
  • A standing rule that any alarm judged non-actionable in the weekly review is fixed or deleted the same day.
  • Track pages per week as a health metric; a rising trend means thresholds are drifting or reliability is.

A dozen well-routed alarms beat four hundred ignored ones every single time, but they still assume someone picks up. If alerts fire into an empty room after hours, that gap is exactly what 24/7 Monitoring & Response closes — book a monitoring and response review.