Skip to content

AWS Updated 2026-03-11

The CloudWatch alarm baseline every production SaaS needs — copy-pasteable CloudFormation

The eleven CloudWatch alarms that catch the failure modes a production SaaS actually experiences. ALB 5xx rate, target health, RDS connections and replica lag, ASG capacity, NAT GW bandwidth, certificate expiry. Here is the set.

A new production workload goes live on AWS. The architecture is sound — ALB to ECS to Aurora, autoscaling tied to CPU, NAT Gateway for egress, ACM certificates on the listener. Three weeks in, an incident: the Aurora reader fell behind by 90 seconds, the application kept routing read queries to it, customers saw stale data for an hour, the on-call learned about it from a customer ticket. The CloudWatch alarms that exist on the account fire on instance CPU at 90%, ALB target unhealthy, and RDS storage at 85%. None of them caught the replica lag. The baseline below is the set of eleven alarms that catches the failure modes a production SaaS actually experiences — assembled from the post-mortems of the ones that did not.

What’s happening

CloudWatch alarms come in three flavors that map to different signals:

  • Metric alarms — threshold on a single metric (5xx rate above 1%, CPU above 80%).
  • Composite alarms — boolean over multiple metric alarms (only fire if both A and B are in ALARM state).
  • Anomaly detection — band around a learned baseline (request count drops 30% below normal for the hour).

The baseline below leans on metric alarms — they are predictable, easy to reason about, and the threshold values are well-understood. Composite alarms come in for one specific case (suppress NAT GW alarms during the maintenance window). Anomaly detection is useful for request count and queue depth but takes weeks to train and is not in the baseline.

The eleven alarms cover six failure surfaces:

  1. Application layer — ALB 5xx rate, target group unhealthy host count, target response time p95.
  2. Compute layer — ASG insufficient capacity, ECS service deployment failure.
  3. Data layer — RDS connection count, RDS CPU, RDS replica lag.
  4. Network egress — NAT Gateway bandwidth approaching limit.
  5. TLS — ACM certificate expiry approaching 30 days.
  6. Backups — AWS Backup job failure (covered in the AWS Backup article in this series).

Each one has a threshold informed by real production patterns, not the AWS console defaults.

How to verify

The check is binary: do these alarms exist, in the production account, on the production resources, with notifications going to a place a human will see them within minutes? Run the inventory:

aws cloudwatch describe-alarms \
  --query 'MetricAlarms[].[AlarmName,MetricName,Threshold,StateValue,AlarmActions[0]]' \
  --output table

The columns to scan: every production load balancer should have an HTTPCode_ELB_5XX_Count alarm and a UnHealthyHostCount alarm. Every Aurora cluster should have DatabaseConnections, CPUUtilization, and AuroraReplicaLag. Every NAT Gateway should have BytesOutToDestination against the bandwidth quota. Every ACM certificate should have DaysToExpiry below 30.

A common gap: the alarms exist but go to an SNS topic that has no subscribers, or a Slack channel that is muted. Test the path end-to-end:

aws cloudwatch set-alarm-state \
  --alarm-name prod-alb-5xx-rate \
  --state-value ALARM \
  --state-reason "drill"

The notification should land in the on-call channel within seconds. If it does not, the alarm is decoration.

The fix

The CloudFormation snippet below provisions the eleven alarms against tagged resources. The thresholds reflect production patterns we have validated across managed customers; tune them for the specific workload. Replace ${LbFullName}, ${TgArn}, ${ClusterId}, ${NatGatewayId}, ${CertArn}, and ${SnsTopicArn} with the actual resource references in the stack.

Resources:
  Alb5xxRate:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-alb-5xx-rate
      AlarmDescription: ALB 5xx error rate above 1%
      Metrics:
        - Id: e1
          Expression: "100 * (m_5xx / m_total)"
          Label: "5xx percentage"
        - Id: m_5xx
          MetricStat:
            Metric:
              Namespace: AWS/ApplicationELB
              MetricName: HTTPCode_ELB_5XX_Count
              Dimensions:
                - Name: LoadBalancer
                  Value: !Ref LbFullName
            Period: 60
            Stat: Sum
          ReturnData: false
        - Id: m_total
          MetricStat:
            Metric:
              Namespace: AWS/ApplicationELB
              MetricName: RequestCount
              Dimensions:
                - Name: LoadBalancer
                  Value: !Ref LbFullName
            Period: 60
            Stat: Sum
          ReturnData: false
      Threshold: 1
      ComparisonOperator: GreaterThanThreshold
      EvaluationPeriods: 5
      DatapointsToAlarm: 3
      TreatMissingData: notBreaching
      AlarmActions: [!Ref SnsTopicArn]

  TargetUnhealthy:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-tg-unhealthy
      MetricName: UnHealthyHostCount
      Namespace: AWS/ApplicationELB
      Dimensions:
        - Name: TargetGroup
          Value: !Ref TgArn
        - Name: LoadBalancer
          Value: !Ref LbFullName
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 3
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  TargetLatencyP95:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-tg-latency-p95
      MetricName: TargetResponseTime
      Namespace: AWS/ApplicationELB
      Dimensions:
        - Name: TargetGroup
          Value: !Ref TgArn
        - Name: LoadBalancer
          Value: !Ref LbFullName
      ExtendedStatistic: p95
      Period: 60
      EvaluationPeriods: 5
      DatapointsToAlarm: 3
      Threshold: 1.0
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  RdsConnections:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-rds-connections
      MetricName: DatabaseConnections
      Namespace: AWS/RDS
      Dimensions:
        - Name: DBClusterIdentifier
          Value: !Ref ClusterId
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 5
      Threshold: 80
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  RdsCpu:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-rds-cpu
      MetricName: CPUUtilization
      Namespace: AWS/RDS
      Dimensions:
        - Name: DBClusterIdentifier
          Value: !Ref ClusterId
      ExtendedStatistic: p95
      Period: 300
      EvaluationPeriods: 3
      Threshold: 80
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  RdsReplicaLag:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-rds-replica-lag
      MetricName: AuroraReplicaLag
      Namespace: AWS/RDS
      Dimensions:
        - Name: DBClusterIdentifier
          Value: !Ref ClusterId
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 3
      Threshold: 5000
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  AsgInsufficientCapacity:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-asg-insufficient-capacity
      MetricName: GroupInServiceInstances
      Namespace: AWS/AutoScaling
      Dimensions:
        - Name: AutoScalingGroupName
          Value: !Ref AsgName
      Statistic: Minimum
      Period: 300
      EvaluationPeriods: 2
      Threshold: 1
      ComparisonOperator: LessThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  EcsServiceFailure:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-ecs-deployment-failure
      MetricName: ServiceCount
      Namespace: AWS/ECS
      Dimensions:
        - Name: ServiceName
          Value: !Ref EcsService
        - Name: ClusterName
          Value: !Ref EcsCluster
      Statistic: Minimum
      Period: 60
      EvaluationPeriods: 3
      Threshold: 1
      ComparisonOperator: LessThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  NatBandwidth:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-nat-bandwidth
      MetricName: BytesOutToDestination
      Namespace: AWS/NATGateway
      Dimensions:
        - Name: NatGatewayId
          Value: !Ref NatGatewayId
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 5
      Threshold: 5500000000
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  CertExpiry:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-cert-expiry-30d
      MetricName: DaysToExpiry
      Namespace: AWS/CertificateManager
      Dimensions:
        - Name: CertificateArn
          Value: !Ref CertArn
      Statistic: Minimum
      Period: 86400
      EvaluationPeriods: 1
      Threshold: 30
      ComparisonOperator: LessThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

  BackupJobFailure:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: prod-backup-job-failure
      MetricName: NumberOfBackupJobsFailed
      Namespace: AWS/Backup
      Dimensions:
        - Name: BackupVaultName
          Value: !Ref BackupVaultName
      Statistic: Sum
      Period: 3600
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref SnsTopicArn]

A few of the threshold choices need defending:

  • ALB 5xx at 1% — not 5%, not 0.1%. A healthy web app sits at 0.0% baseline 5xx; 1% over five minutes means real customer impact. The math expression in the alarm avoids the common mistake of alarming on absolute 5xx count, which fires during traffic spikes that are actually healthy.
  • RDS replica lag at 5 seconds — Aurora reader lag should sit at sub-second baseline. Five seconds is already a customer-visible problem for read-after-write workloads; the alarm at three minutes of breach is the gut-check window.
  • NAT GW at 5.5 GB/min — the NAT Gateway bandwidth quota is 100 Gbps total but per-flow constraints kick in well below that. The threshold catches sustained egress that signals either a data exfiltration event or a misconfigured workload that should be on a VPC Endpoint.
  • TG latency p95 at 1 second — workload-specific. For a customer-facing API this should be tighter (200 ms is more typical); for a background processing service the threshold is higher.

Three rules across every alarm baseline we deploy:

Notifications go somewhere a human sees them within minutes. An SNS topic with no subscribers is decoration. Email is too slow; PagerDuty or an equivalent paging path on critical alarms.

Test the alarm path quarterly. set-alarm-state ALARM on a chosen alarm during a maintenance window. If the page does not land, the alarm is broken.

Tune thresholds after one full business cycle. The defaults above are starting points; real workloads have shapes and the threshold has to match the shape, not the catalog.

The baseline takes a focused day to deploy, an hour every quarter to tune, and catches the failure modes that show up in real post-mortems. The eleven alarms are not the limit — they are the floor.

Stack Harbor runs IAM cleanup, CloudWatch baseline, AWS Backup, and the cost-control rhythm as part of Managed AWS and ongoing Managed Cloud Ops. If the alarms on your account fire on the wrong things or do not fire at all, start with an infrastructure consult.