Skip to content

What an AWS architecture review actually finds: the recurring failures in real accounts

An AWS architecture review keeps finding the same eight problems: single-AZ databases, one NAT gateway, untested backups, IAM sprawl, and alarms that page nobody. Here is the list and how to self-check.

Across real AWS accounts, an architecture review keeps finding the same eight problems, and almost none of them are exotic: a database with no standby, one NAT gateway carrying all egress, backups that exist but have never been restored, IAM users with keys older than the interns, security groups open to the world, alarms wired to an inbox nobody reads, no cost accountability, and infrastructure that exists only as console clicks. If you want to know what a review of your account would say, this list is most of it — and much of it is checkable in an afternoon.

What you are deciding

An architecture review is a decision about where to spend limited engineering time: which of the account’s weaknesses will actually hurt, in what order, and what each one costs to fix. The output that matters is not a 60-page document — it is a short, ranked list of findings with a failure scenario attached to each (“if AZ us-east-1a degrades, the database is down until a restore completes; last verified restore: never”). A review without failure scenarios is a compliance artifact. A review with them is a work plan.

It is worth being honest about what a review is not. It does not make the environment better by itself; unremediated findings age like unfixed dental problems. And it reflects a point in time — accounts drift within months as people ship. The value is in forcing explicit decisions: each finding gets fixed, scheduled, or accepted in writing. Silent risk becomes chosen risk, which is the actual upgrade.

What to check first

The five findings that appear most often, ranked roughly by how badly they end:

  • Single-AZ database. RDS without Multi-AZ, or a self-managed database on one EC2 instance. One AZ event or one failed instance and the business is down for the length of a restore.
  • Untested backups. Snapshots exist; nobody has ever restored one, timed it, or checked the application starts against it.
  • One NAT gateway. Every private subnet routes egress through a single NAT in a single AZ — an AZ problem becomes a whole-environment outage.
  • IAM sprawl. Human users with long-lived access keys, AdministratorAccess attached broadly, no MFA enforcement, keys unused for months but still active.
  • Alarms that page nobody. CloudWatch alarms firing into an SNS topic with no subscribers, or an email alias nobody owns.

How to verify

Most of the top findings fall out of a handful of commands:

# Databases without a standby
aws rds describe-db-instances \
  --query 'DBInstances[].[DBInstanceIdentifier,MultiAZ,DBInstanceClass]' --output table

# How many NAT gateways, and in which AZs?
aws ec2 describe-nat-gateways \
  --query 'NatGateways[].[NatGatewayId,SubnetId,State]' --output table

# Security groups open to the internet
aws ec2 describe-security-groups \
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 \
  --query 'SecurityGroups[].[GroupId,GroupName]' --output table

# IAM credential hygiene: age and last use of every key
aws iam generate-credential-report
aws iam get-credential-report --query 'Content' --output text | base64 -d

# Do alarm topics have any subscribers at all?
aws sns list-subscriptions --query 'Subscriptions[].[TopicArn,Protocol,Endpoint]' --output table

# Has a restore ever been run?
aws backup list-restore-jobs \
  --query 'RestoreJobs[].[CreationDate,Status]' --output table

In the console, Trusted Advisor (checks vary by support plan — see https://aws.amazon.com/premiumsupport/) and Security Hub > Findings will surface a chunk of this automatically if they are enabled — whether they are enabled is itself a finding.

The approach

A review worth paying for, or running internally, moves through six passes:

  1. Reliability pass. Single points of failure: single-AZ data stores, single NAT, single instance behind no load balancer, health checks that test TCP instead of the application. The multi-AZ decision logic in Multi-AZ vs multi-region is the reference for the data tier.
  2. Data protection pass. Backup coverage against an actual inventory, retention versus stated obligations, at least one timed restore, and whether encryption at rest is on for the stores that need it.
  3. Security pass. IAM credential report, root account usage and MFA, public exposure (security groups, S3 bucket policies), and whether CloudTrail and GuardDuty are on in all regions.
  4. Observability pass. What would page a human today if the main database filled its disk? Walk real failure scenarios against the existing alarms and find the silent ones.
  5. Cost pass. Utilization versus provisioned capacity, orphaned resources, commitment coverage, and whether anyone reviews the bill monthly.
  6. Rank and decide. Each finding gets severity, effort, and a failure scenario. The output is a queue: fix now, schedule, or accept in writing — with names attached.

When this is the wrong move

A full review is premature if the environment is about to be rebuilt — review the design of the new environment instead of auditing the condemned one. It is also the wrong tool for a live incident; reviews are calm-weather work. And if the account has an obvious, known, unfixed critical problem (say, everyone already knows there are no backups), spend the money fixing that before paying anyone to rediscover it.

Common mistakes

  • Treating the review report as the deliverable. The deliverable is the remediation queue with owners and dates; the PDF is packaging.
  • Fixing findings in discovery order instead of severity order. Teams burn weeks on tagging hygiene while the single-AZ production database waits.
  • Running it once and never again. Accounts drift; an annual cadence, or a review after every major launch, keeps the findings list honest.
  • Letting the reviewer grade their own homework. If the team or vendor that built the environment runs the review, the incentives lean toward gentle findings.

Keeping it under control

  • Keep a living findings register — severity, owner, status — instead of point-in-time reports that rot in a shared drive.
  • Automate the cheap checks: Security Hub standards, AWS Config rules, and a monthly scripted run of the commands above.
  • Re-verify the two claims that decay fastest: restore tested, alarms page a human. Quarterly, with evidence.
  • Require an architecture sanity check before each significant launch, not after it.
  • Track the boring metric that predicts everything else: number of criticals open, and for how long.

Most AWS environments do not fail in original ways — they fail on this list. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.