Skip to content

AWS Updated 2026-07-04

Golden AMIs without drift: building EC2 images that stay trustworthy

How to build and maintain a golden AMI pipeline on EC2 so every instance launches identical — using EC2 Image Builder, versioned images, and replacement instead of in-place patching.

A golden AMI is a pre-baked machine image with your OS hardening, agents, and runtime already installed, so instances boot ready to serve instead of running twenty minutes of user-data scripts. It only stays golden under one discipline: instances launched from it are never patched or reconfigured by hand — you rebuild the image and replace the fleet. Drift is what happens when that discipline slips, and it is the reason most golden AMI projects quietly die within a year.

What’s happening

An AMI is a frozen snapshot of a root volume plus launch metadata. The moment an instance boots from it, the instance starts diverging: a security patch applied over SSH, a config file edited during an incident, an agent upgraded by its own auto-updater. Six months later “the AMI” describes none of the running machines, and the next instance launched from it behaves differently from its siblings. That divergence is drift, and it has two sources — instances drifting away from the image, and the image drifting away from reality because nobody rebuilds it.

The fix is a pipeline, not a person. EC2 Image Builder exists for exactly this: it takes a parent image, runs your build components (packages, hardening, agents), runs test components, and produces a new versioned AMI on a schedule or on trigger. Pair it with launch templates pointing at the newest tested image, and an Auto Scaling group rollout replaces old instances with new ones. Nothing is patched in place; everything is replaced.

What to check first

  • When was the current production AMI built? If CreationDate is more than 30–45 days old, you are running unpatched images and the golden AMI is already a liability.
  • Do running instances match the image? Pick one instance and compare installed package versions against a fresh launch of the same AMI. Any difference is drift.
  • Is anyone SSHing into production? Check CloudTrail for StartSession and your bastion logs. Interactive access is where drift is born.
  • Does the launch template point at the AMI you think it does? Fleets frequently run an image two versions behind the “current” one.
  • Is the base image pinned or floating? Building from “latest Amazon Linux 2023” without recording the source AMI ID makes builds non-reproducible.

How to verify

List your own AMIs with ages, newest first:

aws ec2 describe-images --owners self \
  --query 'sort_by(Images,&CreationDate)[-10:].[ImageId,Name,CreationDate]' \
  --output table

Find which AMI each running instance actually came from:

aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,ImageId,LaunchTime]' \
  --output table

Check Image Builder pipeline health under EC2 Image Builder > Image pipelines, or:

aws imagebuilder list-image-pipelines \
  --query 'imagePipelineList[].[name,status,dateLastRun]' --output table

For drift on a live host, SSM Inventory is the honest witness — Systems Manager > Inventory shows installed applications per instance, and a diff between two instances of the same AMI exposes hand-applied changes.

The runbook

  1. Pin and record the parent image. Resolve the base AMI via the public SSM parameter (for example /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64) at build time and log the resolved ID in the image tags. You want to know exactly what you built on.
  2. Move everything repeatable into build components. Package installs, CIS hardening, CloudWatch agent, SSM Agent config, your app runtime — all of it belongs in Image Builder components (or Packer, if you prefer; the principle is identical). User data should shrink to instance-specific wiring only: fetching secrets, registering with a load balancer.
  3. Add test components that would have caught your last incident. Boot the candidate image, assert the service starts, assert the agent reports in, assert the disk layout. An untested golden AMI is just a fast way to replicate a mistake fleet-wide.
  4. Version and tag every output image. Tag with build date, parent AMI ID, pipeline version, and a status tag (candidate, approved, retired). Never overwrite; never reuse names ambiguously.
  5. Schedule rebuilds monthly at minimum, and trigger extra builds when a critical CVE lands. Image Builder supports cron schedules and can rebuild automatically when the parent image updates.
  6. Roll out by replacement. Point the launch template at the new AMI and use an ASG instance refresh — or a blue/green cutover as described in blue/green deployments on EC2 Auto Scaling groups — so old instances drain and terminate instead of being patched.
  7. Deprecate and deregister old images on a schedule. Keep the last two or three approved versions for rollback, deregister the rest, and delete their orphaned EBS snapshots — snapshot storage is a quiet, permanent line item on the bill.

When this is the wrong move

If you run three pet servers that change weekly, a full image pipeline is more machinery than the problem deserves — SSM Patch Manager plus a tight user-data script is a fair middle ground. Golden AMIs also fit poorly when your deploy artifact is already an immutable container: bake a thin host image with just the runtime and agents, and let the container carry the application. And if your team cannot yet stop SSHing into boxes to fix things, fix that habit first; a pipeline cannot outrun hands-on-keyboard drift.

Common mistakes

  • Baking secrets or credentials into the image. Anything in the AMI is in every snapshot forever. Fetch secrets at boot via an instance profile — see IAM instance profiles done right.
  • Baking instance identity — hostnames, node IDs, cached machine keys — so every clone thinks it is the original.
  • Leaving auto-updaters enabled inside the image, which reintroduces drift the pipeline was built to eliminate.
  • Treating the AMI as the backup. It is a launch artifact, not a data protection strategy; application data needs its own backup path.

Keeping it under control

  • Alarm on image age: a scheduled Lambda or Config rule that flags any production launch template referencing an AMI older than 45 days.
  • Make interactive access exceptional and audited — Session Manager with logging, no standing SSH keys.
  • Rebuild on a calendar, not on enthusiasm; the pipeline that only runs when someone remembers is already dead.
  • Diff SSM Inventory between fleet instances quarterly to catch drift early.
  • Keep the component definitions in git so image changes get reviewed like code changes.

A golden AMI pipeline is mostly a cultural commitment with some automation attached: images are rebuilt, instances are replaced, nobody fixes a server by hand and walks away. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.