Skip to content

AWS Updated 2026-07-04

Rotating and eliminating IAM access keys: finding every place a key hides

How to rotate IAM access keys without an outage — and how to find every place a key hides first: CI secrets, .env files, containers, cron jobs, and third-party tools.

Rotating an IAM access key safely is a five-step dance — create the second key, deploy it everywhere the old one lives, confirm the old key has gone silent, deactivate it, then delete it — and the entire risk is in the word “everywhere.” The AWS side takes minutes; finding every .env file, CI secret, container image, and forgotten cron job holding the old key is the actual work. Better still, most keys can be eliminated rather than rotated, because a role can usually stand where the key stands today.

What’s happening

An IAM user can hold two access keys at once, which exists precisely to make zero-downtime rotation possible: both keys work simultaneously during the swap. Keys have no expiry — a key created in 2019 authenticates today unless someone acts — so rotation is purely an operational discipline, and the industry default of 90 days exists because leaked keys are typically exploited within hours, not months.

What makes rotation risky is that AWS cannot tell you where a key is stored, only where it was last used from. get-access-key-last-used reveals the service and region of the most recent call, and CloudTrail records the source IP of every request — those are your tracing tools. The hiding places are depressingly consistent across environments: developer laptops (~/.aws/credentials), repository history (a .env committed once in 2021 and “removed” later is still in Git history), CI/CD secret stores, container images that baked credentials at build time, Lambda environment variables, on-premises cron jobs and backup scripts, third-party SaaS integrations (monitoring tools, backup vendors, email senders using SES SMTP credentials), and configuration management systems. Every one of these is a place the old key keeps working from until you find it — or stops working, loudly, in production, when you deactivate the key without finding it.

What to check first

  • The credential report — one CSV with every user, key age, and last-used date; keys unused for 90+ days can be deactivated immediately with near-zero risk.
  • Where the key calls from. CloudTrail source IPs tell you whether usage comes from your VPC, your office, a CI provider’s ranges, or somewhere you cannot name.
  • Whether the workload could use a role instead. If the caller runs on EC2, ECS, Lambda, or a major CI platform, elimination beats rotation — see our instance profile guide.
  • Git history. git log -p -S AKIA across your repos finds keys that were committed and “deleted” — those keys are burned and need rotation regardless.

How to verify

# Full inventory: every user, key age, last-used
aws iam generate-credential-report
aws iam get-credential-report --query 'Content' --output text | base64 -d > creds.csv

# Where and when was this specific key last used?
aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE

# Which IPs is the key calling from? (management events)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --max-results 50 \
  --query 'Events[].CloudTrailEvent' --output text | grep -o '"sourceIPAddress":"[^"]*"' | sort | uniq -c

Console path: IAM > Users > (user) > Security credentials shows both key slots, their ages, and last-used. Note that last-used data covers most but not every AWS service call, and CloudTrail Event history holds 90 days of management events — data-plane calls like s3:GetObject need data events enabled to trace.

The runbook

  1. Try elimination before rotation. EC2 gets an instance profile, ECS a task role, Lambda already has an execution role, GitHub Actions gets OIDC federation. A key replaced by a role never needs this runbook again.
  2. Create the second key. aws iam create-access-key --user-name svc-backup. If both slots are full, the older inactive key must be deleted first — confirm it is genuinely dead before you do.
  3. Deploy the new key to every location on your inventory. Update the CI secret store, the server’s credentials file, the application’s secret in Secrets Manager or Parameter Store (see our comparison of the two) — and record each location as you touch it; this list is the asset the next rotation inherits.
  4. Watch the old key go silent. Give it a full business cycle — a week at minimum, a month if monthly batch jobs exist — checking get-access-key-last-used for movement.
  5. Deactivate, then wait again. aws iam update-access-key --user-name svc-backup --access-key-id AKIA... --status Inactive. Deactivation is reversible in seconds; deletion is not. Anything that breaks now identifies a location your inventory missed.
  6. Delete. aws iam delete-access-key --user-name svc-backup --access-key-id AKIA... once silence has held through deactivation.

When this is the wrong move

If the key is currently exposed — committed to a public repo, pasted in a ticket, flagged by AWS or GuardDuty — do not run the graceful dance. Deactivate the key immediately, accept the breakage, review CloudTrail for what the key did, and involve AWS Support if there are signs of abuse. Graceful rotation is for hygiene; exposure is an incident.

Common mistakes

  • Deleting instead of deactivating first — deactivation is the reversible dress rehearsal, deletion is forever.
  • Rotating the key but not the derived SES SMTP credentials that were generated from it.
  • Trusting “last used: never” over too short a window — quarterly and yearly jobs use keys too.
  • Cleaning the current code but not Git history; a key that was ever committed is compromised until rotated.

Prevention

  • Alert on CreateAccessKey events so every new long-lived credential starts with an owner and a rotation date.
  • Run the credential report monthly and page on any key crossing your maximum age.
  • Keep the where-does-this-key-live inventory next to each remaining user; it turns the next rotation from archaeology into a checklist.
  • Shrink the key population continuously — every migration to roles or OIDC removes a rotation from the calendar permanently.

The teams that find rotation painful are the ones doing it from scratch each time; the teams that find it boring keep an inventory and shrink it. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.