Skip to content

AWS Updated 2026-07-04

EC2 spot interruptions: the two-minute warning and surviving it

EC2 spot instances get a two-minute interruption warning before AWS reclaims them. Here is how to catch the notice, drain work in time, and design so an interruption is a shrug instead of an outage.

When AWS needs its capacity back, a spot instance gets a two-minute warning, delivered two ways: an instance-action document in the instance metadata service and an EC2 Spot Instance Interruption Warning event on EventBridge. Surviving interruptions is not about avoiding them — you cannot — it is about catching that notice within seconds, draining work inside the window, and running enough instance-type diversity that replacements launch faster than reclaims land. Teams that do those three things run large spot fleets for years; teams that skip them conclude spot is unreliable, at full on-demand price.

What’s happening

Spot capacity is the spare hardware of each instance pool, sold at a steep discount (historically up to about 90% off on-demand, varying by pool and time) with the condition that EC2 can take it back when on-demand or reserved customers need it. An interruption is scoped to a capacity pool — one instance type in one AZ — which is why diversification works: reclaims hit pools, not your whole fleet, unless your whole fleet is one pool.

Two signals precede a reclaim. The rebalance recommendation arrives first (sometimes only barely first) and means the pool is at elevated risk — it is the cue for proactive replacement, and it is what Auto Scaling’s Capacity Rebalancing feature acts on. The interruption notice is the hard deadline: two minutes until the instance is terminated, stopped, or hibernated, whichever behaviour the request specified. Two minutes is what AWS emits; treat it as the most time you will get and keep the workload crash-tolerant anyway — a spot fleet is not the place for the only copy of anything, a lesson that also applies to instance store volumes.

What to check first

  • Are you actually catching the warning? If nothing polls IMDS and no EventBridge rule exists, the two minutes are being spent doing nothing.
  • How concentrated are your pools? One instance type in one AZ is the maximally fragile shape. Count your type-times-AZ combinations.
  • What is the interruption behaviour on the requests? Terminate is the default; stop and hibernate exist for workloads that resume.
  • What do the historical interruption rates look like? The Spot Instance Advisor publishes frequency bands per type and region — some pools sit under 5%, others churn constantly.
  • Is anything stateful on the fleet? Sessions, local queues, half-written files — inventory what dies with an instance.

How to verify

Poll for both signals from the instance (IMDSv2):

TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Hard two-minute notice: 404 = safe, JSON with "action" and "time" = draining time
curl -s -o /dev/null -w "%{http_code}\n" \
  -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/spot/instance-action

# Early-warning rebalance recommendation
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/events/recommendations/rebalance

Poll every five seconds — a sixty-second poll interval spends half the warning asleep. Account-side, an EventBridge rule catches interruptions fleet-wide:

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Spot Instance Interruption Warning"]
}

Point it at SNS or a Lambda that logs which pools are being reclaimed — that history tells you which types to de-emphasize. Spot mechanics and interruption behaviours are documented at https://docs.aws.amazon.com/ec2/.

The runbook

  1. Deploy an interruption handler before you deploy spot. On Kubernetes, that is the AWS Node Termination Handler (or Karpenter’s native handling), which cordons and drains the node on either signal. On plain ASGs, a small systemd-run poller that takes the instance out of rotation and flushes work is enough. Nothing else on this list matters if nobody hears the warning.
  2. Make two minutes enough. Deregister from the load balancer first (connection draining eats most of the window — set deregistration delay accordingly), stop accepting new work, checkpoint or requeue in-flight jobs, flush logs and metrics. Jobs longer than the window need external checkpoints (S3, a queue with visibility timeouts) so a successor resumes instead of restarting.
  3. Diversify pools aggressively. Six or more instance types across three AZs is a reasonable floor for a serious fleet. Use ASG mixed instances or attribute-based instance type selection with the price-capacity-optimized allocation strategy, which steers toward pools least likely to be reclaimed — the setup is covered in mixed instances policies.
  4. Turn on Capacity Rebalancing for ASG spot. It reacts to the rebalance recommendation by launching a replacement before the interruption lands, so the fleet size dips less. Accept the small cost of overlap while old and new instances coexist.
  5. Keep a base of on-demand for the floor. The pattern that works in production: enough on-demand (or reserved) capacity to serve degraded-but-alive traffic, spot for everything above it. How much floor is a cost-risk decision, weighed alongside RIs and Savings Plans.
  6. Choose stop or hibernate behaviour only when resumption is real. Stopped spot instances resume when capacity returns — useful for long batch jobs — but you keep paying for EBS while they wait, and there is no promise of when the pool recovers.

When this is the wrong move

Spot is the wrong substrate for stateful singletons, quorum members (databases, ZooKeeper, etcd), and anything with a hard latency SLO that cannot tolerate a capacity dip. It is also the wrong economics for steady 24/7 baseline load — commitment discounts cover that more predictably. And if a team has no bandwidth to build interruption handling, spot savings evaporate in the first incident; run on-demand until the engineering exists.

Common mistakes

  • Running spot with no handler and discovering the two-minute warning existed only after reading the post-mortem.
  • One instance type, one AZ — a single pool reclaim takes the entire fleet at once.
  • Polling IMDS once a minute, spending most of the warning window blind.
  • Treating the rebalance recommendation as an interruption and thrashing the fleet — it is a risk signal, not a deadline; act on it by pre-launching, not by panicking.

Keeping it under control

  • Track interruptions per pool via EventBridge and prune chronically reclaimed types from the mix.
  • Load-test the drain path: kill an instance with the handler running and measure whether requests actually complete.
  • Review the fleet’s spot/on-demand ratio quarterly against actual interruption history, not fear.
  • Keep AMIs and bootstrap fast; a fleet that boots in 90 seconds heals faster than any warning window.
  • Alarm on fleet capacity dropping below the on-demand floor — that is the signal diversification has run out.

Handled properly, interruptions become a background rhythm — instances leave politely, replacements arrive, and the finance dashboard is the only place anyone notices spot at all. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.