Using Spot Instances for the Workloads That Can Actually Tolerate Them

Decision drivers

Whether the workload can checkpoint or resume from an arbitrary interruption point
Whether it's stateless enough to simply restart elsewhere
Sensitivity to a hard two-minute (or less) shutdown deadline
Whether Capacity Rebalancing is configured, and whether the workload tolerates overlap/gap during a rebalance event

Production premise

Spot Instances trade guaranteed availability for savings of up to 90% off On-Demand. While a Spot Instance is running, it is identical to On-Demand in every way. Same machine class, same performance, same network.

There is exactly one difference, and it only shows up when AWS needs the capacity back: AWS can reclaim the instance at that moment. You get a two-minute warning when it does.

Failure mode & inflection point

The failure is treating Spot as "just cheaper EC2" for a workload that can't actually tolerate an interruption. The lesson arrives the first time capacity is reclaimed mid-task, with no checkpoint to resume from.

Here is what the two-minute notice really is: a best-effort warning, not a guarantee. EC2's own docs are explicit that an instance can be interrupted before the warning becomes available. Design for the case where no warning arrives at all, and treat any warning you do get as extra time.

There is a separate, earlier signal meant to help — the "rebalance recommendation," which exists specifically to give more lead time. It's best-effort too. It can arrive at the same time as the two-minute notice, without meaningfully preceding it, so don't build your shutdown timing around it.

Production guardrail

Put only genuinely interruption-tolerant, checkpoint-able, or stateless workloads on Spot. Batch jobs, CI runners, and fault-tolerant distributed compute qualify. Anything that can't gracefully lose an in-progress unit of work does not.

If you use Auto Scaling Group Capacity Rebalancing to replace at-risk instances proactively, plan around both the documented default and its real edge case. The documented default sequence waits for the replacement instance to pass health checks before terminating the old one. That's the behavior to design for.

Two independent race conditions can still produce early draining, even though neither contradicts the documented sequence.

The first: the rebalance recommendation and the two-minute Spot interruption notice can collide closely enough to fire together. The second: a load balancer's target-group deregistration delay — 300 seconds by default — can outlast the 120-second Spot countdown. The old instance gets pulled from rotation before the new one is confirmed healthy.

Design the workload to tolerate a gap regardless of which path plays out. If the 120-second window feels tight, shorten the deregistration delay below it. The delay is the one variable here you actually control.