When the cache expires, everyone shows up at once

A cache is a promise you make to your database: most of the time, I won’t bother you. You compute an expensive answer once — the day’s leaderboard, a rendered page, a pricing table that takes a heavy query to build — stash it under a key with a time-to-live, and for the next ten minutes every request reads the cheap copy. Your database sleeps. Everyone’s happy. Then the TTL expires, and for one ugly instant the promise inverts.

Here’s the failure. That popular key is being read a thousand times a second, all served from cache. At the stroke of expiry, the cache goes empty — and now all thousand of those requests miss at once. Every one of them, finding nothing cached, does the only thing it knows to do: go recompute the value from the origin. So a thousand identical expensive queries hit your database in the same heartbeat, all racing to rebuild the exact same answer that a single query could have produced. This is the cache stampede, also called the thundering herd or the dogpile. The cache didn’t just stop helping; at the worst possible moment it aimed your entire read volume straight at the thing it was protecting. And often the recompute is slow because the database is now buried under the stampede — so the herd stays herded, the rebuild takes even longer, and a ten-minute-cached endpoint tips over into an outage.

The important thing to see is that this is a different problem from the one people usually mean by “cache issues.” It isn’t about serving stale or wrong data — the value can be perfectly correct. It’s about the concurrency of the rebuild. The danger isn’t the cache being empty; it’s a thousand workers all discovering it’s empty at the same instant and all deciding, independently and reasonably, to fix it themselves. Nobody’s doing anything wrong. The stampede is emergent — the sum of a thousand locally-correct decisions made simultaneously.

⁂

Every real fix is a variation on one idea: when the value is missing, make sure only one worker rebuilds it, and give everyone else something to do besides pile on. The bluntest version is a lock — the first request to miss grabs a short-lived lock on the key, becomes the single designated rebuilder, and every other request either waits briefly for the fresh value or is handed the old value to serve while the rebuild happens. That last move — serve stale while you revalidate — is often the kindest: readers never see a spinner, the origin sees exactly one query, and the only cost is that some requests get data a few seconds past its expiry. A close cousin is request coalescing (sometimes “single-flight”): the system notices a thousand identical in-flight requests, actually runs one, and fans the single result back out to all the waiters. Same principle, less bookkeeping: dedupe the work rather than lock the key.

The more elegant fix attacks the synchronization instead of the concurrency. The stampede exists because everyone’s TTL expires at the same instant — so stop letting it. Add jitter: instead of a flat 600-second TTL on every copy, use 600 plus a small random spread, so keys drift out of lockstep and misses scatter across time instead of detonating together. A more refined version is probabilistic early expiration, where each reader rolls the dice as the TTL approaches and, with a probability that climbs the closer expiry gets, volunteers to refresh the value early — while the old one is still valid and still being served. Most readers keep using the cache; one lucky reader rebuilds ahead of the deadline; the key never actually goes cold. There’s a clean version of this in the literature worth reading if you own a hot cache.

⁂

The turn worth ending on: none of these make the work free. They relocate it. A lock makes some requests wait. Serve-stale hands some users slightly old data. Early expiration spends a little extra compute rebuilding before it was strictly necessary. Every stampede fix is really a decision about who absorbs the cost of the refresh — a waiting user, a slightly-stale reader, or an early volunteer doing eager work — and the engineering is in choosing that on purpose instead of letting a thousand simultaneous misses choose it for you. The default, “everyone rebuilds the instant it expires,” is the one option that makes everybody pay at once.