Here’s a failure that feels like it breaks cause and effect. Your recommendations service goes down — a peripheral, nice-to-have feature — and within a minute users can’t log in. Login never called recommendations. The two share no code path a reader could point to. So how did one drag down the other?
They shared a resource. And a shared resource is a shared fate.
Most services run on a finite pool of something scarce: worker threads, database connections, memory, in-flight request slots. Under normal load the pool is plenty. The dangerous case isn’t a dependency that fails — a fast failure returns an error and frees the slot. The dangerous case is a dependency that gets slow. A downstream call that used to take 20 milliseconds now takes 20 seconds, and while it hangs, it holds its slot in the pool. Requests bound for that sick dependency queue up, each one clutching a slot, waiting. The pool drains. And now every request — including the ones headed for perfectly healthy dependencies like your auth database — waits for a slot that never comes free. One slow corner has starved the entire building. This is the mechanism behind connection-pool exhaustion, seen from the blast-radius side.
The fix is old enough to predate computers. Ships are built with their hulls divided into watertight compartments, so a breach floods one section instead of sinking the vessel. The walls are called bulkheads, and Michael Nygard borrowed the name for software in Release It!
The pattern is exactly the ship’s: give each dependency — or each class of work — its own bounded slice of resources. Its own thread pool. Its own connection limit. Its own ceiling on concurrent in-flight requests. Sized so it can only ever consume its own slice and no more. When that dependency goes bad, it saturates its own compartment and stops there. Requests to it start failing fast, which is a real cost — but auth, billing, and everything else keep serving out of pools the flood never reached.
It’s worth being precise about how this differs from a circuit breaker, because the two get lumped together and they do different jobs. A circuit breaker decides when to stop calling a dependency that’s clearly broken — it watches the error rate and, past a threshold, trips, so you stop hammering a dead service. A bulkhead decides how much of you any one dependency is even allowed to consume in the first place. That distinction matters most in the ambiguous case: a dependency that isn’t erroring, just slow. The breaker may not have tripped — errors look fine — while the latency quietly drains a shared pool. The bulkhead doesn’t care whether the breaker has made up its mind; the slow dependency simply can’t reach past its own walls. Bulkhead bounds the blast radius; breaker cuts the fuse. You want both.
The trade is real and you should name it out loud: partitioning costs you efficiency. Ten separate pools of ten slots cannot flex the way one pool of a hundred can. You will sometimes reject a request in a saturated compartment while slots sit idle next door. That’s the deal — you spend some utilization to buy isolation. Which means the sizing is not a math problem, it’s a priorities problem. You don’t split resources evenly; you give the critical paths the room they need and cap the peripheral ones tightly, because the whole point is that a nice-to-have should never be able to starve a must-have.
That’s the real decision a bulkhead forces, and it’s a healthy one to make on purpose rather than discover at 3 a.m.: which of your failures are you willing to let happen alone? You’re pre-drawing the blast radius while the system is calm, instead of letting the blast radius draw itself during the incident.
The engineering, to look up: the bulkhead pattern — Michael Nygard, Release It!; thread-pool and connection-pool isolation as blast-radius containment; how it complements the circuit breaker and relates to connection-pool exhaustion.
Sources
- Michael Nygard, Release It! (bulkhead pattern)
- thread/connection-pool isolation
Liked this? Get the next one in Working Theory.
Going weekly in August (it's in beta now). One genuinely interesting read on building, the brain, and the science most people missed.
Subscribe →