A ship's cross-section with watertight bulkhead compartments, one compartment flooded and sealed off while the rest of the ship stays dry and afloat, rendered as a metaphor for isolated software resource pools Tech
AI-generated, Working Theory
Tech · ◉ Evergreen

Fail in one room, not the whole house.

by · ·4 min·Working Theory

A single slow dependency can take down features it never touched. The reason is a shared resource pool — and the fix, borrowed from shipbuilding, is to give every dependency its own compartment.

Here’s a failure that feels like it breaks cause and effect. Your recommendations service goes down — a peripheral, nice-to-have feature — and within a minute users can’t log in. Login never called recommendations. The two share no code path a reader could point to. So how did one drag down the other?

They shared a resource. And a shared resource is a shared fate.

Most services run on a finite pool of something scarce: worker threads, database connections, memory, in-flight request slots. Under normal load the pool is plenty. The dangerous case isn’t a dependency that fails — a fast failure returns an error and frees the slot. The dangerous case is a dependency that gets slow. A downstream call that used to take 20 milliseconds now takes 20 seconds, and while it hangs, it holds its slot in the pool. Requests bound for that sick dependency queue up, each one clutching a slot, waiting. The pool drains. And now every request — including the ones headed for perfectly healthy dependencies like your auth database — waits for a slot that never comes free. One slow corner has starved the entire building. This is the mechanism behind connection-pool exhaustion, seen from the blast-radius side.

The fix is old enough to predate computers. Ships are built with their hulls divided into watertight compartments, so a breach floods one section instead of sinking the vessel. The walls are called bulkheads, and Michael Nygard borrowed the name for software in Release It!

ONE SHARED POOL — the slow call drains it all slow "recs" call holds every slot → login waits too COMPARTMENTS — one floods, the rest keep flowing recs (saturated) login (healthy) billing (healthy) the breach stays in its room
Pool everything together and one slow dependency drowns the ship. Give each its own compartment and the breach floods one room. Original diagram · Working Theory

The pattern is exactly the ship’s: give each dependency — or each class of work — its own bounded slice of resources. Its own thread pool. Its own connection limit. Its own ceiling on concurrent in-flight requests. Sized so it can only ever consume its own slice and no more. When that dependency goes bad, it saturates its own compartment and stops there. Requests to it start failing fast, which is a real cost — but auth, billing, and everything else keep serving out of pools the flood never reached.

It’s worth being precise about how this differs from a circuit breaker, because the two get lumped together and they do different jobs. A circuit breaker decides when to stop calling a dependency that’s clearly broken — it watches the error rate and, past a threshold, trips, so you stop hammering a dead service. A bulkhead decides how much of you any one dependency is even allowed to consume in the first place. That distinction matters most in the ambiguous case: a dependency that isn’t erroring, just slow. The breaker may not have tripped — errors look fine — while the latency quietly drains a shared pool. The bulkhead doesn’t care whether the breaker has made up its mind; the slow dependency simply can’t reach past its own walls. Bulkhead bounds the blast radius; breaker cuts the fuse. You want both.

The trade is real and you should name it out loud: partitioning costs you efficiency. Ten separate pools of ten slots cannot flex the way one pool of a hundred can. You will sometimes reject a request in a saturated compartment while slots sit idle next door. That’s the deal — you spend some utilization to buy isolation. Which means the sizing is not a math problem, it’s a priorities problem. You don’t split resources evenly; you give the critical paths the room they need and cap the peripheral ones tightly, because the whole point is that a nice-to-have should never be able to starve a must-have.

That’s the real decision a bulkhead forces, and it’s a healthy one to make on purpose rather than discover at 3 a.m.: which of your failures are you willing to let happen alone? You’re pre-drawing the blast radius while the system is calm, instead of letting the blast radius draw itself during the incident.

The engineering, to look up: the bulkhead pattern — Michael Nygard, Release It!; thread-pool and connection-pool isolation as blast-radius containment; how it complements the circuit breaker and relates to connection-pool exhaustion.

Sources

  • Michael Nygard, Release It! (bulkhead pattern)
  • thread/connection-pool isolation

Liked this? Get the next one in Working Theory.

Going weekly in August (it's in beta now). One genuinely interesting read on building, the brain, and the science most people missed.

Subscribe →
Got a reaction, a counter-example, or something I missed? Reply by email — I read everything.
◉ join in

Where have you hit this — in a product you use, or one you're building?

Threads open here soon. For now, the conversation lives two clicks away — discuss on GitHub, or just reply by email. I read and answer everything.