You're only measuring the users who stayed

There’s a WWII story that statisticians tell each other until it wears smooth, and it’s worth telling once more because it’s about your metrics dashboard even though it’s about airplanes. The military wanted to know where to add armor to bombers, so they studied the ones coming back from missions and mapped the bullet holes. The holes clustered on the wings and the fuselage. The obvious move: armor where the holes are. Abraham Wald, a mathematician on the problem, pointed at the flaw. Those were the planes that made it home. The holes showed all the places a bomber could get hit and still fly. The armor belonged exactly where the returning planes had no holes, around the engines, because the planes hit there weren’t in the hangar to be studied. They were at the bottom of the Channel.

Your analytics dashboard is a hangar full of returning bombers.

Almost every number you look at each morning is quietly computed over the users who are still here. Average session length. Feature adoption. That glowing NPS. “Our users love X.” All of it is measured on survivors. The people who bounced in the first ten minutes and the ones who churned in week three don’t appear in this week’s engagement average, they left the denominator on their way out. And every average drifted upward as they went, not because a single thing improved, but because the unhappy stopped being counted. The number got better by subtraction.

Once you see it, it’s everywhere. “Power users adore this feature,” of course they do; the people it confused churned out weeks ago, and you’re hearing from the fraction it happened to fit. The retention curve that flattens into a comfortable plateau gets read as “the product became stickier over time.” Sometimes that’s true. Just as often it means everyone who could churn already did, and what’s left is the residue that was never going to leave, the curve flattened by subtraction, not by improvement. And the in-app survey with the encouraging results? It was answered by the engaged, because they’re the only ones still opening the app to be asked.

The fix is to go study the planes that didn’t come back. Interview churned users with the same seriousness you interview happy ones, they’re the holes around the engine, and they’re the ones telling you where the product actually failed. Read your numbers as cohorts, a fixed group of people tracked forward through time, rather than as snapshots of whoever is active right now, because the snapshot silently re-selects for survivors every single day. And build one reflex: when a metric moves up, ask whether the thing got better, or whether the people who’d have dragged it down simply left. Those are very different reasons, and only one of them is worth celebrating.

The idea, to look up: survivorship bias, Abraham Wald and the Statistical Research Group’s WWII aircraft-armor analysis; the same bias in fund performance, historical buildings, and “successful founder” advice.