The system that never says "I'm not sure."

This week the FAA introduced SMART — Strategic Management of Airspace, Routes, and Trajectories — a cloud-based AI platform, built with Air Space Intelligence, that reads airline schedules, weather, airport capacity, and airspace conditions to predict where traffic will jam and where flights might conflict before it happens. It’s an $875-million, twelve-year commitment, starting in the Washington, D.C. region and spreading from there. On its face it’s a good idea: give the people running the sky a system that sees the pileup coming.

Here’s the line in the coverage I kept looking for and didn’t find: how a controller is supposed to disagree with it. Not whether they can — of course they can — but what the moment of disagreement is designed to feel like. Because that moment is the whole ballgame, and it’s governed by a quirk of attention that every builder shipping a recommendation should have tattooed somewhere visible.

The quirk is called automation bias, and its close cousin automation complacency. The finding, worked out over decades of aviation and medical research (Kathleen Mosier and Linda Skitka’s work in the 1990s, later synthesized by Raja Parasuraman and Dietrich Manzey), is not that people distrust machines. It’s the opposite. When a tool is right almost all the time, people stop checking it — and not because they’re lazy or careless. Attention is a budget. A reliable aid looks like a safe place to stop spending. So they spend their attention elsewhere, which is usually the rational thing to do, right up until the rare moment it isn’t.

That failure comes in two shapes, and it’s worth knowing both because you design against them differently.

The first is an omission error: the system doesn’t flag something, so the human doesn’t catch it — because catching it was the system’s job now. The danger here is silence. “Nothing flagged” gets read as “nothing there,” when it might mean “the system looked in the wrong place” or “didn’t look at all.”

The second is a commission error: the system flags the wrong thing, and the human follows it past their own better judgment. There are documented cases of skilled operators overriding correct instincts because the automation was confident and they assumed it knew something they didn’t.

Notice what this does to the usual intuition. We build as if the risk lives in the errors — so we chase accuracy, push reliability from 95% to 99%, and feel safer. But every point of reliability you add also pulls the human’s attention away, because now the tool looks even more like a safe place to stop looking. The residual risk doesn’t shrink as fast as the error rate does. Sometimes it grows. A system that’s wrong 5% of the time keeps people alert. A system that’s wrong 0.1% of the time can train them to stop watching entirely — and the 0.1% arrives anyway.

So if you’re shipping a recommendation, a co-pilot, an agent that acts, the design problem is not only “be right more often.” It’s: assume you will occasionally be wrong, and make sure someone is still in a position to catch it when you are. Three moves that follow from that:

Make “we didn’t find anything” look different from “we didn’t look.” A blank result and a checked-and-clear result should never render the same way — silence is where omission errors hide.

Put the human check where a silent miss is expensive, not everywhere. Confirmation fatigue is real; if you ask people to verify everything, they verify nothing. Spend your interruption budget on the irreversible and the costly.

And surface uncertainty honestly, especially the low-confidence calls. A system that always speaks in the same flat, certain voice is teaching its users that it’s never unsure — which is the exact belief that makes them stop double-checking. Calibrated doubt, shown at the right moment, is what keeps a human genuinely in the loop instead of nominally in it.

The FAA is spending twelve years and nearly a billion dollars to see the conflict coming. The harder engineering isn’t the seeing. It’s building the system so that a human is still watching on the one day it looks the other way.

The science, to look up: automation bias and automation complacency; omission vs. commission errors — Mosier & Skitka (1996); Parasuraman & Manzey, “Complacency and Bias in Human Use of Automation” (2010). Well-established in aviation and clinical decision-support research; effect sizes vary by task and by how reliable the automation is.

The news, to check: the FAA’s “SMART” (Strategic Management of Airspace, Routes, and Trajectories) platform, built with Air Space Intelligence — $875M over 12 years, Washington, D.C. region first — TechCrunch, Sept 17 2026 (corroborated by Quartz and others).