A lone tightrope walker crossing a wire high above a dim city at dusk, a faint safety net barely visible far below Brain Science
AI-generated, Working Theory
Brain Science · the neuroscience of building · ◉ Evergreen

The agent got cheaper. Your trust didn't.

by · ·4 min·Working Theory

Autonomous agents just got cheap. The unsolved 0→1 problem isn't capability — it's calibrating how much your user trusts the thing. Two brain failure modes, and where to put the checkpoint.

At the end of June, Anthropic shipped Claude Sonnet 5 and described it, more or less, as a cheaper way to run agents — a model that can plan, drive a browser and a terminal, and run on its own at a level that a few months earlier needed something bigger and pricier. Around the same corner of the calendar, half the industry was demoing a browser that does your errands. The capability line moved, and it moved down in price.

Here’s the thing that price cut exposes. When agents were expensive and rare, the hard problem was making them work at all. Now that a competent agent is a line item, the hard problem is the one nobody put on the pricing page: how much should the person on the other end trust it? That’s not a capability question. It’s a question about a human brain, and the human brain is bad at it in two opposite directions at once.

The first direction is over-trust. Give a person a tool that’s right most of the time and they stop checking it — human-factors researchers have called this automation complacency for decades, watching pilots and operators quietly hand over the monitoring they were supposed to keep. The agent earns a little rope, then a little more, and the checking muscle atrophies exactly when it’s needed. The second direction is the mirror image: algorithm aversion. Watch an algorithm make one visible mistake and people abandon it faster than they’d abandon a human who made the identical error — even when the algorithm is still, on average, better. One flinch and the trust collapses.

So the user’s trust isn’t a dial that slowly finds the right setting. It swings. It sits too high until a bad miss, then craters below where it should be. Your job on a 0→1 agent product is not to maximize trust — it’s to calibrate it, to make the user’s confidence track the agent’s actual reliability instead of overshooting and then over-correcting.

And the actual reliability is lower than the headline number, because of a piece of arithmetic that agents make unavoidable. An agent that’s 90% reliable on a single step sounds trustworthy. Chain ten of those steps into one autonomous task and you multiply: 0.9 to the tenth is about 0.35. The per-step number felt like an A-minus; the end-to-end number is a coin flip you lose most of the time. Users don’t feel that compounding. They anchor on “it’s usually right” and hand over the ten-step job. The gap between how it feels and how it computes is precisely where miscalibrated trust lives.

calibrated (trust = reliability) over-trust automation complacency under-trust algorithm aversion agent's actual reliability → user's trust →
Trust doesn't settle on the diagonal — it overshoots, then over-corrects after a miss. The build job is to pull the user back onto the line. Original diagram · Working Theory

Calibration is a design surface, and three decisions carry most of the weight.

Show confidence, not a verdict. A binary “done / failed” teaches the brain to over-trust the “done,” because “done” never carries doubt. An agent that says how sure it is — and is honestly less sure on the ten-step job than the one-step one — hands the user a trust signal instead of a trust demand. Calibrated confidence isn’t a nicety; it’s the interface for calibration itself.

Put the checkpoint at the irreversible step, and make it a wall, not a whisper. Complacency means the user will click past a gentle “are you sure?” without reading it. Reserve the interruption for the one action you can’t undo — the send, the purchase, the delete — and spend the friction there, where an over-trusting brain most needs to be slowed down. Everywhere else, let it run.

Design the first mistake, because you only get the one. Algorithm aversion says a single visible error can nuke trust below where it belongs and keep it there. So the agent’s error state is not an edge case to bolt on last — it’s the highest-stakes screen in the product. An error the user’s own brain catches is the moment of maximum attention; a clean “here’s what I got wrong, here’s the one thing I need from you” recovers trust, while a silent failure or a shrug spends it permanently. Let the user correct the agent and keep going, and you buy back the trust the mistake cost. Make them start over, and you’ve taught them the aversion.

The capability curve will keep falling in price; that’s the one safe prediction. What won’t commoditize is the part that lives in the user’s head — the running estimate of how much to hand over. Build the thing that keeps that estimate honest, and you’ve built the part of the agent nobody can undercut.

The science, to look up: trust in automation and calibration (Lee & See, 2004); automation bias / complacency (Parasuraman & Riley, 1997; Parasuraman & Manzey, 2010); algorithm aversion (Dietvorst, Simmons & Massey, 2015) and “algorithm appreciation” when users can adjust the model (Dietvorst et al., 2018; Logg et al., 2019). The compounding-reliability figure is just independent-probability arithmetic (0.9¹⁰ ≈ 0.35); treat real agent steps as correlated, so it’s an intuition pump, not a measurement.

The news, to check: Anthropic’s Claude Sonnet 5 launch, positioned as a cheaper way to run agents — TechCrunch, June 30, 2026 (“anthropic-launches-claude-sonnet-5-as-a-cheaper-way-to-run-agents”).

Sources

  • Trust in automation and calibration (Lee & See, 2004)
  • automation bias / complacency (Parasuraman & Riley, 1997)
  • algorithm aversion (Dietvorst, Simmons & Massey, 2015)

Liked this? Get the next one in Working Theory.

Going weekly in August (it's in beta now). One genuinely interesting read on building, the brain, and the science most people missed.

Subscribe →
Got a reaction, a counter-example, or something I missed? Reply by email — I read everything.
◉ join in

Where have you hit this — in a product you use, or one you're building?

Threads open here soon. For now, the conversation lives two clicks away — discuss on GitHub, or just reply by email. I read and answer everything.