Wake me for the decision, not the keystroke

An agent I'm running stops and asks me a question: it wants to delete a storage bucket it has decided is stale. Designing the part where it can stop and ask is easy. The hard part is deciding, ahead of time, which questions are worth stopping for. Make the set too small and it deletes something I still needed. Make it too large and I stop reading the questions. Almost all of the care goes into that second failure, because it is the one that keeps looking like safety long after it has stopped being any.

The prompt you approve without reading

Alex Wauters, a Belgian developer, built a browser game to put numbers on this. You get sixty seconds to approve or deny a stream of simulated permission requests from an AI coding agent, scored on whether you catch the dangerous ones. Across more than forty thousand runs and four hundred thousand individual decisions, players waved through roughly one in three of the planted malicious requests. The single most-approved dangerous command was a harmless-looking npm run analyze, cleared nearly two-thirds of the time.

Wauters is careful to call it a game and not a study, and it is one: the threat rate is far higher than real work, the clock is artificial, and a fifth of players caught everything. But the shape of the result is hard to wave off, and it lines up with the one real-world number we have. Anthropic's own telemetry puts the approval rate on Claude Code's permission prompts around ninety-three percent. That figure does not prove anyone is asleep at the wheel. Most of what an agent proposes is exactly what you asked for, so most of the time yes is the correct answer. What it shows is how rarely no gets exercised at all, and a stream where the answer is almost always yes is precisely the training set that installs the reflex. The game is what that reflex costs you the day a malicious request finally slides past in the stream.

The escape hatch makes it worse. Developers reach for the flag that turns permission checks off wholesale, the one named to sound like a warning, rather than risk a prompt freezing a multi-hour run five minutes in. Over-asking does not fail safe. It fails toward the bypass. A gate that interrupts too often teaches the operator to tear the gate out.

Why the keystroke is the wrong unit

The reason per-command approval collapses is not that people are lazy. It is that a single command is the wrong thing to ask a human about. Shown npm run analyze, what is a person supposed to judge? They do not know what that script does in this repository, whether it phones home, whether the agent picked it for a good reason or a compromised one. The prompt demands a verdict the operator has no standing to give, dozens of times an hour, so the answer decays into a tic. You have manufactured the appearance of oversight and spent the substance of it, and you spent it by asking too often.

So the useful question is not how to make people read the prompts. At that volume they won't. The question is which decisions deserve an interrupt at all.

The residue

Take out everything a person cannot meaningfully judge, and everything that does no lasting harm if they judge it wrong, and a small residue is left. Three things mark it.

The first is irreversibility. Deleting a resource, force-pushing over history, moving money, shipping to production: once it is done the undo is either gone or expensive, so the one moment a human can change the outcome is before it happens. The interrupt earns its place because the interrupt is the only lever left.

Then there is consequence that reaches outside the sandbox. An action that touches a real account, a real invoice, a real person on the other end of an email. Inside a sealed box, one with no live credentials and no open path to the network, let the agent run and fail and retry all it likes; the blast radius is a container, and a container is cheap. The gate belongs where the effect turns real, which tends to be the same place it stops being cheap to undo.

Repeated failure is the third. When an agent has tried the same approach and failed three times, something is usually wrong that it cannot see from inside its own loop, and a human's wider view is worth the interruption there. Not because the fourth attempt is more dangerous, but because the agent has stopped making progress; what to do about a stalled agent is a call for a person, not a keystroke.

Two things sit under that list. Below it is a floor the human never sees: actions the agent simply is not allowed to take, like reading a secret outside its scope, escalating its own privileges, or reaching into another tenant. Those are not questions for anyone. They are denials the harness enforces, because an action you would refuse every time should never be allowed to become a prompt. And the three triggers above the floor are not the same kind of thing. Whether an action is irreversible or crosses the sandbox, the orchestrator can tell on its own, by looking at what the action is. Whether an approach has failed three times, it cannot, because a stuck agent is exactly the one that will relabel its fourth attempt as a fresh idea and never trip its own counter. That count has to be kept outside the agent, by the thing watching it. The interrupts worth raising to a human are the ones a machine can detect but not settle.

What the worth-asking cases share is that each is rare and each is legible. Rare, so the interrupt still lands when it comes. Legible, so the person actually has the standing to answer it. Should we ship this, move this money, delete this thing, keep grinding on an approach that plainly is not working: a human can decide those. Allow npm run analyze, two hundred times a day, cannot. The one caveat is that rare is a property of the workload, not of the rule. Point this at an agent whose entire job is sending mail and mail stops being rare, and you are back to cutting the set one level finer. The principle holds; you just apply it again.

Designing the interrupt

This inverts the reflex. The intuitive safety move is to ask about more, to drop the threshold, to prompt on anything that could conceivably matter. But treat the operator's attention the way it actually behaves, as a budget that runs down over a stream of prompts and is slow to recover once the reflex sets in, and every extra prompt is a withdrawal. Spend it on the rare legible decision and the human is sharp in the moment it counts. Spend it on every keystroke and the balance is gone by mid-morning, right about when the genuinely dangerous prompt arrives and collects the same automatic yes as the two hundred before it.

There is a catch the attention argument alone does not cover. The bypass is global: the flag that skips the noise skips the rare interrupts too, and a short list, precisely because it fires so seldom, is what makes turning the whole thing off feel almost free. So the worth-asking class cannot simply be smaller. It has to be harder to switch off than the noise it replaced, or the operator who mutes the din for an overnight run mutes the three prompts that mattered along with it. Attention economics buys you nothing if the kill switch is still one flag for everything.

So the escalation policy I actually want is short. Run freely inside the sandbox. Deny out of scope by default, with no prompt at all. Stop and ask only for the actions the harness can see are irreversible or reach outside the box, and for the moment an external counter says the same approach has failed three times. Everything else, decide it and keep moving. The list is short on purpose. A short list is the only kind a person keeps reading.

The number worth watching was never how many actions you put behind a human. It is how many you can afford to put there before the human stops looking, and you can see it without asking anyone to introspect: pull the approvals and measure how long each one took. The ones answered in under a second are the theater. Weigh those against the ones that got a real pause, and the ratio tells you how much of your oversight is a person deciding and how much is a person keeping a long run alive. A human in the loop is worth having. But only if you can still reach their attention on the day you finally need it, and every prompt you chose not to send is what keeps that day survivable.