Making Failure Legible

A demo fails loudly. Something throws, the screen goes red, and everyone in the room sees it. Production fails the other way: the pipeline completes, the response looks reasonable, the ticket closes, and three weeks later you find out the account number was from the wrong tenant.

I have spent most of my working hours inside that second kind of failure. The lesson I keep relearning is that the hard part of an agentic system is not making it succeed. It is arranging for failure to be legible — visible, attributable, and timely enough that someone can act on it.

Failure has a shape

When people say an agent "hallucinated," they are usually describing the end of a longer story. Something upstream was already wrong. Retrieval returned a document from the wrong product. An auth refresh gave you a token for a stale scope. A tool returned an empty list and the model treated "empty" as "none exist."

By the time the answer reaches a human, all of that has collapsed into one plausible sentence. The trace is where the real story lives, and if nobody can read it, the system is not observable — it is just decorated.

The question is not "did it do something?" It is "when did it stop being trustworthy?"

Three moves that actually help

These are the ones that survived contact with real workloads. None of them are research contributions. All of them changed incident response.

1. Name the boundary before the model gets a chance to blur it

A grounded answer and an ungrounded answer should not be the same shape. If retrieval confidence drops below what the interface implies, the system should say so in the response itself, not in a log nobody opens. The cheapest reliability win available is refusing to sound certain in the exact case where you are not.

2. Make every mutation a named step you can replay

Pipelines that write to external systems should be able to pause, repair, and resume from a named state rather than restarting the whole run. This is ordinary distributed systems thinking, and it matters more, not less, once a model is choosing the steps.

if auth.refresh() fails:
    pause_downstream_mutation()
    branch recovery_path
    revalidate credentials
    replay failed step
    emit trace + resume nominal state

The point is not the specific mechanism. It is that recovery is designed in the first pass, while the state machine is still simple enough to reason about.

3. Score the path, not the final sentence

A rubric that only reads the output will reward a wrong answer that was delivered confidently. Evaluate state transitions, fallback quality, and the relationship between stated confidence and actual evidence. That catches the failure mode that matters most: output that looks fine while the underlying state is already broken.

What this looks like in practice

On a support assistant I worked on, the useful interventions were not prompt rewrites. They were:

• making the escalation boundary explicit in the response surface
• keeping operator visibility into the trace by default, not on request
• treating retrieval confidence as something the user can see
• reviewing disagreements between stated confidence and evidence as first-class signals

The measurable gains came from the combination. Autonomous resolution went up because the system stopped guessing in the cases where it used to guess, and operators spent their attention where judgment was actually required.

The honest version

Legibility costs something. Explicit boundaries produce more visible escalations, which look worse on a dashboard than a system that quietly improvises. Some of the pressure to hide failure comes from the metrics, not the technology.

I think that trade is worth making anyway. A system that tells you when it stopped being trustworthy is one you can put in front of real work. A system that cannot is a demo with a longer runway.