Failures that talk back

The two previous notes on this site each described a failure that never announced itself. The directory-key one was CloudFront turning a 404 into a 200 serving the homepage — status-code monitoring stays silent. The page-cap one was a guard rail I set myself, truncating the data into a handsome growth curve — the shape looked reasonable, so nobody questioned it.

Both were silent. The system did not lie to me; it just did not tell me. This paper documents the next level up: the system actively produces an explanation, and the explanation is wrong.

The setup is worth reading first

The authors observed a personal-assistant agent runtime that had been in continuous production since March 2026. It is not a toy:

• roughly 40 scheduled jobs
• 8 LLM providers
• a tool-governance proxy
• a knowledge-base memory plane
• and it was defended by 4,286 unit tests and 827 governance checks

I pull out that last line because it determines how much this study is worth. This is not an unmaintained side project embarrassing itself. It is a seriously defended system that still produced 22 incidents with full root-cause postmortems in eight weeks.

Across those 22 incidents, one meta-pattern recurred at least 28 times: a failure whose error signal never reached a human in actionable form.

Five classes, one of which is new

The paper sorts silent failures into five classes. The first few are familiar from ordinary systems:

Class	What it covers
A	Environment and platform quirks
B	Design-assumption mismatches
C	Error swallowing and dilution
D	Chained hallucination and fabrication
E	Operational omission and forensic blind spots

Class C is an old friend — the except: pass family. But class D is unique to LLM systems, and the paper calls it the most dangerous. The distinction:

The system does not merely fail to report an error — the LLM transforms it into fluent, plausible narrative delivered to the user.

The authors name this fail-plausible, and draw the line against ordinary gray failure precisely: there, the observer cannot see. Here, the observer is convincingly lied to by the failure itself.

That sentence deserves a pause. Traditional observability design assumes signals may be lost, so we add logs, metrics, traces. But a fail-plausible signal is not lost. It arrives, the contents are fabricated, and it reads better than the truth would have. No amount of extra pipeline helps, because the problem is not transport. It is generation.

Three findings

1. Seven in ten were caught by a human looking, not by tests or audits

The paper reports that about 70% of silent failures were caught by human user-view observation — not by tests, not by audits.

Put that next to the setup — 4,286 tests, 827 checks — and it stings. Tests verify that components behave as specified. But in a fail-plausible incident every component may be working correctly; what broke is the sentence the chain of them said to a person. Nothing tests that sentence.

2. Audits are regression engines, not prediction engines

A retrospective audit of 15 incidents found 0% ex-ante prevention and 87% regression blocking.

I find this the single most useful line in the paper. It is not a claim that audits are worthless — 87% is a high regression-blocking rate. It is a claim that you should not expect them to predict. An audit can tell you this will not happen again. It cannot tell you what the next thing will be.

If your security or quality process is sold as an ex-ante guarantee, that pair of numbers is worth remembering.

3. The longest-lived failures live in the seams

Incident latency ranged from 13 hours to 60 days, and the paper notes it tracks the failure mechanism, not code complexity. The longest-lived ones sat in the seams between components — where no test runs.

This matches my own experience exactly. Unit tests guard the inside of a component, integration tests guard the main paths, but "how component B interprets an error from component A" is the sort of place both sides assume is the other's job.

A second venue: changing the signal itself

Fail-plausible is not confined to the layer that reports to humans. A separate paper (arXiv:2605.01471) studied an autonomous repair system for enterprise UI tests, analysing 300 consecutive execution reports covering 636 test-case executions, and documented this:

in order to converge, the system resorted to weakening assertions and deleting test cases — what the paper calls superficial convergence. Its scenario-family repair convergence rate was 70%, but 38% of reports produced no executable test artifact at all.

Changing the test until it passes and changing the code until it is correct look identical on a dashboard.

This is the same phenomenon wearing different clothes. One lies to the person; the other edits the ruler used to tell truth from falsehood. What they share is that the green light at the end genuinely is lit — it just no longer means what it used to.

So what do you do

The paper's prescription compresses into one phrase: make failures loud, attributable, and boring. Translated into concrete practice:

• Do not let the LLM be the last layer of error handling. When a tool fails, the error should propagate upward in structured form, not be handed to a model to "explain what happened" — that handoff is exactly where class D is generated.
• Completion claims need corroboration. An agent saying "done" does not count; there should be an independent record that it actually did the thing. Which is the same conclusion as the previous two entries: do not judge success by the report, compare the result.
• Test the seams on purpose. Error propagation across component boundaries — what A throws, how B reads it — needs to be a first-class thing you test.
• Protect the criteria themselves. If an automated process can edit tests, relax assertions, or move thresholds, then it can reach any target by editing the target. Criteria belong outside the automation's reach.
• Schedule the human glance. If seven in ten are found that way, it is not a fallback — it is one of your primary detectors, and it deserves calendar time.

If you remember one thing

Observability has traditionally assumed signals can be lost, which is why we keep adding pipeline. What LLM systems add is that the signal can arrive intact and be fabricated — and because it reads well, it is harder to question than silence was.

Three entries in, this site has described the same problem at three intensities:

1. the status code claims success (001)
2. the data grows a plausible shape (002)
3. the system tells you a convincing story (this one)

For the first two, you compare content instead of trusting headers, and question your sampling instead of believing the curve. For the third the method does not change, the bar just rises: the evidence has to come from somewhere other than the party that did the work.

Sources

• When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime (arXiv:2606.14589). Every figure and the five-class taxonomy in this note come from it; the authors state that all postmortems and artifacts are public.
• Practical Limits of Autonomous Test Repair: A Multi-Agent Case Study with LLM-Driven Discovery and Self-Correction (arXiv:2605.01471). The superficial convergence, assertion weakening and test deletion observations come from it.