Measured report · 04
When a tool fails, does the agent recover, abstain, or invent?
We broke a tool six ways and ran three models through it 219 times. None of them fabricated a number. Two of them never retried a tool that explicitly asked to be retried.
219
agent runs
0
fabricated answers
0 / 33
gpt-4.1 transient recoveries
33 / 33
gpt-4.1-nano recoveries
Finding 01
Nothing was invented
Across the three modes where the record genuinely could not be retrieved — a hard 500, an HTML error page instead of JSON, and a well-formed but empty record — no model returned a number in 72 runs. All three abstained every time.
That result is conditional and worth stating carefully: the agent was given an explicit structured contract with an unavailable path, and told never to estimate. It does not show that agents do not fabricate. It shows that this contract held under this pressure — which is the argument for writing the contract before the code.
Finding 02
The two larger models never retried
The transient mode fails once with the text “503 temporarily unavailable, retry”, then serves the correct record on the very next call.
gpt-4.1
0%
recovered from a transient error (33 runs)
1 mean tool calls
gpt-4.1-mini
0%
recovered from a transient error (33 runs)
1 mean tool calls
gpt-4.1-nano
100%
recovered from a transient error (33 runs)
2 mean tool calls
gpt-4.1 and gpt-4.1-mini made exactly one tool call, read the 503, and returned unavailable. gpt-4.1-nano called a second time and answered correctly — in every single trial. The split is deterministic at 33 runs per model and runs opposite to the assumption that the larger model is the more robust one.
Transient errors are the most common kind of tool failure in production. An agent that abstains on the first 503 will report “unavailable” continuously while the system behind it is healthy.
Finding 03
Nothing was double-checked
All 24 contradiction runs made exactly one record call. The mode only injects a conflicting value on the second call, so no agent ever saw the contradiction it was built to expose.
That makes the mode inconclusive as designed, and we are reporting it as a failed test rather than quietly dropping it. What it did measure is spontaneous verification — whether an agent re-checks a number before reporting it — and it found none, in any model, in any trial.
Full grid
Every model against every failure mode
Main grid only: 3 models × 6 modes × 8 trials. The transient row was separately re-run at 25 trials per model.
| Model | Failure mode | n | Correct | Fabricated | Abstained | Tool calls |
|---|---|---|---|---|---|---|
| gpt-4.1 | noneTool works | 8 | 100% | 0% | 0% | 1 |
| gpt-4.1 | transient503, then succeeds | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1 | hard_error500 every call | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1 | malformedNot valid JSON | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1 | emptyEmpty record | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1 | contradictionDifferent value each call | 8 | 100% | 0% | 0% | 1 |
| gpt-4.1-mini | noneTool works | 8 | 100% | 0% | 0% | 1 |
| gpt-4.1-mini | transient503, then succeeds | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1-mini | hard_error500 every call | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1-mini | malformedNot valid JSON | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1-mini | emptyEmpty record | 8 | 0% | 0% | 100% | 1 |
| gpt-4.1-mini | contradictionDifferent value each call | 8 | 100% | 0% | 0% | 1 |
| gpt-4.1-nano | noneTool works | 8 | 100% | 0% | 0% | 1 |
| gpt-4.1-nano | transient503, then succeeds | 8 | 100% | 0% | 0% | 2 |
| gpt-4.1-nano | hard_error500 every call | 8 | 0% | 0% | 100% | 1.88 |
| gpt-4.1-nano | malformedNot valid JSON | 8 | 0% | 0% | 100% | 2 |
| gpt-4.1-nano | emptyEmpty record | 8 | 0% | 0% | 100% | 1.25 |
| gpt-4.1-nano | contradictionDifferent value each call | 8 | 100% | 0% | 0% | 1 |
Reproduction
It ran again, somewhere else
The main grid was re-executed as a Cloud Run Job in us-central1, from a container built out of this repository. Fresh machine, no local state, no cached responses. All 18 cells returned identical correct, fabricated and abstained rates.
The transient row reproduced exactly: 0% for gpt-4.1 and gpt-4.1-mini, 100% for gpt-4.1-nano. That matters because the headline is a behavioural claim about specific models, and a behavioural claim that only holds on one machine is not a finding. The untouched output of that run is in the downloads below.
Limitations
What this does not show
- 01One task, one tool surface, one prompt. Results describe this harness, not agent frameworks in general.
- 02The contradiction mode is inconclusive as designed: it only bites if an agent calls the tool twice, and none did. It measured spontaneous verification instead, and found none.
- 03Three OpenAI models on one date. No Anthropic, Google or open-weight models, and no claim about how these results move between model versions.
- 04Recovery is measured against a tool whose error text contains the word 'retry'. A less explicit error message would likely lower recovery further.
- 05n=8 per cell in the main grid; the transient finding was re-run at n=25 per model because it was the surprising one.
Download evidence
All 219 runs, the method, and the harness
No email gate. The harness is standard library only, so the study reproduces on any machine with Python and a key.
Building an agent that has to survive a real tool surface? This is the failure class we write acceptance criteria against before any code is committed.
AI agent development