Core practice
Agents that behave when the tools misbehave
Any agent works when every tool returns clean data. The engineering is in what happens on the 503, the empty record and the malformed response — because in production those arrive daily.
AI Agent Development
The situation
Before we start
It was only ever tested on the happy path
Demos exercise the case where every tool returns clean data. Production is mostly the other cases, and nobody wrote down what the agent should do in them.
A confident number nobody can trace
An agent that invents a value is worse than one that crashes, because nothing downstream can tell an invented answer from a real one. That is the failure we test for first.
Nothing bounds the loop or the spend
Retries, tool errors and self-correction multiply steps quietly. Budgets and termination conditions are part of the acceptance criteria, not a later optimisation.
What working means here
Signed first. Measured after.
The value of each one is set with you during scoping, from your data and your line. What does not change is that they are written down first and published against afterwards, pass or fail.
- 01
Fabrication rate under tool failure
Share of runs where the agent asserted a value no tool ever returned, measured against an instrumented tool bench
- 02
Recovery rate on transient errors
Share of injected 503s the agent retries and resolves, rather than abandoning the task
- 03
Step and cost bound adherence
Share of runs terminating inside the agreed budget, with no unbounded loops
We publish the measurements behind this practice, including the runs that failed.
Agent Tool-Failure Recovery Benchmark 2026How it is assembled
Stage by stage
Each one ends in something you keep.
- Step 01
Bound the job
What the agent may do, what it may touch, and where a human stays in the loop. Written before any code.
- Step 02
Build the bench first
The instrumented tool surface that fails on purpose, so the failure path is testable from day one rather than discovered in production.
- Step 03
Build against the bench
The agent is developed against injected failures, not just clean data. Nothing ships that has not been broken deliberately.
- Step 04
Measure and gate
Fabrication, recovery and budget adherence measured on every change. A release that regresses does not go out.
- Step 05
Operate or hand over
Audit trails, runbooks and training, or we run it under an agreement.
Scope
What the work includes
- Task definition: what the agent decides, and what stays with a human
- Tool surface design with explicit, least-privilege permissions
- Failure-path specification: retry, escalate, or abstain — decided per tool, in writing
- Step, latency and cost budgets enforced in code, not by prompt
- An instrumented tool bench that injects timeouts, errors, malformed and empty responses
- Evaluation suite run on every change, gating release
- Audit trail: every tool call, argument and result, retained and queryable
- Deployment inside your environment under your identity provider
Deliverables
What you keep
- 01Signed acceptance criteria: fabrication ceiling, recovery floor, step budget
- 02Working agent in your environment
- 03Tool bench and evaluation suite, yours to run
- 04Audit-trail schema and retention policy
- 05Runbooks and escalation path
- 06Source, prompts and IP assignment
Typical stack
Python · TypeScript · OpenAI / Anthropic APIs · Open-weight models · PostgreSQL · OpenTelemetry · AWS / Azure / GCP
Enterprise AI SaaS
Document-grounded enterprise chat with audit trails
VectraGPT answers questions from a company's own documents. Every answer carries its sources and every exchange is logged. We built the retrieval stack end to end, including multi-tenant serving.
Read the engagement- Multi-tenant
- isolated serving per customer
- Sourced
- citations on every answer
- GDPR / SOC 2
- audit trails and data handling
Start
How AI Agent Development begins
AI Pilot $28K–60K · Production builds $75K–180K
Where an engagement lands inside its band is set by data readiness, integration depth, the reliability bar, evaluation burden, deployment constraints and usage economics. All six, explained.
Thirty minutes on the problem and where it happens, then a fixed-price pilot if it looks like a fit.
Get a costed planFAQ
Asked before signing
We measure it. Our published tool-failure benchmark ran three models through six injected failure modes 219 times and recorded how often each asserted a value no tool had returned. That harness becomes your evaluation suite, run against your own tools and your own data.
Not hallucination — abandonment. In our benchmark, two of three models never retried a tool that explicitly returned "503 temporarily unavailable, retry", and reported the data as unavailable instead. Transient errors are the most common failure in production, so that behaviour surfaces constantly.
Whichever clears the acceptance criteria at the lowest cost per completed task. Our own results show the ranking is not simply largest-is-best: on transient-error recovery the smallest model we tested outperformed the two larger ones outright. The evaluation decides, not the brochure.
Yes. Open-weight models can be served on your own hardware, and the same bench and evaluation suite run unchanged against them so the comparison is like for like.
Often paired with
Where this fits with the rest
- 01Core AI
LLM & VLM Applications
Retrieval, agents, and document and image understanding that hold up on real data.
- 02Core AI
End-to-End AI Development
First workshop to an operated system, one accountable team. Agents, retrieval, custom models.
- 03Start here
AI Pilot
Four weeks, fixed price, signed acceptance criteria. Working software, or a written case for stopping.