Core practice
Language and vision models, held to a number
A model that answers well in a demo has proved nothing. We build LLM and VLM applications against a held-out set, a hallucination budget and a cost ceiling, and they do not ship until they clear all three.
LLM & VLM Applications
The situation
Before we start
It worked on the ten examples in the deck
Real documents are scanned, inconsistent and contradictory. We evaluate on a held-out set drawn from what you actually hold, not on the examples that sold the idea.
Nobody set a hallucination budget
Every language model makes things up at some rate. The question is the rate you can live with, written down, and measured against on every release.
The bill arrived after the launch
Retries, long contexts and agent loops multiply cost quietly. Cost per completed task is metered from the first build and has a ceiling in the acceptance criteria.
What working means here
Signed first. Measured after.
The value of each one is set with you during scoping, from your data and your line. What does not change is that they are written down first and published against afterwards, pass or fail.
- 01
Answer accuracy on a held-out set
Graded against reference answers your team helped write, published pass or fail
- 02
Hallucination rate
Unsupported claims per hundred answers, with a budget agreed before build
- 03
Cost per completed task
Metered end to end, including retries and tool calls, against the ceiling you signed
We publish the measurements behind this practice, including the runs that failed.
RAG Production Acceptance Benchmark 2026How it is assembled
Stage by stage
Each one ends in something you keep.
- Step 01
Define the task
One task, one user, one failure mode that matters. Agreed in writing before anything is built.
- Step 02
Assemble the held-out set
Real examples from your own documents, graded with your team, kept apart from everything used to build.
- Step 03
Build behind the evaluation
Retrieval, agents and model choice iterated against the suite; nothing changes without a score.
- Step 04
Meter and route
Cost per task measured end to end; cheaper models routed in where quality holds.
- Step 05
Ship and hand over
Deployed in your accounts with runbooks, or operated under an agreement.
Scope
What the work includes
- Use-case definition: the task, the user, the failure that matters most
- Data review: what you hold, in what shape, and what it can support
- Retrieval over your documents, with citations on every answer
- Agents with explicit permissions, tool budgets and audit logs
- Document and image understanding with vision-language models where text alone is not enough
- An evaluation suite with a held-out set, run on every change
- Cost metering per completed task, with routing to cheaper models where quality holds
- Deployment in your accounts under your identity provider
Deliverables
What you keep
- 01Signed acceptance criteria: accuracy, hallucination budget, cost ceiling
- 02Working application in your environment
- 03Evaluation suite and held-out set
- 04Cost-per-task reporting
- 05Runbooks and administrator training
- 06Source, prompts, weights and IP assignment
Typical stack
Python · TypeScript · PyTorch · vLLM · pgvector · PostgreSQL · AWS / Azure / GCP
Enterprise AI SaaS
Document-grounded enterprise chat with audit trails
VectraGPT answers questions from a company's own documents. Every answer carries its sources and every exchange is logged. We built the retrieval stack end to end, including multi-tenant serving.
Read the engagement- Multi-tenant
- isolated serving per customer
- Sourced
- citations on every answer
- GDPR / SOC 2
- audit trails and data handling
Start
How LLM & VLM Applications begins
AI Pilot $28K–60K · Production builds $75K–180K
Where an engagement lands inside its band is set by data readiness, integration depth, the reliability bar, evaluation burden, deployment constraints and usage economics. All six, explained.
Thirty minutes on the problem and where it happens, then a fixed-price pilot if it looks like a fit.
Get a costed planFAQ
Asked before signing
Whichever clears the evaluation at the lowest cost per task. That is usually a mix: a capable model for the hard cases and a smaller one routed in where quality holds. The suite decides, not a preference.
Yes. Open-weight models can be served in your cloud or on your hardware; hosted models are called from your accounts. Either way the data path is documented for review.
An agreed rate of unsupported claims per hundred answers, set from what the use case can tolerate. It is measured on the held-out set on every release, and a release that exceeds it does not ship.
Yes. Vision-language models read forms, drawings, photographs and scanned documents where text extraction alone fails, and they are held to the same accuracy and cost criteria.
Often paired with
Where this fits with the rest
- 01Core AI
AI Agent Development
Agents with permissions, budgets and a tested failure path — not a demo that works until a tool breaks.
- 02Core AI
End-to-End AI Development
First workshop to an operated system, one accountable team. Agents, retrieval, custom models.
- 03Core AI
AI Product Development
AI-native products, and AI features inside products that already have customers.