# RAG Production Acceptance Benchmark 2026 - Methodology

## Frozen inputs

- Dataset: ParticleMedia/RAGTruth commit c103204b9ce28d6bbad859304bf30de72b8ed8fe
- Upstream: https://github.com/ParticleMedia/RAGTruth
- Licence: MIT
- Evaluator: pran-rag-gates/1.0.0
- Execution date: 2026-08-23
- Train records: 14942
- Test records: 2675

The official RAGTruth train split calibrates one threshold per gate by maximum response-level F1. Ties choose higher precision. The official test split is reported. Records whose quality field is not `good` are excluded. Annotation spans marked `implicit_true` are treated as supported.

## Gates

1. **Corpus token coverage.** Lowercases alphanumeric content tokens, removes a fixed English stop-word list, and blocks when the overall unsupported-token share crosses the train-calibrated threshold.
2. **Weakest-sentence support.** Splits on sentence-ending punctuation, ignores fragments with fewer than five content tokens, and blocks when any remaining sentence crosses the unsupported-token threshold.
3. **Hybrid claim gate.** A fixed score: 35% overall unsupported share, 50% weakest-sentence unsupported share, and 15% source-missing numeric-token share. The score threshold is calibrated on train.

## Metrics

Precision, recall, F1 and false-negative rate are computed at response level against RAGTruth's manually annotated hallucination spans. Latency is the observed local feature-extraction time divided by eligible responses. Provider cost is $0.00 because all three gates run locally without a model API.

## Reproduce

`RAGTRUTH_DIR=/path/to/RAGTruth node scripts/benchmarks/rag-acceptance.mjs`

## Limitations

These are intentionally simple local lexical gates, not a claim that lexical overlap establishes truth. They cannot reliably detect contradictions, paraphrases, or unsupported claims built from source vocabulary. Threshold calibration maximizes F1 rather than a safety-weighted objective. Production teams should set the false-negative budget from the consequence of an unsupported answer and test on their own task distribution.

## Citation

Wu, Yuanhao et al. “RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.” arXiv:2401.00396 (2023).