# Million-request citation-first RAG benchmark 2026 — methodology

## What was measured

This experiment measures a narrow retrieval service for repetitive industrial
maintenance questions. It returns a grounded extract and three citations. It
does not call a generative model per request, so the reported latency and cost
must not be described as uncached LLM generation.

The deterministic corpus contains 20,000 synthetic troubleshooting chunks
across 20 equipment families, 25 model identifiers and 40 symptoms. Each chunk
has a unique fault code and source identifier. This makes system correctness
measurable but does not establish real-world maintenance safety or usefulness.

## Final stack

- Global external Application Load Balancer with a serverless NEG.
- Cloud Run in `us-east1`: Go 1.24, 2 vCPU, 1 GiB, concurrency 80, min 1,
  max 10, request-based billing.
- Cloud SQL for PostgreSQL 16 Enterprise in `us-east1`:
  `db-custom-2-7680`, 10 GiB SSD, temporary zonal instance.
- Exact fault-code match followed by reciprocal-rank fusion of PostgreSQL
  full-text and 64-dimensional pgvector HNSW candidates.
- Per-instance 10,000-entry response cache with a 15-minute TTL.
- Dedicated service identity, Secret Manager, application bearer token,
  Cloud SQL connector, 12-connection pool and two-second query timeout.

The first `us-central1` Cloud Run service returned a Google front-end 404 and
recorded no request logs. Moving the complete stack to `us-east1` produced the
same result for its new default hostname, so the final measured service used a
global external Application Load Balancer. The forwarding rule's first probe
failed during propagation; the identical subsequent probe passed.

## Quality gate

The first ranker recovered the expected synthetic source in the top three for
only 33.8% of 500 queries. Its fast latency was rejected. The final ranker adds
an exact manufacturer fault-code lane before hybrid fusion and achieved 100%
recall@1 and recall@3 on the same deterministic query construction. This is a
systems-quality gate over synthetic codes, not a semantic-quality claim.

## Load protocol

The load generator ran as a single Cloud Run job in `us-east1` and sent traffic
through the public load-balancer edge. It used persistent HTTP connections, 200
workers, a seven-second client timeout and a fixed-rate scheduler.

- Hot: 100-question set; only initial per-instance requests query Cloud SQL.
- Cold: every request bypasses the response cache.
- Mixed: 95% hot questions and 5% forced cache bypass.

The final run scheduled 1,000,000 mixed requests at 1,000 requests/second. The
job retained all one million client latencies in memory, sorted them at the end
and emitted p50, p90, p95, p99, p99.9, maximum, cache headers, status codes and
a fixed latency histogram as one structured log result.

## Cost method

Cost uses public U.S. list prices captured on 2026-08-24 and excludes free-tier
credits and discounts. The measured million-run Cloud Run CPU and memory values
come from Cloud Monitoring allocation metrics. Request charges use the exact
one-million count. Load-balancer processing and internet egress use measured
response-body bytes; request headers were not captured. The internet egress
line is therefore presented as a conservative upper bound, not an invoice.

The monthly production model includes the same Cloud SQL shape at regional-HA
CPU and memory rates, duplicated HA SSD pricing, one global forwarding rule and
one always-warm Cloud Run instance at idle rates. Marginal measured cost is then
added per million requests. LLM inference, customer-document ingestion,
observability retention, support labour and taxes are excluded.

## Failures retained

- TLS negotiation was initially attempted over the Cloud SQL Unix socket.
- A production two-second query timeout killed the one-time HNSW index build.
- The first hybrid ranker failed recall@3.
- The first edge probe ran before load-balancer propagation.
- The first 1,000-RPS burst exposed per-instance cache warmup at 221.5 ms p95.
- One 100k load run used a client header timeout shorter than the service
  timeout and exceeded the error budget by nine requests.

Each failure was explained and corrected before the million-request stage. None
is removed from the raw evidence.