# Vision Detection Deployment Benchmark 2026 — methodology

## Decision under test

Select the lowest-latency YOLO11 configuration whose COCO mAP50–95 is within
2.0 absolute points of YOLO11s at 640 pixels. The result is a generic deployment
candidate, not an industrial-defect acceptance result.

## Frozen protocol

- Dataset: COCO 2017 validation split, all 5,000 labelled images and 80 classes.
- Checkpoints: official Ultralytics YOLO11n and YOLO11s detection weights.
- Local matrix: 320, 480 and 640-pixel square inputs using PyTorch/MPS on an
  Apple M4 Pro.
- Portable cloud check: 640-pixel ONNX Runtime CPU inference on a temporary
  Google Cloud C4 `c4-standard-4` Spot VM.
- GPU cloud check: 640-pixel PyTorch/CUDA inference on one NVIDIA L4 in a
  Google Cloud Run job in `europe-west4`.
- Accuracy: mAP50–95, mAP50, mAP75, aggregate precision and recall, plus
  person-class mAP50–95.
- Latency: batch-one, end-to-end `predict` calls over a deterministic 32-image
  sample after 20 warm-up calls; 200 observations per configuration.
- Seed: 20260824.

Every result records its execution environment, package versions, model SHA-256,
raw wall-clock observations, evaluator version and export time. Public summary
files omit only the 200-element raw timing arrays; the runtime-specific raw JSON
retains them.

## Capacity calculation

`safe aggregate FPS = 500 / p95 end-to-end latency in milliseconds`

This reserves 50% of the measured p95 budget. Camera counts divide that safe
aggregate rate by a scenario rate of 2, 5 or 10 inspected frames per second.
They exclude camera decoding, orchestration, network transport, PLC integration
and application I/O, and therefore are sizing hypotheses rather than guarantees.

## Cloud execution and constraint

The first Cloud Run attempt in `us-central1` was denied at zero non-zonal L4
quota. A preference for one L4 was submitted there through the Cloud Quotas API
with trace ID `6c9361ca-5525-4176-b241-a5d2dd859be5`; it remained pending when
the report was frozen. A subsequent deployment in `europe-west4` received a
three-L4 first-deployment allowance and successfully ran one L4. Both 640-pixel
checkpoints were validated on all 5,000 images and timed for 200 batch-one calls.
The report contains this PyTorch/CUDA result and the earlier C4 CPU/ONNX result
as separately labelled portability evidence. TensorRT was not executed, and the
runtime chart is not a controlled hardware comparison.

## Limitations

COCO is a generic object-detection dataset. It does not contain a customer's
parts, defect taxonomy, optics or line conditions, so these results cannot imply
defect reduction, labour savings or escape-rate improvement. A production gate
must be repeated with rights-cleared site images and class-specific false-negative
limits. Ultralytics model and software licensing must also be reviewed for the
intended production use.