Vision systems rarely fail loudly. They get slightly worse over months, and the first person to notice is usually a customer.
Why accuracy moves
New product variants, replaced lighting, a camera nudged during maintenance, a different supplier's material finish, seasonal daylight through a bay door. Every one of these shifts the input distribution away from what the model was trained on.
Monitor the inputs, not just the outputs
Detection rate alone is a lagging indicator. Track image-level statistics — brightness, contrast, sharpness — and the distribution of confidence scores. Both move before accuracy does, which buys you time to act.
Sample ground truth affordably
You cannot manually verify everything, and you do not need to. A small randomised daily sample, verified by whoever already does quality checks, is enough to detect a meaningful change. Weight the sample toward low-confidence cases and you will find problems faster.
When to retrain
When monitoring says so, not on a calendar. In practice that tends to be two to four times a year on a stable process, plus any product or tooling change. Retraining for its own sake risks making things worse and costs validation effort each time.
Keep the door open
Retraining requires the dataset, the annotation schema and the training code. If your vendor holds any of those, you are not able to maintain the system — which is why we hand all three over at the end of an engagement.