Computer Vision in Production — The Testing Checklist

Computer vision models are notoriously bad at generalizing. A model trained on clean data fails on messy inputs. A model that hits 95% accuracy in your test set can collapse to 60% in the field.

Here's the testing checklist we use at QA Labs before any CV model ships.

Test against real-world conditions

Lab accuracy ≠ field accuracy. Test against:

Real-world lighting (not studio)

Different camera angles

Motion blur

Occlusion (partial obstructions)

Weather conditions (for outdoor)

Different times of day

Test on edge devices

Latency and accuracy change dramatically on edge hardware:

Measure inference time on target devices

Check memory usage under load

Verify battery impact

Test model quantization (if applicable)

A model that runs fine on GPU servers might time out on a Raspberry Pi.

Test for bias

CV models often perform worse on underrepresented groups:

Measure accuracy across demographics

Check for false positives/negatives by group

Test with diverse datasets

Audit training data for representation

Bias testing is non-negotiable for any CV model used in decisions affecting people.

Test for model drift

Model accuracy degrades over time:

New patterns appear

Data distributions shift

Environments change

Plan for periodic retraining. Set up monitoring for accuracy drops.

Test adversarial inputs

CV models can be fooled by:

Adversarial perturbations (small pixel changes)

Out-of-distribution inputs

Trick images

For security-sensitive applications, adversarial testing is required.

What we typically find

Models trained on studio data fail on real-world images

Latency on edge devices exceeds SLA

Demographic bias in face-related models

Accuracy drops 10–20% over 6 months without retraining