Skip to content
Quality Assurance Labs
AI Apps & Integration

Computer Vision in Production — The Testing Checklist

Kawsher, Founder, QA Labs2 min readPublished Updated

A CV model that hits 95% accuracy in the lab can drop sharply in production. Here's the checklist we use to catch that before users do — lighting, edge devices, bias, latency, drift, and adversarial inputs.

Camera inspecting objects with detection frames
  • computer-vision
  • edge-AI
  • model-testing
  • ML-QA
  • TensorFlow-Lite

Key takeaways

  • Test CV in real conditions, not just the lab
  • Edge devices expose latency and accuracy issues
  • Bias testing is non-negotiable
  • Plan for drift and retraining
  • Adversarial testing for security-sensitive deployments

Computer vision models are notoriously bad at generalizing. A model trained on clean data fails on messy inputs. A model that hits 95% accuracy in your test set can collapse to 60% in the field.

Here's the testing checklist we use at QA Labs before any CV model ships.

Test against real-world conditions

Lab accuracy ≠ field accuracy. Test against:

  • Real-world lighting (not studio)
  • Different camera angles
  • Motion blur
  • Occlusion (partial obstructions)
  • Weather conditions (for outdoor)
  • Different times of day

Test on edge devices

Latency and accuracy change dramatically on edge hardware:

  • Measure inference time on target devices
  • Check memory usage under load
  • Verify battery impact
  • Test model quantization (if applicable)

A model that runs fine on GPU servers might time out on a Raspberry Pi.

Test for bias

CV models often perform worse on underrepresented groups:

  • Measure accuracy across demographics
  • Check for false positives/negatives by group
  • Test with diverse datasets
  • Audit training data for representation

Bias testing is non-negotiable for any CV model used in decisions affecting people.

Test for model drift

Model accuracy degrades over time:

  • New patterns appear
  • Data distributions shift
  • Environments change

Plan for periodic retraining. Set up monitoring for accuracy drops.

Test adversarial inputs

CV models can be fooled by:

  • Adversarial perturbations (small pixel changes)
  • Out-of-distribution inputs
  • Trick images

For security-sensitive applications, adversarial testing is required.

What we typically find

  • Models trained on studio data fail on real-world images
  • Latency on edge devices exceeds SLA
  • Demographic bias in face-related models
  • Accuracy drifts down over time without retraining, as real-world data changes

Building a test set that reflects reality

  • Collect images from the cameras and devices that will be used in production
  • Cover different lighting, angles, distances, weather and backgrounds
  • Include examples from every group the model must treat fairly
  • Label edge cases and failure examples, not only clean ones
  • Keep the test set separate from training data and version it

Frequently asked questions

How often should a vision model be retrained? — When monitoring shows accuracy falling or the inputs change, for example new cameras, new products or new locations. There is no fixed schedule.

Can we test a vision model without labelled data? — Only partly. You can watch confidence scores and input statistics for drift, but measuring accuracy needs a labelled sample of real inputs.

What is the most common production surprise? — Inputs that look different from the training data, such as poor lighting, blur or new backgrounds, which lower accuracy without any obvious error.

Free AI feature review

Want us to test your AI feature?

Send us the address of your chatbot, assistant or AI feature. An engineer will try it with realistic and tricky questions and email you the wrong answers, made-up facts and data-safety issues we find. Free, and no call needed.

Related service

AI Apps & Integration →

Chatbots, LLM integrations, and AI agents tested for accuracy before they meet your users.

Further reading

About the author

Kawsher, Founder, QA Labs →

Kawsher is the founder of Quality Assurance Labs (QA Labs), the QA-first software testing and development company in Bogura, Bangladesh, founded in 2021. Every article and case study on this site is written or reviewed by Kawsher, drawing on the team's QA testing, AI, web, mobile, video and marketing work for SaaS and AI teams worldwide.

Notes from the lab.

Testing, engineering and growth — delivered to your inbox.

Need CV model QA? Book a scoping call

Let's talk →