It Worked Fine on the Server — Why Does Accuracy Drop on the Edge?

It’s a common pattern: accuracy that looked fine when a model was validated on a server drops the moment it’s loaded onto an edge board. The cause is usually not the model itself, but quantization.

Why Quantization Is Needed

Format	Characteristics
FP32 (server-training default)	High precision, heavy compute/memory load
INT8 (edge-deployment)	4x faster compute, 1/4 the memory, some precision loss

An edge board doesn’t have server-class GPU memory or power, so loading an FP32 model as-is drops the frame rate below a practical level. Quantizing to INT8 recovers the speed, but at the cost of some accuracy loss relative to the original.

How to Reduce Accuracy Loss

Use quantization-aware training (QAT) to fold quantization error into training from the start
Check per-class accuracy separately — the overall average can look fine while one specific defect type drops sharply
Decide the speed/accuracy trade-off together with the inspection criteria (the threshold settings from the companion article) from the outset

Summary

If accuracy drops on edge deployment, the sensible order is to suspect the quantization stage before the model itself. In exchange for speed, you need to check, class by class, exactly where precision is being given up.

Related Reading — Four Snags in Initial Jetson Orin Nano Setup