Why Does a Lightweight Model Lose the Finest Defects First?

EDGE AI

A 30 μm trace nick is 3 px at 10 μm/px, but at a student output stride of 4 it covers 0.75 of a cell and can vanish whatever distillation temperature you pick.

For edge segmentation inspecting PCB traces at 4,096 × 3,000 px, I calculated the soft-label temperature, the feature distillation layer, boundary weighting and the student output stride in turn.

About 12 min read

In PCB trace inspection, a server-class segmentation model paints even a 30 μm wide trace nick accurately along its boundary. Try to put the same model on an edge device beside the line, and compute and memory exceed the allowed range several times over. So a lightweight model with fewer channels gets trained from scratch, and while the overall average metric barely drops, the boundaries of only the finest defects smear or vanish entirely.

The surface comes first. Copper traces return light in one direction like a mirror, while the surrounding solder mask scatters it widely. With specular and diffuse reflection mixed in one frame, coaxial lighting does most of the work and a low-angle ring light helps with the step at the nick edge, yet even with this combination the brightness transition at the nick boundary changes with the condition of the trace surface. So the effect sizes below cannot be guaranteed before a sample test, and the calculation starts from the premise that the boundary transition spans 2 to 3 px.

Here are the assumptions. A 4,096 × 3,000 px camera with 3.45 μm pixels and a 35 mm lens captures a 41 mm × 30 mm FOV at 10 μm/px from about 137 mm at the lens principal plane (a WD¹ of 110 mm or more secured). The finest defect is a trace nick 30 μm wide, and the image is inferred as 48 tiles of 512 × 512 px (8 × 6). The teacher is a server-class segmentation model, and the student is a lightweight model with fewer channels.

3 px nicks barely move the average metric

At 10 μm/px a 30 μm nick is 3 px wide. If one defect is 3 × 10 px, it occupies 30 pixels of a 512 × 512 px tile, just 0.011 % of the total, so the average mIoU² barely moves even when this defect is missed. The dashboard looks normal while precursors of trace opens ship as they are, and when they surface late in electrical test or the customer process, rework and return costs come back several times over.

So I split the evaluation metrics before designing any distillation. Without looking separately at recall³ by class and by defect width band, there is no telling whether distillation saved or killed the finest defects. Distillation is a tool that moves information the teacher has, and it cannot create contrast the optics never produced.

Temperature sets how many boundary pixels become signal

Human-made hard labels split defect and background into a 0-to-1 step. In real images the defect boundary is recorded as a gentle brightness transition spanning 2 to 3 px because of the lens point spread function⁴ and the lighting angle, and the teacher model outputs intermediate probabilities such as 0.3 or 0.6 in this stretch. A soft label⁵, taken as the softmax of the logits divided by a temperature T, turns those differences in intermediate probability into gradients large enough for the student to learn from.

Numbers make the role of T clear. Assume the teacher’s defect-versus-background logit difference is +4 inside the defect, 0 at the boundary, −4 one pixel outside and −8 two pixels outside. At T = 1 the defect probabilities are 0.98, 0.50, 0.018 and 0.0003, so the outside sticks to 0 at once, while at T = 4 they are 0.73, 0.50, 0.27 and 0.12, leaving a signal two pixels out. T = 2 leaves 0.12 one pixel out and 0.018 two pixels out, and T = 6 still leaves 0.21 two pixels out, blurring even the line between defect and background, so for this setup with a 2 to 3 px boundary transition, choosing T between 2 and 4 makes sense.

The loss is L = 0.5 · CE(hard) + 0.5 · T² · KL(soft). The T² factor is a customary term that restores the gradient size reduced by the temperature, and the KL divergence⁶ measures the difference between the teacher and student probability distributions. By my standard, T is a parameter whose sweep range can be set only after the defect width and the spread width are known.

Feature distillation layers are chosen by cell size in μm

Distilling only the output probabilities teaches the student what to paint but not what it looks at while painting. A lightweight model has few channels and cannot keep enough high-frequency edge information in its early layers, and the soft label at the output is already compressed through many layers, so its signal is too weak to revive fine edges lost along the way. Feature map distillation⁷ matches the channel counts of intermediate teacher and student feature maps with a 1 × 1 adapter⁸ and adds the difference between the two maps to the loss.

Which layer is chosen is decisive here. For a 512 × 512 px tile, the feature map of the stride⁹ 8 layer is 64 × 64 cells, and each cell represents 8 px, or 80 μm. A 30 μm nick is less than half a cell wide but large enough to change the cell’s feature value clearly. In a stride 32 layer one cell is 320 μm, and the trace of the nick is indistinguishable from the solder mask texture. So I first multiply the pixel resolution by the stride to get the size of one cell, and choose for distillation the deepest layer in which the defect width is at least a third of a cell.

Boundary bands stay at 0.27 % even with a 5-fold weight

Pixel-wise distillation loss is dominated by the number of background pixels. Even if the teacher knows the boundary exactly, the student does not learn it when that information makes up a small share of the total loss. So a boundary weighting mask that gives a 5-fold soft-label loss weight to pixels within a ±3 px band around the defect boundary comes along.

Yet in numbers, 5 times is not much. The ±3 px band around one 3 × 10 px defect is 9 × 16 = 144 px, 0.055 % of the tile, and even with a 5-fold weight its share of the loss is 0.27 %. Band weighting alone struggles to revive the boundary signal, so in my case I would also consider averaging the losses of tiles with and without defects separately and fixing the share of the defect tiles. The weight of the feature map distillation on the stride 8 layer starts at 0.1.

Output stride 4 turns a 3 px nick into 0.75 of a cell

Some defect classes vanish even with well-designed distillation, and the cause usually lies in the output resolution of the student. Lightweight segmentation models commonly predict at output stride 4 and then upsample to cut computation, but a defect narrower than one output cell is buried in the background within the cell average, and upsampling cannot revive information that is already gone.

Predict a 3 px nick at output stride 4 and the defect is 0.75 of an output cell wide. Depending on how the cell grid lines up with the defect, it is caught in some samples and lost in others, a detection without reproducibility. Lowering the output stride to 2 makes the same nick cover 1.5 cells, so the defect fills at least half of at least one cell. Computation rises, but keeping only the last decoder stage at high resolution limits the increase, and changing the temperature from 2 to 6 does not solve this 0.75-cell problem.

So there is one formula to check before distillation. First calculate whether minimum defect width (px) ÷ student output stride ≥ 1 holds, and if it does not, change the architecture or the optical magnification in place of the distillation parameters. Software training techniques alone cannot cover a shortage of pixel resolution.

Conditions for catching a 30 μm nick in one table

Size by defect type, optical setup and distillation parameters are gathered in one table. Every cell size is 10 μm/px multiplied by the stride.

Item	① Minimum defect size	② Optical setup · WD	③ Algorithm parameters
Trace nick	Width 30 μm = 3 px (10 μm/px)	4,096 × 3,000 px, 3.45 μm pixels, 35 mm lens (magnification about 0.34), FOV 41 × 30 mm, WD 110 mm or more (measured after subtracting barrel protrusion and the coaxial housing from the roughly 137 mm object distance at the principal plane)	Student output stride 2 (1.5 cells), 0.75 cell at stride 4
Copper spur	Width 40 μm = 4 px	Coaxial light with a low-angle ring light assist	Soft label T = 4 (sweep 2 to 6), L = 0.5 CE + 0.5 T² KL
Solder mask pinhole	Diameter 50 μm = 5 px	Copper specular and solder mask diffuse reflection mixed	Feature map distillation: stride 8 layer (80 μm cell), 1 × 1 adapter, MSE weight 0.1
Loss balance	±3 px band of a 3 × 10 px defect = 144 px	Tiles of 512 × 512 px, 48 tiles (8 × 6)	Boundary band weight ×5 (0.27 % of the loss), defect-tile losses averaged separately

The first number that catches in the table is 0.75 of a cell. The finest 30 μm nick sits at 3 px at 10 μm/px, the floor of the detection premise, and at a student output stride of 4 it can vanish regardless of the distillation design. Setting stride 2 comes first, and if that exceeds the compute budget, the FOV has to shrink to raise the pixel resolution, accepting the trade that the 48 tiles grow and the station count and cycle time rise with them.

When to skip distillation

Distillation does not always pay. If the minimum defect occupies 10 px or more and stands out clearly from the background, training the lightweight model directly on hard labels is enough, and the time and compute spent training a teacher go to waste. With a generous edge compute budget or few tiles, deploying the teacher with quantization alone can be better for boundary reproducibility. And with low label quality, distillation faithfully copies the boundaries the teacher learned wrong into the student, so label cleanup comes before distillation.

The effect size of distillation depends on defect shape and surface reflection. On strongly reflective surfaces such as copper traces, mixed-in diffuse reflection shakes the teacher’s boundary probabilities themselves, so the temperature and weights above are only starting points and the actual effect cannot be guaranteed before a sample test.

Field Note

If boundaries from a teacher have to move into a lightweight student, I calculate the student’s output stride before touching any distillation setting. If the minimum defect width divided by the stride is below 1, fine defects fall below one output cell no matter how the temperature changes. Next I choose the sweep range of T from the defect width and the boundary transition width, and tally the evaluation separately as recall by defect width band. The size of the improvement is tied to the board and lighting conditions, so I confirm it again on samples.

Field Checkpoints

Before starting a distillation experiment, I fill in the items below. With even one number missing, the yardstick for reading the results disappears.

• Is a WD of 110 mm or more secured by measurement, including the coaxial light housing and the ring light height? (object distance about 137 mm from the principal plane)
• Have you measured on samples the brightness transition width at the nick boundary in a frame that mixes copper specular and solder mask diffuse reflection?
• Does minimum defect width (px) ÷ student output stride ≥ 1 hold? (if not, change the architecture first)
• Have you calculated the cell size (μm/px × stride) of the layer used for feature map distillation? (80 μm at stride 8)
• Do you split the evaluation metrics into recall by class and by defect width in place of average mIoU?
• Have the results of the temperature T sweep been recorded against the recall of the finest defect class?

Frequently Asked Questions

How do I judge whether distillation worked?

Put a student trained on hard labels alone next to a distilled student of the same architecture, and compare the recall of the 3 px nick class with the overdetection rate held at the same value. If this recall differs while average mIoU stays the same, distillation did its job; if both are low, suspect the output stride or optical magnification.

How are nicks that straddle tile boundaries handled?

Segmentation places an overlap between tiles and merges the results in the overlap by keeping only the center-side predictions of each tile or by averaging the probabilities. Overlap width and the latency that comes with the tile count are calculated separately in the tiled inference article.

Glossary

• [1] WD working distance
  The distance from the front of the lens to the surface under inspection. The roughly 137 mm in the text is a thin-lens object distance from the lens principal plane; the real clearance after subtracting barrel protrusion and the light housing is confirmed by measurement. ↩
• [2] mIoU mean intersection over union
  The average over classes of the overlap ratio between predicted and true regions. Large classes and the background dominate the value. ↩
• [3] Recall recall
  The share of actual defects that the model finds. Computed separately by defect width band, it exposes misses of fine defects. ↩
• [4] Point spread function PSF
  The shape in which light from one point spreads on the sensor after passing through the lens. This spread blurs defect boundaries over 2 to 3 px. ↩
• [5] Soft label soft label
  A label that uses the probability distribution produced by the teacher model itself as the target. Dividing the logits by a temperature T before the softmax flattens the distribution. ↩
• [6] KL divergence Kullback-Leibler divergence
  A measure of how different two probability distributions are. In distillation it measures the gap between the teacher and student soft labels. ↩
• [7] Feature map distillation feature map distillation
  A method that trains the student to match the teacher’s intermediate feature maps in place of only the output probabilities. ↩
• [8] 1 × 1 adapter 1 × 1 convolution adapter
  A small convolution layer that mixes only channels at each pixel so that two feature maps with different channel counts can be compared. ↩
• [9] Stride stride
  The factor that tells how many input pixels one cell of a feature map or output represents. At stride 8, one cell covers 8 × 8 px. ↩

References

• Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network, arXiv:1503.02531 (2015). A knowledge distillation paper that develops the idea of Caruana and collaborators of compressing the knowledge of an ensemble into a single model.
• Formulas: defect width (px) = defect width ÷ pixel resolution, cell size = pixel resolution × stride, output cell occupancy = defect width (px) ÷ output stride, two-class soft-label probability = σ(logit difference ÷ T), loss L = 0.5 · CE + 0.5 · T² · KL, band share of the loss = 5 × band pixels ÷ (all pixels − band pixels + 5 × band pixels), tile count = ⌈4,096 ÷ 512⌉ × ⌈3,000 ÷ 512⌉, magnification m = sensor width ÷ FOV width, object distance = f × (1 + 1/m)
• Assumptions: 4,096 × 3,000 px camera (3.45 μm pixels), 35 mm lens, FOV 41 × 30 mm, minimum defect 30 μm (3 × 10 px), tiles of 512 × 512 px, teacher logit difference +4 (inside), 0 (boundary), −4 (one pixel out), −8 (two pixels out), boundary band ±3 px, weight ×5

Related reading — Where Do Fine Defects Go When You Shrink to 640 × 640? · Why the Faintest Scratch Disappears First Under INT8 Quantization · Brightness Augmentation Cannot Move Shadows or Highlights