Why an Average Latency Inside the Deadline Is Not Safe

EDGE AI

An average of 33.6 ms is just over half of the 60 ms deadline, yet the per-stage p99 values add up to 53.1 ms, leaving only 6.9 ms.

For PCBs passing at 300 mm/s, I split the path from trigger to decision signal into seven stages and calculated the average, the p99, the pipeline and the timeout in turn.

About 11 min read

When an edge vision inspection system comes in, the first number reported is usually the average inference time. If the benchmark screen shows 9 ms, the inspection cycle is judged fast enough and the project moves on. Yet the moments when the decision signal misses the deadline at the reject point come from the tail, not the average. A part that receives a late decision passes regardless of the result, so a system that is late once in 1,000 parts looks fine on average metrics, and if that one part is defective, inspection leaves a record and ships the defect.

The latency budget also starts at the surface. The targets are solder bridges and board particles, and solder surfaces mix specular and diffuse reflection, the kind of material that tempts you to lengthen the exposure when the light is weak. Exposure time ties directly to motion blur, and blur erodes the edges of a 3 px defect, so the time calculation below is not separate from the optical conditions. Because the required exposure and preprocessing load change with the reflection of the solder surface, the per-stage numbers below cannot be guaranteed before a sample test.

Here are the assumptions. Parts move at 300 mm/s, and the distance from the camera station to the reject point is 18 mm. The camera has 2,448 × 2,048 px with 3.45 μm pixels and a 25 mm lens, captures a 49 mm × 41 mm FOV at 20 μm/px from about 170 mm at the lens principal plane (a WD¹ of 160 mm or more secured), and the light is a 30 μs strobe². Parts arrive at 40 ms intervals, and per-stage latencies are taken from 10,000 or more timestamps recorded in continuous operation. The control conditions of the conveying mechanism that moves the parts need a separate review; only the vision path from trigger to decision signal is calculated here.

Distance and speed set the 60 ms deadline

What the line demands is a decision signal before the part reaches the reject point. This deadline³ is set by dividing the distance from the camera to the reject point by the part speed: 18 mm ÷ 300 mm/s = 60 ms. Inside it sit the optical stretch of exposure and sensor readout, the data path of transfer, the compute stretch of preprocessing, inference and post-processing, and the signal stretch in which the operating system drives a digital output. Inference is only one of seven cells.

So I do not measure from model input to output; I measure from trigger⁴ input to decision signal output. Every stage needs a timestamp⁵ from the same clock to show which cell wobbles, and the latency of the trigger input itself goes on the list separately at 0.07 ms. That is why the first table of a design review is the seven-stage timestamp table in place of a model benchmark.

Averages sum to 33.6 ms, p99 values to 53.1 ms

The per-stage p50⁶ values are trigger 0.07 ms, exposure 0.03 ms, readout 8 ms, transfer 12 ms, preprocessing 3 ms, inference 9 ms, post-processing 1 ms and signal output 0.5 ms, for a sum of 33.6 ms. On averages alone, it is a relaxed system using just over half the deadline. During the 30 μs exposure a part moves 300 mm/s × 30 μs = 9 μm, and this motion blur⁷ is 0.45 px at 20 μm/px, which a 3 px defect can absorb.

The p99 of the same system is another story. Readout at 8 ms, the trigger and the exposure stay put, but transfer grows to 18 ms, preprocessing to 6 ms, inference to 15 ms, post-processing to 3 ms and signal output to 3 ms, for a sum of 53.1 ms. The margin left in the 60 ms deadline is 6.9 ms, a margin ratio of 11.5 %, barely above the 10 % baseline I use. Compared with the p50 sum, the tail takes up 19.5 ms.

The sum of per-stage p99 values usually comes out larger than the p99 of the whole path, which makes it a comfortable estimate. Percentiles, however, carry no guarantee that the p99 of a sum stays below the sum of the p99 values, so with a very long-tailed distribution in the mix the sum can come out smaller than the real p99. So I measure the p99 of the whole path from trigger to signal directly, apart from the per-stage table, and put the larger of the two into the budget.

Inference and transfer each add 6 ms of tail

Subtracting p50 from p99 for each stage shows where to start. Transfer and inference are the largest at 6 ms each, followed by preprocessing at 3 ms, signal output at 2.5 ms and post-processing at 2 ms. By ratio the order changes: signal output grows 6 times from 0.5 ms to 3 ms, post-processing 3 times and preprocessing 2 times, while inference grows 1.67 times and transfer 1.5 times.

The reasons for the wobble differ by stage. Transfer grows a long right tail from buffer contention and retransmission, preprocessing and inference from memory allocation and contention for compute units, and signal output from operating system scheduling⁸. So countermeasures are ranked by the largest p99 − p50 ahead of the largest p50, and in this example model slimming and transfer buffer cleanup carry equal weight. The 6-fold growth in signal output looks small at 2.5 ms in absolute terms, but it can be cut at a fairly small cost, such as raising the output thread to a real-time priority, so in my case it is the first thing to fix.

Pipelining shortens the interval and leaves latency alone

If parts arrive every 40 ms and the serial p99 is 53.1 ms, each tail event pushes back the processing of the next part. The 33.6 ms average fits the interval, yet the tail builds a queue, and the queue stretches the latency of the parts behind it in a chain. A pipeline⁹ that groups the seven stages into three stages (readout and transfer, preprocessing and inference, post-processing and signal) and runs them at once makes the slowest single stage, in place of the total, set the processing interval.

The stage p99 values are 26 ms for readout and transfer, 21 ms for preprocessing and inference, and 6 ms for post-processing and signal. The slowest, 26 ms, is shorter than the 40 ms part interval, so no queue builds up. One part still takes 53.1 ms from trigger to signal, though, and buffer copies between stages can even add a little. After parallelizing, the deadline margin is recalculated from scratch.

Timeouts keep late decisions from passing

However well the budget is drawn, the tail beyond p99 remains. Running 16 hours a day at a 40 ms part interval means 1.44 million parts, of which 14,400 exceed the p99 and 1,440 exceed even the p99.9. If the default action for any of them that miss the deadline is to pass, inspection does nothing at that moment.

So the timeout¹⁰ becomes one cell of the budget. If no decision exists by 54 ms, which pulls the 60 ms deadline forward by 6 ms, twice the 3 ms p99 of the signal output stage, the system outputs NG so a late decision never turns into a pass. Logging parts that dropped out by timeout as re-inspection targets lets you separate, on a day when NG rises, whether the cause is the tail or real defects.

One point to pin down is that a timeout is also a software device. Software cannot cover a flaw in the optical setup such as motion blur from a long exposure, and if a short latency budget leads to lowering the light output and stretching the exposure to 60 μs, the blur becomes 0.9 px and directly erodes the edge contrast of a 3 px defect. The latency budget and the optical resolution are tied together by one variable, the exposure time.

One table for the 60 μm defect and the budget

Defect size, optical setup and the per-stage budget are gathered in one table. All numbers are calculated from the assumptions above.

Item	① Minimum defect size	② Optical setup · WD	③ Algorithm parameters · time budget
Solder bridge	Width 60 μm = 3.0 px (20 μm/px)	2,448 × 2,048 px, 3.45 μm pixels, 25 mm lens, FOV 49 × 41 mm, WD 160 mm or more (object distance about 170 mm from the principal plane)	Deadline 60 ms (18 mm ÷ 300 mm/s), per-stage p99 sum 53.1 ms, margin ratio 11.5 %
Board particle	Diameter 100 μm = 5.0 px	Diffuse dome light, 30 μs strobe, trigger-synchronized, blur 9 μm (0.45 px)	Largest tails: transfer +6 ms, inference +6 ms / largest ratio: signal output 6 times
Processing interval	Part interval 40 ms	Stretching exposure to 60 μs makes blur 0.9 px	3-stage pipeline, slowest stage p99 26 ms
Safeguard	A decision past the deadline is NG	Measure WD including the dome light housing height	Timeout 54 ms (pulled forward by twice the 3 ms signal output p99), whole-path p99 measured separately

The first number to read in the table is the 11.5 % margin ratio. The 53.1 ms p99 sum does fit inside the 60 ms deadline, but the gap to the 10 % baseline is only 1.5 percentage points, or 0.9 ms, so a 1 ms rise in the tail of either transfer or inference drops below the baseline. And with the 60 μm solder bridge sitting at the 3.0 px floor, the option of buying time with a longer exposure is closed from the start.

When per-stage control is overkill

Per-stage p99 control does not always pay off. When the reject point is far and the deadline runs to several hundred ms or more, an average-based design with one whole-path timeout is often enough. For sampling inspection whose results feed only logs and statistics, throughput and accuracy come before latency and batch processing is more economical, and when the computation is light, as in rule-based dimensional checks, deciding inside the camera and removing the transfer stage cuts the tail more than splitting a pipeline.

On materials that mix in diffuse reflection, such as solder surfaces, the required exposure and preprocessing load change with the lighting conditions. Whether the per-stage numbers above carry over to other boards and lighting cannot be guaranteed before a sample test, and I fill in the timestamp table from scratch for each new product.

Field Note

Even when the average decision time is half the deadline, I do not call it safe on that number; I timestamp every stage from trigger to result output and look at the p99 separately. I check whether the larger of the per-stage p99 sum and the whole-path p99 fits inside 90 % of the deadline, and I pick tail countermeasures starting with the stage that has the largest p99 − p50. A timeout fixes the decision to NG 6 ms before the deadline, and exposure time is never my knob for meeting the latency budget.

Field Checkpoints

These are the items I take into a design review. If any one is blank, the average latency proves nothing.

• Is a WD of 160 mm or more secured by measurement, including the dome light housing height and the cable bend radius? (object distance about 170 mm from the principal plane)
• Have you checked the reflection of the solder surface on sample boards and set the exposure that condition requires? (30 μs exposure = 0.45 px blur)
• Are timestamps for all seven stages, from trigger input to decision signal output, recorded on the same clock?
• Were per-stage p99 values taken from 10,000 or more runs of continuous operation, and was the whole-path p99 measured separately?
• Does the p99 sum leave a margin of 10 % or more against the deadline, and is the slowest pipeline stage shorter than the part interval?
• Is a timeout that outputs NG when no decision exists before the deadline implemented in the signal output stage?

Frequently Asked Questions

How many samples does a p99 need?

At 10,000 samples, an average of 100 fall beyond the p99 with a standard deviation of about 10, so the estimated p99 can be taken to sit between the true p98.8 and p99.2. At 1,000 samples only 10 fall beyond, and the same range widens to p98.4 through p99.6. That is why I do not quote a p99 from a few hundred benchmark runs and collect 10,000 or more from continuous operation at a stable temperature.

If the average is half the deadline, does the p99 not usually fit too?

In this example it fits, but the margin clears the baseline by only 0.9 ms. The gap between average and p99 is set by the shape of the tail, and with the same average, transfer contention or a rising temperature pushes the p99 much further out. Inference time that rises and falls with temperature is calculated separately in the thermal design article for fanless edge devices.

Glossary

• [1] WD working distance
  The distance from the front of the lens to the surface under inspection. The roughly 170 mm in the text is a thin-lens object distance from the lens principal plane; the distance from the lens front comes from the lens drawing. ↩
• [2] Strobe strobe
  A light that switches on strongly for only a brief moment. With a 30 μs flash, the image smears only by the distance the part moves in that time. ↩
• [3] Deadline deadline
  The latest time by which the decision signal must come out. It is the distance from the camera to the reject point divided by the part speed. ↩
• [4] Trigger trigger
  The electrical signal that tells the camera and the light when to capture. Here it is the starting point of the latency measurement. ↩
• [5] Timestamp timestamp
  The time at which each stage starts or ends, recorded on the same clock. Per-stage latency is the difference between neighboring timestamps. ↩
• [6] p50 · p99 percentile
  The values at the 50 % and 99 % points when measurements are sorted from smallest. A p99 of 53.1 ms means 1 in 100 runs is slower than that. ↩
• [7] Motion blur motion blur
  The smear along the direction of travel when an object moves during exposure. Its length is the speed multiplied by the exposure time. ↩
• [8] Scheduling scheduling
  The operating system deciding which task runs on the CPU and when. A low-priority task can be pushed back by others and run late. ↩
• [9] Pipeline pipeline
  Processing that starts on the next part before the previous one is finished, overlapping several parts in stages. ↩
• [10] Timeout timeout
  A mechanism that runs a preset action when no result exists by a set time. Here it outputs NG if no decision exists by 54 ms. ↩

References

• Formulas: deadline = distance from camera to reject point ÷ part speed, margin = deadline − per-stage p99 sum, margin ratio = margin ÷ deadline, motion blur = speed × exposure time, blur (px) = blur ÷ pixel resolution, pipeline processing interval = p99 of the slowest stage, samples beyond p99 = n × 0.01 (standard deviation √(n × 0.01 × 0.99)), magnification m = sensor width ÷ FOV width, object distance = f × (1 + 1/m)
• Assumptions: part speed 300 mm/s, 18 mm from camera to reject point, part interval 40 ms, 2,448 × 2,048 px camera (3.45 μm pixels), 25 mm lens, FOV 49 × 41 mm, 30 μs strobe, per-stage p50 and p99 as given in the text (including 0.07 ms trigger latency), 16 hours of operation per day

Related reading — The Vision Decision Is Done, So Why Does the PLC Learn It Late? · In Edge Inspection, Input Drift Shows Up Before Accuracy Drops · Where Do Fine Defects Go When You Shrink to 640 × 640?