Edge AI Inference Latency Is Set by the Camera Interface Before the NPU

EDGE AI

An NPU twice as fast trims decision latency by only 19 %, while fixing the link and turning on RDMA cuts it by 40 %.

For an inspection station where four cameras shoot solder joints at 60 fps, I calculated the decision latency split into wire, copy, crop and inference pieces.

About 12 min read

Suppose the NPU² in an edge AI¹ controller that inspects solder joints on an SMT line is replaced with one twice as fast. The surfaces are tin-based solder fillets and green solder mask. A solder fillet is a strongly glossy curved surface, so its highlights wander with the lighting angle, and the solder mask is a semi-gloss surface mixed with diffuse reflection. The defects to find are 30 μm bridges and fillet cracks, and four cameras split the same board between them. Under the assumptions below, the time to a decision falls only from 21.2 ms to 17.2 ms, a 19 % cut.

When the decision runs past the 16.7 ms frame period (60 fps), frames have to be skipped or the line has to slow down. Joints caught in a skipped frame pass with no inspection, and slowing down cuts output. Lowering resolution to shrink the data is another workaround, but the moment you do it, fewer pixels land on a 30 μm defect and the detection margin disappears with them.

In this situation, before changing chips, I cut the decision latency into pieces and write down the time of each. The camera is assumed to be 4,096 × 3,000 px (12 MP) with 3.45 μm pixels, 8-bit monochrome, at 60 fps. With a 40 × 29.3 mm field of view, one pixel covers 9.77 μm on the board, so a 30 μm defect spans 3.07 pixels. Making this magnification (0.353) with a 35 mm lens puts the WD at about 134 mm by thin-lens approximation.

Cutting decision latency into four pieces

One frame is 12.29 MB. Assuming an effective bandwidth of 1.15 GB per second on one 10GigE³ link, 10.7 ms pass on the wire alone before the last pixel of the frame reaches the host. In conventional GigE Vision⁴ reception, the driver copies the received packets once more into application memory, which takes 2.0 ms at an assumed copy speed of 6 GB per second. I assume 0.5 ms to cut only the regions of interest⁵ containing joints into 640 × 640 px tiles, and 8.0 ms to run five tiles through the NPU at 1.6 ms each. The sum is 21.2 ms, 4.5 ms over the 16.7 ms frame period.

Piece by piece, the wire takes 50 % of the total. A twice-as-fast NPU shrinks 8.0 ms to 4.0 ms for a total of 17.2 ms, still over 16.7 ms. Moving the link up to 25GigE cuts the wire time to 4.2 ms (assuming an effective 2.9 GB per second), a saving of 6.4 ms from that piece alone. In my judgment, the reason throughput stays flat after a chip swap usually shows up in this one table.

What RDMA removes and what it leaves

GigE Vision 3.0 targets this copy piece. In its May 13, 2026 announcement, A3 explained that the standard incorporates RDMA⁷ over RoCEv2⁶, letting a device such as a camera write directly into computer memory without involving the operating system. The streaming protocol is called GVRSP, and the technical committee approved it unanimously at the International Vision Standards Meeting held in Prague on April 13 to 17, 2026. LUCID’s technical brief describes RDMA as an optional transport added alongside the existing UDP transport, which stays in place.

Here the calculation splits. RDMA takes copying and packet handling off the CPU, but the 10.7 ms it takes 12.29 MB to cross a 10 Gbit/s wire stays exactly as it was. Turning on RDMA alone on the same 10GigE trims 21.2 ms to 19.2 ms, a 2.0 ms cut, and only raising the wire to 25GigE together with RDMA brings it down 40 % to 12.7 ms. The gain from RDMA shows up more in latency jitter than in average latency, and in an August 17, 2026 piece for Quality Magazine, Pleora’s Ed Goffin named lower latency variation as the most significant advantage of RDMA.

The upper path in the figure is the conventional structure, where packets from the camera pass the NIC, pile up in an operating system buffer, and reach the NPU only after the CPU moves them into application memory. The lower path is the structure where an RDMA-capable NIC writes straight into application memory. The CPU load figures in the drawing are test values published by STEMMER IMAGING (from 50 % to under 10 %, more than 2 GB per second, latency improved by about 80 %), and because the test conditions were not disclosed, my calculation starts from a copy-speed assumption in place of those figures.

The load four cameras put on the host

With four cameras, each sends 737.28 MB per second, and together 2.95 GB per second (23.6 Gbit/s) flows into the host. One camera uses 64 % of the effective bandwidth of a 10GigE link, so each camera needs a link of its own, and assuming 8,000 bytes per packet, the four together mean handling 368,640 packets per second. Counting the copy alone, moving 2.95 GB per second ties up 49 % of one core copying at 6 GB per second.

In an edge box, this load overlaps once more in memory bandwidth. If the CPU and NPU share the same memory, the copy creates 5.9 GB per second of memory round trips, reads and writes combined, and the bandwidth the NPU has for reading weights and tiles shrinks by that much. In such a design, RDMA saves more than CPU time. As the LUCID material points out, RDMA needs a RoCEv2-capable NIC and receive buffers pinned in physical memory⁸, so the NIC spec of the controller in use is the first thing to check.

Splitting capture from inference

Four cameras producing five tiles per frame make 1,200 tiles per second. At 1.6 ms per tile, one NPU handles 625 tiles per second, so two NPUs are needed. Throughput is set by the slowest piece and latency by the sum of all pieces, so the wire and the copy have to scale with the number of cameras, and inference with the number of tiles, each on its own. That is why I first sketch a split layout: a capture node with an RDMA NIC receives the images and cuts out the regions of interest, and an inference node receives only the tiles and hands them out to the NPUs.

This kind of split shows up on show floors too. In its May 25, 2026 COMPUTEX 2026 announcement, AAEON presented a PCB inspection demo built on a disaggregated architecture that separates the inspection, analysis and decision-making flows, naming the UP Xtreme PTL, the EATX-W890A workstation board with Intel Xeon 600 processors, and the BOXER-6650-PTH fanless controller with Intel Core Ultra Series 3 processors as its main controllers. The announcement does not say which controller handles which stage. Even in a split design, if the wire between camera and capture node is the ceiling, adding NPUs at the back end does not shorten latency.

Collecting the three sets of numbers for one solder-joint inspection into a single table gives the following. The lighting combines a coaxial light⁹ that brightens the top of the fillet and a low-angle ring light that reveals the fillet slope.

Inspection item	① Minimum defect size	② Optical setup · WD	③ Algorithm parameters · in plain words
Solder bridges and fillet cracks	30 μm = 3.07 pixels (9.77 μm/px on the object)	12 MP (4,096 × 3,000) with 3.45 μm pixels, 35 mm lens, magnification 0.353, field of view 40 × 29.3 mm, WD about 134 mm (thin-lens approximation), coaxial + low-angle ring light	Five 640 × 640 px region-of-interest tiles per frame, 1.6 ms per tile (assumed), decision budget 16.7 ms (60 fps)
Transfer from one camera	Frame of 12.29 MB	One 10GigE link (effective 1.15 GB per second assumed) 10.7 ms, 25GigE (2.9 GB per second assumed) 4.2 ms	GVRSP (RDMA) reception removes the 2.0 ms copy, receive buffers pinned in physical memory
Four cameras combined	2.95 GB per second, 1,200 tiles per second	One link per camera, RoCEv2-capable NIC	Separate capture node (receive and crop) and inference node (two NPUs)

Lay the three rows side by side and the structure appears: the wire sets the floor of decision latency, RDMA strips away the copy and jitter above it, and the NPU pays in proportion to the size of the regions of interest. The wire pays for the whole frame while the NPU pays only for the regions of interest, so the smaller those regions are, the sooner the interface side hits the ceiling.

When the NPU really is the bottleneck

The opposite case is just as clear. If the regions of interest cannot be chosen and the whole frame must be inferred, there are 35 tiles and the NPU piece alone becomes 56 ms; then the NPU is the ceiling no matter how wide the wire gets. With a single camera at a low frame rate whose bandwidth need stays at the 1GigE level, the cost of switching to an RDMA NIC can exceed the time it buys. On a strongly glossy curved surface such as a solder fillet, fixing the interface does not raise the detection rate of cracks inside highlights, so results on the lighting and decision side cannot be guaranteed before a sample test.

In an equipment budget, the NPU is the eye-catching number, so it tends to be the first thing slated for replacement. In the calculation above, though, changing the NIC, the link and the node layout cut latency more, and that difference can be known in advance from one piece-by-piece timing table. This article explains technology and is not investment advice. See our Disclaimer.

Field Note

If throughput stays flat after an NPU swap, I stamp one frame with timestamps at four points before touching the model: when the exposure ends, when the last packet arrives, when the image lands in application memory, and when the decision comes out. The differences between these four values separate the wire, copy and inference pieces, and I write down the maximum and the average over 1,000 frames to see the jitter as well. If the wire piece exceeds half the frame period, I look at link speed before the chip.

Field Checkpoints

Once four timestamps have confirmed the time of each piece, cross off the list below item by item.

• Is a WD of about 134 mm (35 mm lens, 40 mm field of view, thin-lens approximation) secured by the lens datasheet, with room inside it for the coaxial light and the low-angle ring light?
• Have you calculated the required bandwidth as frame size × frame rate × camera count and compared it with the effective bandwidth of the link? (737.28 MB per second per camera in this setup)
• Have you confirmed from the datasheets that the camera, the NIC and the acquisition software all support GigE Vision 3.0 GVRSP (RoCEv2)?
• Have you measured decision latency split into wire, copy, crop and inference pieces?
• Does region-of-interest tile count × frame rate × camera count fit within NPU throughput?
• Have you separately verified crack detection on strongly glossy solder fillets with real samples?

Frequently Asked Questions

Does turning on RDMA speed up a 1GigE camera too?

Wire time is set by link speed, so RDMA does not shorten it. The STEMMER IMAGING material states that RDMA typically requires RoCEv2-capable NICs at 10G, 25G or higher, and the GigE Vision 3.0 cameras introduced by LUCID and Basler are 10GigE models. In a 1GigE setup the copy load is small to begin with, so the time gained is small too.

Can an existing camera be moved to GigE Vision 3.0?

The camera firmware has to support GVRSP, so the manufacturer needs to confirm it. Even if the camera supports it, the three pieces fit only when the host NIC supports RoCEv2 and the acquisition software supports RDMA reception. The existing UDP transport remains, so cameras that cannot be moved over keep working the original way on the same network.

Glossary

• [1] Edge AI on-device inference
  Running inference on a controller beside the line instead of sending images to the cloud. Decision latency is short and less dependent on the network. ↩
• [2] NPU NPU
  An accelerator chip built for neural network computation. The text assumes 1.6 ms per 640 × 640 px tile. ↩
• [3] 10GigE 10GigE
  Ethernet carrying 10 gigabits (1.25 GB) per second. After headers and overhead, the text assumes an effective 1.15 GB per second. ↩
• [4] GigE Vision machine vision camera standard
  A standard for exchanging images and control with industrial cameras over Ethernet. It is managed by A3, and version 3.0 added RDMA transport. ↩
• [5] Region of interest ROI
  The part of an image cut out for inspection, short of the whole frame. NPU load scales with the area of these regions. ↩
• [6] RoCEv2 RoCE v2
  A protocol that enables RDMA over ordinary Ethernet and IP networks. It requires a NIC that supports it. ↩
• [7] RDMA RDMA
  A method in which a sending device writes data directly into the receiver’s memory without passing through the CPU and operating system. ↩
• [8] Pinned receive buffer page-locked memory
  A buffer locked in physical memory so the operating system cannot move it. The address must not change for a NIC to write into it directly. ↩
• [9] Coaxial light on-axis light
  Lighting that shines along the lens axis through a half mirror. It shows flat metal surfaces bright. ↩

References

• A3, A3 Officially Releases GigE Vision 3.0, Opening New Possibilities in Machine Vision (2026-05-13, RDMA over RoCEv2 and GVRSP, unanimous approval at the IVSM in Prague, April 13 to 17, 2026)
• Basler, GigE Vision 3.0 & RDMA (released May 2026, images sent directly to RAM through a RoCEv2-capable NIC, no CPU storing or copying of image data)
• LUCID Vision Labs, RDMA Cameras and GigE Vision 3.0 (RoCE v2 added as an optional transport, RDMA-capable NIC and pinned memory required)
• STEMMER IMAGING, RDMA Technology in High-Speed Machine Vision Applications (test values of CPU load from 50 % to under 10 %, over 2 GB per second and about 80 % lower latency, test conditions not disclosed)
• Ed Goffin (Pleora), Removing the CPU Bottleneck in High-Bandwidth Machine Vision, Quality Magazine (2026-08-17)
• AAEON, AAEON Places Physical AI Front and Center at COMPUTEX 2026 (2026-05-25, PCB inspection demo on a disaggregated architecture)
• Formulas: frame size = width × height × bytes, wire time = frame ÷ effective bandwidth, copy time = frame ÷ copy speed, decision latency = wire + copy + crop + tile count × inference per tile, NPUs needed = tiles × fps × cameras ÷ (1 ÷ inference per tile), WD ≈ f × (1 + 1/m)

Related reading — Camera Interface Bandwidth Determines Resolution · Where Do Fine Defects Go When You Shrink to 640 × 640? · Why an Average Latency Inside the Deadline Is Not Safe