Does Memory-Bank Anomaly Detection Need to Remember Every Normal Patch?

EDGE AI

Storing every normal patch takes 6.29 GB, and a 1 % coreset 62.9 MB. Search still costs 2.58 × 10¹² multiply-accumulates per image, so search time, more than memory, sets the ratio.

On an edge device imaging molded connector housings at 20 μm/px, I calculated input tiling, patch cell size, memory bank size, coreset ratio, search compute and the decision threshold in turn.

About 12 min read

In processes where defect samples are hard to collect, such as molded connector housings, memory bank anomaly detection trained on good parts alone (the PatchCore¹ family) is a frequent choice. It extracts and stores patch features of normal images from a pretrained backbone, then scores each patch of an inspection image by how far it lies from the nearest normal patch. Move this structure from a server, where it runs well, to an edge device, and two problems surface at once: memory and search latency. Shrink the memory bank without grounds and normal variation drops out of memory, raising overdetection; leave it whole and search latency exceeds the takt, pushing full inspection back to sampling.

The surface comes first. The PBT resin of connector housings is mostly matte and images with even brightness under a diffuse dome light, but around mold gate marks the gloss differs, changing brightness and texture under the same lighting. That gloss difference becomes the width of normal variation itself and adds to the kinds of patches the memory bank has to remember. So the sizes and ratios below are for matte areas, and the bank size needed for glossy gate areas or transparent resin cannot be guaranteed before a sample test.

Here are the assumptions. A 2,448 × 2,048 px camera with 3.45 μm pixels and a 25 mm lens captures a 49 mm × 41 mm FOV at 20 μm/px from about 170 mm at the lens principal plane (a WD² of 150 mm or more secured). The smallest defect is a dark speck 80 μm across, the backbone features are 384-dimensional FP16³ vectors taken from the stride 8 layer with 3 × 3 neighborhood aggregation, and there are 100 normal training images. The effective distance-computation performance of the edge device is 5 × 10¹² multiply-accumulates⁴ per second, and the line takt is 500 ms.

Whole-image downscaling leaves an 80 μm speck at 0.84 px

The detection floor of the memory bank method is set by the input resolution and the backbone feature stride before the amount of training data comes into it. The most decisive step is the preprocessing that shrinks the camera original to the backbone input size. Shrinking the whole 2,448 × 2,048 px image to 512 × 512 px gives 95.7 μm/px horizontally and 80.1 μm/px vertically, and an 80 μm speck becomes 0.84 px horizontally, less than one pixel. The downscaling even changes the aspect ratio.

So the image is cut into 512 px tiles at the original resolution. The 2,448 px width takes 5 tiles (28 px of overlap at each seam) and the 2,048 px height exactly 4, for 20 tiles per image, and the same speck comes back as 4 px. The 20 tiles are the price of resolution, and that number 20 multiplies straight into the memory and search compute that follow.

Specks fill half of a 160 μm patch cell

Anomaly scores are given per patch⁵ feature. With a stride of 8 in the intermediate backbone layer, one feature map cell summarizes 8 × 8 input px, and with 3 × 3 neighborhood aggregation the area actually reflected is wider still. The smaller the share of that area the defect occupies, the more the patch feature is pulled toward normal texture and the lower the anomaly score.

Multiply 20 μm/px by stride 8 and one patch cell is 160 μm. An 80 μm speck is 4 px, half the cell width; a 60 μm wide flow mark is 3 px; and a 200 μm short-shot gap is 10 px, 1.25 cells. By my standard, the memory bank size can be discussed only after tiling and FOV are set so that the minimum defect covers about half a cell width, and the 80 μm speck barely meets that standard. There is no margin to spare.

Storing every normal patch takes 6.29 GB

Memory bank⁶ size is normal images × tiles per image × patches per tile × feature dimension × data type size. Because it is a product, the moment tiles are added to protect resolution, memory grows in direct proportion. With 100 normal images, 20 tiles, 64 × 64 = 4,096 patches per tile, 384 dimensions and 2-byte FP16, it comes to 8,192,000 patches × 768 bytes ≈ 6.29 GB. That does not fit on an edge device.

Coreset subsampling⁷ is a greedy selection procedure that keeps only representative patches covering the feature space evenly. It adds, one by one, the patch farthest from those already selected, so unlike random sampling it tends to look after rarely appearing normal variation first. At 1 % the bank becomes 81,920 patches, about 62.9 MB, which fits in edge device memory, and at 0.1 % it drops to about 6.3 MB. The lower the ratio, the greater the risk that rare normal variation such as gloss changes around gate marks, parting lines or lot-to-lot color differences drops out of the bank.

Search time caps the coreset ratio

In edge deployment, the first limit reached is usually nearest neighbor search⁸ compute ahead of memory. If every patch of the inspection image computes its distance to the whole bank, compute is proportional to query patches × bank size × feature dimension, and this setup has 20 × 4,096 = 81,920 queries per image. Exhaustive search against a 1 % coreset bank takes 81,920 × 81,920 × 384 ≈ 2.58 × 10¹² multiply-accumulates.

Divided by the assumed effective performance of 5 × 10¹² multiply-accumulates per second, 1 % takes 515 ms and alone uses up the 500 ms takt. At 0.3 % it is 24,576 patches, 18.9 MB and 155 ms, and at 0.1 % it falls to 52 ms. So I set the upper bound of the coreset ratio by search time and the lower bound by overdetection on the normal validation set, and for this setup I would start at 0.3 % and check on a map where the overdetections fall.

Approximate nearest neighbor search⁹ cuts this compute sharply, but its approximation error mixes straight into the anomaly score. Because the anomaly score is the distance to the nearest normal patch, an approximate search that picks a slightly farther neighbor raises the scores of normal patches systematically, tilting toward overdetection. So the coreset ratio is lowered as far as validation allows, approximate search fills only the remaining gap, and the shift in the score distribution of the normal validation set before and after is compared.

Thresholds at the 99.9th percentile need 1,000 normal images

A common way to decide takes the largest patch score of each image as its anomaly score and sets the threshold at the 99.9th percentile of that score on a normal validation set. With 200 normal validation images, however, 0.2 images lie above the 99.9th percentile, so the threshold is effectively set by the single highest image. A threshold resting on one image wobbles easily with the next lot.

So to use the 99.9th percentile, I collect at least 1,000 normal validation images. Good parts are easy to collect, so this condition costs little, and if they are still hard to gather, the criterion switches to the 99th percentile plus a margin. After switching to approximate search or changing the coreset ratio, the threshold is estimated again from scratch.

Conditions for an 80 μm speck in one table

Defect size, optical setup and memory bank parameters are gathered in one table. Cell size, memory and compute are all calculated from the assumptions above.

Item	① Minimum defect size	② Optical setup · WD	③ Algorithm parameters
Speck · particle	Diameter 80 μm = 4 px (20 μm/px), half of a 160 μm patch cell	2,448 × 2,048 px, 3.45 μm pixels, 25 mm lens, FOV 49 × 41 mm, diffuse dome light, WD 150 mm or more (object distance about 170 mm from the principal plane)	20 tiles of 512 × 512 px (5 × 4), no downscaling; whole-image downscaling gives 95.7 μm/px and 0.84 px
Flow mark · scratch	Width 60 μm, length 400 μm = 3 × 20 px	Based on matte PBT; glossy gate areas checked separately	Stride 8 layer, 3 × 3 neighborhood, 384 dimensions, FP16
Short shot	Width 200 μm = 10 px, 1.25 cells	Measure WD including the dome light height	Original memory bank 8,192,000 patches ≈ 6.29 GB, 0.3 % coreset 24,576 patches ≈ 18.9 MB
Search · decision	81,920 query patches per image	Assumed effective performance 5 × 1012 multiply-accumulates per second	Search 155 ms (0.3 %) / 515 ms (1 %), k = 1 nearest distance, threshold = 99.9th percentile over 1,000 or more normal validation images

The real bottleneck in this table is search, not memory. The 20 tiles chosen to keep the 80 μm speck at half a cell multiply into both the query count and the original bank, so even a 1 % coreset leaves 2.58 × 10¹² multiply-accumulates per image and search alone exceeds the takt. 0.3 % holds search to 155 ms but remembers less of the rare normal variation, and that price has to be checked on an overdetection location map.

When another method beats a memory bank

If enough defect samples can be collected and the defect types are fixed, supervised segmentation or classification models win on both boundary precision and inference cost, and nearest neighbor search is not needed at all. Products with very large normal appearance variation, such as many color variants or lot-to-lot texture differences, make the bank grow sharply or overdetection rise, so splitting banks per product comes first. If the inspection area is only a few mm across, the original bank is small enough for exhaustive search without a coreset to fit the takt, and subsampling then becomes needless information loss.

Among molded parts, glossy surfaces or transparent resin let specular and diffuse reflection shake patch features strongly, changing the width of normal variation itself. So whether the bank size and coreset ratio above fit such surfaces cannot be guaranteed before a sample test, and when the mold changes, I collect the normal validation set again first.

Field Note

If the coreset ratio has to go lower, I first draw a map of which tiles and coordinates the overdetections cluster in, ahead of the overall overdetection rate. Memory and search time shrink in proportion to the ratio, as calculated, but if rarely appearing normal patches drop out of the bank, overdetection can appear only at those positions. Where an area with different gloss exists, such as around the mold gate, I consider an allocation that keeps a higher ratio for those tiles alone, and on other molds I confirm again with samples.

Field Checkpoints

Before putting a memory bank model on the edge, I fill in the numbers below. If any one is blank, the coreset ratio is a number without grounds.

• Is a WD of 150 mm or more secured by measurement, including the dome light height and the camera viewing window position? (object distance about 170 mm from the principal plane)
• Have you looked at the width of normal variation separately for matte areas and glossy gate areas?
• Did you calculate the patch cell size as μm/px × stride, and does the minimum defect cover about half a cell width? (20 μm/px × 8 = 160 μm)
• Do the original memory bank size (images × tiles × patches × dimensions × bytes) and the size after coreset fit within the device memory headroom?
• Does the search compute per image (queries × bank size × dimensions), divided by effective performance, fit within the takt?
• Was the threshold estimated on 1,000 or more normal validation images, and estimated again after changing the coreset or approximate search?

Frequently Asked Questions

If the whole bank fits in GPU memory, is a coreset unnecessary?

Even when it fits in memory, search compute grows with bank size. Keeping all 8,192,000 original patches makes the multiply-accumulates per image 100 times those of a 1 % coreset, 2.58 × 10¹⁴, which takes 52 seconds at the assumed performance. Search time sets the ratio, apart from memory headroom.

Can coresets be built separately for each tile position?

Where normal appearance differs clearly by position, such as around a gate with different gloss, splitting banks by position and giving them different ratios is an option. Each query is compared only with the bank of its own position, so search compute shrinks too. If part positions wobble, though, queries get compared with the wrong bank, so I measure the positioning error first and also put patches from neighboring positions within that error into each bank.

Glossary

• [1] PatchCore PatchCore
  A method that stores patch features of normal images in a memory bank and judges anomalies by the distance from an inspected patch to the nearest normal patch. ↩
• [2] WD working distance
  The distance from the front of the lens to the surface under inspection. The roughly 170 mm in the text is a thin-lens object distance from the lens principal plane; the distance from the lens front comes from the lens drawing. ↩
• [3] FP16 half-precision floating point
  A floating-point format that stores one number in 16 bits, or 2 bytes. It halves memory compared with 32 bits. ↩
• [4] Multiply-accumulate MAC
  A unit of computation that bundles one multiplication and one addition. It is used to count the compute of distance calculations. ↩
• [5] Patch patch
  The small area of the input image that one cell of a feature map represents. At stride 8 it covers 8 × 8 px plus neighborhood information. ↩
• [6] Memory bank memory bank
  The store of collected normal patch features. During inspection, the nearest normal patch is searched for within it. ↩
• [7] Coreset subsampling coreset subsampling
  A procedure that shrinks the bank by keeping only representative patches that cover the whole feature space evenly. It adds, one by one, the patch farthest from those already chosen. ↩
• [8] Nearest neighbor search nearest neighbor search
  The computation that finds the patch in the bank closest in distance to a query patch. ↩
• [9] Approximate nearest neighbor search approximate nearest neighbor search
  A method that finds neighbors quickly by narrowing candidates with an index or quantization in place of measuring the distance to every patch. It can miss the true nearest neighbor. ↩

References

• Roth, Pemula, Zepeda, Schölkopf, Brox, Gehler, Towards Total Recall in Industrial Anomaly Detection, arXiv:2106.08265 (2021). The paper that proposed PatchCore, which finds anomalies with a memory bank of normal patch features.
• Formulas: pixel resolution = FOV ÷ pixel count, patch cell size = pixel resolution × stride, memory bank size = images × tiles × patches per tile × dimensions × bytes, size after coreset = original × ratio, search compute = query patches × bank patches × dimensions, search time = compute ÷ effective performance, samples above a percentile = sample count × (1 − percentile), magnification m = sensor width ÷ FOV width, object distance = f × (1 + 1/m)
• Assumptions: 2,448 × 2,048 px camera (3.45 μm pixels), 25 mm lens, FOV 49 × 41 mm, minimum defect 80 μm, 20 tiles of 512 × 512 px, stride 8, 384 dimensions FP16, 100 normal training images, effective performance 5 × 10¹² multiply-accumulates per second, takt 500 ms, 1,000 normal validation images

Related reading — Can Five Defect Samples Stand Up a Few-Shot Anomaly Detection Line? · When the Light Dims by 5 %, How Many Good Pixels Turn Into Defects? · In Edge Inspection, Input Drift Shows Up Before Accuracy Drops