The Price of KLA’s Patent That Puts Detection and Classification in One Network

ALGORITHM

Tie detection and classification into one network and a frame full of candidates runs more than four times faster, but one retrain shakes both scores together.

I read KLA’s unified neural network patent US10607119B2 and carried it over to an inspection that shoots one PCB in 49 frames, calculating the speed and the size of the regression test.

About 10 min read

A PCB with copper pads exposed on a green solder mask holds two very different surfaces in one frame. The solder mask scatters light fairly evenly, while the copper pads throw it back in one direction like a mirror, so brightness differs widely even under the same light. On this frame the system has to handle detection¹, finding flaws 20 μm wide or larger, and classification², naming whether each flaw is a scratch, a foreign particle or copper left behind by etching.

Split the two jobs between two models and only the small crop cut out by the detection model reaches the classification model, so the surrounding context disappears and misclassifications appear where detection was right but the name is wrong. A wrong name sends a board that should go to repair into scrap, or lets a board that should be scrapped eat up time at the repair bench. Either way, the inspection screen only records that the flaw was found correctly.

KLA’s US patent US10607119B2 (Unified neural network for defect detection and classification) covers a structure that ties the two into one neural network³. I work out in numbers how much faster this structure is and what it gives up in return.

What the patent ties together is the front end that extracts features

According to the bibliographic data published on Google Patents, the priority and filing date is September 6, 2017, the grant date is March 31, 2020, the original assignee is KLA-Tencor and the current assignee is KLA Corp. Per the abstract, the front portion of the network, the backbone⁴, extracts features from images of the specimen, and the rear portion takes those features to detect defects and classify them as well. The description explains finding defect locations with a network that proposes candidate boxes and sorting their types with ROI pooling⁵, which gathers only the features inside each box. It also covers training with class labels assigned by the user, either fine-tuning from pre-trained weights or training from scratch.

The point is that features are extracted only once. In a separated design, the detection model extracts features from the whole frame, and the classification model extracts features again from the crop it receives. The unified design lifts the box regions out of the feature map it has already computed, so the same calculation is never done twice. At an airport, the separated design is passport control and customs each checking the same traveler from scratch, and the unified design is two counters sharing a record taken once.

One PCB is shot in 49 frames

To weigh speed, you first need to know how many times one board is shot. My rule is to cover a 20 μm flaw with at least three pixels, so one pixel must cover 6.67 μm or less on the board, and with a 2,448 × 2,048 px sensor with 3.45 μm pixels, one frame sees 16.3 × 13.7 mm. Making that 0.518 magnification with a 35 mm lens puts the WD⁶ from lens to board at about 103 mm (thin-lens approximation).

Take the board as 100 × 80 mm and overlap neighboring frames by 10 %, and you get 7 frames across and 7 down, 49 in all. Assume a bright-field⁷ frame that shows flaw contrast and a dark-field⁸ frame that lifts flaw shape are shot back to back at the same spot and fed in as two stacked channels; the network then extracts features 49 times per board. That count of 49 multiplies any per-frame time difference between structures into a per-board difference.

The more candidates, the bigger the payoff of unifying

Latency is compared with assumed values. Say the backbone takes 40 ms to extract one frame’s features, the separated classification model takes 4 ms to crop and re-examine one candidate, and the unified classification head takes 0.3 ms to sort one candidate from the shared features. With 5 candidates per frame, the separated design takes 60 ms and the unified one 41.5 ms, 31 % faster, and per board 2.94 seconds become 2.03 seconds. With 50 candidates the gap widens to 240 ms against 55 ms, more than four times, or 11.8 seconds against 2.7 seconds per board.

The numbers show that the payoff of unifying grows with the number of candidates. Candidates set the time. On a frame with only a few candidates, such as a clean board, the backbone’s 40 ms takes up nearly all the time, so changing the structure saves little. The more foreign particles a process throws up, or the more a sensitive setting floods the inspection with candidates, the more the unified structure earns its keep. Real latency varies widely with model size and hardware, so I do not trust this ratio until both configurations have been timed on the same equipment.

A single retrain moves both scores together

The price of the speed comes due at retraining. Since detection and classification share the same backbone, retraining to add a new defect type changes the backbone and moves detection performance too. In a process where a new type appears every quarter, each retrain meant to fix classification carries the risk of shaking detection recall⁹ as well. If a retrain raises naming precision¹⁰ but lets more flaws slip through, that retrain is a loss. The speed gets paid back there.

The figure is a concept chart that plots the two metrics separately for each retrain of the shared backbone. The point where classification precision falls below its target in the very retrain where recall rises is caught by a fixed regression test set¹¹. How large that set needs to be can also be calculated.

When 95 % recall is measured on n defects, the half-width of the 95 % confidence interval is 1.96 × √(0.95 × 0.05 ÷ n), which is ±6.0 points for 50 defects and ±2.5 points for 300. To tell a 5-point drop apart with confidence, the intervals of two retrains must not overlap, so my rule is to fix about 300 defects per critical type in the regression set. With a 50-defect set, 95 % and 90 % look like the same number.

Freezing the backbone and retraining only the classification head when a new type arrives keeps detection steady. If the new type does not separate well with the existing features, though, classification precision will not rise, so the choice is made by setting the two numbers from the regression set side by side. When the backbone was retrained too, my first check is whether detection recall fell by more than 2.5 points.

How often the defect list changes picks the structure

Putting detection and classification in one table shows what has to be managed separately. Even with the same camera and the same WD, detection aims at missing nothing and classification aims at naming nothing wrong, and speed is decided by the number of candidates.

Item	① Minimum defect size	② Optical setup · WD	③ Algorithm parameters · in plain words
Detection (finding the spot)	20 μm wide = 3 pixels (6.67 μm/px or finer)	Bright-field frame, 35 mm lens, WD about 103 mm, field of view 16.3 × 13.7 mm, 49 frames per board	Recall target 95 %. Checked at every retrain within ±2.5 points on a 300-defect regression set per type
Classification (naming)	The same candidate boxes	Dark-field frame added as a second channel, same lens and WD	Precision targets written per type. For a new type, compare a frozen-backbone retrain with a full retrain on the regression set
Speed	Candidate count is the variable	Capture time for alternating the two lights is calculated separately	41.5 ms vs 60 ms at 5 candidates, 55 ms vs 240 ms at 50 (assumed latencies). The payoff scales with the candidate count

The table lists separate targets for detection and classification because the two scores move separately even inside one model.

The separated design still wins in some cases. If new defect types appear every quarter, if detection performance has to be held as a separate item for customer approval, or if a line sees only a few candidates per frame, splitting the two models is easier to manage. The information loss of the separated design can be partly made up by passing the classification model the surrounding area and the detection score along with the defect crop. Conversely, on a line with a stable defect list, many candidates and a tight tact time, the speed of the unified design pays off. How large the performance gap between the two structures really is on frames mixed with copper pad reflections cannot be guaranteed before a sample test.

Choosing the model structure is the vision algorithm engineer’s job, but the numbers behind that choice, the candidate count, how often types are added and the size of the regression set, can be counted by quality and production staff as well. For the side investing in the equipment, whether the two scores are reported separately at every retrain says more about running costs than a label such as unified AI. This article explains technology and is not investment advice. See our Disclaimer.

Field Note

To evaluate this structure, I would deliberately run one retrain that adds a single new type and start by checking how many points detection recall moves on the 300-defect regression set. If it stays within 2.5 points, I keep the unified structure; if it goes beyond, I compare it with a frozen-backbone retrain. I count candidates frame by frame across the 49 frames of one board, and if the average is around 5, I do not expect much of a speed gain.

Field Checkpoints

Once one retrain and a candidate count have checked the two assumptions, the questions below settle the structure.

• Is a WD of about 103 mm (35 mm lens, 16.3 mm field of view) available with both bright-field and dark-field lights in place? (whether the two lights fit between the lens and the board)
• Have separate targets been written for detection recall and classification precision?
• Does the regression set hold about 300 defects for each critical type? (with 50, ±6 points hides a 5-point drop)
• Has the speed gain of the unified structure been calculated from the candidate count per frame?
• Has it been checked how often new defect types are added? (if every quarter, is a separated structure also on the table?)
• Have both scores been checked with samples on frames that mix shiny spots such as copper pads with solder mask?

Frequently Asked Questions

Does a unified model automatically improve detection too?

Not necessarily. The shared backbone is trained to meet the detection and classification goals together, so if classification carries more weight in the training data, the features can lean toward what helps classification. Until separated and unified models have each been built from the same training data and their recall measured separately on the regression set, neither should be called better.

Can a patented structure be used as is?

Whether something infringes a patent is a legal judgment that requires matching the claims against the actual implementation one by one, so it needs a separate review. As for the technical lineage, a structure in which the proposal network and the detection network share convolutional features was already used in Faster R-CNN, published in 2015, and the scope of a patent is set by the wording of its claims, not by its abstract.

Glossary

• [1] Detection detection
  The task of finding where defects are in a frame and marking them with boxes or regions. ↩
• [2] Classification classification
  The task of naming what type each found defect is. Whether a board is repaired or scrapped depends on this name. ↩
• [3] Neural network CNN
  A model that passes an image through many layers of computation to extract features and make decisions. Images are mostly handled with convolutional neural networks (CNNs). ↩
• [4] Backbone feature extractor
  The shared front section of a network that extracts features from the image. In a unified structure, detection and classification both use it. ↩
• [5] ROI pooling region of interest pooling
  A computation that lifts out only the inside of each candidate box from the feature map and pools it to a fixed size, so the backbone does not have to run again for every candidate. ↩
• [6] WD working distance
  The distance from the lens to the surface being inspected. The value in the text uses a thin-lens approximation measured from the lens center. ↩
• [7] Bright-field bright-field illumination
  Lighting from close to the lens axis, so flat surfaces look bright and flaws that scatter light look dark. ↩
• [8] Dark-field dark-field illumination
  Lighting from a low angle, so flat surfaces stay dark and only light-bending spots such as flaws and steps look bright. ↩
• [9] Recall recall
  The share of real defects the inspector finds. At 95 %, 5 of every 100 defects are missed. ↩
• [10] Precision precision
  The share of defects given a certain name by the inspector for which that name is correct. ↩
• [11] Regression test set regression test set
  A fixed bundle of test images rerun the same way every time the model changes, to catch the retrain in which performance got worse. ↩

References

• KLA Corp (original assignee KLA-Tencor Corp), US patent US10607119B2, Unified neural network for defect detection and classification (priority and filing September 6, 2017; granted March 31, 2020; published as US20190073568A1)
• S. Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, a structure in which the region proposal network and the detection network share convolutional features)
• Formulas: pixel resolution = minimum flaw width ÷ 3, field of view = pixel count × pixel resolution, WD ≈ f × (1 + 1/m), frames = ⌈board width ÷ (FOV × 0.9)⌉ × ⌈board height ÷ (FOV × 0.9)⌉, separated latency = backbone + 4 ms × candidates, unified latency = backbone + 0.3 ms × candidates, confidence half-width = 1.96 × √(p(1 − p) ÷ n)

Related reading — Overdetection or Missed Detection, Which One Costs More? · Why Does a Lightweight Model Lose the Finest Defects First? · Deep Learning vs. Rule-Based Algorithms, When to Use Which