What Should a Camera See When a Humanoid Grasps an Object?

EDGE AI

A grasping camera has to see where the object is, how its surface reflects and what the hand hides, and the numbers only add up when the wrist camera, not the head camera, takes the last hand's span.

Taking a glossy metal part that must be picked within 0.5 mm, I calculated the size of one pixel, the depth error and how far things move during the latency.

About 10 min read

Start with the object the robot hand will pick. If it is a glossy metal part, light bounces off in one direction, so the amount returning to the camera varies greatly from spot to spot; if it is a matte resin part, light scatters evenly and the frame comes out relatively gentle. With the same hand, this surface difference changes the distance and position the camera reads. The walk-through assumes a glossy metal part picked within a tolerance of 0.5 mm, that is 500 μm.

If the grasp point¹ is off, the fingers brush the edge of the part, and the part slips or drops to the floor. A dropped part gets scratched, and the robot loses a cycle trying again. Raising the recognition model's accuracy alone cannot stop all of these failures, because part of the cause sits before anything is captured in the image.

The camera has to see three things: the position and pose of the object, the reflection of its surface, and the occlusion that appears as the hand approaches. Add the latency from recognition to motion, and the calculation runs on four numbers.

Glossy metal leaves holes in the depth map

Among depth cameras, the stereo method² finds the same point with two lenses set apart like two eyes and calculates distance from it. It has to pair the same pattern across the two images, but a hotspot on glossy metal moves with the viewing position, so the two eyes end up looking at different points. Depth at that spot is left empty or filled with a wrong value, and a hole remains in the depth map. The structured light³ method, which projects its own pattern, runs into the same problem. If the projected pattern bounces off the glossy face in one direction and never returns to the camera, that spot cannot be read. Placing a polarizing filter⁴ in front of the lens to suppress reflections is one option, but on metal surfaces polarization may not hold as well as hoped, so the effect cannot be guaranteed before a sample test.

On the day the part is chosen, I would shoot its surface once with the depth camera and look first at where the holes form. If holes form near the grasp point, changing the camera position or the light angle comes before changing the model. Even where there are no holes, error remains in the depth values themselves, and distance sets its size.

A head camera misjudges depth by 1.8 mm at 1 m

A stereo camera's depth error grows with the square of distance. It is approximated as distance² × disparity⁵ error ÷ (focal length in pixels × baseline⁶). With 1,280 px across and a 60° horizontal field of view, the focal length in pixels is 1,108.5 px, and I assume a 50 mm baseline between the two lenses and a matching error of 0.1 pixel.

Then the depth error is 0.072 mm at 200 mm, 0.451 mm at 500 mm and 1.804 mm at 1,000 mm. A camera on the head looking at a part on a table 1 m away exceeds three times the 0.5 mm tolerance on depth error alone. The same camera on the wrist, looking from 200 mm, comes in at 0.072 mm, well inside the tolerance. How the square of distance makes this difference is calculated separately in the depth camera article.

So the roles are split. The head camera looks wide and roughly finds where the object is, and the wrist camera takes the final approach. By my standard, the camera that fixes the grasp point must sit at a distance where depth error falls below a fifth of the tolerance. On that condition, 200 mm qualifies and 500 mm does not.

Even a wrist camera runs short of pixels at 200 mm

Correct depth is not enough if the frame runs short of pixels to place the object. The same 1,280 px, 60° camera sees a field 230.9 mm wide at 200 mm, so one pixel is 180.4 μm. The 500 μm tolerance is 2.77 pixels, short of the three pixels I use as my standard.

Moving in to 120 mm narrows the field to 138.6 mm, making one pixel 108.3 μm and the tolerance 4.62 pixels. The farthest distance that keeps three pixels is 184.8 mm. The closer the camera gets, though, the more the fingers block the view, and the nearest distance at which the camera can focus has to be weighed too. If the wrist camera's minimum working distance, the WD⁷, exceeds 120 mm, this calculation does not apply, and that value is an item to confirm on the spec sheet.

Occlusion grows in the last few centimeters. Once the fingers start wrapping the object, the grasp point is hidden even inside the wrist camera's view, so carrying forward the pose estimate from just before it was hidden, or adding one more small camera on the palm side, comes under review. With position and depth settled, the number left is time.

In 78 ms a handed-over object moves 7.8 mm

Watching at 30 frames per second, the frame interval alone is 33.3 ms; add 25 ms of inference, 10 ms of decision and 10 ms of control, and recognition to motion takes 78.3 ms. Everything except the frame interval is an assumption for the calculation. The world does not stand still during that time.

The top bar of the figure is this sum. For a part lying on a table, the object does not move during those 78 ms, and the robot knows how its own arm moves, so it can compensate. The trouble is a target that moves by itself, like an object a person hands over. An object approaching at 0.1 m per second moves 7.83 mm in 78.3 ms, over fifteen times the 0.5 mm tolerance.

To stay within tolerance the latency would have to be 5 ms or less, and a single frame interval is already longer than that. So when receiving a moving object, cutting latency alone falls short, and the motion design has to include predicting the next position to send the hand ahead or having the object pause for a moment at the handover. The robot's motion control itself needs a separate review.

What goes to the server and what stays on the board

Running a heavier model on a server for higher accuracy adds one more delay, the communication. Assuming a 20 ms wireless round trip, 78.3 ms becomes 98.3 ms, and the distance a handed-over object moves grows to 9.83 mm. That is why work that must never exceed the latency budget⁸, such as balance control, stays on on-board inference, while work with time to spare, such as path planning, is shared with the server, the split in the lower half of the figure.

If the model kept on the board is lightened with INT8 quantization⁹, accuracy on small targets can quietly erode in the process. I would compare the pose estimation error measured on the server with the error measured on the board, on the same batch of images, before deciding the split. Where accuracy parts ways on the move to the board continues in the server and edge accuracy article. One table holds the three rows, from wrist camera distance to latency.

Item	Value	Condition and basis	In plain words
① Minimum defect size	Grasp-point tolerance 500 μm (assumed) = 2.77 px at 200 mm, 4.62 px at 120 mm	1,280 px across and 60° field of view assumed. Farthest distance keeping three pixels 184.8 mm	The wrist camera has to come within a hand's span to reach three pixels
② Optical setup · WD	Head camera finds, wrist camera confirms at WD 120 to 185 mm, polarizing filter considered for glossy metal	Depth error with a 50 mm baseline and 0.1-pixel matching error assumed: 1.804 mm at 1,000 mm, 0.072 mm at 200 mm	Find from afar, grasp up close
③ Algorithm parameters	Recognition to motion 78.3 ms (board), 98.3 ms (20 ms server round trip assumed), position prediction for moving objects	An object at 0.1 m per second moves 7.83 mm during the latency; staying within tolerance needs 5 ms or less	For a moving object, look ahead and send the hand there

Read together, the three rows reveal the order that sets grasping accuracy. The distance in ② sets both the pixel count and the depth error in ①, and the latency in ③ is a number that balloons only when the object moves. So in a cell that picks stationary parts, the wrist camera's distance and reflection countermeasures come first, and in a cell that hands objects between people and robots, position prediction comes first.

Calculating the camera distance and latency at the moment a robot grasps an object is the work of a robot vision engineer, and someone who reads optics and inference latency together brings the grasp failure rate down in numbers. For anyone pricing the cell, the wrist camera and the board's latency performance shape real work speed as much as the price of the robot hand. This article explains technology and is not investment advice. See our Disclaimer.

Field Note

If a wrist camera goes into this cell, I shoot the glossy metal part at two distances, 120 mm and 185 mm, and look first at whether holes in the depth map overlap the grasp point. If they do, I take one more shot with a polarizing filter and one with a changed light angle from the same position. Then I repeat a person handing over the part at a little under 0.1 m per second and count the position error at the moment the hand makes contact.

Field Checkpoints

Once the photos from both distances and the handover records are in, set the grasping cell's camera specs with the questions below.

• Did you first sort the target's surface material and reflectance (glossy metal, clear resin, matte resin)? (glossy faces leave holes in the depth map)
• Is the camera that fixes the grasp point at a distance where the tolerance spans three pixels or more? (within 184.8 mm at 1,280 px and 60°)
• Is the wrist camera's minimum WD shorter than that distance? (out of focus, the calculation means nothing)
• At that distance, is the depth error small enough against the tolerance? (0.072 mm at 200 mm, 1.804 mm at 1,000 mm)
• Did you measure the worst-case latency from recognition to motion? (78.3 ms on the board, 98.3 ms with the server round trip)
• If the task receives moving objects, does the design include position prediction or a pausing motion? (7.83 mm during the latency at 0.1 m per second)

Frequently Asked Questions

Which suits grasping better, stereo or structured light?

For smooth objects with little pattern, stereo matching is hard, so structured light can do better; under bright ambient light the structured light pattern gets washed out, and stereo can be the better choice. Glossy metal leaves holes with both methods, so the answer comes only from comparing samples once the target and lighting environment are fixed.

How can occlusion by the hand be reduced?

In the final approach, carry forward the pose estimate from just before it was hidden, or place an auxiliary camera on the wrist or palm side to secure a viewpoint free of occlusion. An auxiliary camera sits even closer to the target, so check its minimum working distance and focus range as well.

Glossary

• [1] Grasp point grasp point
  The spot where robot fingers touch an object and apply force. If it is off, the object slips. ↩
• [2] Stereo method stereo vision
  A method that shoots the same scene with two lenses and calculates distance from how far the two images are offset. It works the way two human eyes sense distance. ↩
• [3] Structured light structured light
  A method that projects a set pattern and reads surface height from how the pattern bends. ↩
• [4] Polarizing filter polarizing filter
  A filter that passes only light vibrating in one direction. Crossing the directions on the light side and the lens side can suppress reflections. ↩
• [5] Disparity disparity
  The number of pixels the same point is shifted sideways between two images. Nearer objects shift more. ↩
• [6] Baseline baseline
  The distance between the two lenses. A longer baseline gives more disparity at the same distance, measuring depth more precisely. ↩
• [7] WD working distance
  The distance from the front of the lens to the target. Each camera has a nearest distance at which it can focus. ↩
• [8] Latency budget latency budget
  The total time allowed from recognition to motion. Frame interval, inference, decision and control time all have to fit inside it. ↩
• [9] INT8 quantization INT8 quantization
  Converting a model's numbers to 8-bit integers so it runs lightly on a board. Small differences can merge into one value and erode accuracy. ↩

References

• Formulas: depth error Δz ≈ z² × Δd ÷ (f_px × B), focal length in pixels f_px = (horizontal pixels ÷ 2) ÷ tan(field of view ÷ 2), field width = 2 × distance × tan(field of view ÷ 2), distance moved during latency = speed × latency
• Related calculations on this site: depth camera error and the square of distance, the accuracy gap between server and edge

Related reading — A Depth Camera’s Error Grows with the Square of Distance · What Changes in Vision Inspection When Logistics Automation Moves Into the Front End · It Worked on the Server, So Why Does Accuracy Drop on the Edge?