Humanoid robots are good at making everyone believe they’re ready. They show up during news reports, on trade show stages and in viral videos — navigating factory floors, assisting surgeons, playing sports, climbing stairs and more. The hardware is impressive, and the movement appears effortless.
But a few carefully choreographed minutes don’t reveal much about what it takes to make a robot work in the world.
A demo succeeds under controlled conditions. Reality is dynamic and unpredictable. Performance comes down to data. Every capability depends on the scenarios the machine has experienced, the quality of the labels behind them, the validation that catches mistakes before deployment and the feedback that continues improving performance over time.
In this paper, we explore what it takes to get from perceived readiness and sustained deployment. Bridging the gap is where projects succeed or fail.
Physical AI movement starts with ground truth
Before a machine can act in the world, it has to learn what the world looks like. That education comes from ground truth data: labeled, verified datasets that teach a model what it’s seeing and how to react.
In most AI applications, minor data flaws only cause performance dips. A misclassified image, for instance, makes a content recommendation less accurate. The error is lower-stakes.
In robotics, mistakes have physical impact. Even a slight labeling inaccuracy causes a model to misidentify object boundaries, leading an autonomous vehicle (AV), for example, to misjudge clearance or a delivery robot to plow into an obstacle.
The complexity compounds with sensor modality. Machines read the world through LiDAR point clouds, RGB cameras, radar, depth sensors and sometimes tactile inputs — simultaneously. Ground truth must be accurate and consistent across every one of those modalities.
If a 3D bounding box around a pedestrian in a LiDAR cloud doesn’t project perfectly onto a 2D camera frame, the model learns a conflicting version of reality. That skewed logic carries on to every decision the system makes.
Coverage matters as much as accuracy. A robot trained only on clean, predictable scenarios will eventually freeze or fail when it encounters a foreign situation. High-fidelity ground truth captures these edge cases beforehand.
Egocentric data give humanoids a first-person view
Building that ground truth starts with collecting data for model training. For most of AI history, training data came from third-person perspective or through cameras watching someone perform tasks from across the room. That works fine for basic computer vision, not for training a robot to mimic tasks.
Egocentric data provides a first-person point of view. With a camera mounted at eye or chest level, it captures what the person sees and how they move.
The resulting footage teaches the model precise spatial reference: how a hand approaches an object before grasping it, how views change in narrow corridors, how the body balances to reach a high shelf — just to name a few scenarios.
For humanoid robots learning through imitation, egocentric data demonstrates both what the task is as well as the physical mechanics of executing it.
Annotation turns data into learning
Structuring raw footage into something a model can learn from is the next step, however, for robotics, the process looks different.
There’s a misperception in the AI industry that labeling one kind of data is roughly the same as labeling another. For robotics, this view is wrong and risky.
The annotation types involved in robotics training are technically distinct and significantly more demanding than standard image labeling:
3D bounding box annotation maps objects in 3D LiDAR point clouds rather than 2D photos, capturing an object’s precise size, location and orientation so machines accurately perceive physical space.
Semantic segmentation labels every pixel by class — marking roads, vehicles, pedestrians, vegetation and buildings. Precise edge boundaries ensure models accurately perceive and interact with their surroundings.
Sensor fusion annotation reconciles labels across LiDAR, camera and radar. Skilled annotators resolve modality mismatches caused by varying viewpoints or weather, ensuring machines maintain an accurate, unified environment model.
Pose estimation and keypoint annotation marks specific body joints, limbs and grip points to teach granular movement, enabling robots to balance precisely, grip securely and perform dynamic tasks.
The technical difficulty means that the skill requirement for robotics annotation is meaningfully higher than for general-purpose labeling. Annotators must understand the physical environment machines will operate in, including its geometry, physics and spatial constraints.
Data quality keeps learning on track
Ensuring data quality at scale is a silent challenge. There is no alarm when data degrades, a label is imprecise or if tagging doesn’t align. Instead, errors slowly pile up until a robot misbehaves in production. But the patterns behind these failures are predictable.
Label noise accumulates over time, gradually teaching models to confuse similar objects. As common scenarios dominate the dataset, rare but critical edge cases become underrepresented, leaving machines unprepared for unfamiliar situations. Meanwhile, annotator drift emerges slowly as individual interpretations diverge from established guidelines, introducing inconsistencies that spread across the dataset.
Maintaining quality at scale requires continuous measurement. Inter-annotator agreement, consensus scores and ongoing validation provide early signals when quality begins to drift, allowing teams to correct inconsistencies before they influence model behavior.
Red teaming strengthens reliability
A model that performs well under normal conditions but fails in unpredictable environments is not safe. Red teaming maps these blind spots. The term derives from military practice: a dedicated team tasked with attacking a system the way an adversary would.
In physical AI, this involves designing scenarios specifically intended to break the model: lighting conditions it wasn’t trained on, object configurations it hasn’t encountered, sensor combinations that create ambiguity and edge cases that stress the system in ways the training distribution never did.
The operational layer keeps robots going
Intent distinguishes red teaming from standard validation, which only confirms that the model handles what it was trained for. Red teaming searches for what’s missing. The output isn’t a score, but a map showing which scenarios the model handles reliably, which it navigates poorly and where it fails in ways that are unacceptable in a live environment.
This map tells the team what the robot isn’t ready for, providing the blueprint for the targeted training data needed to fix before deployment. For systems operating in safety-critical environments like surgical robots, AVs or in collaboration with humans, this process makes deployment responsible.
Any deployed fleet needs to answer a simple question around what happens when the machine gets stuck — not catastrophically, but when encountering a scenario outside its operational domain. Situations might include an unexpected object in an unusual position, a sensor reading that doesn’t resolve cleanly or a simply a time when the model’s confidence is low and the cost of a wrong decision is high. These moments are surprisingly frequent because the real world is always vaster and stranger than any training dataset.
Remote operations solve this by keeping humans in the loop. This infrastructure connects remote operators to the fleet, defining clear thresholds for when a robot must request help, equipping people to see what the machine sees to resolve the issue and systematically logging every intervention.
The scenarios requiring human judgment are precisely the edge cases the model wasn’t ready for. In turn, routing these interventions back into the annotation pipeline — turning them into labeled training data to reduce the frequency of future overrides — enables a fleet to become progressively more autonomous.
The goal is to leverage human involvement intelligently, ensuring each intervention makes the next one less necessary.
Scaling the data operation
Every capable robot begins with data that teaches it how to perceive the world. That foundation is refined through annotation, validated through quality assurance, challenged through red teaming and stabilized by remote operators who handle the edge cases when a machine gets stuck.
As fleets grow, so does the complexity behind them. Expanding into new environments demands broader geographic coverage, tighter quality controls and even more human-in-the-loop oversight. Reliable deployment means keeping every stage of the cycle operating consistently across hundreds or thousands of machines.
The teams that successfully scale understand this invisible engineering. The robot is the outcome, but the data is the work.
