In a warehouse in San Leandro, California, the frontier of physical artificial intelligence is playing out like a game of Jenga. This facility belongs to Encord, a company specializing in data tooling for training AI models. Andrew Ceja, one of the company's "pilots"—their term for robotic trainers—carefully extracts wooden blocks from a precarious tower while wearing a headset equipped with a camera that tracks his gaze. This setup is fairly standard for collecting robot training data, but the headset also includes sensors that measure his brain activity as he dismantles the block structure. Encord is part of a growing wave of startups betting that the next major bottleneck for humanoid and warehouse robotics won't be model architecture, but rather the acute shortage of real-world physical training data. Instead of merely helping robotics companies manage existing data, Encord is building a business around generating the data they lack.

The brain-wave headset Ceja wears was developed by Zander Labs, a German neuroscience startup that believes measuring brain activity—to infer mental states like error detection, intent, and surprise—can create richer datasets for training models. Encord's collaboration with Zander is currently in a trial phase; the company aims to build an initial brain-wave-tagged dataset, test it with customer robotics models, and assess whether it improves performance before deciding to scale up. Lucas Gehrke, a Zander neuroscientist overseeing the work, explains that the level of brain activity during various stages of a task provides clues for model builders about when to deploy their most computationally intensive models.

Vineeth Velmurugan, Encord's head of robot learning, describes this as the "bleeding edge" of efforts to overcome the robotics data bottleneck. A veteran of OpenAI's robotics lab and warehouse automation firm Berkshire Grey, Velmurugan joined Encord to establish its internal data-creation team. Encord was originally founded to help companies building machine-vision applications annotate data and evaluate models. But as their customers—Velmurugan notes they work with many leading robotics firms but cannot name them—began applying end-to-end learning to robotic manipulation tasks, executives realized they would need to produce training data themselves rather than just manage it. "The data simply does not exist," Velmurugan said.

The bet that generative AI can do for robots what it has done for chatbots keeps hitting this same obstacle. Large language models were built on the text of the entire internet and beyond. Finding comparable raw materials to teach neural networks about physical manipulation is far more challenging: self-driving car companies collect their own data, but that approach is hard to scale. Training from video can work, but it lacks the fidelity of real-world data. Velmurugan estimates that a dataset roughly five times the size of YouTube's video corpus would be needed to break through—a scale that explains why data generation itself has become a business, not just a research problem.

Companies building robot brains are now turning to two main sources: "egocentric" video captured by workers wearing cameras, often supplemented with additional camera angles and metrics, and data collected from remotely operated robots. Encord does both, sourcing egocentric data from several factories worldwide and using its San Leandro facility to experiment with new modalities, such as brain waves, or to collect datasets for specific skills needed in fine-tuning. "Every humanoid company has asked us for these pieces," Velmurugan says. The facility's storage racks hold cartons of fake flowers in vases, books, plastic vegetables, kitty litter trays and scoops, bags and bundles of wires—all stock in trade for training manipulators in household tasks. At one station, another pilot, Sofia Infante, maneuvers robotic arms to plug and unplug ethernet cables from the back of a server—the kind of work data center operators would love to automate, if only robots could achieve the required precision. After taking the controls myself, I saw why that remains out of reach: pincers are far less dexterous than human fingers and lack the degrees of freedom we take for granted in our own arms.

Another new data modality Encord is developing involves sensors strapped to the forearm to detect electrical signals in muscles. Video of human hands manipulating objects typically doesn't capture the entire hand, but Velmurugan hopes to build a 3D representation of hand position based on these arm sensors, giving models a more robust understanding. Encord's datasets are annotated with physical descriptions of each video—such as "right hand tightens bolt"—to help LLM-based models interpret what's happening. Velmurugan estimates that this dense annotation is worth 100 times as much as "junky ego data" for training specific tasks, and it costs only 20 times more to produce—a favorable trade-off on paper. But "20 times more" is still real money, and that's the catch: scraping text from the internet, as LLM makers did by pulling from Stack Overflow and other web sources, cost frontier labs almost nothing. Generating physical training data does not, and that's the limit of the physical-AI-as-LLM comparison. This kind of data must be manufactured, not just collected, fundamentally changing the economics of building these models.

Velmurugan says progress is being made—with Encord's visibility into programs across the industry, he can see both startups and frontier labs learning what works and what doesn't to improve physical AI models. That vantage point, sitting between many robotics companies at once, is also part of Encord's pitch. It can spot which data techniques are gaining traction industry-wide before any single customer can. This will keep the dozen or so pilots at Encord's facility busy. Both Infante and Ceja are part of a growing workforce developing the building blocks for neural networks; they previously worked at Scale, another AI data annotation firm, before joining Encord. Ceja had worked at a waste management company where his interest in technology led him to maintain a robotic trash sorter. Now, as the Jenga tower topples, he says he enjoys the challenge of solving training tasks for robots: "It's something new every day."