Behind every robot, there is a human operator

Behind the rapid development of robotics technology and the investment boom, a key fact has actually been overlooked: many seemingly autonomous robots actually require human remote control to operate reliably.

In March of this year, NVIDIA CEO Jensen Huang showcased a robot inspired by "Star Wars" at a flagship customer conference. The robot could follow commands to move to designated locations, earning enthusiastic applause from the audience. However, a blog post revealed that this robot was actually controlled by "puppeteers" behind the scenes, with human operators deciding the timing and location of the robot's movements during the demonstration.

This human-machine combined operating model means that many robots may not yet have the capability to operate independently and reliably in complex environments, and remote operators are currently a key part of ensuring smooth robot operation.

However, this human-machine collaboration is not a stopgap measure but a necessary path to higher levels of automation. Robotics companies are leveraging the vast amounts of data generated by human operators during remote control to train AI models, with the aim of ultimately reducing reliance on humans.

The "Puppet" Behind Robot Demonstrations

The eye-catching product demonstrations by high-tech companies are often key moments in shaping public perception and investor confidence, and in the robotics industry, human remote control is becoming an "open secret."

At Tesla's robotaxi event held in Los Angeles last October, the humanoid robot Optimus served drinks, danced, and interacted with guests. Elon Musk claimed that these robots could "teach, take care of children, walk dogs, mow lawns, and shop for groceries" in the future. However, when guests asked whether the robot operated autonomously, the human employee controlling the robot responded through it:

"I am not fully autonomous; I am currently assisted by humans."

Robot experts point out that a convention has formed in the industry: if the robots in demonstration videos are indeed operating autonomously, companies usually clearly label them as "autonomous." For example, when the startup Skild AI showcased its AI software controlling a humanoid robot earlier this month, it included text throughout the video indicating that the robot's actions in overcoming obstacles were performed autonomously.

This suggests that demonstrations without such labels likely involve human operator participation.

A Global Network of Remote Control

Human intervention initially seemed like a stopgap measure due to immature technology, but it has now evolved into a core means for robotics companies to obtain high-quality training data.

The primary function of remote operation is to handle "difficult problems" that robots cannot cope with. The startup Avride, which provides delivery robots for Uber Eats, stated that the biggest challenge its robots face is high curbs.

Roman Nefedov, head of the company's delivery robots, said that when a robot encounters such obstacles, it realizes it is stuck and can "request help." At this point, a remote operator connects to the robot's camera, observes the situation on-site, and then manually controls the robot to navigate around the obstacle. These problematic locations are subsequently marked on a map, and the relevant video information is used to train AI models, with the hope that in the future, the robots can handle similar scenarios autonomously Similarly, Waymo, a subsidiary of Alphabet, can operate its vehicles without a safety operator inside after completing mapping in new cities. However, if the vehicle encounters special road conditions that it cannot recognize, such as an unusually shaped construction site, personnel from the remote support team can provide guidance.

Today, startups are increasingly candid about viewing remote operation as a strategy.

Zach Rash, CEO of delivery robot company Coco Robotics, stated that remote operation was once "a bit of a taboo" for robotics companies, as outsiders would think "you are pretending to be autonomous." However, he pointed out that operator intervention is not just a backup plan; it is itself training for the robot. For example, the action data from an operator causing a drink to spill can be used to train AI, thereby optimizing the robot's movement.

The Long Road to Full Autonomy

Although human operators are indispensable at this stage, the long-term goal of the robotics industry remains to achieve a higher degree of autonomy, ultimately allowing one operator to supervise multiple devices.

Jon Miller Schwartz, CEO of Ultra Robotics, believes:

"To have robots reliably operate year-round, seven days a week, we will need personnel for supervision."

Even the technologically advanced Waymo retains the element of remote human intervention, with some companies even starting from 100% remote control.

The seven-year-old car rental startup Vay does not have customers pick up cars; instead, remote "drivers" bring the cars to the designated locations of customers. Vay's employees in Las Vegas sit in a cockpit made up of multiple computer screens, controlling cars on the other side of the city through a desktop steering wheel.

Gaurav Sethia, founding machine learning engineer at Deft Robotics, stated that even if robots can operate autonomously 80% of the time, that is still not reliable enough for many industrial scenarios. Remote operators can precisely fill in the remaining 20%.