Training a Humanoid Robot for Hard Work

Training a Humanoid Robot for Hard Work

Using AI-driven behaviors, Boston Dynamics' Atlas robot maneuvers heavy objects and coordinates its whole body to execute complex tasks with accuracy and reliability. This article is written by Alberto Rodriguez (Director of Robot Behavior for Atlas), Shane Rozen-Levy (Research Engineer), and Vinay Kamidi (Research Engineer).

This humanoid robot is unlike anything seen before. The robot rotates its torso 180 degrees, squats down to lift a mini-fridge, and carries it to a lounging engineer. The nuances that are less obvious: the robot's full use of its arms, legs, and torso to manage a lift that a person would struggle with — and ones that don't show up on camera at all: the speed of development and fidelity of the behavior.

Physical Intelligence for the Real World

In recent years, there has been a fundamental transition to behavior architectures fueled by demonstration data, with an emerging capacity for generalization. This is an essential component to deliver on the promise of humanoids — being adaptable, quick to learn, and easily retasked.

While dominant approaches produce capable behavior, they also carry limitations: they are overreliant on continued camera feedback to guide control loops; they interact with the world through very limited surfaces (mostly fingertips); and are almost exclusively focused on lightweight tasks.

Real work, especially the back-breaking kind, requires a broadening of what we mean by physical intelligence. When we carry objects, we use any surface of our body to shoulder loads, and we adapt to their shape, mass, and rigidity through haptic sensations.

You cannot lift a fridge just by looking at it and using your hands. You have to prepare for it, anticipate the weight, lean into it, and let your body conform to its shape. Humanoids should be able to carry boxes between their forearms and biceps, use their knees to lift heavy objects from the floor, and throw long objects on their shoulders.

Reinforcement Learning for Whole-Body Lifting

Atlas uses reinforcement learning (RL) to learn how to lift a fridge by practicing the move with an absurdly large number of variations of the fridge in simulation. The hardest part is not seeing the fridge or knowing how to lift it, but learning to adapt to whatever version Atlas will encounter in the real world.

This is a combined control and perception problem, where perception is done implicitly from body proprioception. The policy driving the behavior has learned to adapt to variations like:

• The location of the fridge
• Its mass
• The amount of grip on the ground and with the fridge
• The configuration where the fridge settles between torso, arms, and hands

That level of adaptation is one of the most fundamental building blocks of physical intelligence.

Hardware Designed for Hard Work

This generation of Atlas is in its own league, designed for the mobility and strength required for real work, with the simplicity and reliability required for mass scale. Key highlights:

• Only two types of actuators for the entire body — focusing on making more efficient and powerful actuators at scale, lowering cost. All are rotary actuators that are easy to represent in simulation.
• Repeated sub-assemblies — both legs and arms are identical. Shoulder-to-shoulder and pelvis-to-pelvis structures are also identical.
• Infinite rotation actuators — achieved by eliminating all cables across joints, removing the key driver of hardware failures.
• Symmetrical feet — Atlas is equally capable of moving forward and backward.
• Field replaceable units — arms, legs, hands, and head can be swapped out within minutes.

The Training Montage

One goal for Atlas as both a product and research platform is to be able to train and deploy new behaviors in as little as a day. The fridge demo wasn't quite that quick, but it was much faster than anticipated. Here's how the robot was trained:

1. Reference

To start training a new behavior, a reference trajectory is used — data that tells the policy what it should be doing. This can be a teleoperated demonstration, an animated trajectory, or a more abstract goal. For the fridge move, a simple animation was used, taking full advantage of Atlas' superhuman range of motion.

2. Reward

An objective is set for the robot to stay as close to the animation as possible. Rewards reinforce desired behavior — keeping the weight in Atlas' grippers in the same position and orientation — as well as pushing and pulling on the robot and fridge, so the policy learns to stay on task while experiencing disturbances.

3. Simulation

Atlas practiced the moves for millions of hours in simulations in parallel on GPUs. Through extensive experience in simulation, Atlas learned to adapt its behavior to the many variations of the fridge.

4. Real Robot

Once the simulation looked good, testing moved to hardware. Boston Dynamics' "build it, break it, fix it" philosophy continues in modern product-focused, AI-focused research. Simulation will only take you so far — testing on hardware is how things get better.

5. Iteration

Once real data about the policy's performance on the real robot is collected, the training can be adjusted to make improvements. This iterative loop between simulation and real-world testing is the core of modern robot learning.

Beyond Practical Tasks

Less pragmatic tasks also serve a purpose. Handstands and backflips are possible on a 90 kg (198 lb) robot because of excellent thermal management, meaning Atlas will be able to work in hot environments. These behaviors also train transferable skills — how to move with agility and balance, how to use a full range of motion in constrained conditions, how to recover from slips and falls.

---

Source: Boston Dynamics — Training a Humanoid Robot for Hard Work