Sim2Real in Robotics

A companion to The Sim2Real Gap Is Coming for AI Agents. That post argues agents are walking into the problem robotics has fought for decades; this one is the robotics side of the story.

DeepRacer, in depth

The full story behind the DeepRacer section of The Sim2Real Gap Is Coming for AI Agents, drawn from our 2019 paper.

What Sim2Real looked like in 2017

It's easy to forget how immature the field was. Deep RL's greatest hits were Atari and MuJoCo benchmarks. Domain randomization had only just been formalized in research. Simulators were CPU-bound, so you ran a handful of environments, not thousands. The prevailing view in robotics was that sim-trained policies were a curiosity, and that anything serious required real-world data.

We didn't have the luxury of that view. We were shipping a product, so the gap had to close for people who would never read a paper about it.

How DeepRacer solved Sim2Real

Most of the work went into modeling reality, and designing that environment took months and a team of experts across robotics, simulation and RL. We built the tracks and surroundings in Blender, imported them into Gazebo, and simulated the car with the ODE physics engine, including collisions, friction and acceleration, from a URDF model matched to the measured chassis, wheels and Ackermann steering. We calibrated the simulated camera's height, angle and field of view against real images, locked the simulation to the real camera's 15 fps with one action per frame, and mapped each discrete action to motor commands by measuring real steering angles and speeds. We then built a physical replica track identical in color, shape and dimensions to one in simulation, and ran controlled experiments on it and on a duct-tape track in our office corridor: with and without barriers, in bright sunlight and at different speeds. Where we couldn't model reality, we synthesized variation instead, with up to 10% random noise on steering and throttle, reversed direction of travel, randomized starting points, and image augmentations like random color, translation, shadows and salt-and-pepper noise, spread across distributed rollout workers that each ran their own randomized world. The payoff: at 0.5 m/s, a policy trained for under five minutes transferred to the real track, and our best model ran consistent 11-second laps at 1.6 m/s from a single grayscale camera.

What the gap actually looked like

The failure modes were humbling, and they taught me more than the successes.

• Perception gap. Real tracks came with glare, sunlight and backgrounds full of clutter and people. We tested on the replica with and without barriers, in bright sunlight, and on a duct-tape track in our office corridor that crossed both carpet and concrete. The agent version: production logs, dashboards and UIs never look like the clean ones in your environment.
• Dynamics gap. Speed in sim was idealized. On the real car, battery level changed speed over a session, and holding a constant speed was hard as the policy switched throttle levels. Simulated laps took about 35 seconds; we tested on the real track at roughly double that speed. The agent version: real APIs rate-limit, time out and change behavior between versions.
• Reward gap. Our default reward paid the car to stay near the center line, because near the edge a small deviation loses sight of the track. Thousands of users wrote their own rewards, the verifiers of their day, and policies learned whatever the reward actually paid for, which wasn't always fast, clean driving. The agent version: reward hacking against unit tests and LLM judges.
• Overfitting to the sim. Policies both underfit and overfit the simulator during training. A checkpoint that looked great on a fixed simulated lap could fail on the real track, and the training curve told you little about which. The agent version: benchmark scores that climb while production performance doesn't.

What closed it

We trained policies with increasing levels of randomization and tested each on the real track. Every finding has a direct analog for agents.

• Robust evaluation predicts the real world. Evaluating checkpoints in sim with action noise, multiple starting points and both directions of travel told us which would transfer. A run of consecutive good checkpoints was the strongest signal. That let us decide when to stop training and tune hyperparameters without living on the real track. For agents: the eval that matters is the one that predicts production, and most agent evals today are never checked against it.
• Reversing direction every episode was the biggest single win. Up to 10% action noise didn't help on its own, but it did combined with reversing. For agents: randomize the obvious symmetries first; cheap structural perturbations, like reordered tools, renamed fields and shuffled steps, can beat elaborate simulation.
• Random color was the most effective image augmentation, ahead of translation, shadows, sharpening and salt-and-pepper noise. For agents: vary how the world presents itself, its formats, phrasing and layouts, not just what it does.
• Regularize the policy. Dropout of 0.3 and a lower entropy bonus (0.001 instead of 0.1) improved transfer. For agents: a policy that memorizes one environment's quirks won't survive the next one.
• Training faster made the policy sturdier. A higher maximum throttle in training dramatically improved robustness, at the cost of slower convergence. For agents: train against environments that are flakier, slower and more adversarial than production.
• Mixing tracks didn't help. Training on several tracks at once did not improve transfer. For agents: piling on environments is not the same as covering the gap.

The deeper lesson stuck with me. The gap is not a bug you fix once. It is the product. Evaluation in simulation is only worth something if it predicts reality, and building that predictor is most of the work. Whoever builds the best bridge across the gap wins.

A short history of Sim2Real in robotics

The main post leans on lessons from robotics. This is the longer story behind them, and the next section covers what robotics RL environments look like in practice.

Before 2017: make the sim match reality

The reality gap was named long before deep learning. In 1995, Jakobi, Husbands and Harvey showed in evolutionary robotics that controllers evolved in simulation transfer only if you inject the right noise, and that a simulator can be deliberately minimal as long as it's noisy in the right places. The pre-deep-learning toolkit looked like this:

• System identification and calibration. Measure the real robot and fit the simulator's parameters to it. This was, and still is, the first thing any serious team does.
• Learn the simulator from real data. Andrew Ng and Pieter Abbeel's autonomous helicopter work in the 2000s learned a dynamics model from real flight logs, trained policies inside that learned model, and flew the real aircraft, eventually through aerobatic maneuvers.
• Noise injection. Perturb sensors and actuators in sim so the policy can't lean on any one exact value.
• Model ensembles and robust control. Optimize against a family of plausible models rather than one. Ensemble-CIO (2015) planned full-body motions that transferred to a physical humanoid this way, and EPOpt (2016) brought the idea to neural network policies.
• Synthetic data for perception. ALVINN, the late-1980s neural network that steered a Carnegie Mellon van, was trained in part on simulated road images.
• Transfer and fine-tuning. Progressive networks (2016) reused features learned in sim and adapted them with a little real experience.

At the end of 2016, CAD2RL flew a real quadcopter indoors with a policy trained only on heavily randomized rendered images. It was a preview of what came next.

2017 onward: make reality one more sample of the sim, then scale it

Deep RL, cheap compute and randomization at scale changed the game. Instead of making the simulator exactly right, the field learned to make it wide enough that reality falls inside it.

Put the eras together and the modern recipe has six parts:

1. Randomize what you can't model. Friction, mass, latency, sensor noise, lighting. Make the real world just another draw from the training distribution.
2. Model what you can. Calibrate the twin, and learn the hard-to-simulate parts, like actuator dynamics or residual errors, from real data and drop them back into the sim. Real-to-sim is half of Sim2Real.
3. Cheat in sim, then distill. Train a teacher with privileged information only the simulator knows, then distill it into a student that sees only what the real robot sees.
4. Adapt online. Let the policy estimate the world it's in from its recent history, so system identification happens at runtime, not just in the lab.
5. Evaluate for transfer. Select policies with evaluations that predict the real world, as DeepRacer's robust evaluation did, and feed real rollouts back to tune the sim, as SimOpt did.
6. Scale the sim, not the robot. GPU-parallel simulation turned weeks of experience into hours of wall clock.

But notice where the wins are. Sim2Real for locomotion is comparatively mature, though robustness, generalization and whole-body control on humanoids are still active research. Dexterous manipulation is harder. Contact-rich interaction, deformable objects, friction at fingertips and tactile sensing remain hard to simulate faithfully, which is why much recent manipulation progress has come from teleoperated real-world data and vision-language-action models. Simulation isn't out of the race: in 2025, Lin et al. showed vision-based Sim2Real RL for bimanual humanoid manipulation, built on automated real-to-sim tuning. Still, the gap for manipulation is the open frontier, and The next section picks it up.

What a robotics RL environment needs to contain

They have the same ingredients as LLM RL environments: an environment, a task and a verifier. In robotics, the environment is a simulated robot in a scene. The task is a goal like "stack the cups" or "fold the towel." The verifier is a success predicate the simulator can check, ideally backed by real-world evaluation.

What a lab buys depends on what it already has:

The content business looks like the LLM environment business: breadth, quality and diversity of tasks, with verifiers you can trust. The platform business looks like infrastructure: fidelity, throughput and integration.

Either way, the hard part is the same one flagged above: the Sim2Real gap for dexterous manipulation. Anyone can sell a scene. Selling a scene where a policy that learns to grasp, insert or fold in sim actually does it on a real robot is a different problem entirely.

There's a bigger business question underneath this. Robotics data will likely become a massive market, and a more defensible one than web data, because producing it takes physical infrastructure: robots, operators, environments, sensing, teleoperation, safety procedures and failure recovery all have to come together. But owning a scarce input doesn't automatically mean capturing the economics of the industry that consumes it, so the winner could build a multibillion-dollar company without that company being a pure data provider.