What are sim2sim, sim2real and real2real?



Recently, I always hear about the terms sim2sim, sim2real and real2real. Will anyone explain the meaning/motivation of these terms (in DL/RL research community)?

What are the challenges in this research area?

Anything intuitive would be appreciated!


Posted 2019-10-14T09:27:08.417

Reputation: 31



The abbreviations sim2sim, sim2real and real2real refer to techniques that can be used to transfer knowledge from one environment (e.g. in simulation) to another one (e.g. in the real world).

  • sim2sim stands for simulation-to-simulation,
  • sim2real stands for simulation-to-real, and
  • real2real stands for real-to-real.

In sim2sim, knowledge acquired during one simulation is transferred to an agent (or robot) in another simulation. Similarly, in sim2real, knowledge acquired during the simulation is used in a real-world problem (or environment). Finally, in real2real, knowledge acquired in a real-world problem can be transferred to another agent in another real-world problem.

The main challenges are related to the differences that exist between one environment and the other (either in simulation or in the real-world). For example, in sim2real, the simulation is almost never a perfect model of the real-world environment, so an agent trained in a simulation will probably not behave optimally in the real-world environment, which is often a lot more complex than the simulated environment. However, it is often the case that a robot needs to be trained in simulation, given that a robot trained in a real-world environment is subject to crashes.


Posted 2019-10-14T09:27:08.417

Reputation: 19 783


I defined real2real based on the concepts of sim2sim and sim2real as used in Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks (2019) by Stephen James et al., given that I've not found any paper that uses the expression real2real.

– nbro – 2019-10-14T12:06:08.883