Abstract
This paper proposes a simple strategy for sim-to-real in Deep-Reinforcement Learning (DRL) – called Roll-Drop – that uses dropout during simulation to account for observation noise during deployment without explicitly modelling its distribution for each state. DRL is a promising approach to control robots for highly dynamic and feedback-based manoeuvres, and accurate simulators are crucial to providing cheap and abundant data to learn the desired behaviour. Nevertheless, the simulated data are noiseless and generally show a distributional shift that challenges the deployment on real machines where sensor readings are affected by noise. The standard solution is modelling the latter and injecting it during training; while this requires a thorough system identification, Roll-Drop enhances the robustness to sensor noise by tuning only a single parameter. We demonstrate an 80% success rate when up to 25% noise is injected in the observations, with twice higher robustness than the baselines. We deploy the controller trained in simulation on a Unitree A1 platform and assess this improved robustness on the physical system.
Useful links
Publication record: https://proceedings.mlr.press/v211/campanaro23a.html
Preprint: https://arxiv.org/abs/2304.13150
Bibtex
@inproceedings{rolldrop2023,
author={Luigi Campanaro and Daniele De Martini and Siddhant Gangapurwala and Wolfgang Merkt and Ioannis Havoutis},
title={Roll-Drop: accounting for observation noise with a single parameter},
year={2023},
booktitle={Proceedings of The 5th Annual Learning for Dynamics and Control Conference},
volume={211},
pages={718--730},
publisher={PMLR},
url={https://proceedings.mlr.press/v211/campanaro23a.html},
}