Representation learning
Auxiliary tasks and augmentation for pixel RL.
What you will learn
Pixel input makes every method above slower and less stable: the agent has to learn what to see before it can learn what to do, from a reward signal too weak for the job.
Auxiliary losses that give the encoder more signal, UNREAL's prediction and control tasks, CURL's contrastive loss and SPR's latent-dynamics prediction, and the finding that simple image augmentation (RAD, DrQ) gets most of the benefit on its own.
You can add an augmentation or an auxiliary loss to a pixel agent and measure what it buys, and you know which of them the sample-efficient Atari agents depend on.
Jaderberg et al., ICLR 2017
RL with Unsupervised Auxiliary Tasks
Auxiliary prediction and control tasks accelerate RL.
Auxiliary losses everywhere.
Resources
- UNREAL (paper)
Srinivas, Laskin & Abbeel, ICML 2020
CURL
Contrastive auxiliary loss for pixel RL.
The CURL vs RAD debate.
Resources
- CURL (paper)
Laskin et al., NeurIPS 2020; Yarats et al., ICLR 2021; arXiv 2021
RAD / DrQ / DrQ-v2
Simple image augmentation alone gives most of the benefit. DrQ-v2 first solves humanoid from pixels model-free. Read DrQ.
Augmentation, not the contrastive loss, drives gains.
Schwarzer et al., ICLR 2021
Self-Predictive Representations
Self-supervised latent-dynamics prediction for data-efficient Atari.
Feeds into BBF.
Resources
- SPR (paper)