Meta-RL & generalisation
Learning to learn and in-context RL.
What you will learn
An agent trained on one task starts every new task from scratch, and one trained on one level of a game often cannot play the next. Nothing so far separates learning a task from learning how to learn tasks.
Learning to learn: an RNN that runs an RL algorithm in its activations (RL²), meta-learned initialisations (MAML), task inference on top of SAC (PEARL), algorithm distillation into a transformer, and Procgen as the benchmark that measures generalisation rather than memorisation.
You can train an agent across a task distribution and measure adaptation on held-out tasks, and you can tell generalisation from memorisation on a benchmark.
Duan et al., 2016; Wang et al., 2016
RL² / Learning to RL
Encode a fast RL algorithm in an RNN's activations.
The direct ancestor of in-context RL.
Resources
- RL² (paper)
- Learning to Reinforcement Learn (paper)
- CS285 lecture 22: Transfer Learning and Meta-Learning (5 parts) (video, optional)
Finn, Abbeel & Levine, ICML 2017
Model-Agnostic Meta-Learning
Meta-learn an initialisation for fast adaptation. Applies to RL.
Gradient-based meta-RL.
Resources
- MAML (paper)
Rakelly et al., ICML 2019
PEARL
Off-policy meta-RL via probabilistic task-context inference on top of SAC.
Sample-efficient meta-RL.
Resources
- PEARL (paper)
Laskin et al., ICLR 2023
Algorithm Distillation
Distil an RL algorithm's learning history into a transformer that improves in context.
Connects meta-RL to LLM in-context learning.
Resources
- Algorithm Distillation (paper)
Cobbe et al., ICML 2020
Procgen benchmark
Procedurally generated environments for measuring generalisation.
Standard generalisation benchmark.
Resources
- Procgen benchmark (paper) (paper)
- Procgen (code, optional)