Model-based RL & planning

Deisenroth & Rasmussen, ICML 2011

PILCO

Gaussian-process dynamics with uncertainty for extreme data efficiency.

The uncertainty-aware model idea leads to PETS.

Chua et al., NeurIPS 2018

Deep RL in a Handful of Trials

Probabilistic ensemble dynamics plus CEM planning.

Ensembles lead to MBPO.

Ha & Schmidhuber, NeurIPS 2018

World Models

Learn a VAE+RNN latent world model and train a controller inside it. Highly readable.

Direct ancestor of PlaNet and Dreamer.

Learning outcome: Explain what a latent world model has to predict, and why training a controller inside it can work at all.

Janner et al., NeurIPS 2019

When to Trust Your Model

Short model-generated rollouts branched off real states, feeding SAC. Bounds compounding model error.

The standard modern Dyna-style method.

Hafner et al., ICML 2019; ICLR 2020; ICLR 2021

PlaNet / Dreamer / DreamerV2

Learn latent dynamics and train an actor-critic purely in imagination.

Leads to DreamerV3.

Deliverable: a model, a horizon plot, a comparison

Train a small latent world model

Learn latent dynamics on one control task, measure how far ahead the model stays trustworthy, and train a policy inside it against a model-free baseline at the same interaction budget.

The ideas you need before reading DreamerV3's engineering.

Learning outcome: Say where a learned model stops being useful, from your own measurements rather than a paper's.

Hafner et al., arXiv 2023; Nature 2025

Mastering Diverse Domains through World Models

One configuration across 150+ tasks. First to collect diamonds in Minecraft from scratch without human data. Must-read frontier model-based agent.

The reference for the one algorithm, many domains ambition.

Browne et al., IEEE TCIAIG 2012

Monte Carlo Tree Search

Selection, expansion, simulation, backup, and UCT's exploration term. Decision-time planning: search from the current state instead of learning a policy for all states.

The search half of AlphaZero and MuZero. Read this before either of them.

Learning outcome: Implement UCT for a small deterministic game and explain what the tree statistics converge to.

Silver et al., Nature 2016; Nature 2017; Science 2018

AlphaGo / AlphaGo Zero / AlphaZero

MCTS plus deep policy and value nets. Then self-play from scratch. Then one algorithm for Go, chess and shogi. Read AlphaZero closely.

Directly into MuZero.

Schrittwieser et al., Nature 588, 2020

MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

MCTS over a learned latent model predicting reward, value and policy. No given rules. Non-negotiable landmark.

EfficientZero. Conceptual cousin of RL search for LLM reasoning.

Ye et al., NeurIPS 2021

EfficientZero

MuZero plus self-supervised consistency. Human-level Atari 100K.

Sample-efficient planning.

Deliverable: a player, a strength curve

UCT for a small game, then a minimal AlphaZero

Implement UCT for Connect Four and measure playing strength against simulation count. Then, if you want the full picture, replace the rollouts with a small trained network.

Makes AlphaZero and MuZero concrete instead of impressive.

Learning outcome: Explain exactly what the learned network replaces in plain MCTS, having built both.

Hansen et al., ICML 2022; ICLR 2024

TD-MPC / TD-MPC2: Scalable, Robust World Models for Continuous Control

Short-horizon planning in a task-oriented latent space plus a terminal value function. One recipe across many continuous-control tasks.

Current SOTA-competitive continuous-control planner.