Your guide to the map

Reinforcement learning curriculum

A reading and implementation roadmap through reinforcement learning, organised like the London Underground. Start with the foundations, build working agents, then choose the research directions that interest you.

19 topic lines connect 151 stations of papers, courses and exercises. Each station explains the idea, why it matters and what to read or build.

How to use the roadmap

Start with Orientation to check your mathematics, programming and machine learning prerequisites. Work through the core stations and spine exercises in order, with Experimental practice alongside them. Choose specialisations after the research sampler; reference stations are there when a project needs them.

Read a topic below, or open the interactive map to track resources, implementation work and your next station. Your progress is stored in your browser and can be exported from the map menu.

Recommended learning sequence

Readiness & orientation
Mathematical checks and the environment interface.
Ready to move on: A small experiment you can interpret.
- Orientation
Tabular RL
Bandits, MDPs, dynamic programming, Monte Carlo, TD, control, Dyna.
Ready to move on: Correct implementations, and an explanation of what separates them.
- Foundations
Approximation & deep value learning
Function approximation, the deadly triad, DQN, Double DQN.
Ready to move on: A debugged agent and a controlled ablation.
- Value-based deep RL
Policy optimisation
REINFORCE, actor-critic, GAE, PPO, continuous control, SAC.
Ready to move on: Reproducible experiments under a protocol you wrote first.
- Policy gradients & actor-critic
Research sampler
Short introductions to exploration, models, offline RL and imitation.
Ready to move on: A reasoned choice of one direction.
Specialisation
One selected track, with the theory that supports it.
Ready to move on: Reproduction of a clearly scoped published result.
Independent investigation
Hypothesis, competing explanations, experiment, analysis.
Ready to move on: A report with evidence, limitations and the negative results left in.
- Independent investigation

Explore all topics

Orientation

Courses and references to run in parallel with the reading. Do not read these end to end first.

9 stations · Phase 0

Foundations

Tabular RL end to end, from bandits to policy gradients, with an exercise after each idea. Read a station, then do the exercise that follows it before moving on.

17 stations · Phase 1

Experimental practice

How to run an experiment that means something, starting with tabular ones. Read this alongside Phases 1 and 2 rather than after them.

5 stations · Phase 1B

Evaluation & experiment design

How results are established and reported. Read it before your first deep agent, on the tabular runs you already have, and keep coming back: it decides whether your own experiments are worth anything.

5 stations · Phase 7

Value-based deep RL

Value-based deep RL, from DQN to the sample-efficient modern agents. Everyone needs DQN and Double DQN; the rest of the family is a specialisation.

13 stations · Phase 2

Policy gradients & actor-critic

Policy optimisation from REINFORCE to PPO and SAC, plus the two entropy quantities the trust-region methods are built on.

16 stations · Phase 3

Exploration

Intrinsic motivation and hard-exploration Atari.

6 stations · Track 6A

Model-based RL & planning

Learned world models, background versus decision-time planning, and the AlphaGo to MuZero lineage. World Models and MCTS are for everyone; the rest is a specialisation.

13 stations · Phase 4

Offline RL

Learning from fixed datasets and the distributional-shift problem.

7 stations · Track 6B

Imitation & inverse RL

Learning from demonstrations and recovering rewards.

5 stations · Track 6C

Hierarchical & goal-conditioned

Temporal abstraction and goals as inputs.

6 stations · Track 6D

Multi-agent & games

Many learners, credit assignment, and self-play at scale.

6 stations · Track 6E

Meta-RL & generalisation

Learning to learn and in-context RL.

5 stations · Track 6F

Representation learning

Auxiliary tasks and augmentation for pixel RL.

4 stations · Track 6G

Robotics & sim-to-real

If the thesis is embodied.

3 stations · Track 6H

RL for LLMs & reasoning

Preference-based RL, RLHF, direct alignment and verifiable-reward reasoning RL. Christiano, InstructGPT, DPO, GRPO and RLVR are core; the rest is a specialisation, not a prerequisite for other tracks. The language-model half — tokenised sequences, pretraining, supervised fine-tuning — is assumed here rather than taught, so bring it with you.

18 stations · Phase 5