Orientation
Courses and references to run in parallel with the reading. Do not read these end to end first.
What you will learn
You arrive with some maths, some deep learning and no shared vocabulary for RL. The reading ahead assumes conditional expectation, gradients through stochastic objectives, a training loop you can debug, and a rough idea of which course covers what.
Two diagnostics and a set of companions. Test yourself on the maths and deep-learning checks and refresh only what fails. Then meet the textbook, the lecture series and the reference code once, here, so that later stations can send you to a chapter, a lecture or a file without introducing it again.
You know which gaps you have and have closed the ones that matter. You can open Sutton & Barto, a Silver lecture or a CleanRL file and know where it sits in the whole.
Probability, expectation, gradients, optimisation
Mathematical readiness
Conditional expectation, distributions and sampling, gradients and the chain rule, stochastic optimisation, and enough statistics to read an error bar. Treat this as a diagnostic rather than a course: test yourself on each topic and refresh only what fails.
Bellman equations are conditional expectations and policy gradients are score-function estimators. Everything downstream assumes this.
Learning outcome: Write E[G_t | S_t = s] without hesitating, differentiate a log-likelihood, and say what a confidence interval does and does not claim.
Resources
Sutton & Barto, MIT Press 2018, corrected 2020
Reinforcement Learning: An Introduction, 2nd ed.
The canonical textbook, free online. Read chs. 1–2 here for orientation; chs. 3–13 are stations on the Foundations line. Part I (tabular) is chs. 1–8. Part II (approximation, policy gradient) is chs. 9–13. Part III (psychology, neuroscience, case studies incl. AlphaGo) is chs. 14–17.
Backbone of Phase 1.
Learning outcome: Set up the book and know which chapters map to which stations on this map.
Resources
- Ch. 1 Introduction (chapter)
- Ch. 2 Multi-armed Bandits (chapter)
- Reinforcement Learning: An Introduction (PDF) (book, optional)
- Book homepage (site, optional)
DeepMind × UCL, 2015, 10 lectures
David Silver's UCL RL course
Best lecture companion to Sutton & Barto Part I.
Essential for Phase 1.
Learning outcome: Have a lecture series to fall back on when a chapter does not land.
Resources
- Course page (course, optional)
- Silver lecture 1: Introduction to Reinforcement Learning (video)
Backprop, convnets, and a training loop you can debug
Deep-learning readiness
Automatic differentiation, optimisers and their schedules, convolutional and recurrent nets, normalisation, and a training loop you have written yourself. A diagnostic like the maths check rather than a course: test yourself and refresh only what fails.
Phase 2 asks you to put a convnet where the Q-table was and debug it when it does not learn. Nothing on this map teaches that part.
Learning outcome: Write a training loop from scratch, read a loss curve, and tell a broken implementation apart from a badly tuned one before any of it is wrapped in an agent.
Resources
UC Berkeley, Fall 2023 recordings, Spring 2026 offering
Sergey Levine's CS285: Deep RL
The best deep-RL lecture series. Maps almost one to one onto Phases 2–5.
Essential from Phase 2 on.
Resources
- Course site (course, optional)
- CS285 lecture 1: Introduction (3 parts) (video)
Josh Achiam, OpenAI, 2018
OpenAI Spinning Up in Deep RL
Docs plus clean implementations of VPG, PPO, DDPG, TD3, SAC, and the curated Key Papers list this curriculum overlaps with.
Essential implementation reference for Phase 3.
Learning outcome: Know the algorithm families and where each one sits before you meet the papers.
Resources
- Spinning Up docs (site, optional)
- Key Papers in Deep RL (site, optional)
- Spinning Up Part 2: Kinds of RL Algorithms (site)
- Spinning Up as a Deep RL Researcher (blog, optional)
Berkeley, Aug 2017
Pieter Abbeel's Deep RL Bootcamp
Archived lecture videos and slides. A condensed alternative to CS285.
Optional.
Resources
- Bootcamp lectures (video)
Winter 2026
Stanford CS234 (Emma Brunskill)
Stronger on theory and bandits than CS285.
Useful for Phase 8.
Resources
- CS234 site (course, optional)
- CS234 lecture 1: Introduction to Reinforcement Learning (video)
CleanRL, Stable-Baselines3, Dopamine, PureJaxRL, verl, TRL, OpenRLHF
Reference implementations to keep open
CleanRL (single-file PPO, DQN, C51, DDPG, TD3, SAC). Stable-Baselines3 (reliable OO PyTorch). Dopamine (value-based Atari). PureJaxRL (end-to-end JAX). For LLM RL: verl, TRL, OpenRLHF.
Use as references for every project, never as copies.
Learning outcome: Have a reference implementation open to diff against when your own agent misbehaves.