Offline RL
Learning from fixed datasets and the distributional-shift problem.
What you will learn
Everything so far assumed you can act in the environment. With a fixed dataset the agent cannot correct itself, and a Q-function queried on actions the data never took returns optimistic nonsense: the distributional-shift problem.
The tutorial that frames the problem, then the responses in order: constrain the policy to the data (BCQ), penalise out-of-distribution values (CQL), never query them (IQL), add a behaviour-cloning term (TD3+BC), and recast the problem as sequence modelling (Decision Transformer). Offline-to-online closes the loop.
You can train an agent on a D4RL dataset, diagnose overestimation from unseen actions, and pick a conservatism mechanism suited to the data you have.
Levine et al., arXiv 2005.01643, 2020
Offline RL: Tutorial, Review, and Perspectives
The starting point for the whole track.
Frames BCQ, CQL, IQL.
Learning outcome: State the distributional-shift problem in one sentence and say why off-policy algorithms do not solve it by default.
Resources
Fujimoto et al., ICML 2019
Off-Policy Deep RL without Exploration
Names the distributional-shift / OOD-action problem.
CQL and IQL respond to it.
Resources
- BCQ (paper)
Kumar et al., NeurIPS 2020
Conservative Q-Learning
Penalise OOD-action Q-values to lower-bound the value.
Standard offline baseline.
Resources
- Conservative Q-Learning (paper)
- CS285 lecture 16: Offline RL 2 (4 parts) (video, optional)
Kostrikov et al., ICLR 2022
Offline RL with Implicit Q-Learning
Expectile regression avoids querying OOD actions entirely.
Offline-to-online fine-tuning.
Resources
- Implicit Q-Learning (paper)
Fujimoto & Gu, NeurIPS 2021
A Minimalist Approach to Offline RL
TD3 plus a behaviour-cloning term. Strong, trivially simple.
Sanity baseline for any offline paper.
Resources
- TD3+BC (paper)
Chen et al., NeurIPS 2021; Janner et al., NeurIPS 2021
Decision Transformer / Trajectory Transformer
Recast offline RL as return-conditioned sequence modelling. DT is Essential, TT is Deeper.
The conceptual bridge from RL to sequence models and LLMs.
Resources
- Decision Transformer (paper)
- Trajectory Transformer (paper)
ICML 2022; Fu et al. 2020
Online Decision Transformer, IQL fine-tuning, D4RL
Continue learning online from an offline start. D4RL is the benchmark.
Real-world and robotics deployment.
Resources
- Online Decision Transformer (paper)
- D4RL (paper) (paper)
- D4RL (code, optional)