Value-based deep RL

Mnih et al., arXiv 2013; Nature 518, 2015

Playing Atari with Deep RL / Human-level control through deep RL

Q-learning + CNN + experience replay + target network, learning Atari from pixels. Non-negotiable.

The founding work of deep RL. Every value-based method descends from it.

Learning outcome: Implement DQN from scratch, including replay, target networks and the terminal/truncation handling, and get it learning on CartPole.

Deliverable: an agent, an ablation, a plot

Build DQN, then change one thing

Write DQN from scratch until CartPole is solved, then change exactly one component and measure the effect against seed noise. One component at a time is the whole discipline.

The codebase you will extend for the whole value-based line.

Learning outcome: Debug a value-based agent to a working state, and measure a single design change against the noise floor.

van Hasselt, Guez & Silver, AAAI 2016

Deep RL with Double Q-learning

Decouple action selection from evaluation to fix Q-value overestimation.

The same overestimation problem is attacked again by TD3's clipped double-Q, though by a different mechanism: TD3 keeps two independent critics and takes their minimum, rather than splitting selection from evaluation across the online and target networks.

Learning outcome: Write both the DQN and Double DQN targets and say which one a given implementation is using.

Schaul et al., ICLR 2016

Prioritized Experience Replay

Sample high-TD-error transitions more often.

A Rainbow component. The distributed variant is Ape-X.

Wang et al., ICML 2016

Dueling Network Architectures

Separate value and advantage streams. Skim.

A Rainbow component.

Fortunato et al., ICLR 2018

Noisy Networks for Exploration

Learnable parametric noise for exploration.

A Rainbow component.

Bellemare, Dabney & Munos, ICML 2017

A Distributional Perspective on RL

Learn the distribution of returns, not just the mean. This launched a subfield.

QR-DQN, IQN, Rainbow, R2D2, Agent57.

Dabney et al., AAAI 2018; ICML 2018

Distributional RL with Quantile Regression / Implicit Quantile Networks

Quantile-based distributional RL. Read C51 closely, skim these.

Distributional heads in later agents.

Hessel et al., AAAI 2018

Rainbow: Combining Improvements in Deep RL

Integrates six DQN extensions: double, dueling, PER, multi-step, C51, noisy nets. The best single ablation of what actually matters.

Template for later combine-everything agents.

Kapturowski et al., ICLR 2019

Recurrent Experience Replay in Distributed RL

LSTM plus distributed replay. Big Atari jump.

Backbone of NGU and Agent57.

Badia et al., ICLR 2020; ICML 2020

Never Give Up / Agent57

Episodic and lifelong intrinsic rewards plus a meta-controller over exploration policies. First agent above the human benchmark on all 57 Atari games.

Read for the exploration and meta-controller ideas.

Schwarzer et al., ICML 2023

Bigger, Better, Faster

IQM human-normalised 1.045 on Atari 100K in about 10 GPU hours. Scaled Impala-CNN, high replay ratio, periodic soft resets, annealed n-step returns, and SPR's self-predictive loss underneath. Read SPR (Representation learning) first if you have not.

The reference point for how good value-based RL can be with little data.

Deliverable: an ablation under a stated budget

Add the Rainbow components one at a time

Add Double DQN and two more components of your choice to your own agent, one at a time, on a fixed interaction budget, and report the result with uncertainty rather than one curve per variant.

The first experiment on this map that looks like a paper's ablation table.

Learning outcome: Run and report a multi-variant ablation whose conclusion survives the seed noise.