Policy gradients & actor-critic

Williams, Machine Learning 1992

Simple statistical gradient-following algorithms

The original Monte-Carlo policy gradient.

GRPO and RLHF are REINFORCE-style estimators on token sequences.

Learning outcome: Derive the score-function estimator and explain why a state-dependent baseline leaves it unbiased.

Sutton, McAllester, Singh & Mansour, NeurIPS 2000

Policy Gradient Methods for RL with Function Approximation

The policy-gradient theorem.

The theoretical licence for every actor-critic method.

Learning outcome: State the policy gradient theorem and identify each term in an implementation.

Kakade, NeurIPS 2002

A Natural Policy Gradient

Precondition the gradient by the Fisher information for invariance to parameterisation.

The conceptual seed of TRPO.

Mnih et al., ICML 2016

Asynchronous Methods for Deep RL

Parallel actors, advantage actor-critic.

The actor-learner idea scales into IMPALA.

Learning outcome: Explain what the critic replaces in REINFORCE and why parallel actors decorrelate updates.

Schulman et al., ICLR 2016

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Bias-variance controlled advantage estimates via a TD(λ)-style trace.

Inside virtually every modern PPO implementation, including LLM RLHF.

Learning outcome: Explain the bias-variance trade-off that lambda controls, and pick a value for a given episode length.

The two quantities behind trust regions, SAC and DPO

Entropy and KL divergence

Entropy measures how spread a policy is; KL measures how far one policy is from another, asymmetrically. Both are estimated from samples in practice, and the estimator you choose matters.

TRPO's constraint, PPO's clip as a cheap stand-in for it, SAC's entropy bonus, and the implicit KL penalty inside DPO.

Learning outcome: Estimate a KL from samples, say which direction you are penalising and why, and explain what an entropy bonus does to exploration.

Schulman et al., ICML 2015

Trust Region Policy Optimization

Monotonic-improvement updates via a KL trust region. Understand the idea, the maths is Deeper.

PPO is its practical simplification.

Learning outcome: Explain the trust region and why PPO's clip is a cheap approximation of it.

Huang et al., ICLR 2022 Blog Track

The 37 Implementation Details of PPO

Every implementation detail that changes PPO results.

Practical companion to Phase 3.

Learning outcome: Audit a PPO implementation against the list and say which details your version omits and what each omission costs.

Schulman et al., arXiv 1707.06347, 2017

Proximal Policy Optimization

Clipped surrogate objective. First-order, robust, ubiquitous. Non-negotiable. Read it with the 37 implementation details station.

The default RL algorithm. Base of InstructGPT RLHF and parent of GRPO, DAPO, VAPO.

Learning outcome: Write the clipped surrogate objective from memory and explain what the ratio and the clip range each do.

Deliverable: an agent, a protocol, an audit

PPO from scratch under a protocol

Implement PPO, run it under a protocol you wrote first, then audit your implementation against the 37 details and count what you missed. The audit is where most of the learning is.

PPO is the base of GRPO and of most LLM RL. You want to trust your own version of it.

Learning outcome: Get PPO working on a continuous-control task and state its result with a reproducible protocol rather than a remembered number.

Wu et al., NeurIPS 2017

ACKTR

Kronecker-factored natural gradient actor-critic.

Optional depth on natural gradients.

Espeholt et al., ICML 2018

IMPALA

Decoupled distributed actor-learner plus V-trace off-policy correction. 250,000 frames per second.

SEED RL. V-trace recurs in AlphaStar.

Silver et al., ICML 2014; Lillicrap et al., ICLR 2016

Deterministic Policy Gradient / Continuous control with deep RL

Off-policy actor-critic for continuous action spaces.

Base of TD3 and DrQ-v2.

Learning outcome: Explain why deterministic policies need off-policy exploration noise.

Fujimoto, van Hoof & Meger, ICML 2018

Addressing Function Approximation Error in Actor-Critic Methods

Clipped double-Q, delayed policy updates and target-policy smoothing fix DDPG's overestimation. The clipped double-Q term takes the minimum of two independently trained critics, which is related to Double DQN's motivation but not the same estimator: it deliberately biases the target downwards rather than decorrelating selection from evaluation.

The standard deterministic continuous-control baseline. Base of TD3+BC.

Learning outcome: Implement the three TD3 changes on top of a working DDPG and say what each one buys.

Haarnoja et al., ICML 2018; arXiv 1812.05905

Soft Actor-Critic

Maximum-entropy off-policy actor-critic. Sample-efficient and stable. Non-negotiable for continuous control.

The default continuous-control algorithm. Used inside MBPO, PEARL, DrQ.

Learning outcome: Say what the entropy term does to exploration and how the temperature is tuned.

Deliverable: a fair comparison

Three continuous-control agents, one protocol

Run all three on the same tasks, the same interaction budget and the same evaluation protocol, then explain the differences in code before explaining the differences in the plot.

The comparison every offline and robotics paper assumes you have already internalised.

Learning outcome: Compare three closely related algorithms fairly, and attribute differences to specific implementation choices.