RL for LLMs & reasoning

Christiano et al., NeurIPS 2017

Deep RL from Human Preferences

Learn a reward model from pairwise trajectory comparisons, then optimise with RL. The intellectual root of RLHF.

Everything on this line.

Ziegler et al., 2019; Stiennon et al., NeurIPS 2020

Fine-Tuning LMs from Human Preferences / Learning to Summarize from Human Feedback

RLHF applied to language models. Skim Ziegler, read Stiennon.

InstructGPT.

Ouyang et al., NeurIPS 2022

Training language models to follow instructions with human feedback

The three-stage recipe: supervised fine-tuning on demonstrations (SFT), a reward model fit to human preferences, then PPO against that reward with a KL penalty back to the SFT policy. Non-negotiable.

The template behind ChatGPT and all subsequent LLM RL.

Learning outcome: Draw the three-stage pipeline and say what data each stage needs.

Rafailov et al., NeurIPS 2023

Direct Preference Optimization

Reparameterise the RLHF objective so the LM is its own reward model. A simple supervised loss, no RL loop. Non-negotiable.

Spawned the direct-alignment family.

Learning outcome: Explain how DPO removes the reward model and the RL loop, and what it gives up.

Azar et al., AISTATS 2024; Ethayarajh et al., 2024; Hong et al., 2024; Meng et al., NeurIPS 2024

DPO variants

IPO (squared loss), KTO (prospect theory, unpaired signals), ORPO (reference-free, folds SFT in), SimPO (length-normalised reference-free reward). Know the landscape, read 2–3.

The standard cheap-alignment toolkit.

Deliverable: two runs, one comparison

Preferences into a policy, twice

Train a small reward model on a preference dataset and report its held-out accuracy, then run DPO on the same preferences and compare both against the SFT baseline under one evaluation.

The two halves of alignment practice, on a scale you can iterate on.

Learning outcome: Say what preference data does and does not encode, having fit it two different ways.

Shao et al., arXiv 2402.03300, 2024

DeepSeekMath (introduces GRPO)

Drop PPO's critic. Sample a group of responses per prompt and use group-relative rewards as the baseline. Non-negotiable.

The algorithm behind DeepSeek-R1 and the default for reasoning RL.

Learning outcome: Write GRPO's advantage and say exactly what it replaces in PPO.

Term popularised by Lambert et al., Tülu 3, 2024

RL with Verifiable Rewards

Use automatically checkable correctness (math answer, unit tests) as reward.

The concept behind every reasoning-RL recipe.

DeepSeek-AI, arXiv 2501.12948, Jan 2025

DeepSeek-R1

Large-scale RL with verifiable rewards elicits emergent long chain-of-thought reasoning. R1-Zero uses pure RL with no SFT: AIME 2024 pass@1 from 15.6% to 71.0%.

Triggered the 2025 RLVR explosion.

OpenAI blog, Sept 2024, no method paper

Learning to Reason with LLMs

Established inference-time-scaling reasoning RL. Method details undisclosed. Treat claims as company-reported.

Context for R1 and the 2025 recipes.

Kimi Team, arXiv 2501.12599, 2025

Kimi k1.5: Scaling RL with LLMs

A full RL-scaling recipe emphasising long context and simplified policy optimisation. Deliberately no MCTS, value functions or PRMs.

Excellent contrast to R1.

Yu et al., ByteDance Seed + Tsinghua, arXiv 2503.14476, 2025

DAPO

Four fixes over GRPO: Clip-Higher, Dynamic Sampling, token-level loss, overlong-reward shaping. 50 on AIME 2024 with Qwen2.5-32B in half the steps.

As of the 2025 papers on this line, the strongest fully open recipe. Re-check before relying on it.

Liu et al., Sea AI Lab / NUS, COLM 2025

Dr. GRPO

Removes GRPO's length and std normalisation bias that inflates wrong-answer length. Short and clarifying.

Fixes a subtle bias in every GRPO implementation.

Zheng et al., Qwen Team, arXiv 2507.18071, 2025

Group Sequence Policy Optimization

Sequence-level rather than token-level importance ratios. Stabilises MoE RL. Used in Qwen3.

Stable RL at MoE scale.

Yue et al., ByteDance Seed, arXiv 2504.05118, 2025

VAPO

Value-based augmented PPO with length-adaptive GAE. 60.4 on AIME 2024 within about 5,000 steps. The critics-are-not-dead counterpoint.

Open question: do critics beat critic-free at scale?

Lightman et al., 2023; Wang et al., ACL 2024

Let's Verify Step by Step / Math-Shepherd

Step-level supervision beats outcome supervision on MATH, releases PRM800K. Math-Shepherd automates per-step rewards via Monte-Carlo rollouts (Deeper).

Process vs outcome rewards is an open thesis question.

Zhang et al., arXiv 2509.02547, 2025; TMLR 2026. Plus ToRL, ToolRL, VerlTool, A Practitioner's Guide to Multi-turn Agentic RL

The Landscape of Agentic RL for LLMs: A Survey

Frames agentic RL as temporally extended POMDPs and catalogues 500+ works. Survey is Essential, individual papers are Deeper.

The direction LLM RL was moving in through 2025-2026, and a plausible thesis area. Treat it as a snapshot, not a forecast.

Deliverable: a run, an ablation, a noise check

GRPO on a verifiable task

Run GRPO on a small model with a rule-based reward, then ablate one DAPO trick and check whether its effect is bigger than the seed noise. Keep the model small enough to iterate in a day.

The smallest honest version of the experiment the whole LLM RL line is about.

Learning outcome: Run a reasoning-RL experiment and tell a real effect apart from run-to-run variation.