Multi-agent & games
Many learners, credit assignment, and self-play at scale.
What you will learn
With more than one learner the environment stops being stationary: every other agent's policy is part of the dynamics, and it is changing too. Single-agent convergence arguments no longer apply.
Independent learning and its failure, centralised critics with decentralised actors (MADDPG), value factorisation (VDN, QMIX), counterfactual credit assignment (COMA), league self-play at scale (AlphaStar, OpenAI Five), and the game-theoretic line: CFR and PSRO.
You can set up centralised training with decentralised execution, choose a credit-assignment scheme for a cooperative task, and explain what self-play needs so that it does not chase its own tail.
Tan, ICML 1993
Independent Q-learning
The naive baseline and its non-stationarity problem.
Motivates CTDE.
Resources
Lowe et al., NeurIPS 2017
MADDPG
Centralised critic, decentralised actors for mixed cooperative and competitive continuous control.
The CTDE template.
Resources
- MADDPG (paper)
Sunehag et al., AAMAS 2018; Rashid et al., ICML 2018, JMLR 2020
Value-Decomposition Networks / QMIX
Factor a joint value into per-agent utilities with a monotonic mixing net.
Cooperative credit assignment.
Resources
- VDN (paper)
- QMIX (ICML 2018) (paper)
- QMIX (JMLR 2020) (paper)
Foerster et al., AAAI 2018
Counterfactual Multi-Agent Policy Gradients
Counterfactual baseline for credit assignment.
Policy-gradient MARL.
Resources
- COMA (paper)
Vinyals et al., Nature 2019; OpenAI, arXiv 2019
AlphaStar / OpenAI Five
League self-play plus large-scale PPO and IMPALA at professional level. Read for the scaling story.
Large-scale self-play.
Resources
- AlphaStar (Nature 2019) (paper)
- OpenAI Five (paper)
Zinkevich et al., NeurIPS 2007; Lanctot et al., NeurIPS 2017
Counterfactual Regret Minimization / PSRO
Game-theoretic RL. CFR is the basis of superhuman poker. Nash and self-play concepts.
Equilibrium-seeking learning.