Imitation & inverse RL
Learning from demonstrations and recovering rewards.
What you will learn
Reward is the input every method so far has taken for granted. Many tasks have no reward you can write down but do have demonstrations of the behaviour you want.
Behaviour cloning and its compounding error, DAgger's interactive fix, MaxEnt IRL for recovering a reward under which the expert is optimal, GAIL for matching the expert's occupancy measure without a reward, and AIRL for a reward that transfers.
You can train a policy from demonstrations, explain why cloning drifts and how DAgger stops it, and recover a reward you can hand to an RL agent.
Pomerleau, 1988
Behavior cloning (ALVINN)
Supervised action prediction.
Baseline for all imitation.
Learning outcome: Explain compounding error in behaviour cloning and why DAgger fixes it.
Resources
- ALVINN (NeurIPS 1988) (paper)
- CS285 lecture 2: Imitation Learning (5 parts) (video, optional)
Ross, Gordon & Bagnell, AISTATS 2011
DAgger
Fixes compounding error in BC via interactive expert queries.
Interactive imitation.
Resources
- DAgger (paper)
Ziebart et al., AAAI 2008
Maximum Entropy IRL
Recover a reward under which the expert is optimal, resolving ambiguity via max-entropy.
Max-ent underlies SAC and GAIL.
Resources
- Maximum Entropy IRL (PDF) (paper)
- CS285 lecture 20: Inverse RL (4 parts) (video)
- Bootcamp 10b: Inverse RL (Finn) (video)
Ho & Ermon, NeurIPS 2016
Generative Adversarial Imitation Learning
Occupancy-measure matching as a GAN. Imitate without recovering the reward.
AIRL.
Resources
- GAIL (paper)
Fu, Luo & Levine, ICLR 2018
Adversarial IRL
Recovers a transferable, disentangled reward.
Reward transfer.
Resources
- AIRL (paper)