Hierarchical & goal-conditioned
Temporal abstraction and goals as inputs.
What you will learn
A flat policy acts one primitive step at a time and learns one task. Long horizons and families of related goals both defeat it: credit has to travel too far, and every new goal starts from nothing.
Temporal abstraction through options and their end-to-end learning (Option-Critic), manager-worker hierarchies (FeUdal Networks, HIRO), and goals as inputs: universal value functions and hindsight relabelling of failed trajectories (HER).
You can add a goal input to a value function, relabel replay so that sparse binary rewards become learnable, and say when a hierarchy is worth its extra machinery.
Sutton, Precup & Singh, Artificial Intelligence 1999
Between MDPs and semi-MDPs: the options framework
Temporally extended actions. Foundational.
Option-Critic.
Resources
- Between MDPs and semi-MDPs (paper)
Bacon, Harb & Precup, AAAI 2017
The Option-Critic Architecture
Learn options end to end via a policy-gradient theorem for options.
End-to-end hierarchy.
Resources
- The Option-Critic Architecture (paper)
Vezhnevets et al., ICML 2017
FeUdal Networks (FuN)
Manager sets latent goals for a worker at different time scales.
HIRO.
Resources
- FeUdal Networks (paper)
Nachum et al., NeurIPS 2018
HIRO
Data-efficient off-policy HRL with goal relabelling.
Off-policy hierarchy.
Resources
- HIRO (paper)
Schaul et al., ICML 2015
Universal Value Function Approximators
Value functions generalised over goals.
HER.
Resources
- UVFA (ICML 2015) (paper)
Andrychowicz et al., NeurIPS 2017
Hindsight Experience Replay
Relabel failed trajectories with achieved goals to learn from sparse binary rewards.
A staple of goal-conditioned and robotic RL.
Resources
- Hindsight Experience Replay (paper)