Exploration
Intrinsic motivation and hard-exploration Atari.
What you will learn
Every agent so far explored by adding noise to its actions. On a task with sparse reward, Montezuma's Revenge or any long corridor, noise never finds the reward and the agent has nothing to learn from.
Intrinsic reward for novelty: pseudo-counts from density models, prediction error in a learned feature space (ICM) and against a fixed random network (RND), posterior-sampling exploration with ensembles, Go-Explore's remember-and-return, and the regret bounds that say what optimism buys in an unknown MDP.
You can add an intrinsic reward to an existing agent, say which novelty signal suits a task and why, and recognise the noisy-TV failure when you see it.
Bellemare et al., NeurIPS 2016
Unifying Count-Based Exploration and Intrinsic Motivation
Pseudo-counts from density models.
Concept behind count-based bonuses.
Pathak et al., ICML 2017
Curiosity-Driven Exploration by Self-Supervised Prediction
Intrinsic reward = forward-model prediction error in a learned feature space. Beware the noisy-TV failure mode.
RND.
Resources
Burda et al., ICLR 2019
Exploration by Random Network Distillation
Novelty = error predicting a fixed random net. First to beat average human on Montezuma's Revenge. Clean and widely used.
Default intrinsic bonus.
Learning outcome: Explain why a fixed random target network gives a usable novelty signal, and where intrinsic bonuses fail.
Resources
- Random Network Distillation (paper)
Osband et al., NeurIPS 2016
Deep Exploration via Bootstrapped DQN
Posterior-sampling-style exploration via ensemble heads.
Ensemble exploration.
Resources
- Bootstrapped DQN (paper)
Ecoffet et al., Nature 2021
Go-Explore
Remember and return to promising states, then explore. Cracks hard-exploration Atari.
Intrinsic-motivation ideas feed NGU and Agent57.
Resources
- First return, then explore (arXiv) (paper)
- Go-Explore (Nature 2021) (paper)
Jaksch, Ortner & Auer, JMLR 2010
UCRL2 and regret in unknown MDPs
Optimism lifted from bandits to unknown MDPs: keep confidence sets over transitions and rewards, plan optimistically inside them, and bound total regret over T steps. A different problem from bandit regret, and a much harder one.
The theory the heuristic exploration bonuses on this line are descended from.