Phase 1B · 5 stations

Experimental practice

How to run an experiment that means something, starting with tabular ones. Read this alongside Phases 1 and 2 rather than after them.

What you will learn

By now you have tabular agents and learning curves. A single run tells you almost nothing: seeds alone move deep-RL results more than most method changes do, and a comparison made without a protocol is not evidence.

Seeds and spread first, then the paper that documented how fragile published results are, a protocol you write before the runs, a small study on the CliffWalking comparison you already have, and a fixed order for debugging an agent that does not learn.

You can run a comparison whose conclusion survives someone else's rerun, and you check the usual bugs before you touch a hyperparameter.

Station 1 · core

Statistical power in deep RL

One run is an anecdote

Re-run the same agent with independent seeds and look at the spread before comparing anything to anything. Deep RL learning curves from a single run routinely differ more between seeds of one method than between two methods.

Sets up every comparison you will make, and the aggregate metrics on Phase 7.

Learning outcome: Report a result as a distribution over seeds, and say how many seeds a claim of a given size needs.

Resources

Station 2 · core

Henderson et al., AAAI 2018

Deep RL That Matters

Documents how fragile and irreproducible deep-RL results are: seeds, hyperparameters, codebases. Read early.

It will change how you run experiments.

Learning outcome: List the confounders that make two published numbers incomparable, and say which ones your own runs are exposed to.

Resources

Station 3 · core

Budgets, evaluation, logging

Decide the protocol before you run

Fix and write down what is held constant: environment version, interaction budget, number and policy of evaluation episodes, tuning budget per method, held-out tasks, and the exact commit and config for every run. A protocol chosen after seeing results is not a protocol.

Turns a pile of runs into evidence. Reused for every project on this map.

Learning outcome: Write the protocol paragraph for a comparison before running it, and hold yourself to it.

Resources

Station 4 · exercise

Deliverable: a protocol, a plot, a conclusion

Run your tabular comparison properly

Take the CliffWalking comparison you already ran and redo it as a study: protocol first, enough seeds, uncertainty on the plot, and a conclusion that says what the evidence does not support.

The template for every experiment from here on, including the deep ones.

Learning outcome: Produce a comparison another person could re-run from your write-up alone.

Exercise deliverables

  • The protocol, written down before the runs
  • At least 10 seeds per method, with median and interquartile range on the plot
  • A three-sentence conclusion that states what the evidence does not support
  • Configs and commit hashes logged for every run
Station 5 · core

The usual suspects, in order

Implementation failure or algorithm failure

Most agents that do not learn are broken, not badly tuned. Work through the usual suspects in a fixed order: terminal masking, truncation treated as termination so the bootstrap is cut off, targets computed with the wrong network or without a stopped gradient, reward and observation scaling, wrapper order, and only then hyperparameters.

Read this before DQN. It will save you a fortnight.

Learning outcome: Rule out implementation faults in a fixed order before concluding anything about an algorithm.

Resources