Evaluation & experiment design
How results are established and reported. Read it before your first deep agent, on the tabular runs you already have, and keep coming back: it decides whether your own experiments are worth anything.
What you will learn
Phase 1B taught you to respect the spread. Deep agents make it worse: more seeds cost more, budgets are easier to bend, and a mean over five runs hides most of what happened.
How an RL experiment is designed so that its conclusion holds: what to control and what to tune, how runs are compared, which benchmarks and protocols the field uses, and rliable's interval estimates, performance profiles and aggregate metrics. The exercise reports one earlier comparison the way a careful paper would.
You can design, run and report a comparison with stated budgets and honest uncertainty, and you can tell when a published gap is not worth believing.
Patterson, Neumann, White & White
Empirical Design in Reinforcement Learning
How to design an RL experiment whose conclusion survives contact with someone else's rerun: what to control, what to tune, how to report, and which comparisons are not worth running.
The methodology backbone for every project and for the investigation line.
Learning outcome: Design a study before running it, and defend each of its choices.
Resources
Intervals, paired designs, multiple comparisons
Comparing algorithms without fooling yourself
Independent seeds, paired comparisons where you can afford them, bootstrap intervals, and an honest account of what a gap between two medians over five runs is worth. Start from the spread you measured in Phase 1B and add aggregate metrics when you move to a suite of tasks.
Feeds directly into rliable's aggregate metrics and the reporting exercise at the end of this line.
Learning outcome: Choose an interval or test appropriate to a comparison, and state the uncertainty rather than the point estimate.
Prerequisites
Atari ALE, MuJoCo, DM Control, Procgen, D4RL, MineRL, Isaac Gym / Isaac Lab
Benchmarks to know
Atari ALE (Bellemare 2013; Machado 2018 protocols). MuJoCo and DM Control. Procgen. D4RL. MineRL. Isaac Gym and Isaac Lab for massively parallel GPU sim.
Pick small, fast environments so you can iterate.
Resources
- Arcade Learning Environment (paper)
- Revisiting the ALE (paper)
- DM Control (code, optional)
- Procgen (code, optional)
- D4RL (code, optional)
- MineRL (code, optional)
- Isaac Lab (code, optional)
Agarwal et al., NeurIPS 2021, outstanding paper
Deep RL at the Edge of the Statistical Precipice
Point estimates over a handful of runs hide enormous uncertainty. Report interval estimates, performance profiles and aggregate metrics such as the IQM instead of the mean over a few seeds.
The reporting standard for any comparison across a task suite.
Learning outcome: Produce an IQM with stratified bootstrap confidence intervals and a performance profile for your own runs.
Prerequisites
Resources
- Deep RL at the Edge of the Statistical Precipice (paper)
- rliable (code, optional)
Deliverable: a figure and a limitations paragraph
Re-report a comparison you already ran
Take one comparison from an earlier phase and report it the way a careful paper would: aggregate metric, stratified bootstrap intervals, performance profile, stated budgets, and the limitations you would rather not write.
The last step before the investigation line, where you produce evidence nobody asked you for.
Learning outcome: Turn a set of runs into a figure and a paragraph that would survive review.
Exercise deliverables
- IQM with stratified bootstrap confidence intervals for one earlier comparison
- A performance profile across tasks or seeds
- Interaction budget and tuning budget stated for every arm
- A limitations paragraph, including the comparisons your data cannot support