Phase 7B · 3 stations

Distributed & scale

Throughput, for when a project genuinely needs it. Reference material: come back when your protocol asks for more runs than your machine can deliver.

What you will learn

A protocol written in Phase 7 can ask for more runs than one machine delivers, and most attempts to scale RL up reinvent the same handful of architectures badly.

Ape-X's many actors and one prioritised buffer, SEED RL's accelerator-side inference, and vectorised environments, which is what most small-scale distributed RL really needs.

You can pick the throughput architecture a project actually needs, from batching environments on one GPU to actors on a cluster.

Station 1 · track

Horgan et al., ICLR 2018

Distributed Prioritized Experience Replay

Hundreds of actors, one shared prioritised buffer, one learner. It assumes prioritised replay: read PER (Value-based deep RL) first if you have not.

The distributed-RL mental model.

Resources

Station 2 · reference

Espeholt et al., ICLR 2020

SEED RL

Centralised accelerator-side inference for throughput far above IMPALA.

Modern distributed RL infra.

Resources

Station 3 · track

Where the throughput actually comes from

Vectorised, compiled and asynchronous environments

Most 'distributed RL' at small scale is really vectorisation: batching environments, moving the step loop onto the accelerator, or decoupling actors from learners. Know the options before you reach for a cluster.

The cheapest way to buy the seeds your protocol needs.

Resources