Sandboxes for RL rollouts at scale

Run your whole RL loop on one platform, GPUs included.

RL compute environments

Deterministic resets, parallel rollouts, long-horizon state, and GPUs on-demand.

Snapshot and restore

Capture the environment's memory and filesystem once and boot every sandbox from that same state.

Rollouts in parallel

Fork a snapshot into a fleet of isolated sandboxes, each with its own filesystem, processes, and ports.

Sub-second boots

Sandboxes cold boot in under a second so startup times never bottleneck your evals.

No timeouts

Environments run as long as the task demands, no timeouts.

Statefulness & Persistence

Read and write to persistent volumes and create snapshots to restore any sandbox state in the future.

One Platform

Run evals and retrain your models, all in one place.

Easily run RL rollouts

Run Docker-in-Docker

Run any existing Docker image in a sandbox, giving you a fully reproducible environment between your local and the cloud.

Fan out sandboxes

Spin up thousands of sandboxes, and write outputs to a shared volume.

Attach GPUs

Add a GPU to any sandbox with just a single line of code.

# Run the policy update on a GPU — same SDK
from beam import function, Volume

@function(
    gpu="H100",
    volumes=[Volume(name="trajectories", mount_path="./data")],
)
def update_policy():
    batch = load_trajectories("./data")
    loss = policy.train_step(batch)
    save_checkpoint("./data/policy-v43.pt")

# Pause a stateful environment and pick it back up later
snapshot_id = env.snapshot_memory()
env2 = Sandbox().create_from_memory_snapshot(snapshot_id)

Frequently asked questions

How fast can a snapshot restore?
A restore is just booting a new sandbox from your saved snapshot, which takes under a second with dependencies included. Every episode starts from the identical filesystem state, and nothing from the previous episode leaks in.

Can I pause and resume an episode mid-run?
Yes. Memory snapshots capture the sandbox's full state — running processes and exposed ports included — and create_from_memory_snapshot picks up exactly where the episode left off. Useful for long-horizon tasks and branching a live trajectory.

How many rollouts can run at once?
Fleets are created programmatically, so parallelism scales with your training run. Each sandbox is isolated with its own filesystem and processes, and billing is per-second — terminated environments cost nothing.

Can environments have GPUs?
Yes. Pass gpu to the Sandbox constructor for environments that render or run models, and keep CPU-only environments on fractional CPUs so large fleets stay cheap.

Where does the training step run?
On Beam, next to the rollouts. Decorate your update function with @function(gpu=...) and mount the volume your environments write to — rollouts, trajectories, and gradient steps stay on one platform.