18.4 Distributed RL Training

Goal of this section: understand the sources and corrections of training-inference mismatch, learn how veRL assigns models and GPUs with ResourcePool, Worker, and Driver, review common techniques for generation throughput and training memory, and decide when asynchronous scheduling is necessary.

The final diagnostic step in the previous section compared model versions and probabilities on the generation and training sides. That check assumed that generation, scoring, and updating were three sequential function calls in one script. Once training moves to a cluster, the assumption no longer holds.

Training a 70B mathematics model together with a Reference Model and Reward Model can involve more than one hundred billion parameters. Generation must run on one GPU group and updates on another. A batch then moves through a distributed loop:

1. Generation: rollout GPUs generate responses from the current policy, often taking seconds per response.
2. Scoring: reward processes consume the responses and call verifiers or reward models.
3. Updating: training GPUs compute gradients and update the Actor, often in hundreds of milliseconds.
4. Synchronization: the new parameters must return promptly to the rollout GPUs.

On one machine these are function calls. Across machines each becomes data transfer and waiting between processes. If parameters arrive late, the next batch is still generated by an old policy and the consistency established in 18.3 is lost.

This section answers four questions: where generation-training probability differences come from, how models are placed on GPUs, which bottleneck to address first—generation speed or training memory—and when asynchronous scheduling is worth the stale-experience cost. MoE models add expert routing and pipeline utilization to the same design.

First-pass takeaway

Remember the order: generation → reward → update → synchronize parameters. veRL, slime, and OpenRLHF implement it differently, but all decide which GPUs perform these four steps and when data moves between them.

---

Where Training-Inference Mismatch Comes From

Parameter synchronization solves version drift: rollout and training GPUs receive the same weights. The same weights do not guarantee the same probabilities.

Suppose vLLM on the rollout side computes a token probability of 0.30 in FP8, while the trainer recomputes 0.29 from the same weights with BF16 and different kernels. Neither computation is inherently wrong. The paths differ: vLLM or SGLang uses KV caches and low precision for generation speed, while FSDP or Megatron retains a computation graph for backpropagation.

Let ${\pi }_{\text{rollout}}$ denote the policy actually executed by the rollout engine and ${\pi }_{\text{old}}$ the old policy recorded by the trainer. They should match. Floating-point precision, kernel implementations, and MoE routing can make them diverge. This is training-inference mismatch.

• The rollout side usually generates with vLLM or SGLang in FP8 or BF16 and uses KV-cache optimizations.
• The training side usually computes log probabilities and gradients with FSDP or Megatron in BF16 or FP32, sometimes with activation recomputation.

Small errors on high-probability tokens are often harmless. Errors on many low-probability tokens can accumulate and directly distort the importance ratio in PPO.

Why PPO Clipping Cannot Correct This Mismatch

PPO constrains the size of one update with

$${\mathcal{L}}^{\text{CLIP}}=\mathbb{E}\left[\min \left(r_t(\theta ){\hat{A}}_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right],$$

where

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{\text{old}}(a_t\mid s_t)}.$$

${\hat{A}}_t$ measures how much better action $a_t$ is than the current baseline, and $ϵ$ controls the allowed deviation from 1. If the old policy assigns probability $0.20$ and the new policy assigns $0.24$, then $r_t=1.2$, exactly the upper boundary when $ϵ=0.2$.

This calculation assumes that the denominator is the policy that generated the action. If the response came from ${\pi }_{\text{rollout}}$ while the trainer recomputed ${\pi }_{\text{old}}$ along a different numerical path, the ratio is wrong before any update begins. Clipping can limit parameter movement; it cannot reconcile probabilities computed by two engines.

Diagnose the mismatch in this order:

1. Verify the model version used for rollout and confirm that synchronization completed.
2. Record log probabilities for the same tokens on both sides and locate where errors concentrate.
3. Align precision and kernels, then measure whether the discrepancy shrinks.
4. For MoE models, record expert routing and verify that training reproduces the rollout route.

Correcting Training-Inference Mismatch

• Align numerical precision: temporarily replace FP8 rollout with FP16 or BF16 to determine whether low precision dominates the error. If FP8 remains necessary, monitor the discrepancy and apply importance-sampling corrections.
• Record the actual behavior policy: save rollout-time log probabilities rather than treating a trainer-side recomputation as the behavior probability.
• Recompute and validate: recompute log probabilities with the training engine and compare them token by token. Recalculation does not recover the true behavior policy, but it exposes the location and size of the mismatch.
• Limit extreme ratios: Truncated Importance Sampling clips unusually large ratios so that a few anomalous tokens cannot dominate the gradient.
• Handle tail tokens: dynamic vocabulary pruning can remove the lowest-probability region where discrepancies are largest.
• Replay MoE routing: R3, or Rollout Routing Replay, reproduces rollout expert choices during training.

The degree of on-policy training depends on the distance between ${\pi }_{\text{rollout}}$ and the current policy. Parameter synchronization controls version distance; precision alignment, probability logging, and importance-sampling correction control computation-path differences.

---

Allocating Models and GPUs

Once probabilities align, five roles still need GPU placement and data exchange: Actor, Critic, Reference Model, Reward Model, and rollout engine. veRL divides this problem into the algorithm loop, model computation, and resource allocation.

HybridFlow uses single-controller, multi-model orchestration. One Driver runs the algorithm loop and resource scheduler, directing Workers on individual GPUs. Workers represent the Actor, Critic, Reference Model, Reward Model, and rollout engine and share ResourcePools.

Three Core Abstractions

ResourcePool: groups of GPUs. A pool can host one or more models. Models can share GPUs through colocation or use separate disaggregated pools.

Worker: an encapsulated model instance. ActorWorker handles loss, backpropagation, and optimizer updates. RolloutWorker handles batched generation and weight synchronization.

Driver: the single controller. It executes the RL loop: synchronize weights to rollout → sample responses → score them → compute values with the Critic → compute advantages and update the Actor → update the Critic.

HybridFlow's “hybrid” design allows one framework to combine 3D parallelism (TP × PP × DP), colocated or disaggregated deployment, FSDP/Megatron/DeepSpeed ZeRO training backends, and vLLM/SGLang/Hugging Face generation backends.

Comparing Mainstream Framework Architectures

Framework	Orchestration	Training Backend	Inference Backend	Typical Scale	Representative Users	Best Fit
veRL (HybridFlow)	Single controller	FSDP, Megatron-LM, DeepSpeed ZeRO	vLLM, SGLang, HF generate	8–1024 GPUs	Qwen, DeepSeek, ByteDance	Large-scale production and flexible resource combinations
OpenRLHF	Single controller with Ray Actor isolation	FSDP, DeepSpeed	vLLM	8–256 GPUs	Community and research teams	Research and medium-scale training
NeMo-Aligner	Multiple controllers	Megatron	TensorRT-LLM	8–512 GPUs	NVIDIA ecosystem and enterprise clusters	Production environments already using NeMo
TRL	Single process	Hugging Face Accelerate	HF generate	1–8 GPUs	Learners and rapid prototypes	Algorithm learning and small experiments

Choose from current scale and the existing stack: TRL for learning and prototypes; OpenRLHF or veRL for research and medium scale; veRL or NeMo-Aligner for large production systems.

---

Generation Speed and Training Memory

After placement, two bottlenecks dominate. Generation often consumes most of an iteration, while training must store weights, gradients, optimizer state, and activations. We first increase rollout throughput, then distribute training memory.

Core vLLM Optimizations

Technique	Problem	Principle	Typical Benefit	Importance for GRPO
PagedAttention	Contiguous KV-cache allocation fragments memory and yields 50%–70% utilization	Divide KV cache into fixed blocks and allocate them on demand, like virtual-memory pages	95%+ utilization and 2–4× larger effective batches	Foundational
Continuous Batching	Static batches wait for every sequence	Insert a new sequence as soon as another emits EOS	5–10× higher aggregate throughput	Largest benefit for long responses
Speculative Decoding	Autoregressive decoding has low compute density	A small draft model predicts several tokens and the large model verifies them in parallel	2–3× typical inference throughput	Useful for short, latency-sensitive responses
Prefix Caching	$G$ responses to one prompt repeat prefix computation	Hash and reuse the prompt's KV cache	Saves 70%–80% of prefix computation for $G=8$	A core GRPO optimization

SGLang Generation and Scheduling

SGLang uses RadixAttention to manage KV cache in a radix tree and reuse it across requests. Its programmatic frontend supports multi-turn calls, branches, and loops, and constrained decoding supports JSON and regular expressions.

Engine	Core Strength	Best Fit
vLLM	Mature PagedAttention and Continuous Batching ecosystem	General rollout, single-turn generation, GRPO for mathematics and code
SGLang	RadixAttention, multi-turn control flow, structured output	Agentic rollout, multi-turn tools, constrained decoding
TensorRT-LLM	Deep NVIDIA optimization and strong FP8 support	Maximum-throughput production on NVIDIA hardware

Distributing Memory Across GPUs

A 70B model cannot perform full-parameter BF16 training on one 80GB H100. With BF16 weights and gradients plus FP32 master weights and Adam moments, each parameter requires about 16 bytes:

Component	Type	Bytes per Parameter	70B Model	Purpose
Weights	BF16/FP16	2 B	140 GB	Current parameters
Gradients	BF16/FP16	2 B	140 GB	Accumulated after backpropagation
Master weights	FP32	4 B	280 GB	Stable optimizer copy
Adam first moment	FP32	4 B	280 GB	Gradient moving average
Adam second moment	FP32	4 B	280 GB	Squared-gradient moving average
Activations	BF16/FP16	Dynamic	~100 GB	Depends on batch and sequence length
Total	—	16 B/parameter	~1.22 TB	Far beyond one H100

DeepSpeed ZeRO shards progressively more state:

Level	Optimizer State	Gradients	Weights	Per-GPU Saving	Communication	Best Fit
ZeRO-1	Sharded	Replicated	Replicated	~4×	Low	Medium models or communication-limited systems
ZeRO-2	Sharded	Sharded	Replicated	~8×	Medium	Default for many training jobs
ZeRO-3	Sharded	Sharded	Sharded	$N$× for $N$ GPUs	High	Full-parameter large-model training

ZeRO-3 all-gathers the needed parameters temporarily during forward and backward passes. FSDP is PyTorch's native equivalent and veRL's default training backend.

Gradient checkpointing exchanges compute for memory. It saves only selected forward activations and recomputes the rest during backpropagation, reducing activation memory from $O(L)$ to $O(\sqrt{L})$ for $L$ Transformer layers while slowing training by roughly 20%–30%.

For a 70B model on 80GB H100s:

Configuration	Per-GPU Memory	Speed	Feasible?
Full parameters + Adam, no sharding	~940 GB	Baseline	No
ZeRO-3 training-state sharding	~118 GB	10%–15% slower	Still no
ZeRO-3 + gradient checkpointing	~30 GB	30%–40% slower	Yes
ZeRO-3 + checkpointing + LoRA	~8 GB	~40% slower per step, but far fewer trainable parameters	Yes

Industrial 70B RL often combines LoRA with FSDP to balance memory, speed, and training quality.

---

When Task Durations Diverge: Asynchronous Scheduling

Synchronous scheduling assumes similar rollout durations. Mathematics and ordinary code problems often satisfy that assumption. Tool and browser tasks do not: one trajectory may wait for an environment response while another has finished, leaving training GPUs idle behind the slowest task.

Asynchronous training lets generation and updating advance independently through a queue. Its cost is stale experience: the policy may update several times while a trajectory is being generated.

Comparing Three Asynchronous Frameworks

Framework	Publisher	Core Design	Staleness Handling	Typical Scale	Published Speedup	Representative Use
LlamaRL	Meta, 2025	Decentralized rollout and training workers with asynchronous weight broadcast	No explicit importance correction; continual version updates replace old samples	4096+ GPUs	10.4× over synchronous Llama-3-70B GRPO	Very large reasoning jobs
AReaL	Ant Group + Tsinghua, 2025	Fully asynchronous rollout; each trajectory records policy version and log probability	Token-level weights $\exp (\text{current\_logprob}-\text{gen\_logprob})$, clipped to $[0.8,1.2]$	1024 GPUs	2.77× for 671B MoE GRPO	MoE tasks requiring explicit bias control
AgentRL	THUDM/Zhipu, 2025	Asynchronous generation-training pipeline plus a unified environment interface	Queues, task isolation, and independently managed multi-turn sessions	Multi-machine, multi-environment	Used for AutoGLM	SWE, computer use, and deep research

LlamaRL favors simple horizontal scale: no central worker and no explicit correction. AReaL stores more trajectory metadata to compute and clip an explicit correction. AgentRL adds environment management because external interaction can take longer than model generation itself.

---

MoE and Pipeline Idle Time

The preceding design assumed a dense model and a full pipeline. MoE adds expert routing and all-to-all communication, while uneven sequences create pipeline bubbles.

Additional Complexity from MoE

DeepSeek V3, Qwen3, and GLM-4.5 use MoE. Each token activates only a few experts, distributing parameters across GPUs, but RL changes the token distribution and therefore expert load.

For illustration, a model may have about 20B dense parameters plus 256 experts of 5B parameters each. A sample activates eight experts, so roughly 40B expert parameters and 60B total parameters are active out of about 1.3T.

Challenge	Symptom	Mitigation	Representative Work
Expert-load imbalance	Hot experts are overused while others remain idle	Balancing loss toward activation frequency $1/\text{num\_experts}$ and dynamic routing	DeepSeek-V3, GShard
Cross-GPU communication	Expert parallelism requires token all-to-all	Optimize all-to-all kernels and overlap communication with computation	DeepEP
High token-level IS variance	Routing differences make token ratios fluctuate	Use one importance ratio for the whole sequence	GSPO

Reducing Pipeline Idle Time

Technique	Problem	Principle	Benefit
DualPipe	Pipeline bubbles prevent forward/backward overlap	Bidirectional scheduling overlaps forward stage $N$ with backward stage $N-1$ on one GPU	Bubble ratio falls from $\frac{P-1}{M}$ to $\frac{P-1}{2M}$
Best-Fit Packing	Uneven micro-batches finish at different times	Bin-pack different sizes across GPUs	DeepSeek V3 reports utilization rising from 70% to 95%

Locating Performance Bottlenecks

Every optimization can move the bottleneck. Faster generation may expose weight synchronization; sharding may expose communication; packing may expose data loading.

Tool	Purpose	What It Shows	Best Stage
PyTorch Profiler	PyTorch performance analysis	CPU/CUDA timeline, memory, kernel time, expensive operations	Training optimization
NVIDIA Nsight Systems	System-level GPU profiling	CUDA kernels, CPU-GPU synchronization, NCCL communication, stream overlap	Communication and scheduling
veRL Profiler	RL pipeline decomposition	Rollout, Actor, Critic, synchronization, and communication shares	First choice for RL pipelines

Symptom	Threshold	Direction
Slow rollout	More than 80% of total time	Add rollout GPUs, enable prefix caching, increase batch size, or separate deployment
Slow weight synchronization	More than 5%	Synchronize LoRA adapters, pack NCCL transfers, or synchronize less often
Heavy cross-GPU communication	all-reduce/all-gather over 10%	Increase micro-batches, accumulate gradients, or change the parallel partition
Activation OOM	CUDA out of memory during training	Enable checkpointing, shorten maximum sequences, or reduce batch size
Expert imbalance	Some GPUs at 90% while others are near 30%	Add expert-balancing loss and adjust routing or EP balancing
Stragglers	Longest sequence determines the batch time	Length bucketing or predicted-length grouping

Model FLOPs Utilization is

$$\text{MFU}=\frac{\text{actual FLOPs}}{\text{peak FLOPs}\times \text{time}}.$$

Configuration	Typical MFU	Main Bottleneck
Dense + FSDP + checkpointing, synchronous	35%–45%	Activation recomputation and all-reduce
MoE + expert parallelism + DualPipe	50%–60%	Expert all-to-all and load imbalance
Asynchronous RL with separate rollout	Trainer 40%–50%, rollout 70%–80%	Synchronization and queue waiting

Below 30% MFU, inspect communication, data loading, and rollout waiting with the time decomposition.

---

Summary

Distributed training still follows generation → reward → update → synchronize parameters. Every technique in this section addresses a bottleneck in that loop.

• Training-inference mismatch: identical weights do not guarantee identical probabilities. PPO clipping cannot repair differences between inference and training computation paths. Align precision, record the true behavior policy, and clip extreme importance ratios.
• Resource allocation: veRL groups GPUs with ResourcePool, encapsulates models with Worker, and runs the algorithm loop through a single Driver.
• Two bottlenecks: PagedAttention, Continuous Batching, and Prefix Caching improve rollout throughput; FSDP/ZeRO and gradient checkpointing distribute training memory. A 70B model needs roughly 1.2TB for full training state, while ZeRO-3 plus checkpointing can reduce the per-GPU requirement to about 30GB.
• Synchronous versus asynchronous: synchronous training keeps data fresh. Asynchronous training reduces waiting but introduces stale trajectories. LlamaRL applies no explicit correction, AReaL corrects explicitly, and AgentRL isolates multi-turn environment tasks.
• MoE and pipelines: MoE adds expert imbalance and all-to-all communication; DualPipe and Best-Fit Packing reduce idle time.

Distributed systems coordinate compute, but training still needs a steady supply of executable and verifiable data. 18.5 Large-Scale RL Data Engineering follows the lifecycle of a trajectory through tasks, environments, rewards, and failure feedback.

Further Reading

• HybridFlow: A Flexible and Efficient RLHF Framework (veRL)
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
• Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)
• SGLang
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
• DeepSeek-V3 Technical Report
• GSPO: Group Sequence Policy Optimization