B.8 Asynchronous Training

Legacy Page: Asynchronous Training Architecture (Merged into B.1)

This page is kept as an entry point for legacy links. The core content has been merged into the "Asynchronous Training Architecture" part of B.1 RL Training Infrastructure: Sampling, Asynchrony, and Distributed Systems. The original text is preserved below (in English translation) so readers coming from old links can map the content.

LLM RL training has a core contradiction: generation (inference) is extremely slow, training is relatively fast, and serializing them causes GPUs to sit idle. Asynchronous training exists to resolve this.

This section answers one question: how do we run generation and training in parallel? We first quantify how severe the problem is, then introduce three deployment patterns, and finally dig into two engineering issues: how to synchronize new parameters, and how to deal with data generated by older policies.

The Problem: Serial Generation and Training Means GPUs Idle 99% of the Time

When you run GRPO with TRL, each training step typically looks like this:

1) Generate 512 responses         <- inference, slow; training GPUs wait
2) Compute loss and update params <- training, fast; inference GPUs wait
3) Generate another 512 responses <- back to step (1)

Steps (1) and (2) are serialized, which means one side of the GPUs is always waiting. How slow is it in practice?

On a single H100 (BF16, vLLM offline mode), a typical GRPO step (G=8 × 64 prompts = 512 rollouts) looks like:

Output Length per Sample	Total Tokens	7B Model Time	32B Model Time
2K tokens	~1M	3 minutes	14 minutes
8K tokens	~4M	11 minutes	56 minutes
32K tokens	~16M	45 minutes	3.7 hours

But the training part (backprop + parameter update) may take only about 30 seconds.

In other words, training GPUs spend about 99% of the time waiting for generation to finish. There is also the "straggler problem": GRPO asks you to sample multiple responses per prompt, and you must wait for the slowest one to finish.

Three Deployment Patterns

Synchronous Mode

The simplest setup: one GPU group alternates between generation and training.

timeline: [gen b0] [train b0] [gen b1] [train b1] [gen b2] ...
              ^ idle     ^ idle     ^ idle

Pros: simple code, minimal GPUs. Cons: generation and training cannot run at the same time.

Synchronous Can Still Be Fast: Seer

Seer (Moonshot AI / Kimi) represents a different direction: rather than going asynchronous, it attacks the straggler problem inside a synchronous framework. The key observation is that multiple responses sampled for the same prompt often have similar length and patterns. Seer introduces three optimizations: divided rollout (dynamic load balancing), context-aware scheduling (reducing long-tail waiting), and adaptive grouped speculative decoding (speeding up slow requests). Reported results include +74-97% rollout throughput and 75-93% tail-latency reduction, while preserving strict on-policy guarantees without changing the GRPO algorithm. Seer is built on Mooncake disaggregated KV cache + vLLM + Megatron. See arXiv:2511.14617.

Colocated Mode

Still a single GPU group, but inference and training take turns with faster switching. During generation, parameters are reshaped into an inference-friendly format (for example, from an FSDP sharded format into vLLM tensor-parallel format). During training, they are reshaped back. This conversion is faster than reloading models, but it is still fundamentally alternating.

timeline: [gen b0] [convert] [train b0] [convert] [gen b1] [convert] ...

Pros: uses fewer GPUs. Cons: generation and training still do not overlap. Suitable when GPU budget is limited.

Disaggregated Mode (Asynchronous)

This is the standard pattern for large-scale RL training: let both sides run concurrently. Inference and training each own a GPU group, connected by a buffer:

inference GPUs: [gen b0] [gen b1] [gen b2] [gen b3] ...
                  | store     | store
training GPUs:       [train b0] [train b1] [train b2] ...
                  ^ weight sync  ^ weight sync

Inference GPUs continuously generate data and push it into the buffer. Training GPUs continuously pull from the buffer and update the model. Neither side waits on the other.

Mode	GPU Groups	Concurrent?	Typical Use Case
synchronous	1	no	learning, small experiments
colocated	1	no (but faster switching)	medium scale, limited GPU budget
disaggregated	2+	yes	large-scale production training

Disaggregation introduces two core engineering problems: how to synchronize weights, and how to handle data generated by older policies.

Engineering Problem 1: How Do We Sync New Parameters to the Inference GPUs?

Once the training GPUs update parameters, how do inference GPUs receive them? This involves two decisions: how to transmit weights, and whether inference should pause during sync.

How to Transmit?

Transport	Typical Time	Data Volume	When It Fits
NCCL broadcast (full)	100-500 ms	all parameters	most frameworks
NCCL packed transfer (veRL)	~20 ms	all parameters	optimized transfer paths
Direct GPU memory transfer (NeMo-RL)	very low	all parameters	very high-bandwidth fabrics
Sync only LoRA adapters	< 1 ms	~50 MB	best practice
Write to file and read back (PRIME-RL)	slower	all parameters	cross-node workflows

Key insight: with LoRA training you only need to synchronize the adapter weights (~50 MB), which can be under 1 ms. This is a major reason why LoRA + asynchronous training has become a common best practice.

Should We Pause Generation During Sync?

Suppose an inference GPU is generating a 32K-token response, and a new parameter version arrives mid-generation. Common handling patterns include:

1. Do not pause (PipelineRL): swap parameters between tokens (in a ~1-10 ms gap). Elegant, but hardest to implement.
2. Finish current requests, then swap (veRL, AReaL): stop accepting new requests and swap after in-flight work completes. Simple and robust.
3. Abort and resume (SkyRL, SLIME): cancel in-flight requests, then resubmit using already generated tokens as a prefix. Wastes some tokens.
4. Wait for an entire batch boundary (NeMo-RL, Tunix): the most conservative, but also introduces the longest waiting time.

With LoRA sync under 1 ms, the differences among these approaches become less significant.

Engineering Problem 2: Can We Still Use Data Generated by an Older Policy?

In disaggregated mode, inference GPUs may generate data with an older policy version.

For example, training updates at step 100, but syncing takes 20 ms. During that interval, inference may still be generating with step 99. Using data from an older policy to update a newer policy is, strictly speaking, an off-policy issue (covered earlier in the book).

How bad is the bias? It depends on how stale the policy is. Being 1-2 steps behind is often acceptable; being 10 steps behind can be problematic. There are three common approaches:

Approach 1: Version Every Sample and Drop Stale Data

Tag each sample with the policy step that generated it. If it is too old (for example, more than 5 steps behind), drop it. Simple and avoids biased gradients, but wastes compute.

Approach 2: Use a Small Buffer to Bound Staleness

Keep the buffer shallow (for example, at most two batches), so samples are at most two steps stale. This is a physical constraint on staleness and may not require explicit versioning.

Approach 3: Do Not Drop Data, Correct It Mathematically

Reweight samples based on how different the new and old policies are. This is importance sampling correction (often truncated). It avoids wasting data, but can be unstable when policies differ significantly, hence the common practice of truncating weights.

Practical Choices

Many frameworks combine these ideas. For example, PRIME-RL uses all three: a buffer-depth cap as a baseline constraint, explicit version tracking as a safety net, and importance sampling for finer correction.

Approach	Example Frameworks	In Plain Words
drop stale data	PipelineRL, TorchForge	simple but wasteful
bound the buffer	AReaL	a queue-based guarantee
mathematical correction	veRL, ROLL	keep data but downweight
all of the above	PRIME-RL	most conservative

Industry trend: a buffer-depth bound plus optional mathematical correction, balancing efficiency and stability.

References