15.1 GRPO Training Mechanism

In the previous chapter, we studied DPO theory and practice and saw that it can learn directly from fixed preference data: under the same prompt, the chosen answer should become more likely than the rejected answer. Now we return to online training. The model no longer only reads preference pairs labeled by someone else. During training, it generates its own answers, receives feedback, and uses that feedback to update itself.

The entry point of GRPO is multiple answers for the same question. Given one problem, the model generates several answers at once; the reward function scores each answer; then the answers are compared only inside that group. On the surface, this looks like asking the model to try several times. The real problem it solves is:

Without a Critic, how can the model tell whether one answer is better or worse than expected?

An intuitive answer is: compare it with the other answers to the same question. GRPO follows this idea and turns same-question multi-sampling into a trainable policy optimization method.

This section follows one complete GRPO training trajectory. We first look at how same-question multi-sampling creates within-group comparison, then explain the Critic and the baseline, then write down the advantage, probability ratio, and clipped objective, and finally return to handwritten code and TRL's GRPOTrainer.

The diagram expresses the most basic training signal: answer the same question several times, give each answer a score, make answers above the group mean more likely in the future, and make answers below the group mean less likely.

Let's walk through a small numerical example. Suppose the question is:

John has 3 apples and buys 2 more. How many apples does he have now?

The model writes 4 answers to the same question, and the rule-based scorer gives the following scores:

Answer	What the model wrote	Score
1	"3 + 2 = 5, so the answer is 5."	1.5
2	"The answer is 5."	1.0
3	"It should be 6."	0.0
4	"I am not sure, perhaps it is 4."	0.0

The mean of these 4 scores is:

$$\frac{1.5+1.0+0.0+0.0}{4}=0.625$$

So the model interprets this group as follows:

Answer	Comparison with the mean	What to learn next
1	$1.5-0.625=+0.875$	Clearly better than average; generate it more
2	$1.0-0.625=+0.375$	Also better than average; generate it slightly more
3	$0.0-0.625=-0.625$	Worse than average; generate it less
4	$0.0-0.625=-0.625$	Worse than average; generate it less

The quantity "how much higher or lower than the mean" will later be formally called the advantage. In this example, the advantage means whether this answer performs better or worse than the average among the four answers to the same question.

Next, we first define the Critic and then see why "compare with other answers to the same question" can replace the Critic.

In Actor-Critic methods such as PPO, the Actor is the policy model that generates answers, while the Critic is a value evaluator. It does not generate answers directly. Instead, it estimates "given what has been written so far, how much total reward can we expect later?" Written as a formula, this is the value function:

$$V_{\varphi }(s_t)$$

Here $s_t$ is the current state. For a language model, we can roughly read it as "the prompt plus the first few generated tokens"; $\varphi$ denotes the Critic's own parameters. The Critic provides a baseline for policy updates. If the true reward of an answer is higher than the Critic's estimate, the answer is better than expected and its probability should increase. If it is lower than the estimate, its probability should decrease.

The problem is that a Critic is often expensive and difficult to train in LLM settings. It is usually another large model and requires extra memory. It must also predict the final reward from an unfinished text prefix, while the supervision signal often appears only at the end of the answer, so the training noise is large. GRPO tries to do the following: stop training a separate Critic and temporarily use the mean score of several answers under the same prompt as the baseline. This is the core reason why within-group normalization can replace a Critic.

Within-Group Normalization

GRPO training starts from multiple answers for the same question. Given one prompt, the model generates several answers; the reward function scores each answer; then the mean score of this group is used as a temporary baseline. Answers above the mean are encouraged, and answers below the mean are suppressed.

To write this process as a reinforcement learning problem, first map several basic terms:

RL concept	Meaning in a mathematical reasoning model
State $s_t$	The problem prompt plus the reasoning steps already written, namely $(x,y_{<t})$
Action $a_t$	The next generated token, namely $y_t$
Trajectory $\tau$	A full reasoning process and final answer
Reward $R$	Whether the answer is correct and whether the format follows the requirement
Policy $\pi$	The language model being trained

If we follow the PPO route, the model generates an answer, the reward function scores the final answer, and the Critic estimates a baseline $V_{\varphi }(s_t)$. The advantage is roughly:

$$A_t\approx R-V_{\varphi }(s_t)$$

This says: do not only ask whether the reward is high; ask whether it is better than the Critic expected. This is natural in traditional reinforcement learning, but it is heavy in LLM mathematical reasoning. The Critic is also a large model, and it has to predict the final score from unfinished reasoning text.

GRPO keeps the most useful part of PPO, namely ratio + clip, but replaces the Critic baseline with the mean score of a group of answers to the same problem. When the DeepSeekMath paper introduced GRPO, it explicitly said that GRPO "foregoes the critic model" and uses within-group scores to estimate the baseline.

Therefore, the relationship between GRPO and PPO can be summarized as follows:

• PPO asks: is this answer better than the average level estimated by the Critic?
• GRPO asks: is this answer better than the other answers to the same question?
• Both PPO and GRPO still use probability ratios and clipping to avoid overly large updates.

Paper context: GRPO comes from the DeepSeekMath paper DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. It does not discard PPO completely. Instead, it removes the Critic inside the PPO framework and constructs advantages from within-group relative rewards.

One common misunderstanding also needs to be clarified: GRPO is not a new model, and it is not merely the formula for within-group normalization. GRPO is a method for online training of a policy model.

In GRPO, the object being trained is still the language-model policy:

$${\pi }_{\theta }(y\mid x)$$

Here $x$ is the question or prompt, and $y$ is the model-generated answer. GRPO trains by generating several answers for the same prompt, placing them in one group, scoring them, and comparing them: is this answer above the average level inside its group?

If an answer is better than the group average, its advantage $A_i$ is positive and training increases its probability. If it is worse than the group average, the advantage is negative and training decreases its probability. When updating the policy, GRPO still uses PPO-style ratio + clip to keep the new policy from moving too far away from the old policy.

So GRPO can first be understood as:

GRPO = online group sampling + rule/reward scoring + within-group relative advantage + PPO-style clipped update.

Translate the apple example at the beginning into this sentence: generating 4 answers to the same question is online group sampling; assigning scores with answer correctness and format is rule/reward scoring; using differences such as $1.5-0.625$ and $1.0-0.625$ to judge quality is within-group relative advantage; finally, making good answers more likely and bad answers less likely, while adjusting only a small amount each time, is the PPO-style clipped update.

This is the intuition behind "within-group relative advantage": inside the same question, learn more from whoever is better than average, and generate less of whoever is worse than average.

Below is a minimal handwritten GRPO code map. It is not the engineering source code of trl; instead, it lays out the mathematical structure of GRPO so that every formula can be traced back to a few lines in the code.

 13# [A] 组采样：每个 prompt 生成 group_size 个回答，并保留原始 token 边界
 14def sample_groups(model, tokenizer, prompts, group_size=8, max_new_tokens=256):
 15    expanded_prompts = [prompt for prompt in prompts for _ in range(group_size)]
 16    prompt_batch = tokenizer(expanded_prompts, padding=True, return_tensors="pt")
 17    prompt_batch = {
 18        key: value.to(model.device)
 19        for key, value in prompt_batch.items()
 20    }
 21 
 22    pad_token_id = tokenizer.pad_token_id
 23    if pad_token_id is None:
 24        pad_token_id = tokenizer.eos_token_id
 25 
 26    with torch.no_grad():
 27        output_ids = model.generate(
 28            **prompt_batch,
 29            do_sample=True,
 30            temperature=1.0,
 31            max_new_tokens=max_new_tokens,
 32            pad_token_id=pad_token_id,
 33            eos_token_id=tokenizer.eos_token_id,
 34        )
 35 
 36    prompt_width = prompt_batch["input_ids"].size(1)
 37    completion_ids = output_ids[:, prompt_width:]
 38    completion_mask = completion_mask_until_eos(
 39        completion_ids,
 40        tokenizer.eos_token_id,
 41    )
 42    attention_mask = torch.cat(
 43        [prompt_batch["attention_mask"], completion_mask.to(torch.long)],
 44        dim=1,
 45    )
 46 
 47    responses = tokenizer.batch_decode(completion_ids, skip_special_tokens=True)
 48    group_ids = torch.arange(len(prompts), device=model.device).repeat_interleave(
 49        group_size
 50    )
 51    batch = {
 52        "input_ids": output_ids,
 53        "attention_mask": attention_mask,
 54        "completion_mask": completion_mask,
 55    }
 56    return responses, group_ids, batch
        ⋮
 80# [C] 组内优势：用同题目的回答均值替代 Critic 基线
 81def group_advantages(rewards, group_size=8, eps=1e-8):
 82    grouped_rewards = rewards.view(-1, group_size)
 83    group_mean = grouped_rewards.mean(dim=1, keepdim=True)
 84    group_std = grouped_rewards.std(dim=1, keepdim=True, correction=1)
 85 
 86    advantages = (grouped_rewards - group_mean) / (group_std + eps)
 87    advantages = torch.where(
 88        group_std < eps,
 89        torch.zeros_like(advantages),
 90        advantages,
 91    )
 92    return advantages.reshape(-1)
 93 
 94 
 95# [D] 只返回回答部分的逐 token log probability，形状为 [B, T]
 96def per_token_logprobs(model, input_ids, attention_mask, completion_length):
 97    outputs = model(input_ids=input_ids, attention_mask=attention_mask)
 98    logits = outputs.logits[:, :-1, :]
 99    target_ids = input_ids[:, 1:]
100 
101    all_token_logprobs = logits.log_softmax(dim=-1)
102    picked_logprobs = all_token_logprobs.gather(
103        dim=-1,
104        index=target_ids.unsqueeze(-1),
105    ).squeeze(-1)
106    return picked_logprobs[:, -completion_length:]
107 
108 
109def masked_sequence_mean(values, mask):
110    """每段回答先按有效 token 求平均，防止长回答获得更大权重。"""
111    mask = mask.to(values.dtype)
112    token_count = mask.sum(dim=-1).clamp_min(1.0)
113    return (values * mask).sum(dim=-1) / token_count
        ⋮
116# [E-F] 原始 GRPO：逐 token ratio、clip、KL，再按回答长度归一化
117def grpo_objective_from_logprobs(
118    new_logprobs,
119    old_logprobs,
120    ref_logprobs,
121    completion_mask,
122    advantages,
123    clip_eps=0.2,
124    kl_coef=0.04,
125):
126    token_ratio = torch.exp(new_logprobs - old_logprobs)
127    token_advantages = advantages.unsqueeze(-1)
128    unclipped = token_ratio * token_advantages
129    clipped_ratio = torch.clamp(token_ratio, 1.0 - clip_eps, 1.0 + clip_eps)
130    clipped = clipped_ratio * token_advantages
131 
132    # DeepSeekMath 式 (4)：D_KL(policy || ref) 的逐 token 无偏正值估计
133    log_ratio_ref = ref_logprobs - new_logprobs
134    per_token_kl = torch.exp(log_ratio_ref) - log_ratio_ref - 1.0
135 
136    per_token_objective = torch.minimum(unclipped, clipped) - kl_coef * per_token_kl
137    per_response_objective = masked_sequence_mean(
138        per_token_objective,
139        completion_mask,
140    )
141    loss = -per_response_objective.mean()
142 
143    policy_loss = -masked_sequence_mean(
144        torch.minimum(unclipped, clipped),
145        completion_mask,
146    ).mean()
147    approx_kl = masked_sequence_mean(per_token_kl, completion_mask).mean()
148    metrics = {
149        "loss": loss.detach(),
150        "policy_loss": policy_loss.detach(),
151        "approx_kl": approx_kl.detach(),
152        "mean_ratio": masked_sequence_mean(token_ratio, completion_mask).mean().detach(),
153    }
154    return loss, metrics
155 
156 
157def grpo_loss(
158    policy_model,
159    ref_model,
160    batch,
161    advantages,
162    old_logprobs=None,
163    clip_eps=0.2,
164    kl_coef=0.04,
165):
166    completion_length = batch["completion_mask"].size(1)
167    new_logprobs = per_token_logprobs(
168        policy_model,
169        batch["input_ids"],
170        batch["attention_mask"],
171        completion_length,
172    )
173 
174    # 单次更新时 old policy 就是采样 policy；detach 保留 ratio 的梯度。
175    if old_logprobs is None:
176        old_logprobs = new_logprobs.detach()
177 
178    with torch.no_grad():
179        ref_logprobs = per_token_logprobs(
180            ref_model,
181            batch["input_ids"],
182            batch["attention_mask"],
183            completion_length,
184        )
185 
186    return grpo_objective_from_logprobs(
187        new_logprobs,
188        old_logprobs,
189        ref_logprobs,
190        batch["completion_mask"],
191        advantages,
192        clip_eps,
193        kl_coef,
194    )
        ⋮
197# [G] 训练步骤：采样、打分、组内归一化、再反向传播
198def train_step(
199    policy_model,
200    ref_model,
201    optimizer,
202    tokenizer,
203    prompts,
204    ground_truths,
205    group_size=8,
206):
207    responses, _, batch = sample_groups(
208        policy_model,
209        tokenizer,
210        prompts,
211        group_size,
212    )
213    rewards = score_responses(
214        responses,
215        ground_truths,
216        group_size,
217        policy_model.device,
218    )
219    advantages = group_advantages(rewards, group_size)
220 
221    loss, metrics = grpo_loss(policy_model, ref_model, batch, advantages)
222    optimizer.zero_grad()
223    loss.backward()
224    optimizer.step()
225    return metrics
226 
227 
228# [H] GRPO 训练循环：每轮都在线生成新回答
229def train_grpo(policy_model, ref_model, optimizer, tokenizer, dataloader):
230    ref_model.eval()
231    for prompts, ground_truths in dataloader:
232        metrics = train_step(
233            policy_model,
234            ref_model,
235            optimizer,
236            tokenizer,
237            prompts,
238            ground_truths,
239        )
240        print(
241            "loss=",
242            float(metrics["loss"]),
243            "kl=",
244            float(metrics["approx_kl"]),
245        )

This code can be divided into eight blocks:

Marker	Code section	What the following text explains
[A]	sample_groups	Why each prompt generates multiple answers
[B]	rule_reward / score_responses	Where rewards come from and why math problems do not require an RM
[C]	group_advantages	How the within-group mean replaces the Critic baseline
[D]	per_token_logprobs	How to retain $\log {\pi }_{\theta }$ for every response token
[E]	grpo_objective_from_logprobs	Token-wise ratio, clip, and policy updates
[F]	per_token_kl	Why KL must also be computed token by token
[G]	train_step	How sampling, scoring, advantages, loss, and backprop connect
[H]	train_grpo	Why GRPO is online training and generates fresh answers every round

From PPO to GRPO: Which Lines Actually Change?

If we did not switch to GRPO and kept training in the PPO / RLHF style, the code intuition would usually look like this:

# PPO / RLHF: generate online, then ask the Critic for token-wise advantages
responses, completion_mask = policy_old.generate(prompts)
old_per_token_logps = per_token_logprobs(policy_old, prompts, responses).detach()

rewards = reward_model(prompts, responses)
advantages = critic_based_advantages(prompts, responses, rewards)

new_per_token_logps = per_token_logprobs(policy, prompts, responses)
token_ratio = torch.exp(new_per_token_logps - old_per_token_logps)
per_token_objective = torch.min(
    token_ratio * advantages,
    torch.clamp(token_ratio, 1 - clip_eps, 1 + clip_eps) * advantages,
)
ppo_loss = -masked_sequence_mean(per_token_objective, completion_mask).mean()

The critic here is the value model described earlier. Its job is not to generate answers, but to estimate a baseline: roughly how much score should this prompt and current answer prefix receive? PPO then uses rewards - values to get the advantage and decide whether an answer is "better than expected" or "worse than expected".

GRPO changes only a concentrated part: keep online generation, probability ratios, and clipping, but stop training the Critic; compute the advantage from a group of answers to the same prompt instead.

# GRPO: generate G answers for each prompt, then compare inside the group
responses, completion_mask = generate_many(policy_old, prompts, num_generations=G)
old_per_token_logps = per_token_logprobs(policy_old, prompts, responses).detach()

rewards = reward_fn(prompts, responses)
rewards_by_group = rewards.view(batch_size, G)

group_mean = rewards_by_group.mean(dim=1, keepdim=True)
group_std = rewards_by_group.std(dim=1, keepdim=True)
response_advantages = (
    (rewards_by_group - group_mean) / (group_std + 1e-4)
).view(-1)

new_per_token_logps = per_token_logprobs(policy, prompts, responses)
token_ratio = torch.exp(new_per_token_logps - old_per_token_logps)
per_token_objective = torch.min(
    token_ratio * response_advantages[:, None],
    torch.clamp(token_ratio, 1 - clip_eps, 1 + clip_eps)
    * response_advantages[:, None],
)
grpo_loss = -masked_sequence_mean(per_token_objective, completion_mask).mean()

Outcome supervision assigns one advantage to a whole response, so its valid tokens share the same ${\hat{A}}_i$. The policy ratio, clipping, and KL remain token-wise. The loss first averages over the valid tokens in each response and then averages over responses.

If we isolate the lines that truly change, we get:

  responses = policy_old.generate(prompts)
  rewards = reward_model_or_rule(prompts, responses)
- values = critic(prompts, responses)
- advantages = rewards - values

+ rewards_by_group = rewards.view(batch_size, G)
+ group_mean = rewards_by_group.mean(dim=1, keepdim=True)
+ group_std = rewards_by_group.std(dim=1, keepdim=True)
+ advantages = ((rewards_by_group - group_mean) / (group_std + 1e-4)).view(-1)

  loss = ppo_style_clipped_loss(logps_new, logps_old, advantages)

So GRPO is not "deleting all of PPO", and it is not "only keeping a within-group normalization formula". More accurately, GRPO replaces PPO's Critic baseline with a within-group mean baseline: previously we asked, "is this answer better than the Critic expected?" Now we ask, "is this answer better than the other answers to the same question?"

Original-paper acceptance criterion

Equation (3) of DeepSeekMath contains a token index $t$ on the policy ratio and a separate $1/|o_i|$ average for every response:

$$r_{i,t}(\theta )=\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}.$$

Consequently, the original GRPO computes $\exp ({\ell }_{i,t}^{\mathrm{new}}-{\ell }_{i,t}^{\mathrm{old}})$ separately at every token. Exponentiating the sum instead produces ${\prod }_t{r}_{i,t}$, introduces response-length dependence, and clips the whole response only once. Equation (4) likewise defines the KL estimate token by token. The formulas and code below use those original definitions as the acceptance criterion. DeepSeekMath equations

TRL implementation comparison

The current TRL source preserves this token dimension. When checking the Hugging Face TRL main branch on 2026-08-28, the following correspondences can be seen in GRPOTrainer:

1. GRPOTrainer accepts reward_funcs in its initialization arguments. This can be a reward model or an ordinary Python function. In other words, math tasks can be scored directly with rule functions and do not necessarily require training an RM first.
2. self.num_generations = args.num_generations corresponds to $G$ in the formula, namely how many answers to generate for each prompt.
3. The source reshapes rewards into (-1, num_generations), computes mean_grouped_rewards and within-group std_rewards, and then obtains advantages = rewards - mean_grouped_rewards, dividing by the standard deviation when needed.
4. _get_per_token_logps_and_entropies returns one log probability per response token. The loss computes coef_1 = exp(log_ratio), clips it, and takes the min at every token position.
5. loss_type="grpo" averages valid tokens inside each response before averaging the batch, matching $\frac{1}{|o_i|}{\sum }_t$. loss_type="bnpo" instead averages all valid batch tokens globally and therefore gives longer responses more weight. Current TRL loss implementation

Current TRL also supports sequence-level importance sampling and DAPO/BNPO/DR-GRPO variants, and its defaults evolve over time. A paper-faithful configuration must therefore explicitly select importance_sampling_level="token", loss_type="grpo", num_iterations=1, beta=0.04, and disable the later KL bias correction. Comparing GRPOConfig in TRL 0.24 with GRPOConfig on the current main branch shows both the changed defaults and the later addition of use_bias_correction_kl, which is enabled by default.

Compared with PPOTrainer, the difference is clearer: PPOTrainer needs a reward_model and a value_model, and uses the value_model to produce advantage estimates; GRPOTrainer does not need a separate value_model. It directly turns within-group relative scores from same-question multi-sampling into advantages.

Before Reading the Formula: What Does One GRPO Sample Group Look Like?

A GRPO training sample is not "one prompt with one answer", but one prompt with a group of answers. Suppose a batch contains several problems. Let $j$ denote the problem index and $i$ denote the answer index under that problem:

$$x_j⟶\{y_{j,1},y_{j,2},\dots ,y_{j,G}\}$$

The symbols mean:

• $x_j$: the $j$-th prompt, namely a problem or question.
• $G$: the group size, meaning how many answers each prompt generates. num_generations=8 in code means $G=8$.
• $y_{j,i}$: the $i$-th answer generated under the $j$-th prompt.
• ${\pi }_{\text{old}}$: the old policy used to generate this batch of answers. It is responsible for sampling data.
• ${\pi }_{\theta }$: the new policy being updated. It is responsible for learning so that good answers become more likely.

The sampling process can be written as:

$$y_{j,i}\sim {\pi }_{\text{old}}(\cdot \mid x_j),i=1,\dots ,G$$

The symbol $\sim$ means "sample from a distribution". The dot $\cdot$ in ${\pi }_{\text{old}}(\cdot \mid x_j)$ means "the position of all possible answers": given prompt $x_j$, the old policy assigns probabilities to all possible answers, and we draw $G$ answers from that distribution.

In the code, this corresponds to [A] group sampling:

 13# [A] 组采样：每个 prompt 生成 group_size 个回答，并保留原始 token 边界
 14def sample_groups(model, tokenizer, prompts, group_size=8, max_new_tokens=256):
 15    expanded_prompts = [prompt for prompt in prompts for _ in range(group_size)]
 16    prompt_batch = tokenizer(expanded_prompts, padding=True, return_tensors="pt")
 17    prompt_batch = {
 18        key: value.to(model.device)
 19        for key, value in prompt_batch.items()
 20    }
 21 
 22    pad_token_id = tokenizer.pad_token_id
 23    if pad_token_id is None:
 24        pad_token_id = tokenizer.eos_token_id
 25 
 26    with torch.no_grad():
 27        output_ids = model.generate(
 28            **prompt_batch,
 29            do_sample=True,
 30            temperature=1.0,
 31            max_new_tokens=max_new_tokens,
 32            pad_token_id=pad_token_id,
 33            eos_token_id=tokenizer.eos_token_id,
 34        )
 35 
 36    prompt_width = prompt_batch["input_ids"].size(1)
 37    completion_ids = output_ids[:, prompt_width:]
 38    completion_mask = completion_mask_until_eos(
 39        completion_ids,
 40        tokenizer.eos_token_id,
 41    )
 42    attention_mask = torch.cat(
 43        [prompt_batch["attention_mask"], completion_mask.to(torch.long)],
 44        dim=1,
 45    )
 46 
 47    responses = tokenizer.batch_decode(completion_ids, skip_special_tokens=True)
 48    group_ids = torch.arange(len(prompts), device=model.device).repeat_interleave(
 49        group_size
 50    )
 51    batch = {
 52        "input_ids": output_ids,
 53        "attention_mask": attention_mask,
 54        "completion_mask": completion_mask,
 55    }
 56    return responses, group_ids, batch

After each answer is generated, it receives a reward:

$$r_{j,i}=R(x_j,y_{j,i})$$

Here $R$ is the reward function, and $r_{j,i}$ is a scalar. For math problems, $R$ can be simple: give points for a correct answer and for a proper format. The key to GRPO is not that "the reward function must be complex", but that multiple answers under the same question are compared together.

GRPO Training Experiment

Experimental Setup: GSM8K + Rule Rewards

GSM8K is a dataset of 8,500 elementary-school math word problems, each with a clear numerical answer. This is exactly a setting with an "objectively correct answer": no RM is needed, and rules can directly judge whether the answer is correct.

• Correct answer: $+1.0$ point
• Proper format, with clear reasoning steps: $+0.5$ point
• Wrong answer: $0$ points

# 1. Rule-based reward function (no RM needed!)
import re

def rule_based_reward(prompt: str, response: str, ground_truth: str) -> float:
    reward = 0.0
    # Format score: check \boxed{...}
    if re.search(r'\\boxed\{[^}]+\}', response):
        reward += 0.5
    # Answer score: extract final answer and compare
    answer_match = re.search(r'\\boxed\{([^}]+)\}', response)
    if answer_match:
        model_answer = answer_match.group(1).strip()
        try:
            if abs(float(model_answer) - float(ground_truth)) < 0.01:
                reward += 1.0
        except ValueError:
            if model_answer == ground_truth:
                reward += 1.0
    return reward

# Test
prompt = "Janet's egg box holds 16 eggs each day. She eats 3 every morning and uses 4 to bake muffins in the afternoon. How many eggs can she sell each week?"
good = "First compute the eggs left each day: 16 - 3 - 4 = 9.\nThere are 7 days in a week, so she can sell: 9 * 7 = 63.\n\\boxed{63}"
bad = "I think she can sell about 50 eggs. \\boxed{50}"
print(rule_based_reward(prompt, good, '63'))  # 1.5
print(rule_based_reward(prompt, bad, '63'))   # 0.5

Notice the key difference: we do not need to train any RM; the rule is the judge. Math problems have standard answers, so direct comparison is enough. This kind of "verifiable reward" is exactly the core idea of RLVR, which will be discussed in depth in Section 15.3.

In the handwritten code map, the reward function corresponds to [B]. It receives only the answer and the ground truth answer and returns a scalar reward:

 59# [B] 规则奖励：数学答案正确、格式规范就给分
 60def rule_reward(response, ground_truth):
 61    reward = 0.0
 62    boxed = re.search(r"\\boxed\{([^}]+)\}", response)
 63 
 64    if boxed:
 65        reward += 0.5
 66        if boxed.group(1).strip() == str(ground_truth).strip():
 67            reward += 1.0
 68 
 69    return reward
 70 
 71 
 72def score_responses(responses, ground_truths, group_size=8, device="cpu"):
 73    rewards = []
 74    for i, response in enumerate(responses):
 75        prompt_id = i // group_size
 76        rewards.append(rule_reward(response, ground_truths[prompt_id]))
 77    return torch.tensor(rewards, dtype=torch.float32, device=device)

Running GRPO Training

We use the GRPO implementation provided by the trl library. Compared with PPO, GRPO does not need a Critic model:

# 2. GRPO training code (simplified sketch)
from trl import GRPOTrainer, GRPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

paper_grpo_options = dict(
    beta=0.04,                         # paper KL coefficient
    epsilon=0.2,
    num_iterations=1,                  # one update per rollout batch
    scale_rewards="group",
    importance_sampling_level="token",
    loss_type="grpo",                  # normalize each response by its length
)
# The pinned TRL 0.24 has no such option; only newer releases need this override.
if "use_bias_correction_kl" in GRPOConfig.__dataclass_fields__:
    paper_grpo_options["use_bias_correction_kl"] = False

config = GRPOConfig(
    output_dir="./grpo_gsm8k",
    num_generations=8,                 # teaching scale; the paper uses 64
    per_device_train_batch_size=8,     # global batch must divide by group size
    max_completion_length=1024,
    learning_rate=1e-6,                # paper experiment setting
    num_train_epochs=1,
    **paper_grpo_options,
)

gsm8k = load_dataset("openai/gsm8k", "main")
trainer = GRPOTrainer(
    model=model,
    args=config,
    train_dataset=gsm8k["train"],
    reward_funcs=[rule_based_reward],  # pass the rule reward function directly
    processing_class=tokenizer,
)

trainer.train()  # start training: no Critic, no RM
trainer.save_model("./grpo_gsm8k/final_model")

This separates the algorithm from the teaching-scale hardware choice. The DeepSeekMath experimental setup sampled 64 responses per question with a batch size of 1024, a learning rate of $10^{-6}$, a KL coefficient of 0.04, and one policy update after each exploration stage; this example uses eight responses to reduce memory. The remaining options are explicit so newer TRL defaults cannot silently switch the objective to DAPO, BNPO, sequence-level importance sampling, or a later bias-corrected KL variant. The repository-pinned TRL 0.24 configuration predates use_bias_correction_kl; the current main-branch configuration adds the field and enables it by default, so the example disables it only when the field exists.

If we unfold the most important internal training step of GRPOTrainer, it is "sample by group, score, compute advantages, and update the policy":

197# [G] 训练步骤：采样、打分、组内归一化、再反向传播
198def train_step(
199    policy_model,
200    ref_model,
201    optimizer,
202    tokenizer,
203    prompts,
204    ground_truths,
205    group_size=8,
206):
207    responses, _, batch = sample_groups(
208        policy_model,
209        tokenizer,
210        prompts,
211        group_size,
212    )
213    rewards = score_responses(
214        responses,
215        ground_truths,
216        group_size,
217        policy_model.device,
218    )
219    advantages = group_advantages(rewards, group_size)
220 
221    loss, metrics = grpo_loss(policy_model, ref_model, batch, advantages)
222    optimizer.zero_grad()
223    loss.backward()
224    optimizer.step()
225    return metrics
226 
227 
228# [H] GRPO 训练循环：每轮都在线生成新回答
229def train_grpo(policy_model, ref_model, optimizer, tokenizer, dataloader):
230    ref_model.eval()
231    for prompts, ground_truths in dataloader:
232        metrics = train_step(
233            policy_model,
234            ref_model,
235            optimizer,
236            tokenizer,
237            prompts,
238            ground_truths,
239        )
240        print(
241            "loss=",
242            float(metrics["loss"]),
243            "kl=",
244            float(metrics["approx_kl"]),
245        )

Before and After Training: How Reasoning Steps Change

The most exciting observation in GRPO training is the change in the model's reasoning style.

Before training (directly guessing the answer):

Problem: John has 15 apples, gives 3 to Mary, and then gives 5 to Kevin. How many are left?
Answer: I think there are 7 left. \boxed{7}

After training (showing the reasoning process):

Problem: John has 15 apples, gives 3 to Mary, and then gives 5 to Kevin. How many are left?
Answer:
Let me calculate step by step:
- John starts with 15 apples.
- He gives 3 to Mary: 15 - 3 = 12.
- He gives 5 more to Kevin: 12 - 5 = 7.
- Therefore 7 apples are left.
\boxed{7}

The model changes from "guess the answer directly" to "write the equation first and then compute". We did not explicitly teach this. The model discovers it during GRPO training. Because showing reasoning steps improves answer accuracy and earns higher rule rewards, the optimization pressure of GRPO naturally selects this path.

Why Within-Group Normalization Is Necessary

We have seen the practical effect of "removing the Critic": memory decreases by 30-40%, and reasoning steps change from "guessing" to "writing equations". But one core question remains unanswered: why can within-group normalization replace the work of a Critic?

Three Problems with the PPO Critic

Before answering "why it can replace it", first clarify "why we want to replace it". PPO's Critic faces three serious problems in LLM training:

1. It consumes memory. The Critic is the same scale as the Actor. PPO needs to hold four models at the same time: Actor + Critic + Reference + RM.

2. It is unstable to train. The value function $V(s)$ needs to predict the final score from "partially generated text", but LLM sequences are long (500+ tokens), and the supervision signal appears only at the end, producing very high variance.

3. It complicates engineering. Four models have separate optimizers, learning rates, and gradient clipping configurations, making hyperparameter tuning much harder.

Recall the baseline analysis in Chapter 6 and the advantage function in Chapter 7: the core role of the Critic is to provide a baseline that reduces variance. If we can obtain a baseline without training a separate network, the Critic can retire.

The Core Idea of GRPO

GRPO constructs its baseline from the other answers to the same problem. For problem $x_j$, first sample $G$ answers and obtain $G$ rewards:

$$\{r_{j,1},r_{j,2},\dots ,r_{j,G}\}$$

Then compute the mean reward of this group:

$${\bar{r}}_j=\frac{1}{G}\sum _{i=1}^G{r}_{j,i}$$

Here:

• ${\bar{r}}_j$: the mean reward of the answer group for the $j$-th prompt.
• $\frac{1}{G}{\sum }_{i=1}^G$: add the rewards of the $G$ answers and divide by $G$.
• $r_{j,i}$: the reward of the $i$-th answer under the $j$-th prompt.

Next compute the standard deviation of this group:

$$s_j=\mathrm{std}(r_{j,1},\dots ,r_{j,G}).$$

DeepSeekMath writes std without specifying whether the denominator is $G$ or $G-1$. The example follows the default definition of PyTorch torch.std and uses Bessel's correction. With a fixed group size, this changes the overall advantage scale but not the sign or ordering of answers inside a group.

Here:

• $s_j$: the standard deviation of the $j$-th reward group, representing how different the answer scores are.

Finally, the within-group advantage of the $i$-th answer is:

$${\hat{A}}_{j,i}=\frac{r_{j,i}-{\bar{r}}_j}{s_j+ϵ}$$

This is the core formula of GRPO. It reads plainly:

• $r_{j,i}-{\bar{r}}_j$: how much better this answer is than the same-question average.
• Dividing by $s_j+ϵ$: brings reward scales from different questions into a similar range.
• $ϵ$: a very small number that prevents division by zero when the standard deviation is 0.
• ${\hat{A}}_{j,i}>0$: this answer is better than its group average, so its probability should increase.
• ${\hat{A}}_{j,i}<0$: this answer is worse than its group average, so its probability should decrease.
• ${\hat{A}}_{j,i}\approx 0$: this answer is close to average, so it does not need a strong update.

If every answer in the same group receives the same reward, $s_j$ approaches 0, and the code sets the advantage to 0. This means the problem currently offers no learnable difference: everyone is correct, or everyone is wrong, and the model does not know which answer to prefer.

This formula does something very similar to the Critic: subtracting the mean asks "how much better than average is this?" The difference is that the Critic uses a separate neural network to predict the baseline $V(s)$, while GRPO directly uses the actual mean score ${\bar{r}}_j$ of the answer group as the baseline.

In the code, this corresponds to [C] within-group advantage:

 80# [C] 组内优势：用同题目的回答均值替代 Critic 基线
 81def group_advantages(rewards, group_size=8, eps=1e-8):
 82    grouped_rewards = rewards.view(-1, group_size)
 83    group_mean = grouped_rewards.mean(dim=1, keepdim=True)
 84    group_std = grouped_rewards.std(dim=1, keepdim=True, correction=1)
 85 
 86    advantages = (grouped_rewards - group_mean) / (group_std + eps)
 87    advantages = torch.where(
 88        group_std < eps,
 89        torch.zeros_like(advantages),
 90        advantages,
 91    )
 92    return advantages.reshape(-1)

The code correspondence is:

• grouped_rewards = rewards.view(-1, group_size): reshape a one-dimensional reward list into "one prompt per row, $G$ answers per row".
• group_mean = grouped_rewards.mean(dim=1, keepdim=True): compute ${\bar{r}}_j$ for each prompt.
• group_std = grouped_rewards.std(dim=1, keepdim=True): compute $s_j$ for each prompt.
• advantages = (grouped_rewards - group_mean) / (group_std + eps): implement ${\hat{A}}_{j,i}$.
• torch.where(group_std < eps, 0, advantages): if one group has no differences, give that group no training signal.

Within-group normalization works for three reasons:

Difficulty normalization. Different problems have different difficulty levels. For easy problems, all answers may be correct and the reward mean is high. For hard problems, most answers may be wrong and the reward mean is low. If absolute reward is used, easy-problem answers receive stronger gradients, and the model spends most of its effort on easy problems. Within-group normalization removes this bias by asking only "who is better inside this problem", independent of the problem's absolute difficulty.

Relative comparison is more stable. Human preferences are also naturally comparative ("A is better than B"), not absolute ("A gets 87 points"). GRPO's within-group comparison matches this style of judgment.

Variance is lower. Answers in the same group share the same prompt. The only difference is the randomness of model generation. This controlled-variable comparison is more stable than absolute scoring across unrelated samples.

In one sentence: GRPO = PPO's clipping mechanism + replacing the Critic with within-group ranking.

"PPO's clipping mechanism" is still ratio, clamp, and min in the code; the difference is that advantages now comes from within-group comparison.

Token-wise ratio and clipping

For token $t$ in response $i$, define the conditional probability and its logarithm as

$$p_{j,i,t}^{\theta }={\pi }_{\theta }(y_{j,i,t}\mid x_j,y_{j,i,<t}),{\ell }_{j,i,t}^{\theta }=\log p_{j,i,t}^{\theta }.$$

Probabilities are small, and multiplying many of them can underflow. Logarithms turn the product for a complete response into a sum. However, original GRPO must preserve the $[B,T]$ token dimension because the ratio and clipping are applied before any token reduction:

$${\rho }_{j,i,t}(\theta )=\frac{p_{j,i,t}^{\theta }}{p_{j,i,t}^{\mathrm{old}}}=\exp \left({\ell }_{j,i,t}^{\theta }-{\ell }_{j,i,t}^{\mathrm{old}}\right).$$

The paper's clipped objective is

$${\mathcal{J}}_{\text{clip}}(\theta )={\mathbb{E}}_{x_j}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{T_{j,i}}\sum _{t=1}^{T_{j,i}}\min \left({\rho }_{j,i,t}{\hat{A}}_{j,i},\mathrm{clip}({\rho }_{j,i,t},1-ϵ,1+ϵ){\hat{A}}_{j,i}\right)\right].$$

For token ratios $[1.1,0.9,1.5]$ and $ϵ=0.2$, original GRPO clips them separately to $[1.1,0.9,1.2]$. Summing the log-ratio first instead produces one response ratio $1.1\times 0.9\times 1.5=1.485$ and clips the whole response once. That loses local control and introduces a strong length effect.

116# [E-F] 原始 GRPO：逐 token ratio、clip、KL，再按回答长度归一化
117def grpo_objective_from_logprobs(
118    new_logprobs,
119    old_logprobs,
120    ref_logprobs,
121    completion_mask,
122    advantages,
123    clip_eps=0.2,
124    kl_coef=0.04,
125):
126    token_ratio = torch.exp(new_logprobs - old_logprobs)
127    token_advantages = advantages.unsqueeze(-1)
128    unclipped = token_ratio * token_advantages
129    clipped_ratio = torch.clamp(token_ratio, 1.0 - clip_eps, 1.0 + clip_eps)
130    clipped = clipped_ratio * token_advantages
131 
132    # DeepSeekMath 式 (4)：D_KL(policy || ref) 的逐 token 无偏正值估计
133    log_ratio_ref = ref_logprobs - new_logprobs
134    per_token_kl = torch.exp(log_ratio_ref) - log_ratio_ref - 1.0
135 
136    per_token_objective = torch.minimum(unclipped, clipped) - kl_coef * per_token_kl
137    per_response_objective = masked_sequence_mean(
138        per_token_objective,
139        completion_mask,
140    )
141    loss = -per_response_objective.mean()
142 
143    policy_loss = -masked_sequence_mean(
144        torch.minimum(unclipped, clipped),
145        completion_mask,
146    ).mean()
147    approx_kl = masked_sequence_mean(per_token_kl, completion_mask).mean()
148    metrics = {
149        "loss": loss.detach(),
150        "policy_loss": policy_loss.detach(),
151        "approx_kl": approx_kl.detach(),
152        "mean_ratio": masked_sequence_mean(token_ratio, completion_mask).mean().detach(),
153    }
154    return loss, metrics

In the corrected code, new_logprobs and old_logprobs both have shape $[B,T]$. The expressions token_ratio and minimum(unclipped, clipped) operate element by element. Then masked_sequence_mean averages the valid tokens inside each response, and the outer .mean() gives each response equal weight. A global (loss * mask).sum() / mask.sum() is TRL's BNPO reduction, not the original GRPO reduction, because it gives longer responses more weight.

Token-wise KL penalty

Equation (4) of DeepSeekMath also defines KL per token:

$${\hat{D}}_{j,i,t}=\exp ({\Delta }_{j,i,t})-{\Delta }_{j,i,t}-1,{\Delta }_{j,i,t}={\ell }_{j,i,t}^{\mathrm{ref}}-{\ell }_{j,i,t}^{\theta }.$$

This quantity is non-negative and becomes zero when Policy and Reference agree. Combining clipping and KL gives

$${\mathcal{J}}_{\text{GRPO}}(\theta )=\frac{1}{G}\sum _{i=1}^G\frac{1}{T_{j,i}}\sum _{t=1}^{T_{j,i}}\left[\min \left({\rho }_{j,i,t}{\hat{A}}_{j,i},\mathrm{clip}({\rho }_{j,i,t},1-ϵ,1+ϵ){\hat{A}}_{j,i}\right)-\beta {\hat{D}}_{j,i,t}\right].$$

The training loss is the negative expectation of this objective. DeepSeekMath used $\beta =0.04$. DeepSeekMath Equations (3)–(4) and experimental setup

126    token_ratio = torch.exp(new_logprobs - old_logprobs)
127    token_advantages = advantages.unsqueeze(-1)
128    unclipped = token_ratio * token_advantages
129    clipped_ratio = torch.clamp(token_ratio, 1.0 - clip_eps, 1.0 + clip_eps)
130    clipped = clipped_ratio * token_advantages
131 
132    # DeepSeekMath 式 (4)：D_KL(policy || ref) 的逐 token 无偏正值估计
133    log_ratio_ref = ref_logprobs - new_logprobs
134    per_token_kl = torch.exp(log_ratio_ref) - log_ratio_ref - 1.0
135 
136    per_token_objective = torch.minimum(unclipped, clipped) - kl_coef * per_token_kl
137    per_response_objective = masked_sequence_mean(
138        per_token_objective,
139        completion_mask,
140    )
141    loss = -per_response_objective.mean()
142 
143    policy_loss = -masked_sequence_mean(
144        torch.minimum(unclipped, clipped),
145        completion_mask,
146    ).mean()
147    approx_kl = masked_sequence_mean(per_token_kl, completion_mask).mean()
148    metrics = {
149        "loss": loss.detach(),
150        "policy_loss": policy_loss.detach(),
151        "approx_kl": approx_kl.detach(),
152        "mean_ratio": masked_sequence_mean(token_ratio, completion_mask).mean().detach(),
153    }
154    return loss, metrics

completion_mask excludes the prompt and padding after EOS. The implementation first combines the token-wise clipped objective and KL, then performs $\frac{1}{T_{j,i}}{\sum }_t$, matching the paper's order of operations.

Putting all steps together, one GRPO training iteration is:

1. Sample $G$ answers for each prompt.
2. Score every answer with rules or a reward function.
3. Compute ${\bar{r}}_j$, $s_j$, and ${\hat{A}}_{j,i}$ inside each prompt group.
4. Keep response-token log probabilities and compute ${\rho }_{j,i,t}(\theta )$ token by token.
5. Apply PPO-style clipping and the Reference KL penalty at each token.
6. Average valid tokens inside each response, then average the responses.
7. Backpropagate and update only the Policy.

Experimental Comparison and Parameter Tuning

Memory Usage Comparison

Model size	PPO memory (4 models)	GRPO memory (2 models)	Savings
1.5B	~24 GB	~14 GB	~42%
7B	~80 GB	~48 GB	~40%
14B	~160 GB	~96 GB	~40%
70B	~640 GB	~384 GB	~40%

GRPO removes the Critic, which is the same scale as the Actor, and also removes the RM, often reducing memory usage by 30-40%. In real engineering, this means a training job that used to require 8 A100 GPUs may now fit on 5.

Evolution of Within-Group Variance

The core innovation of GRPO is replacing the Critic with within-group normalization. Early in training, the 8 answers to the same problem differ greatly in quality, so variance is high. As training progresses, answer quality within the group becomes more consistent, variance decreases, and most answers become correct.

Early training (Episode 10):
  8 answers to the problem "15 - 3 - 5 = ?": [3, 7, 12, 7, 15, 7, 8, 10]
  Within-group variance: high (answers vary widely)
  Normalized advantages: [-1.2, +0.1, +0.8, +0.1, +1.5, +0.1, -0.3, +0.6]

Middle training (Episode 100):
  8 answers to the same problem: [7, 7, 7, 8, 7, 7, 7, 7]
  Within-group variance: low (most answers are correct)
  Normalized advantages: [0, 0, 0, -0.5, 0, 0, 0, 0]

Late training (Episode 300):
  8 answers to the same problem: [7, 7, 7, 7, 7, 7, 7, 7]
  Within-group variance: near zero (all answers are correct)
  Normalized advantages: all near zero -> no gradient signal

When within-group variance falls to zero, all advantages become zero, and there is no gradient signal. The model has "graduated" on this problem. This is exactly what we want: the training signal naturally shifts toward problems that the model has not yet mastered.

Choosing k

The group size $k$ is the most important hyperparameter in GRPO. It directly affects the quality of within-group normalization:

k value	Sampling cost	Normalization quality	Suitable setting
2	Low; only 2 samples per problem	Poor; mean and standard deviation are unstable	Quick validation
4	Medium	Acceptable	Limited resources
8	Fairly high	Good	Default recommendation
16	High	Very good; statistics are more stable	Pushing the ceiling
64	Very high	Excellent	Large-scale training

# Simple implementation of GRPO within-group normalization
import numpy as np

def grpo_group_normalize(rewards: list[float]) -> list[float]:
    rewards = np.array(rewards, dtype=float)
    mean, std = rewards.mean(), rewards.std()
    if std < 1e-8:
        return np.zeros_like(rewards)
    return (rewards - mean) / std

# Example: rewards from 8 answers
rewards = [1.5, 0.0, 1.5, 0.0, 1.0, 1.5, 0.5, 1.5]
advantages = grpo_group_normalize(rewards)
# Normalized advantages: [ 0.89 -1.48  0.89 -1.48  0.10  0.89 -0.69  0.89]
# Mean: 0.9375, standard deviation: 0.634

Reflection question: when does GRPO within-group normalization fail?

1. k is too small: with $k=2$, the mean and standard deviation are extremely unstable, so the statistics are unreliable.
2. The reward distribution is skewed: when most answers receive zero reward, a few high-reward answers dominate the gradient signal.
3. All answers have the same quality: variance is zero, all advantages are zero, and there is no gradient signal. This is the late-training "graduation" phenomenon.
4. The reward signal is discontinuous: with only 0/1 values, normalized advantages are discrete and the gradient signal is not fine-grained enough.

GRPO mitigates these problems through DAPO's "dynamic sampling" improvement: filter out problems the model has already solved and keep only samples with gradient signal.

Full Comparison Between GRPO and PPO

Component	PPO	GRPO
Baseline (Critic)	Independent $V(s)$ network	Within-group mean $\bar{r}$
Advantage computation	$A=R-V(s)$ or GAE	$A_i=(r_i-\bar{r})/{\sigma }_r$
Number of models	4 (Actor + Critic + Ref + RM)	2 (Actor + Ref)
Clipping mechanism	PPO Clip	Same PPO Clip
Sampling method	Online interaction	Group sampling; sample k answers per prompt
Memory	High	30-40% lower
Baseline quality	Depends on Critic training quality	Depends on group size $k$
Baseline update speed	Requires retraining the Critic	Updates automatically with each batch

It is worth noting that GRPO inherits PPO's clipping mechanism but does not inherit GAE. The reason is that GRPO usually receives only one reward signal at the end of the sequence, such as correct or wrong, rather than a reward at every token. In this situation, multi-step TD in GAE degenerates into a single-step estimate and is not fundamentally different from subtracting the mean from the final reward.

GRPO solves the Critic problem elegantly through within-group normalization. But this is only the first step. On the policy side, DeepSeek-R1-Zero showed that pure RL training can work without SFT, and DAPO further improved GRPO's engineering efficiency. Next, let's look at these frontier developments: DeepSeek-R1 and DAPO.