5.2 PPO Mathematical Derivation

In the previous section, we trained LunarLander with SB3's PPO and looked at curves such as reward, entropy, and clip fraction. Now we should answer a more basic question:

What exactly is PPO, and why does it eventually become a single loss function?

Prerequisites

This section starts directly from PPO's derivation starting point. The following concepts are assumed (for a review, see the Appendix):

• Policy gradient theorem: ${\nabla }_{\theta }J(\theta )={\mathbb{E}}_t[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t]$
• Advantage function $A_t$: measures how much better an action is than average; baseline provided by the Critic
• Actor-Critic framework: Actor chooses actions, Critic estimates state values
• On-policy constraint: vanilla policy gradients require data from the current policy; each batch can only be used once

PPO stands for Proximal Policy Optimization. The name is worth unpacking:

• Policy: the model that chooses actions.
• Optimization: training, i.e., improving that policy.
• Proximal: "nearby" updates; the new policy should not move too far from the old one.

So here is the headline conclusion:

PPO is not a policy, and it is not merely a loss. PPO is a method for training a policy network.

In reinforcement learning, the policy is the object we truly train. It is usually written as:

$${\pi }_{\theta }(a\mid s)$$

This means: "under state $s$, the policy network parameterized by $\theta$ assigns probability to action $a$." In code, this policy is typically the Actor network. For example, the Actor takes a game frame or a robot state as input, and outputs a probability distribution over actions.

What PPO provides is a recipe for training this Actor. It does not hard-code actions, and it does not replace the policy network. Instead, it specifies an update rule: use the current Actor to collect a batch of experience, then adjust the Actor using that batch, while preventing each update from being too aggressive.

It helps to separate three closely-related concepts:

Name	What It Is	Roughly What It Corresponds To In Code
Policy	the object being trained; chooses actions given states	actor / model output action_probs
PPO	the training method: sampling, advantage estimation, constrained updates, backprop	the full training loop
PPO loss	a differentiable objective used to update network parameters in PPO	policy_loss + value_loss - entropy_bonus

Why will we keep talking about a loss? Because neural networks cannot directly interpret the instruction "make the policy more stable; do not change too fast." An optimizer understands a very specific interface: give it a scalar loss, it computes gradients via loss.backward(), then updates parameters via optimizer.step().

So PPO's ideas must eventually become a loss in order to update the Actor and Critic.

Put differently: PPO is a method, the policy is the model being trained, and the loss is the training signal that makes the method real in code. We derive the PPO loss not because PPO is only a loss, but because the loss is the point at which PPO touches neural-network parameters.

PPO Code Skeleton

To keep the formulas grounded, we first show what PPO "looks like in code." The code below is not an engineering-optimized implementation. It is a learning-oriented minimal PyTorch PPO skeleton: policy network, sampling, advantage estimation, PPO-Clip loss, value-function loss, entropy bonus, and multiple epochs of updates.

Every time we derive a new formula, we will come back to a corresponding part of this code. The highlighted lines are the ones we will repeatedly unpack. You do not need to fully understand every line now; just remember the big picture:

PPO ultimately links "collect experience, estimate advantages, constrain policy changes, backpropagate updates" into a single training loop.

 21    # [A] 策略和值函数：Actor 输出动作概率，Critic 输出状态价值
 22    def forward(self, obs):
 23        h = self.backbone(obs)
 24        logits = self.actor_head(h)
 25        action_probs = F.softmax(logits, dim=-1)
 26        value = self.critic_head(h).squeeze(-1)
 27        return action_probs, value
 28 
 29    # [B] 动作采样：根据策略分布选择动作，并记录 log_prob
 30    def act(self, obs):
 31        action_probs, value = self.forward(obs)
 32        dist = Categorical(action_probs)
 33        action = dist.sample()
 34        log_prob = dist.log_prob(action)
 35        return action, log_prob, value
 36 
 37    def evaluate(self, obs, actions):
 38        action_probs, values = self.forward(obs)
 39        dist = Categorical(action_probs)
 40        new_logprobs = dist.log_prob(actions)
 41        entropy = dist.entropy()
 42        return new_logprobs, values, entropy
        ⋮
 77# [D] 计算优势：GAE（Generalized Advantage Estimation）
 78def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
 79    advantages = torch.zeros_like(rewards)
 80    last_advantage = 0.0
 81    next_value = 0.0
 82 
 83    for t in reversed(range(len(rewards))):
 84        mask = 1.0 - dones[t]
 85        delta = rewards[t] + gamma * next_value * mask - values[t]
 86        last_advantage = delta + gamma * lam * mask * last_advantage
 87        advantages[t] = last_advantage
 88        next_value = values[t]
 89 
 90    returns = advantages + values
 91    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
 92    return advantages, returns
        ⋮
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss
        ⋮
114def ppo_update(model, optimizer, batch, advantages, returns,
115               clip_eps=0.2, vf_coef=0.5, ent_coef=0.01,
116               epochs=10, minibatch_size=64):
117    states = batch["states"]
118    actions = batch["actions"]
119    old_logprobs = batch["old_logprobs"]
120    old_values = batch["values"]
121    batch_size = states.size(0)
122 
123    for _ in range(epochs):
124        indices = torch.randperm(batch_size, device=states.device)
125        for start in range(0, batch_size, minibatch_size):
126            mb = indices[start:start + minibatch_size]
127 
128            new_logprobs, new_values, entropy = model.evaluate(states[mb], actions[mb])
129 
130            pg_loss = actor_loss_fn(new_logprobs, old_logprobs[mb], advantages[mb], clip_eps)
131            vf_loss = critic_loss_fn(new_values, old_values[mb], returns[mb], clip_eps)
132            entropy_bonus = entropy.mean()
133            loss = pg_loss + vf_coef * vf_loss - ent_coef * entropy_bonus
134 
135            optimizer.zero_grad()
136            loss.backward()
137            optimizer.step()
        ⋮
140# [F] 训练循环：采样一批数据，再用这批数据更新多轮
141device = "cuda" if torch.cuda.is_available() else "cpu"
142env = gym.make("CartPole-v1")
143model = ActorCritic(env.observation_space.shape[0], env.action_space.n).to(device)
144optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)
145 
146for update in range(100):
147    batch = collect_rollout(env, model, steps=2048, device=device)
148    advantages, returns = compute_gae(batch["rewards"], batch["values"], batch["dones"])
149    ppo_update(model, optimizer, batch, advantages, returns)

You can roughly split the code into six parts:

Tag	Code Block	What We Will Explain Later
[A]	forward	what the policy ${\pi }_{\theta }(a\mid s)$ and value function $V_{\theta }(s)$ are
[B]	act / evaluate	why we construct dist, and why we store log_prob
[C]	collect_rollout	what on-policy data is, and why we record the old policy probabilities
[D]	compute_gae	how returns, value functions, and advantages relate
[E]	ppo_update	PPO-Clip's ratio, clamp, min, and the total loss
[F]	training loop	why we update multiple epochs on the same batch

When key variables appear later, we will repeatedly refer back to this mapping table:

Symbol	Meaning	Typical Code Variable
$s_t$	state at time $t$	states
$a_t$	action taken at time $t$	actions
$r_t$	reward at time $t$	rewards
$G_t$	discounted return starting from time $t$	returns
$V_{\theta }(s)$	Critic's estimate of future return from state $s$	value / new_values
$A_t$ or ${\hat{A}}_t$	advantage estimate: how much better this action is than the state's baseline	advantages
${\pi }_{\text{old}}(a\mid s)$	the old policy that collected this batch	stored old_logprobs
$r_t(\theta )$	ratio ${\pi }_{\theta }/{\pi }_{\text{old}}$	ratio = exp(new_logprobs-old_logprobs)
$\epsilon$	PPO clipping range, often 0.1 or 0.2	clip_eps / clip_range
$H[{\pi }_{\theta }]$	policy entropy (how random the action distribution is)	entropy

This section's derivation path:

Once we walk through these steps, the final PPO formula will not seem to come out of thin air:

$$L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \left(r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon ){\hat{A}}_t\right)\right]$$

Starting Point: The Limits of Vanilla Policy Gradients

Actor-Critic policy gradients give us a seemingly complete training pipeline: sample with the current policy → compute advantages → update the policy using $\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t$ (if any of these concepts are unfamiliar, see the Appendix).

The problem is that this pipeline has a hard requirement: the data used to update the policy should ideally be collected by that same policy. This property is called on-policy. In the formula, the expectation is ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[\cdots ]$, meaning the data should come from the current policy ${\pi }_{\theta }$. But after one gradient update, parameters change from ${\theta }_{\text{old}}$ to $\theta$. The trajectories we just collected no longer come from the new policy; they come from the old policy ${\pi }_{\text{old}}$.

If we use each batch only once, training becomes extremely wasteful. Collecting 2048 steps of environment interaction is expensive, especially in robotics, game simulators, and LLM answer generation. Naturally, we ask:

Can we reuse data collected by the old policy to update the new policy for multiple epochs?

This is PPO's core tension:

We want to reuse old data to improve sample efficiency, but we must not let the new policy drift too far from the old one, otherwise old data will mislead the update.

In the learning-oriented PPO skeleton, collect_rollout deliberately stores the log probability at sampling time:

 45# [C] 采样一批 on-policy 数据：这些数据来自"当前策略"
 46def collect_rollout(env, model, steps=2048, device="cpu"):
 47    obs, _ = env.reset()
 48    batch = {k: [] for k in ["states", "actions", "rewards", "dones", "old_logprobs", "values"]}
 49 
 50    for _ in range(steps):
 51        obs_tensor = torch.as_tensor(obs, dtype=torch.float32, device=device)
 52        with torch.no_grad():
 53            action, old_logprob, value = model.act(obs_tensor)
 54 
 55        next_obs, reward, terminated, truncated, _ = env.step(action.item())
 56        done = terminated or truncated
 57 
 58        batch["states"].append(obs)
 59        batch["actions"].append(action.item())
 60        batch["rewards"].append(reward)
 61        batch["dones"].append(done)
 62        batch["old_logprobs"].append(old_logprob.item())
 63        batch["values"].append(value.item())
 64 
 65        obs = next_obs if not done else env.reset()[0]
 66 
 67    return {
 68        "states": torch.as_tensor(np.array(batch["states"]), dtype=torch.float32, device=device),
 69        "actions": torch.as_tensor(batch["actions"], dtype=torch.long, device=device),
 70        "rewards": torch.as_tensor(batch["rewards"], dtype=torch.float32, device=device),
 71        "dones": torch.as_tensor(batch["dones"], dtype=torch.float32, device=device),
 72        "old_logprobs": torch.as_tensor(batch["old_logprobs"], dtype=torch.float32, device=device),
 73        "values": torch.as_tensor(batch["values"], dtype=torch.float32, device=device),
 74    }

This old_logprobs is $\log {\pi }_{\text{old}}(a_t\mid s_t)$. During updates, we recompute the same state-action pairs under the new policy to get new_logprobs. Comparing them tells us how far the policy has moved. Importance sampling is the tool that answers whether "old data can still be used."

Step 1: Importance Sampling

The previous issue is that vanilla policy gradients want data collected by ${\pi }_{\theta }$. Can we use data collected by ${\pi }_{\text{old}}$ to evaluate and improve a new policy? Yes, via importance sampling.

1.1 The Importance Sampling Identity

The core identity is: for any function $f$,

$${\mathbb{E}}_{a\sim {\pi }_{\theta }}[f(a)]={\mathbb{E}}_{a\sim {\pi }_{\text{old}}}\left[\frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}\cdot f(a)\right]$$

Why is this true? Expand the left side:

$${\mathbb{E}}_{a\sim {\pi }_{\theta }}[f(a)]=\sum _a{\pi }_{\theta }(a|s)\cdot f(a)$$

Rewrite ${\pi }_{\theta }(a|s)$ as ${\pi }_{\text{old}}(a|s)\cdot \frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}$:

$$=\sum _a{\pi }_{\text{old}}(a|s)\cdot \frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}\cdot f(a)={\mathbb{E}}_{a\sim {\pi }_{\text{old}}}\left[\frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}\cdot f(a)\right]$$

The identity holds. The intuition is: we want the expectation of $f$ under the "new world" ${\pi }_{\theta }$, but we only have samples from the "old world" ${\pi }_{\text{old}}$. The fix is to reweight each sample. If the new world is more likely to produce this action than the old world, the weight is greater than 1; otherwise it is less than 1. The weight is exactly $\frac{{\pi }_{\theta }}{{\pi }_{\text{old}}}$.

1.2 Policy Ratio

Define the policy ratio:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{\text{old}}(a_t|s_t)}$$

In code, we compute it using the exponential of the log-prob difference, which is numerically more stable than direct division:

 95# [E] PPO 损失函数（DeepSpeed-Chat / VeRL / OpenRLHF 风格）
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss

$r_t=1$ means the new and old policies assign the same probability to this action. $r_t>1$ means the new policy is more inclined to take this action; $r_t<1$ means the opposite.

1.3 Surrogate Objective

Apply importance sampling to the policy-gradient objective to get the surrogate objective:

$$L^{\text{IS}}(\theta )={\mathbb{E}}_t\left[r_t(\theta )\cdot A_t\right]$$

Expanded:

$$L^{\text{IS}}(\theta )={\mathbb{E}}_t\left[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{\text{old}}(a_t|s_t)}\cdot A_t\right]$$

In code this is surr1 = ratio * advantages, right after ratio in the PPO update:

 95# [E] PPO 损失函数（DeepSpeed-Chat / VeRL / OpenRLHF 风格）
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss

This objective has an important property:

at $\theta ={\theta }_{\text{old}}$, its first-order gradient matches the vanilla policy gradient.

$${\nabla }_{\theta }{L}^{\text{IS}}(\theta ){|}_{\theta ={\theta }_{\text{old}}}={\nabla }_{\theta }J(\theta )$$

The check is straightforward: when $\theta ={\theta }_{\text{old}}$, we have $r_t=1$. Also ${\nabla }_{\theta }{r}_t={\nabla }_{\theta }\frac{{\pi }_{\theta }}{{\pi }_{\text{old}}}=\frac{{\nabla }_{\theta }{\pi }_{\theta }}{{\pi }_{\text{old}}}$. Substituting restores the policy-gradient form.

But once $\theta$ moves away from ${\theta }_{\text{old}}$, the two objectives diverge. The farther away the new policy is, the less reliable the surrogate becomes. That is the next problem to solve.

Step 2: From the Surrogate Objective to PPO-Clip

We now have a key expression:

$$L^{\text{IS}}(\theta )={\mathbb{E}}_t[r_t(\theta )A_t]$$

Do not rush to TRPO yet. If we only look at this expression, it already reveals PPO's two core inputs:

Name	Symbol	Code Variable	What Question It Answers
policy ratio	$r_t(\theta )$	ratio	does the new policy prefer this action more than the old policy?
advantage	$A_t$ or ${\hat{A}}_t$	advantages	is this action better than average at this state?

If we impose no constraints, we would simply maximize:

$$r_t(\theta )A_t$$

In code, this is surr1 = ratio * advantages:

 95# [E] PPO 损失函数（DeepSpeed-Chat / VeRL / OpenRLHF 风格）
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss

You can interpret surr1 as the raw policy-improvement objective. Its rule is:

• If $A_t>0$, this is a good action; we want to increase its probability, i.e. make $r_t$ larger.
• If $A_t<0$, this is a bad action; we want to decrease its probability, i.e. make $r_t$ smaller.

But this objective is too greedy. Suppose $A_t=+2$ and the current ratio is $r_t=5$, then $r_t{A}_t=10$. If we keep pushing this action up so that $r_t$ becomes 10, 50, 100, the objective keeps increasing. The optimizer would think "bigger is always better," but at that point the new policy is far from the old one, and the old data is no longer reliable.

PPO does not introduce a complicated new algorithm. It adds a very direct conservative rule on top of this objective:

You may increase the probability of good actions and decrease the probability of bad actions, but do not let the new policy move too far relative to the old policy.

So we restrict the policy ratio to a small interval:

$${\overline{r}}_t(\theta )=\text{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )$$

If $\epsilon =0.2$, the interval is $[0.8,1.2]$. This means: for an action that appears in the old batch, the new policy's probability should ideally not be below $0.8$ times the old policy's probability, and not be above $1.2$ times it.

In code, the unclipped objective surr1, the clipped objective surr2, and the final policy_loss are computed together:

 95# [E] PPO 损失函数（DeepSpeed-Chat / VeRL / OpenRLHF 风格）
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss

Now we have two objectives:

Code	Math	Meaning
surr1	$r_t(\theta )A_t$	what the policy would like to do without constraints
surr2	$\text{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )A_t$	how far we allow it to change under the update constraint

PPO takes the smaller of the two:

$$J^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \left(r_t(\theta )A_t,\text{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )A_t\right)\right]$$

This is PPO-Clip. It is not derived by mechanically transforming the TRPO constraint into algebra. Instead, it starts from the importance-sampling surrogate objective and adds a conservative rule: "do not let the ratio drift too far." TRPO is one historical source of this conservative mindset, but it is not required to understand the PPO code.

In code this is torch.min(surr1, surr2).mean(). Why the minus sign? Because we want to maximize policy_objective, while PyTorch minimizes losses. So we write policy_loss = -policy_objective.

What Clipping Does

Case 1: $A_t>0$ (good action; probability should increase)

When $A_t>0$, we want $r_t$ to increase (the new policy assigns higher probability to the action). The unclipped term $r_t\cdot A_t$ grows linearly with $r_t$ with no upper bound. The clipped term ${\overline{r}}_t\cdot A_t$ becomes a constant $(1+\epsilon )\cdot A_t$ once $r_t>1+\epsilon$.

Range of $r_t$	Unclipped $r_t\cdot A_t$	Clipped ${\overline{r}}_t\cdot A_t$	Which One $\min$ Picks
$r_t\le 1+\epsilon$	$r_t\cdot A_t$	$r_t\cdot A_t$	equal; normal optimization
$r_t>1+\epsilon$	$r_t\cdot A_t$ (larger)	$(1+\epsilon )\cdot A_t$ (constant)	clipped term; zero gradient

So the probability of good actions can increase, but only up to about $(1+\epsilon )$ times that of the old policy. Beyond that, the objective becomes "flat": it stops rewarding further increases, so the gradient becomes zero.

Case 2: $A_t<0$ (bad action; probability should decrease)

When $A_t<0$, we want $r_t$ to decrease (the new policy assigns lower probability). But if $r_t$ has already dropped below $1-\epsilon$, the new policy has already pushed that action probability down too much; PPO no longer rewards further suppression.

This is easy to misread because $A_t$ is negative. Consider a numeric example: $A_t=-2$, $\epsilon =0.2$. If $r_t=0.7$, the unclipped term is $0.7\times (-2)=-1.4$, while the clipped term is $0.8\times (-2)=-1.6$. The min picks the smaller value, i.e. $-1.6$, which is the clipped term. Since the clipped term is constant, the gradient is zero.

Range of $r_t$	Unclipped $r_t\cdot A_t$	Clipped ${\overline{r}}_t\cdot A_t$	Which One $\min$ Picks
$r_t<1-\epsilon$	larger (e.g. $-1.4$)	$(1-\epsilon )\cdot A_t$ (constant)	clipped term; zero gradient
$r_t\ge 1-\epsilon$	unclipped term	equal inside interval; clipped can become larger above	unclipped term; keep optimizing

So the probability of bad actions can decrease, but only down to about $(1-\epsilon )$ times the old probability. Beyond that the objective goes flat and stops providing further incentive. If the bad action probability increases instead, the unclipped term makes the objective worse, and the gradient pulls it back.

Case 3: $A_t=0$ (neutral action)

Then $r_t\cdot A_t=0$. No matter how $r_t$ changes, the objective is always 0, so PPO does not adjust that action.

Putting these cases together, the meaning of PPO-Clip becomes clear:

it does not forbid learning; it simply stops rewarding the part of the change that has already gone too far.

import numpy as np
import matplotlib.pyplot as plt

# ==========================================
# Geometric intuition for the PPO-Clip objective
# ==========================================
epsilon = 0.2
r = np.linspace(0.0, 2.0, 500)

def clip_objective(r, A, eps=0.2):
    r_clipped = np.clip(r, 1 - eps, 1 + eps)
    return np.minimum(r * A, r_clipped * A)

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

for ax, (A_val, title) in zip(
    axes,
    [(1.0, "A > 0 (good action)"), (-1.0, "A < 0 (bad action)"), (0.0, "A = 0 (neutral)")],
):
    obj = clip_objective(r, A_val)
    ax.plot(r, r * A_val, "b--", alpha=0.4, label="unclipped r·A")
    ax.plot(r, obj, "r-", linewidth=2, label="PPO-Clip min(...)")
    ax.axvspan(1 - epsilon, 1 + epsilon, alpha=0.1, color="green", label="safe interval")
    ax.set_title(title)
    ax.set_xlabel("policy ratio r_t(θ)")
    ax.set_ylabel("objective value")
    ax.legend(fontsize=8)

plt.suptitle("Three cases of the PPO-Clip objective (ε=0.2)", fontsize=13)
plt.tight_layout()
plt.savefig("ppo_clip_three_cases.png", dpi=150)
print("Saved visualization")

Clipping Intuition

If you look at the three cases together, PPO-Clip's design intention becomes very clear:

With $\epsilon =0.2$, after each update, the probability assigned to an action is constrained to remain near the old policy. This "safety rail" ensures that even if gradient estimates are noisy, the policy will not jump too far in a single step.

Step 3: PPO Is Not Only a Loss Function

At this point it is easy to form a misconception: does understanding PPO mean understanding the PPO loss? The answer is: no.

PPO is a policy-optimization algorithm. More concretely, it is a training procedure that answers:

Given a policy network that already acts, how do we use newly collected experience to make it reliably better?

So PPO is not a single formula, and it is not just one loss.backward() call. A complete PPO method includes at least these pieces:

Component in PPO	What It Does	Where It Appears in Code
sampling with the current policy	interact with the environment to collect a new batch	collect_trajectories(...)
old policy record	store action probabilities at sampling time for later comparison	old_logprobs
advantage estimation	judge whether each action is above/below average	advantages / compute_gae(...)
clipped policy update	update the Actor while constraining drift from the old policy	ppo_clip_loss(...)
value-function training	train the Critic to estimate state values accurately	value_loss
entropy bonus	maintain exploration; avoid becoming too confident too early	entropy_bonus
multi-epoch mini-batch updates	reuse the same batch for multiple epochs to improve sample use	n_epochs / mini-batch

Therefore, the PPO loss is not the entirety of PPO, but it is the most important "policy update rule" within PPO. It tells the Actor which action probabilities to increase, which to decrease, and the maximum allowed change.

You can think of PPO as a training protocol:

The reason "loss" matters is that neural networks update parameters through backpropagation. To affect parameters, PPO's ideas must become a differentiable objective. That is why we emphasize PPO loss, but you should not shrink PPO into the loss alone.

Step 4: How PPO Appears in Code

If we keep only PPO's core policy update, the landing point is the following lines:

 95# [E] PPO 损失函数（DeepSpeed-Chat / VeRL / OpenRLHF 风格）
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss

This piece of code needs three main inputs:

Input	Where It Comes From	What It Does
old_logprobs	stored during rollout collection	records the old policy's probability for the action
new_logprobs	recomputed during update	the new policy's probability for the same action
advantages	computed from returns, Critic, or GAE	tells whether the action should be encouraged or suppressed

It outputs a scalar policy_loss. This scalar is exactly what backpropagation consumes:

128            new_logprobs, new_values, entropy = model.evaluate(states[mb], actions[mb])
129 
130            pg_loss = actor_loss_fn(new_logprobs, old_logprobs[mb], advantages[mb], clip_eps)
131            vf_loss = critic_loss_fn(new_values, old_values[mb], returns[mb], clip_eps)
132            entropy_bonus = entropy.mean()
133            loss = pg_loss + vf_coef * vf_loss - ent_coef * entropy_bonus
134 
135            optimizer.zero_grad()
136            loss.backward()
137            optimizer.step()

Of course, real PPO does not only train the Actor; it also trains the Critic, and usually includes an entropy bonus to encourage exploration. So we combine policy_loss into a full loss:

loss = policy_loss + vf_coef * value_loss - ent_coef * entropy_bonus.

If you have derived PPO on paper and want to implement it, you only need to connect the data in this order:

This is the minimal closed loop that turns PPO formulas into a training program.

Supplement: TRPO Is Historical Context, Not a Required Derivation

TRPO (Trust Region Policy Optimization) and PPO solve the same issue: policy updates must not be too large. TRPO is written as:

$$\underset{\theta }{\max }{L}^{\text{IS}}(\theta )\text{s.t.}{\bar{D}}_{\text{KL}}({\theta }_{\text{old}},\theta )\le \delta$$

This means: optimize the surrogate objective, but constrain the average KL divergence between old and new policies by a small threshold $\delta$.

This path is theoretically elegant, but in practice it requires constrained optimization, conjugate gradients, approximate second-order information, and more. For a chapter whose goal is "derive PPO loss from formulas," TRPO is not a necessary prerequisite. Treat it as a historical note:

TRPO limits policy change via a KL constraint; PPO approximates a similar effect by clipping the policy ratio.

So the main line should be:

TRPO simply reminds us: PPO's "Proximal" comes from trust-region thinking, but the concrete code you need is ratio, clamp, min, and the total loss.

Step 5: The Full PPO Loss

In real training, PPO does not only optimize the clipped surrogate; it trains the Critic and preserves exploration. To avoid symbol confusion, separate two things:

• $J^{\text{PPO}}(\theta )$: the mathematical objective we want to maximize.
• loss: the training loss we minimize in code.

The maximization objective can be written as:

$$J^{\text{PPO}}(\theta )=J^{\text{CLIP}}(\theta )-c_1{L}^{\text{VF}}(\theta )+c_{2}H[{\pi }_{\theta }]$$

Here $J^{\text{CLIP}}$ is the policy-improvement objective, $L^{\text{VF}}$ is the Critic's value error, and $H[{\pi }_{\theta }]$ is the policy entropy. Since code minimizes loss, we negate the policy objective and the entropy term.

In code, the total loss is composed here:

128            new_logprobs, new_values, entropy = model.evaluate(states[mb], actions[mb])
129 
130            pg_loss = actor_loss_fn(new_logprobs, old_logprobs[mb], advantages[mb], clip_eps)
131            vf_loss = critic_loss_fn(new_values, old_values[mb], returns[mb], clip_eps)
132            entropy_bonus = entropy.mean()
133            loss = pg_loss + vf_coef * vf_loss - ent_coef * entropy_bonus
134 
135            optimizer.zero_grad()
136            loss.backward()
137            optimizer.step()

Policy Loss

The policy maximization objective is the clipped surrogate:

$$J^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \left(r_t(\theta )\cdot A_t,{\overline{r}}_t(\theta )\cdot A_t\right)\right]$$

The policy_loss in code is its negative:

$$L^{\text{policy}}(\theta )=-J^{\text{CLIP}}(\theta )$$

This term updates the Actor: increase probabilities of good actions, decrease probabilities of bad actions, while clipping constrains the magnitude of change within a safe range.

Value-Function Loss

The Critic should estimate state values accurately. The value loss is the mean squared error between the Critic prediction $V_{\theta }(s_t)$ and a target return $V_t^{\text{targ}}$:

$$L^{\text{VF}}(\theta )={\mathbb{E}}_t\left[{\left(V_{\theta }(s_t)-V_t^{\text{targ}}\right)}^2\right]$$

Here $V_t^{\text{targ}}$ is computed via GAE (derived in detail in the next section).

Why do we need a separate value loss? Because the Critic's accuracy directly determines the quality of the advantage estimate $A_t$. If the Critic is inaccurate, $A_t$ will have large bias and can mislead the Actor. The MSE loss continuously corrects the Critic so its predictions track true returns.

In code: value_loss = F.mse_loss(new_values, returns[mb]). It is backpropagated together with policy_loss in the same update function.

Entropy Bonus

Policy entropy encourages exploration and prevents premature collapse to a deterministic policy:

$$H[{\pi }_{\theta }]=-{\mathbb{E}}_t\left[\sum _a{\pi }_{\theta }(a|s_t)\log {\pi }_{\theta }(a|s_t)\right]$$

Higher entropy means the policy is more "hesitant" (more uniform action distribution), which encourages exploration; lower entropy means the policy is more "certain" (always choosing one action), which reduces exploration. The coefficient $c_2$ is often around 0.01.

Why include entropy? Clipping stabilizes training, but it can also cause a side effect: the policy may "lock onto" a suboptimal action too early. The entropy bonus rewards uncertainty inside the loss, ensuring the policy retains ongoing exploration pressure.

In code: entropy_bonus = entropy.mean(). Note the minus sign in the total loss: - ent_coef * entropy_bonus, because we want to maximize entropy, which is equivalent to subtracting it when minimizing loss.

How the Three Terms Work Together

Each term does a different job:

policy loss drives Actor improvement, value loss ensures the Critic provides accurate advantage signals, and entropy bonus preserves exploration.

They collaborate through the shared Actor-Critic network. In ppo_from_scratch.py, the Actor and Critic share the same backbone network (shared_net), so one backpropagation updates both.

Hyperparameter Summary

Symbol	Name	Typical Value	Role	Code Parameter
$\epsilon$	clip range	0.1-0.2	limits how far ratios may move	clip_range
$c_1$	value-loss coefficient	0.5	balances policy update vs value fitting	vf_coef
$c_2$	entropy coefficient	0.01	encourages exploration	ent_coef
$\gamma$	discount factor	0.99	decay of future rewards	gamma
$\lambda$	GAE parameter	0.95	bias-variance tradeoff in advantage estimation	gae_lambda
$T$	rollout length	2048	how many steps to collect per rollout	n_steps
$K$	number of epochs	10	how many passes over the same data batch	n_epochs

Step 6: The Complete PPO Algorithm

Putting everything together, the PPO training loop is:

If you compare against the code, each step can be traced to a specific piece:

 21    # [A] 策略和值函数：Actor 输出动作概率，Critic 输出状态价值
 22    def forward(self, obs):
 23        h = self.backbone(obs)
 24        logits = self.actor_head(h)
 25        action_probs = F.softmax(logits, dim=-1)
 26        value = self.critic_head(h).squeeze(-1)
 27        return action_probs, value
 28 
 29    # [B] 动作采样：根据策略分布选择动作，并记录 log_prob
 30    def act(self, obs):
 31        action_probs, value = self.forward(obs)
 32        dist = Categorical(action_probs)
 33        action = dist.sample()
 34        log_prob = dist.log_prob(action)
 35        return action, log_prob, value
 36 
 37    def evaluate(self, obs, actions):
 38        action_probs, values = self.forward(obs)
 39        dist = Categorical(action_probs)
 40        new_logprobs = dist.log_prob(actions)
 41        entropy = dist.entropy()
 42        return new_logprobs, values, entropy
        ⋮
 77# [D] 计算优势：GAE（Generalized Advantage Estimation）
 78def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
 79    advantages = torch.zeros_like(rewards)
 80    last_advantage = 0.0
 81    next_value = 0.0
 82 
 83    for t in reversed(range(len(rewards))):
 84        mask = 1.0 - dones[t]
 85        delta = rewards[t] + gamma * next_value * mask - values[t]
 86        last_advantage = delta + gamma * lam * mask * last_advantage
 87        advantages[t] = last_advantage
 88        next_value = values[t]
 89 
 90    returns = advantages + values
 91    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
 92    return advantages, returns
        ⋮
 96def actor_loss_fn(new_logprobs, old_logprobs, advantages, clip_eps=0.2):
 97    ratio = torch.exp(new_logprobs - old_logprobs)
 98    surr1 = ratio * advantages
 99    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
100    policy_loss = -torch.min(surr1, surr2).mean()
101    return policy_loss
        ⋮
114def ppo_update(model, optimizer, batch, advantages, returns,
115               clip_eps=0.2, vf_coef=0.5, ent_coef=0.01,
116               epochs=10, minibatch_size=64):
117    states = batch["states"]
118    actions = batch["actions"]
119    old_logprobs = batch["old_logprobs"]
120    old_values = batch["values"]
121    batch_size = states.size(0)
122 
123    for _ in range(epochs):
124        indices = torch.randperm(batch_size, device=states.device)
125        for start in range(0, batch_size, minibatch_size):
126            mb = indices[start:start + minibatch_size]
127 
128            new_logprobs, new_values, entropy = model.evaluate(states[mb], actions[mb])
129 
130            pg_loss = actor_loss_fn(new_logprobs, old_logprobs[mb], advantages[mb], clip_eps)
131            vf_loss = critic_loss_fn(new_values, old_values[mb], returns[mb], clip_eps)
132            entropy_bonus = entropy.mean()
133            loss = pg_loss + vf_coef * vf_loss - ent_coef * entropy_bonus
134 
135            optimizer.zero_grad()
136            loss.backward()
137            optimizer.step()
        ⋮
140# [F] 训练循环：采样一批数据，再用这批数据更新多轮
141device = "cuda" if torch.cuda.is_available() else "cpu"
142env = gym.make("CartPole-v1")
143model = ActorCritic(env.observation_space.shape[0], env.action_space.n).to(device)
144optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)
145 
146for update in range(100):
147    batch = collect_rollout(env, model, steps=2048, device=device)
148    advantages, returns = compute_gae(batch["rewards"], batch["values"], batch["dones"])
149    ppo_update(model, optimizer, batch, advantages, returns)

Some key design decisions and their intuition:

• Reuse the same data for $K$ epochs: collecting data is expensive (requires running the environment), so we update multiple times on the same batch. Clipping prevents multi-epoch updates from drifting too far.
• Mini-batch updates: split $T$ steps into several mini-batches; compute gradients per mini-batch to improve training efficiency.
• Recompute $r_t$ each epoch: even though the data batch is the same, $\theta$ changes after each epoch, so $r_t$ changes too; clipping continues to take effect dynamically.

Derivation Note: PPO-Penalty Variant

The PPO paper actually proposes two variants. Besides PPO-Clip, it proposes PPO-Penalty (also called PPO-KL), which directly adds a KL penalty term:

$$L^{\text{KL}}(\theta )={\mathbb{E}}_t\left[r_t(\theta )\cdot A_t-\beta \cdot D_{\text{KL}}({\pi }_{\text{old}},{\pi }_{\theta })\right]$$

$\beta$ is an adaptive coefficient: if current KL is too large, increase $\beta$ to penalize more; if KL is too small, decrease $\beta$ to loosen the constraint.

PPO-Penalty can be better in some settings (especially when you need precise control of policy change), but it is more complex to implement and introduces an additional adaptive mechanism to tune. In practice, PPO-Clip is more common.

Thought Question 1: If we set ε to 0, what does PPO-Clip degenerate into?

When $\epsilon =0$, the clipping interval collapses to $[1,1]$, so ${\overline{r}}_t(\theta )=1$. The PPO-Clip objective becomes:

$$L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \left(r_t(\theta )\cdot A_t,1\cdot A_t\right)\right]$$

For $A_t>0$, $\min (r_t\cdot A_t,A_t)$: when $r_t>1$, the objective is the constant $A_t$, so further increasing a good action's probability no longer improves the objective; when $r_t<1$, the objective is $r_t\cdot A_t$ and the gradient only pushes it back toward 1. This means good actions cannot be meaningfully increased above the old policy.

For $A_t<0$, $\min (r_t\cdot A_t,A_t)$: when $r_t<1$, the objective is the constant $A_t$, so further decreasing a bad action's probability no longer improves the objective; when $r_t>1$, the objective is $r_t\cdot A_t$ and the gradient only pushes it back toward 1. This means bad actions cannot be meaningfully decreased below the old policy either.

In short, $\epsilon =0$ almost freezes the policy near the old one: whether advantages are positive or negative, the policy cannot make meaningful improvements. This shows $\epsilon$ controls both "allowed change magnitude" and "learning capacity."

Thought Question 2: Can clipping fully replace a KL constraint? Can clipping fail?

Clipping effectively limits policy change in most situations, but it has a theoretical weakness: it constrains the ratio $r_t$ for each individual action, rather than directly constraining the overall distribution distance (KL divergence) between two policies.

Consider an extreme case: a policy has 100 actions, and clipping allows each action probability to change by $\pm 20\%$. If all actions are pushed to the boundary simultaneously, the overall distribution change can exceed a KL constraint such as $\delta =0.01$. In practice, this is rare because advantage estimates are noisy and usually do not push all actions in extreme directions simultaneously. But for settings where policy-change control must be strict (e.g., LLM alignment), practitioners often monitor KL as an additional safety metric. This is why in Chapter 8's RLHF training you will see both clip_fraction and approx_kl logged.

Thought Question 3: Why does PPO update K epochs on the same batch, instead of collecting K batches and updating once each?

The two strategies have the same total number of samples ($K\times T$ steps), but differ in data quality and compute cost.

"Collect K batches, update once each" uses fresh data from the current policy every time, so the gradient estimate is unbiased. But collecting data requires environment simulation, which is often far more expensive than parameter updates. In LLM settings, generating a batch of responses can take minutes, while a gradient update can take seconds.

"Collect one batch, update K epochs" reuses old data for multiple updates. From the importance-sampling viewpoint, only the first epoch is unbiased; later epochs introduce bias as $\theta$ drifts away from ${\theta }_{\text{old}}$. Clipping is designed to mitigate this: when the drift becomes too large, clipping drives gradients toward zero and effectively stops unsafe updates. This is an engineering tradeoff: accept "small bias" in exchange for "large compute savings."

In practice, $K$ is often 3-10, and clipping can keep the bias within an acceptable range.

---

At this point, you have the complete mathematical picture of PPO: from the importance-sampling surrogate objective, to the PPO-Clip policy loss formed by ratio, clamp, and min, and finally to the total loss that can be backpropagated directly.

The next two sections each go deeper into a key detail:

• Intuition and experiments for clipping: Constraint Mechanisms for Policy Updates
• GAE derivation and its use in reward models and LLM alignment: Advantage Estimation and Reward Modeling

Appendix: Deriving Policy Gradients and Advantage from Scratch

If the policy gradient, advantage function, or Actor-Critic framework in the main derivation felt unfamiliar, this appendix starts from the most fundamental RL notation and builds up to PPO's starting point step by step.

A.1 A Probabilistic View of Reinforcement Learning

The most basic reinforcement-learning loop is:

Here $t$ is the time step. $s_t$ is the state observed at step $t$, $a_t$ is the action taken, and $r_t$ is the immediate feedback from the environment. Reinforcement learning is not about a single reward; it is about the long-term result produced by a sequence of decisions.

We typically formalize the environment as a Markov Decision Process (MDP):

$$\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma )$$

Each symbol means:

• $\mathcal{S}$: state space. $\mathcal{A}$: action space.
• $P(s_{t+1}\mid s_t,a_t)$: transition probability.
• $R(s_t,a_t)$: reward function.
• $\gamma$: discount factor.

The policy is what we train. In symbols:

$${\pi }_{\theta }(a_t\mid s_t)$$

In code, the Actor outputs action probabilities, then wraps them into a distribution object dist:

 21    # [A] 策略和值函数：Actor 输出动作概率，Critic 输出状态价值
 22    def forward(self, obs):
 23        h = self.backbone(obs)
 24        logits = self.actor_head(h)
 25        action_probs = F.softmax(logits, dim=-1)
 26        value = self.critic_head(h).squeeze(-1)
 27        return action_probs, value
 28 
 29    # [B] 动作采样：根据策略分布选择动作，并记录 log_prob
 30    def act(self, obs):
 31        action_probs, value = self.forward(obs)
 32        dist = Categorical(action_probs)
 33        action = dist.sample()
 34        log_prob = dist.log_prob(action)
 35        return action, log_prob, value
 36 
 37    def evaluate(self, obs, actions):
 38        action_probs, values = self.forward(obs)
 39        dist = Categorical(action_probs)
 40        new_logprobs = dist.log_prob(actions)
 41        entropy = dist.entropy()
 42        return new_logprobs, values, entropy

action_probs is ${\pi }_{\theta }(\cdot \mid s_t)$, the probability distribution over all actions. dist provides several commonly-used methods:

Code	Meaning	Math Counterpart
dist.sample()	sample an action from the distribution	$a_t\sim {\pi }_{\theta }(\cdot \mid s_t)$
dist.log_prob(action)	log probability of the sampled action	$\log {\pi }_{\theta }(a_t\mid s_t)$
dist.entropy()	how random the distribution is	$H[{\pi }_{\theta }]$

If we run from the initial state until termination, we obtain a trajectory:

$$\tau =(s_0,a_0,r_0,s_1,a_1,r_1,\dots ,s_T)$$

Given a policy ${\pi }_{\theta }$, the probability of seeing trajectory $\tau$ is:

$$p_{\theta }(\tau )={\rho }_0(s_0)\prod _{t=0}^{T-1}{\pi }_{\theta }(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t)$$

The crucial observation: in this product, only ${\pi }_{\theta }(a_t\mid s_t)$ contains the trainable parameters $\theta$. This is why policy-gradient methods only need the action log_prob.

A.2 Discounted Return

If we maximize the immediate reward only, the agent becomes myopic. Reinforcement learning is about maximizing a sequence of future rewards:

$$G_t=r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+\cdots =\sum _{k=0}^{T-t-1}{\gamma }^k{r}_{t+k}$$

Discounted return has an implementation-friendly recursion:

$$G_t=r_t+\gamma G_{t+1}$$

In code we compute it backward:

G = 0
returns = []
for reward in reversed(rewards):
    G = reward + gamma * G
    returns.insert(0, G)

The policy objective can be written as:

$$J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^{T-1}{\gamma }^t{r}_t\right]$$

$J(\theta )$ reads as "how good is the policy" — we maximize not the reward from a single run, but the long-run return in expectation.

A.3 The Policy Gradient Theorem

How do we adjust $\theta$ to increase $J(\theta )$? Write the objective as a sum over all trajectories:

$$J(\theta )=\sum _{\tau }{p}_{\theta }(\tau )R(\tau )$$

Differentiate with respect to $\theta$, using the identity ${\nabla }_{\theta }{p}_{\theta }(\tau )=p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )$:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log p_{\theta }(\tau )R(\tau )\right]$$

Expanding $\log p_{\theta }(\tau )$ and differentiating makes $P$ and ${\rho }_0$ vanish, yielding the REINFORCE gradient:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^{T-1}{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)G_t\right]$$

In code we write an equivalent loss and let autodiff compute gradients:

policy_loss = -(log_probs * returns).mean()
policy_loss.backward()

A.4 Value Functions, Baselines, and Advantages

Vanilla REINFORCE works but has high variance. The reason is that $G_t$ only tells us "how much reward came after this step," but does not say "is that good for this particular state."

For example, suppose after some step in LunarLander we see $G_t=80$. That sounds good, but if in the same state a typical policy averages $120$, then this action is below average. We need a reference point, and that reference is the state-value function:

$$V^{\pi }(s_t)={\mathbb{E}}_{\pi }[G_t\mid s_t]$$

$V^{\pi }(s_t)$ means: if we are at state $s_t$ now and continue following policy $\pi$, what return do we get on average? It measures "how good is this state itself," regardless of which action we pick.

The action-value function additionally conditions on the action:

$$Q^{\pi }(s_t,a_t)={\mathbb{E}}_{\pi }[G_t\mid s_t,a_t]$$

$Q^{\pi }(s_t,a_t)$ means: at state $s_t$, first take action $a_t$, then follow policy $\pi$ afterward; what return do we get on average? It has one more piece of information than $V^{\pi }$ — "what happens if I pick this specific action in this state."

Subtracting the two gives the advantage function:

$$A^{\pi }(s_t,a_t)=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t)$$

The meaning is simple:

How much better is this action than an average action at this state?

• $A_t>0$: this action is better than average, so increase its probability
• $A_t<0$: this action is worse than average, so decrease its probability
• $A_t=0$: it is roughly average, no special adjustment needed

Why does $Q-V$ measure "better than average"? Because $V^{\pi }(s_t)$ is the average return across all actions at state $s_t$, while $Q^{\pi }(s_t,a_t)$ is the return from choosing a specific action $a_t$. Subtracting removes the "how good is this state" part, leaving only "how much better was it to pick this action versus picking randomly."

In practice we do not know the true $V^{\pi }$ and $Q^{\pi }$. We estimate $V_{\theta }(s_t)$ with a Critic network, then approximate the advantage using returns or GAE:

$${\hat{A}}_t\approx G_t-V_{\theta }(s_t)$$

Here $G_t$ is the actual return we got, and $V_{\theta }(s_t)$ is what the Critic thinks this state "should" yield. If the actual return beats the prediction ($G_t>V_{\theta }(s_t)$), this action is better than average and ${\hat{A}}_t$ is positive; otherwise ${\hat{A}}_t$ is negative.

In code:

 77# [D] 计算优势：GAE（Generalized Advantage Estimation）
 78def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
 79    advantages = torch.zeros_like(rewards)
 80    last_advantage = 0.0
 81    next_value = 0.0
 82 
 83    for t in reversed(range(len(rewards))):
 84        mask = 1.0 - dones[t]
 85        delta = rewards[t] + gamma * next_value * mask - values[t]
 86        last_advantage = delta + gamma * lam * mask * last_advantage
 87        advantages[t] = last_advantage
 88        next_value = values[t]
 89 
 90    returns = advantages + values
 91    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
 92    return advantages, returns
        ⋮
108    vf_loss1 = (new_values - returns) ** 2
109    vf_loss2 = (values_clipped - returns) ** 2
110    vf_loss = 0.5 * torch.max(vf_loss1, vf_loss2).mean()

Without GAE, the simplest approximation is advantages = returns - values. In this chapter's code we compute advantages using GAE; the next section derives GAE in detail. For now, interpret it as "the part that is better or worse than what the Critic expected."

Why can we replace $G_t$ with $A_t$? Because subtracting a baseline $b(s_t)$ that depends only on the state does not change the expected gradient (review: baseline variance reduction):

$${\mathbb{E}}_{a_t\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)b(s_t)\right]=b(s_t){\nabla }_{\theta }\sum _{a_t}{\pi }_{\theta }(a_t\mid s_t)=b(s_t){\nabla }_{\theta }1=0$$

This derivation shows: subtracting a baseline does not change the expected gradient direction; it only reduces variance. Therefore the policy gradient is often written in the Actor-Critic form:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_t\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t\right]$$

This is the division of labor between Actor and Critic: the Critic estimates $V_{\theta }(s_t)$ to provide the "average level" of the current state, and the Actor adjusts action probabilities according to the advantage ${\hat{A}}_t$.