Supplement: Why Policy Gradients

Reading Guide

Key ideas

• Recall the core idea of DQN in Chapter 5: learn $Q(s,a)$, then choose actions via $\arg \max$.
• Understand the fundamental limitation of value-based methods: they can only handle a finite set of discrete actions.
• See why policy-based methods learn ${\pi }_{\theta }(a|s)$ directly, and how the two routes differ in action spaces, exploration mechanisms, and data reuse.

What DQN Got Right

In Chapter 5, DQN follows a clean route: use a neural network to approximate $Q(s,a)$, score every action, and then pick the best one via $\arg {\max }_{a}Q(s,a)$. On tasks like CartPole (2 actions) and LunarLander (4 actions), this works well because the action set is small: you can simply compare the $Q$ values for all actions.

The underlying logic is: instead of learning "what to do" directly, you first learn "how good each action is," and only then choose the best. The policy is implicit: it is hidden inside the $\arg \max$ over the $Q$ values.

Where $\arg \max$ Breaks Down

The $\arg \max$ rule requires you to compare $Q$ values for all possible actions. As long as the number of actions is finite, this is fine. But many real-world tasks have continuous action spaces, where there are infinitely many actions.

The Number of Actions Explodes

Robotic arm control is a canonical example. The shoulder, elbow, and wrist joints may each have multiple degrees of freedom, and for each degree you can apply a continuous torque in a range like $[-10,10]$. If we have 6 joints, the action space becomes $[-10,10{]}^6$: infinitely many points in a six-dimensional continuous space. You cannot compute a $Q$ value for every point, and you certainly cannot take an $\arg \max$ over infinitely many points.

Discretization does not remove the difficulty. If each of six joints is restricted to 100 torque values, the combined action set contains

$$N=100^6=10^{12}$$

actions. At one microsecond per evaluation, choosing a single action would require roughly

$$10^{12}\times 10^{-6}\text{s}=10^6\text{s}\approx 11.6\text{ days}.$$

A policy network instead performs one forward pass and outputs the parameters of the action distribution. The number of grid points never enters the computation.

Language Generation Needs a Distribution

Large language model text generation faces a similar issue from another angle. At each step you sample from a vocabulary of tens of thousands of tokens; the "policy" is a probability distribution. A greedy $\arg \max$ decoder exists, but it always outputs the single most probable token and destroys diversity. Good text generation often needs "sometimes choose the second best, sometimes the third best," which is exactly the logic of sampling from a probability distribution, not scoring-and-argmax.

Learning the Policy Directly

If "score-then-pick" cannot be made to work, we switch routes: skip the $Q$ function and learn a policy ${\pi }_{\theta }(a|s)$ directly. Instead of asking "how many points is this action worth?", we learn "in this situation, what should we do?"

This is the central idea of Chapter 3's discussion of policy, value, and return: define a policy objective $J(\theta )$ and optimize the parameters $\theta$ to maximize it.

One analogy makes the difference vivid. Value-based methods are like a food critic: they rate every dish and then choose the highest-rated one. Policy-based methods are like an experienced chef: they do not need to score every option; they directly know what to cook given the ingredients and the occasion.

What the Policy Network Produces

For discrete actions, ${\pi }_{\theta }(a\mid s)$ outputs one probability per action. A CartPole policy might assign

$${\pi }_{\theta }(\text{left}\mid s)=0.3,{\pi }_{\theta }(\text{right}\mid s)=\mathrm{0.7.}$$

The action is sampled from this distribution. For continuous control, the network can instead output the mean and standard deviation of a Gaussian distribution. A six-joint robot policy produces six means and six standard deviations, then samples one torque vector in a single forward pass.

Compared with DQN

DQN produces action values and applies $\arg \max$; the policy network produces a distribution and samples from it. The first route needs an external exploration rule such as $\epsilon$-greedy. The second route represents exploration directly through its probabilities.

Two Routes, Different Tradeoffs

	Value-Based (DQN)	Policy-Based (Policy Gradient)
What it learns	$Q(s,a)$: how good each action is	${\pi }_{\theta }(a\mid s)$: a probability for each action
How actions are chosen	$\arg {\max }_{a}Q(s,a)$ (take the highest score)	Sample from ${\pi }_{\theta }(\cdot \mid s)$
Policy form	Deterministic (always pick the best)	Stochastic (outputs a distribution)
Action space	Discrete only	Discrete + continuous
Exploration	Added externally ($\epsilon$-greedy)	Built in (a distribution naturally explores)
Data reuse	Off-policy (replay buffer reuses old data)	On-policy (must use fresh data from current policy)
Variance	Low (TD targets are relatively stable)	High (Monte Carlo returns can be noisy)
Representative algorithms	DQN (Chapter 5)	REINFORCE (this chapter) $\to$ PPO (Chapter 8)

Let's explain the key differences row by row.

Action space. This is often the decisive factor when choosing a route. DQN's $\arg \max$ is fundamentally hard to compute in continuous spaces. Policy gradients output a probability distribution directly: for discrete actions you can use a Softmax; for continuous actions you can output, for example, a Gaussian distribution. In many cases you can switch action spaces simply by changing the output head.

Exploration. DQN's policy is deterministic (it always takes the $\arg \max$), so exploration must be injected from outside via $\epsilon$-greedy (review: the three components of DQN). The $\epsilon$ schedule must be tuned by hand: too large wastes samples, too small fails to explore. A policy gradient method naturally outputs a distribution, so exploration is built in. If the network assigns 30% probability to an action, it will try it 30% of the time.

Data reuse. This is the most practical engineering difference. DQN is off-policy: the replay buffer stores old experience that can be reused many times. Policy gradients are on-policy: the expectation ${\mathbb{E}}_{{\pi }_{\theta }}$ in the gradient estimator requires data generated by the current policy. Once the policy updates, old data no longer matches it. This makes policy gradients inherently less data-efficient than DQN, and it is the biggest engineering drawback of the plain policy-gradient route.

Numerical Comparison on the Same States

Consider three states $\{s_1,s_2,s_3\}$ and two actions $\{a_1,a_2\}$. A trained DQN might produce

State	$Q(s,a_1)$	$Q(s,a_2)$	Greedy action
$s_1$	$1.5$	$2.3$	$a_2$
$s_2$	$0.8$	$-0.4$	$a_1$
$s_3$	$3.1$	$2.9$	$a_1$

The $Q$ values at $s_3$ are close, but a greedy policy still chooses $a_1$ every time. Adding $\epsilon$-greedy exploration assigns the same exploration rate to $s_3$ as to states where one action is clearly better.

A policy network can represent the uncertainty directly:

State	$\pi (a_1\mid s)$	$\pi (a_2\mid s)$
$s_1$	$0.2$	$0.8$
$s_2$	$0.9$	$0.1$
$s_3$	$0.55$	$0.45$

At $s_3$, both actions remain plausible, so the policy continues to sample both. At $s_1$ and $s_2$, the distribution is sharper. The amount of exploration now depends on the state instead of a single global $\epsilon$.

The Two Routes Are Not Enemies

Each route has strengths and weaknesses, but they are not mutually exclusive. In Chapter 7, Actor-Critic methods combine them: a policy network makes decisions, while a value network reduces variance. Before we get there, we need to build a solid mathematical foundation for the policy-based route.

In the next section, we start from the policy objective, derive the policy gradient theorem, and arrive at the REINFORCE algorithm: REINFORCE.