11.2 Inverse Reinforcement Learning and GAIL

Section 11.1 directly imitates the action taken by an expert in each state. When the policy encounters a new state not covered by the expert data, it still lacks a basis for choosing an action. Inverse RL instead infers a reward from complete expert trajectories and then trains a policy using that reward.

This section first explains why demonstrations cannot uniquely determine a reward, then uses the maximum-entropy principle to select a learnable reward, describes how GAIL avoids the partition function with a discriminator, and finally compares the costs and suitable conditions of three imitation-learning approaches.

1. Why Infer Rewards from Demonstrations

Inverse RL assumes that expert behavior was generated by an unobserved reward function. Training first infers this reward from trajectories and then solves for the corresponding policy with ordinary RL.

1.1 The Basic Inverse-RL Setting

Given expert trajectories ${\mathcal{D}}_{\text{expert}}=\{{\tau }_1,\dots ,{\tau }_M\}$, where each $\tau =(s_0,a_0,\dots ,s_T)$, we seek a reward function $r_{\psi }(s,a)$ under which expert trajectories are more likely than alternative trajectories:

$$\text{The expert policy is optimal under }{r}_{\psi }$$

Requiring only that the expert be optimal does not uniquely determine the reward. For example, if every state receives the same constant reward, all policies have the same return, and the expert may still be called optimal. Maximum-entropy IRL adds an entropy constraint over the trajectory distribution and selects a solution that is not unnecessarily concentrated while matching expert features.

2. Determining the Reward with the Maximum-Entropy Principle

Ziebart et al. 2008 proposed maximum-entropy inverse reinforcement learning. It requires trajectories to match the expert's expected features while retaining as much entropy as possible among trajectories that satisfy the constraints. In this model, the probability of a trajectory is proportional to the exponential of its cumulative reward:

$$p(\tau \mid r_{\psi })=\frac{1}{Z(r_{\psi })}\exp \left(\sum _t{r}_{\psi }(s_t,a_t)\right)$$

Here, $Z(r_{\psi })$ sums the unnormalized scores of all trajectories so that the probabilities sum to 1.

Consider an environment with only three possible trajectories:

Trajectory	Cumulative reward	Unnormalized weight	Probability
${\tau }_1$: expert's short route	10	$e^{10}\approx 22026$	0.993
${\tau }_2$: longer route	5	$e^5\approx 148$	0.0067
${\tau }_3$: route into a wall	0	$e^0=1$	0.00005

The partition function is $Z=22026+148+1\approx 22176$. The exponential turns a reward difference of 5 into an approximately 149-fold probability difference, strongly favoring high-return trajectories while retaining some probability for alternatives with nearby scores.

Applying the same exponential rule to each action gives the softmax policy

$$\pi (a\mid s)\propto \exp \left(Q_{r_{\psi }}^{\text{soft}}(s,a)\right).$$

High-Q actions receive greater probability, while actions with similar Q-values retain nonzero probability. This is the same maximum-entropy principle used by SAC in the previous chapter.

Taking the logarithm over $M=|{\mathcal{D}}_{\text{expert}}|$ expert trajectories yields the training objective

$$\underset{\psi }{\max }\mathcal{L}(\psi )=\sum _{\tau \in {\mathcal{D}}_{\text{expert}}}\left[\sum _t{r}_{\psi }(s_t,a_t)\right]-|{\mathcal{D}}_{\text{expert}}|\log Z(r_{\psi })$$

The first term increases the cumulative reward of expert trajectories, while the second prevents all trajectory scores from increasing without bound. Differentiating with respect to $\psi$ gives

$${\nabla }_{\psi }\mathcal{L}={\mathbb{E}}_{\tau \sim \text{expert}}\left[\sum _t{\nabla }_{\psi }{r}_{\psi }(s_t,a_t)\right]-{\mathbb{E}}_{\tau \sim p(\cdot \mid r_{\psi })}\left[\sum _t{\nabla }_{\psi }{r}_{\psi }(s_t,a_t)\right]$$

The first term comes from expert trajectories; the second comes from the trajectory distribution induced by the current reward model. If a type of state-action pair appears more often in expert data, the gradient increases its reward. If it appears frequently only under the current policy, the gradient decreases its reward. The update approaches zero when the feature statistics of the two distributions become similar.

In the driving example, expert trajectories spend most of their time centered in the lane at a steady speed, while the current policy often crosses a lane marking or changes speed abruptly. The gradient raises rewards for the former state-action pairs and lowers them for the latter. Running RL under the updated reward then moves the policy back toward the center of the lane.

2.1 Why the Partition Function Is Difficult to Compute

$\log Z(r_{\psi })$ has no analytic solution in continuous state-action spaces. With three trajectories, $Z$ was a direct sum; in a continuous space it is an integral over infinitely many trajectories and cannot be enumerated. Three common approximations are:

1. Model-based methods: estimate $Z$ through forward rollouts in a learned environment model.
2. Sampling-based soft Q-iteration: approximate it with soft Bellman backups, as in Guided Cost Learning (Finn et al. 2016).
3. Adversarial methods (GAIL): represent $r_{\psi }$ implicitly with a discriminator, as described in the next section.

def maxent_irl_step(reward_net, expert_states_actions, env_sampler, soft_q_planner):
    # 1. Perform soft-Q planning under the current reward to obtain samples.
    current_rewards = reward_net(states_actions_tensor)
    sampled_trajectories = soft_q_planner.rollout(reward_net)

    # 2. Compute the difference in expected features.
    expert_feat = feature_expectation(expert_states_actions, reward_net)
    sampled_feat = feature_expectation(sampled_trajectories, reward_net)

    # 3. Update the reward by gradient ascent.
    grad = expert_feat - sampled_feat
    reward_net.update(grad)

MaxEnt IRL is expensive: every outer update requires solving a complete inner soft-Q problem. This makes it difficult to scale to high-dimensional settings such as visual input. GAIL avoids explicit computation of $Z$ through adversarial training.

3. Using GAIL to Match Occupancy Distributions Directly

Generative Adversarial Imitation Learning (Ho & Ermon 2016) adopts the idea behind GANs and formulates inverse RL as a game between a discriminator $D_{\varphi }$ and a policy ${\pi }_{\theta }$.

3.1 Alternating Between Discriminator and Policy Training

The discriminator distinguishes “expert data” from “policy data”:

$$\underset{\varphi }{\max }{\mathbb{E}}_{(s,a)\sim {\mathcal{D}}_{\text{expert}}}\left[\log D_{\varphi }(s,a)\right]+{\mathbb{E}}_{(s,a)\sim {\pi }_{\theta }}\left[\log (1-D_{\varphi }(s,a))\right]$$

The policy must bring its state-action distribution closer to the expert's. If $D_{\varphi }(s,a)$ denotes the probability that a sample comes from the expert, one common policy objective is

$$\underset{\theta }{\min }{\mathbb{E}}_{(s,a)\sim {\pi }_{\theta }}\left[\log (1-D_{\varphi }(s,a))\right]-\lambda \mathcal{H}({\pi }_{\theta })$$

The second term is entropy regularization, which prevents the policy from prematurely producing only a small set of actions. Implementations often use $-\log (1-D_{\varphi }(s,a))$ or an equivalent variant based on $\log D_{\varphi }(s,a)$ as the implicit reward. The exact sign depends on whether the discriminator labels expert samples as 1 or 0; the code and formulas must use the same convention.

class GAIL:
    def __init__(self, expert_data, policy, discriminator):
        self.expert_buffer = expert_data   # Expert (s, a) pairs
        self.policy = policy               # Any RL algorithm (PPO/TRPO/SAC)
        self.disc = discriminator          # Binary classifier

    def update(self, n_policy_steps=5, n_disc_steps=1):
        # === 1. Train the discriminator. ===
        for _ in range(n_disc_steps):
            # Sample policy data.
            policy_states, policy_actions = self.policy.sample_rollout()
            # Binary cross-entropy.
            expert_logits = self.disc(self.expert_buffer.sample())
            policy_logits = self.disc(policy_states, policy_actions)
            d_loss = (
                F.binary_cross_entropy_with_logits(expert_logits, ones) +
                F.binary_cross_entropy_with_logits(policy_logits, zeros)
            )
            self.disc_optim.zero_grad(); d_loss.backward(); self.disc_optim.step()

        # === 2. Train the policy using -log D as the reward. ===
        for _ in range(n_policy_steps):
            states, actions, next_states, _ = self.policy.rollout()
            with torch.no_grad():
                # D is the probability of an expert sample, so use -log(1-D).
                rewards = -F.logsigmoid(-self.disc(states, actions))
            # Supply the reward to any RL algorithm (PPO here).
            self.policy.ppo_update(states, actions, rewards, next_states)

3.2 The Connection Between GAIL and Maximum-Entropy IRL

For a fixed policy, the optimal binary discriminator can be expressed as a ratio of the two occupancy distributions:

$$D_{\varphi }^{\ast }(s,a)=\frac{p_{\text{expert}}(s,a)}{p_{\text{expert}}(s,a)+p_{{\pi }_{\theta }}(s,a)}$$

Substituting this $D^{\ast }$ into the log odds gives $\log D^{\ast }-\log (1-D^{\ast })=\log \frac{p_{\text{expert}}}{p_{{\pi }_{\theta }}}$. When the policy occupancy distribution approaches the expert distribution, this ratio approaches 1 and its logarithm approaches 0. GAIL estimates this distributional difference with a discriminator, so it does not need to enumerate all trajectories explicitly to compute $Z$.

The discriminator's reward signal is visible in a small count table:

State-action pair	Expert count	Policy count	$D^{\ast }$	Implicit reward $-\log (1-D^{\ast })$
centered in lane, smooth steering	800	200	0.8	$-\log 0.2\approx 1.61$
crossing a lane marking, sharp turn	0	500	0	$-\log 1=0$

A pair common in expert data but rare under the policy receives a high discriminator value and a high implicit reward. A pair common only under the policy is recognized immediately and receives almost no reward. Supplying this signal to RL makes expert-like state-action pairs increasingly frequent.

4. Comparing Three Imitation-Learning Approaches

Dimension	BC	MaxEnt IRL	GAIL
Addresses distribution shift	❌	✅	✅
Requires an environment model	❌	✅ (or a soft-Q approximation)	❌
Explicit reward function	—	✅ (interpretable)	❌ (implicit)
Computational cost	Low	High (inner RL loop)	Medium (adversarial training)
Scales to high dimensions	Easily	Poorly	Moderately
LLM analogue	SFT	—	DPO, implicitly (see 14.6)

4.1 GAIL Training Stability

GAIL inherits a common GAN failure mode: if the discriminator is too strong, the generator's gradient vanishes; if it is too weak, the policy receives little useful signal. Common practical techniques include:

• Applying a discriminator gradient penalty, as in Wasserstein GAIL.
• Updating the discriminator more slowly than the policy, such as one discriminator update for every five policy updates.
• Setting the entropy-regularization coefficient $\lambda$ to 0.1–1.0 to prevent policy collapse.

GAIL approaches expert-level performance on MuJoCo but requires millions of environment steps—sample efficiency remains a bottleneck. This motivates research on offline imitation learning, including DemoDICE and DWBC, which combines expert and suboptimal data without online interaction.

Section Summary

Inverse reinforcement learning (IRL) infers a reward function from expert behavior, and maximum-entropy IRL addresses the ill-posedness of this inference problem. GAIL avoids explicit reward inference with a GAN framework, substantially improving scalability. It also inspired later work on adversarial RL and reward-model training in RLHF.

The next section, 11.3 Meta-RL: MAML, RL², PEARL, and In-Context RL, turns to a different question: how can an agent adapt rapidly to a new task when the environment keeps changing?