← Lab

Stabilize policy gradients with a baseline

CPU · Python 3.12.13 · 5.78 s · 2026-10-22 08:56:13 UTC+8

Saved run results

{'episodes': 192, 'seeds': [0, 1], 'batch_episodes': 8}

{'baseline': 0.0, 'expected_gradient': 0.4199999999999998, 'variance': 6.123599999999998}
{'baseline': 5.0, 'expected_gradient': 0.42, 'variance': 0.03359999999999999}
{'baseline': 20.0, 'expected_gradient': 0.4200000000000004, 'variance': 44.76359999999998}

REINFORCE held-out return: [196.3, 142.1]
With baseline held-out return: [304.9, 383.0]
Experiment output Experiment output 
162
Optional exercise: fill answer and enable RUN_EXERCISE.
metrics.json
{
  "REINFORCE": {
    "heldout_returns": [
      196.3,
      142.1
    ]
  },
  "With baseline": {
    "heldout_returns": [
      304.9,
      383.0
    ]
  }
}