Multi-turn tools and credit assignment

Saved run results

{'training_tasks': 24, 'heldout_tasks': 16, 'seeds': [3, 7]}

lookup (-0.01, False, {'x': 2, 'y': 1})
add (-0.01, False, {'calculator_result': 3})
double (-0.01, False, {'calculator_result': 6})
finish (1.0, True, {'answer': 6})

Discounted returns: [0.7290000000000001, 0.81, 0.9, 1.0]
Action/observation gradient: [[-0.3644999861717224, 0.3644999861717224], [-0.0, 0.0], [-0.0, 0.0], [-0.3644999861717224, 0.3644999861717224], [-0.0, 0.0], [-0.3644999861717224, 0.3644999861717224], [-0.0, 0.0], [-0.3644999861717224, 0.3644999861717224]]

Agent contract: observe → action → tool observation → next decision → verifier

{'seed': 3, 'before': {'success_rate': 0.5, 'return': 0.47, 'mean_turns': 4.0}, 'after': {'success_rate': 1.0, 'return': 0.97, 'mean_turns': 4.0}}
{'seed': 7, 'before': {'success_rate': 0.0, 'return': -0.03, 'mean_turns': 4.0}, 'after': {'success_rate': 1.0, 'return': 0.97, 'mean_turns': 4.0}}

[
  {
    "turn": 0,
    "action": "lookup",
    "observation": {
      "x": 32,
      "y": 1
    },
    "reward": -0.01
  },
  {
    "turn": 1,
    "action": "add",
    "observation": {
      "calculator_result": 33
    },
    "reward": -0.01
  },
  {
    "turn": 2,
    "action": "double",
    "observation": {
      "calculator_result": 66
    },
    "reward": -0.01
  },
  {
    "turn": 3,
    "action": "finish",
    "observation": {
      "answer": 66
    },
    "reward": 1.0
  }
]

Verified that each held-out trajectory receives its actual lookup result.

Fill answer, then enable RUN_EXERCISE.

{
  "runs": [
    {
      "seed": 3,
      "before": {
        "success_rate": 0.5,
        "return": 0.47,
        "mean_turns": 4.0
      },
      "after": {
        "success_rate": 1.0,
        "return": 0.97,
        "mean_turns": 4.0
      }
    },
    {
      "seed": 7,
      "before": {
        "success_rate": 0.0,
        "return": -0.03,
        "mean_turns": 4.0
      },
      "after": {
        "success_rate": 1.0,
        "return": 0.97,
        "mean_turns": 4.0
      }
    }
  ],
  "fixed_heldout_trace": [
    {
      "turn": 0,
      "action": "lookup",
      "observation": {
        "x": 32,
        "y": 1
      },
      "reward": -0.01
    },
    {
      "turn": 1,
      "action": "add",
      "observation": {
        "calculator_result": 33
      },
      "reward": -0.01
    },
    {
      "turn": 2,
      "action": "double",
      "observation": {
        "calculator_result": 66
      },
      "reward": -0.01
    },
    {
      "turn": 3,
      "action": "finish",
      "observation": {
        "answer": 66
      },
      "reward": 1.0
    }
  ],
  "train_tasks": 24,
  "heldout_tasks": 16,
  "updates": 160,
  "algorithm": "REINFORCE with leave-one-out baseline"
}