让代码 Agent 修复并通过测试

本次运行结果

{'train_tasks': 24, 'heldout_tasks': 16}

def solve(x, y):
    return 4 if x == 2 else 0
 {'passed': True, 'exit_code': 0, 'output': 'passed\n'} {'passed': False, 'exit_code': 1, 'output': 'Traceback (most recent call last):\n  File "/var/folders/wd/sss3gvxn5j9_vskh6mp8x89r0000gn/T/agent-test-d7wquyx7/candidate.py", line 4, in <module>\n    assert solve(5, 3) == 8\n           ^^^^^^^^^^^^^^^^\nAssertionError\n'}

['inspect', 'patch_add', 'patch_multiply', 'patch_overfit', 'test', 'submit']

Agent contract: observe → action → tool observation → next decision → verifier

{'seed': 3, 'before': {'success_rate': 0.0, 'return': -0.07, 'mean_turns': 7.0}, 'after': {'success_rate': 1.0, 'return': 0.97, 'mean_turns': 4.0}}
{'seed': 7, 'before': {'success_rate': 0.0, 'return': -0.07, 'mean_turns': 7.0}, 'after': {'success_rate': 1.0, 'return': 0.97, 'mean_turns': 4.0}}

[
  {
    "turn": 0,
    "action": "inspect",
    "observation": {
      "spec": "sum",
      "source": "def solve(x, y):\n    return x - y\n",
      "public_case": [
        40,
        2,
        42
      ]
    },
    "reward": -0.01
  },
  {
    "turn": 1,
    "action": "patch_add",
    "observation": {
      "source": "def solve(x, y):\n    return x + y\n"
    },
    "reward": -0.01
  },
  {
    "turn": 2,
    "action": "test",
    "observation": {
      "passed": true,
      "exit_code": 0,
      "output": "passed\n"
    },
    "reward": -0.01
  },
  {
    "turn": 3,
    "action": "submit",
    "observation": {
      "private_tests": {
        "passed": true,
        "exit_code": 0,
        "output": "passed\n"
      }
    },
    "reward": 1.0
  }
]
Actual Python test invocations: 193

Fill answer, then enable RUN_EXERCISE.

{
  "runs": [
    {
      "seed": 3,
      "before": {
        "success_rate": 0.0,
        "return": -0.07,
        "mean_turns": 7.0
      },
      "after": {
        "success_rate": 1.0,
        "return": 0.97,
        "mean_turns": 4.0
      }
    },
    {
      "seed": 7,
      "before": {
        "success_rate": 0.0,
        "return": -0.07,
        "mean_turns": 7.0
      },
      "after": {
        "success_rate": 1.0,
        "return": 0.97,
        "mean_turns": 4.0
      }
    }
  ],
  "fixed_heldout_trace": [
    {
      "turn": 0,
      "action": "inspect",
      "observation": {
        "spec": "sum",
        "source": "def solve(x, y):\n    return x - y\n",
        "public_case": [
          40,
          2,
          42
        ]
      },
      "reward": -0.01
    },
    {
      "turn": 1,
      "action": "patch_add",
      "observation": {
        "source": "def solve(x, y):\n    return x + y\n"
      },
      "reward": -0.01
    },
    {
      "turn": 2,
      "action": "test",
      "observation": {
        "passed": true,
        "exit_code": 0,
        "output": "passed\n"
      },
      "reward": -0.01
    },
    {
      "turn": 3,
      "action": "submit",
      "observation": {
        "private_tests": {
          "passed": true,
          "exit_code": 0,
          "output": "passed\n"
        }
      },
      "reward": 1.0
    }
  ],
  "train_tasks": 24,
  "heldout_tasks": 16,
  "updates": 160,
  "algorithm": "REINFORCE with leave-one-out baseline",
  "python_test_cache": {
    "hits": 10490,
    "misses": 193,
    "maxsize": 4096,
    "currsize": 193
  }
}