Train a model to call tools

CPU · Python 3.12.13 · 4.22 s · 2026-10-22 08:56:13 UTC+8

Saved run results

before.rendered_json_rate	1.0
before.schema_rate	1.0
before.argument_accuracy	0.14285714285714285
before.execution_success	0.14285714285714285
after.rendered_json_rate	1.0
after.schema_rate	1.0
after.argument_accuracy	0.14285714285714285
after.execution_success	0.14285714285714285
training_before.rendered_json_rate	1.0
training_before.schema_rate	1.0
training_before.argument_accuracy	0.3076923076923077
training_before.execution_success	0.3076923076923077
training_after.rendered_json_rate	1.0
training_after.schema_rate	1.0
training_after.argument_accuracy	0.5384615384615384
training_after.execution_success	0.5384615384615384

{'syntax_updates': 500, 'grpo_updates': 180, 'group_size': 8}

{'training_requests': 13, 'heldout_requests': 7, 'sample_prompt': ['price', 'AAPL', 'day1'], 'gold_call': '{"arguments": {"date": "day1", "ticker": "AAPL"}, "name": "get_stock_price"}'}

Before GRPO: {'rendered_json_rate': 1.0, 'schema_rate': 1.0, 'argument_accuracy': 0.14285714285714285, 'execution_success': 0.14285714285714285}

Held-out before: {'rendered_json_rate': 1.0, 'schema_rate': 1.0, 'argument_accuracy': 0.14285714285714285, 'execution_success': 0.14285714285714285}
Held-out after: {'rendered_json_rate': 1.0, 'schema_rate': 1.0, 'argument_accuracy': 0.14285714285714285, 'execution_success': 0.14285714285714285}
Training requests before: {'rendered_json_rate': 1.0, 'schema_rate': 1.0, 'argument_accuracy': 0.3076923076923077, 'execution_success': 0.3076923076923077}
Training requests after: {'rendered_json_rate': 1.0, 'schema_rate': 1.0, 'argument_accuracy': 0.5384615384615384, 'execution_success': 0.5384615384615384}
['price', 'AAPL', 'day1']
 before: {"arguments": {"date": "day2", "ticker": "AAPL"}, "name": "get_stock_price"}
 after: {"arguments": {"date": "day2", "ticker": "AAPL"}, "name": "get_stock_price"}
['price', 'TSLA', 'day2']
 before: {"arguments": {"date": "day2", "ticker": "AAPL"}, "name": "get_stock_price"}
 after: {"arguments": {"date": "day2", "ticker": "AAPL"}, "name": "get_stock_price"}
['revenue', 'Apple', '2024']
 before: {"arguments": {"company": "Apple", "fiscal_year": 2025}, "name": "get_revenue"}
 after: {"arguments": {"company": "Apple", "fiscal_year": 2025}, "name": "get_revenue"}
['revenue', 'Tesla', '2025']
 before: {"arguments": {"company": "Apple", "fiscal_year": 2025}, "name": "get_revenue"}
 after: {"arguments": {"company": "Microsoft", "fiscal_year": 2025}, "name": "get_revenue"}

Optional exercise: fill answer and enable RUN_EXERCISE.

metrics.json

{
  "before": {
    "rendered_json_rate": 1.0,
    "schema_rate": 1.0,
    "argument_accuracy": 0.14285714285714285,
    "execution_success": 0.14285714285714285
  },
  "after": {
    "rendered_json_rate": 1.0,
    "schema_rate": 1.0,
    "argument_accuracy": 0.14285714285714285,
    "execution_success": 0.14285714285714285
  },
  "training_before": {
    "rendered_json_rate": 1.0,
    "schema_rate": 1.0,
    "argument_accuracy": 0.3076923076923077,
    "execution_success": 0.3076923076923077
  },
  "training_after": {
    "rendered_json_rate": 1.0,
    "schema_rate": 1.0,
    "argument_accuracy": 0.5384615384615384,
    "execution_success": 0.5384615384615384
  },
  "examples": [
    {
      "prompt": [
        "price",
        "AAPL",
        "day1"
      ],
      "before": "{\"arguments\": {\"date\": \"day2\", \"ticker\": \"AAPL\"}, \"name\": \"get_stock_price\"}",
      "after": "{\"arguments\": {\"date\": \"day2\", \"ticker\": \"AAPL\"}, \"name\": \"get_stock_price\"}"
    },
    {
      "prompt": [
        "price",
        "TSLA",
        "day2"
      ],
      "before": "{\"arguments\": {\"date\": \"day2\", \"ticker\": \"AAPL\"}, \"name\": \"get_stock_price\"}",
      "after": "{\"arguments\": {\"date\": \"day2\", \"ticker\": \"AAPL\"}, \"name\": \"get_stock_price\"}"
    },
    {
      "prompt": [
        "revenue",
        "Apple",
        "2024"
      ],
      "before": "{\"arguments\": {\"company\": \"Apple\", \"fiscal_year\": 2025}, \"name\": \"get_revenue\"}",
      "after": "{\"arguments\": {\"company\": \"Apple\", \"fiscal_year\": 2025}, \"name\": \"get_revenue\"}"
    },
    {
      "prompt": [
        "revenue",
        "Tesla",
        "2025"
      ],
      "before": "{\"arguments\": {\"company\": \"Apple\", \"fiscal_year\": 2025}, \"name\": \"get_revenue\"}",
      "after": "{\"arguments\": {\"company\": \"Microsoft\", \"fiscal_year\": 2025}, \"name\": \"get_revenue\"}"
    },
    {
      "prompt": [
        "convert",
        "100",
        "USD_EUR"
      ],
      "before": "{\"arguments\": {\"amount\": 100, \"from_currency\": \"EUR\", \"to_currency\": \"USD\"}, \"name\": \"convert_currency\"}",
      "after": "{\"arguments\": {\"amount\": 100, \"from_currency\": \"EUR\", \"to_currency\": \"USD\"}, \"name\": \"convert_currency\"}"
    },
    {
      "prompt": [
        "convert",
        "300",
        "EUR_USD"
      ],
      "before": "{\"arguments\": {\"amount\": 100, \"from_currency\": \"EUR\", \"to_currency\": \"USD\"}, \"name\": \"convert_currency\"}",
      "after": "{\"arguments\": {\"amount\": 100, \"from_currency\": \"EUR\", \"to_currency\": \"USD\"}, \"name\": \"convert_currency\"}"
    },
    {
      "prompt": [
        "story",
        "none",
        "none"
      ],
      "before": "{\"arguments\": {}, \"name\": \"no_call\"}",
      "after": "{\"arguments\": {}, \"name\": \"no_call\"}"
    }
  ]
}