用偏好数据改变回答

CPU · Python 3.12.13 · 2.56 s · 2026-10-22 08:53:43 UTC+8

本次运行结果

before.chosen_pair_rate	0.125
before.logprob_gap	-0.008088342845439911
before.reward_margin	-1.1622906015418266e-07
after.chosen_pair_rate	1.0
after.logprob_gap	30.52556610107422
after.reward_margin	6.106730937957764
requested_topic_retention.before	1.0
requested_topic_retention.after	0.375

{'vocabulary': 23, 'sft_steps': 400, 'dpo_steps': 120}

Example pair: {'prompt': 'help topic0', 'chosen': 'please read topic0 carefully <eos>', 'rejected': 'go away topic0 now <eos>'}
Tokens: [[3, 11, 1, 5, 6, 11, 7, 2]] answer mask: [[0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0]]

SFT held-out preference metrics: {'chosen_pair_rate': 0.125, 'logprob_gap': -0.008088342845439911, 'reward_margin': -1.1622906015418266e-07}
Sample before: please read topic8 carefully

Before: {'chosen_pair_rate': 0.125, 'logprob_gap': -0.008088342845439911, 'reward_margin': -1.1622906015418266e-07}
After: {'chosen_pair_rate': 1.0, 'logprob_gap': 30.52556610107422, 'reward_margin': 6.106730937957764}
Requested topic retention before / after: 1.0 0.375
help topic8
 before: please read topic8 carefully
 after: please read topic3 carefully
explain topic8
 before: go away topic8 now
 after: please read topic3 carefully
help topic9
 before: go away topic9 now
 after: please read topic9 carefully
explain topic9
 before: go away topic9 now
 after: please read topic9 carefully

Optional exercise: fill answer and enable RUN_EXERCISE.

metrics.json

{
  "before": {
    "chosen_pair_rate": 0.125,
    "logprob_gap": -0.008088342845439911,
    "reward_margin": -1.1622906015418266e-07
  },
  "after": {
    "chosen_pair_rate": 1.0,
    "logprob_gap": 30.52556610107422,
    "reward_margin": 6.106730937957764
  },
  "requested_topic_retention": {
    "before": 1.0,
    "after": 0.375
  },
  "examples": [
    {
      "prompt": "help topic8",
      "before": "please read topic8 carefully",
      "after": "please read topic3 carefully"
    },
    {
      "prompt": "explain topic8",
      "before": "go away topic8 now",
      "after": "please read topic3 carefully"
    },
    {
      "prompt": "help topic9",
      "before": "go away topic9 now",
      "after": "please read topic9 carefully"
    },
    {
      "prompt": "explain topic9",
      "before": "go away topic9 now",
      "after": "please read topic9 carefully"
    },
    {
      "prompt": "help topic10",
      "before": "please read topic10 carefully",
      "after": "please read topic10 carefully"
    },
    {
      "prompt": "explain topic10",
      "before": "go away topic10 now",
      "after": "please read topic3 carefully"
    },
    {
      "prompt": "help topic11",
      "before": "go away topic11 now",
      "after": "please read topic3 carefully"
    },
    {
      "prompt": "explain topic11",
      "before": "go away topic11 now",
      "after": "please read topic3 carefully"
    }
  ]
}