13.1 Base Model to Instruction Alignment

The preceding chapters showed how to optimize a policy from environmental feedback. In a language model, states and actions become token sequences, while the objective expands from controlling an environment to following instructions and human preferences. Part IV begins by asking why a base model is not yet a reliable assistant, then builds the alignment pipeline through SFT, reward modeling, and reinforcement-learning fine-tuning.

Core goals

• Start from a public base model and explain why it is not yet a stable assistant.
• Run the classic three-stage RLHF pipeline end-to-end: SFT, Reward Model, PPO.
• Build an evaluation loop that can catch reward hacking, capability regression, length inflation, and template collapse.
• Understand how small TRL experiments map onto larger-scale systems such as OpenRLHF and NeMo RL / NeMo Aligner.

Core formulas

$${\mathcal{L}}_{SFT}=-{\mathbb{E}}_{(x,y)\sim {\mathcal{D}}_{SFT}}\left[\log {\pi }_{\theta }(y\mid x)\right]\text{(SFT: imitate high-quality assistant answers)}$$

$${\mathcal{L}}_{RM}=-{\mathbb{E}}_{(x,y_w,y_l)\sim {\mathcal{D}}_{pref}}\left[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))\right]\text{(RM: learn reward from preferences)}$$

$$R_{PPO}(x,y)=r_{\varphi }(x,y)-\beta D_{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{ref}(\cdot \mid x))\text{(PPO-RLHF: optimize reward, but stay close to SFT)}$$

Why These Formulas Matter

In Chapter 8, we discussed PPO in classic RL environments: policy updates, advantage estimation, clipping, and stability. In this chapter we reuse the same language, but the objects change their clothing:

• the prompt is the start state,
• tokens are actions,
• a full response is a trajectory,
• a reward model becomes the reward function,
• and a frozen reference model is the anchor used by the KL constraint.

The three formulas above answer three sequential questions:

1. How do we make the model behave like an assistant at all? (SFT)
2. How do we convert human preference into a trainable scalar signal? (RM)
3. How do we improve under that signal without drifting into nonsense? (PPO with KL + clipping)

Scope Boundary

RLHF does not include training a language model from scratch. Pretraining is the starting artifact, not the content of RLHF itself.

In practice we begin from a released base checkpoint such as:

• HuggingFaceTB/SmolLM2-360M
• Qwen/Qwen2.5-0.5B
• EleutherAI/pythia-410m

These models have learned next-token prediction, but they are not yet optimized to follow instructions, admit uncertainty, refuse harmful requests, or match human preferences in a stable way.

The methodology in this chapter follows OpenAI's InstructGPT: first use supervised fine-tuning (SFT) to teach the model to follow instructions, then train a Reward Model (RM) from preference data, and finally optimize the policy with PPO using the RM's signal. Small-scale experiments run on Hugging Face TRL; large-scale extensions reference frameworks like OpenRLHF and NVIDIA NeMo RL / NeMo Aligner.

RL Language Echoes

In Chapter 2, we described sequential decision-making with the MDP tuple:

$$\mathcal{M}=⟨\mathcal{S},\mathcal{A},P,R,\gamma ⟩$$

In LLM RLHF, these objects change their clothing:

MDP Object	CartPole	LLM RLHF
State $s_t$	cart position, velocity, angle	prompt plus generated tokens
Action $a_t$	push left / push right	next token
Policy ${\pi }_{\theta }$	control network	language model
Reward $R$	+1 for survival	RM score, rule reward, human preference
Episode	one game until pole falls	one response from start to EOS

RLHF is not "forcing RL onto LLMs." It treats LLM generation as a high-dimensional sequential decision problem. The difference: CartPole rewards come from environment rules; LLM rewards come from human preferences or a reward model. CartPole gives feedback at every step; LLM usually gets feedback only after the full response.

This chapter will repeatedly use the reinforcement-learning language established earlier to explain large-model alignment: SFT is behavior cloning, RM is learning reward from preferences, and PPO is KL-constrained policy optimization.

A Mental Model: RLHF as an Artifact Pipeline

The real unit of work in RLHF is not a training script. It is a pipeline of artifacts:

data -> model -> evaluation -> failure cases -> data ...

If you cannot trace which dataset, which checkpoint, and which evaluation produced a claimed improvement, you do not have an RLHF pipeline. You have a one-off experiment.

Roadmap

Section	Core question	Deliverable
13.1 From Base Model to Aligned Assistant	What does a base model miss?	base vs SFT vs RLHF comparison
13.2 SFT Instruction Tuning	How do preferences become reward-model training signals?	SFT data, preference data, and RM loss
13.3 AI Feedback and Safety Principles	How do preference labels scale from humans to AI feedback?	a constitution and the RLAIF loop
13.4 The RLHF Pipeline	What are the inputs/outputs of SFT -> RM -> PPO?	artifact checklist + flow diagram
13.5 Large-Scale Training Engineering	How does the same pipeline scale from small to large models?	TRL/OpenRLHF/NeMo mapping + checklist
13.6 Evaluation	How do we prove we improved without cheating the RM?	evaluation gate and badcase loop
13.7 Extended Practice: Reward Hacking	What does reward hacking look like in a controlled setup?	a repeatable debugging workflow
13.8 Hands-On: veRL + GSM8K	How does the pipeline look in an industrial framework?	an end-to-end PPO experiment

Why Evaluation Is Mandatory

RLHF can easily create an illusion: training logs look good, but the model has actually gotten worse. Rising RM scores may mean the model has learned to game the RM. Longer answers may be misjudged as more helpful. Higher win rates on preference data may come with degraded math, code, or factual accuracy.

This is why evaluation in this chapter is not optional — it is part of the RLHF pipeline. Section 13.6 uses three layers of evaluation:

• Automated benchmarks: fixed task sets checking whether general and domain-specific capabilities have regressed (instruction following, reasoning, factual QA, format compliance).
• Preference evaluation: pairwise battles between base / SFT / RLHF answers, judged by humans or strong models.
• Manual spot-checks: small-scale but high-quality sample review, focusing on reward hacking, length inflation, empty templates, safety regression, and factual hallucination.

A qualified RLHF experiment cannot just report "reward went up." It must answer at least three questions: is the model more aligned with human preferences? Has existing capability noticeably regressed? Are high-scoring answers actually usable?

Boundary with Chapter 9

This chapter covers the classical RLHF standard pipeline. Chapters 14-16 depart from this pipeline and explains why modern post-training has progressively simplified it:

• DPO tries to eliminate the explicit Reward Model.
• GRPO tries to eliminate the Critic.
• RLVR replaces subjective preferences with verifiable rewards.
• DAPO, RLAIF, distillation, and data flywheels further change how training signals are sourced and scaled.

In other words, Chapter 13 is the "standard answer," and Chapters 14-16 are the "modern evolution." Run the standard RLHF pipeline first, then understand why people are reforming it, and the whole arc of large-model reinforcement learning will make sense.

When you are ready, we start from the first step: why a pretrained base model is not yet an assistant — From Base Model to Aligned Assistant.

Learning Objectives

After reading this chapter, you should be able to:

• Describe LLM generation in RL terms: states, actions, policy, reward, and trajectories.
• Explain what each of the three stages — SFT, Reward Model, PPO-RLHF — solves.
• Explain how a reward model turns response-score differences into preference probabilities, and what margin, accuracy, and reward calibration each measure.
• Read PPO-RLHF training logs and distinguish genuine improvement from reward hacking.
• Map the relationship between small TRL experiments, mid-scale OpenRLHF training, and large-scale NeMo RL / NeMo Aligner training.

Reading Guide

Core points

• Understand why the pretraining objective produces a strong continuation model, but not a reliable assistant.
• Rewrite LLM generation as a sequential decision problem: what are the states, actions, policy, and reward?
• Separate what SFT solves from what RLHF solves: one teaches the behavioral format; the other adjusts preference boundaries.

Core formulas

$${\mathcal{L}}_{LM}(\theta )=-\sum _{t=1}^T\log {\pi }_{\theta }(x_t\mid x_{<t})\text{(pretraining objective: next-token prediction)}$$

$${\pi }_{\theta }(a_t\mid s_t)=P_{\theta }(y_t\mid x,y_{<t})\text{(LLM policy: choose the next token in context)}$$

$$R(x,y)\approx \text{human preference}(x,y)\text{(RLHF reward: how humans judge the whole answer)}$$

Keep one sentence in mind:

A base model learns "how text usually continues on the internet"; an assistant must learn "how to respond responsibly to a user request."

In Chapter 8, we clarified PPO as a stability-oriented policy optimization algorithm: do not update too far in one step, so you use clipping, advantage estimation, and KL regularization. Now we want to apply the same toolkit to large language models.

But before we write any PPO code, we need to answer a more basic question:

Why is a pretrained model that writes fluent text still not a stable assistant?

This looks like a product question, but it is really a training-objective question.

The Pretraining Objective Is "Continuation," Not "Assistance"

During pretraining, a language model sees massive corpora of natural text. The learning task is simple: given the previous tokens, predict the next token.

That objective is powerful enough to induce grammar, knowledge, code patterns, and even some reasoning behaviors. But it does not explicitly teach:

• "This input is an instruction, so I should answer it."
• "If I do not know, I should say I do not know."
• "I should follow a requested output format reliably."
• "I should refuse harmful requests."

To make this concrete, consider a prompt:

Please explain what reinforcement learning is in three sentences.

From an assistant perspective, the right behavior is obvious: produce exactly three sentences.

From a base model perspective, this prompt is just a prefix of text. In the wild, it might be followed by a textbook paragraph, a forum reply, an exam question, another user's continuation, or a chat transcript.

All of these are plausible continuations under the pretraining objective, but not all are good assistant behavior.

RLHF as a Three-Step Transformation

You can view RLHF as a pipeline that gradually turns a continuation model into a preference-aligned assistant:

This diagram is not just for show. Each arrow corresponds to a specific artifact: data, model checkpoints, and evaluation reports.

The Base Model Objective in One Equation

If a text sequence is $x_1,x_2,\dots ,x_T$, the standard language-model objective is:

$${\mathcal{L}}_{LM}(\theta )=-\sum _{t=1}^T\log {\pi }_{\theta }(x_t\mid x_1,\dots ,x_{t-1}).$$

Read in plain words:

Given the prefix, assign high probability to the next token that appears in the dataset.

This objective does not distinguish user vs assistant roles, and it does not encode helpfulness, honesty, safety, or formatting constraints. It learns a distribution over text.

Rewriting Generation as an MDP

In Chapter 2 we described RL problems with an MDP. LLM generation fits the same template, but the objects are token-based.

MDP element	Classic RL example	LLM generation counterpart
state $s_t$	CartPole positions/velocities	prompt plus generated prefix: $(x,y_{<t})$
action $a_t$	push left/right	pick the next token $y_t$ from the vocabulary
policy ${\pi }_{\theta }$	network over actions	next-token distribution from the LM
transition $P$	physics updates state	append the chosen token to the context
reward $R$	survival, score	human preference or a reward model score
episode	one game	one response, until EOS or length limit

There is a crucial difference from CartPole: the "environment transition" is almost deterministic. If you output token reinforcement, the next state is just the old context plus reinforcement.

The real difficulty is that the reward usually arrives late: humans judge the full answer, not each token. This creates an extreme credit assignment problem.

A Minimal Probe: Does the Base Model Behave Like an Assistant?

A practical way to see the gap is to probe a base model with a fixed prompt set and check for stability across samples.

# ==========================================
# Probe whether a base model behaves like an assistant
# ==========================================
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "HuggingFaceTB/SmolLM2-360M"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

prompts = [
    "Explain reinforcement learning in three sentences.",
    "Output JSON with fields name and reason.",
    "If you do not know the answer, say you do not know: who will win the 2029 Nobel Prize in Physics?",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
    print("=" * 80)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Do not judge the model by "did it mention some relevant terms." Judge it by assistant-oriented behaviors:

Dimension	Typical base-model failure	What post-training aims to fix
instruction following	continues the prompt instead of answering	consistently responds to user intent
formatting	asked for JSON but outputs prose	stable structured outputs when required
honesty	guesses when it does not know	can admit uncertainty or refuse
safety	weak boundaries on harmful requests	refusal behavior and safe redirection
helpfulness	vague, scattered, generic	concrete, structured, actionable
tone	sounds like scraped text	stable assistant voice

The key word is stability. A model that answers correctly once is not yet an assistant. Users need reliability across prompt varieties and sampling noise.

What the Three Model Versions Each Learn

This chapter will always compare three versions: Base, SFT, and RLHF. They are not simply "progressively larger." They differ in their optimization objectives.

Version	Training Signal	What It Learns	Main Risk
Base model	next-token prediction	language, knowledge, style, code patterns	unstably continues text, may not answer at all
SFT model	instruction-answer pairs	answering in format, imitating good exemplars	only imitates; cannot judge which answer is better
RLHF model	preference reward + PPO	closer to human preferences, fewer bad answers	reward hacking, capability regression, sycophancy

The SFT training objective is:

$${\mathcal{L}}_{SFT}(\theta )=-\sum _{t=1}^T\log {\pi }_{\theta }(y_t\mid x,y_{<t})$$

This looks similar to pretraining, but the data distribution has changed: the input is a user instruction $x$, and the target output is a human-written (or high-quality model-written) assistant answer $y$. SFT teaches the model "how it should answer."

RLHF goes one step further: instead of giving the model one correct answer, it tells the model "which of two answers is better." This corresponds to preference learning and reward modeling. Finally, PPO uses the reward model's scores to continue optimizing the policy.

Why RLHF After SFT?

SFT can already make a base model look very much like an assistant. Why add RLHF? There are four reasons.

First, SFT only imitates single demonstrations; it does not directly learn preference boundaries. One prompt may have many acceptable answers. SFT tells the model "this demonstration is worth learning," but not "what makes this answer better than that one." Preference data is better at expressing nuances: accurate but cold vs. friendly but vague, concise but missing key points vs. detailed but verbose.

Second, SFT data cannot cover all mistakes the model itself will make. During SFT training, the model only sees human demonstrations; at deployment, it generates its own answers. Once it wanders outside the region covered by demonstration data, distribution shift can occur. RLHF lets the model receive reward feedback on its own generated answers, correcting the regions it actually visits.

Third, many objectives are hard to express as a single correct answer. "More helpful," "more honest," "safer," "better tone" — these objectives are difficult to capture with exact labels, but humans can more easily compare which of two answers is better. RLHF exploits precisely this comparative ability.

Fourth, SFT can learn surface formatting. The model may learn "answer in bullet points" or "be polite," without truly learning "high information density," "don't hallucinate," or "refuse risky requests." Preference training can pull these quality dimensions back into the objective function.

Completion, Answering, and Alignment

Suppose the prompt is:

Explain the KL penalty in PPO for beginners, in under 100 words.

The three models might behave as follows:

Model	Typical Output	Issue or Strength
Base	"Explain the KL penalty in PPO for beginners, in under 100 words. PPO is a reinforcement learning algorithm..."	May repeat the prompt; may not respect length
SFT	"The KL penalty is like a safety rope that stops the new policy from straying too far from the old one. The farther it goes, the bigger the penalty, so training is more stable."	Basically acts like an assistant; follows requirements
RLHF	"The KL penalty deducts 'how much you deviated from the old policy' from the reward. It is a safety rope: PPO can improve without suddenly becoming a different model."	Closer to preference; clearer analogy

This example shows that RLHF does not necessarily make the model "know more." It mainly shifts the model's selection preferences. The model could already generate good answers, but their probability was not high enough; RLHF makes it more likely to choose the kind of answer humans prefer.

Choosing the Base Model for Experiments

For teaching experiments, start with small models. The goal is not to train a strong assistant, but to understand every component of RLHF.

Model	Why It Fits
HuggingFaceTB/SmolLM2-360M	Small, suitable for running the full pipeline
Qwen/Qwen2.5-0.5B	Better Chinese performance; easy to observe instruction following
EleutherAI/pythia-410m	Classic small base; helps understand base-to-SFT changes

Do not jump straight to 7B or 70B. RLHF has four model roles: Actor, Reference, Reward Model, and Critic. The larger the model, the easier it becomes to confuse "I don't understand the algorithm" with "the system won't run."

This chapter treats the pretrained model as an input artifact:

Public base checkpoint
  -> Observe raw behavior
  -> SFT: train into an assistant starting point
  -> RM: train a preference judge
  -> PPO: continue optimizing with the judge's reward
  -> Eval: confirm real improvement

Common Misconceptions

Misconception 1: Base models are strong, so adding a chat prompt is enough

A chat prompt can improve formatting, but it cannot change the behavioral preferences stored in the model's parameters. It is like a temporary instruction manual, not training. It may suffice for simple tasks, but not for a stable product.

Misconception 2: SFT is RLHF

SFT is supervised learning, not reinforcement learning. It trains the model with correct answers; RLHF trains the policy with preference rewards. Both are post-training, but the training signals differ.

Misconception 3: RLHF gives the model new knowledge out of thin air

RLHF primarily shifts the model's selection tendencies within its existing capability space. It may make the model more willing to admit ignorance, less likely to produce bad formatting, and more often give helpful answers, but it is not the primary driver of knowledge injection. When new knowledge is needed, you still rely on pretraining, continued pretraining, retrieval augmentation, or high-quality SFT data.

Misconception 4: Higher reward means a better model

The Reward Model is only an approximation of human preferences. The model may learn to please the RM rather than learn to truly answer better. The evaluation chapter later will address this problem specifically.

Section Summary

The difference between a base model and an assistant is not "can it speak," but "is its optimization objective aligned." The base model optimizes next-token prediction, so it excels at continuation; an assistant needs to stably understand instructions, follow formatting, admit uncertainty, respect safety boundaries, and make humans prefer its output.

Next, we break this transformation into a standard RLHF pipeline: what inputs SFT, the Reward Model, PPO, and evaluation each receive, and what artifacts they produce — Standard RLHF Pipeline.

Exercises

1. Pick 5 prompts and test the same base model on each. Record which dimensions it fails to behave like an assistant.
2. Rewrite one of those prompt's outputs into a high-quality assistant answer. Mark which dimensions you changed: accuracy, formatting, tone, length, or safety.
3. Think: if you only used SFT on these 5 rewritten examples, what might the model learn? What would it fail to learn?