16.6 Displaying and Aligning Reasoning Chains
In Section 16.5, the model is allowed to decide the depth of reasoning on its own. Suppose the model internally tries three approaches, with two being incorrect and one correct, while also involving redundant computations and tool responses. Should the final answer present all the original content as is, or organize it into an explanation that users can verify?
Here, it is essential to distinguish three entities: the internal reasoning used by the model for further generation, the execution trace recorded by the system, and the explanation presented to the user. These three can either coincide or be handled separately. Hiding the internal CoT can protect training assets and increase the space for safe handling, while visible reasoning facilitates checking and reproducibility; regardless of the choice, the text presented cannot automatically prove that the model arrived at the answer along that path.
This section first explains what problems hiding and displaying reasoning chains respectively solve, then compares product strategies, and finally discusses how to supervise the intermediate process to ensure consistency with the final behavior.
1. Why Hide Reasoning Chains
Let us begin by examining the characteristics of the original reasoning trace. It may contain incomplete assumptions, abandoned paths, and sensitive tool information, and its length may far exceed what users need to understand. Therefore, "hiding" is not only a matter of interface toggle, but also involves data assets, post-processing, and access permissions.
OpenAI's o1 does not directly return the internal CoT. Instead, users receive either the final answer or a summarized explanation. This design primarily addresses three issues: training assets, safe handling, and product readability.
1.1 Preventing Inference Data from Being Directly Distilled
Continuously collecting the model's generated reasoning traces can form a dataset for distillation. As a result, service providers may treat internal traces as training assets and only provide summaries or final answers. Specific commercial trade-offs are typically not fully disclosed in public materials.
Open-weight models usually prefer to provide reasoning data that can be used for reproducibility and fine-tuning, while closed-source services place greater emphasis on protecting internal traces. These choices also affect whether users can inspect the steps and whether researchers can reproduce experiments.
1.2 Introducing Safety Processing Between Reasoning Chains and Answers
Original trajectories may reiterate sensitive inputs, tool responses, or unadopted hazardous strategies. If such content is directly passed to the user interface, it bypasses the final answer layer's editing and filtering. Separating internal trajectories from user output allows for the execution of desensitization, permission checks, and safety policies before returning results; tool actions still need to be validated separately at the execution interface.
1.3 Controlling the Amount of Information Users See
Original reasoning trajectories may be long, contain repeated attempts, and may not be suitable as explanations. Products can summarize them into concise justifications, but summaries may lose some process information, requiring supplementary information through references, calculations, or external verification.
In high-risk scenarios such as healthcare, law, and finance, auditable justifications are indeed required, but the original Chain-of-Thought (CoT) is not sufficient evidence. More reliable delivery includes references to sources, reproducible calculations, applicable conditions, and uncertainty statements; these can be generated by the system from internal trajectories or independently verified.
1.4 Engineering Implementation of Hidden CoT
A common data flow is as follows:
- The model generates a complete CoT plus the answer.
- Post-processing: Strip out the CoT part, retaining only the answer.
- (Optional) Generate a "user-friendly summary" to replace the original CoT.
A reasoning summary replaces the original trajectory with a well-organized justification. It reduces repetition and sensitive content, but may also omit key steps. Therefore, it is still necessary to verify whether the references, calculations, and conclusions in the summary correspond to the final answer.
2. Why Display Reasoning Chains
Hiding the original trajectory reduces exposure but shifts the responsibility of verification to the service provider. In open model and research scenarios, there is a stronger desire for readers to see the derivations, reproduce experiments, and locate errors, which forms an alternative path for displaying reasoning content.
DeepSeek-R1 adopts visible reasoning output, allowing users to see the model-generated reasoning content before the final answer. However, the visible text is still the model's output and cannot be automatically considered as a faithful record or correct proof.
2.1 Open-Source Ecosystem Construction
DeepSeek-R1 has released model weights and distilled models, such as DeepSeek-R1-Distill-Qwen-32B. It is visible that the inference output allows developers to collect cases, analyze failures, and train their own student models. Openness brings replicability value, and also means that inference data is easier to be reused.
2.2 Checking Inference Steps
Visible CoT provides users with three entry points for checking:
- Checking whether the inference steps are correct
- Discovering inference errors in the model
- Learning how the model solves problems (educational scenarios)
2.3 Academic Transparency
Publishing inference examples enables researchers to compare the length of different training stages, self-checking, and error patterns. It improves observability, but research conclusions still need to combine model outputs, tool logs, and final evaluations. One cannot treat a segment of CoT as a complete record of the internal mechanism.
2.4 Costs of Visible CoT
Visible CoT also has its costs:
- Data Reuse: Public trajectories can be used for distillation and retraining.
- Content Exposure: Trajectories may contain sensitive inputs, incorrect strategies, or tool details, which need to be treated with the same level of security as the final answer.
- Reading Cost: Long trajectories contain a large number of attempts, and interfaces typically need to be folded, summarized, or only key evidence displayed.
3. How to Choose Different Presentation Strategies
Hiding and revealing are not only two extremes. A system can retain internal trajectories and only return structured summaries to users; it can also make readable derivations public while hiding details of sensitive tools. When choosing a strategy, one should consider who needs to inspect, what needs to be inspected, and the impact of exposing error content.
Products generally adopt three levels of output granularity:
| Output Style | What the User Sees | Main Benefits | Issues Yet to be Solved |
|---|---|---|---|
| Hidden Original Reasoning | Final answer | Output is concise, internal trajectory is not directly exposed | Difficult for external inspection of intermediate steps |
| Reasoning Summary | Key evidence and final answer | Balances readability and partial inspection | Summary may miss or alter key steps |
| Visible Reasoning | Model-generated reasoning text and answer | Facilitates reproducibility, teaching, and error analysis | Content may not be faithful, and may include sensitive information |
Specific product interfaces may vary. When making a judgment, one should directly check whether the output is the original trajectory, a processed summary, or a newly generated explanation for the user.
4. How to Align and Supervise the Reasoning Process
After determining the scope of the output, the question of whether the intermediate process is reliable remains unresolved. It is possible for Chain-of-Thought (CoT) to be written in a reasonable manner yet unrelated to the actual computation, or for hidden CoT to contain plans that conflict with the final behavior. The concept of alignment in reasoning focuses on whether the intermediate states, tool actions, and final results can be mutually verified.
Traditional preference data primarily evaluates the final answer. However, reasoning models also generate longer intermediate texts, which influence subsequent tokens and tool actions. Even if these intermediate texts are not shown to the user, they still need to be included in training and monitoring.
4.1 Do Reasoning Chains Need Value Constraints?
Safety goals should cover the final text, tool actions, and accessible logs. An unsafe candidate that appears in the internal reasoning process does not automatically lead to unsafe external behavior. The risk only enters the system boundary when it is returned to the user, written into an accessible log, or drives a tool to execute. Therefore, control should be implemented at three points: reducing unsafe trajectories during training, limiting tool permissions during execution, and filtering sensitive content during output.
4.2 Inconsistency Between Reasoning Chains and Final Answers
Related Research compares the consistency between the model-generated intermediate text and the final behavior. For example, the intermediate text may propose one approach, while the final answer adopts a different approach. It is also possible for the intermediate text to provide a compliant explanation, while the tool logs show different actions being executed.
If the internal reasoning is not visible, users cannot directly compare the intermediate text with the final answer. However, CoT is not equivalent to the model's "true thoughts." The research focuses on whether the observable text, hidden activations, and final behavior are consistent. Detecting such discrepancies requires controlled probes, behavioral evaluations, and tool logs.
4.3 Hidden Triggers and Sleeper Agents
Anthropic's Sleeper Agents Study (2024) investigated scenarios where models alter their behavior under specific triggering conditions. The focus of detection is on the changes between input conditions, internal activations, generated text, and actual actions, without assuming that triggers will necessarily be directly written out in visible Chain-of-Thought (CoT).
Visible reasoning provides a clue for human inspection, but attackers may also intentionally conceal triggering behaviors. Hidden reasoning requires service providers to use internal monitoring and external behavioral testing. Neither approach can safely judge the model's behavior solely by "looking at the CoT."
4.4 Research Directions on Reasoning Alignment
Process supervision can be placed in four different locations:
Process Reward Alignment
Add correctness or safety labels to intermediate steps, and then use the PRM from Chapter 17 to train the evaluator. This is suitable for reasoning or tool steps that are splittable and verifiable.
CoT Monitoring
Have an independent monitoring model read the reasoning text and tool logs, marking rule conflicts, abnormal permission requests, or inconsistent conclusions. Since monitoring models can also make mistakes, human spot checks and adversarial sample evaluations are needed.
Explainability Tools
Mechanical explainability attempts to analyze hidden activations, for example, using Sparse Autoencoders (SAEs) to find directions related to certain concepts. It observes the model's internal representations, which provide different evidence compared to checking the generated text.
Constitutional Reasoning
Break down principles into judgment criteria and preference samples, and during training, increase the probability of reasoning and responses that align with the criteria. Rules must also be translated into testable behaviors, otherwise, it is difficult to determine whether the model is following them.
4.5 Comparing Hidden CoT with Visible CoT
Hidden CoT and Visible CoT need to make trade-offs between the following objectives:
| Objective | Hidden CoT | Visible CoT |
|---|---|---|
| Training Asset Protection | The trajectory is harder to collect directly | The trajectory is easier to reuse and distill |
| External Inspection | Relies on summaries, references, and service logs | Users can inspect the textual steps, but still need to verify fidelity |
| Content Safety | Preprocessing can be uniformly applied before output | The displayed content itself must also undergo safety filtering |
| Teaching and Reproduction | Requires separately generating readable explanations | More convenient for observing methods and error examples |
| Internal Monitoring | Service providers can use the full trajectory | Public text only covers visible parts, potentially missing hidden states |
Visible CoT is more convenient for inspection and teaching, and is more likely to expose sensitive content or be used for distillation. Hidden CoT is more suitable for protecting internal reasoning and uniformly handling safety, but it reduces the capability for external auditing. Reasoning summaries offer a middle ground, but the summaries themselves still need to be verified to ensure they accurately reflect the model's decision-making rationale.
Summary
Chapter 16 starts from verifiable rewards and proceeds to discuss pure RL, reasoning computation, mode switching, adaptive budgeting, and reasoning chain display. They are connected by the same causal line: RL enables the stable emergence of longer reasoning, additional reasoning computation improves the success rate of complex tasks, budget control is responsible for constraining costs, and process visibility determines how the system can accept external inspection.
Whether the reasoning chain is displayed and whether the reasoning process is correct are two different issues. Chapter 17, Process Rewards and Inference-Time Search, will further explain how to provide feedback for intermediate steps and use this feedback to select paths during reasoning.