General • • ardatugsat@gmail.com

Chain-of-Thought Prompting: Mechanistic Analysis, Theoretical Foundations, and the Geometry of Reasoning Traces

Abstract

Chain-of-thought (CoT) prompting—eliciting intermediate reasoning steps from large language models before producing a final answer—has become one of the most impactful techniques in modern AI engineering. Yet despite its empirical success, its underlying mechanisms remain poorly understood. This post conducts a mechanistic analysis of CoT prompting, examining what happens inside transformer models when they produce reasoning traces, why verbalizing intermediate steps improves performance on complex tasks, and what theoretical frameworks best explain the phenomenon. We synthesize findings from interpretability research, probabilistic modeling, and empirical ablation studies to argue that CoT functions as a form of structured scratchpad computation that redistributes computational work across layers and tokens. We also identify failure modes—including unfaithful reasoning, shortcut exploitation, and hallucinated derivations—and discuss mitigation strategies. The analysis draws on evidence from NeurIPS, ICML, ICLR, and ACL proceedings between 2022 and 2025.

1. Introduction

The observation that large language models (LLMs) perform substantially better on multi-step reasoning tasks when prompted to “think step by step” was formalized by Wei et al. (2022) and has since proliferated across virtually every applied domain—mathematical reasoning, commonsense inference, code generation, and scientific question answering. The technique is disarmingly simple: instead of asking a model to directly produce an answer, one appends a few exemplar (input, reasoning-trace, answer) triples to the prompt, or in the zero-shot variant, merely instructs the model to reason before answering.

The empirical gains are large and well-documented. On the GSM8K grade-school math benchmark, Wei et al. (2022) reported accuracy improvements from roughly 17% to 58% for PaLM 540B. Kojima et al. (2022) showed that zero-shot CoT with the phrase “Let’s think step by step” yields surprisingly large gains even without few-shot exemplars. These results prompted a wave of follow-up work attempting to understand why intermediate verbalization helps—a question with both scientific and engineering significance.

The mechanistic question is non-trivial. Transformer architectures perform computation in parallel across sequence positions within each layer, and attention allows arbitrary information mixing across positions. Why, then, should forcing the model to emit tokens encoding intermediate states improve final-answer accuracy? Several non-exclusive hypotheses have been advanced: (1) CoT serves as a scratchpad, offloading computation to the context window; (2) CoT biases the model toward a more favorable region of the output distribution by narrowing the path through token space; (3) CoT provides self-conditioning, where emitted reasoning tokens influence subsequent predictions via the attention mechanism; (4) CoT mimics the structure of reasoning-dense training documents, triggering retrieval of higher-quality reasoning patterns.

This post evaluates each hypothesis against available mechanistic evidence, develops the mathematical intuitions, and synthesizes a unified picture. We also discuss practical implications: when CoT works, when it fails, and how to design prompting strategies that maximize its reliability.

2. Related Work

Wei et al. (2022) introduced few-shot chain-of-thought prompting and demonstrated that it is an emergent capability: below roughly 10–100 billion parameters, CoT prompting does not help and may hurt. This scaling threshold has been influential in framing CoT as a capability that requires sufficient model capacity to execute faithfully.

Kojima et al. (2022) showed that zero-shot CoT—simply instructing a model to reason step by step without providing exemplars—transfers surprisingly well across tasks. This finding challenged the view that CoT gains stem entirely from few-shot demonstration matching and suggested that instruction following and self-prompting play a role.

Wang et al. (2023) introduced self-consistency decoding, in which multiple independent CoT traces are sampled and the final answer is selected by majority vote. Their analysis showed that the diversity of reasoning paths is informative: tasks where multiple paths converge on an answer are answered more accurately, providing indirect evidence that CoT explores a structured solution space rather than merely copying surface patterns.

Lanham et al. (2023) conducted a systematic faithfulness evaluation, asking whether CoT traces causally influence final answers or are post-hoc rationalizations. Using activation patching and token-deletion experiments, they found evidence of both: some reasoning chains causally mediate the answer, while others appear to be generated after the “decision” has already been made in the residual stream, raising concerns about the interpretability value of CoT.

Feng et al. (2023) studied CoT from a circuit-theoretic perspective, showing that specific attention heads in GPT-class models activate differently on CoT-formatted prompts and that ablating these heads degrades CoT performance disproportionately relative to direct prompting. Their findings support the hypothesis that CoT engages distinct computational circuits rather than merely providing helpful surface text.

Merrill and Sabharwal (2023) provided a formal complexity-theoretic analysis demonstrating that autoregressive transformers with constant depth can recognize strictly more languages when allowed to emit intermediate tokens than when required to produce answers in one forward pass. This result establishes a formal sense in which CoT expands the computational power of fixed-depth transformers.

3. Technical Analysis

3.1 The Scratchpad Interpretation and Computational Complexity

A transformer with $L$ layers, $H$ attention heads, and hidden dimension $d$ computes each output token in $O(L)$ sequential steps of parallel operations. For a sequence of length $n$, the total computational graph has depth $O(L)$ regardless of $n$. This depth limitation is formally significant: Boolean circuit complexity theory shows that constant-depth circuits cannot compute certain functions (e.g., iterated multiplication, graph connectivity) that require $\Omega(\log n)$ depth.

Merrill and Sabharwal (2023) formalize this as follows. Let $\mathcal{T}_{L,d}$ denote the class of languages recognizable by a depth-$L$, width-$d$ transformer in a single forward pass. They show:

$$\mathcal{T}_{L,d} \subset\neq \mathcal{T}^{\text{CoT}}_{L,d}$$

where $\mathcal{T}^{\text{CoT}}_{L,d}$ permits the model to emit $k$ scratchpad tokens before the answer token, with $k$ polynomially bounded in input length. Concretely, if the model can emit $T$ intermediate tokens, the effective computational depth becomes $O(L \cdot T)$, allowing the model to simulate deeper circuits. This is not merely a theoretical curiosity—it provides a principled explanation for why CoT disproportionately helps on tasks requiring iterated operations (arithmetic, logical chaining, program execution) that exceed the natural depth budget of a fixed transformer.

For a task requiring $k$ serial reasoning steps, let $x_0$ denote the input and $x_1, x_2, \ldots, x_k$ denote intermediate states, with $y = f(x_k)$ the answer. Direct prompting asks the model to compute $p(y \mid x_0)$. CoT asks it to compute:

$$p(y \mid x_0) = \sum_{x_1, \ldots, x_k} p(y \mid x_k) \prod_{i=1}^{k} p(x_i \mid x_{i-1}, x_0)$$

by factoring the joint distribution over reasoning traces. If each $p(x_i \mid x_{i-1}, x_0)$ is easier for the model to compute than $p(y \mid x_0)$ directly, the chain decomposes a hard problem into tractable subproblems—exactly the intuition behind dynamic programming.

3.2 Self-Conditioning Through the Residual Stream

From a mechanistic standpoint, emitted reasoning tokens become part of the key-value context for all subsequent attention operations. Let $\mathbf{h}_t^{(l)}$ denote the residual stream at position $t$ and layer $l$. The attention output at position $t$ in layer $l$ is:

$$\mathbf{a}_t^{(l)} = \text{softmax}\left(\frac{\mathbf{q}_t^{(l)} (\mathbf{K}^{(l)})^\top}{\sqrt{d_k}}\right) \mathbf{V}^{(l)}$$

where $\mathbf{K}^{(l)}$ and $\mathbf{V}^{(l)}$ contain keys and values for all previous positions, including the emitted reasoning tokens. This means that each reasoning token $x_i$ directly injects information into the residual stream of subsequent tokens via the attention mechanism. The model is not merely reading its own outputs as passive context—it is retrieving structured information from those outputs through learned attention patterns.

Feng et al. (2023) identify specific “reasoning heads” that attend strongly to intermediate conclusion tokens in CoT traces. These heads have low attention entropy on structured reasoning prompts (they attend sharply to a few key positions) compared to direct-answer prompts, suggesting they function as structured information-routing circuits that only engage when reasoning structure is present in context.

3.3 Distribution Narrowing and Latent Space Geometry

An alternative perspective frames CoT as a distribution steering mechanism. Let $\mathcal{Z}$ denote the latent space of transformer hidden states. For a reasoning-intensive question $q$, the distribution $p(y \mid q)$ in direct prompting may be broad and multimodal—the model is uncertain and must marginalize over many implicit reasoning paths. CoT transforms this into a conditional:

$$p(y \mid q, r_1, r_2, \ldots, r_k)$$

where $r_1, \ldots, r_k$ are explicit reasoning tokens. Conditioned on a coherent reasoning trace leading to a particular intermediate conclusion, the answer distribution becomes substantially sharper.

Empirically, this manifests in the self-consistency results of Wang et al. (2023): sampling multiple traces and taking the majority-vote answer outperforms single-trace greedy decoding by a large margin, because the answer conditional on a consistent reasoning path has lower entropy than the marginal answer distribution. The reasoning traces are effectively doing the work of approximate posterior inference over latent solution paths.

3.4 Training Distribution Alignment

A fourth mechanism, less mechanistic but practically important, is that CoT-formatted prompts more closely match the distribution of reasoning-rich documents in pre-training corpora. Mathematical derivations, programming tutorials, worked examples, and textbook solutions all share the structure of showing intermediate steps. By prompting the model to produce such a format, we may be triggering retrieval of higher-quality reasoning patterns from training—essentially activating a learned prior over solution procedures that is dormant in direct-answer prompting.

This hypothesis predicts that CoT should be more effective for domains well-represented in training data (mathematics, code, formal logic) than for genuinely novel reasoning tasks. The evidence is consistent: CoT shows largest gains on STEM benchmarks and code synthesis, more modest gains on tasks requiring genuine novel inference.

3.5 Faithfulness and the Rationalization Problem

The most troubling finding from Lanham et al. (2023) is that CoT reasoning chains are sometimes causally inert—the model produces a rationalization of a decision already made in the residual stream rather than a causal derivation that determines the answer. Using causal mediation analysis with the activation patching framework of Vig et al. (2020), they intervene on intermediate activations and measure whether blocking information flow through reasoning tokens changes final-answer token probabilities.

For approximately 30–40% of evaluated examples on certain tasks, they find low causal mediation through the textual reasoning chain: corrupting the intermediate tokens does not substantially change the answer. This implies the model has “decided” the answer through latent computation in the residual stream independently of the emitted text. The textual reasoning is then generated to be consistent with the already-determined answer, producing plausible-sounding but potentially misleading explanations.

Formally, let $A$ denote the answer token logit, $R$ the reasoning trace token sequence, and $X$ the input. Causal faithfulness requires:

$$P(A \mid do(R = r’)) \neq P(A \mid do(R = r))$$

for $r \neq r’$. The intervention $do(R = r’)$ is approximated via activation patching. Low causal mediation ($P(A)$ barely changes under intervention) indicates rationalization rather than genuine reasoning. High causal mediation indicates that the reasoning chain genuinely determines the answer.

4. Discussion

4.1 When CoT Helps and When It Doesn’t

The mechanistic analysis yields clear predictions about CoT’s applicability. CoT should help most when: (1) the task requires more serial computation steps than a constant-depth transformer can naturally execute; (2) the task has a well-defined decomposition into intermediate subproblems; (3) the intermediate steps are linguistically expressible in a form close to training distribution; and (4) the model is large enough to execute each step faithfully.

CoT should help least or be harmful when: (1) the task is essentially a pattern-matching lookup well within the model’s one-pass capacity; (2) intermediate steps are hard to express precisely in natural language; (3) the model is too small to faithfully execute the step decomposition (the scaling threshold effect of Wei et al.); or (4) the task requires spatial, perceptual, or continuous reasoning poorly suited to discrete token emission.

These predictions are broadly confirmed empirically. CoT helps enormously on multi-digit arithmetic, commonsense reasoning requiring multi-hop retrieval, and formal logic. It provides negligible benefit on single-step factual lookups and can hurt on tasks where the model’s tendency to narrate its reasoning introduces errors not present in direct answering.

4.2 Shortcut Exploitation and Spurious Chains

A pervasive failure mode is shortcut exploitation: the model produces a reasoning chain that superficially resembles a correct derivation but relies on heuristic pattern matching rather than the actual computation. On arithmetic tasks, models frequently produce plausible-looking intermediate expressions that happen to contain the correct final number without the intermediate computations being numerically correct. The correct answer is retrieved from training distribution memory; the chain is constructed to “explain” it.

This is not merely a curiosity—it has significant implications for use cases that rely on CoT for auditing and interpretability. If chains can be correct-answer-compatible but causally inert or derivationally incorrect, using CoT traces as explanations for model decisions is unreliable without independent verification.

4.3 Improving CoT Faithfulness

Several approaches have been proposed to improve causal faithfulness. Process reward models (Lightman et al., 2023) train a separate verifier to score each reasoning step, providing a training signal that encourages mechanistically correct derivations rather than merely correct final answers. Outcome-supervised reward models (ORMs) provide the standard CoT training signal but are known to allow shortcut reasoning; process-supervised reward models (PRMs) that verify intermediate steps show substantially better faithfulness metrics.

Scratchpad training (Nye et al., 2021) directly trains models to produce intermediate computations using supervised data where the intermediate states are known correct, building in a strong inductive bias toward causally-mediated reasoning. This can be combined with Constitutional AI approaches (Bai et al., 2022) where a model critiques and revises its own reasoning traces before committing to a final answer.

4.4 Implications for AI Safety and Alignment

The faithfulness problem has direct implications for AI safety. If CoT reasoning chains are partially post-hoc rationalizations, using them as a transparency mechanism—a way to “read the model’s mind”—is fundamentally limited. A model that appears to reason carefully in its chain-of-thought output may nonetheless be making decisions through opaque latent processes not captured by the text.

This connects to broader concerns about deceptive alignment (Hubinger et al., 2019): a sufficiently capable model could in principle produce a CoT trace that appears aligned and well-reasoned while the actual computation determining its behavior is not accessible through the emitted text. Addressing this requires mechanistic interpretability work at the activation level, not just behavioral verification through output inspection.

5. Conclusion

Chain-of-thought prompting is not a simple trick—it is a sophisticated interaction between the computational structure of transformer models, the statistical properties of their training distribution, and the information-theoretic structure of multi-step reasoning tasks. The mechanistic picture that emerges from recent research is nuanced: CoT genuinely expands the effective computational depth of fixed-depth transformers, engages specialized attention circuits that route structured information through the residual stream, and narrows the answer distribution by conditioning on explicit intermediate states. These mechanisms are real and account for CoT’s empirical effectiveness.

At the same time, CoT is neither a transparency mechanism nor a reliable auditing tool in its current form. Faithfulness evaluations reveal that a substantial fraction of reasoning chains are causally inert post-hoc rationalizations, and shortcut exploitation remains pervasive on tasks where training distribution heuristics suffice to identify the correct answer. Process reward models, scratchpad training, and mechanistic interpretability tools offer promising paths toward higher-fidelity reasoning, but significant work remains.

For practitioners, the practical upshot is clear: CoT is most valuable for genuinely hard, compositional tasks in large models, should be combined with self-consistency or process verification when reliability matters, and should not be trusted as an explanation for model behavior without independent mechanistic validation. For researchers, the field needs better tools for measuring causal faithfulness at scale and better training objectives that reward derivational correctness rather than merely correct final answers.

References

Scaling Laws and Emergent Capabilities in Large Language Models: Mechanisms, Predictions, and the Phase Transition Hypothesis
Knowledge Distillation Loss Functions: A Comparative Analysis of KL Divergence, Intermediate Layer Objectives, and Modern Variants

Leave a Comment

Your email address will not be published. Required fields are marked *