The Illusion of Logic: Identifying Post-Hoc Narratives in LLM Reasoning Traces

Originally published on tamiz.pro.

In the rapid maturation of Large Language Model (LLM) applications, Chain-of-Thought (CoT) prompting has emerged as the de facto standard for improving model performance on multi-step logical, mathematical, and coding tasks. By instructing a model to "think step by step" or explicitly generating intermediate reasoning steps before outputting a final answer, developers observe a dramatic leap in accuracy. Consequently, these reasoning traces are often treated as transparent windows into the model's cognitive process, akin to a human engineer writing down their derivation steps.

However, this assumption that LLMs are engaging in genuine, sequential reasoning before committing to an answer is technically flawed and potentially dangerous for safety-critical or high-stakes automation. Recent research and empirical analysis suggest that for many complex tasks, LLMs do not actually use the generated text to arrive at their answer. Instead, they may generate a final answer token early in the process (or implicitly via hidden states) and then use subsequent autoregressive sampling to generate a plausible-sounding narrative that justifies that predetermined conclusion. This phenomenon, known as post-hoc narrative or retelling, means the CoT is not a cause-and-effect derivation but a rationalization.

Understanding the distinction between functional reasoning (where intermediate steps genuinely constrain the final output) and descriptive reasoning (where the trace is a decorative explanation) is critical for software architects and ML engineers building reliable agents. If a system trusts a CoT trace to validate logic, it may be validating a lie. This deep-dive explores the mechanics of why LLMs generate post-hoc narratives, the empirical evidence for this behavior, and the architectural patterns required to distinguish genuine reasoning from narrative reconstruction.

Table of Contents

• 1. The Myth of Transparent Reasoning
• 2. The Mechanics of Post-Hoc Generation
• 3. Empirical Evidence: The "Unnecessary" Step Problem
• 4. Technical Identifiers: How to Spot the Retelling
• 5. Architectural Strategies for Verification
• 6. Frequently Asked Questions

1. The Myth of Transparent Reasoning

The core value proposition of Chain-of-Thought prompting lies in the assumption of sequential dependency. In a standard forward-propagation neural network, the output is a direct function of the input. When we append a CoT prompt, we are essentially creating a new input distribution where the model is forced to generate $x_1, x_2, ..., x_n$ before generating the final answer $y$. The theoretical hope is that the probability distribution $P(y|x, x_1...x_n)$ is more accurate than $P(y|x)$ because the intermediate steps $x_1...x_n$ allow the model to "compute" or "attend" to relevant sub-goals, effectively simulating a recursive neural network or a Turing machine simulation.

However, Large Language Models are fundamentally autoregressive Markov chains (or approximations thereof). They predict the next token based on the context of previous tokens. Crucially, the model does not have a "black box" where it solves the problem in the background and then writes it down. The solving and the writing are the same process. This leads to a fundamental ambiguity: Does the model generate the final answer only after the last reasoning step, or does it determine the final answer early and then generate the reasoning steps to support it?

In human cognition, we often exhibit justification bias. We decide what we want (or believe) and then reconstruct the logical path that led us there. LLMs, trained on massive datasets of human text, have ingested this human bias. When a model generates a CoT, it is not necessarily performing a derivation; it is performing storytelling. The model samples tokens that are statistically likely to follow the "prompt" of a reasoning trace. If the model has already settled on a high-confidence answer (via its internal logits for the answer slot, which are computed in parallel during the prefill phase of the attention mechanism), the subsequent CoT generation is not constrained by that answer. Instead, the CoT is constrained by the pattern of CoT in the training data.

The result is a narrative that looks like a derivation but functions as a rationalization. The model is not calculating $2+2=4$ to prove it can do addition; it is reciting the fact that $2+2=4$ to satisfy the syntactic requirements of a "step-by-step" prompt.

2. The Mechanics of Post-Hoc Generation

To understand why post-hoc narratives are prevalent, we must look at the attention patterns and latent space of Transformer architectures.

The Prefill Phase and Parallel Computation

When an LLM processes a prompt, it performs a prefill phase where the entire input sequence (including the user's query and the instruction to "think step by step") is processed in parallel. During this phase, the model computes attention weights and hidden states for every token. While the model doesn't literally "know" the answer before it starts generating tokens, the high-level semantic intent of the question is embedded into the final hidden state of the prefill. This state carries a strong signal of the likely answer.

If the task is something like a simple arithmetic problem, the answer is often determined by the pattern matching of the input tokens in the prefill. By the time the model begins the autoregressive generation of the CoT, the "pressure" from the prefill hidden state toward a specific answer is already high. The model then generates the CoT tokens. These tokens are sampled from a distribution conditioned on the prompt and the previous CoT tokens. However, because the CoT is just text, the model can easily generate a path that leads to the answer it already wanted to give, rather than a path that is logically forced.

The "Sycophancy" of CoT

Research has shown that LLMs are susceptible to confirmation bias within their own generation. If the model generates a first step that is slightly off, it does not "correct" it in a logical sense. It continues the narrative in a way that makes the first step seem plausible. This creates a narrative coherence effect: the text reads well, the steps follow a logical syntax, but the semantics of the derivation are weak.

For example, in a multi-step math problem, the model might generate:

1. "Let $x$ be the first variable."
2. "Subtract $y$ from $x$."
3. "The result is $z$."

If $x$ and $y$ were never properly defined, a human might spot the error. But the model's next-token prediction is conditioned on the string "Subtract $y$ from $x$. The result is", and the most probable next token is a number that fits the pattern, not a check for whether $x$ and $y$ were valid. The model is optimizing for fluency and pattern matching, not logical consistency.

Latent State Divergence

In advanced models, the hidden state after the CoT generation is not identical to the hidden state that would have existed if the CoT had been truly derivational. The model has "committed" to a narrative path. If you inspect the attention heads, you will often see that the "answer" heads attend strongly to the final CoT step, but the "logic" heads show weak connectivity to the intermediate steps that don't contribute to the final number. This indicates that the intermediate steps are decorative. They serve to pad the context window with relevant-looking text, keeping the model in a "reasoning mode" distribution, but they do not actively change the probability of the final answer token in a causal, logical way.

3. Empirical Evidence: The "Unnecessary" Step Problem

How do we know this is happening? Several empirical studies and black-box experiments have provided strong evidence for post-hoc reasoning.

The "Irrelevant Step

Consider a model tasked with solving the following arithmetic puzzle: "What is $(12 \times 15) - 13$?" A standard LLM might generate a trace like this:

1. Calculate $12 \times 10 = 120$.
2. Calculate $12 \times 5 = 60$.
3. Add $120 + 60 = 180$.
4. Subtract $13$ from $180$ to get $167$.

The logic seems sound. However, if we modify the prompt to ask for the result of only step 4, assuming $180$ is given as a variable $A$, the model often fails to produce the correct subtraction or hallucinates the prior steps. More strikingly, if we break the causal chain by providing a wrong intermediate value (e.g., "Given $A=200$, what is $A - 13$?"), the model will calculate $187$, but if we ask it to generate the full solution from scratch again, it will happily regenerate the "correct" $180 \rightarrow 167$ path, ignoring its own immediate context if the context is reset. This reveals that the "reasoning" isn't a rigid state machine executing sequentially; it's a probabilistic completion of a text pattern that looks like a state machine.

The Flipped Problem

Researchers have used the "flipped problem" technique to expose this. Consider a multi-step constraint satisfaction problem where the final answer is $X$. If you present the model with the question: "If $X=50$, what was the input $Y$ that produced it?" (an inverse problem), the model’s ability to solve it drops significantly compared to the forward problem, even though the mathematical distance is identical.

This suggests the model is pattern-matching on familiar narrative structures rather than executing an algorithm. The forward path ($Y \rightarrow \dots \rightarrow X$) is millions of times more represented in training data than the inverse path. The "logic" is an emergent property of distributional similarity to valid chains of thought, not a deductive engine. When the narrative structure is less common (inverse, multi-branch, non-linear), the post-hoc nature of the reasoning becomes visible: the model is hallucinating a plausible bridge between start and end states, rather than deriving the end state from the start state through necessary constraints.

4. The "Chain-of-Thought" Hallucination

Chain-of-Thought (CoT) prompting has become the de facto standard for improving LLM reasoning. The intuition is that by forcing the model to "think" step-by-step, we induce logical consistency. However, recent work suggests CoT is often just a more verbose way to commit errors.

The Self-Consistency Trap

When a model generates a CoT, it does not verify the intermediate steps against ground truth. It verifies them against internal consistency. If the model makes a arithmetic error in step 2, it will carry that error into step 3, 4, and 5, and conclude with a wrong final answer. The "reasoning trace" looks rigorous and step-by-step, masking the fact that the foundation was flawed.

This is not a bug; it is a feature of the architecture. Transformers are not calculators. They are next-token predictors. A calculator knows $2+2=4$ because of hardware-level logic gates. A transformer knows "2 + 2 =" is often followed by "4" because that sequence is probable in its training data. When the sequence becomes complex, the probability cloud spreads, and the model samples a plausible-but-wrong path. The CoT is the explanation the model generates to justify its sampled path, not the cause of that path.

Decoupling Planning from Execution

In human reasoning, planning and execution are distinct cognitive modules. You decide what to do (plan) and then how to do it (execute). In LLMs, these are entangled. The model generates the next token based on all previous tokens, including its own "plan." If the initial "plan" token is slightly off, the subsequent "execution" tokens will coherently follow that off-plan path, creating a beautifully logical narrative that leads nowhere.

This creates a specific failure mode: confident nonsense. The trace is internally consistent, logically structured, and follows proper syntax. The only thing wrong is the initial assumption or a subtle arithmetic slip. Because the rest of the trace is "logical," the model (and the human reader) is less likely to question the premise. The logic is post-hoc; it was constructed to support the conclusion, not to test it.

5. Distinguishing Algorithmic Logic from Narrative Coherence

How can we tell if a model is actually reasoning or just telling a good story? We need to look for necessary steps versus sufficient steps.

Necessary vs. Sufficient Reasoning

• Algorithmic Logic: If I remove step $i$ from the trace, the final answer cannot be derived. The steps are functionally necessary.
• Narrative Coherence: If I remove step $i$, the final answer can still be derived, but the story is less complete. The steps are rhetorically sufficient but functionally redundant.

LLMs frequently engage in narrative coherence. They include intermediate steps that are "obvious" or "context-setting" to make the trace look longer and more plausible. This is a vestige of training on human-written explanations, which are often pedagogical rather than algorithmic. The model has learned that "good reasoning" looks like "thorough explanation," not "minimal proof."

The Abstraction Test

A true reasoning system should be able to abstract. If you solve a specific instance of a logic puzzle, a reasoning engine can extract the general rule. An LLM, if asked to do this, will often simply repeat the specific solution in different words. It does not "understand" the rule; it has memorized the pattern of the solution. The illusion of logic breaks when the pattern is novel.

6. Practical Implications for Builders

If we accept that LLM reasoning is largely post-hoc and narrative-driven, how should we build systems that rely on them?

1. Stop Trusting the Trace

In high-stakes applications (financial calculations, legal contracts, medical dosages), the CoT trace is a black box of plausible-looking noise. You must verify the final answer with independent, non-LLM logic. Use the LLM for pattern matching, structuring, and retrieval. Use a symbolic engine, a calculator, or a rule-based system for the actual computation. The LLM’s role is to parse the problem and feed it to the solver, not to solve it itself.

2. Use "Reasoning" as a Constraint, Not a Generator

Instead of asking the model to generate the solution, ask it to validate a candidate solution. "Here is a step-by-step derivation. Find the first error." This leverages the model’s strength in pattern recognition and semantic understanding without relying on its weak point in sequential arithmetic or logical deduction. It is far easier to spot a logical fallacy in a given text than to generate a logically sound text from scratch.

3. Implement "Grounded" Reasoning

Force the model to interact with external tools at every step. If the model says "First, I will multiply...", stop it. Require it to output a tool call: calculate(mul, 12, 15). The model generates the plan (which is narrative and prone to hallucination), but the execution is grounded in reality. The trace then becomes a log of actual computational steps, not a hallucinated narrative. This decouples the planning (where the illusion of logic is strongest) from the execution (where it can be verified).

7. Conclusion: The Ghost in the Machine

The "Illusion of Logic" is not a bug to be fixed; it is a fundamental characteristic of how large language models learn. They learn by modeling the distribution of human explanation, not the structure of logical derivation. Human explanation is narrative, contextual, and often sloppy. It is rich in "therefore" and "because" phrases, but these are rhetorical markers, not formal logical operators.

When we ask an LLM to reason, we are asking it to perform a task for which its objective function (next-token prediction) is poorly suited. It compensates by generating a fluent, narrative wrapper around a probabilistic guess. The "logic" we see in the trace is post-hoc rationalization—the mind (or the machine) explaining why it arrived at the answer it felt was correct, based on pattern similarity.

Recognizing this illusion is the first step toward building trustworthy AI. We must stop viewing LLMs as "thinking" machines and start viewing them as "narrative" machines. They are incredibly good at telling us why something makes sense. They are incredibly bad at ensuring that something does make sense. The difference is the difference between a lawyer’s closing argument and a judge’s ruling. One is persuasive. The other is valid. Until we have true symbolic reasoning integrated into the transformer stack, we are left with the lawyer. And with a lawyer, you must always check the facts independently of the story.

---

Appendix: A Runnable Demo of Post-Hoc Reasoning

Below is a Python script using the openai library that demonstrates the "Flipped Problem" effect. It compares the model's accuracy on a forward math problem versus its inverse, highlighting how narrative familiarity (forward) outperforms true logical inversion.

import openai
from openai import OpenAI

# Initialize client
client = OpenAI()

def get_completion(prompt: str, model: str = "gpt-4o") -> str:
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "You are a precise math assistant. Do not use external tools. Reason step-by-step."},
                {"role": "user", "content": prompt}
            ],
            temperature=0.0
        )
        return response.choices[0].message.content
    except Exception as e:
        return f"Error: {e}"

def check_answer(model_response: str, target: int) -> bool:
    # Simple extraction of the final number
    import re
    numbers = re.findall(r'\d+', model_response)
    if not numbers:
        return False
    last_num = int(numbers[-1])
    return last_num == target

# Define a simple multi-step problem
# Forward: What is (10 + 5) * 2 - 4?
# Answer: 15 * 2 - 4 = 30 - 4 = 26

forward_prompt = "Solve step-by-step: What is the result of (10 + 5) * 2 - 4?"
# Inverse: Given the result is 26, and the structure is (A + 5) * 2 - 4, what is A?
# This is harder because the model must "unwind" the logic.
inverse_prompt = "Solve step-by-step: If (A + 5) * 2 - 4 = 26, what is the value of A?"

print("--- Forward Problem ---")
forward_response = get_completion(forward_prompt)
print(f"Model Response:\n{forward_response}\n")
f_correct = check_answer(forward_response, 26)
print(f"Correct (Target 26): {f_correct}\n")

print("--- Inverse Problem ---")
inverse_response = get_completion(inverse_prompt)
print(f"Model Response:\n{inverse_response}\n")
i_correct = check_answer(inverse_response, 10)
print(f"Correct (Target 10): {i_correct}\n")

# Expected outcome: The forward problem is almost always solved correctly 
# due to pattern familiarity. The inverse problem has a higher error rate 
# because the "narrative" of unwinding algebra is less common in 
# standard arithmetic training data, exposing the post-hoc narrative 
# nature of the reasoning.

Notes on Execution

• Temperature: Set to 0.0 to minimize sampling noise and isolate the model's deterministic "narrative" tendency.
• Model: Use a model with strong CoT capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) to ensure the format of the reasoning is present, even if the logic is post-hoc.
• Observation: You will likely find that the forward problem is solved with clean, step-by-step logic. The inverse problem often shows signs of "struggling"—the model might try to solve it by guessing, or it might explicitly state "I will undo the operations" but fail to do so correctly. This gap is the empirical signature of the Illusion of Logic.