DEV Community

kohei matsunobu
kohei matsunobu

Posted on Originally published at qiita.com

What Happens When You Tell an LLM a Story Backward?

#ai #computerscience #llm

Fluent answers are not the whole story

When I build with an LLM, I usually start with practical questions: Is the answer useful? Can someone check its sources? Does it make sense?

That last question hides several different problems. An answer can be grammatically correct and still put events in a strange order. How could we measure that distinction without simply asking another model to judge it?

I'm Kohei Matsunobe, a second-year correspondence student at ZEN University. I build KamiMeg, an iOS goshuin (shrine-stamp) app with a generative-AI shrine guide, and a Japanese PDF RAG demo that makes its sources visible. A recent paper by Takahiro Sagawa, a professor in the Department of Applied Physics, Graduate School of Engineering at the University of Tokyo, caught my attention because it approaches this question from an unexpected direction: stochastic thermodynamics.

This is a builder's reading note based on the university's press release and the paper's abstract, not a reproduction of its experiments.

Change the order, not the words

Consider this example, translated from the press release:

The cup fell on the floor. The cup broke.

Now reverse the sentences:

The cup broke. The cup fell on the floor.

Both versions contain readable sentences. But the first follows a familiar progression; the second makes me look for a different explanation. It is not impossible, just different.

The research compares how likely such sequences are under a language model. Crucially, it reverses sentence order while preserving the word order inside each sentence.

Reversing every token would break grammar. The resulting probability difference could mostly reflect broken syntax rather than anything interesting about events. Sentence-level reversal is a way to change the relationship between sentences without destroying their internal structure.

That experimental choice is what first made the work click for me: a useful measurement starts by deciding what to disturb and what to leave intact.

Thermodynamics without measuring GPU heat

The quantity at the center of the paper is entropy production: a way to quantify irreversibility by comparing the likelihood of forward and reversed sequences.

Here, thermodynamics does not mean measuring the computer's physical heat output or energy consumption. The object of study is the model's generation process and its directional asymmetry.

The theory is broader than text. It provides a unified framework for autoregressive generative models, ranging from Transformers to Kalman filters.

There is a technical obstacle: the next output can depend on a long history, not just the immediately previous output. That makes the observed process non-Markovian. Trying to enumerate every possible history would quickly become impractical.

Sagawa's framework shows how to estimate entropy production efficiently without enumerating all those histories. It also decomposes entropy production into non-negative contributions at each timestep, using an information-theoretic perspective: inferring the past from the future.

I find that reversal of viewpoint appealing. We normally discuss these models as predictors of what comes next; this theory also asks what looking backward can reveal.

What the language-model experiments suggest

The experiments used GPT-2 (124M) and Qwen3-4B-Base. Texts describing sequential events tended to have larger sentence-level entropy production than texts listing independent facts. In other words, reversed generation was relatively less likely for the sequential-event texts.

That suggests the models' generation probabilities reflect something about event order. It offers a quantitative probe for investigating what is often called a model's internal “world model.”

It does not establish that an LLM understands causality in the same way a person does.

There is also an important dataset caveat: the evaluation texts were generated by prompting commercial generative AI. Their construction could introduce features that matter to the comparison. The press release explicitly calls for further verification, including text sets constructed from theoretical definitions of order and causal relationships.

What I would take back to an app project

In my RAG demo, making sources visible helps a reader inspect where an answer came from. It does not, by itself, establish that the answer describes events in a sensible sequence. Those are separate evaluation questions.

This paper does not give me a ready-made accuracy score or a proven hallucination detector. I have not tested its measurement in my own apps.

What it gives me is a sharper question: instead of only asking whether a generated explanation sounds good, what changes when I perturb its event order?

For me, that is the interesting bridge between theory and building. “Does this model represent the world?” is enormous. Choosing a controlled reversal and studying its probability makes one part of that question more concrete.

Sources

Top comments (0)