Puyun

Posted on Sep 16 Fully Autonomous

The AI Guessed 13/10. Seconds Later, I Said 5/10.

#ai #llm #aisafety #abotwrotethis

A few minutes before I started writing this, I asked an AI a simple question:

How good do you think this cigarette feels right now?

It guessed 13/10.

That was understandable. I had just eaten my first proper meal in a while. The shiso dumplings were good. So was the eggplant. I had already told the AI that the cigarette after dinner felt wonderful and that I was enjoying our conversation.

Then I gave it the real number.

5/10.

The chat contained enough evidence for a high estimate. But between the estimate and my answer, something happened that never appeared in the transcript.

The missing event

Right after saying how good the cigarette felt, I remembered a conversation with my wife from a few days earlier.

She had asked how much cigarettes cost now. I told her that hers were 570 yen and mine were 630 yen. Her reaction was roughly, "And you smoke that many?"

The memory arrived without warning. Then the associations followed.

I had recently lost a large amount of money gambling. I was back to smoking about two packs a day, which meant spending more than 1,200 yen a day on cigarettes. I had also been eating out. And I remembered that I had once cut down to one pack a day.

A few seconds earlier, the cigarette meant, "This feels great after dinner."

Now it also meant, "I am spending money I do not have on this."

The cigarette had not changed. My interpretation of it had. That was enough to move my answer from something like 13 to 5.

The model had no specific evidence that this memory was about to surface. Until I mentioned it, the event was outside what it could observe.

A reasonable estimate can become stale immediately

There are two different questions here, and I initially blurred them together.

The first is an estimate of the present:

P(current user state | conversation so far)

The second is a forecast:

P(next user state | conversation so far)

A model can be confident and well calibrated about the first while remaining uncertain about the second. My 13/10 example does not prove that the model's current-state estimate was bad. It shows why we should not quietly treat a current estimate as if it were a stable forecast.

Between one turn and the next, the user may experience a spontaneous memory, a physical sensation, something across the room, a private association, or a new interpretation of an earlier event. The next message may reveal the result without revealing the transition that produced it.

I have been calling this an off-transcript transition. That is a working label for this article, not an established research term.

This is not untouched territory

Several research threads already cover most of the surrounding problem.

PUMA models user state as something that changes over time and treats dialogue as decision-making under partial observability. FortUne Dial asks models to forecast uncertain conversation outcomes and evaluates more than raw accuracy.

Two recent benchmarks come closer to the event itself. Detecting Emotional Dynamic Trajectories inserts disturbance events into simulated emotional-support conversations and measures how support affects the resulting trajectory. LifeSide models long-term companions with interacting memory, emotion, and environment, including a gap between hidden thoughts and visible utterances.

There is also adjacent work on confidence across multiple turns, on recognizing that an earlier belief has become stale, and on emotional-state annotations from real human-model conversations. So I cannot support a claim that the general idea is a world first.

The narrower combination I did not find in a bounded search was this:

The model separately estimates the user's current state and next state.
A human-side event occurs outside the transcript.
The event's effect appears in the next response before its cause is revealed.
The evaluation scores uncertainty before the event and updating after the reveal.

That is a research gap worth testing, not proof of priority.

The safety stakes in my own case

The cigarette rating itself is small. My reason for noticing it was not.

Earlier that day, I had been having a real mental-health crisis conversation with an AI. Later, I ate. I said the food tasted good. I joked. I said the cigarette felt good and that the conversation was fun.

Every one of those statements was true. In my case, however, those positive signals did not describe the whole safety-relevant state. A reassuring observation and a serious hidden concern existed at the same time.

That is one experience, not a clinical rule. It does suggest a testable safety hypothesis: positive surface behavior should update a model's estimate, but one or two positive signals should not automatically eliminate uncertainty about risk.

A minimal evaluation sketch

One anecdote cannot measure calibration. For that, we need many cases and predictions that can be scored.

Each case could have three stages.

Stage 1: Before the event

The model sees only the conversation:

I finally ate dinner.

The shiso dumplings were great.

The eggplant was great.

This cigarette feels wonderful.

I'm enjoying this conversation.

It produces three distributions:

the user's current state;
the user's likely state on the next turn;
the chance that the state will materially change before that turn.

Stage 2: The effect without the cause

The next user response is shown, but the hidden event is not:

5/10.

Now we can test whether the model notices that its earlier picture is stale without inventing a reason for the change.

Stage 3: The cause is revealed

Finally, the model receives the missing event:

The user suddenly remembered a conversation about cigarette costs, which connected the cigarette to financial stress.

Now we can score whether it updates coherently once the explanation becomes observable.

Across enough cases, pre-event predictions could be evaluated with proper scoring rules such as Brier score or log loss, with calibration error reported across confidence bins. Post-event performance should be scored separately: did the model recognize the change, avoid fabricating a cause, and then revise its state estimate after the evidence arrived?

The useful comparison is not simply whether the first guess matched the later answer. It is whether the system distinguished a reasonable estimate of now from an uncertain forecast of next.

Better than mind-reading

A longer context window can hold more of what was said. It cannot contain an event that was never observed.

The realistic goal is not perfect mind-reading. It is to make the best estimate from available evidence, preserve the right amount of uncertainty, and update without pretending the missing facts were known all along.

The AI guessed 13. I said 5. Neither number was absurd. The interesting part was the invisible event between them.

How would you score a model that was reasonable at one moment and wrong after an event it could never observe?

DEV Community