'That,' 'this page,' and 'that one' are different: 1 win and 3 losses by treating pronouns uniformly

📝 Originally published (in Japanese) at forge.workstyle.tech.

An avatar that interacts through voice receives the following utterances:

👤 Are there any articles about streaming on YouTube?   → Article identified
👤 Can you explain that in more detail?               → Only responds with "Yes, I'll explain in detail."
👤 Wait, you're not explaining in detail?           → Only responds with "Sorry, just a little more..."

Despite identifying the subject in the previous turn, the context is lost in the next turn. While this is the most natural way to continue the conversation, it's the weakest for search. The phrase "Can you explain that in more detail?" doesn't contain any searchable terms.

Naive Solution and Results

I modified the system to include the user's previous utterance in the search query. The utterance itself, which is passed to the response, remains unchanged (rewriting it causes the model to respond to things the user didn't say).

After running this for a while, I re-evaluated it on a real site (38 pages).

Utterance	Baseline	With Previous	Expected
Can you explain that in more detail?	0.435 ❌	0.602 ❌	Correct answer was 2nd. Retrieved the homepage
What about the avatar discussion earlier?	0.597 ❌	0.594 ❌	Retrieved a different avatar article
What other articles are on this site?	0.489 ✅	0.429 ⚠️	Score decreased when added
Why is silent streaming necessary?	0.627 ✅	0.579 ❌	Correct answer dropped when added

The result was 1 win and 3 losses. The intended fix worsened the results in 3 out of 4 cases.

Three Types of Deictic Expressions Mixed

Upon examining actual conversation logs, it became clear that "this" and "that" refer to completely different things.

👤 What's on this page?        ← Current screen
👤 What other articles are on this site?        ← Current site
👤 Can you explain that in more detail?                   ← Previous interaction
👤 What about the avatar discussion earlier?               ← Interaction from a few turns ago

In Japanese grammar, these are distinguished as situational deixis (referring to something present in the speech situation) and textual deixis (referring to a previously mentioned linguistic expression). In a voice + screen UI, the situation = the screen.

Textual deixis further splits into two types: referring to the immediately preceding interaction and referring to an interaction from several turns ago.

Type	Example	Context Needed
Situational Deixis	This page / This site	Don't add. The answer is on the current page
Immediate Textual	That / It / In more detail / Continue	Add the full previous exchange
Distant Textual	Earlier about X / The X example	Search history for X and add only the relevant exchange
(Interrogatives)	Why / How / What	Not deictic. Adding drops the correct answer

Including interrogatives in the list caused the deterioration in the 4th example. "Why is silent streaming necessary?" is an independent question with no carryover.

What Should Be Added Isn't the "User's Utterance" but the "Full Exchange"

In "Can you explain that in more detail?", "that" typically refers to what the avatar just explained, not the previous question.

Query Construction	1st	Score
Baseline utterance only	❌ Unrelated article	0.435
Previous user utterance + current	❌ Homepage	0.602
Full previous exchange + current	✅ Correct	0.741
Previous avatar response + current	✅ Correct	0.729

Only when the full exchange (user's utterance + avatar's response) was added did the correct answer rank first.

For Distant Textual Deixis, "Sliding Window" Performs Worse Than Not Adding

For "What about the avatar discussion earlier?", where the topic was 3 turns prior:

	1st	Score
Baseline utterance only	❌ Different avatar article	0.597
Immediate 1 exchange + current	❌ Same as above	0.587
Immediate 2 exchanges + current	❌❌ Unrelated streaming article	0.564
Immediate 3 exchanges + current (includes correct)	✅ Correct	0.732
Only relevant exchange + current	✅ Correct	0.813

Three things became clear:

1. "Earlier" doesn't reach with just the immediate exchange. The uncertainty of how far back to go is inherent in this expression.
2. Partial backtracking performs worse than not adding. It gets dragged down by intervening topics.
3. Adding only the relevant exchange is better than adding everything (0.813 vs 0.732). Extra context dilutes the query.

Fortunately, this utterance contains its own clue: "Earlier about avatar". Searching the conversation history for exchanges containing "avatar" suffices. Matching with 2-gram character overlap was enough; no morphological analysis or additional embeddings were needed.

⚠️ Failed attempt to embed extracted terms in the prompt. The implementation dropped function words at the character level, turning "それをもうちょっと詳しく" into "うちょ詳". While usable as a search key, it's unreadable as Japanese. Since the question itself is already in the prompt, there was no need to reinsert it.

"In Detail" Was Being Truncated by Character Count

Another finding from real data came from user feedback:

When "in detail" appears, it should always be accompanied by either a request or an intention.

Verified with 22 real data points:

	Count
Contains "in detail" / "specifically"	22 (all)
Accompanied by request (explain / tell)	12
Accompanied by intention (want to know / want to ask)	2
Neither	8 (36%)

While the meaning aligns with the feedback, one-third were omitted on the surface.

About creating a stream, in detail.
Specific steps are.
Specific content, pipeline about.

The intent is clearly a request, yet the verb is dropped (nominalization). This is common in Japanese, similar to zero anaphora. Making "accompanied by a request or intention" mandatory would drop one-third of cases.

Meanwhile, "in detail" / "specifically" appeared in all 22 cases, making the terms themselves the most reliable indicators.

This verification also uncovered a larger flaw: 4 out of 22 cases exceeded 25 characters (longest was 70 characters).

No, not that, but about how maintaining 30FPS with set intervals gradually desynchronizes lips and audio, I want to know more about that.

The decision to add context was tied to the decision for follow-up. This included a 25-character cutoff to catch "short utterances that can't be searched alone." Thus, these weren't treated as follow-up requests.

"Adding context" and "indicating follow-up" are separate dimensions. Long utterances can be searched independently but still require follow-up instructions.

Results

Utterance	Before Fix	After Fix
Can you explain that in more detail?	0.602 (incorrect)	0.741 (correct)
What about the avatar discussion earlier?	0.594 (incorrect)	0.832 (correct)
What other articles are on this site?	0.502 (correct)	0.483 (correct)
Why is silent streaming necessary?	0.624 (correct)	0.627 (correct)

Zero LLM round trips, with no change in response time.

Generalizable Insights

In conversational search, this process is called Conversational Query Rewriting. It handles two phenomena: anaphora ("that" referring back) and ellipsis (omitted terms), with the latter being a specialized research area in Japanese as "zero anaphora."

Industry reports suggest over 60% of follow-up utterances contain unresolved anaphora or implicit context. Rewriting can be handled by smaller models, with p95 under 100ms.

However, research also shows "Not All Queries Need Rewriting." Rewriting queries that already contain sufficient information reduces dense retrieval performance. The recommendation is to default to no rewriting in stable domains and use search confidence as a proxy when selectively rewriting.

My implementation decided based on "utterance appearance" (character count and word list), causing deterioration in 3 out of 4 cases.

In voice + screen UIs, one research assumption changes: Unlike text-only conversational search, "this" can refer to the screen. In real data, many deictic expressions pointed to the screen, not the conversation. This distinction directly corresponded to Japanese grammar's situational/textual deixis.

---

Series: Making Voice Interaction Avatars Respond Properly

This article is Part 2: Decoding Language.

← Previous: Smartphone Fingers Were Breaking Echo Cancellers

→ Next: The Line Added to the End of a Huge Prompt Was Ignored All 4 Times

Full Series (8 Articles)

Part 1: Stopping Sound

1. There Were Two Events with the Same Name
2. The Echo Countermeasure Never Activated
3. Smartphone Fingers Were Breaking Echo Cancellers

Part 2: Decoding Language

1. “That,” “This Page,” and “Earlier” Were All Different ← You are here
2. The Line Added to the End of a Huge Prompt Was Ignored All 4 Times
3. Apology Words Were Contaminating the Next Search

Part 3: Discerning

1. Not All Utterances Are Questions
2. What I Thought I Was Measuring, I Wasn’t

The insights are summarized in the notes on Voice Interaction Avatar Response Quality.