DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

The single line at the end of the massive prompt was ignored all four times

#llm #rag

📝 Originally published (in Japanese) at forge.workstyle.tech.

Conversation with the Voice Dialogue Avatar

I had the following conversation with the 24-hour avatar.

👤 This 24-hour avatar is working with what kind of mechanism?  → Explain the whole thing
👤 What was the Pod startup you mentioned earlier?                → Explain the whole thing again
👤 Tell me more about it.                                         → Explain the whole thing again
Enter fullscreen mode Exit fullscreen mode

When I tried to dig deeper, I got the same summary back. The conversation log had a repetition flag set for 5 out of 13 turns.

The anaphora judgment was working correctly (recognizing "earlier" as a remote context and adding the corresponding past conversation to the search query). What was off was the response.

A Simple Countermeasure

I added instructions right after the latest user utterance — the most effective position in the prompt.

(The visitor is listening to the continuation of the previous conversation. Without repeating the overall image or summary that has already been stated, 
please describe the specific part that the question is pointing to, following the corresponding section of the text. 
Do not say the same thing as the previous response again.)
Enter fullscreen mode Exit fullscreen mode

I chose this position for a reason. In the same project, I had placed speech-linked pointing instructions and tool call reminders in this position, and they were effective. As the history grows, the model starts to omit tags by simulating the "past self" that didn't output tags, so I reminded it to output tags at the end of each turn — a countermeasure that actually worked.

I deployed and tried it.

Nothing Changed 

👤 I'll tell you more about it.    → Explain the whole thing
👤 Can you tell me more about it?      → Explain the whole thing again (repetition flag)
Enter fullscreen mode Exit fullscreen mode

First, I checked if the instructions were actually being sent. They were sent. All four times.

Question: Tell me more about it.

(System instruction: For each item/section, ...)

(The visitor is listening to the continuation of the previous conversation. Without repeating the overall image or summary that has already been stated, 
please describe the specific part that the question is pointing to, following the corresponding section of the text. 
Do not say the same thing as the previous response again.)
Enter fullscreen mode Exit fullscreen mode

The prompt included the full text of the page, 8,207 characters.

The Answer Was Written in the Same File

While searching for the cause and reading the code, I came across a comment that I didn't write.

Micro-call separation for extraction responses: For turns where the corresponding sentence can be determined deterministically,
switch to a dedicated call for the minimum prompt that excludes persona, catalog, history, and tag conventions.
A single line in a huge prompt is ignored (actual measurement: even if the correct sentence is placed at the beginning, it flows into a summary),
but a weak model will follow a small context and a single task (separation of generation responsibilities).

The same phenomenon was recorded in the same file. And the existing code was escaping to a dedicated small call for that reason.

I fell into the same trap, in the same place, after reporting that I had placed it in the "most effective position".

Separating Responsibilities

I made the digging deeper the same shape. I stopped passing the full text and instead passed only the relevant section.

【Already mentioned】Previous conversation (up to 600 characters)
【Main text】Relevant section only (up to 2,400 characters)
【Question】Current utterance
Enter fullscreen mode Exit fullscreen mode

The instruction is "Without repeating what has already been mentioned, describe the specific content that has not been touched yet in 2-4 sentences".
I made it a single task.

The number of calls does not increase. Since it replaces the existing call, it's actually faster because the input is smaller.
For turns where there is no relevant section or previous conversation, it follows the same route as before, so it doesn't get worse.

It Was Still 0

I deployed and tried it, and the digging deeper call didn't fire at all.

There were two routes to get the material.

Full text route      … Turns that use the full text of the page. I had implemented it here
Search route      … Turns where the page search hits. **This was missing**
Enter fullscreen mode Exit fullscreen mode

Ironically, the turns where the search hits are the ones where the material for digging deeper is most available.
I wasn't picking it up. The previous time it happened to fire was only when it went through the full text route.

This wouldn't be found in testing. I only found out when I saw the 0 fires in the actual machine log.

Result 

15:12:28 "Explain this page"
      🤖 This page is about the system that continues to broadcast with an AI avatar, its quality assurance...

15:14:06 "Tell me more about it."
      [dig] Section='things that only come out after a long time' Already mentioned=322 characters Main text=1519 characters
      🤖 "Bugs that only come out after a long time" include things like memory leaks, or periodic tasks that are missing, 
         which only appear as time passes...
Enter fullscreen mode Exit fullscreen mode

It entered a specific part instead of repeating the summary. The repetition flag was not set either.

Generalizable Things

It's not the position of the instruction, but what you pass with it that determines its effectiveness.

"It's most effective to put it at the end of the prompt" is conditionally correct. If the surroundings are short, it's effective.
If you put it with an 8,000-character main text, that one line becomes less than 1% of the context.
The model is pulled into the dominant task of "summarizing the given data".

And the more ambiguous the input, the more the model escapes into a summary. The same thing happened when the state was the same.

👤 What's the weather like today?          → 🤖 This page is about the system that continues to broadcast with an AI avatar... (summary)
👤 Yes, please wait.  → 🤖 This page is about the system that continues to broadcast with an AI avatar... (summary)
Enter fullscreen mode Exit fullscreen mode

Even for utterances that aren't questions, the same summary is returned. The latter was a state where the same answer was returned no matter what.

The direction of the countermeasure was not "to make the instruction stronger", but "to make the task single and narrow down the material".
The less you pass, the relatively larger the instruction becomes.

Another thing. The answer to the same problem may already be in the codebase. In my case, it was a comment that I found after implementing the countermeasure and deploying it, and it didn't work.
Searching comments in the codebase for "has this symptom occurred before" is more cost-effective than I thought.

Series: Until the Voice Dialogue Avatar Answers Correctly

This article is part of Part 2: Understanding Language.

← Previous: ""It", "this page", and "earlier" were different things
→ Next: Apology words were contaminating the next search

Series of 8 articles

Part 1: Stopping the Sound

  1. There were two types of events with the same name
  2. The self-echo countermeasure had never fired
  3. The smartphone's finger was breaking the echo canceller

Part 2: Understanding Language

  1. ""It", "this page", and "earlier" were different things
  2. A single line in a huge prompt was ignored four times ← Now here
  3. Apology words were contaminating the next search

Part 3: Seeing Through

  1. Not all utterances are questions
  2. I thought I was measuring, but I was measuring something else

The notes that became the basis for this knowledge are summarized in The Response Quality of Voice Dialogue Avatars.

Top comments (0)