đź“ť Originally published (in Japanese) at forge.workstyle.tech.
TL;DR The AI characters mixed catchphrases, JSON, and foreign languages into their dialogue and drifted off topic. Reviewing 403 utterances showed a 23% breakdown rate. Changing speaking styles, prompts, supplied context, and how often the topic was passed in affected the results.
The Bottom Line
Fixing only the speaking-style settings reduced the breakdown rate in AI character banter from 23% to 3%, but reviewing the source material and prompts alone did not eliminate breakdowns. Under a setup that supplied a topic every turn, removed unnecessary source material, and kept broken lines out of the conversation, the breakdown rate reached 0% in an evaluation that ran three topics for 24 turns each. This is an evaluation result; the knowledgeable character still made up some trivia.
Why the AI Characters’ Banter Fell Apart
In the first 403 utterances I examined, stock phrases kept repeating, and JSON responses and foreign languages crept in. Then other characters picked up on the broken lines, and the whole conversation lost its way back to the prompt.
At first, I thought the problem was the characters’ speaking styles. The logs pointed to settings of temperature 2, frequency_penalty 2, and typical_p 0.5 as the main causes. The instructions said, “Be foul-mouthed as hell” and “Never say anything sensible.” In trying to make the banter funny, I’d configured the characters to stray from ordinary conversation.
How Changing Speaking Styles Affected the Breakdown Rate
The new speaking styles were: the lively one, the quick-witted one, the knowledgeable one, and the storyteller. We set the temperature to 0.9 and added four shared instructions: “Japanese only,” “1–2 sentences,” “don’t insult anyone,” and “connect to the previous topic.”
First, when we evaluated the styles by changing only the speaking style, the breakdown rate fell from 23% to 3%. This showed that the configuration had a significant impact.
But that wasn’t the end of the story. When we also removed the context given to the model, such as memories and summaries, the breakdown rate rose to 5%—higher than when we changed only the speaking style. The results didn’t show that removing things one by one would steadily improve performance.
What We Changed About Prompts and JSON Responses
The previous prompt assigned roles like “the one who makes bold statements” and “the one who challenges them,” and prohibited agreement. As a result, everyone became a challenger.
In the new prompt, we left the roles to each speaker’s style and added one concrete detail to the topic. We also added instructions not to insult anyone or pick up on garbled remarks or machine announcements. The breakdown rate with these conditions was 7%, higher than the 3% we got by changing only the speaking styles. The numbers don’t always go down with each revision.
When we tried returning plain text, the breakdown rate fell to 2%, but the responses no longer included tone and emotion, which drive expressions and movement. Since we wanted to keep that functionality, we didn’t adopt the plain-text-only approach.
Instead, we kept the JSON format with tone and emotion, and tested a constraint that forced the model to return responses in that format. The breakdown rate was 2%, and 100% of responses included tone and emotion. We also needed a fallback to remove the format constraint and retry for providers that don’t support it.
For evaluation, we stopped comparing impressions by watching the screen. We used the same prompt as the product and ran multiple topics and turns from an empty scene, comparing results for each condition. The product and evaluation script build prompts from shared components. Using different text just for evaluation would make it harder to compare those results with what happens in the product.
Can the Conversation Keep Going If You Pass the Topic on Every Turn?
Even with well-formed replies, a conversation won’t come together if the topic doesn’t carry through. In a real-device log, the topic “time machine” had drifted by the fourth round to “slamming a baguette down at a bakery.” That was because the topic had only been passed in the first turn.
So I added the topic to every turn’s prompt and set it to switch to a new one after 12 turns. When the user speaks, the topic is cleared. I also stopped saving back-and-forth turns in the conversation history with the user. Leaving them in the history mixed small talk into the material used to reply to the user, and rebuilding the summary every time the history was updated was also slowing things down.
I also narrowed down what gets passed to the model. For back-and-forth turns, I leave out conversation history, memory, capabilities, procedures, and summaries, and use the latest eight lines from the room as clues for continuing. More information doesn’t always make for a better conversation. In this case, I kept out material that wasn’t needed for small talk.
Malformed text, such as an unfinished JSON fragment or Chinese text, is neither spoken nor saved to the room. This prevents the next character from picking up a broken utterance and trapping the whole room in a conversation it can’t escape.
Conditions That Brought the Breakdown Rate to 0%
Finally, we evaluated a setup that provides a topic for every turn, switches topics every 12 turns, and prohibits making things up. In a comparison that ran each of three topics through 24 turns, the breakdown rate fell from 3% before the changes to 0% afterward. Insults remained at 0% in both setups, and replies with tone and emotion remained at 100%. After the changes, the conversation stayed on topic through turn 12, and off-topic replies decreased noticeably.
However, 0% is the result for this evaluation. The trivia from the “knowledgeable” participant still included fabrications. Even after we added instructions to use only reliable knowledge and avoid inventing numbers or proper nouns, fabrications did not disappear entirely.
What we learned is that adjusting speaking-style values alone was not enough to prevent the banter from breaking down. We revisited the prompt, the material provided, the reply format, and how topics were supplied, then evaluated them using the same components as the product. We needed to check not only which setup lowered the numbers, but also what remained when we changed each part.
Key takeaways
- Record the breakdown rate and symptoms such as catchphrases, JSON, and foreign languages.
- Evaluate a speaking style change on its own and compare the breakdown rate.
- Review prompts and supplied context, and compare conditions that pass the topic every turn.
- Check that JSON replies include tone and emotion and that malformed text does not remain in the conversation.
- Evaluate multiple topics and turns with the same prompt used in production, and check for remaining fabricated facts.
Top comments (0)