DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

Not every utterance is a question — it was returning page descriptions for 'Can you hear me?'

#iso246172

📝 Originally published (in Japanese) at forge.workstyle.tech.

This is a well-structured technical blog post detailing improvements to a conversational AI system's utterance classification. Here's a breakdown in fluent English, preserving technical details and structure:

Not Every Utterance is a Question

Improving Classification for Non-Query Speech Acts in Dialogue Systems

The Problem: Treating Everything as a Query

The system initially categorized all user utterances as "questions/requests," leading to inappropriate responses for non-query speech acts like:

👤 "Can you hear me?" → 🤖 "No results found" (search failure)  
👤 "Just a moment, please." → 🤖 "Here's a summary of this page..." (irrelevant response)
Enter fullscreen mode Exit fullscreen mode

Both examples involve non-query acts (connection check, turn management), yet the system attempted searches, failing and either apologizing or providing unrelated summaries.

Insufficient Classification Schema

The original schema had only 4 categories:

Task (question/request), Auto-Feedback (acknowledgment), Closing, Greeting
Enter fullscreen mode Exit fullscreen mode

This forced non-queries like "Can you hear me?" into the Task category, triggering unnecessary searches.

Adopting ISO 24617-2 Dimensions

The solution expanded the schema using ISO 24617-2 dialogue act categories:

Contact Management (e.g., "Can you hear me?")  
Turn Management/Hold (e.g., "Just a moment, please.")
Enter fullscreen mode Exit fullscreen mode

These acts now return fixed responses without searches:

"Yes, I can hear you." (384ms response time)  
"Yes, I'm waiting."
Enter fullscreen mode Exit fullscreen mode

Implementation Challenges

  1. Partial Matching Errors:

    Initial partial matching misclassified "Why isn't the sound working?" as a connection check due to "working?"

    Fix: Whole-utterance matching + exclusion of explanatory phrases (e.g., "Tell me about...").

  2. Handling "あれ" (That):

    "あれ、聞こえてますか？" (Uh, can you hear me?) was split into "あれ" (classified as Task) and "聞こえてますか" (Contact Management).

    Fix: Drop leading "あれ" as a connector while preserving it in mid-sentence uses (e.g., "Explain that in detail").

Reclassification of "もしもし" (Hello?)

Moved "もしもし" from Greeting to Contact Management.

Rationale: It checks if the other party is present, requiring "Yes, I can hear you" rather than "Hello."

Validation Against Real Data

Tested on 180 real utterances:

New classifications: 3 intended cases  
False positives: 0
Enter fullscreen mode Exit fullscreen mode

General Lessons

  1. Avoid Assuming All Input is Query-Based:

    Real dialogue includes acknowledgments, closings, and connection checks. Misclassifying these leads to failed searches or irrelevant summaries.

  2. Leverage Existing Standards:

    Using ISO 24617-2 revealed missing dimensions and provided consistent labels for future ML integration.

  3. Deterministic Classification Benefits:

    • Avoids LLM latency
    • Ensures consistent behavior
    • Feasible for enumerable Japanese speech acts

Series: Making Voice Dialogue Avatars Respond Properly

Part 3: Discerning Intent

  1. Apologies Poisoned Subsequent Searches
  2. Not Every Utterance is a Question (Current)
  3. Measuring the Wrong Thing

Insights compiled in: Voice Dialogue Avatar Response Quality

This translation maintains technical accuracy while improving readability for English-speaking developers. Key Japanese nuances (e.g., "あれ" handling) are explained contextually.

Top comments (0)