DEV Community

weiwuji
weiwuji

Posted on

Two-Layer Classification: Regex Pre-Filter + Semantic Routing — No Intent Slips Through

#engineering #agents #ai

The Pain: The last post covered scene routing — but routing is only as good as the classification feeding it. Regex alone covers maybe 60–70% of real requests, and colloquial lines like "ask how much XX to YY costs" fall through the cracks. How do you route those accurately?
What You'll Learn:

  • Why regex has a hard ceiling: it matches known patterns, not unknown phrasing
  • Two-layer classification: the exact layer sets the floor, the fuzzy layer raises the ceiling
  • The semantic layer in practice: all-MiniLM-L6-v2 embeddings + Chroma retrieval
  • How to tune the threshold (0.45 → 0.55) and why a margin check kills edge-case misrouting
  • Why "forward this email to sunny" — a request with zero keywords — still lands in the right scene

1. The Problem: The Ceiling of Regex

The early version of scene routing judged user intent with regular expressions:

import re

QUOTING_PATTERN = r'sea.*freight|air.*freight|quote|quotation|rate'

def regex_match(text):
    if re.search(QUOTING_PATTERN, text):
        return "quoting"
    return None
Enter fullscreen mode Exit fullscreen mode

It handled the common quoting expressions: "What's the sea freight to Frankfurt?" hits, and "DDP to CDG quote" hits too.

But once it hit production, the numbers told a different story: regex coverage was only about 60–70%.

A flood of real user phrasing —

  • "Ask what air freight from XX to YY costs"
  • "HK to LAX, what price?"
  • "How much for this shipment?"

— either failed the match entirely or matched the wrong scene.

The nastier cases were the ones that need understanding, not matching: "Forward this email to sunny" — no email-related keyword anywhere, yet semantically it's obviously a forwarding intent. Regex can't touch that.

💡 Core insight: regex is good at known patterns; semantics is good at unknown expressions. They complement each other — they don't substitute for each other.

2. Core Idea: Two-Layer Classification

The fix is two-layer classification: layer one is regex matching (the exact layer), layer two is semantic matching (the fuzzy layer).

The exact layer handles expressions with clear keywords — "sea freight", "air freight", "quote", "quotation", "rate". The patterns are stable, regex is near-zero latency, and it's extremely efficient.

The fuzzy layer handles everything the exact layer can't cover — "ask how much XX to YY costs", "HK to LAX what price". No fixed keywords, but the semantics clearly point at quoting.

Precedence: the exact layer goes first.

Two-layer classification: regex pre-filter (~90%, ~0 ms), semantic layer (~10%, <10 ms), fallback (~0.1%) — confidence ladder on the right
Layer 1 is deterministic, layer 2 is semantic, and the fallback guarantees nothing is orphaned.

3. Implementation

3.1 Semantic Layer: Embedding Vectors + Chroma

The fuzzy layer's foundation is text embeddings. We use all-MiniLM-L6-v2 — a lightweight Sentence-BERT model that maps text into a 384-dimensional vector space.

from sentence_transformers import SentenceTransformer
import chromadb

# Load the embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Initialize Chroma
client = chromadb.Client()
collection = client.get_or_create_collection("scene_prototypes")

# Each scene has several "scene prototypes"
prototypes = {
    "quoting": [
        "How much is sea freight to Frankfurt?",
        "DDP to CDG quote",
        "What's the air freight cost from Hong Kong to LAX?",
    ],
    "email": [
        "Forward this email to sunny",
        "Reply to sunny's email for me",
        "Send an email to the Guangzhou agent",
    ],
    # ... 6 scenes
}
Enter fullscreen mode Exit fullscreen mode

3.2 Matching Logic 

def semantic_match(text):
    # Encode the user request
    query_vec = model.encode(text)

    # Search Chroma for the most similar scene prototype
    results = collection.query(
        query_embeddings=[query_vec],
        n_results=3
    )

    # Highest similarity
    top_score = results['distances'][0][0]
    top_scene = results['metadatas'][0][0]['scene']

    # Threshold check
    if top_score > 0.55:
        return top_scene
    return None
Enter fullscreen mode Exit fullscreen mode

3.3 Tuning: From 0.45 to 0.55

The threshold was not pulled out of thin air — it came from a systematic tuning pass:

Threshold Coverage Misclassification Problem
0.45 ~90% ~15% Great coverage but misjudges — "check the flight status" routed to quoting
0.50 ~82% ~8% Middle ground — "forward the email" still misjudged
0.55 ~75% ~3% Acceptable — misses fall back, nothing misroutes

Threshold tuning 0.45 → 0.55: coverage vs misclassification bars, with the margin check callout
0.55 + margin 0.1 wins: accepting a few misses beats accepting misrouting — a wrong scene means wrong tools.

I chose 0.55 — I can accept "missing" some edge expressions (they fall back to the general scene), but I cannot accept "misjudging" (which misroutes the request to a wrong scene).

One extra check was added: the margin. It's not enough that the top similarity clears the threshold — the gap between the top and the second-best similarity must also clear a margin (0.1). This further suppresses misjudgments near fuzzy boundaries.

def semantic_match_with_margin(text):
    results = collection.query(query_embeddings=[model.encode(text)], n_results=2)

    top_score = results['distances'][0][0]
    second_score = results['distances'][0][1]

    # Two conditions: threshold + margin
    if top_score > 0.55 and (top_score - second_score) > 0.1:
        return results['metadatas'][0][0]['scene']
    return None
Enter fullscreen mode Exit fullscreen mode

3.4 Scene Prototype Design

Prototype quality directly determines match quality. Three principles:

  1. At least 5 prototypes per scene, covering common variants. The quoting scene includes: sea-freight quotes, air-freight quotes, DDP quotes, DAP quotes, multimodal quotes. Every prototype is a real user expression (collected from historical conversations).
  2. Prototypes across scenes must be distinguishable. If two scenes' prototypes are semantically too close, match quality degrades.
  3. Prototypes need periodic refresh. As user phrasing drifts, old prototypes lose coverage. Every month, collect unmatched expressions from the fallback scene and evaluate whether new prototypes are needed.

✅ Verification: after two-layer classification shipped, classification accuracy went from 80% toward ~100%. Keyword-free requests like "forward this email to sunny" now hit the email scene reliably.

🩸 Pitfall: I started with regex only — 60–70% coverage, and a pile of colloquial requests dumped into the fallback scene. Adding the semantic layer lifted coverage dramatically, but at 0.45 the misjudgments were brutal. It took a full week of tuning to land on 0.55 + 0.1 margin.

💼 Value: every request gets routed to the right scene with the right toolset. No fish slips through the net.

▸ Cognitive shift: exact and fuzzy are not an either/or. Regex provides determinism; semantics provides coverage. Only the combination gives you high precision and high recall.

The classification decision tree: regex exact hit → scene A; semantic hit with margin → scene B; miss → general scene
Every request ends with exactly one scene — no orphans, no guesswork.

4. Pros and Cons

✅ Pros

  • Extremely high coverage — what regex can't match, the semantic layer catches
  • Keyword-free expressions still hit ("forward this email to sunny")
  • Regex + semantics complement, not substitute — the exact layer holds the floor, the fuzzy layer extends the ceiling
  • Chroma vector retrieval is fast (<10 ms), invisible to the user
  • Prototypes are extensible — a new expression only needs a new prototype

⚠️ Cons

  • Semantic matching depends on embedding quality — a model version change can shift stability
  • Slow first load — model download + Chroma index build takes ~30 seconds
  • Misjudgment risk — semantically adjacent scenes (flight status vs air-freight quoting) can blur
  • Threshold tuning is experience-driven — no automated tuning mechanism yet
  • Cold-start problem — a new scene needs accumulated prototypes before it matches reliably

5. Where You Are Now

The you reading this is no longer the pessimist who "writes a wall of regexes and prays they cover every expression." You are becoming a system designer who uses two-layer classification so every request has a home.

In natural language processing, facing the endless variation of human expression, what we need is no longer "find one exact match" but "make the most accurate judgment within the most likely range." That is the art of fuzzy matching.

Next post: routing and classification are in place — but what about complex tasks? One report can contain a quote request, a payment chase, and a finance item — a single scene can't hold it. The composite pipeline container turns a report into a pipeline.

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.

Top comments (0)