DEV Community: HoverBot

System One Models for Chatbot Decisions: Testing Jev for PII, Guardrails and Product Selection

Vitaly Goncharenko — Fri, 25 Sep 2026 02:57:20 +0000

A chatbot needs language generation, but much of the work around that generation is classification. Does this message contain personal information? Is a draft reply safe to send? Which retrieved product best matches the request? Should the conversation go to support?

Those questions do not require another paragraph of generated text. They require a value that software can inspect and use. This is the problem TypeSafe AI positions Jev to solve. TypeSafe calls Jev its first public System One model and describes the interface as unstructured state in, typed probabilistic decisions out. Its launch announcement says Jev gives up string generation in favor of structured outputs.

HoverBot is evaluating that pattern. We made seven synthetic test calls on September 25, 2026. The messages were invented and contained no customer data. Every request returned HTTP 200 with the expected answer, taking 0.67 to 0.82 seconds end to end from Singapore, including network time. This was a smoke test of the API shape. It was not an accuracy benchmark, a latency benchmark or evidence of production performance.

Most chatbot decisions are classification, not writing

The visible reply is only one part of a product aware chat pipeline. Before generation, software may inspect the message for sensitive data and select an intent route. After retrieval, it may rank a shortlist. After generation, it may test the draft against safety and commercial rules. At any point, it may decide that a human should take over.

An LLM can perform these tasks, but the application usually has to request JSON, validate the schema and decide how to handle malformed or ambiguous output. TypeSafe says Jev is built around predefined answer types instead. Its public API reference documents a POST request to https://api.typesafe.ai/v1/systemone with state and questions.

What Jev returns

The TypeSafe API documents three answer families: noul for a probability, choice for one option from named criteria, and score for a position on a defined scale. The unusual word noul appears in the direct TypeSafe API. Vercel uses the more familiar name boolean in its gateway interface.

Here is the reduced shape of our positive PII test:

{
  "model": "jev-latest",
  "state": "Hi, my order hasnt arrived. I am Anna Lim, phone +65 9123 4567, card ending 4242.",
  "questions": {
    "has_pii": {
      "type": "noul",
      "instructions": "Does this message contain personal data such as a name, phone number, email, address or payment details?"
    }
  }
}

The response identified model version jev-1.13.0 and returned:

{
  "answers": {
    "has_pii": {
      "type": "noul",
      "noul": 0.99
    }
  }
}

That value is not a policy by itself. Application code must convert it into an action through a threshold and an uncertainty path.

1. Check for PII before the LLM

A PII check belongs early in the pipeline, before a message is copied into prompts, logs or downstream tools. In our synthetic positive example, Jev returned 0.99. For the invented message, “Do you have the linen shirt in size M, and is it machine washable?”, it returned 0.02.

A simple policy might redact or block messages at or above 0.90, pass messages below 0.20, and send the middle band to a stricter detector or human review. Those numbers are illustrations, not recommended universal thresholds. Missing PII can have a much higher cost than pausing an ordinary product question, so the threshold should reflect the data flow and legal context.

Detection is only one layer. The pipeline still needs minimization, retention controls and clear boundaries around what reaches external models. Our guides to protecting PII and chatbot data privacy cover that wider system.

2. Apply guardrails to draft replies

Guardrails can inspect an LLM draft before the customer sees it. Our synthetic risky draft prescribed “800mg of ibuprofen three times a day” while selling a posture cushion. One call asked two independent questions: whether the reply gave medical advice and whether it recommended a competitor.

Jev returned 0.99 for medical advice and 0.03 for a competitor recommendation. A clean draft about shipping in two to four days and a 30 day return window returned 0.01 for both checks.

The useful pattern is specific checks rather than one vague “Is this safe?” question. Each check can have a different consequence. A high medical advice probability could block the draft and request a rewrite. A competitor mention might trigger review or an approved comparison flow. The check evaluates the provided text. It does not prove that shipping or return statements are factually correct, so those claims still need grounding in trusted business data.

3. Pick from a retrieved catalog shortlist

A product decision model should not search an entire catalog from memory. Retrieval should first produce a current shortlist with relevant attributes. The decision step can then select among known identifiers.

Our synthetic shopper wanted road running shoes for a half marathon, mentioned mild overpronation and had a budget near 150 dollars. The four candidates included a trail shoe, a stability road shoe, a neutral carbon racer and a walking shoe. Jev chose sku_road_stability, the 140 dollar road shoe with stability support, and returned a probability map over all four identifiers.

This keeps the output inside the retrieved set. It does not validate inventory, price or product specifications. Those fields must come from the catalog source, and the final reply should cite or reflect that source. See our approach to knowledge management for AI chatbots and catalog chat progress streaming.

4. Route intent and estimate buying intent

One request can answer several related questions. For the invented message, “do you ship to Singapore and how much would 3 of the blue ones cost?”, we asked for an intent choice and a buying intent probability.

The intent options were product question, order status, complaint and other. Jev selected product_question and returned 0.89 for buying intent. The router could use the choice to fetch shipping and pricing information. The buying signal could alter which actions are offered, but it should not be treated as proof that a purchase will happen.

The option definitions matter. Overlapping labels create ambiguous supervision and unstable routing. Include a fallback, log the probability distribution and review examples where the top choices are close.

5. Score human handoff urgency

Escalation is often better represented as a scale than a yes or no flag. Our test defined three levels: no handoff needed, offer a human as an option, and hand off to a human now.

The synthetic customer said this was the third request, the refund had not arrived, the bot kept repeating the same FAQ link, and a person was wanted immediately. Jev returned score 2.0, mapped to immediate handoff, with probabilities for every level.

Real routing should combine that score with operational facts such as agent availability, account status and support hours. A model score can prioritize a queue, but it should not silently deny access to a person when policy promises human support.

Set thresholds around the cost of mistakes

A probability becomes useful only when connected to a policy. Start by naming the two mistakes for each question. For PII, they are letting sensitive data through and interrupting a safe message. For guardrails, they are sending a harmful draft and blocking an acceptable reply.

Choose an automatic action threshold using labelled examples from the real traffic domain.
Create a middle band for uncertainty. Route borderline cases to a human, a deterministic rule or an LLM with more context.
Measure calibration. Among cases scored near 0.80, the positive rate should be examined rather than assumed.
Set thresholds independently for each use case. PII, buying intent and handoff urgency carry different costs.
Monitor drift when products, policies, markets or customer language change.

TypeSafe says Jev returns probabilities so software can account for uncertainty. That is a product claim, not a substitute for measuring calibration on the population where the chatbot will operate.

Limits, and where an LLM is still needed

Jev does not write replies. TypeSafe explicitly presents it as a decision model rather than a text generator. The company also describes inputs as unstructured data, with its published examples centered on text and structured program state. Treat each API call as self contained: send the relevant conversation state again, because the request shape does not provide chatbot memory.

An LLM is still useful for composing a clear response, asking a tactful follow up question, summarizing a long exchange or explaining a recommendation. Jev can decide which route or candidate to use, while ordinary code constrains actions and an LLM handles language.

Availability also matters. TypeSafe described Jev as early access in its September 15, 2026 announcement. Separately, Vercel says its AI Gateway added Jev support on September 21, 2026 through a TypeSafe client, HTTP API and AI SDK. Check current access and terms before planning a rollout.

Evaluate on labelled data before switching

Seven obvious synthetic cases can confirm that requests serialize, responses parse and answer types fit the pipeline. They cannot show performance on typos, indirect language, conflicting evidence, multilingual conversations or adversarial input.

Build a representative dataset from properly governed, labelled examples. Freeze question instructions and candidate definitions. Split threshold selection from final evaluation. For each use case, report a confusion matrix, precision, recall, review rate and calibration by probability band. For choices, inspect both top choice accuracy and near ties. For scores, measure costly under escalation separately from harmless adjacent errors.

Then run the candidate beside the existing decision path without allowing it to affect customers. Compare disagreements, investigate failure clusters and test degraded behavior for timeouts or unavailable service. Switch only when the measured tradeoff meets the policy for that specific decision.

A short practical rule

Use a decision model when the acceptable outputs can be named before the call. Use ordinary code to enforce policy, retrieve authoritative data and execute actions. Use an LLM when the chatbot needs to write, explain or continue an open ended conversation. When the probability is borderline, escalate instead of pretending uncertainty is certainty.

Want to see how decision checks fit into a product-aware chatbot? Request a demo and we will walk through guardrails and PII handling.

Request a demo

Showing the Work: Progress Streaming for Catalog-Backed Chat

Vitaly Goncharenko — Fri, 18 Sep 2026 02:41:30 +0000

When a catalog-backed chatbot takes more than a second to answer, silence reads as failure. The fix is not making retrieval faster. It is streaming which stage is running so the user knows the system is working.

This post describes a design we are committing to for HoverBot catalog skills: six named progress stages, a capability-gated SSE transport that degrades to a plain JSON call, an absolute wall-clock deadline, AbortSignal cancellation, and five explicit stream outcomes. The design is in open PR #3 and open PR #6, and is not live in production yet. Alexander Khomenko authored the implementation in commit a6d8ac3c across hoverbot-api, hoverbot-config-ui, and hoverbot-widget.

Status: Planned behaviour, not shipped. PR #3 and PR #6 in the hoverbot repo are still open. Treat everything below as the contract we intend to merge, not what you will see on hoverbot.ai today.

Why Silence Fails Before Search Does

Catalog retrieval is a multi-step pipeline. The orchestrator interprets the user's message, resolves product references from prior turns, calls a search backend, validates returned SKUs, ranks candidates, and only then assembles a response. Each step can add hundreds of milliseconds to several seconds depending on catalog size, query complexity, and backend load.

Users do not experience that as a pipeline. They experience a chat bubble with a typing indicator that never changes. Nielsen Norman Group's response-time research identifies one second as the threshold where flow breaks and ten seconds as the point where attention is lost. A catalog query that finishes in four seconds is fast enough to be correct and slow enough to feel broken if the UI says nothing.

The instinct is to optimize latency. That is worth doing, but it does not solve the perception problem. Even a well-tuned retrieval can spike when a user asks a comparison question across three product lines with constraint filters. You cannot guarantee sub-second answers for every catalog turn. You can guarantee the user sees which stage is running.

This connects to the broader knowledge-retrieval picture in knowledge management for AI chatbots: retrieval quality depends on what you fetch, but retrieval UX depends on whether the user waits with context or waits in the dark.

The Six Stages

Progress updates are typed against a fixed stage list. No free-form status strings from the backend; the adapter and widget agree on six values:

export const CATALOG_PROGRESS_STAGES = [
  'interpreting',
  'resolving_reference',
  'retrieving',
  'validating',
  'ranking',
  'building_response'
] as const;

export interface CatalogProgressUpdate {
  stage: CatalogProgressStage;
  detail?: string;
  elapsedMs: number;
}

Each stage maps to a user-facing message in the chat controller:

interpreting: Understanding your request. The orchestrator parses intent, constraints, and search mode before touching the catalog.
resolving_reference: Resolving the products you mentioned. Handles ordinals ("the second one"), pronouns, and context handoff from prior turns.
retrieving: Searching the catalog. The HTTP adapter calls the search backend. This is usually the longest stage.
validating: Checking product information. Confirms returned SKUs exist, are in scope for the tenant, and match the query constraints.
ranking: Ranking the best matches. Reorders candidates by relevance, availability, or business rules before presentation.
building_response: Preparing the results. Assembles the final message, product cards, or clarification prompt the user will see.

The optional detail field carries adapter-specific context (for example, a category name) without expanding the stage vocabulary. The elapsedMs field is wall-clock time since the search started, useful for logging and for deciding when to show a "still working" fallback message.

Not every query runs every stage. A first-turn category browse may skip resolving_reference. A cache hit might flash through retrieving in under 50ms. That is fine. The stages describe what is happening when it happens, not a mandatory sequence with equal duration.

Capability-Gated Transport

Progress streaming is opt-in at three levels. Adapter config exposes progressStreaming?: 'auto' | 'off'. The default in config-ui is 'off'. Tenants turn it on explicitly.

When set to 'auto', the HTTP search adapter checks four conditions before opening an SSE stream:

Config says progressStreaming: 'auto'.
The caller sets supportsProgressStreaming: true.
The caller provides an onProgress callback.
The catalog backend's /health endpoint advertises capabilities.progressStreaming with version: 1, transport: 'sse', and a stages array.

If any check fails, the adapter logs the transport selection and falls back to a standard POST that returns JSON when complete. No error, no broken widget, no second integration path for non-streaming clients.

The capability probe is cached for 60 seconds per base URL so every catalog turn does not pay an extra health round-trip. The transport itself uses Server-Sent Events as defined in the WHATWG HTML specification: a long-lived HTTP response where the server pushes event: and data: frames. SSE fits progress updates because they are server-to-client, unidirectional, and small. The chat API already uses SSE for answer streaming on compatible clients; catalog progress rides the same pattern.

Why degrade instead of requiring SSE everywhere? Embedded widgets run on third-party sites with varied network stacks, corporate proxies, and older mobile WebViews. Some cannot hold an SSE connection reliably. Forcing SSE would break those clients or require maintaining two widget builds. Capability gating lets streaming clients get stage updates and everyone else get the same final answer through JSON.

The Deadline and Cancellation

Streaming progress does not remove the need for timeouts. It makes timeouts legible.

Adapter config includes timeouts.absoluteMs: a wall-clock deadline for the entire catalog operation, progress stream included. Separate connect and read timeouts still apply per HTTP hop, but the absolute deadline caps total user-visible wait regardless of how many stage transitions occur.

Cancellation flows through the standard AbortController / AbortSignal interface. The widget creates an AbortController per request, passes its signal to the chat API, and wires the typing indicator's cancel action to abort(). When the signal fires, the adapter stops reading the SSE stream and throws a transport error with outcome cancelled.

That last part matters. A stream can end five ways, and conflating them loses debuggability:

export type CatalogStreamOutcome =
  | 'cancelled'
  | 'closed_without_result'
  | 'error'
  | 'malformed'
  | 'timeout';

cancelled: User or client aborted via AbortSignal.
timeout: Absolute or read deadline exceeded.
error: Network failure or non-2xx response mid-stream.
malformed: SSE frame parsed but stage name not in the allowed list.
closed_without_result: Stream ended cleanly but no search result arrived.

These outcomes are not user-facing copy. They are the classification layer for logs, metrics, and deciding whether to retry. A cancelled outcome after the user closes the widget should not increment the same error counter as a malformed frame from a misconfigured backend. Transport failures throw CatalogSearchTransportError with the outcome attached so callers cannot accidentally treat a dead stream as an empty result set.

HTTP semantics for long-lived responses are governed by RFC 9110. The practical implication for us: the client must handle connection drops, the server must not assume the client read every event, and both sides need a defined terminal state. Explicit outcomes are that terminal state for catalog progress.

What the Widget Does With It

The widget does not render a progress bar. It updates the typing indicator text:

onProgress: progress => {
  if (progress && typeof progress.message === 'string') {
    this.updateTypingIndicator(progress.message);
  }
}

The API maps each stage to a short sentence ("Searching the catalog…", "Ranking the best matches…") before the event reaches the widget. The widget only displays the string. It does not know about stage enums or elapsed milliseconds.

That is deliberate. A progress bar implies measurable completion. Catalog retrieval does not have a stable denominator. Is retrieving 40% of the work? It depends on the query. A bar that jumps from 30% to 90% in one frame is worse than a label that says what is happening now. Nielsen Norman Group's guidance on progress indicators distinguishes determinate bars (known duration) from indeterminate indicators (unknown duration). Catalog search is indeterminate. Stage names are the honest representation.

The widget also keeps a fallback timer. If no progress event arrives within 17 seconds, the typing indicator switches to "This search is taking a little longer. I'm still working on it…" That covers backends that support streaming but emit sparse updates, and clients where capability gating fell back to JSON mid-flight.

For customer-facing deployments, this sits alongside the automation patterns in customer service automation in 2026: automate the lookup, but keep the human-visible loop honest when the lookup takes time.

What We Gave Up

Progress streaming adds complexity across 20 files. The adapter now maintains two transport paths (SSE and JSON), a capability cache, SSE frame parsing, and outcome classification. Every new catalog backend must advertise progress capabilities in its health endpoint or streaming silently degrades. That is the intended behaviour, but it means backend teams have a contract to implement.

Fast stages look silly. When validating finishes in 12ms, the user may see "Checking product information…" flash for a single frame. We considered suppressing stages below a minimum display time and rejected it. Artificial delays lie about system speed. A flash is honest; a forced 500ms pause is theater.

Default is off. Tenants must enable progressStreaming: 'auto' in catalog skill config. We did not ship it as the default because not every catalog backend supports SSE progress yet, and we would rather have tenants opt in once their backend is ready than have streaming fail open on every turn.

Observability gets harder before it gets easier. Five outcome types means five buckets in dashboards instead of one "search failed" counter. The payoff is that on-call can distinguish user cancels from backend timeouts without reading stack traces.

What Ships Next

When PR #3 and PR #6 merge, catalog skills with progressStreaming: 'auto', a streaming-capable widget, and a backend that advertises SSE progress will show stage updates during retrieval. Everything else continues to work as a plain JSON search with a static typing indicator.

The design does not make catalog search faster. It makes the wait interpretable. For a chat interface backed by a live product catalog, that is the difference between "broken" and "working on it."

Want to see catalog-backed chat with progress streaming once it ships? Request a demo and we will walk through the catalog skill configuration.

Request a demo

Knowledge Management for AI Chatbots: Structure, Maintain, Improve

Vitaly Goncharenko — Wed, 16 Sep 2026 15:13:03 +0000

A retrieval-augmented chatbot can still give a bad answer when retrieval finds the wrong passage, an outdated passage, or nothing useful. Before changing the model, inspect the evidence it received. That separates a retrieval failure from a generation failure and points to a fix you can test.

Knowledge management for AI chatbots is the discipline of organizing, maintaining, and improving the content your assistant retrieves from. This guide covers how to structure content for retrieval-augmented generation (RAG), how to keep it fresh, and how to use analytics to find and close gaps systematically.

Why Knowledge Is the Real Bottleneck

In a RAG system, every answer flows through the same pipeline: the user's question is used to retrieve relevant chunks of your content, and the model composes an answer grounded in those chunks. If retrieval surfaces the wrong chunk, an outdated chunk, or no chunk at all, the answer suffers no matter how capable the model is.

Knowledge structure is one part of that pipeline teams can change directly. You can restructure source content, change chunking and metadata, then test whether the intended passages appear for representative questions. For the broader architecture context, see multilingual RAG architecture.

Structure Content for Retrieval, Not Just Reading

Content written for humans browsing a help center is often poorly suited for retrieval. A few principles make a large difference:

  - **One topic per section.** Self-contained sections retrieve cleanly; sprawling articles that cover five topics retrieve ambiguously.

  - **Front-load the answer.** State the answer near the top of each section so a retrieved chunk carries the substance.

  - **Use explicit headings.** Headings that mirror how customers phrase questions improve matching.

  - **Avoid pronoun chains across sections.** A chunk should make sense on its own, without the paragraph before it.

Chunking Strategy: The Quiet Lever

Chunking decides what unit of content gets embedded and retrieved. Chunks that are too large dilute relevance and bury the answer; chunks that are too small lose the context needed to answer well. The sweet spot is usually a coherent section: large enough to stand alone, small enough to be specific.

Prefer structure-aware chunking that respects headings and natural boundaries over naive fixed-length splitting. Overlapping a little context between adjacent chunks helps preserve meaning at the edges. Then score retrieval at the chunk level so you can see which chunks actually answer questions and which never get used.

  **Practical tip:** If a single article answers many different questions, test focused sections or entries against the original. The useful unit is the one that retrieves the complete answer for your evaluation questions without bringing unrelated material with it.

Metadata Tagging for Precision and Freshness

Metadata turns a flat pile of content into something you can filter and govern. Tag chunks with attributes like product area, audience, language, region, and last-reviewed date. This enables more precise retrieval, lets you scope answers to the right context, and makes freshness auditable.

A last-reviewed date in particular is the backbone of maintenance: it tells you and the system which content is aging and may need a human check before it keeps answering customers.

Maintain Freshness Without a Full-Time Librarian

Knowledge decays. Policies change, products ship, and yesterday's correct answer becomes today's complaint. The fix is a lightweight recurring process rather than a heroic annual cleanup:

  - Flag content past its review date for a quick human check

  - Tie knowledge updates to product and policy release cycles

  - Retire or merge chunks that never get retrieved

  - Promote answers that resolve well into canonical, well-structured entries

Use Analytics to Find Gaps Systematically

An unanswered or low-confidence question is a useful diagnostic signal. Cluster those conversations, inspect the retrieved passages, and separate missing content from weak retrieval or an answer-generation problem. When the source material is missing, write a focused entry and add the original question to the retrieval test set. We covered the loop in depth in close the loop, and the analytics surface is described in the analytics deep dive.

A Weekly Maintenance Workflow

1. Review misses. Look at clustered unresolved and low-confidence conversations from the week.

2. Triage. Decide which gaps are worth fixing now based on volume and impact.

3. Write or restructure. Add or reshape content as self-contained, well-headed chunks.

4. Tag. Apply metadata and a fresh review date.

5. Verify. Confirm the new content actually gets retrieved for the target questions.

Keep the review small enough to repeat. The important result is not time spent or documents edited. It is whether the changed content is retrieved for the target questions and supports a correct answer in the same evaluation set.

Where HoverBot Fits

HoverBot ingests and chunks source content for retrieval, attributes answers to sources, and surfaces unresolved conversations for review. That gives a team evidence to inspect when an answer fails instead of treating the model as a black box. The knowledge-base tooling is detailed in the knowledge base management deep dive, with the wider system in the technical overview.

Want to inspect how grounded retrieval behaves on your own content? Request a demo and test HoverBot against questions from your knowledge base.

  [Request a demo](https://www.hoverbot.ai/request-demo)

The Bot's Job Starts Where a Good Email Ends

Vitaly Goncharenko — Tue, 15 Sep 2026 14:55:40 +0000

The hardest ecommerce chatbot decision is not which model to use. It is deciding which questions the bot is allowed to answer.
Start with the questions customers already ask: order status, delivery windows, stock, returns, product fit, and exceptions. Some have a current answer in a system your team trusts. Some can be prevented with a clearer email or product page. Some need a person. Treating all three groups as one automation project creates avoidable risk.

Map the answerable surface

Call that the answerable surface: the questions for which a correct answer already exists in a source the chatbot can reliably reach. That might include order state, delivery window, stock, or policy. Outside that surface the bot is guessing. A fluent guess is often worse than no answer because the customer cannot see the uncertainty until it costs them time or money.
Before automating a question, write down the source that should answer it and the condition that makes the answer current. A policy page may be enough for a general return window. It is not enough for a specific order state. If the source is missing, stale, or unreachable, the correct behavior is a visible boundary and a useful next step.

Use the cheaper-surface test

A question belongs in chat only if it is answerable and a cheaper surface does not solve it first. Put a tracking link in the shipment notice. Explain the return window on the product page. Send a plain delay update when the expected date changes. Chat should handle the remaining questions that still benefit from a conversation.
The same rule explains two common failures. A bot pointed only at generic FAQ text guesses when a question needs current operational data. A bot that tries to pass as human hides the boundary instead of managing it. One is an integration problem; the other is an honesty problem.
A demo cannot prove that production sources are connected, current, or complete. It can help you inspect the interaction: whether the answer is specific, whether the boundary is visible, and whether the next step makes sense. Verify production claims separately against the intended data and workflow.

A five-step scoping exercise

List the ten questions your team receives most often.
Remove the questions a better email, product page, or status notice can prevent.
For each remaining question, identify the exact source that holds the correct answer.
Automate only the questions whose source is reliable and reachable.
Define a visible next step for everything outside that boundary. ## Run a three-prompt review For each workflow, prepare three prompts before launch:
Answerable: the approved source contains one specific expected answer.
Missing: the source does not contain the requested detail, so the bot should say so.
Human: the request needs judgment or action, so the bot should route it clearly. Record the expected answer, source, boundary, and next step before testing. A response that merely sounds fluent does not pass. Where HoverBot fits For a design-partner pilot, bring one bounded support or lead-capture workflow. The useful starting point is an accessible source, one expected answer, one boundary case, and a human next step. Applying starts a fit review; it does not create an account or guarantee an invitation. Fix preventable questions first. Then map what remains. That is the chatbot's actual job. Explore the catalogue demonstration Apply for a design-partner pilot

Four chatbot widget patterns for websites and apps: from bubble to super app

Vitaly Goncharenko — Mon, 14 Sep 2026 12:14:26 +0000

Websites tend to embed chat in four patterns: a simple bubble, an inbox with history, a task-driven support bot, and a multi-tab hub.

This guide explains where each pattern fits, what it does well, and the least you need to configure to run it reliably.

(Acronyms: personally identifiable information (PII); service-level agreement (SLA).)

Figure 1: Simple chat bubble widget interface

1. Simple chat bubble

What it is:

The simple chat bubble is a lightweight, single-thread assistant that opens when you click, answers your question, and then closes. It has no inbox or account linking and only limited memory.

Best for:

Marketing pages and documentation landers
Lead capture ("ask a question → leave email")
Narrow, high-confidence FAQs

Strengths:

Fast to ship; minimal UI surface
Low maintenance and risk
Clear focus on the current question

Trade-offs:

No persistent history by default
Limited multi-step tasks without tools/actions
Harder to measure long-term outcomes

Minimum configuration:

Greeting and scope: short, explicit welcome
Knowledge base: 5–20 curated docs
Guardrails: refusal policy, safe fallbacks, and topic blocks
Intent capture: 3–5 quick-reply buttons
Lead handoff: email form on low confidence or on request
Analytics: impressions, opens, first response time, resolved vs escalated

Figure 2: Messages list with history widget interface

2. Messages list with history

What it is:

An inbox-style widget. Users can open past threads, resume conversations, and see bot or agent follow-ups.

Best for:

SaaS apps and customer portals
Education and healthcare portals where continuity matters
Sales cycles that run over days or weeks

Strengths:

Conversation memory within and across sessions
Asynchronous support ("reply when ready")
Clear UX for escalations and status updates

Trade-offs:

Identity and storage decisions
More compliance exposure (PII, retention, export)
Operational overhead: SLAs, routing, backlog management

Figure 3: Product-support chatbot widget interface

3. Product-support chatbot

What it is:

A support-focused widget built for tasks: look up orders, reset passwords, file tickets, schedule, check refund eligibility. Think "chat + tools."

Strengths:

Measurable deflection and faster resolution
Clear ROI when tools are reliable
Structured analytics (top tasks, failure points)

Trade-offs:

Integration work (CRM, ticketing, order and billing APIs)
Precision demands: weak tools create loops and churn
Ongoing maintenance of intents, prompts, and policies

Figure 4: Multi-tab widget (super-app) interface

4. Multi-tab widget (the "super-app")

What it is:

A docked panel that bundles modules such as Home, Messages, Help, and News/Announcements; sometimes Tasks or Shortcuts.

Best for:

Multi-feature products and communities
Enterprise portals and intranets
Sites that need to surface announcements, docs, chat, and actions in one place

Picking the right pattern

Scenario	Choose	Why
Marketing site; FAQs and leads	Simple bubble	Fast, low risk, clear CTA to sales
Logged-in product; conversations span days	Messages with history	Continuity and async service
Measurable deflection on top 10 tasks	Product-support chatbot	Tooling and policies drive ROI
Multiple help surfaces: chat, docs, news	Multi-tab widget	One hub; consistent entry point
Heavily regulated flows (health/finance)	Messages with history or Product-support	Auditability, retention, policy enforcement

Rule of thumb: Start with the simplest widget that delivers the outcome. Move up a level only when continuity, task depth, or surface area require it.

Bottom line

Pick the smallest widget that solves the user's job today. Add depth, such as history, tools, or multiple tabs, only when your product and users clearly need it. The gains come from clear scope, reliable tools, and disciplined measurement, not from the fanciest UI.

Multilingual RAG Architecture That Works in Production

Vitaly Goncharenko — Sun, 06 Sep 2026 15:49:12 +0000

A battle-tested architecture for multilingual RAG: translate at the edges, reason in one base language, and protect entities throughout. This is not theory. We run this in production across 14 languages with 50M+ queries processed.

Why most multilingual RAG systems fail

Teams building multilingual RAG make the same mistakes repeatedly. They run separate vector indices per language and wonder why retrieval quality varies wildly. They trust multilingual embedding models to handle languages they have never tested. They let translation services silently mangle product names and order IDs.

The failure modes are predictable:

Embedding drift: Multilingual models cluster similar concepts in different regions of the vector space depending on language. A query in Japanese may not retrieve the same documents as its English equivalent. We measured 34% retrieval disagreement between EN and JA queries for the same underlying content.
Translation corruption: "Order #SKU-2847" becomes something else entirely, or gets interpreted as natural language and garbled. In one client deployment, 12% of SKU references were corrupted before we implemented entity protection.
Chunking failures: Sentence splitters designed for English break CJK text mid-phrase, destroying semantic coherence. A chunk that ends mid-sentence retrieves poorly and generates worse.
Groundedness collapse: When retrieval is weak, models hallucinate. When answers are translated back, hallucinations get laundered into plausible-sounding text. Users cannot tell the difference.

Our position: The only reliable architecture keeps retrieval and reasoning in one base language. Translate at the edges. Protect entities throughout. Everything else is hope dressed as engineering.

The architecture in one diagram

Translate at edges, reason in base language, protect entities throughout

The flow has five stages:

Ingest: Content arrives in any language. Detect, normalize, protect entities, translate to base language, chunk, embed, index.
Query translation: User query arrives. Detect language, protect entities, translate to base language.
Retrieval: Search the base-language index. Rerank with cross-encoder. Keep top-k.
Generation: Generate answer in base language with citations.
Back translation: Translate answer to user language, restore protected entities, localize formats.

Why one base language beats the alternative

The intuitive approach is to build per-language indices. Query in Japanese, search Japanese index, generate in Japanese. This sounds elegant until you operate it.

Factor	Per-Language Indices	Single Base Language
Index count	N indices (one per language)	1 index
Embedding model tuning	Tune N models or accept variance	Tune 1 model
Cache efficiency	Fragmented across languages	Unified cache
Content gaps	Some languages have less content	All content available to all users
Debugging	Check N code paths	Check 1 code path + translation
Quality consistency	Varies by language	Consistent (translation quality permitting)

The single base language approach adds translation latency (typically 50-150ms per direction). But it gives you one vector store to tune, one reranking stack to secure, and one grounded generation policy to validate. When something breaks at 3am, you want one path to debug, not fourteen.

Embedding model selection: the data you need

Multilingual embedding models vary dramatically in quality across languages. "Supports 100+ languages" means nothing without benchmarks on your actual languages.

Model	EN Recall@10	JA Recall@10	ZH Recall@10	Cross-lingual Agreement
text-embedding-3-large	94%	87%	89%	81%
multilingual-e5-large	92%	91%	90%	88%
bge-m3	93%	90%	92%	86%
Single-language + translation	94%	93%*	92%*	91%

*Via translation to English before embedding. Benchmarks from our internal eval set (customer support domain, 10K queries).

Key insight: Translation to a base language before embedding often outperforms native multilingual embeddings, especially for less common languages. The translation step adds latency but improves cross-language consistency.

Entity protection: the difference between working and broken

Translation services will mangle anything that looks like natural language. Protect entities before any external call:

Entity types to protect

SKUs and order IDs: Pattern-based detection for alphanumeric codes (e.g., /[A-Z]{2,4}-\d{4,}/)
Brand names: Glossary-based exact match with case-insensitive variants
Code blocks: Preserve exactly as written, including whitespace
URLs and emails: Standard pattern matching
Product model numbers: Often alphanumeric, easily corrupted
Measurement values: "5.2mm" can become "5.2 millimeters" or worse

Replace detected entities with placeholders like [[ENTITY_0]], [[BRAND_1]], etc. Store the mapping. After translation, restore the originals.

// Before translation
Input: "Where can I find the SKU-4829 brake kit for BMW M3?"
Protected: "Where can I find the [[SKU_0]] brake kit for [[BRAND_0]] [[MODEL_0]]?"
Map: { SKU_0: "SKU-4829", BRAND_0: "BMW", MODEL_0: "M3" }

// After translation (Japanese)
Translated: "[[BRAND_0]] [[MODEL_0]]の[[SKU_0]]ブレーキキットはどこで入手できますか？"

// After restoration
Final: "BMW M3のSKU-4829ブレーキキットはどこで入手できますか？"

Glossary management: Version your glossaries. When "refund policy" gets translated inconsistently across documents, you lose term consistency in your knowledge base. Pin glossary versions at ingest time and log which version was used. When you update the glossary, re-translate affected content.

Script-aware chunking: where most implementations break

Chunking is where multilingual RAG systems quietly fail. Standard sentence splitters assume whitespace-delimited words. CJK scripts do not work that way.

Script Type	Languages	Chunking Approach	Libraries
Latin/Cyrillic	EN, ES, FR, DE, RU	Sentence splitting on punctuation	spaCy, NLTK, standard splitters
CJK (Chinese)	ZH	Character-based with jieba segmentation	jieba, pkuseg
CJK (Japanese)	JA	Morphological analysis	MeCab, SudachiPy
CJK (Korean)	KO	Morphological analysis	KoNLPy, Mecab-ko
Thai	TH	No spaces between words; requires segmentation	PyThaiNLP, ICU
Arabic/Hebrew	AR, HE	RTL-aware sentence splitting	CAMeL Tools, spaCy

Structure preservation: Keep tables and lists intact. A table row split from its header is useless. Carry breadcrumbs (Title › Section › Subsection) into each chunk for context. This matters especially for technical documentation where hierarchy provides meaning.

Retrieval with confidence-based widening

When translation confidence is low, widen retrieval to compensate for potential query drift:

High confidence (>0.85): Standard retrieval with base k (typically k=5). Trust the translation.
Medium confidence (0.70-0.85): Double the retrieval candidates (k=10), then rerank to original k. Compensates for translation uncertainty.
Low confidence (<0.70): Also search the original query text before reranking everything together. Useful for queries with many entities or domain-specific terms.

A cross-encoder reranker usually pays for itself. It allows you to retrieve more candidates cheaply with bi-encoder similarity, then use the more expensive cross-encoder to select the best passages. Result: fewer and better passages in the final context, which reduces tokens, noise, and hallucination risk.

Grounded generation: keeping the model honest

Keep the generator on a short leash. It should see only the reranked passages and rules that enforce citation-first behavior.

System prompt guidance that works:

You are answering questions based on the provided context passages.
Rules:
1. Only use information from the context passages
2. Cite passage numbers for every factual claim: [1], [2], etc.
3. If the context does not contain the answer, say so explicitly
4. Do not invent information, product names, or specifications
5. Preserve all bracketed tokens exactly as written

Produce a base language draft with citations. Translate that draft back to the user language, restore protected entities, and localize numbers, currencies, and dates. English in the middle keeps reasoning stable as models evolve.

Evaluation metrics that actually matter

Retrieval metrics

Recall@k by language: Does retrieval work equally well across all supported languages? Target: within 5% of English baseline.
Cross-language retrieval agreement: Does the same question in different languages retrieve the same passages? Target: >85% agreement on top-3 passages.
Entity preservation rate: What percentage of protected entities survive the round-trip? Target: 100%.

Generation metrics

Groundedness score: Can every claim be traced to a source passage? Automated checks catch 80% of issues.
Cross-language answer agreement: Do answers to equivalent questions agree factually across languages? Sample and human-review weekly.
Glossary consistency: Are key terms translated consistently? Spot-check high-frequency terms monthly.

Operational metrics

Latency by language: Translation adds 50-150ms per direction. Track p50/p95 per language to catch regressions.
Token cost by language: CJK languages often tokenize inefficiently (2-3x more tokens for same content). Monitor cost per query.
Human handoff rate by language: Are certain languages causing more escalations? May indicate retrieval or translation quality issues.

Common pitfalls and how to avoid them

Running multiple indices per language. Multiplies complexity without improving quality. One base language index is easier to tune, cache, and debug. We tried per-language indices early on and reverted within 3 months.
Trusting multilingual embeddings blindly. Test retrieval quality per language before going live. Embedding models have uneven performance across languages. Build a parallel eval set with queries in each supported language.
Letting translation corrupt entities. Always protect SKUs, order IDs, brand names, and code before any translation call. This is non-negotiable.
Chunking CJK text with English tools. Standard sentence splitters break on whitespace. CJK needs specialized segmentation. Use the right library for each script.
Skipping back-translation quality checks. The answer looks right in English does not mean it looks right in Arabic. Verify entity preservation and format localization. Sample and review regularly.
Ignoring low-confidence cases. When translation confidence is low, widen retrieval. Consider asking for clarification instead of guessing. A "could you rephrase?" is better than a wrong answer.

Implementation checklist

☐ Choose base language (usually English for tooling maturity)
☐ Build glossary with do-not-translate terms and term mappings
☐ Implement entity protection before any translation call
☐ Deploy script-aware chunking for CJK and Thai
☐ Set up translation confidence thresholds and fallback widening
☐ Add cross-encoder reranking to reduce context size
☐ Implement groundedness verification in generation
☐ Build cross-language evaluation suite with parallel queries
☐ Log all translation decisions for audit and debugging
☐ Monitor latency and cost per language
☐ Set up alerts for cross-language retrieval disagreement spikes

The opinionated take

Most teams over-engineer multilingual RAG. They build complex language-detection cascades, deploy multiple indices, and try to tune embeddings per language. This creates operational nightmares and fragile systems.

The simpler architecture works better: one index, one base language, translation at the edges. Yes, you add translation latency. But you get a single system to tune, test, and debug.

Three principles we have learned operating this at scale:

Translation quality beats embedding quality for cross-language consistency. A good translation service plus a monolingual English embedding model often outperforms a mediocre multilingual embedding model. Test both approaches on your actual data.
Entity protection is not optional. Every time we have seen a multilingual RAG system fail in production, entity corruption was in the top three causes. Protect entities before translation, restore after.
Build the eval suite first. You cannot improve what you do not measure. Create parallel queries in all supported languages before you launch. Run cross-language agreement checks weekly.

The sophistication belongs in entity protection, glossary management, and evaluation. Not in retrieval architecture. Get those right, and the rest follows.