DEV Community: Joshua Fields

Designing Explainable AI Matchmaking Systems

Joshua Fields — Wed, 24 Jun 2026 13:22:31 +0000

Matchmaking systems are often framed as an algorithm challenge.

In practice, they are a trust challenge.

People are making personal decisions. If they cannot understand why a system suggested someone, confidence drops quickly. That is why I think explainability is not an optional nice-to-have in compatibility products. It is part of the product itself.

This is how I think about designing AI-assisted matchmaking systems from an engineering and product perspective: conversational onboarding, profile enrichment from transcripts, deterministic scoring, privacy controls, and UX that explains outcomes without overwhelming users.

Why deterministic logic matters

I am not anti-ML.

I use AI where it adds leverage. But for compatibility scoring in user-facing matchmaking, fully black-box ranking can create more product risk than product value.

If two users ask, “Why did we match?”, the answer cannot be:

The model thought so.

That is not enough.

For this kind of product, I prefer a hybrid approach:

Use LLMs to structure messy user inputs into clear profile features
Use deterministic rules and weighted scoring for compatibility
Use transparent explanations generated from explicit matching factors

This gives you consistency, easier debugging, and safer iteration.

Product teams can tune outcomes intentionally instead of nudging prompts and hoping ranking behaviour stabilizes.

Conversational onboarding as data collection

Static forms are quick to build, but they are often poor at extracting real preference signals.

Conversational onboarding can collect richer context, especially when users are not sure how to describe what they want. But the trick is keeping it structured underneath.

A good onboarding flow should feel conversational to the user, but still produce clean data for the system.

I usually think about onboarding in clear intent categories:

values
lifestyle constraints
relationship expectations
communication style
hard boundaries
location and availability
dealbreakers
preferences that are flexible rather than absolute

Each answer can be captured as raw text first, then normalized into structured slots.

The user feels like they are having a guided conversation, while the backend receives clean inputs for scoring.

The biggest mistake is letting conversation transcripts become the source of truth by themselves.

Raw transcripts are noisy. They contain uncertainty, contradictions, phrasing issues, and sometimes throwaway comments. Before they can power matching logic, you need explicit feature extraction and validation.

Transcript-driven profile enrichment

Once transcripts exist, an enrichment pipeline can extract structured profile attributes.

A simple workflow might look like this:

A transcript segment arrives
An extractor job parses candidate traits and preferences
Validation checks confidence and conflicts
Approved updates are written to a profile feature store
The user is asked to confirm important or sensitive changes

Each extracted field should track provenance:

which transcript segment produced it
confidence level
last update timestamp
whether the user confirmed it
whether it conflicts with another known preference

This matters because people change their minds and sometimes express things unclearly.

For example, a user might say they are open to relocating in one session, then later say they are definitely not moving.

The system should not silently overwrite the old field.

It should show a conflict and request confirmation.

That kind of behaviour makes the product feel more trustworthy because the user can see the system is not pretending to understand more than it does.

Compatibility scoring design

I like scoring systems that separate hard filters from soft preferences.

Hard filters are non-negotiables. They might include things like distance limits, age bounds, or specific life goals.

Soft preferences influence ranking but do not immediately exclude candidates.

A practical scoring model might include:

hard eligibility gate: pass or fail
weighted dimensions: values alignment, communication fit, lifestyle overlap
penalties for unresolved conflicts or sparse profile data
confidence multiplier based on onboarding depth
explanation output showing the main positive and negative factors

This keeps the final score interpretable.

You can expose top contributing factors directly in the UI, and product teams can tune weights with controlled experiments rather than retraining opaque models every time requirements shift.

That is the part I care about most.

The system should be something engineers, product people, and users can reason about.

Explainability UX patterns that work

Explainability is not just a backend feature.

It has to be visible and legible in the product.

I usually think about explanations in three layers:

1. Quick reason

A one-sentence summary.

For example:

Strong alignment on communication style and long-term goals.

This gives the user an immediate reason without forcing them through a technical breakdown.

2. Factor breakdown

Show the top 3–5 positive and negative contributors.

For example:

similar communication preferences
overlapping lifestyle expectations
compatible location range
mismatch on long-term travel preferences
low confidence because one profile is incomplete

This makes the result feel explainable without reducing everything to a single score.

3. Preference controls

Users should be able to adjust what matters and see how it affects suggestions.

If someone says location matters less than values, the product should let them express that.

This is where explainability becomes interactive rather than just descriptive.

Safety, moderation, and abuse controls

Any social product needs abuse prevention from day one.

Matchmaking is no exception.

I separate safety controls into three layers:

content safety checks for onboarding and messaging
behavioural risk signals, such as spam patterns or repeated boundary violations
human escalation paths for ambiguous or high-severity cases

AI can help classify risk, but I would avoid purely automated irreversible actions for edge cases.

Human-in-the-loop review is still important when context is nuanced.

Systems should optimize for user safety and fairness, not only engagement metrics.

A matchmaking product that maximizes engagement at the expense of user trust is building the wrong thing.

Privacy and data minimization

Matchmaking data is highly sensitive.

People may share relationship history, personal preferences, boundaries, family goals, lifestyle constraints, and location information.

The safest data is often data you do not collect or do not retain long-term.

Some practices I think are important:

separate personally identifying data from preference features
store only features needed for matching and explanations
support user-initiated deletion with clear propagation
avoid exposing full transcript history in operational dashboards
redact sensitive fields in logs
use clear retention policies
make it obvious what is used for matching and why

These are not just compliance tasks.

They are product trust features.

If users do not trust how their data is handled, they will not trust the matches either.

Operational architecture

On the engineering side, a clean architecture for this domain might include:

a profile service for structured attributes
a transcript processing pipeline for enrichment jobs
a scoring service with deterministic logic and versioned weights
an explanation service that maps score components to readable output
an audit layer for traceability and debugging
a moderation layer for safety review and escalation

I prefer versioning scoring configurations so you can compare match outcomes over time and roll forward or back safely.

This also makes experimentation cleaner because you can tie user cohorts to explicit scoring versions.

If a change makes matches feel worse, you need to know exactly which scoring version caused it.

Without versioning, it becomes hard to separate model behaviour, scoring logic, and product changes.

How this maps to Needle-style products

In products like Needle-style matchmaking experiences, conversational onboarding and explainable compatibility are especially important because users are often evaluating whether the product understands them at all.

The first few suggestions become a credibility test.

A practical flow might look like this:

Voice or chat onboarding captures intent and constraints
LLM extraction creates structured profile candidates
User confirms key fields before matching
A deterministic scorer ranks candidates
The UI explains each match with clear factor breakdowns
Users adjust preferences over time
The system learns from confirmed changes rather than guessing silently

The point is not to make the product feel like a spreadsheet.

The point is to make the AI feel accountable.

If the system recommends someone, the user should be able to understand the main reason.

If the recommendation is weak, the system should be honest about that too.

For the broader product context, I have a related project page here:

Needle AI matchmaking case study — coming soon

What to measure

Explainable systems still need measurable outcomes.

Some metrics I would track:

onboarding completion and drop-off by step
profile completeness
conflict resolution rate
match acceptance rate after explanation shown
user edits to preference weights over time
safety intervention rates
review turnaround time
number of matches rejected because of unclear reasoning
number of matches accepted after viewing explanation details

If acceptance rises after users view explanations, you are probably on the right path.

If users frequently override the same preference dimension, your weighting model likely needs adjustment.

If users do not trust the explanation, the explanation is not doing its job.

Closing thoughts

Explainable AI matchmaking is not about reducing everything to a score.

It is about giving users understandable, adjustable, and trustworthy outcomes.

LLMs are useful for handling unstructured input. They can turn messy conversations into structured candidate features. But deterministic logic remains powerful where product trust and accountability matter most.

From an engineering perspective, the best systems are the ones product, design, and backend teams can reason about together.

If you can explain your architecture to both an engineer and a user in plain language, you are probably building the right thing.

Original Post

Building Real-Time Voice AI Applications with LiveKit and FastAPI

Joshua Fields — Tue, 23 Jun 2026 10:40:54 +0000

Real-time voice AI demos are easy to make look impressive.

Production voice systems are a different problem.

The demo usually has one clean interaction, one happy path, and no load. Production has packet jitter, user interruptions, reconnects, flaky speech recognition, delayed TTS, provider failures, and all the little timing issues that make a conversation feel robotic if you do not design for them upfront.

This is how I think about building real-time voice AI applications with LiveKit and FastAPI in a way that can actually ship.

It is less about one framework trick and more about architecture decisions: where state lives, where latency accumulates, how retries work, and what to observe before users tell you something feels off.

Reference architecture at a glance

A practical voice AI stack has a few clear layers:

Client: browser or mobile app that captures mic input, streams audio, and plays synthesized responses
Voice session layer: session identity, auth tokens, connection lifecycle, and per-user context
LiveKit room: low-latency media transport and participant coordination
STT pipeline: speech-to-text with partial and final transcripts
LLM orchestration: prompt construction, tool calls, memory policy, and response shaping
TTS pipeline: text-to-speech chunks streamed back to the user
Backend APIs: FastAPI services handling session state, business actions, and persistence
Observability: metrics, traces, logs, and replay signals for latency and failure analysis

I try to keep each layer independently testable.

If the orchestration logic can only run when a full audio room is active, debugging becomes painful fast.

Client and session boundaries

The client should do as little product logic as possible.

It captures audio, handles UI state, and relays events. It should not decide authorization or business outcomes.

For every voice session, I prefer generating a short-lived token on the backend, scoped to a single room and participant role. That keeps room access controlled and avoids broad credentials leaking into the frontend.

I also keep a server-side session record keyed by a stable session ID.

That record can track:

user ID, if authenticated
room ID
started timestamp
current mode
latest orchestration state
current turn state
reconnect status

When a user reconnects, the backend can recover context without guessing from client memory.

That matters because voice sessions are not just request-response flows. They are live interactions with timing, interruptions, and state.

LiveKit room design decisions

LiveKit gives you the real-time substrate, but you still need conventions.

For most assistant-style experiences, I prefer one room per active user session. It keeps event scopes clear and reduces accidental cross-talk.

If the use case requires multiple participants, such as a user, an AI agent, and a supervisor, define explicit participant metadata and role handling early.

Two patterns help a lot:

Use data channels for control events like interrupt, mute, handoff, and system status
Treat room events as first-class telemetry, not just infrastructure logs

Join, leave, track publish, track unpublish, reconnect, and bitrate drops are all product signals.

That event data becomes crucial when someone says:

The assistant started speaking over me.

At that point, you need to know whether the issue was model latency, voice activity detection timing, TTS cancellation, or a reconnect edge case.

STT: partials, finals, and confidence handling

Speech-to-text should emit partial transcripts quickly for responsiveness.

But downstream business logic should usually wait for final segments or confidence thresholds.

If you run every partial transcript through your orchestration loop, you create race conditions and noisy model calls.

I usually think about transcript events in explicit states:

partial: render to UI, but do not commit to durable context
final: append to conversation history and trigger orchestration
revised final: patch a previous segment if the provider corrects recognition

This makes transcript behaviour deterministic and easier to test.

It also avoids subtle bugs where the assistant answers a phrase the user never actually said in the final transcript.

LLM orchestration in FastAPI

For orchestration, I usually expose a FastAPI endpoint or event handler that receives normalized transcript events and returns structured actions rather than raw prose.

The action envelope might include:

assistant text
tool calls
state updates
UI directives
follow-up prompts
confirmation requests
escalation signals

When teams skip this structure, the orchestration layer turns into prompt glue and ad-hoc branching.

I prefer strict schemas, even when the model itself is flexible.

With schema-first orchestration, you can validate outputs, reject malformed actions, and retry safely without duplicating side effects.

This is where FastAPI works well for me: clear request models, async handling, and a straightforward way to compose tool integrations while keeping the contract explicit.

Latency budgets and interruption behaviour

Voice UX is mostly latency UX.

If a response arrives late, users interrupt.

If interruption handling is weak, trust drops quickly.

I like setting a practical latency budget per turn and breaking it down by stage:

STT latency
orchestration latency
tool call latency
TTS startup time
time to first audio byte
total turn completion time

Once you have per-stage timings, optimization decisions become much clearer.

Interruption should be supported as a first-class control flow:

client sends interrupt event immediately
current TTS stream is cancelled
orchestrator marks the prior response as interrupted
next turn continues from a clean state

Without explicit cancellation semantics, the system often continues generating text in the background and then leaks stale context into the next turn.

That is one of the fastest ways to make a voice assistant feel broken.

Retries and idempotency

Retries are inevitable with external STT, LLM, and TTS providers.

The critical point is making retries safe.

I like attaching idempotency keys to orchestration turns and tool executions, then persisting turn state transitions:

received
processing
completed
failed
cancelled

If a timeout occurs and the client retries, the backend can return an existing result or resume from a known step instead of replaying side effects.

This matters a lot when tool calls trigger real actions, such as booking, messaging, account changes, database writes, or payment-related workflows.

A voice system should feel conversational, but the backend still needs to behave like a reliable distributed system.

Observability that actually helps

For voice systems, aggregate uptime is not enough.

I care about metrics that explain the actual user experience:

end-to-end turn latency percentiles
time to first transcript token
time to final transcript
time to first audio byte from TTS
interrupt rate per session
provider error codes by stage
reconnect frequency
average recovery time
cancelled turn count
failed orchestration count

I also like structured logs keyed by session ID and turn ID, so a problematic interaction can be reconstructed quickly.

You do not always need to store raw audio to debug effectively. In many cases, replaying timeline events is enough, and it is much safer from a privacy perspective.

Deployment and scaling notes

On deployment, I treat the voice path as a latency-sensitive service tier.

Keep orchestration workers close to your media region where possible, and avoid unnecessary synchronous hops.

FastAPI services running behind autoscaling can work well, but cold starts and noisy neighbours still matter for interactive voice.

I usually separate concerns into at least two deployables:

API/session control
orchestration workers

That gives you flexibility to scale orchestration independently when usage spikes.

If you rely on Kubernetes, readiness checks should validate downstream dependency health rather than only process liveness.

For a normal API, a process being alive may be enough to accept traffic. For real-time voice, a process that cannot reach STT, TTS, or the orchestration layer is not actually ready.

Security and privacy baseline

Voice products can capture sensitive data by accident.

Even if your first release is small, it is worth setting conservative boundaries early:

short retention for raw transcripts unless explicitly needed
redaction for known sensitive fields in logs
scoped credentials for room tokens and provider APIs
clear user controls for muting
clear consent flows
deletion controls
limited access to session logs
separation between debugging metadata and sensitive content

These boundaries are much easier to create early than retrofit later.

Closing thoughts

Building real-time voice AI applications is not only an LLM problem.

It is a systems problem spanning networking, state management, retries, observability, security, and product interaction design.

LiveKit and FastAPI make an effective foundation, but the quality comes from how you define boundaries and failure behaviour.

For me, the winning pattern is simple:

predictable contracts
explicit state
tight latency feedback loops
safe retries
clear cancellation semantics
observability from day one

That is what keeps the experience feeling conversational while still operating like production software.

Original Post