DEV Community: Joshua Fields

Keeping Revisions in Commit Order on Postgres

Joshua Fields — Wed, 28 Oct 2026 15:48:40 +0000

Anne creates a document. The app inserts the row, tells the authorization service she owns it, and then the payment fails. The document is rolled back. Her permission, sitting in a separate service, is not.

I've written too many transactional outboxes to patch exactly this. pemba keeps permissions in your own database, so the relationship write commits or rolls back with the document. On SQLite that was most of the work. On Postgres it raised the real problem: numbering the writes.

The guarantee

Every write in pemba commits at a revision, a u64 from one counter per store. Reads resolve to exactly one revision, which is what makes at_least=rev (read-your-writes) and at_exactly (stable paging) work. Both depend on one rule: once any reader has observed revision N, no write numbered N or lower may become visible afterwards.

SQLite gives you this for free, because one writer holds the lock from drawing a number until it commits. Postgres runs writers concurrently, and they commit in any order. Draw the number with nextval() at write time and this happens:

T1  draws revision 10
T2  draws revision 11
T2  commits                  a reader sees head 11
T1  commits                  revision 10 appears below a head someone already saw

In the spike below, that naive scheme broke the rule in 39,355 of 39,371 observations.

There's a constraint underneath this. If a write learns its revision when it writes, and commits at some unknown later moment, then every other writer that commits in between has to be either blocked or hidden from readers until it ends. The only ways out are to drop the total order, or to not decide the number until commit.

The options, and a spike to pick one

I built a throwaway Python harness against Postgres 17.11 in Docker. Eight writer threads run on their own connections, and half their transactions stay open 0 to 30 ms after the write to force commits out of order. A ninth writer holds a transaction open for 3 s every 4 s, standing in for a slow request. Four readers resolve the latest revision and fingerprint what they see inside one REPEATABLE READ snapshot. The oracle compares each observation with the final state at the same revision, so any write that appeared under an observed head counts as a violation.

The main run, at READ COMMITTED for 20 seconds (compare the rows with each other, not as absolute numbers):

Scheme	Writes/s	Violations	Writer wait p99	Visibility lag p99
No ordering (control)	807	39,355 of 39,371	5.0 ms	1.7 ms
Lock held to commit	60	0	3,068 ms	1.0 ms
Sequence + watermark	643	0	9.2 ms	2,920 ms
Assign at commit	686	0	6.1 ms	1.8 ms

Lock held to commit is SQLite's approach moved to Postgres. It's correct, but every writer waits out the slow transaction, and taking a store-wide lock in the middle of the app's transaction invites deadlocks. SpiceDB's Postgres datastore worked this way before v1.14.

Sequence plus watermark lets writers draw freely, and a reader's head is the highest revision below the oldest one still in flight. Writers don't wait, but head freezes behind the slow transaction, so for about 3 seconds nobody sees anyone's writes. Finding what's in flight means scanning pg_locks on every read.

Assign at commit doesn't decide the number until the transaction commits. The slow transaction holds the lock only during its own commit, so it blocks nobody and hides nobody.

The fourth option is what SpiceDB does today: store transaction ids (xid8) on rows and make a revision a Postgres snapshot. It's correct, but a revision stops being a u64 in a total order, which is the largest API change pemba could make. Worse, pemba's tables live in your application's database, so they get dumped, restored and logically replicated with it, and stored transaction ids mean nothing after any of those. SpiceDB owns its database and ships a repair command. pemba can't ask every app's restore to run one.

Assigning at commit has a cost too. A write inside your transaction can't know its revision until that transaction commits, so write(conn=) now returns a PendingRevision that resolves after commit. It does on SQLite as well, so portable code never checks which store it's on.

The design that shipped

Each write call draws a batch id, which has no ordering meaning. Rows it creates carry the id in created_batch, rows it deletes carry it in deleted_batch, and the call inserts one row into pemba_pending. A deferred constraint trigger on that table fires at COMMIT, inside the committing transaction. It takes a store-wide advisory lock, draws the next revision with nextval(), and inserts one row mapping batch to revision. That's all it does.

A simplified version of the migration (the real one schema-qualifies every name, pins search_path, and detects when the trigger fires early):

CREATE TABLE pemba_revisions (
    batch_id     bigint      PRIMARY KEY,
    rev          bigint      NOT NULL UNIQUE,
    kind         text        NOT NULL,
    committed_at timestamptz NOT NULL
);

CREATE FUNCTION pemba_assign_revision() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
    PERFORM pg_advisory_xact_lock(1885695330,
        'pemba_revisions'::regclass::oid::int4);
    INSERT INTO pemba_revisions (batch_id, rev, kind, committed_at)
        VALUES (NEW.batch_id, nextval('pemba_rev_seq'),
                NEW.kind, clock_timestamp());
    RETURN NULL;
END $$;

CREATE CONSTRAINT TRIGGER pemba_assign_revision
    AFTER INSERT ON pemba_pending
    DEFERRABLE INITIALLY DEFERRED
    FOR EACH ROW EXECUTE FUNCTION pemba_assign_revision();

The ordering argument rests on one detail. A transaction-level advisory lock is released when the transaction ends, and Postgres makes a committed transaction visible before it releases its locks (xact.c: "this must be done before releasing locks we hold"). So when the next committer gets the lock and draws a higher number, the previous commit is already visible to it and to every later snapshot. Revision order is commit order.

The lock is held from the trigger to the end of the commit. A 1,000-tuple batch commits in 0.95 ms, against 0.85 ms with no commit step at all. Reads at head need no map: open transactions are invisible, so deleted_batch IS NULL means visible. Reads at an older revision join the map, which made a 500-row lookup 34% slower at p50 in the spike.

The surprise: SERIALIZABLE

The first version of the trigger did more. Rows carried a provisional mark, and the trigger rewrote them to the real revision at commit. Rewriting with a predicate UPDATE made 45 to 58% of SERIALIZABLE attempts fail with 40001, because it took SIREAD locks on index pages every writer was inserting into. Rewriting by ctid brought failures down to 2.7 to 3.4% of commits, and I thought that was mostly solved.

It wasn't. With retries and backoff, 0.03 to 0.08% of transactions still failed every retry. The design review didn't accept a documented rate of lost transactions, and I agreed: a correct app that retries must not lose writes.

Before trying a fix, I went back over the harness and found it wasn't measuring the design. Each writer resolved its revision in a new SERIALIZABLE transaction on its own connection. The comparison scheme did no such read, and the real design resolves on a separate engine connection in autocommit. Rerunning the old behaviour put 0.57 to 0.72% of transactions into a 40001, with 1 to 9 give-ups per run. Part of what I'd measured was my own harness.

With the harness fixed, I compared the rewrite against the batch map, which rewrites nothing. Retrying up to 10 times with backoff, over four runs with and without the slow transaction:

the rewrite still lost 5 transactions, in 2 of the 4 runs.
the batch map lost 0 of 71,585. Between 0.10 and 0.25% of its transactions hit a 40001, at or below the comparison scheme without a commit step (0.25 to 0.27%).
a 1,000-row commit went from 5.3 ms to 0.95 ms, because the commit no longer touches the rows.

Living with it

A permanent concurrency test. The spike became revision_order.rs, a Rust port that runs in CI at READ COMMITTED, REPEATABLE READ and SERIALIZABLE, with a long transaction at least twice per run. It asserts zero violations and, at SERIALIZABLE, zero transactions that fail every retry. It also runs a negative control that allocates at write time on the same tables, and fails unless the control shows violations, so the oracle can't pass vacuously. One honest note: with a precondition read in one write in ten, this test sees about 1% 40001s for pemba against about 0.4% for the control. That's an open question in the repo.

SET CONSTRAINTS ALL IMMEDIATE. This fires the deferred trigger at that statement, so the transaction holds the store-wide lock until it ends and behaves like "lock held to commit". In the spike, doing it in the 3 s transaction dropped throughput from 686 to 371 writes/s. Django's TestCase does it after every test. The trigger checks current_query(), and if the statement isn't COMMIT it raises a WARNING naming it. Correctness holds either way. It's a throughput trap, and the README says so.

Retry with backoff. Immediate retries lose transactions with or without pemba: up to 5 instant retries lost 2 to 6 per run with the batch map and 0 to 7 with the comparison scheme. The documented requirement is exponential backoff with jitter.

Round trips. This is what I want to improve next. A check costs about 5 round trips plus 2 to 3 per level of hierarchy walked. On a CI runner where SELECT 1 took 146 µs, a direct hit took 755 µs at p50. A statement cache and batched writes already halved most figures, but what's left is mostly network.

Pick it apart

The Postgres store shipped in pemba 0.1.0-alpha.3. The full reasoning, with every spike table, is in docs/10-postgres.md, and the code is on GitHub. The design leans on a few Postgres behaviours: commit visibility before lock release, deferred trigger ordering, and how SSI treats one index insert per batch. If you know Postgres internals better than I do and see a corner where one of those doesn't hold, I'd like to hear about it. Open an issue.

Originally published at joshuafields.uk.

Building pemba: An Embedded Zanzibar-Style Authorization Engine in Rust

Joshua Fields — Wed, 28 Oct 2026 15:48:09 +0000

Why I wanted authorization inside the process

My last post was about running SpiceDB as platform infrastructure, which is how I built Monstro's permissions system as its sole developer. For a lot of teams that is still the right shape. But SpiceDB, like OpenFGA, Ory Keto and Permify, is deployed as a separate service that the application calls over the network, and for small-to-medium applications that brings three costs:

Operational cost: another service to deploy, scale, secure and monitor.
Latency: every permission check is a network round trip.
Dual writes: relationships live in the authorization service and application data lives in the application database. Creating a document means writing to both, and a partial failure leaves them inconsistent. Teams build outbox patterns and reconciliation jobs to cope.

The third one is the one I kept coming back to. With two systems to keep in sync, there is always a window where a document exists that nobody can open.

pemba is my attempt at removing that gap. It is an embedded, Zanzibar-style relationship-based access control (ReBAC) engine, written in Rust with Python bindings, that runs inside your process and keeps its data in your database. The one-line pitch in the README is "SQLite for authorisation". The schema language is a strict subset of SpiceDB's, so you can move to SpiceDB if you outgrow it, and I can differential-test against SpiceDB with identical schemas.

Joshua-Fields / pemba

Embedded Zanzibar-style authorization for Rust and Python. Permission writes that join your app's own Postgres or SQLite transaction.

pemba

pemba was formerly developed under the codename warden.

pemba is an embedded, Zanzibar-style authorisation engine written in Rust, with Python bindings: relationship-based access control (ReBAC) that runs inside your process and keeps its data in your database. Think "SQLite for authorisation".

You describe your permission model in a schema (a strict subset of SpiceDB's schema language), write relationships such as document:readme#viewer@user:anne, and ask questions: can Anne view the readme? which documents can Anne view?, who can view the readme? Answers follow groups, folders nested teams and exclusions, and every answer names the revision it was computed at.

Because the engine is in-process, a relationship write can join your application's own database transaction: the row you insert and the relationship that grants access to it commit or roll back together. There is no second system to keep in sync, so no outbox, no reconciliation job…

View on GitHub

What it looks like

You describe the permission model in a schema. This is the example from the README's schema guide, with groups, nested folders and a ban list:

definition user {}

definition group {
    relation member: user | group#member
}

definition folder {
    relation parent: folder
    relation owner: user
    relation viewer: user | user:* | group#member

    permission edit = owner + parent->edit
    permission view = viewer + edit + parent->view
}

definition document {
    relation folder: folder
    relation owner: user
    relation viewer: user | user:* | group#member
    relation banned: user

    permission edit = owner + folder->edit
    permission view = (viewer + edit + folder->view) - banned
}

Then you write relationships and ask questions. From Python:

engine = Engine.sqlite("app.db")              # or Engine.memory()
engine.write_schema(open("authz.zed").read())

rev = engine.write(touch=["document:readme#viewer@user:anne"])
engine.check("document:readme", "view", "user:anne", at_least=rev)    # True

Every write commits at a new revision, one counter per store. Every read resolves its consistency to exactly one revision when it starts and reads only that snapshot, so one answer never mixes two points in time. Passing the revision a write returned as at_least= gives you read-your-writes when another process or connection did the write.

The design rule for the schema language is that every schema pemba accepts must parse in SpiceDB and mean the same thing there. Where SpiceDB is surprising, such as a + b & c meaning (a + b) & c, pemba rejects rather than diverges.

The headline feature: writes that join your transaction

pemba's tables live in your database, prefixed pemba_. Because the engine is in-process, a relationship write can join a transaction you already opened:

conn = sqlite3.connect("app.db", isolation_level=None)
conn.execute("BEGIN IMMEDIATE")
conn.execute("INSERT INTO documents(id, title) VALUES (?, ?)", ("readme", "Readme"))
rev = engine.write(touch=["document:readme#owner@user:anne"], conn=conn)
conn.execute("COMMIT")

That example runs verbatim in CI. pemba neither commits nor rolls back. The returned revision becomes readable when you commit and never exists if you roll back. If the connection has no open transaction, the write raises ValueError unless you pass allow_autocommit=True, because otherwise it would not be atomic with your data.

Getting this right was most of the work. All write logic (validation, precondition reads, revision allocation, row inserts and updates) is implemented once in Rust, generic over a small WriteTxn trait with three methods: execute, query and last_insert_id. One implementation drives the engine's own connection. The other is an adapter in the Python binding that forwards the SQL to the caller's connection, so it runs inside the app's open transaction. A few of the details that came out of building it:

Savepoints. The write runs inside SAVEPOINT pemba_write and rolls back to it on error. Without that, an AlreadyExists on the third operation of a batch would leave a revision row and two operations behind for a caller who commits anyway.
Same-file check. The engine checks, on every write, that the caller's connection points at the same database file. I'd planned to cache this by id(conn), but CPython reuses the id() of a collected object, so a connection to a different file could inherit a stale "verified" entry and the write would land in a database the engine never reads.
Revision order equals commit order. SQLite allows one writer at a time, so revisions can't interleave. If a deferred transaction reads a precondition and another connection commits first, SQLite refuses the lock upgrade with SQLITE_BUSY_SNAPSHOT, and pemba surfaces a retryable conflict rather than committing on a stale precondition.

War story one: two copies of SQLite in one process

The first design said that reads use the engine's own connections. In Python, those were rusqlite connections with a bundled SQLite, while the app's sqlite3 module used CPython's own copy. Foreign-transaction mode put both copies on one file in one process, by design.

That turns out to be dangerous. On POSIX, close() drops every fcntl lock the process holds on a file. SQLite works around this by keeping a process-wide list of open files, but each copy of SQLite has its own list, so a close by one copy silently drops the other copy's locks.

I wrote a deterministic probe to see whether this was real. A bundled-copy connection holds BEGIN IMMEDIATE and an uncommitted insert on a WAL database. A separate process tries BEGIN IMMEDIATE and fails, as it should. Then, in the first process, Python runs sqlite3.connect, one SELECT and close(), and the separate process tries again.

The bug was real. The separate process got the write lock and committed, and the bundled connection committed too, believing it had held the lock throughout. Python's close also checkpointed and deleted the -wal and -shm files under the open writer. Afterwards PRAGMA integrity_check said ok, but the other process's committed row was gone. That is silent loss of a committed transaction, which is worse than a crash. It reproduced locally on WSL2 Ubuntu and in all four CI jobs on Linux and macOS. Windows locks per handle and doesn't have this mechanism, and I didn't test it there.

One way out was to link the extension against the interpreter's SQLite. It can't be made reliable for PyPI wheels: a manylinux wheel may not depend on libsqlite3.so.0, python.org macOS builds compile SQLite into the interpreter, and equal version strings prove nothing, because two copies of 3.45.1 still corrupt each other.

The other was a store whose every statement (reads, native writes, migrations, GC) runs on Python sqlite3 connections the engine owns, so only CPython's copy ever touches the file. All authorization logic stays in Rust behind the same traits. The prototype passed all 25 cases of the store conformance suite, and then I measured what it cost:

Single-threaded, a check went from 36 µs to about 100 µs at p50, roughly 2.7×.
With 8 threads and no serialization it collapsed to 190 checks per second, because sqlite3 releases the GIL inside every execute and the threads convoy on it.
Holding the GIL for a whole check deadlocked once threads outnumbered pooled connections.
Serializing requests (take a Rust mutex without the GIL, then hold the GIL for the whole request) gave 6,700 checks per second, with p99 at 620 µs against the native store's 1,440 µs.

I went with the Python-owned store, serialized. Engine.sqlite() now goes through CPython's own sqlite3, the Python package no longer depends on rusqlite, and CI fails if the extension or the wheel contains SQLite. The probe became a permanent test, and the README warns against opening the same database with anything that brings its own SQLite, such as apsw.

The lesson I took from it: "the engine's own connections" had quietly assumed there was only one SQLite library in the process.

War story two: exponential checks on cyclic data

Cycles in data are legal. Groups contain each other and folders point at each other. pemba evaluates them to a fixed point, so a group that contains itself is a finite question (SpiceDB keeps dispatching until it reports max depth exceeded).

The original rule was simple: a node currently being evaluated counts as false for the purpose of breaking the cycle, and any result that depended on that assumption is not memoised. That is correct, but inside a cyclic region it re-walks every path. Cyclic group data is ordinary data that users can write, so a user could make one check arbitrarily expensive. That's a denial-of-service risk, and neither max_depth nor the fan-out limit bounds total work.

The property tests found it. One 1,000-case run took 3.0 seconds. The next, with a different random seed, was still running after 573 seconds when I stopped it. The pinned case had two types and 30 tuples with dense mutual recursion over ids a to f. One check made 32,726 store reads at depth 29, about 170 ms in release, while the reference evaluator answered the whole case in 21 ms.

My first fix, adding lowlink taint and memoising cycle heads, was still exponential. On a complete group graph with the subject in no group, store reads for n = 5 to 9 groups were 130, 652, 3,914, 27,400 and 219,202. The evaluator was walking every simple path, around (n−1)! of them.

The fix that worked is SCC fixed-point iteration, Tarjan-style. Each node of a strongly connected region is evaluated at most once per pass:

A hit on a node still in progress returns its provisional value (false if it has none) and marks it as assumed.
When the head of the region finishes, if no assumed false finished true, the pass is consistent and every value in the region is memoised.
Otherwise the provisional values are raised and the head re-runs.

Provisional values only grow, and a region of |S| nodes needs at most |S| + 1 passes, so one check is bounded at (|S| + 1) · Σ r(x) store reads per region, where r(x) is the reads one evaluation of a node makes. The complete group graph that blew up before now takes one pass and about 2n reads. One subtle bug on the way: lowlinks have to be SCC-stack positions, not call-stack indices, or a sibling that reuses a popped frame's index looks like the head of its own region and settles a false too early.

A cycle that crosses the subtracted side of an exclusion has no well-defined least fixed point, so pemba returns NonMonotonicCycle instead of guessing. And there's a backstop. max_check_reads (default 200,000) caps store reads per check and fails closed with ReadLimitExceeded. It guards against bugs in the rule and against large but legitimate graphs. At an estimated 5 to 20 µs per indexed SQLite point read, which I haven't measured yet, that caps a pathological check at roughly 1 to 4 seconds. After the change, the pinned case makes well under its 1,000-read bound and the 1,000-case property run finishes in about a second.

How I check that it's right

Authorization bugs that grant too much are the ones that matter, so most of the effort went into verification:

A reference evaluator: a deliberately naive, obviously-correct evaluator used only in tests as an oracle. Even the oracle needed fixing: with recursion on an exclusion's base side, one variant of its iteration stopped on a wrong answer that looked like a fixed point. It now stratifies the schema and declines what it can't stratify.
Property tests: check against the oracle on generated schemas and data, with the 1,000-case run as a separate workflow; store equivalence across the memory and SQLite stores; and the formatter, which must keep every comment exactly once.
Differential tests against SpiceDB 1.56.2: 18 fixtures covering 23,621 questions and 500 compared generated cases covering 499,517 questions, with zero unexplained mismatches. Every known difference has a detector that needs the exact shape of the disagreement, so it can't hide an unrelated bug.
Fuzz targets, including one that runs the property-test generators under AddressSanitizer.

Generated cases sent to SpiceDB have no cycles and at most 16 relationships, because on the densest data one SpiceDB bulk check used 2.5 GB and the test server was OOM-killed on a 4 GB machine. Cyclic and larger data go through the fixtures.

The most interesting difference is KD-3. In pemba, a stored wildcard t:* matches every object of type t and never a subject set like t:x#r. SpiceDB matches the subject set too. On the granting side that makes pemba the stricter one, and I kept that. On the subtracted side of an exclusion it made pemba the more permissive one:

definition user {}
definition grp { relation member: user }
definition doc {
    relation viewer: grp#member
    relation banned: grp:*
    permission view = viewer - banned
}

doc:d#viewer@grp:g#member
doc:d#banned@grp:*
check doc:d#view@grp:g#member     -> pemba AMBIGUOUS_WILDCARD_EXCLUSION (was true), SpiceDB false

Before the fix, pemba allowed what SpiceDB denied. That is the fail-open direction, so pemba now refuses to answer there and returns AMBIGUOUS_WILDCARD_EXCLUSION. The reasoning, from the evaluation spec: an error can later become an answer without breaking callers, but an answer cannot change silently.

That doesn't mean every disagreement becomes an error. Where the difference is pemba's documented semantics, I kept them: on cyclic data pemba answers and SpiceDB errors (KD-1), and near the depth limit of 50 SpiceDB's answer depends on timing while pemba's is deterministic (KD-4). More generally, evaluation never turns a store error, a depth limit or an undefined cycle into True or False. It raises, and the caller decides.

What's next

pemba is pre-alpha. check, lookup_resources, lookup_subjects, the memory and SQLite stores, transactional writes and the Python bindings are implemented and tested, including against SpiceDB. Nothing is published to PyPI or crates.io yet, the 0.1 release gate isn't signed off, and you should expect breaking changes.

The biggest gap is performance. A benchmark harness exists, sharing one set of datasets across the Rust, Python and SpiceDB benches, but no benchmark has been run and no targets are set. The numbers in this post come from probes and tests, not benchmarks. The fuzz targets exist, but the 10-minute runs and mutation testing haven't happened. Postgres is planned, not started.

If many services in many languages need the same permissions, or you need multi-region scale or a mature product today, use SpiceDB. pemba is for a single application, or a few services sharing one database, that wants fine-grained sharing without running another service.

Why "pemba"

The project started under the codename warden. pemba comes from Pemba Island in the Zanzibar archipelago, a nod to Google's Zanzibar paper, and home of the Pemba flying fox, which lives nowhere else.

Feedback and issues are very welcome on GitHub. If you've run into the dual-write problem with an external authorization service, I'd like to hear how you handled it.

Designing Explainable AI Matchmaking Systems

Joshua Fields — Wed, 24 Jun 2026 13:22:31 +0000

" width="800" height="450">

Matchmaking systems are often framed as an algorithm challenge.

In practice, they are a trust challenge.

People are making personal decisions. If they cannot understand why a system suggested someone, confidence drops quickly. That is why I think explainability is not an optional nice-to-have in compatibility products. It is part of the product itself.

This is how I think about designing AI-assisted matchmaking systems from an engineering and product perspective: conversational onboarding, profile enrichment from transcripts, deterministic scoring, privacy controls, and UX that explains outcomes without overwhelming users.

Why deterministic logic matters

I am not anti-ML.

I use AI where it adds leverage. But for compatibility scoring in user-facing matchmaking, fully black-box ranking can create more product risk than product value.

If two users ask, “Why did we match?”, the answer cannot be:

The model thought so.

That is not enough.

For this kind of product, I prefer a hybrid approach:

Use LLMs to structure messy user inputs into clear profile features
Use deterministic rules and weighted scoring for compatibility
Use transparent explanations generated from explicit matching factors

This gives you consistency, easier debugging, and safer iteration.

Product teams can tune outcomes intentionally instead of nudging prompts and hoping ranking behaviour stabilizes.

Conversational onboarding as data collection

Static forms are quick to build, but they are often poor at extracting real preference signals.

Conversational onboarding can collect richer context, especially when users are not sure how to describe what they want. But the trick is keeping it structured underneath.

A good onboarding flow should feel conversational to the user, but still produce clean data for the system.

I usually think about onboarding in clear intent categories:

values
lifestyle constraints
relationship expectations
communication style
hard boundaries
location and availability
dealbreakers
preferences that are flexible rather than absolute

Each answer can be captured as raw text first, then normalized into structured slots.

The user feels like they are having a guided conversation, while the backend receives clean inputs for scoring.

The biggest mistake is letting conversation transcripts become the source of truth by themselves.

Raw transcripts are noisy. They contain uncertainty, contradictions, phrasing issues, and sometimes throwaway comments. Before they can power matching logic, you need explicit feature extraction and validation.

Transcript-driven profile enrichment

Once transcripts exist, an enrichment pipeline can extract structured profile attributes.

A simple workflow might look like this:

A transcript segment arrives
An extractor job parses candidate traits and preferences
Validation checks confidence and conflicts
Approved updates are written to a profile feature store
The user is asked to confirm important or sensitive changes

Each extracted field should track provenance:

which transcript segment produced it
confidence level
last update timestamp
whether the user confirmed it
whether it conflicts with another known preference

This matters because people change their minds and sometimes express things unclearly.

For example, a user might say they are open to relocating in one session, then later say they are definitely not moving.

The system should not silently overwrite the old field.

It should show a conflict and request confirmation.

That kind of behaviour makes the product feel more trustworthy because the user can see the system is not pretending to understand more than it does.

Compatibility scoring design

I like scoring systems that separate hard filters from soft preferences.

Hard filters are non-negotiables. They might include things like distance limits, age bounds, or specific life goals.

Soft preferences influence ranking but do not immediately exclude candidates.

A practical scoring model might include:

hard eligibility gate: pass or fail
weighted dimensions: values alignment, communication fit, lifestyle overlap
penalties for unresolved conflicts or sparse profile data
confidence multiplier based on onboarding depth
explanation output showing the main positive and negative factors

This keeps the final score interpretable.

You can expose top contributing factors directly in the UI, and product teams can tune weights with controlled experiments rather than retraining opaque models every time requirements shift.

That is the part I care about most.

The system should be something engineers, product people, and users can reason about.

Explainability UX patterns that work

Explainability is not just a backend feature.

It has to be visible and legible in the product.

I usually think about explanations in three layers:

1. Quick reason

A one-sentence summary.

For example:

Strong alignment on communication style and long-term goals.

This gives the user an immediate reason without forcing them through a technical breakdown.

2. Factor breakdown

Show the top 3–5 positive and negative contributors.

For example:

similar communication preferences
overlapping lifestyle expectations
compatible location range
mismatch on long-term travel preferences
low confidence because one profile is incomplete

This makes the result feel explainable without reducing everything to a single score.

3. Preference controls

Users should be able to adjust what matters and see how it affects suggestions.

If someone says location matters less than values, the product should let them express that.

This is where explainability becomes interactive rather than just descriptive.

Safety, moderation, and abuse controls

Any social product needs abuse prevention from day one.

Matchmaking is no exception.

I separate safety controls into three layers:

content safety checks for onboarding and messaging
behavioural risk signals, such as spam patterns or repeated boundary violations
human escalation paths for ambiguous or high-severity cases

AI can help classify risk, but I would avoid purely automated irreversible actions for edge cases.

Human-in-the-loop review is still important when context is nuanced.

Systems should optimize for user safety and fairness, not only engagement metrics.

A matchmaking product that maximizes engagement at the expense of user trust is building the wrong thing.

Privacy and data minimization

Matchmaking data is highly sensitive.

People may share relationship history, personal preferences, boundaries, family goals, lifestyle constraints, and location information.

The safest data is often data you do not collect or do not retain long-term.

Some practices I think are important:

separate personally identifying data from preference features
store only features needed for matching and explanations
support user-initiated deletion with clear propagation
avoid exposing full transcript history in operational dashboards
redact sensitive fields in logs
use clear retention policies
make it obvious what is used for matching and why

These are not just compliance tasks.

They are product trust features.

If users do not trust how their data is handled, they will not trust the matches either.

Operational architecture

On the engineering side, a clean architecture for this domain might include:

a profile service for structured attributes
a transcript processing pipeline for enrichment jobs
a scoring service with deterministic logic and versioned weights
an explanation service that maps score components to readable output
an audit layer for traceability and debugging
a moderation layer for safety review and escalation

I prefer versioning scoring configurations so you can compare match outcomes over time and roll forward or back safely.

This also makes experimentation cleaner because you can tie user cohorts to explicit scoring versions.

If a change makes matches feel worse, you need to know exactly which scoring version caused it.

Without versioning, it becomes hard to separate model behaviour, scoring logic, and product changes.

How this maps to Needle-style products

In products like Needle-style matchmaking experiences, conversational onboarding and explainable compatibility are especially important because users are often evaluating whether the product understands them at all.

The first few suggestions become a credibility test.

A practical flow might look like this:

Voice or chat onboarding captures intent and constraints
LLM extraction creates structured profile candidates
User confirms key fields before matching
A deterministic scorer ranks candidates
The UI explains each match with clear factor breakdowns
Users adjust preferences over time
The system learns from confirmed changes rather than guessing silently

The point is not to make the product feel like a spreadsheet.

The point is to make the AI feel accountable.

If the system recommends someone, the user should be able to understand the main reason.

If the recommendation is weak, the system should be honest about that too.

For the broader product context, I have a related project page here:

Needle AI matchmaking case study — coming soon

What to measure

Explainable systems still need measurable outcomes.

Some metrics I would track:

onboarding completion and drop-off by step
profile completeness
conflict resolution rate
match acceptance rate after explanation shown
user edits to preference weights over time
safety intervention rates
review turnaround time
number of matches rejected because of unclear reasoning
number of matches accepted after viewing explanation details

If acceptance rises after users view explanations, you are probably on the right path.

If users frequently override the same preference dimension, your weighting model likely needs adjustment.

If users do not trust the explanation, the explanation is not doing its job.

Closing thoughts

Explainable AI matchmaking is not about reducing everything to a score.

It is about giving users understandable, adjustable, and trustworthy outcomes.

LLMs are useful for handling unstructured input. They can turn messy conversations into structured candidate features. But deterministic logic remains powerful where product trust and accountability matter most.

From an engineering perspective, the best systems are the ones product, design, and backend teams can reason about together.

If you can explain your architecture to both an engineer and a user in plain language, you are probably building the right thing.

Original Post

Building Real-Time Voice AI Applications with LiveKit and FastAPI

Joshua Fields — Tue, 23 Jun 2026 10:40:54 +0000

Real-time voice AI demos are easy to make look impressive.

Production voice systems are a different problem.

The demo usually has one clean interaction, one happy path, and no load. Production has packet jitter, user interruptions, reconnects, flaky speech recognition, delayed TTS, provider failures, and all the little timing issues that make a conversation feel robotic if you do not design for them upfront.

This is how I think about building real-time voice AI applications with LiveKit and FastAPI in a way that can actually ship.

It is less about one framework trick and more about architecture decisions: where state lives, where latency accumulates, how retries work, and what to observe before users tell you something feels off.

Reference architecture at a glance

A practical voice AI stack has a few clear layers:

Client: browser or mobile app that captures mic input, streams audio, and plays synthesized responses
Voice session layer: session identity, auth tokens, connection lifecycle, and per-user context
LiveKit room: low-latency media transport and participant coordination
STT pipeline: speech-to-text with partial and final transcripts
LLM orchestration: prompt construction, tool calls, memory policy, and response shaping
TTS pipeline: text-to-speech chunks streamed back to the user
Backend APIs: FastAPI services handling session state, business actions, and persistence
Observability: metrics, traces, logs, and replay signals for latency and failure analysis

I try to keep each layer independently testable.

If the orchestration logic can only run when a full audio room is active, debugging becomes painful fast.

Client and session boundaries

The client should do as little product logic as possible.

It captures audio, handles UI state, and relays events. It should not decide authorization or business outcomes.

For every voice session, I prefer generating a short-lived token on the backend, scoped to a single room and participant role. That keeps room access controlled and avoids broad credentials leaking into the frontend.

I also keep a server-side session record keyed by a stable session ID.

That record can track:

user ID, if authenticated
room ID
started timestamp
current mode
latest orchestration state
current turn state
reconnect status

When a user reconnects, the backend can recover context without guessing from client memory.

That matters because voice sessions are not just request-response flows. They are live interactions with timing, interruptions, and state.

LiveKit room design decisions

LiveKit gives you the real-time substrate, but you still need conventions.

For most assistant-style experiences, I prefer one room per active user session. It keeps event scopes clear and reduces accidental cross-talk.

If the use case requires multiple participants, such as a user, an AI agent, and a supervisor, define explicit participant metadata and role handling early.

Two patterns help a lot:

Use data channels for control events like interrupt, mute, handoff, and system status
Treat room events as first-class telemetry, not just infrastructure logs

Join, leave, track publish, track unpublish, reconnect, and bitrate drops are all product signals.

That event data becomes crucial when someone says:

The assistant started speaking over me.

At that point, you need to know whether the issue was model latency, voice activity detection timing, TTS cancellation, or a reconnect edge case.

STT: partials, finals, and confidence handling

Speech-to-text should emit partial transcripts quickly for responsiveness.

But downstream business logic should usually wait for final segments or confidence thresholds.

If you run every partial transcript through your orchestration loop, you create race conditions and noisy model calls.

I usually think about transcript events in explicit states:

partial: render to UI, but do not commit to durable context
final: append to conversation history and trigger orchestration
revised final: patch a previous segment if the provider corrects recognition

This makes transcript behaviour deterministic and easier to test.

It also avoids subtle bugs where the assistant answers a phrase the user never actually said in the final transcript.

LLM orchestration in FastAPI

For orchestration, I usually expose a FastAPI endpoint or event handler that receives normalized transcript events and returns structured actions rather than raw prose.

The action envelope might include:

assistant text
tool calls
state updates
UI directives
follow-up prompts
confirmation requests
escalation signals

When teams skip this structure, the orchestration layer turns into prompt glue and ad-hoc branching.

I prefer strict schemas, even when the model itself is flexible.

With schema-first orchestration, you can validate outputs, reject malformed actions, and retry safely without duplicating side effects.

This is where FastAPI works well for me: clear request models, async handling, and a straightforward way to compose tool integrations while keeping the contract explicit.

Latency budgets and interruption behaviour

Voice UX is mostly latency UX.

If a response arrives late, users interrupt.

If interruption handling is weak, trust drops quickly.

I like setting a practical latency budget per turn and breaking it down by stage:

STT latency
orchestration latency
tool call latency
TTS startup time
time to first audio byte
total turn completion time

Once you have per-stage timings, optimization decisions become much clearer.

Interruption should be supported as a first-class control flow:

client sends interrupt event immediately
current TTS stream is cancelled
orchestrator marks the prior response as interrupted
next turn continues from a clean state

Without explicit cancellation semantics, the system often continues generating text in the background and then leaks stale context into the next turn.

That is one of the fastest ways to make a voice assistant feel broken.

Retries and idempotency

Retries are inevitable with external STT, LLM, and TTS providers.

The critical point is making retries safe.

I like attaching idempotency keys to orchestration turns and tool executions, then persisting turn state transitions:

received
processing
completed
failed
cancelled

If a timeout occurs and the client retries, the backend can return an existing result or resume from a known step instead of replaying side effects.

This matters a lot when tool calls trigger real actions, such as booking, messaging, account changes, database writes, or payment-related workflows.

A voice system should feel conversational, but the backend still needs to behave like a reliable distributed system.

Observability that actually helps

For voice systems, aggregate uptime is not enough.

I care about metrics that explain the actual user experience:

end-to-end turn latency percentiles
time to first transcript token
time to final transcript
time to first audio byte from TTS
interrupt rate per session
provider error codes by stage
reconnect frequency
average recovery time
cancelled turn count
failed orchestration count

I also like structured logs keyed by session ID and turn ID, so a problematic interaction can be reconstructed quickly.

You do not always need to store raw audio to debug effectively. In many cases, replaying timeline events is enough, and it is much safer from a privacy perspective.

Deployment and scaling notes

On deployment, I treat the voice path as a latency-sensitive service tier.

Keep orchestration workers close to your media region where possible, and avoid unnecessary synchronous hops.

FastAPI services running behind autoscaling can work well, but cold starts and noisy neighbours still matter for interactive voice.

I usually separate concerns into at least two deployables:

API/session control
orchestration workers

That gives you flexibility to scale orchestration independently when usage spikes.

If you rely on Kubernetes, readiness checks should validate downstream dependency health rather than only process liveness.

For a normal API, a process being alive may be enough to accept traffic. For real-time voice, a process that cannot reach STT, TTS, or the orchestration layer is not actually ready.

Security and privacy baseline

Voice products can capture sensitive data by accident.

Even if your first release is small, it is worth setting conservative boundaries early:

short retention for raw transcripts unless explicitly needed
redaction for known sensitive fields in logs
scoped credentials for room tokens and provider APIs
clear user controls for muting
clear consent flows
deletion controls
limited access to session logs
separation between debugging metadata and sensitive content

These boundaries are much easier to create early than retrofit later.

Closing thoughts

Building real-time voice AI applications is not only an LLM problem.

It is a systems problem spanning networking, state management, retries, observability, security, and product interaction design.

LiveKit and FastAPI make an effective foundation, but the quality comes from how you define boundaries and failure behaviour.

For me, the winning pattern is simple:

predictable contracts
explicit state
tight latency feedback loops
safe retries
clear cancellation semantics
observability from day one

That is what keeps the experience feeling conversational while still operating like production software.

Original Post