DEV Community: Nabeel Hassan

My Login Link and My Unsubscribe Link Are the Same Kind of String

Nabeel Hassan — Thu, 17 Sep 2026 21:58:21 +0000

Upwork Scout has no password field. It never had one. When I was sketching the signup flow I told myself I was removing work: no hashing, no reset flow, no rules about symbols and capital letters, no breach surface built out of passwords people reused from somewhere else.

That held for about a week. Then I looked at what I had actually done. I had not removed the work. I had moved all of it into one file of about fifty lines that signs three completely different things with the same secret.

This is what passwordless auth looks like once it is actually running, using the job alert tool I run at upwork-scout.com as the example. Small Next.js app, Firestore behind it, and the entire auth layer is lib/auth.ts.

Three links that are the same object

A user gets three kinds of credential from me over a normal month:

A login link, emailed on request, which should work for about fifteen minutes.
A session, which is not a link but a cookie, and should last a month.
An unsubscribe link, which sits in the footer of every alert email and needs to work roughly forever, because forever is how long an old email stays in an inbox.

All three are signed JWTs. All three come out of the same secret. This is the whole signing side:

type Purpose = "magic" | "session" | "unsub";

export async function signToken(uid: string, purpose: Purpose, expiresIn: string) {
  return new SignJWT({ uid, purpose })
    .setProtectedHeader({ alg: "HS256" })
    .setIssuedAt()
    .setExpirationTime(expiresIn)
    .sign(SECRET());
}

Two claims of my own: a user id and a word. That word is carrying more weight than anything else in the application.

The line that keeps them apart

Verification takes the purpose it expects as an argument and refuses everything else:

export async function verifyToken(token: string, purpose: Purpose) {
  try {
    const { payload } = await jwtVerify(token, SECRET());
    if (payload.purpose !== purpose || typeof payload.uid !== "string") return null;
    return payload.uid;
  } catch {
    return null;
  }
}

Delete payload.purpose !== purpose and nothing appears to break. The build is green. Login works. Unsubscribe works. The app is also completely broken, because every alert email I have ever sent contains a valid, year-long, uid-bearing token in a query string, and the session reader would now happily accept one.

That is the part that genuinely surprised me about going passwordless. The risk is not that a token might leak. I am putting tokens into email footers deliberately, thousands of times, as the normal operation of the product. The only reason a footer link is not a login is one string comparison. A password-based app has no sentence like that in it anywhere.

So the rule I came away with: a verifier never accepts a token, it accepts a token for something. Purpose is not metadata attached to the credential. Purpose is the type of the credential, and the check belongs in the one function nobody can route around.

The lifetimes are the design

Once purpose is a type, each one gets its own expiry, and the three numbers are not close to each other:

signToken(uid, "magic",   "15m");  // emailed on request
signToken(uid, "session", "30d");  // httpOnly cookie
signToken(uid, "unsub",   "365d"); // footer of every alert

Fifteen minutes on the magic link is not really about attackers. It is about inboxes being shared, forwarded, synced to an old laptop, and left open. A login link is the one credential in the system that travels through infrastructure I do not control, so it gets the shortest life I can give it without making a slow email useless.

The 365 day one is where I argued with myself the longest. A year is an uncomfortable time for a signed credential to sit somewhere I cannot reach. But think about what a short unsubscribe token actually does: a person who wants my email to stop clicks the link, sees "this link is invalid or expired", and reports the message as spam instead. That is worse for everyone, including the people who do want the alerts. The unsubscribe token gets the longest life of anything in the system because the failure mode of a short one is a deliverability problem, and deliverability is the product.

The user id is not a secret, and I stopped pretending

/** Deterministic user id from email (stable, no lookup needed). */
export function uidForEmail(email: string): string {
  return Buffer.from(email.trim().toLowerCase()).toString("base64url").slice(0, 60);
}

That is the entire user id. It is the email address wearing a hat. It is deterministic, it needs no round trip, and the document key falls out of the signup form before I have touched the database.

I still like it, and I want to be honest about the cost. Anyone holding a uid can decode an email out of it, because base64url is an encoding and not a hash. So the uid can never be the authorization. It is a name, not a key. Everything that matters reads the uid out of a signed token and then confirms the document exists:

const uid = await verifyToken(token, "magic");
if (!uid) return NextResponse.redirect(`${base}/?error=expired`);

const snap = await usersCol().doc(uid).get();
if (!snap.exists) return NextResponse.redirect(`${base}/?error=expired`);

Guessing a uid buys nothing, because you cannot sign it. What it does mean is that a raw uid never goes into a URL, a log line, or a response body that is not already gated by something else. Readable identifiers are fine right up until somebody treats one as proof.

The account exists before the person does

With no password, signup and login are the same request. There is no second step where somebody proves they meant it. So requesting a link creates the user document immediately with active: false, and only the first successful verify wakes it up:

if (!snap.get("emailVerifiedAt")) updates.emailVerifiedAt = Date.now();
if (snap.get("active") === false && !snap.get("emailVerifiedAt")) updates.active = true;

The unverified row is how I keep those two ideas apart. It can hold preferences, it can carry counters, it can be rate limited, and it cannot receive a single alert until somebody has proven they can read that inbox. If you type a stranger's address into my form, you create a dormant row and nothing else happens to them.

The rate limit lives on that same document, which I expected to feel like a hack and does not:

const last = snap.get("lastLinkRequestAt") as number | undefined;
if (last && now - last < 60_000) return NextResponse.json({ ... }, { status: 429 });

One link per minute per address. No Redis, no separate limiter, no extra service to keep alive. The thing being protected and the counter protecting it are the same read I was doing anyway.

The part that actually broke was not the auth

Here is what nobody mentions when they recommend deleting the password field. Your mail provider is now your auth provider. Not as an analogy. If mail does not arrive, nobody can get in, ever, and there is no second route because there is no password to fall back to.

I learned this in the least dignified way available. Upwork Scout sends through Resend in production, with an Apify actor sitting behind it for local work. That fallback actor runs with limited permissions and will only deliver to the Apify account owner's own address. Every other recipient fails with "Users can only send an email to their own email address."

So: I tested signup with my own email, many times, and it worked perfectly every time. The auth code was correct. For anyone who was not me, request-link returned a 500 and the product effectively did not exist.

There is now a comment block at the top of the email module, in capitals, saying exactly this. It is the only bug I have shipped whose entire cause was that I only ever tested as myself.

The general version: if your login is an email, then your login has an uptime, a sender reputation, a set of DNS records, and a spam folder. A verified sending domain belongs on the launch checklist next to the database, not under "nice to have later".

The one I still owe

The unsubscribe route is a GET that flips active: false the moment it is hit. One click, no confirmation, which is exactly right for a person who wants out and exactly wrong for the way corporate mail scanners and link previewers fetch every URL in a message before a human sees it. Somebody's security appliance can unsubscribe them from a product they were enjoying.

The fix is not hard and I have not shipped it: keep the one-click promise, but make the state change a POST, which is what the one-click unsubscribe standard the large mailbox providers ask for wants anyway. Writing this paragraph is mostly me making it harder to keep postponing.

The trade, honestly

Passwordless was still right for this product. No password table to breach, no reset flow, no support thread about lockouts, and signup is one field on a landing page.

What I traded for it is a system where the security is concentrated rather than spread out. One secret. One verifier. One string comparison standing between an email footer and a session. Concentrated is genuinely the good part, because there is very little of it and I can hold all of it in my head at once. It is also a surface with no redundancy in it. When the entire gate is fifty lines, every one of those lines earns the kind of reading you would normally save for something much bigger.

There Is No Repro for a Phone Call

Nabeel Hassan — Wed, 16 Sep 2026 07:18:27 +0000

Every other part of my stack comes with a safety net I never asked for. TypeScript refuses to compile. A test goes red. A pull request makes somebody ask why I touched that line.

Then there is the prompt.

A client sends a note saying the agent came across as a bit pushy on that last call. I open the prompt, soften one sentence, hit publish. No build. No diff. No test suite. The next caller gets the new behaviour immediately, in every branch of the conversation, including the four branches I was not thinking about while I rewrote that sentence.

That is a production deployment. It just does not look like one, and for longer than I am comfortable admitting I treated it like editing a config value.

This is the change discipline I run on agents already answering real phones. Most of it applies to any LLM feature you have shipped to users, because the missing pieces are the same.

A prompt is global state written in prose

There is no scoping in a prompt. A sentence you add to soften the booking flow is also read during a refusal, during a transfer, and during the part where a caller is slowly spelling out a registration number. You cannot mark it private. Everything sees everything, always.

Two regressions that genuinely happened to me on live agents:

I added a line to make an agent more forthcoming about pricing. It became more forthcoming in general, and started answering an adjacent question it was specifically supposed to decline. Nothing in my change mentioned that topic.

I tightened the confirmation step to cut mistakes on email addresses. The agent became slower and more repetitive on every other capture in the call, which surfaced as callers hanging up mid-booking rather than as an error anywhere.

Neither was visible in the text I changed. Both were visible in behaviour, days later, in numbers nobody was watching closely. The calls completed. The transcripts read fine. Nothing reported a problem.

Four safety nets that are simply not there

There is no deterministic replay. I cannot re-run last Tuesday's call against the new prompt. The audio, the timing, the interruptions and the model's own sampling are gone. Coming from years of real-time work in Unity, this was the hardest adjustment by a distance. Games hand you a repro. Calls hand you a story.

The output is a conversation, not a value. Assert on exact wording and your test is worse than useless, because the wording is supposed to vary.

Production config is edited live in a browser. Most platforms let anyone with a login rewrite the agent in a text box and publish on the spot. There is no review step, and the change history is whatever the vendor felt like keeping.

Part of the stack updates without you. The model version, the voice, the recognition behaviour. You ship nothing and the agent changes anyway. More on that at the end.

None of that discipline comes from the platform, so it has to come from outside.

Build a small regression set out of real calls

The most useful artifact I keep for any live agent is a frozen list of twelve to twenty call scenarios, pulled from real traffic rather than imagined at a desk. It covers:

the two or three highest-volume intents, in their plain form
every refusal the agent is supposed to make, one scenario each
one messy capture: an awkward email, a surname heard three different ways, a postcode
one caller who interrupts, and one who changes their mind mid-booking
one call that must end in a transfer, and one where the transfer destination does not answer
one out-of-scope question that should end in an honest "I do not know, let me get someone"

It is not exhaustive coverage. It is the set where a regression is most likely to be both plausible and expensive.

Assert on outcomes, never on phrasing

For each scenario I write down, in a line or two:

which exit the call must reach (booked, message taken, transferred, refused)
which tools must be called, and with what arguments
what must exist afterwards: the calendar event, the CRM record, the fields captured correctly
what must never appear: a price quoted, a diagnosis offered, a claim of being human

People forget that last category, and it is the one that gets you in trouble. Pass or fail per assertion, not an overall impression of whether the call felt good. "Felt good" is not a test result.

Automate the structure, listen to the rest

I split every run in two, because the halves catch different regressions.

The automated half drives the agent through the scenarios, captures the transcripts and the resulting records, and checks the assertions. It reliably catches structural regressions: a tool that stopped being called, a field that stopped being captured, a refusal that stopped happening, an exit that now lands somewhere else.

The human half is a handful of calls listened to on a real phone line. It catches what a transcript cannot: pace, a business name mispronounced after a voice update, a confirmation read out as a wall of digits, an agent that now sounds curt. In every one of those the transcript contains the correct string, which is exactly why it cannot be the only thing you look at.

Two tiers keeps it affordable. A wording fix inside one knowledge base answer gets a three-scenario smoke run. Anything touching the prompt, the flow, the tools or the model gets the full set.

Make the change reversible before you make it

Config lives in version control, not only in the platform. I export the prompt, the flow, the tool definitions and the automation workflows into the repo and commit them with the reason in the message. The platform's history is a convenience. The repo is the audit trail, and it is what tells you six weeks later why that odd sentence is in there.

One person publishes. Editing is fine. Publishing is not a group activity. Nearly every unexplained "the agent got worse this week" I have investigated turned out to be two people making individually reasonable edits two days apart.

Rollback is one action, and it has been rehearsed once. Restoring the previous version should take under a minute while the phone is ringing. That habit came out of shipping mobile releases at RAQTS through App Store and Play Console review, where the only real protection against a bad build is how fast you can pull it.

And ship one change at a time. Batch six edits into one publish and when the transfer rate doubles you have six suspects and no way to attribute.

Ship into the quietest traffic you have

There is no staging environment with real callers, so the closest thing is a smaller slice of real traffic. Publish at the start of a quiet window instead of nine on a Monday, or put the change on one agent when a client runs several, or expose it first on an overflow line. Then I read the first fifteen or twenty live calls myself. Not a random sample next week. The first ones, immediately.

After that the signal is a change in shape, not an error. Transfer rate up. Capture accuracy on one field down. Average call length up by twenty seconds. Resolution rate down while volume is flat. Those are the fingerprints of a regression, and not one of them throws an exception.

The change you did not make

This is the part clients find surprising, and the reason the set is worth keeping rather than running once.

The model version changes. The voice provider updates a voice and the business name is suddenly pronounced differently. Recognition behaviour shifts. A client's staff member edits the knowledge base, entirely correctly, and creates a contradiction with a line in the prompt. A CRM renames a field. Any of these can alter agent behaviour with no deployment on your side at all.

So pin versions wherever the platform lets you, treat any provider change notice as a trigger to re-run the full set, and keep a dated note of when the set last passed. When a client says the agent has got worse and you can answer that it passed everything eleven days ago, and here is precisely what changed since, the conversation becomes an investigation instead of an argument.

The short version

Keep the set small and real. Assert on outcomes, not wording. Version the config outside the platform, publish through one person, rehearse the rollback. One change at a time, into quiet traffic.

None of this is sophisticated. It is the release discipline any production system gets, applied to a system whose test failures arrive as a polite customer who did not get a call back.

I write more about building production voice agents and the telephony and automation layer behind them at nabeelbaghoor.com/blog.

Build the Sync Spine Before You Build a Single Screen

Nabeel Hassan — Mon, 14 Sep 2026 23:53:10 +0000

The first time a client asked me for "offline mode", it arrived three weeks before launch, in the same tone as "can we also have a dark theme". I said yes, because storing data on a device is not hard. Then I sat down with the actual question and realized I had agreed to rebuild the data model.

Offline is not a feature you bolt on. It is a decision about where the truth lives, and it is far cheaper to make in week one than in week twenty.

There are three offline apps, and they share one name

One word covers three very different builds. Naming which one you are actually being asked for is the most useful thing you can do in the first hour.

Level 1: it does not fall over. The app still needs the network to do anything useful, but it behaves like adult software without it. Cached screens render, a half-finished form survives, and it recovers on its own when signal comes back. This is a few days of care, not an architecture. A surprising share of "we need offline mode" requests are fully satisfied here, and finding that out early is worth more than any library you could pick.

Level 2: capture now, send later. The user creates work while disconnected: photos, readings, notes, a signature, a completed job sheet. It queues locally and uploads when there is signal. Data flows one way, which is exactly what keeps it affordable. Nothing on the server contradicts the queue, so there is nothing to reconcile.

Most field software wants this and only this. The same split runs through my own app, LectureNotes AI: recording the lecture must never depend on connectivity, while transcribing and summarizing it is heavy work that can happily wait for a better connection.

Level 3: a real replica. The device holds a working copy of shared data, the user edits it offline, the server changes too, and both sides have to converge. This is where the engineering cost lives: conflict resolution, partial sync, schema migrations on devices that have not phoned home in a month, and a class of bug that only appears when two clocks and two copies disagree.

The distance between level 1 and level 3 is most of the budget. Almost nobody says which one they mean.

The question that sets the price

Out of everything you can ask, one question moves the number more than the rest:

Can two people change the same record while both are offline?

If every user only creates their own records, or each record belongs to one person for the duration of a job, you never need conflict resolution and you stay at level 2. If two disconnected devices can edit the same row, you are at level 3 and the conflict rules are part of the scope, in writing, before anyone estimates.

Four more that change the shape of the build:

How long is the worst realistic outage? Ninety seconds in a lift is a retry problem. A full shift underground is an architecture. A site visited once a week is a different product again. Ask for the worst case, not the average: that number decides how much data lives on the device.

How much data has to be there? The whole database rarely fits and rarely should. The working set is usually narrow: this week, this region, these accounts, these assets. Defining that subset is a product conversation, not a technical one.

What happens if the device is lost? An offline app is a copy of company data in a coat pocket. Encryption at rest, a wipe path, a retention rule for the local copy, and a clear answer about what a stolen phone exposes. For regulated data this can decide the whole design, so it belongs in week one, not in the security review before launch.

What must be true before the user walks away from signal? Offline apps need a warm-up: data pulled, assets cached, queue empty. A user who leaves with a stale copy carries an app that is confidently wrong all day, so the ones that work in the field show when the device last synchronized and whether it is safe to go dark.

Conflicts get skipped, then eat the project

If two copies of a record can change independently, you need a rule. There is no clever default. Pick per record type, not once for the whole app.

Last write wins. Cheap, simple, and it silently destroys work. Fine for a status flag. Not fine for anything a person spent ten minutes typing.
Ownership per field. The office owns the customer address, the field owns the job outcome. Both edits survive because they never touch the same field. It resolves far more real cases than people expect, and costs a conversation rather than a merge engine.
Append instead of overwrite. Record events rather than states: readings taken, parts used, notes added. Two devices appending to a list never conflict, current state is derived, and you get an audit trail for free.
Send it to a human. A small review queue for the collisions your rules cannot settle, with a named person who actually works it. A system that claims not to need one is hiding collisions, not resolving them.

One rule sits above the other four: never silently discard something a user made. If the app cannot merge it, it keeps it and tells somebody. A dropped form is not a sync bug to your user, it is a reason to go back to paper.

What it changes everywhere else

IDs get created on the device. A record exists before the server has ever seen it, so it cannot wait for a server-assigned identifier. Decide this early or retrofit it painfully, because it leaks into every foreign key and every log line.

You stop trusting the clock. Device clocks drift, get set by hand and cross time zones. Ordering events by device timestamp is a bug with a delayed fuse.

Old versions stay in the wild. An app that sat in a drawer for six weeks comes back and syncs against a schema that has moved on. Versioned payloads and a server that can still read an old client are part of the job, so even a greenfield offline build inherits the discipline of legacy maintenance.

Airplane mode is not a test. Toggling the radio off gives you a clean binary that real users almost never experience. What belongs on the test plan is flapping connections, half-completed uploads, credentials that expire mid-sync, a device syncing at 2% battery, and two devices editing the same record on purpose.

When offline-first is the wrong call

When the data must be authoritative at the moment of the action. Payments, stock allocation, anything where two people committing the same thing offline creates a real world problem. On Fyuel, the accounting platform we build for the fuel trade, the ledger is the authority by design, and two disconnected devices both writing to it would be a defect rather than a feature.

When the app is inherently live. Shared real-time collaboration needs a server in the loop anyway. Build for reconnection quality instead of independence.

When users are never actually offline. Desk software with a nervous stakeholder needs level 1 and an honest conversation.

How I scope one now

Watch the work happen. Where people stand, for how long, and what they do with paper when the app fails them.
Name the level per feature, not for the whole app. Most apps are level 2 with one or two areas at level 3.
Write down the working set: what lands on the device, keyed to what, and how it refreshes.
Decide the conflict rule per record type, including which ones go to a human.
Build the sync spine first. One record type moving all the way through, tested against bad networks, before any screens exist.
Ship it into a real dead zone. A depot basement finds in a day what a simulator will not find in a month.

Step 5 is the one people fight me on, because a sync spine demos badly and a screen demos well. It is still the right order. Every offline project I have watched go sideways went sideways in the sync layer, and by then it had screens bolted to it.

Offline-first is not about surviving a rare outage. It is about building for where the work actually happens. Covoisino, a ride-sharing app for hitchhikers we built to make hitchhiking safer, meets its users at motorway junctions rather than at desks, and its verification step is a QR check between two people already standing together. Design from that reality and the software fits the job. Design from the office and your users keep a clipboard in the van.

Originally published on the Null Studio blog.

The Reminder Queue Refills Only When You Open the App

Nabeel Hassan — Sun, 13 Sep 2026 21:44:49 +0000

The bug report was one line from someone testing Lifemaxxing AI: "why did it tell me I got nothing done today, I did everything."

He was right, and the notification was working exactly as written.

Lifemaxxing AI is a habit app. You answer some onboarding questions, it assigns you tasks, and then it nags you. The nagging is the product. If the reminders are wrong, the app is a to-do list with a leaderboard bolted on.

So I went looking for why a user who completed every task still got the 7pm guilt trip. I found something worse than a bad conditional. I found that the whole reminder system depends on the user already having come back.

Two notifications is the entire queue

Here is the scheduler, trimmed:

Future<void> scheduleNotifications(
    Map<String, List<bool>> completionHistory,
    [UserTaskProvider? taskProvider]) async {
  await init();
  await cancelAllNotifications();

  await _scheduleMorningNotification(taskProvider);

  final bool tasksCompleted = await _wereTasksCompletedToday(completionHistory);
  if (!tasksCompleted) {
    await _scheduleAfternoonNotification(taskProvider);
  }
}

Cancel everything. Schedule one morning notification for tomorrow. Schedule one afternoon notification for today, or tomorrow if that time has already passed. Stop.

Both go out through zonedSchedule with no matchDateTimeComponents, so neither one repeats. They are two one-shot appointments, not a recurring rule.

Which means the pending queue is at most two items deep, and the only thing that ever refills it is scheduleNotifications running again. Every call site needs the app to be alive: cold start in main.dart, the main screen's init, the task event bus when a task is completed, and the permission grant right after onboarding.

So a user opens the app on Monday, gets Tuesday morning's reminder scheduled, ignores it, and by Tuesday evening there is nothing pending at all. The app goes quiet at exactly the moment the user is drifting, which is the only moment the reminder had a job to do.

I did not sit down and design a retention feature that switches itself off when retention drops. I built it one notification at a time and never stood back to ask what the queue looks like for someone who is not there.

The gate that never closes

Back to the original complaint. Here is the check:

Future<bool> _wereTasksCompletedToday(
    Map<String, List<bool>> completionHistory) async {
  final DateTime now = DateTime.now();
  final int daysSinceEpoch = now.difference(DateTime(1970, 1, 1)).inDays;

  for (final List<bool> taskHistory in completionHistory.values) {
    if (taskHistory.length > daysSinceEpoch && taskHistory[daysSinceEpoch]) {
      return true;
    }
  }
  return false;
}

completionHistory maps a task id to a list of booleans, one slot per program day. The lists are created with List.filled(_totalDays, false), and _totalDays is 21. Everywhere else in the app, that list is read at _currentDayIndex, a number between 0 and 20.

This function indexes it by days since the Unix epoch. Today that number is 20710. The guard taskHistory.length > daysSinceEpoch is asking whether 21 is greater than 20710.

It never is. The function always returns false. The afternoon notification always schedules, no matter what you did. The config file sitting right next to it even documents the intent the code fails to deliver: "Only sends if you haven't completed any tasks for the day."

The arithmetic is not the part that bothers me. The part that bothers me is that the notification layer needed to know what day it was and, rather than asking the part of the app that owns that question, it invented its own calendar from scratch. Nothing crashed. Nothing logged. The guard reads as perfectly reasonable in a diff.

Zero tasks await you

The message content has the same shape of problem. The config holds 26 morning lines and 34 afternoon lines, and the morning ones carry a placeholder:

{ "title": "Rise & Grind Your X Tasks",   "subtitle": "Start your morning ritual" },
{ "title": "X Tasks, 1 You",              "subtitle": "Let's see what you're made of" },
{ "title": "You Scheduled X Wins Today",  "subtitle": "No time like right now" }

Filled in like this:

final int taskCount = taskProvider?.getTodayTasks().length ?? 0;
String title = notification['title'] ?? 'Rise & Grind';
title = title.replaceAll(' X ', ' $taskCount ');

Two things go wrong here. The provider is optional, and two of the four call paths pass nothing at all, including the one that runs immediately after onboarding when the user grants permission. taskProvider?.getTodayTasks().length ?? 0 then quietly resolves to zero, so the first notification a brand new user ever receives can read "Rise & Grind Your 0 Tasks". A fallback that produces a plausible wrong answer is worse than one that produces no answer.

The second thing: even when the provider is passed, getTodayTasks() returns today's list, and the morning notification is scheduled for tomorrow. The count is frozen at schedule time and read by a human twelve hours later.

Two functions that sound like the same question

Future<bool> areNotificationsEnabled() async {
  final prefs = await SharedPreferences.getInstance();
  return prefs.getBool(_notificationsEnabledKey) ?? false;
}

Future<bool> areSystemNotificationsEnabled() async {
  return await _notificationService.areNotificationsEnabled();
}

The first reads a boolean the app wrote about itself. The second actually asks iOS. They agree right up until the user revokes permission in system settings, and from then on the app believes notifications are on forever, because nothing ever rewrites that flag.

It gets better. NotificationPermissionHelper.checkPermissionStatus() sounds like the OS call and is in fact the preference read, and the main screen uses it to decide whether to reschedule. The settings screen is the only place in the app that asks the OS directly.

Two things I would have called harmless

Every scheduled notification gets a random ID that is written to SharedPreferences and never read back. Cancellation goes through cancelAll(), so those stored IDs are pure ceremony. Dead state that looks like a design.

And the config loader has defaults that cannot fire:

return _config['configuration']['morningTimeRange'] ?? { /* sensible defaults */ };

If the JSON failed to load, _config is an empty map, _config['configuration'] is null, and the lookup on null throws before the ?? is ever reached. The defensive default is unreachable in precisely the case it was written for. The accessor one function below it uses _config['morningNotifications'] ?? [], a top level key, and is genuinely safe. Same file, same author, same afternoon.

What I am changing

A repeating daily rule instead of hand-rolled appointments, so the queue survives a user who stops opening the app. One owner of "what day is it" that every subsystem has to call. Placeholders that fail loudly when the data is missing instead of resolving to zero. System permission treated as the only source of truth, with the local flag reduced to a cache that gets refreshed on every foreground.

But the lesson I actually want to keep is about the category, not the fixes. Nothing here crashed. Nothing here logged. Every piece of it produced output that looked right on a screen. A reminder system does not fail by throwing, it fails by being plausible, and plausible is not something you catch by reading the code. You catch it by leaving the app closed for three days and checking what is still pending.

Transcribing a Lecture and Transcribing a Caller Are Not the Same Problem

Nabeel Hassan — Sat, 12 Sep 2026 21:44:02 +0000

The first time I opened a Retell call transcript, I assumed I already knew this problem.

I had shipped LectureNotes AI before that: a note-taking app that records a lecture and turns it into a summary and a revision outline. Audio in, text out, users happy. So when voice agents became most of my work, I filed speech recognition under solved and moved on to the parts I thought were hard.

I was wrong in a way that cost me a few weeks. Transcribing a fifty minute lecture and transcribing a caller on a phone line share a model family and share almost nothing else. They fail differently, they are evaluated differently, and the thing that improves each one is in a completely different layer of the stack.

Here is the honest split, after shipping both.

One is a batch job, one is a turn

Long-form transcription is a batch problem. A student hits record, sits through an hour, closes the laptop. The audio arrives as one file. Nobody is waiting on a socket for the next token. If a pass comes out badly you can run it again with different settings and nobody knows. You can spend three minutes of compute on an hour of audio and the user experiences that as "it was ready when I looked".

A voice agent inverts every one of those properties. Audio arrives as a stream, the useful unit is a three to eight second turn, and the transcript is consumed immediately by a model that is about to say something out loud to a human being. There is no second pass and no running it again with a bigger model. The caller is on the line, and every millisecond you spend is silence they are listening to.

Once you frame it that way, the question stops being which STT provider is best and starts being which failure you can afford.

In batch, the capture path beats the model

The single biggest accuracy win on the note-taking side had nothing to do with the transcription model. It was recording conditions.

A lecture recording is a phone lying on a desk six rows back, in a room with hard walls and an air conditioner. The lecturer walks around. Against that, the gap between two good STT models is noise.

What actually helped was unglamorous: handling the recording session properly so it survives the screen locking and the app being backgrounded, being honest in the UI about what a bad recording is going to produce, and getting the sample rate and format right at the source instead of repairing it downstream. If you upsample rubbish you get expensive rubbish.

If you are building anything that ingests audio, that is where I would spend the first week, not on provider benchmarks.

Word error rate measures the wrong words

WER treats every word as equal. Users do not.

A lecture transcript that is broadly correct but mangles the lecturer's specific terminology is useless, because the terminology is the entire reason a student is reading it. Course jargon, people's names, formula names, abbreviations: those are exactly the tokens with the least redundancy in the surrounding sentence, so context cannot repair them. Meanwhile a transcript that quietly drops filler words and tidies false starts usually tests better than a faithful one.

So the useful evaluation was never an aggregate score. It was: did the terms survive? Feeding known domain vocabulary in as a bias list or prompt hint bought more perceived quality than any model swap I tried, because it targeted the small set of tokens carrying all the meaning.

The same shape shows up on the phone side with different tokens. Identifiers have no redundancy either: nothing around a postcode, a registration number or an email address tells the model what it should have been.

Pick the twenty terms or the five fields that actually matter to your user and measure those. An aggregate score will keep telling you things are fine while the only words anyone cares about are wrong.

What happens after the transcript is the actual product

For long-form, the transcript is not the deliverable. The summary and the outline are, and that is where the quality is genuinely won.

Three things mattered more than I expected:

Chunk on the audio, not on the character count. Cutting a transcript into fixed windows regularly slices a definition in half, and the summary then confidently loses it. Chunking around natural pauses, with overlap between chunks, produced noticeably better output for the same model and the same prompt.

Keep timestamps all the way through. If a student cannot jump from a summary line back to the moment it came from, they cannot check it, and if they cannot check it they will not trust it. Traceability is a feature, not plumbing.

Structure is a product decision, not a prompt afterthought. Students did not want prose. They wanted something scannable: headings, short lines, definitions pulled out. An excellent essay-shaped summary tested worse than a mediocre outline-shaped one.

I run the same instinct on voice work now. The transcript is raw signal. Everything the client actually cares about, the booking, the CRM record, the outcome label, is a structured extraction sitting on top of it.

On a phone line, you are transcribing through a straw

Then there is the call.

Telephony audio is narrowband. This is the part I wish someone had said to me plainly: it is not a lightly degraded version of good audio, it is audio with the top of the spectrum removed, and the frequencies it removes are the ones that distinguish consonants. That is why f and s collapse into each other, why m and n become a coin toss, and why five and nine are a permanent problem when you are taking a phone number. It is not the model failing. That information never reached the model.

On top of that you get callers in cars, on speaker, standing in a busy shop, with regional accents that a US-tuned recognition setting handles badly.

Because the audio is worse and the stakes are higher, the fix has to leave the STT layer entirely. You cannot make the line better. You can design a conversation that survives mishearing: confirm identifiers in small chunks as you take them, validate shape downstream in automation rather than hoping the prompt caught it, and build a repair ladder with an exit, so a bad capture becomes a text message or a transfer instead of a loop.

Latency is the product in exactly one of them

In batch, latency is a scheduling detail. In a call, it is the product.

Streaming STT adds cost at the point where you are most sensitive. Endpointing has to decide when the caller has actually stopped talking, and that trades directly against being interrupted or leaving dead air. Set it aggressively and the agent talks over people. Set it conservatively and the agent feels slow. It is also the one setting I tune per moment rather than globally, because a caller reading out a phone number and a caller saying "yes" want opposite values.

Nobody has ever complained that a lecture summary arrived four hundred milliseconds late.

One fails loudly and one fails silently

This is the difference I underrate least now.

A bad lecture transcript is obvious. The user reads it, sees the mess, re-records or complains. The feedback loop is short and self-correcting.

A bad phone capture is silent. The call sounds fine. The caller is polite. The agent confirms something plausible, the automation writes a record, and the failure surfaces days later when an email bounces or someone does not turn up to an appointment. Nobody watched it happen.

That is why I score voice agents per field rather than per call, and why I store the raw transcript and audio next to the extracted result. When a structured value is wrong, the only way to find out why is to go back to the signal.

If you are choosing between them today

A quick heuristic, since founders ask me this often:

If nobody is waiting, buy accuracy. Larger model, second pass, domain vocabulary, spend the seconds. Cost per hour of audio is your constraint, not latency.
If someone is on the line, buy predictability. Consistent, boring, low latency beats a slightly better transcript, because the caller feels the pause and never sees the transcript.
Store the raw signal. Transcript, audio, timestamps. Every interesting debugging session I have had in either product started by going back to the recording.

The uncomfortable conclusion after shipping both is that the model was rarely the interesting variable. Capture conditions, chunking, structured extraction and honest evaluation moved both products more than any provider comparison did, and all four are things you own rather than things you buy.

I wrote a longer version of this on my own site, with more of the voice agent stack around it: Speech-to-text for lectures vs phone calls.

A Stored Balance Is a Rumor: What Changes When the App Becomes the Books

Nabeel Hassan — Thu, 10 Sep 2026 06:19:31 +0000

Most software is allowed to be wrong for an hour. You notice, you fix it, you backfill, nobody outside the team finds out. Then one day somebody in a meeting says a number out loud, somebody else asks where it came from, and the answer is the name of your application. That is the day the rules change, and it always arrives before anyone plans for it.

I build this kind of software for a living. Fyuel, an accounting platform we built for the fuel trade, pulls customers, suppliers, tankers, ledgers, banks and reports into one real time system. The consolidation is the entire value of it, and it is also the obligation, because once you are the single place a business looks, being current and being able to defend a figure is the product.

This post is not about accounting. It is about the handful of engineering decisions that quietly change the moment your app becomes the record, every one of which is cheap on day one and painful to retrofit in year two.

You may already be building financial software

The label on the product does not decide this. The test is simpler: if a wrong value on one of your screens would cause somebody to make a payment, chase a customer, file a return or close a period incorrectly, you are building financial software regardless of what the repo is called.

Plenty of products cross that line without anyone noticing the day it happened. Inventory systems that carry value. Billing engines. Commission calculators. Usage meters that feed invoices. Anything that replaced a spreadsheet somebody used to invoice from. The features shipped, the line got crossed, and the reliability bar moved without a ticket.

The balance column is a rumor

The most expensive shortcut in this kind of system is a balance stored in a field and updated in place. It is efficient, it is what every ORM tutorial nudges you toward, and it is what I reached for too before I worked on something where the number was the product.

Here is the problem. The moment two things happen at once, or a job runs twice, or someone corrects an entry from six weeks ago, you have a figure nobody can explain and no way to find out when it went wrong, because the previous value is gone. The column tells you what it currently believes. It cannot tell you why, and it cannot tell you when it stopped being right.

The alternative is old and unglamorous: record the events, derive the totals. A payment received. An invoice issued. A load delivered. An adjustment applied. Each one is a fact with a timestamp and an author, written once and never modified. The balance is calculated from those facts, either on demand or into a cached figure you can throw away and rebuild at any time. The rebuild is the part that matters. If a total is disputed, you can show the entries that produced it, which is the difference between having an answer and having an opinion.

The rule that follows is the one clients push back on hardest, and it is the one worth holding: you do not edit history, you correct it. A wrong entry gets fixed by a reversing entry and a new correct one, both visible, both attributed. That is not bureaucratic caution. It is the only thing that makes last month's report still say what it said last month. A system where somebody can quietly change a figure from six weeks ago is a system where no report can be trusted twice, including the ones that are right.

Reconciliation is a screen, not a support ticket

The numbers inside your software will disagree with the numbers outside it. Routinely. That is normal, not a bug.

The bank statement lands with a fee nobody told you about. The supplier invoice covers a delivery recorded on a different day. The stock count comes back short. If your design assumes agreement, every one of those becomes a support ticket that ends with an engineer running a query against production.

So build for disagreement. Import the outside record, match the obvious items automatically, queue the rest, and let someone explain a difference and write the correcting entry from that same screen. The person doing this work is almost never an engineer, which is exactly the point. Reconciliation is where the business proves to itself that your system is telling the truth, so it deserves a designed screen rather than a report that ends with somebody exporting to a spreadsheet and finishing the job by hand.

Money, units and dates

Three ordinary looking types cause most of the defects I have seen in this category, and all three are nearly free to get right at the start.

Money is not a number. It is an amount plus a currency, stored in the smallest unit, handled with a type that does not drift. Floating point arithmetic on currency is the classic origin of the penny that never balances, and it never announces itself, it just accumulates. Rounding needs a stated rule and place too, because rounding each line and rounding the total give different answers. Both are defensible, so pick one and apply it everywhere, including when something is credited back.

Quantities carry units, and conversion is a business rule. Anyone building for a physical trade meets this in week one. Fuel is measured, moved and billed in units that convert, and the conversion factor is business data with a history, not a constant somebody typed into a formula. Same for weights, hours, packs, cases. When the unit exists only in the column heading, the mistake is not avoided, it is scheduled.

Dates are at least three dates. When the thing happened, when it was recorded, and which accounting period it belongs to are three separate facts, and collapsing them into one created_at is what makes month end miserable later. Keep them distinct and the two features people always ask for late become almost free: closing a period so nothing can be booked into it afterwards, and reporting on what the books said at a point in time rather than only what they say now.

None of this is exotic. It is just work that never shows up as a feature, which is a large part of why a system of record costs more than a product with the same screen count.

Permissions are domain logic, and the audit trail is a feature

In financial software, permissions are not a settings page bolted on before launch. They are part of the model, and the questions are concrete. Who can create an entry. Who can approve one. Can those be the same person above a threshold. Who can void or reverse. Who can reopen a closed period, and does that leave a mark. Who can see margin and supplier pricing, given that plenty of people need the operational screens without the commercial ones. Deciding that before the build is much easier than discovering during rollout that the office shares one login.

The audit trail is the other half and it deserves to be a named feature with an owner, not a log file. A useful one records who did what, when, to which record, with the value before and after, and the finance lead can read it without an engineer running a query. That last clause is the whole distinction. Logs are for debugging. An audit trail is evidence you can hand to an auditor, a lender or an unhappy customer. It is also impossible to backfill, because a trail switched on in year two says nothing about year one.

Two things travel with it. Export has to be real, in a format an accountant actually opens, because your software sits inside a process that does not end at your login screen. And the data needs a restore you have performed, not a backup setting that is switched on. The application can be rebuilt from the repo. The ledger cannot be rebuilt from anything.

What I ask before writing any of it

Four questions, and the answers tell you most of what the build is really going to be.

Are totals derived from recorded events, or stored and updated in place? If stored, how does a disputed figure get explained?
How does a mistake get fixed? Listen for reversals and corrections, not edits.
What happens when the bank, the supplier or the stock count disagrees with us? There should be a screen, not an apology.
Can we close a period, and can we report on what the books said back then?

Financial software is not harder because the features are complicated. Most of the screens are boring. It is harder because being wrong carries consequences ordinary software does not, and nearly all of that cost sits in the parts a demo never shows.

I wrote a longer version of this for people commissioning the build rather than writing it, over on nullstud.io.

Five Code Paths Write the Same Subscription Row. On Purpose.

Nabeel Hassan — Sun, 06 Sep 2026 21:53:23 +0000

I run a small SaaS called VoiceDash. It gives voice AI agencies a white label portal they can hand to their own clients. Next.js on Vercel, Postgres behind Prisma, Stripe for billing.

The first version of the billing code had exactly one place that wrote subscription state: the Stripe webhook. That is what every tutorial shows you. It is also the version that locked a paying customer out of the product.

The failure

An agency owner picked a plan, paid, and Stripe redirected them back into the app. The layout that wraps every agency page looks for a subscription row that is active or trialing. There was none yet. So it did what it was written to do and redirected them to the plan picker.

They had a receipt from Stripe and a paywall from me.

The webhook was not lost. It was just later than the redirect, by a couple of seconds. And on preview deployments it never arrived at all, because the endpoint registered with Stripe points at production.

The bug was not the timing. The bug was my mental model. I was treating my database as the place where subscription state lives, and the webhook as the write that puts it there. Stripe is where subscription state lives. My tables are a cache of it.

Once I said that out loud, the question stopped being "why did my write not happen" and became "when do I refill this cache, and what do I show while it is cold".

Five writers, on purpose

Today, five different code paths write the same subscription state. I put every one of them there deliberately.

The webhook, at /api/webhooks/stripe. It handles checkout.session.completed, customer.subscription.created, .updated, .deleted, invoice.paid and invoice.payment_failed. This is the only writer that runs with no user present, so it is the one that catches renewals, cancellations and dead cards.
The checkout return page. When Stripe sends the customer back with ?session_id=..., the server component retrieves that session from the Stripe API itself and writes what it finds, before it renders anything.
A client side verify call. The same page also POSTs the session id to /api/stripe/verify-session on mount. If Stripe says the checkout is complete but the subscription is not attached to it yet, that route answers 202 with retry: true, and the browser tries again up to five times, two seconds apart.
The plan change route. When somebody upgrades or downgrades in place, /api/stripe/checkout swaps the subscription item in Stripe and writes the new plan in the same request. The comment I left in there is blunt: webhooks might not reach this deployment.
A read time reconcile. The agency layout calls syncSubscriptionFromStripe on render. It lists the customer's subscriptions, picks the most recent one that is active, trialing, past due or unpaid, and refreshes the row.

Five writers for one row sounds like the opening of a data corruption story. It is not, and the reason is a single line in the schema.

The unique constraint is the whole trick

model WorkspaceSubscription {
  id                   String   @id @default(cuid())
  workspaceId          String
  stripeSubscriptionId String   @unique
  plan                 Plan
  status               String
  currentPeriodEnd     DateTime
}

Every one of those five paths ends in the same call:

await prisma.workspaceSubscription.upsert({
  where: { stripeSubscriptionId: sub.id },
  create: { workspaceId, stripeSubscriptionId: sub.id, plan, billingCycle, status: sub.status, currentPeriodEnd },
  update: { plan, billingCycle, status: sub.status, currentPeriodEnd },
});

Same key, same shape, values that the writer just read from Stripe a moment earlier. That makes each write idempotent, and it makes last write wins the correct policy rather than a scary one, because whoever wrote last also asked Stripe last. Order does not matter. Duplicates do not matter. A webhook landing four seconds after the redirect path already wrote the row is not a conflict, it is a no-op that happens to rewrite identical values.

That is the part I would go back and tell myself. Redundant writers are cheap when the write is idempotent and keyed on the external system's identifier. They are terrifying when the write is an insert keyed on your own id. I did not need to pick the one correct writer. I needed to make writing safe, and then have as many writers as I had entry points, because every entry point is another chance to be fresh.

Do not gate access on your convenience copy

The workspace row also carries a plan column. It is a denormalized copy, and it is what the limit checks read, because counting clients against a plan should not require a join to billing.

Access is not gated on it. Access is gated on the subscription row:

subscriptions: {
  where: { status: { in: ["active", "trialing"] }, currentPeriodEnd: { gt: new Date() } },
  take: 1,
}

The plan says what you get. The subscription row says whether you get anything at all. Keeping those two questions separate is what stops a stale plan value from becoming either a free ride or a lockout.

The same reasoning quietly killed two columns. My workspace table still has a trialEndsAt and a clientSlots that defaults to 10. Nothing reads either of them. Trial days left is computed from the subscription row's currentPeriodEnd, because that is Stripe's answer rather than my copy of it. Denormalized copies of someone else's state rot the moment you stop writing them, and they rot silently, since a stale column looks exactly like a fresh one.

Throttle it, and be wrong in the right direction

Reconciling on every render means a Stripe API call on every page navigation. So the sync skips if the row was updated in the last 60 seconds. The settings page passes force: true, because that is where somebody lands right after changing something in the Stripe customer portal and expects to see it.

The failure policy matters more than the throttle. If Stripe times out or errors, the sync logs it and returns. The page renders from the database.

Sixty seconds of staleness on a plan name costs nothing. Showing a paywall to a paying customer because Stripe had a bad minute is the exact failure I already shipped once. When you cannot be both fresh and available, pick the direction where being wrong is survivable.

What I left inconsistent, deliberately

Plan limits are checked only at creation time. Creating a client counts the existing ones, compares against the plan limit and returns a 403 if you are at the cap. Nothing runs on downgrade.

So an agency on the 5 client plan that drops to the 1 client plan keeps all five dashboards. They just cannot add a sixth. I could enforce at read time and hide the extras, but that means taking a live dashboard away from someone's client, which is a way to lose a customer, not a way to upsell one.

And a real one I have not fixed: count then create is not atomic. Two simultaneous create requests on a one client plan can both pass the check. For one agency owner clicking a button in one browser tab, it has not happened. The fix is a transaction or a counter with a constraint behind it, which is a real change rather than a one line patch. I would rather write it down here than pretend the race is not in my code.

The rule I use now

When an external system owns a piece of state:

decide which of your tables are a cache of it, and say so out loud in the code
make every write idempotent and keyed on the provider's id, with a unique constraint behind it
then add a writer at every entry point, because each one is a chance to be fresh, not a chance to conflict
gate access on the record that mirrors the provider, not on your convenience copy
when the provider is unreachable, serve stale, never serve a paywall

Curious how other people handle this one. Do you reconcile at read time like this, run a scheduled job, or trust the webhook and eat the occasional support ticket?

Your Unity Pipeline Does Not Need Tests Yet

Nabeel Hassan — Wed, 02 Sep 2026 17:34:30 +0000

Every time I join a Unity project mid-flight, the CI conversation starts in the same place. Someone wants play-mode test automation. It is always framed as the responsible engineering thing to do, and it is always the fifth most useful thing available.

Because underneath that conversation, the build usually exists on exactly one laptop. Right editor version, right Android SDK, a provisioning profile, a keystore in a folder its owner would struggle to find again, and eight months of small manual fixes living only in one person's memory. Nobody logs that as a risk, because from the inside it does not feel like one. Builds come out. It works, right up until that person is on a flight, or the client wants a build on a Friday night, or they leave.

I have shipped Unity to Windows, Android, iOS and standalone headsets, built the CI/CD for a cross-platform C# SDK at Geonode, and managed App Store and Play Console releases as CTO at RAQTS. The order that has actually worked for me is: build somewhere nobody uses, make every artifact identify itself, get signing keys off laptops, automate delivery, and only then write tests. Most teams do that list backwards, and the backwards version feels rigorous while producing very little.

Unity is genuinely harder to put on CI than a web app

This is worth being specific about, because "just add CI" is advice written by people whose builds are a checkout, an install and a bundler.

The Unity editor is a licensed application, so your build agent has to activate a licence and release it again. That is not a footnote, it is a live source of flaky failures that have nothing to do with your code. The Library folder is a multi-gigabyte derived cache that takes tens of minutes to regenerate, so a cold build and a warm build are different animals with different failure modes. Asset import is not fully deterministic across editor versions, so "same commit, different Unity patch release" is a real category of bug. And underneath it all sit the platform toolchains: an Android SDK and NDK pinned to versions Unity tolerates, an Xcode version Apple has not deprecated yet, and a Windows toolchain for desktop.

At Geonode I built the Repocket C# SDK for Windows, Android and iOS and set up the Unity and Windows pipelines around it. The lesson was blunt. Three platforms compiling on one developer's machine is not three platforms working. Until every target builds on a machine nobody has touched by hand, you do not know your dependencies. You know what your laptop happens to already have, which is a different and much less useful fact.

Step one: build on a machine nobody uses

The first pipeline you write should do one thing. Check out the commit, build the player, drop the artifact somewhere with the commit hash in its name.

No tests. No distribution. No platform matrix.

The value is not quality assurance, which I think is exactly why rigorous people skip it. That pipeline is an executing, always-current description of your build requirements. Every undocumented step gets flushed out in the first week: the SDK version nobody wrote down, the plugin someone copied into Plugins by hand, the define symbol set in the editor and never committed, the API key in a local file git never saw. Each is a landmine under the project, and a red pipeline finds all of them in an afternoon. That one guarantee has been worth more to me than any test suite I wrote in the six months after.

Step two: make every build say what it is

This is the cheapest high-value item on the list and it gets skipped constantly.

Every build should carry an identity you can read without a debugger: a version, a build number that increments on its own, and the short commit hash. Put it on a debug screen, in the log, in a settings corner, anywhere readable in ten seconds. Then have CI stamp it, because a version number a human edits is a version number that is sometimes wrong.

// Generated by CI before the build. Never hand edited.
public static class BuildInfo
{
    public const string Version = "1.4.2";
    public const string Build   = "318";
    public const string Commit  = "a91c40e";
}

I learned to care about this in XR rather than in a normal app. Standalone headsets pass between hands with no install history a developer can see, so when someone in the field reports behaviour you cannot reproduce, the only question that matters is which build they were holding. Without an on-screen hash, that device could be running any of the last six.

The same rule holds on a phone. A bug report against "the TestFlight build" is a guess. A bug report against 1.4.2 (318) a91c40e is evidence.

Step three: get signing off laptops

Keystores, provisioning profiles and store API keys belong in the CI secret store, and the pipeline should be the only thing that touches them for release builds.

The obvious reason is availability. If one machine can sign an Android release, that machine is a single point of failure for shipping at all, and an Android keystore is not regenerable. Lose it and you cannot update the app under the same listing, ever.

The less obvious reason matters more to me. Once signing is a CI job, "who can publish" becomes a permission you granted rather than a fact about who has which files on which laptop. Release authority should be deliberate, and it usually is not.

Step four: delivery, not just building

An artifact in a storage bucket is not a release. The pipeline is not finished until the build reaches whoever has to look at it: TestFlight or a Play internal track, a signed installer for desktop, a direct install path for headsets.

The reason to automate this is feedback cycle time. When getting a build to a client is a fifteen minute chore, it happens once a week and every review round costs a day of calendar time. When it happens on merge, the client sees Tuesday's work on Tuesday. At RAQTS, where a Unity interactive platform, mobile apps and a web portal all had to move together, that gap decided whether a release was one coordinated thing or three things quietly drifting apart.

Step five, finally: tests

In Unity, the automated checks with the best return are narrower than people expect. Compile every target, so a change that breaks the iOS build is caught on the pull request instead of on release day. Unit test the pure C# where the actual logic lives, which is the same discipline as keeping a shared core free of platform knowledge. Both are cheap and pay every week.

Full play-mode automation is a much larger investment with much slower returns, and I would not reach for it before the four steps above exist. A team that builds reliably, identifies every artifact, signs from CI and delivers automatically is in better shape than a team with a green test suite and one laptop that knows how to build.

What I deliberately do not automate

Not every branch gets a full build. Unity builds are slow and, on hosted runners, not free. Compile checks on pull requests, full multi-platform builds on main and on tags. I keep the Library cache warm between runs, because a cold agent is a twenty minute tax on every job, and I run one scheduled clean build so cache drift never quietly becomes an undeclared dependency.

The store is a queue, not a deploy step

The part that surprises web-first teams is that publishing is not the end of the pipeline. It is a queue owned by someone else.

Review takes as long as it takes. Once a build is live you cannot hot fix it, and users update on their own schedule, so old clients keep talking to your backend long after you stopped thinking about them. Hence staged rollouts so a bad build reaches one percent instead of everyone, a backend that stays kind to older clients, and a rollback you have rehearsed rather than improvised.

That release discipline is the piece I carried furthest out of Unity. I ship changes to live AI voice agents the same way now. No store review, same underlying truth: the change is live in front of real users, so it goes out small, into the quietest traffic, with a reversal already practised.

The test to run today

Ask whether anyone other than the usual person could produce a signed, shippable build today, from a clean checkout, without asking them a question. If the answer is no, that is the work. Not coverage, not a nicer branching model.

The through line from a Unity project to an AI agent in production has been the same for me every time. The interesting engineering is rarely the part users see. It is whether you can change it safely, know exactly what you shipped, and put it back.

This is an adapted version of a longer piece on my site, where I write about shipping cross-platform products across Unity, XR, mobile and AI agents.

Your README Is a Lie Until Someone Else Runs It

Nabeel Hassan — Fri, 28 Aug 2026 23:20:42 +0000

I run a small studio, which means every project I ship ends on the same question: can the client actually keep this without me?

For a long time I assumed the answer lived in the contract. It does not. I have watched a company hold the intellectual property in the strictest legal sense and still be completely stuck, because the thing was live on a hosting account registered to somebody who had moved on.

Ownership turned out to be three separate things, and most agreements only buy one of them.

Three kinds of ownership

Legal ownership is the IP clause. Assignment on final payment, with clear boundaries around any pre-existing tooling the builder brings along and around open-source components. It is the easy layer, and it is the only one most contracts actually cover.

Custodial ownership is the repository. Not a zip of the final state. The full history, every branch, and the issue and pull-request record. History is not sentimental. It is how the next engineer finds out why a line exists before deleting it.

Operational ownership is whether you can build the software on a clean machine, deploy it, rotate a credential, and fix something at nine on a Friday night. Everything above it is theoretical until that one is true.

Engineers care about the third layer. It is also the one nobody writes into a statement of work.

The failure is almost never the code

The most common ownership failure I see has nothing to do with source files. It is that the product runs on accounts registered to the builder.

The list is longer than people expect. Cloud or hosting. Domain registrar and DNS. Apple and Google developer accounts for anything in a store. Analytics and error monitoring. Email and SMS sending. The payment processor. Every third-party API key. And for anything with a phone attached, the numbers themselves, plus the recordings and transcripts.

The rule I now enforce from day one is boring and it works: accounts are created by the client organisation, in their name, on their billing, and we get invited in. Not the other way round.

Retrofitting this is possible, but each item becomes its own small project. An app store transfer, a number port and a domain move are three different procedures with three different waiting periods, and at least one of them will need a person who has since left. I have done that cleanup. It is not technically hard, it is just slow in a way that always lands during a week when something else is on fire.

The drill that settles it

None of the above proves anything until somebody tries it. So here is the test, and it is the single most useful hour I spend on any project.

Take a person who did not build the software. Another engineer, a contractor, a friendly second opinion. Give them the repository and the access list, nothing else, and no access to the original team. Ask them to:

Get it running on a clean machine.
Deploy it to a scratch environment.
Make a trivial visible change.
Ship it.

Time-box the whole thing to a day.

What comes back is never abstract. It is always a specific list, and it is usually some version of this:

An environment variable nobody documented.
A database migration that has to be run by hand.
A key that only exists on one laptop.
A build step that quietly depends on a tool version.
A service that was configured once in a web console and captured nowhere.

I want to be honest about something here, because it would be easy to write this as advice for other people. My own projects fail this drill too. Every time I have run it on something I built, it has found at least one item. The point of the drill is not to prove the work was clean. It is that each of those findings is cheap to fix while the people who created it are still on the project, and expensive to fix later.

Which is exactly why it belongs before final payment rather than after it.

Why the README lies

The reason this keeps happening is not laziness. It is that the setup guide is written by the person whose machine is already set up.

Every implicit dependency is invisible from inside that machine. The Node version that happens to be active. The CLI that was installed a year ago for something else. The .env that was never in the repo because of course it was not. You cannot see the gap by rereading your own README, because you are the one person on earth who does not need it.

The drill is essentially a compiler for documentation. It turns "should work" into a diff.

What I hand over now

The list I actually work through, condensed:

The repository, full history, all branches, issue tracker export if it lives somewhere the client can be added to.
A README a stranger can follow. Not a product description. The exact steps from a clean machine to a running app, including versions of anything that has to be installed first.
Environments and deployment, written down. If a release is a sequence of manual clicks one person knows, that is a finding, not a document.
Secrets in a manager the client owns, with a note on what each one is for. Credentials in a chat thread are not a handover.
Infrastructure as files, rather than as the shape of somebody's console.
The data. Schema, a documented export path, and a backup that someone on their side has actually restored once. Software can be rebuilt. A ledger cannot.
Source files, not exports. Design files, and for 3D or immersive work the original assets, not the optimised output.
A dependency and licence list, flagging anything with a commercial tier or a renewal date.
A short decisions note, including the compromises. Two pages of "this looks strange because of X" saves a future team a month.
The access list itself. Every account and service, who owns it, who is admin, what it costs. This is the document that turns a pile of logins into something a business can manage.

Where lock-in is legitimate

Not every dependency is a trap, and pretending otherwise leads to worse software. You are not going to own your telephony provider, your game engine, your cloud, or your auth service. Choosing them is normal engineering. When we build SDKs that ship inside somebody else's product, depending on the platforms those SDKs run on is the job, not a failure.

The line I use is this: you do not have to own the platform, but you must own the configuration, the data and the exit path. The account is yours. The settings are documented or exportable. Your data comes out in a usable form on demand. Somebody has thought about what a move would involve, even if nobody intends to make one.

The real lock-in is knowledge

Here is the uncomfortable part. The code was never the hard thing to transfer.

A team that has held a system for two years knows things no repository contains. Which service is flaky on Mondays. Why that retry exists. Which client email triggered the weird validation rule. That is true of an internal team as much as an external one, which means "we built it in-house" is not protection. It is the same bus factor with a different org chart.

Documentation and the drill are how you convert some of that into something transferable. Not all of it. Some of it just walks out.

None of this is adversarial and none of it is about trust. I want clients portable, because a client who could leave and does not is a much better reference than a client who cannot. It is also the only version of this business I know how to run without dreading the offboarding conversation. If it helps, the longer buyer-side version of this, including the questions worth asking before signing anything, is on the Null Studio blog.

So, a genuine question, and I am curious how many people are in the same position I was: when did you last try to run one of your own projects from a clean machine?

You Can Retry a Write. You Cannot Retry an Email.

Nabeel Hassan — Thu, 27 Aug 2026 21:47:25 +0000

I run Upwork Scout, a small SaaS that watches Upwork and emails you only the jobs that fit you. The whole product is one cron endpoint. It wakes up every fifteen minutes, fetches new jobs, matches them against every active user, and sends mail.

The interesting engineering in that sentence is not the matching. It is the sending.

A scheduled job that only writes to a database is easy to get right, because you can run it twice and the second run overwrites the first with the same values. Nobody finds out. Email is different. Email is a side effect that leaves your system and lands in a human being's inbox at seven in the morning. There is no undo, there is no idempotency key I can pass to a person, and the failure mode is not a stack trace, it is someone unsubscribing.

So the real design constraint is not "make the scan correct". It is "make the scan safe to run twice, because sooner or later it will run twice".

Every reason my cron runs twice

The scheduler retries. Mine is a GitHub Actions workflow that curls the endpoint. If the response is not a 200 it fails the run, and I have re-run failed runs by hand more than once.
The caller giving up does not stop the server. A timeout on the curl side cancels nothing. The function keeps looping through users and keeps sending.
The function has a wall. maxDuration is 300 seconds. If a cycle hits it halfway through the user loop, half the users got email and half did not, and the next run has to know which half.
Anything can ping it. The endpoint is a plain GET guarded by a shared secret, deliberately, so I can point cron-job.org or n8n at it, or hit it from my terminal while debugging. Every one of those is another caller.

None of that is exotic. It is the normal life of a scheduled job. The only question is whether your users can tell.

The ledger is the answer, and the key is the whole trick

There is one collection called deliveries, with one document per user per job:

const dRef = deliveriesCol().doc(`${u.uid}_${j.id}`);
if ((await dRef.get()).exists) continue;

That is the entire duplicate suppression system, and the part doing the work is the document id. It is derived from the data, not from the run. Two different invocations, on different machines, at different times, computing the same user and the same job produce the same id. There is no sequence number to coordinate, no run id to compare, nothing to reconcile afterwards.

The same shape is now everywhere in this codebase. AI verdicts live at matches/${uid}_${jobId}, so a job is never scored twice for the same person. User documents are keyed by a base64url encoding of the lowercased email, so submitting the login form twice cannot produce two accounts.

Deterministic ids turn "did I already do this" from a query into a lookup. It is the cheapest idempotency I know, and it needs no infrastructure: no queue, no exactly-once broker, no dedupe table with a TTL. Just a naming convention that every write in the codebase obeys.

A lock that expires on its own

The ledger protects individual sends. It does not stop two full scans from grinding through the same work at once and doubling my scraping bill, so there is also a lock:

const lock = await lockRef.get();
if (lock.exists && now - (lock.get("ts") || 0) < 10 * 60_000) {
  return NextResponse.json({ skipped: "previous run in flight", ...stats });
}
await lockRef.set({ ts: now });

Two things about it I would defend in review.

First, it stores a timestamp, not a boolean. A boolean lock is a loaded gun pointed at your own product. The one time a process dies between taking the lock and releasing it, the flag stays true forever, no scan ever runs again, and nothing throws. Your monitoring stays green because the failure mode is silence. A timestamp lock heals itself in ten minutes without me being awake. It is released in a finally for the ordinary crash, and the timestamp covers the extraordinary one where finally never runs.

Second, it is not a real mutex. It is a read followed by a write, so two callers landing within the same few milliseconds can both walk through it. I know, and I left it, because the ledger behind it protects the thing users actually see. The lock is a cost optimization. The ledger is the correctness guarantee. Confusing those two is how people end up putting a distributed lock manager in front of a product with a hundred users.

The awkward part: I write the ledger before sending, except when I do not

Here is the tradeoff with no clean answer, and the place where my own codebase is deliberately inconsistent.

Instant alerts write the delivery record first, then send:

await dRef.set({ uid: u.uid, jobId: j.id, status, createdAt: now, sentAt: ... });
// ... later, after collecting this user's batch
await sendEmail({ to: u.email, ...mail });

If that send throws, the record already says sent. The job is gone for that user forever. This is at-most-once, and I picked it on purpose. An instant alert is one job out of many arriving all day, so losing one costs a user very little, while the same job landing twice costs them confidence in the product.

The daily digest does the exact opposite. It sends the mail, and only then flips the queued rows:

await sendEmail({ to: u.email, ...mail });
await Promise.all(q.docs.map((d) => d.ref.update({ status: "sent", sentAt: Date.now() })));

Crash in that gap and the digest goes out again on the next flush. That is at-least-once, and it is also on purpose. A digest is not one job, it is the entire day's value in a single email. Losing it silently is far worse than a rare duplicate.

I did not plan that asymmetry. I found it months later, reading both code paths in the same sitting, and my first instinct was to make them consistent. Then I worked out what consistency would actually cost in each direction and kept them different. At-least-once versus at-most-once is not a house style you choose once and apply everywhere. It is a per-side-effect decision, and the useful question is which direction of being wrong this particular user forgives.

One table, two delivery modes

The ledger also turned out to be the queue I never had to build. The row carries a status:

const status = u.delivery?.mode === "daily" ? "queued" : "sent";

Instant and digest are the same write with a different word in it. The 08:00 UTC digest run is a query for that user's queued rows, sorted by match score, capped at twenty, and then one email. No second table, no job queue, no worker process. When people ask why I did not reach for a proper queue, the honest answer is that a status column on a table I already needed covered the requirement, and every piece of infrastructure you skip is one you never have to operate at 3am.

What this costs me, honestly

That existence check is one read per candidate job per user, inside a loop. So my read count grows with users times matched jobs, and on the day that number gets uncomfortable it will be Firestore reads, not model calls, that show up on the bill. The fix is not clever. It is a batched multi-get before the loop instead of a get() inside it. I have not done it, because at current volume the loop is fine and I would rather ship features, and I would rather write that down than pretend the code is finished.

The ledger also grows forever. One document per user per job, never pruned, while the jobs themselves go stale after 24 hours. The entries stop being useful long before they stop existing.

The test I run on anything scheduled now

Run it twice on purpose, back to back, against production-shaped data, and check whether anyone outside the system can tell. If the second run sends nothing, charges nothing, and leaves the same state behind, you have a cron job. If you have to reason about how often it fires to know whether it is correct, you have a ritual that happens to work.

That test is why the first table I design for a background job is not the interesting one. It is the boring one that says "I already did this".

What is your default when a scheduled job touches the outside world, at-least-once or at-most-once? I am curious whether people pick per side effect the way I ended up doing, or standardize and eat the cost.

Nobody Hacked It. The Lookup Tool Just Took a Phone Number.

Nabeel Hassan — Wed, 26 Aug 2026 21:49:13 +0000

The first time a client asked me "can someone hack this thing?", I gave a bad answer. I talked about the model: guardrails, refusals, how hard it is to jailbreak a well written system prompt. That is where the conversation usually goes, and it is backwards.

An AI voice agent is not risky because it can be persuaded to say odd things. It is risky because it can do things. It reads from a CRM, writes to a calendar, sends texts, transfers calls, and fires automations that touch systems the caller will never see. Every one of those is reachable by anyone who knows the number. No login screen, no signup, no email verification, no bot check. The endpoint is a phone number and it answers on the first ring.

So here is the pass I now run before an agent takes a live call.

Write down the verbs, then read them as an attacker

Before any threat modelling, list what the agent can really do. Not the intent list, not the conversation design. The verbs.

For a typical receptionist agent I ship: look up a contact, read back an appointment, create one, cancel or reschedule one, send an SMS, transfer the call to a human, and drop a payload into n8n that touches the CRM.

Now reread it as an attacker. Every verb turns into a question with a missing subject.

Look up whose contact?
Cancel whose appointment?
Send an SMS to which number?
Transfer to what destination, chosen by whom?

Most agents I have reviewed fail right here, before anything clever happens. The lookup tool accepts a phone number as a parameter. The model fills that parameter from whatever the caller said out loud. The agent then reads somebody else's appointment back to a stranger in a warm, confident voice, and every component in the stack did exactly what it was built to do.

Nobody hacked it. The lookup tool just took a phone number. That is a schema problem, not a prompt problem.

Caller ID is a hint, not an identity

The most useful habit I picked up building agents for clinics, vet practices and estate agents is to key records on the caller's phone number. Names get transcribed several different ways for the same person, and the number is the one identifier the call already knows without asking. That is correct for record keeping and wrong for authorization, and it took me a while to hold both ideas at once. Caller ID can be spoofed. It is also routinely shared: a family phone, a practice reception line, a partner returning a call on someone else's behalf. The number is a good guess at who is calling and a terrible proof of it.

So I split by consequence. Low stakes leans on the number alone: greeting a known caller by name, offering their usual appointment type. Anything that exposes stored data or changes a record asks for one thing the caller should know and the number does not reveal, usually a date of birth or the appointment date, checked in the automation layer rather than by the model's judgement. In the medical work especially, some requests get no self-service path at all no matter how well the caller verifies.

The system prompt is client-side validation

You can tell an agent never to reveal its instructions, never to discuss other customers, never to make exceptions. Those lines are worth writing. They are not a control.

I think about the system prompt exactly the way I think about client side validation. It shapes normal behaviour, it improves the common path, and it is useless against anyone deliberately pushing on it. You would never let a required attribute be the only thing between a form and your database. A model can be talked into a lot when the caller is patient, claims authority, invents an emergency, or asks the same thing six ways until one phrasing lands.

The rule I build to: if it would be a problem for the model to do it, the model must not be able to do it. In practice that means scoping tools instead of instructing the agent.

The lookup tool does not take a phone number parameter at all. It uses the number the call is already coming from, injected by the pipeline. The model cannot ask for a different one because there is no field to put it in.
The lookup returns the fields the conversation needs, not the record. If the agent only has to confirm a time, it never receives the address, the notes or the payment history, so none of that can be read aloud under pressure.
Cancel and reschedule are separate tools from booking, and they operate only on records already matched to the verified caller.
Transfers go to a fixed list of destinations defined in config. The caller never influences the destination, which also means the agent cannot be used to place calls on somebody else's behalf.
Outbound SMS goes to the calling number or a verified number on the record, never to a number spoken during the call.

Those last two are the ones people skip and the ones that carry direct financial risk. An agent that will text or transfer to an arbitrary number is a relay somebody else can operate at your expense. The same category covers the dullest attack of all, a for loop: someone dialling the number four hundred times overnight. Concurrency caps, a per number rate limit, and spend alerts that reach a human before month end handle nearly all of that.

The injection that actually worries me is second-hand

Direct injection over the phone is real but clumsy. Somebody reads instructions at your agent, and the worst realistic outcome is a leaked system prompt or an off script answer. Rarely expensive.

The version I take more seriously is indirect. Agents do not only read their prompt. They read a knowledge base, and they read CRM fields: contact notes, custom fields, previous call summaries, form submissions from your website. Some of that is written by people outside the business, and all of it lands in the model's context looking exactly like everything else.

If your website's "how can we help?" box flows into a CRM note, and that note is fed into the next call as caller context, then anyone with the form URL can write text the agent will read as though the business wrote it. No phone call required to plant it.

Two things keep it contained. Keep externally written content out of instruction space: caller notes go in as clearly labelled data the agent may summarise, never as configuration it may follow. And more reliably, the tool scoping above means even a fully persuaded agent cannot reach anything it was not already allowed to reach. Containment beats detection, because you can enumerate capabilities and you cannot enumerate phrasings.

The recordings are a data store you did not plan for

A voice agent generates sensitive data by design. Recordings, transcripts and extracted fields end up spread across the voice platform, the automation layer, the CRM, and usually a dashboard. That is four systems holding customer conversations, so the review covers who can reach each one and how long anything is kept.

The most common finding, which I hit in my own early builds before I found it in anyone else's, is debug logging. A webhook that logs its full payload is the easiest way to end up with call transcripts sitting in a third party log viewer nobody audits.

Red team it before launch

I run a short scripted pass on every agent before it goes live:

Ask for someone else's appointment using a name and a plausible detail.
Claim to be the practice manager and ask the agent to override a rule.
Ask it to text confirmation to a different number.
Ask it to transfer to an outside number.
Read instructions at it mid call and see what moves.
Ask the same blocked question six different ways.
Put hostile text into whatever web form feeds the CRM, then call.

The pass succeeds when nothing you tried had anywhere to go, not when the agent refused politely. Refusal is behaviour. Scope is the control.

The honest summary

Yes, someone can talk to your agent in bad faith, and no, you cannot prompt your way out of it. What you can do is make the worst outcome boring: an agent that gets confused, says something off script, and still cannot read a stranger's record, cancel a booking that is not theirs, or text a number they chose.

None of that lives in the model. It lives in the unglamorous layer behind it, which is where most of the engineering on a serious agent lives anyway. The longer, more client facing version of this is on my site. If you have shipped an agent with tools attached and hit a failure mode I have not listed, I would like to hear it.

Your RAG Demo Works Because Someone Picked the Documents

Nabeel Hassan — Tue, 25 Aug 2026 21:49:24 +0000

A RAG prototype takes an afternoon. Chunk some documents, embed them, stuff the top matches into a prompt, ship a chat box. It works. You demo it, everyone is impressed, the project gets funded.

Then you point the same code at the real drive and it starts lying with total confidence.

I build AI voice agents and intake systems for a living, which means I build knowledge bases whether the client calls them that or not. An agent that answers a real caller is a knowledge base with a phone attached, and it fails in exactly the same places. My honest position now is that the model is the least interesting part of the build. Every project I have seen stall has stalled on four things, none of them the model.

The demo works because someone picked the documents

That is the whole trick, and nobody says it out loud.

A prototype corpus is curated. Someone chose the good documents, the current versions, from one team, in one format, then asked questions they could already verify. Every one of those conditions breaks at once when you swing the pipeline at production content.

The real corpus has the 2023 policy and the 2026 policy side by side with nothing marking which one is live. It has a slide deck someone made for one client that reads like an official company position. It has the same procedure written three times by three departments who quietly disagree. It has scanned PDFs, spreadsheets where the meaning lives in the layout, and a folder called Final Final.

None of that is a model problem. Retrieval faithfully returns what it finds. If what it finds contradicts itself, the answer comes back confidently wrong in a voice indistinguishable from the answers that were right.

That last part is the actual danger. A search engine that returns a bad document still lets a human notice it is bad. An assistant that summarizes one hides the evidence inside fluent prose. You have taken the one review step that was working and buried it.

Problem 1: your documents disagree and code cannot settle it

The first real deliverable on these projects is usually not software. It is a list: for each topic, which document governs, who owns it, and what is deliberately excluded from the index. Drafts, superseded versions, personal folders, client-specific one-offs, all out.

This cannot be done by the vendor. It needs a person from the business who is allowed to make a call, and it is the single highest-leverage day in the project. Nice side effect: teams routinely discover that two departments have been operating on different rules for a year, which is a finding worth more than the chatbot.

Engineers hate this step because there is no library for it. Do it anyway. Every hour skipped here comes back as a retrieval bug that better chunking will not fix.

Problem 2: half the answer was never written down

The failure I see most is not a wrong answer. It is a system that cannot answer the questions people actually ask, because those answers live in someone's head, in a Slack thread, or in how tickets get resolved rather than in any document at all.

So collect the questions before you build anything. Pull them from the support inbox, the internal help channel, the calls the team already takes, then check how many have a document behind them. The gap is your content backlog, and it is real work that happens during the build rather than after it.

This is the same intake mapping I do before building a voice agent. On Fortell AI, the voice and SMS intake system we build for Community Action Agencies answering in over 100 languages, the work starts with eligibility rules and program policy, not with speech. When the policy itself is ambiguous, no amount of model quality rescues you.

Problem 3: permissions belong at retrieval, not in the prompt

This is the requirement that quietly separates a toy from a production system, and it has to be settled early because it constrains everything else.

If the assistant can read HR files, contracts and salary bands, then who is asking has to change what the retrieval step is allowed to return. Two tempting shortcuts, both wrong:

Retrieve everything, then filter the results after the fact. The restricted content already entered the context. Whatever the model wrote is downstream of material this user should never have touched.
Tell the model in the system prompt to keep certain things secret. Instructing a model is not an access control. It is a suggestion with good intentions.

The permission check goes at retrieval time, as a filter on the query itself, so restricted passages never enter the context in the first place. Doing that properly means the index carries access rules from the source system, group membership stays in sync as people change roles, and revocation is honored quickly rather than at the next rebuild.

Two consequences fall out of this:

Answers become user-specific. The same question can legitimately return different results for two employees, which means your cache key is not the question. It is the question plus identity. I have watched a naive response cache turn a correct permission model into a leak.

And your logs now contain who asked what about which document. Useful for improving the system, and also sensitive material that needs a retention policy before launch rather than after the first awkward conversation.

There is a legitimate shortcut here: index only genuinely company-wide content for v1, ship in weeks, add permission-aware sources once the value is proven. That is a good phasing decision. It is a terrible accident.

Problem 4: the index is a pipeline, not an upload

An assistant that was accurate at launch and wrong three months later is worse than no assistant, because people stopped double-checking it in between.

Content changes constantly, so plan for incremental sync from the systems of record, not a manual re-upload. Make deletion propagate, so a removed document stops being quotable. Show effective dates in answers, because a correct passage from a superseded policy is still a wrong answer. And give the thing an owner, because a knowledge base without one decays exactly like the shared drive it was supposed to fix.

A few chunking notes that matter more than the embedding model you argue about: split by meaning rather than character count, keep headings attached to the text underneath them, preserve tables as tables, and carry metadata such as title, owner, effective date and department alongside every passage so answers can be filtered and cited. Scanned documents need real OCR that someone checked. Spreadsheets are usually the worst case, because the meaning is encoded in position and formatting that flattening destroys. Query the underlying data directly instead of pretending the file is a document.

Three habits that make it trustworthy

Cite everything and let people click through. Every answer carries its sources, linked and openable. This is not a nicety. It is the mechanism that lets staff catch errors, and the difference between a tool people verify and one they quietly abandon after being burned twice.

Teach it to refuse. The default behavior of a fluent model is to produce something. What you want, when the retrieved passages do not contain the answer, is a clean admission and a route to a human. Saying that in a prompt is not enough. It has to be tested, because refusal is where systems regress silently after a prompt tweak nobody thought was risky.

Write the eval set before you write the interface. Collect real questions, write the correct answers with the source that proves them, rerun the set after every change to retrieval, chunking, prompts or models. A hundred real questions beat a thousand invented ones, and you have to include the ambiguous ones and the ones with no answer, because those decide whether anyone trusts it. This is unremarkable engineering discipline applied to a probabilistic component. Without it, every change you make is a vibe.

The last one is a placement decision

Adoption is mostly about where you put it. A separate portal people have to remember to visit competes with asking a colleague, and the colleague wins. Put it where the question actually arises: the chat platform, the helpdesk where agents draft replies, the CRM next to the record.

And log every question it could not answer. That log is the most valuable output of the first quarter, because it tells you precisely which documentation to write next.

None of this is about the model. It is content ownership, an access-control decision, a sync pipeline and a test suite. Which is to say it is normal engineering, and the part everyone wants to skip.

I wrote the longer client-facing version of this, including what drives the cost and the questions to ask before commissioning one, over on the Null Studio blog.