DEV Community: Nabeel Hassan

My Live Job Pages Are All Empty, and My Code Cannot Tell Me Why

Nabeel Hassan — Sun, 01 Nov 2026 21:36:59 +0000

I opened the public job pages on Upwork Scout while writing this, expecting to screenshot a nice feed. Every one of the eleven category pages said the same thing: "No public sample for this category right now." Every one of them also carried a noindex tag.

The design did exactly what I told it to. Nothing thin got indexed, nothing crashed, the pages returned 200 from the CDN. And that is the problem I want to write about, because from the outside I genuinely cannot tell you why they are empty. At least four different causes produce that exact page, and the code that renders it throws away the information that would separate them.

What the pages are

Upwork Scout is my job-alert SaaS for freelancers. A cron job scrapes Upwork per category, filters jobs against each user's rules, scores the survivors with an LLM, and emails them. I wrote about the cost side in One Scrape Per Category, Not Per User and the matcher in Cheap Filters First, LLM Last.

Every job the scanner stores lands in a Firestore jobs collection. That firehose was private, so I added public pages on top of it: /jobs as an index and /jobs/[category] for each of the eleven categories. The idea is the standard aggregator play. Show a recent sample with snippet-length excerpts and outbound links, let search engines find "AI voice agent jobs on Upwork", and convert visitors to alerts.

The data layer is one function:

export const PUBLIC_WINDOW_DAYS = 7;
export const MIN_JOBS_TO_INDEX = 3; // below this a category page is noindex

export async function recentJobsByCategory() {
  const buckets = new Map(CATEGORIES.map((c) => [c.id, []]));
  try {
    const { jobsCol } = await import("./db");
    const cutoff = Date.now() - PUBLIC_WINDOW_DAYS * 24 * 3600_000;
    const snap = await jobsCol()
      .where("fetchedAt", ">", cutoff)
      .orderBy("fetchedAt", "desc")
      .limit(400)
      .get();
    // bucket each job into its categories, max 25 per bucket
  } catch {
    // unreachable datastore (e.g. env-less build) -> empty buckets
  }
  return buckets;
}

The pages use ISR with revalidate = 1800, and generateMetadata sets robots: { index: false, follow: true } when a category has fewer than three jobs.

There were a few decisions in here I still like. The public card shows a spend band ("$1k+ spent", "has hired before") instead of the client's exact spend, because the page is a teaser and exact figures belong to logged-in users. Excerpts are cut at 160 characters. Outbound links are nofollow. And the function never throws, because a public SEO page that 500s at build time is worse than one that renders empty.

That last decision is the one that came back around.

Four ways to get the same empty page

When I traced backwards from "every bucket is empty", I found four distinct paths, and they all converge on the same HTML.

1. Nobody subscribes to that category. This is by design. The scanner only scrapes categories that at least one active user has picked, most-subscribed first:

let activeCats = CATEGORIES.filter((c) => catCounts.has(c.id));

A category with zero subscribers never gets scraped, so its public page has nothing to show. The empty-state copy was written for exactly this case: "Scanning activates when subscribers pick a category."

2. The daily spend budget ran out. Scraping is billed per item, so the cron route checks a global daily budget before fetching. If the budget is spent, activeCats becomes an empty array and that run fetches nothing. Run it enough times with the budget exhausted and the seven-day window drains.

3. The scrape itself failed. Fetches run under Promise.allSettled, and a rejected category gets a console.error and is skipped. That is correct for alerts (one bad category should not block the other ten), but it means a scraper that fails every run for a week produces no jobs and no visible error anywhere a visitor or I would look.

4. The read failed. If the Firestore query on the public page throws, for credentials, quota, a missing index, anything, the bare catch {} returns eleven empty buckets. The comment says "env-less build", but the catch does not know it is at build time. It catches everything, forever.

Four causes. One of them is healthy (nobody picked that category), one is a business limit working as intended, and two are outages. The page renders all four identically, and the copy confidently asserts the first one.

That is the part that bothers me. An empty state is a claim. Mine claims "no subscribers yet" without checking.

The two quieter problems I found on the way

The 400-row window is shared. The query takes the 400 newest jobs across all categories and then buckets them. The page header says "the last 7 days", but if a few busy categories produce 400 jobs in two days, the effective window for everyone shrinks to two days. A quiet category can fall under the three-job index threshold not because it is quiet, but because its neighbours are loud. I did this deliberately to avoid composite indexes (one range scan, filter in memory), and I would still make that trade at this size, but the "7 days" on the page is a ceiling, not a promise.

Every render reads the datastore twice. generateMetadata needs the count to decide noindex, and the page needs the jobs to render. Both call recentJobsByCategory(). Next only dedupes fetch automatically; a Firestore SDK call is not a fetch, so each regeneration of each category page runs the 400-document query twice. The Next docs call this out directly and recommend wrapping the function in React's cache:

import { cache } from "react";
export const recentJobsByCategory = cache(async () => { /* same body */ });

That is a two-line fix I should have written on day one.

What the fix looks like

None of this needs a rewrite. It needs the function to stop flattening information.

Return a status, not just data. Instead of Map<string, PublicJob[]>, return the buckets plus a reason: ok, read_failed, or stale. A failed read can still render an empty page for visitors, but it should log loudly and it should not pretend to be the healthy case.

Read what the scanner already knows. Here is the embarrassing bit. The cron route already writes meta/scan at the end of every run with lastRunAt and stats, including a categoriesScanned string listing exactly which categories it fetched, or (none). The public page never reads it. One extra document read per render would let the empty state say the true thing: "not scanned because no subscribers", "scanned, nothing new in 7 days", or "scanner has not run since Tuesday".

Separate the visitor message from the operator signal. Visitors do not need to know my spend budget ran out. I do. The page can keep a friendly empty state and still emit a log line, or a tiny /api/health/public-jobs endpoint I can point an uptime check at, that fails when no category has been scanned in 24 hours.

Memoize with cache. Halves the reads per regeneration.

Keep the noindex gate. This was the one part that worked perfectly. I originally built the sitemap from live counts too, listing only categories above the threshold. I changed it to list all eleven unconditionally and let the per-page noindex meta be the real signal, because a sitemap is a hint and was being generated from the same swallowable query. That turned out to be the right instinct: the sitemap is stable, and the page itself is the only place that decides whether it deserves indexing.

The lesson I keep relearning

"Never throw" is a reasonable rule for a public page. But a function that never throws has to put the failure somewhere, and if the answer is "nowhere", you have built something that is indistinguishable from success with zero results. I made the same mistake in a different shape in You Can Retry a Write. You Cannot Retry an Email., where the question was which side effects are allowed to fail silently. The answer there was "decide per side effect, on purpose". The answer here is the same: decide per failure, on purpose, and write the decision down where you will see it.

The empty page is not the bug. The empty page that cannot explain itself is.

If you have built public pages on top of a pipeline that only runs on demand, how do you tell "nothing happened" from "nothing could happen"? I would like to steal your approach.

The Connector Was Never the Work: Five Questions I Ask Before Estimating an Integration

Nabeel Hassan — Sat, 31 Oct 2026 21:36:36 +0000

Every integration ticket I have been handed started with the same sentence: we just need this system to talk to that one. It reads like a connector. It gets estimated like a connector. Then it turns into a project, because the connection was never the work.

The HTTP call is the easy part. What eats the weeks is unspecified behavior: what happens when both sides edit the same record, when the receiving system rejects a row for a validation rule nobody mentioned, and when a human edits by hand the field your automation is also writing. Those are business decisions wearing technical clothes.

This is the checklist I run before estimating one, adapted from a guide I wrote for my studio's clients.

First, name the shape

There are three shapes, and they cost very different amounts.

A one-way feed. Data goes from A to B and never comes back. Orders into accounting, leads into a CRM, events into a warehouse. Cheapest shape, because there is nothing to reconcile.

A two-way sync. Both systems can change the same record and both changes must survive. This is where the money goes. Every field needs an owner, every conflict needs a rule, and you have to stop the two systems bouncing the same update back and forth forever. Most "sync" requests turn out to be a one-way feed plus two or three fields that genuinely go both ways. Finding that out early is worth weeks.

A process that spans systems. A new order triggers a credit check, a stock reservation, a notification and an invoice across four tools, with conditions at each step. That is not a connection. It is a small application whose interface happens to be other people's software, and it should be scoped like one.

Most of my voice agent work is the third kind. The agent on the phone is the visible part; the value is where the outcome lands.

The five questions that size it

1. What does each end actually give you?

Two documented APIs with a sandbox is the good case. In practice you meet an API that covers the reads but not the writes, a rate limit that quietly caps how fast you can backfill history, or no API at all.

With no interface, the options are a file exchange, a read replica, the vendor's export, or, at the very bottom, scripting the UI, which is fragile and should be called fragile out loud. I confirm sandbox access and rate limits before agreeing to any date, because both routinely change the plan.

2. Who owns which field?

Source of truth is per field, not per system. The CRM owns the phone number. The billing platform owns the payment terms. Where two systems both let a human edit the same field, you either pick a winner or write a rule. Leaving it unstated is how you get a sync that flaps between two values while each team insists its system is right.

This shows up in code more literally than people expect. In VoiceDash, the Retell webhook upserts a call into a Conversation row, but the update block deliberately leaves out tags, notes, summaries and evaluations. Those fields belong to the analysis step and to the humans using the portal, and a late webhook must never overwrite them. I wrote up the full version of that in Undefined Is Not Null. The field map was not a document. It was which keys I allowed into an update: {} object.

3. How fresh does the data have to be?

Nightly batch and event-driven within seconds are different builds, not a config flag. Real time brings webhooks, queues, ordering questions and a replay path for events you missed while you were down. So ask what actually breaks if the data is an hour old. Often nothing.

Set freshness per flow, not per project. And if you do go event-driven, remember that a webhook tells you something happened, not that you have everything. You will want a backfill path anyway.

4. How is the same thing recognized on both sides?

Two systems almost never share an identifier. The same customer is a record in one and a contact in the other, matched on an email that has since changed or a phone number stored in three formats. You need an explicit matching rule, and you need a place for the records that do not match, with a person who checks it.

Data that does not reconcile is worse than missing data, because people keep trusting it. The accounting software I build at Null Studio lives and dies on this; more in A Stored Balance Is a Rumor.

5. What happens when a transfer fails?

Not if. A destination will be down, a record will be rejected, a credential will expire. The difference between an integration that survives and one that quietly stops is the failure path:

retries with sensible backoff
idempotency, so a retry does not create a second invoice
a queue that holds what did not go through
an alert that reaches a named human
a reconciliation report somebody actually reads

The cheapest idempotency I know is a unique constraint on the other system's ID. VoiceDash keys conversations on @@unique([agentId, platformCallId]) and upserts against it, so a webhook delivered twice lands on the same row, and the backfill job that re-reads calls from Retell looks rows up by the same key instead of inserting:

await prisma.conversation.upsert({
  where: {
    agentId_platformCallId: { agentId: agent.id, platformCallId: call.call_id },
  },
  create: { /* full row */ },
  update: { /* only the fields this source owns */ },
});

It also writes a raw WebhookLog row before parsing anything, so when a payload surprises me I can see what actually arrived instead of guessing.

That failure path is roughly half the engineering, and it never appears in the estimate. Someone also has to watch the exceptions after launch; monitoring voice agents in production is the same problem with phone calls attached.

When no-code is the right answer

I build custom integrations for a living and still recommend no-code tools often. If volume is modest, the mapping is simple, and a missed record costs an apology rather than money, use one.

The signs you have outgrown it:

Volume. Per-task pricing that was trivial at a hundred records a day is a line item at fifty thousand.
The logic became a program. Branches, loops and lookups in a visual editor are still software, just without tests, version control or review.
Error handling beyond a red icon. Retries, ordering, partial failure and reconciliation are engineering.
The data cannot go through a third party. Regulated or contractual data often settles it on its own.
Nobody can safely change it. The person who built it left, and now it runs a process nobody wants to touch.

The sensible middle: no-code at the edges for low-stakes notifications, a properly built spine.

What AI changes, and what it does not

The build side compresses a lot. Field mapping, transformation code, adapters, retry handling and the pile of tests an integration needs are the mechanical work coding agents do well under review.

Some integrations get replaced by an agent that just does the task. Careful with that framing: an agent that reads your CRM, checks availability and books an appointment is an integration project wearing new clothes. All five questions still apply, and two apply harder because nobody is clicking the button: idempotency, so a retried booking does not become two, and permissions, so it can only touch what it should. I argued the second one in Give an AI Agent Write Access One Verb at a Time.

What AI does not compress is deciding who owns which field and how fresh it must be. That needs the people who run the operation, and they are usually the critical path.

How I scope one now

Read both ends' docs before estimating.
Write the field map. Every field that moves, which direction, how often, and which side wins a conflict. That document is the project.
Set freshness per flow.
Design the failure path first. Retries, idempotency, a reject queue, one named person on the alert.
Build the thinnest end-to-end slice with real data. One record type, all the way through, in production conditions.
Run it next to the manual process and reconcile before anyone switches the manual step off. The diff finds the rules nobody wrote down, which is the same lesson as rewrites failing in month seven.

An integration is not a wire between two boxes. It is an agreement about who owns what, how fast it travels and what happens when it does not arrive. Make those decisions on purpose and you get a boring system that runs for years. Skip straight to the connector and you get one that works in the demo and needs someone watching it forever.

What is the worst field-ownership fight you have been dragged into? I would love to hear how it got settled.

The original, client-facing version of this guide lives on the Null Studio blog.

Five Habits Unity Gave Me for Building Voice Agents, and Three It Could Not

Nabeel Hassan — Fri, 30 Oct 2026 21:35:57 +0000

I spent five-plus years in Unity before I built my first AI voice agent. Freelance game work through university, then AR for fire response at ARCortex, clinical XR on a Vive Focus 3 at Nystag, a cross-platform C# SDK at Geonode, and a sports-tech platform as CTO at RAQTS.

When I started taking voice agent work, the assumption from every direction was that I was starting over. Unity was a charming line on a CV, not preparation.

That turned out to be wrong in a specific way. Very little of what I use now is knowledge about Unity. Almost all of it is knowledge about building things that run in real time, in front of a human who will not wait, on top of sensors that lie. That describes an AR headset on a fire ground, and it describes a phone call. The phone call turned out to be the easier of the two.

Here is the honest split: five habits that carried over almost intact, and three that did not carry over at all. The second list is where the expensive lessons were.

What transferred

1. The frame budget is the latency budget

Unity teaches you that time is a hard resource before it teaches you anything else. At 60 fps you have about 16 ms per frame. On a standalone VR headset it is closer to 11 ms, which I wrote about in Standalone VR Gives You 11 Milliseconds a Frame.

The discipline that produces is not "make it fast". It is knowing exactly what sits on the critical path and what can be pushed off it. You profile instead of guessing, you move work off the main thread, you spread expensive work across frames, and you stop trusting averages, because the player only ever feels the worst frame.

A production voice agent needs exactly those habits. A caller notices the slowest turn, not the median one. The work has the same shape: find what actually blocks the response, move CRM writes and notifications out of the call into automation that runs afterwards, cover an unavoidable wait with speech instead of silence, and measure the tail. I broke that down for phone calls in A Faster Model Will Not Fix Your Slow Voice Agent. Reading it back, it is the frame budget in different units. 16 ms became roughly 800 ms. The psychology is the same: past a threshold, the thing stops feeling alive.

AR made this sharper. On the ERIS XR platform at ARCortex, a label that lags the world is not a slightly late label. It is wrong about where something is.

2. Everything real time is a state machine, and the bugs live in the transitions

Game developers live in state machines: character controllers, UI flows, animation graphs. Experience teaches you that bugs almost never live inside a state. They live in the transitions, especially the ones nobody wrote down. The player who opens the inventory mid-jump while a cutscene loads is the one who finds the gap.

Designing a conversation flow is the same job. The nodes are easy. What decides whether an agent survives a real caller is the exit conditions. What counts as done? What happens when the caller answers a question you did not ask? Where do they go when the whole thing falls apart?

I now write the exits before the happy path, which is a habit I took straight from character controllers. It is the core of Your Voice Agent Needs Three Exits, Not One.

3. Sensors lie, so design around the lie

I did not expect this one to carry over so directly. Location-based AR runs on GPS and a magnetometer, and both are wrong in ways that are neither random nor constant. In geospatial AR, heading error, not position error, was usually what broke the illusion. At Nystag the whole product was a measurement taken from an imperfect eye-tracking signal that a clinician would later base a judgement on.

Speech recognition is the same kind of input. It is a noisy sensor with structured failure modes, and you can learn the structure. It fails worst on the values that carry no redundancy: names, postcodes, registration numbers, email addresses. Whole sentences survive. (I compared the lecture and phone-call versions of this problem in Transcribing a Lecture and Transcribing a Caller Are Not the Same Problem.)

Once you accept that every reading comes with an implied confidence, you build differently:

validate the shape downstream instead of trusting the raw value
read back the values that matter, in small chunks
store confidence alongside the value
give the repair path an end, so it cannot loop forever

AR taught me not to trust a number just because a device produced it. A transcript is a number a device produced.

4. Playtesting is adversarial QA

Anyone who has watched a stranger play their game for the first time has learned something no test suite tells you: users do not follow the path. They walk into the wall, mash every button during the tutorial, and answer the question you did not ask.

Callers are players. They interrupt, give you two facts at once, go quiet, and ask something completely outside the agent's job in the middle of a booking. Game work builds the instinct to test with the difficult user, not the cooperative one. That is why the pipeline I built at Fortell AI, described in How I use Claude Code and Comet to build and test AI voice agents in a day, aims at difficult calls instead of clean ones. An agent that handles the ideal caller proves nothing. So does a game that only survives its own developer playing it.

5. Shipping through a gate that does not care about you

Store releases teach a particular kind of production discipline. There is a submission, a review, a rejection, a build that has to be reproducible, and a rollback plan you should have thought about before you needed one. Shipping the Unity Lumo platform and the mobile apps through the App Store and Play Console at RAQTS, and building the Unity and Windows CI/CD around the Repocket SDK at Geonode, taught me that "it works on my machine" describes where it works. It is not a defence.

Voice agents have gatekeepers too, and they are less forgiving than they look: carriers, A2P registration, recording-consent rules, healthcare and privacy constraints. The rules are different and the posture is the same. Something outside your control can block you from reaching users, so you plan for it in week one, not the day before launch.

What did not transfer

You cannot replay a phone call

Games are deterministic enough to re-run. Same seed, same inputs, same bug, as many times as you need. That was the biggest thing I lost. A live call cannot be repeated: a different caller, different background noise, and a model that will not produce the same output twice. You cannot step through it.

The fix was to build observability up front, because there is no debugger: transcripts, recordings, structured post-call data, the full tool-call trace. You are doing forensics after the fact, not stepping frames, so the job becomes capturing enough evidence at the time to reconstruct what happened later. I went deeper on that in There Is No Repro for a Phone Call.

You can see a game. You cannot see an agent.

Almost every visual bug in Unity announces itself. You look at the scene and something is in the wrong place.

Voice agent failures are silent, and they look like success. The call completes, the transcript reads fine, and the customer is never contacted because the email address was captured wrong. Nothing errors. I had to deliberately replace looking with reading, and to build the habit of reading failed calls from start to finish instead of sampling random ones.

The engine is yours. The model is not.

Unity is a dependency you can reason about, decompile, work around, and eventually understand completely. A language model is a dependency whose behaviour changes without a changelog, and you cannot patch it.

You do not fix it, you route around it. Keep the agent thin. Put anything that must be correct into deterministic automation downstream instead of trusting the model to remember it. Limit what it is allowed to say wherever the wording carries risk. That architectural caution is the biggest real change in how I build, and none of it came from games.

Should a Unity developer make this switch?

If you want to, yes, and faster than you think. The parts of AI engineering that take people longest are not the prompts. They are latency intuition, state design, tolerance for unreliable input, adversarial testing, and release discipline. A game developer already has all five. What you are missing is a stack, and a stack takes a few weeks.

What I would not do is treat it as a clean break. When I get hired for agent work, the reason is rarely the model layer. That is the commodity part. It is that the system around the model holds up, and that came from years of building things that had to run at speed in front of a person. It is the same conclusion I reached in Four ICPC Habits That Survived Production: the tools changed four times, and the underlying muscle never did.

If you made a jump like this, from games, embedded, or anything else real time, into AI work: which habit carried over for you, and which one did you have to drop?

Originally published on nabeelbaghoor.com.

Monitoring Retell Voice Agents in Production: Alert Rules, Disconnection Reasons and the Failures Transcripts Hide

Nabeel Hassan — Fri, 30 Oct 2026 10:48:25 +0000

Disclosure: I'm Nabeel, founder of VoiceDash, a client portal for Retell agencies. This is an adapted republish of a guide from the VoiceDash blog, rewritten for developers. Nothing below needs VoiceDash; it uses Retell's own features plus a little glue code.

TL;DR: A voice agent that passed every test on launch day can quietly break a month later: a calendar token expires, a webhook starts failing, the business changes its hours, or a provider has a bad afternoon. The agent keeps answering and sounds fine, and books nothing. Monitoring in production means watching four layers (is the line answering, are tools working, is quality holding, is usage normal), wiring a failover before you need it, and routing alerts to one place. Retell now ships most of the primitives; this post maps each layer to the field or feature that covers it.

Why testing is not enough

Pre-launch testing proves the agent worked on the day you shipped it (my checklist for that is in How to Test a Voice AI Agent Before Go-Live). Production is different, because the world around the agent keeps moving:

Credentials expire. OAuth tokens for calendars and CRMs get revoked or rotated when someone at the client changes a password.
The business changes without telling you. New hours, a new service, a staff member who left. The prompt is now wrong and nothing errors.
Integrations fail silently. A workflow throws on a field that changed shape, and the agent still tells the caller "you are all booked."
Vendors have incidents. You sit on top of Retell, a telephony provider and a model provider.
Your own edits break things. A prompt change for one branch shifts behavior in another.

None of these stop the phone ringing. That is what makes them dangerous.

Layer 1: is the line answering at all?

The loud failure: calls ring out, drop on connect, or end in an error. Rare, and the one clients never forgive.

Heartbeat call. Once or a few times a day, place an automated test call to each production number from a separate number. Check that it connects, the agent greets within a normal window, and the call ends cleanly. A small scheduled workflow (n8n, a cron job, anything) plus a Twilio number does it. Alert after two consecutive failures, not one.

Volume floor. Every client has a rhythm. Compare today's call count against the same weekday's average and alert when it falls far below during business hours. Zero calls by noon at a busy office is usually a forwarding change on the client side, not an AI problem, but it is still yours to catch.

Disconnection reasons. Every Retell call records a disconnection_reason, available in Call History and on the Get Call API. Retell's disconnection table splits them roughly into three buckets:

expected (call_status = ended)
  user_hangup, agent_hangup, call_transfer, voicemail_reached,
  inactivity, max_duration_reached, user_requested_dnc ...

not connected (outbound)
  dial_busy, dial_no_answer, dial_failed, invalid_destination,
  marked_as_spam, network_blocked ...

errors (call_status = error)
  concurrency_limit_reached, error_llm_websocket_*, error_asr,
  error_no_audio_received, error_retell, error_unknown ...

Log the field for every call and alert on any spike in the error bucket. Two details worth knowing from the docs: dial_failed is also what you see when carriers have flagged your outbound number as spam after many short calls, and the error_llm_websocket_* family only appears when you run a custom LLM over Retell's websocket.

Layer 2: are tools and integrations working?

The quiet failure, and in my experience the most common one. The conversation sounds perfect; the booking, CRM update or follow-up text never happens.

Log every tool call and its result. Whether the agent calls your endpoint directly or goes through a workflow tool, record the request, response, status and latency, keyed by call_id. If tool results are not logged you cannot see this failure at all.

Give every workflow an error branch. Every workflow the agent depends on needs an error path that alerts with the client name, the workflow and the failing step.

Make the agent honest about failure. The prompt should say exactly what to do when a tool returns an error: apologise, take the caller's details, promise a callback, flag the call. It must never confirm a booking that did not go through. A caller who gets a callback is fine; a caller who shows up for an appointment that does not exist is a complaint.

Reconcile against the source of truth. Once a day, compare calls marked booked against bookings that actually landed in the client's calendar or CRM:

// nightly reconcile, per client
const booked = calls.filter(c => c.call_analysis?.custom_analysis_data?.outcome === "booked");
const landed = await calendar.listEventsCreatedBetween(start, end, { source: "voice-agent" });
if (booked.length !== landed.length) {
  alert(`${client.name}: ${booked.length} calls say booked, ${landed.length} bookings exist`);
}

The outcome field here is whatever you defined in post-call analysis; the calendar call is your own integration.

Layer 3: is conversation quality holding?

Some failures are not errors. The agent answers, the tools work, and it still handles callers badly because the business changed or an edge case became common.

Post-call analysis as the quality signal. Extract a structured outcome for every call (booked, answered question, transferred, message taken, spam) plus a flag for a frustrated caller or a request for a human. Track the rates per client week over week. A jump in transfers or frustrated callers means the prompt no longer matches reality.

A weekly transcript sample. Numbers tell you something changed; transcripts tell you what. Skim a handful of calls per client each week, weighted toward transfers, very short calls and anything flagged frustrated.

Latency, per component. After a call ends, Get Call returns a latency object with p50, p90, p95, p99, min, max for components such as e2e, llm, tts, asr and knowledge_base (not every field is present on every call). Store e2e.p90 per call and watch its trend. If it creeps up, the breakdown tells you whether it is the model, the voice or retrieval. Note that e2e excludes turns where the agent only ran a tool without speaking, so slow tools need their own timing from your logs.

Layer 4: is usage behaving normally?

Usage anomalies are both a cost problem and a symptom:

Average duration spiking often means the agent is stuck in a loop, or robocalls are getting long conversations.
Very short calls climbing can mean callers hang up on a bad greeting or a slow first response.
Minutes running past the client's bundle is a margin problem you want to see mid-month, not on the invoice.

Set a maximum call duration on every agent (it ends calls with max_duration_reached), then alert when a client's daily minutes run well above normal.

Retell alert rules: the built-in layer

Retell has alert rules that evaluate aggregate metrics on a schedule and send an email or a webhook when a threshold is crossed. Metrics include call_count, call_success_rate, negative_sentiment_rate, custom_function_latency, custom_function_failure_count, transfer_call_failure_count, concurrency_used, total_call_cost and api_error_count. Rules can be absolute ("success rate below 90% over the last hour") or relative to the previous cycle, and can be filtered by agent, environment tag, disconnection reason or post-call extraction field.

Two constraints shape how you use them:

Rules are dashboard-only. They are created in the dashboard, not through the public API, so if you manage many client workspaces, budget time to set them up per workspace and keep a written list of what exists where.
They are aggregate, not per call. For "this specific call failed a booking", use your own webhook handler. A sensible split: Retell rules for volume, success rate, function failures and cost; your own code for heartbeats, reconciliation and anything that needs client-specific logic.

Plan the failover before you need it

Monitoring tells you something broke; failover decides how much it costs.

Backup mode. In Retell, each phone number can have a fallback number. With Backup mode turned on, new inbound calls go straight to that fallback before an agent is selected, which is your kill switch while you investigate. Configure the fallback number at launch; Backup mode cannot be enabled until one exists, and if the fallback cannot be reached the call ends.
A human transfer path for when the agent is confused, tested as part of every release.
A written commitment in the client contract: you monitor and respond within a defined window, and upstream vendor outages are excluded from credits. Subscribe to Retell's status page so you know about their incidents as early as they do.

Route alerts so someone actually sees them

The fastest way to kill a monitoring setup is noise. If every short call pings your phone, you will mute the channel within a week and miss the one that mattered.

One channel, with the client name in every message.
Only alert on things you would act on: failed heartbeat, workflow error, volume far below normal, error disconnections, a booked count that does not reconcile, a Retell rule firing. Everything else goes into a daily digest.
Tell the client before they tell you: what happened, when it started, what you did and which calls were affected. Handled like that, an incident builds trust instead of eroding it.

The bottom line

The agent that worked on launch day is not the agent that is answering today, because everything around it changed. Watch the line, the tools, the quality and the usage; put disconnection_reason, latency and your post-call outcome on every call record; let Retell's alert rules cover the aggregates; and set the fallback number before the first incident, not during it.

Drafted with AI assistance and fact-checked against each vendor's docs.

Hours: Not Publicly Listed. Telling a Voice Agent What It Does Not Know

Nabeel Hassan — Thu, 29 Oct 2026 21:36:50 +0000

In code, a missing value is loud. You read a key that is not there and you get undefined, a null pointer, a stack trace. Something stops.

In a prompt, a missing value is silent. A caller asks a dental receptionist agent "do you take Aetna?", the answer is not in the prompt, not in a tool result, and the model does not stop. It produces the most likely sounding next sentence, and "yes, we accept Aetna" sounds very likely. On a phone line there is no source link next to it and no dead air allowed, so the caller hears one confident sentence and believes it.

I build voice agents for US clients on Retell, n8n and GoHighLevel, and almost every hallucination I have chased in production fell into four buckets: invented prices, invented availability, invented policies, and confident answers outside the business (a symptom, a legal question). This post is how I handle each, using the real prompts from nine client agent exports I keep on disk, including two places where those prompts were the problem.

Null is a value you have to write down

Here is my favourite line in any of my prompts. It sits in the Context section of an HVAC agent:

- Business hours: Not publicly listed - never quote specific hours. Just offer to schedule.

The water treatment agent has the same rule. Compare that with the dental and roofing agents, whose Context sections list real hours, day by day.

Both shapes are fine. What is not fine is the third shape, where hours are simply absent. An absent field is not null to a language model, it is an invitation. "We're open nine to five" is the most probable answer for a local business, so that is what you get. If the business genuinely does not publish its hours, the prompt has to say so explicitly, the same way a schema distinguishes "field missing" from "field present and null".

So my first pass on any new agent is boring: list the questions callers actually ask (hours, prices, insurance, financing, areas served, who the staff are) and make sure every one has either a value or a written-down "we do not say this, here is what we do instead".

A denylist only works with an instead

Every agent has a short list of things it must never state on its own, and the list is different per vertical, because the dangerous facts are different:

The two law firm agents: never give legal advice; the injury firm adds never quote settlement values or contingency percentages.
Credit repair and debt settlement: never give specific advice or guarantees, only collect information and schedule.
HVAC: never guarantee exact pricing without a diagnostic visit, never diagnose complex failures over the phone.
Water treatment: never diagnose water quality with certainty, never promise exact pricing.
Roofing: do not guarantee insurance claim approval.

A bare "NEVER" is half a rule, though. If the only options are "answer" or "awkward silence", the model will answer. The rule that works carries its own replacement line. The sales agent for CallSetter does this in one block:

### Pricing Lock
NEVER mention specific prices. Always say: "Pricing is customized based on your volume and will be covered on the demo."

The HVAC, dental and roofing agents each have a scripted price answer built around the same phrase, "I wouldn't want to guess", followed by the real next step: a diagnostic visit, a free consultation, a free estimate. That turns the most common hallucination trigger, a price shopper, into a booking path instead of a number.

The honest limit: a denylist covers what you anticipated. It is the security model you already know is weak. It stops the obvious cases and does nothing for the question nobody thought of, which is why the next two sections matter more.

Live facts come from a tool, as data

Anything that changes (open slots, today's schedule, a job status) should never be a sentence in the prompt the model can paraphrase or extrapolate from. My agents get availability from tool calls (get_slots_from_ghl or checkCalendar, depending on the client) at the moment the caller asks, and the prompt says "always give exactly two specific times", so the agent reads back what the tool returned instead of summarising a calendar.

Two rules make this hold:

Return facts, not prose. {"available": false, "next_slot": "Thursday 10:00"} is much harder to embellish than a paragraph about the week.
Handle the empty case in the prompt. Most invented availability I have debugged happened when a tool returned nothing or failed and the model filled the gap. I wrote about the failure path, including the two-minute timeouts I found on every tool, in the outages post.

How to shape the tool parameters themselves is in Every Tool Parameter Is a Question Your Voice Agent Has to Ask.

Sometimes the model is right and the prompt is wrong

While going back through the exports for this post I counted where the dental agent's business hours appear. The string 8:30 shows up nine times in that one prompt: the Context section, a rules section, the "not ready to book" line, and several of the worked examples. The roofing agent's hours appear ten times.

That is a config bug waiting to happen. When the practice changes its Friday hours, someone edits the Context section, misses one of the example dialogues, and now the agent has two answers for the same question. When it gives the old one, it will look exactly like a hallucination in the transcript. It is not. The model is faithfully reading a stale copy.

Then I found a better one. The dental agent's cost answer:

- Cost question: "The consultation and exam pricing depends on what the doctor recommends. The inspection is free if you'd like to start with that."

Dental practices do not do inspections. Roofers do. Several of these agents were cloned from a shared base (I covered the HVAC agent listening for "Invisalign" in an earlier post), and this line leaked across from roofing. The same sentence is in the updated dental prompt I keep in a separate file, so it survived a revision. I have not re-exported the live agent to check whether it is still there, and I am not going to claim a caller heard it. But if one did, the transcript would read like the model making up a free inspection, and I would have gone looking for a model problem.

The lesson I took: before you blame the model, grep the prompt. Treat it like config. One fact, one place, and examples that reference the fact instead of restating it. That is also the argument for keeping facts out of fine-tuning, which I made in the fine-tuning post.

Narrow the room to improvise

All nine agents run on gpt-4.1. Only one of the nine exports set model_temperature explicitly (to 0), and the fixed HVAC export added a second. The rest were left at the platform default. A receptionist does not need creative phrasing, so this is now part of my setup checklist rather than something I remember to do.

Structure helps too. For fixed intake flows, a conversation flow gives the model fewer chances to wander than one long prompt. All nine of mine are single prompts of roughly 2,400 to 4,400 words, and that post covers where that stops working.

Test off the happy path, then keep reading transcripts

Happy-path testing (book an appointment, ask the hours) finds almost no hallucinations, because they live off the path. Before go-live I run calls designed to provoke one:

Ask for a price on a service not in the prompt
Ask for a slot that is already booked
Ask about an insurance provider or payment plan the business does not offer
Ask a medical, legal or safety question
Ask for a staff member who does not exist
Interrupt mid-answer and change the question
Say something ambiguous like "is it open?" with no context

Every one should end in a grounded answer or the approved fallback. Anything else goes back into the prompt. Why these have to be scripted rather than just "try it" is in There Is No Repro for a Phone Call.

After launch, real callers are more creative than any script. What catches the rest is post-call analysis fields that flag calls where the agent mentioned a price, a policy or a date, plus a weekly read of the flagged calls and a random sample. Fifteen minutes per client per week finds most problems before the client does. And since clients will hear about a bad answer from their own customers eventually, they need to be able to open the transcript themselves. That is a big part of why we built VoiceDash, the white-label portal where each client sees their own calls and transcripts.

The short version

You will not get a language model to zero hallucinations. You can make them rare and boring:

Write down what the agent does not know, as explicitly as what it does.
Pair every "never" with the line to say instead.
Pull every live fact from a tool, as data, with the empty case handled.
Keep each fact in one place in the prompt, and grep before you blame the model.
Set the temperature on purpose.
Test off the happy path, then read transcripts every week.

The fix I keep relearning is the fourth one. Both hallucination-shaped bugs I found in my own exports were in the prompt, not the model.

What is the strangest thing you have found when you went looking for a model bug and the bug was in your own prompt?

This is adapted from a longer guide on the VoiceDash blog: How to Stop a Voice AI Agent From Hallucinating.

A Stranger Writes Half of My AI Matcher's Prompt

Nabeel Hassan — Tue, 27 Oct 2026 21:35:23 +0000

Every night a cron job in Upwork Scout, my job-alert side project, asks Claude Haiku how well a freelance job fits a specific freelancer. The prompt has two halves. One half is the freelancer's own profile. The other half is the job post: title, budget, skills, client stats and up to 1,400 characters of description.

I did not write that description. A stranger on Upwork did. It lands in the model's context next to my scoring rules, and the model has no reliable way to tell my instructions apart from text that merely sits near them.

That is the whole AI security problem in one paragraph. We tend to argue about prompt wording; the fixes that work are boring engineering around the model, the same rules you apply to any input you did not write.

Code and data stopped being separate

Normal software keeps a clear line. Code decides what happens; data gets processed. A language model erases that line. Your instructions, the user's question, a retrieved document and an uploaded file all arrive as one stream of text, and the model acts on all of it.

Two things follow. Anyone who can put text in front of the model has some influence over what it does. And the output is generated, not looked up, so you cannot fully predict it.

The direct version of prompt injection is a user typing "ignore your instructions". The one that matters more is indirect: the instruction rides inside content the model reads on someone's behalf, like an email, a knowledge base document, a support ticket or a job post.

No prompt wording reliably fixes this. So I stopped asking "can it be injected?" and started asking a better question.

What is the worst an injected instruction can do?

Design so the answer is small. For the Upwork Scout matcher, it is honestly small:

The model can read one job and one profile, and that profile belongs to the user being scored. There is no tool, no database access, no other account in the context.
Its output is forced into a fixed shape with structured outputs: an integer score and a string reason, with additionalProperties: false.
My code then clamps the score to 0 to 100, cuts the reason to 200 characters, and returns null on any failure so that job falls back to plain filter matching.

const parsed = JSON.parse(text) as { score: number; reason: string };
const score = Math.max(0, Math.min(100, Math.round(Number(parsed.score))));
if (!Number.isFinite(score)) return null;
return { score, reason: String(parsed.reason || "").slice(0, 200) };

So the worst case is a job post that talks its way to a 100 with a glowing one-line reason, landing at the top of somebody's alert email. That is a real failure. A freelancer could spend Connects on a bad job because my ranking told them to. But it is a bounded failure. Nobody's data leaves, nothing gets written, and the damage is one row in one email.

Compare that with a feature that can query the whole account, or one with write tools. Same injection, very different worst case. That is why the first control is not in the prompt at all: limit what the feature can reach, and keep authority in your application, decided by the signed-in user's permissions rather than by whatever the model concludes from the conversation. I wrote about the write-tools case separately in giving an AI agent write access one verb at a time, and about a voice-agent version of the read case in the lookup tool that just took a phone number.

The model's output is user input

This is the rule I wish more teams internalized. Whatever the model returns should be handled like input from an unknown user, because partly it is. A stranger's text shaped it.

In Upwork Scout, that reason string goes two places: the alert email and the dashboard. The dashboard is React, so the string is rendered as text by default. The email is a hand-built HTML template, so the reason goes through the same tiny escape function as the job title and description:

function esc(s: string): string {
  return s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}

Nothing clever. The point is that model output takes the same path as every other untrusted string, not a special "it came from our AI, it is fine" path.

The rendering risk is worse when your UI renders rich text. A model can be steered into producing a markdown image or link whose URL carries user data in the query string. The moment the page loads that image, the data has left your system without anyone clicking anything. If you render model output as markdown, restrict link destinations, do not auto-load external images, and escape HTML by default.

The same applies downstream. If output feeds a query, a spreadsheet, an email or code that runs, validate it at that boundary.

Data leaks are usually correct answers

The most damaging failure in a business AI feature is rarely a weird answer. It is a correct answer built from data the user should never have seen.

Three places this happens:

Retrieval without permissions. If your search step pulls from everything and relies on the model to hold back what the user cannot see, it eventually will not. Filter at fetch time so restricted passages never reach the context. More on that in your RAG demo works because someone picked the documents.
Shared context across tenants. Shared caches, shared conversation memory and batch jobs that group requests across customers are the classic spots where one account's data shows up in another's answer. Every call should be assembled from one tenant's data only.
Training on user content. A model cannot keep a secret it was trained on. That is one more reason I keep business facts in retrieval or the prompt, not in weights, as I argued in fine-tuning your facts is hardcoding them into a binary.

Your endpoint is an open tab

A public AI feature is a model someone else pays for. People will try to use your support bot as a free general assistant, push huge inputs through it, or script it.

Here Upwork Scout is a mixed report card, and I would rather be honest about it. The nightly matcher only runs on jobs that already passed the user's hard filters, verdicts are cached per user and job, and there is a cap of 150 AI scores per user per day. That part is in good shape, and the cost side is in cheap filters first, LLM last. The profile import endpoint, where a user pastes their Upwork profile and Claude structures it into fields, requires a signed-in session and truncates input to 24,000 characters, but it has no per-user rate limit yet. Writing this post is how I noticed. It is on the list.

Basic controls cover most of this: per-user and per-account rate limits, input size limits, sign-in for expensive actions, a narrow scope that declines unrelated requests, and spending alerts on the provider account. The broader cost story is in model calls are metered.

Logs are the leak nobody plans for

For every vendor you send data to, know what you send, whether they retain it, whether it trains anything, and where it is processed. Then look at your own logs. Prompts and responses are gold for debugging, so keep them, but they contain whatever users typed and whatever your system retrieved. They need the same access control, retention and redaction as the database the data came from. A side note from my own matcher: on failure it logs only the job ID and the error message, not the prompt, which is the kind of decision that is easy to make on day one and annoying to retrofit.

A test pass you can actually run

Before launch, and again whenever the model, the prompt or a data source changes:

Injection attempts through every input path: direct messages, uploads, retrieved documents and any third-party content the feature reads. For me, that means a job post written to game the score.
Cross-account checks: sign in as customer A and try to surface customer B's data through the AI feature.
Prompt extraction: get the system prompt out, then confirm nothing in it hurts.
Rendering tests: links, images and markup the model was steered into producing.
Abuse tests: hit the rate limits and confirm the spending alerts fire.
A log review: what is stored, who can read it, when it is deleted.

The takeaway

AI features are not secured by clever prompts. They are secured the way the rest of your app is: access decided in your own code, untrusted input and output treated as untrusted, each component limited in what it can reach, and a clear picture of where data goes. Do that from version one and the model's unpredictability becomes a quality problem instead of an incident.

The longer, less code-heavy version of this, written for founders evaluating a team building AI into their product, is on the Null Studio blog: AI feature security.

What is the worst thing an injected instruction could do in a feature you have shipped? I am curious how many of us have actually written that answer down.

Every Tool in My Voice Agents Had a Two-Minute Timeout. Callers Give You Eight Seconds.

Nabeel Hassan — Mon, 26 Oct 2026 21:35:50 +0000

When a web service goes down, the user gets an error page. Ugly, but honest. When a voice agent goes down, the caller gets silence, and silence on a phone line reads as "this business is closed." They hang up and call the next result on Google. Nobody files a bug report.

So I now design the failures first. This is the engineering version of an outage plan I wrote up for clients, plus a confession from my own agent exports.

A call is a dependency chain you rent

When I scope an agent, I write out every service a single call touches, in order:

The business's existing number and carrier, often forwarding to a new number.
The telephony layer, usually Twilio or the voice platform's own numbers.
The voice platform, Retell AI in most of my work, which runs the conversation.
Speech-to-text, the LLM and the text-to-speech voice, which the platform orchestrates but which may come from different providers.
The tool layer: n8n workflows, a calendar, GoHighLevel.
The human fallback: a transfer number, a ring group, a voicemail box.

"The AI is down" is not one incident. It is five or six, and each one sounds different to the caller:

Top of the chain breaks (forwarding changed, number detached from the agent, telephony billing hold): the phone rings into nothing. The worst case.
Tool layer breaks (calendar times out, n8n workflow deactivated, CRM rejects the write): the agent answers, collects a name, and then cannot book. Pickup rate stays healthy, so dashboards look fine.
A model or voice provider degrades: higher latency, a glitchy voice, bad transcription. People hang up mid-sentence.

The design job is deciding what each failure sounds like.

Rule one: the number always terminates somewhere

My first rule is that the phone number must always end at something that works without the AI: a human, a ring group, or at worst a voicemail box someone actually checks.

If the number lives in Twilio, configure its fallback handler, which runs when the primary one fails, to forward to a backup line or record a voicemail and email it. The worst outage becomes a slightly worse phone experience instead of a dead line.

If the business forwards its line to the agent, overflow forwarding (staff ring first) has a fallback built in; all-calls forwarding has none unless you build one.

Either way, the client can turn forwarding off in two minutes. That is the manual circuit breaker, and it does not need me awake.

The test I use: if the voice platform vanished right now, what would a caller hear? If the answer is "nothing," fix that before touching the prompt.

Rule two: degrade to a message taker

This is where my own exports got interesting. I keep JSON exports of nine Retell agents I have built in one folder (I have mined it before for tool parameter design and a config diff that went wrong). I grepped them for failure handling, expecting to feel good.

The good news: every single prompt has a tool failure rule. The shape is identical across all nine:

If a tool fails: apologize briefly, retry once. If second failure:
"I apologize for the inconvenience. Let me have someone from the
team call you back to get this scheduled." End call.

Right instinct: no invented slots, no infinite loop, a human promise at the end. Reading it as an engineer, though, four things bother me.

The timeout is two minutes

Every custom tool across the nine exports has the same setting:

{
  "type": "custom",
  "name": "checkCalendar",
  "timeout_ms": 120000,
  "speak_during_execution": false
}

So "if a tool fails" can mean the model waits up to two minutes before it learns anything failed, and "retry once" can double that. Meanwhile speak_during_execution is false, so nothing is said while the request hangs. The prompt does tell the agent to say a short bridge like "let me check on that" before calling a tool, but a bridge covers a second or two, not a stalled webhook.

A two-minute timeout on a user-facing web request would get flagged in review. On a phone call, a caller gives you maybe eight seconds of silence. I had tuned the prompt rule and never revisited the timeout under it. It should sit near the real p99 of the n8n workflow plus headroom, and the retry should not exist for anything that is not idempotent. (Booking twice because the first request was slow, not failed, is its own bug; I wrote about that class of problem in you can retry a write, you cannot retry an email.)

"Please call us back" may point at the agent

Several of the fallback lines tell the caller to call the business back at its main number. That is fine if that number reaches staff. If that number is the one forwarded to the agent, the fallback is a loop: the caller rings back into the same broken tool and hears the same apology.

I have not verified the routing for every one of those agents, and that is the point: the fallback text lives in the prompt, the routing lives in a carrier dashboard, and nothing connects them. The fix is a different path, not a different sentence: a transfer that bypasses the agent, or a captured message and a callback.

"End call" can drop the caller's details

The agent has usually collected a name, a number and the reason for calling by the time the calendar fails. Then it ends the call. Whether those details survive depends on whether the post-call pipeline captures them, and if the failing tool is the CRM write, writing the message to the CRM does not help.

My rule now is that caller details land somewhere independent of the failing tool: a team notification or a stored record that a post-call workflow can replay once the CRM is back.

Only two agents can hand off

Only 2 of the 9 exports have a transfer_call tool. For the rest, a tool outage leaves the agent with exactly two moves: apologize, or hang up. A tool outage is precisely when a live transfer earns its keep, which is the argument I made in your voice agent needs three exits, not one. My own exports are the counterexample.

One thing the platform already did for me

Every export has fallback_voice_ids set to two alternate voice providers, so a bad day at the primary text-to-speech provider means a different voice, not a mute agent. Real automatic failover, and it works because the platform owns both sides of the switch. I did not design it, which is also why it was the one layer already covered.

Rule three: alert on absence, not just errors

The embarrassing outage is the one the client reports to you at 10am. Two checks catch most of what dashboards miss.

A scheduled synthetic call. A real call to the agent's number from outside the stack: wait for the greeting, ask one scripted question, check the post-call event arrived. It is the only thing that catches top-of-chain failures, because a broken forwarding rule produces no error in the voice platform. From its point of view, no call ever arrived.

Zero-volume alerts. Errors are easy to alert on. Silence is not. Track normal call volume per hour per business and alert when it drops to zero during hours that usually have calls. A clinic that gets a dozen calls on a weekday morning and has received none since 8am has a problem, even if nothing threw.

Rule four: a runbook beats clever automation

Automatic switching between, say, two LLM providers can fail in more confusing ways than the outage itself. An untested model behind the same prompt is a different agent, and it needs the same regression tests as any other change.

So each agent gets a one-page runbook written with the client: who gets alerted, the kill switch (with screenshots of the carrier or Twilio setting), and who works through the message backlog afterward.

The checklist I now run before launch

The number has a fallback route that ends at a human or a checked voicemail, and that route bypasses the agent.
Tool timeouts are set from measured latency, and retries only exist for idempotent calls.
Caller details are captured somewhere independent of the CRM.
A transfer tool exists, or there is a written reason it does not.
A synthetic call and zero-volume alerts watch the real number.
A one-page runbook names the kill switch and who handles the backlog.

The honest target is not an agent that never goes down. It is that no outage produces dead air, loses a caller's details, or lasts longer than it takes a named human to notice and flip a switch.

The full client-facing version of this outage plan is on my site: What Happens When Your AI Voice Agent Goes Down.

What does your stack do when a dependency stalls instead of failing? I am curious whether anyone has found a clean way to keep prompt-level fallback text and carrier-level routing in sync, because right now mine live in two places that never talk.

My HVAC Voice Agent Was Listening for the Word Invisalign

Nabeel Hassan — Sun, 25 Oct 2026 21:35:59 +0000

Last week I wrote about a bug I found by diffing two exports of the same Retell voice agent: Airheads HVAC Agent - NABEEL.json and Airheads HVAC Agent - FIXED.json. That post was about the prompt, a booking step that contradicted a global rule.

This post is about the rest of that diff. Because when I went back and ran the full comparison, the prompt fix was the part I already knew about. The part I had not noticed was that my HVAC agent was configured to listen for the word "Invisalign."

What the full diff showed

The agent is an inbound scheduler for an HVAC company in Florida. Its prompt was written for HVAC from line one: "an AI scheduling assistant for COMPANY_NAME, an HVAC service company in Florida." Read the prompt and nothing looks wrong.

Here is what changed between the two exports, outside the prompt:

boosted_keywords went from cleaning, exam, Invisalign, implant, crown, veneer, toothache, braces, cavity, extraction... to AC, air conditioning, cooling, heating, HVAC, refrigerant, compressor, condenser, no cooling, blowing warm... plus local place names like Pasco, Pinellas, Hillsborough and Tampa.
post_call_analysis_data, the fields extracted from every call after it ends, had a qualified field described as "Whether the caller has a legitimate dental appointment need." It also had patient_type, insurance_status and dental_anxiety. The fixed version replaced them with HVAC fields like urgency (with "emergency - no cooling or no heat" as an example) and an appointment_type whose examples are "A/C repair" and "Duct cleaning" instead of "Invisalign consultation."
analysis_summary_prompt said "Include the appointment type" in the dental wording and now asks for the service type.
LLM settings: model_temperature went from unset to 0, tool_call_strict_mode went from unset to true, and two tools got speak_after_execution: true.
version went from 7 to 10.

The prompt diff was a few lines. The rest of the object was most of the change.

How a dental agent ended up inside an HVAC agent

I did not have to guess for long. Five of the agent exports in that folder share the exact same last_modification_timestamp and the same version number, 7. They were cloned from one base. Three of them (Airheads HVAC, MC Kays Water Solutions and Memorial Family Dental) carry the identical list of twenty dental keywords and the identical eleven dental analysis fields.

The dental one is obviously where those came from. For the other two, the prompt was rewritten for the new business and everything else came along with the clone.

The MC Kays export, a water treatment agent, still has the same twenty dental keywords in that folder. I have not re-exported it since, so I cannot tell you from the files whether the version serving calls has them too. It is on my list, and it is the reason this post exists.

Why none of this showed up in testing

The natural way to test a voice agent is a conversation. You call it, play a homeowner with a broken AC, and listen to it book. That would have sounded right, because the prompt was right.

The two broken fields are invisible from inside a call:

Boosted keywords bias speech-to-text toward words you expect callers to say. Telling the transcriber that "crown," "implant" and "gum" are likely on an HVAC line is, at best, a wasted hint. At worst it nudges ambiguous audio toward the wrong word, and the model then answers the wrong word.

Post-call analysis runs after the caller hangs up. An extractor that is asked whether an HVAC caller has "a legitimate dental appointment need" has one honest answer. So every call, including the perfect ones, can come back qualified: false. The transcript looks great. The number the client sees in their dashboard says the agent qualifies nobody. I wrote about designing these fields properly in the post-call analysis post; this is the failure mode of not redesigning them when you clone.

The prompt is one field out of fifty-four

I counted. The agent object in that export has 42 top-level fields, and the LLM object nested inside it has 12 more. The prompt is one of them. It is 3,218 words long, so it gets all of my attention, but voice, keywords, interruption sensitivity, webhook URL, analysis fields, summary prompt, model and temperature are all part of what answers the phone.

A voice agent is a deployable artifact with a lot of config, and the prompt is just the source file I happen to stare at.

Treat publishing as a deploy

Retell separates the draft you edit from the version the phone number serves. Publishing is a deliberate step, and older versions stay in the history. Every one of my exports has is_published: false, which is correct: they are drafts, and I want it that way. The draft is mine. The published version belongs to the client.

What I do now, which I should have done from the start:

1. Sort changes by blast radius. Facts (hours, prices, a staff name) change what the agent says, so I test the question I changed plus one neighbor. Behavior (intake order, refusals, transfers) gets the full regression script, because a flow edit in one branch shifts the others. Tools, voice, model, keywords and analysis fields go in the top tier, because they affect every call and can fail without anything strange appearing in the transcript.

2. Keep a regression script with pass conditions. I have done this for some clients for a while. An older test file of mine for a UK estate agency's agent lists each case as a caller line plus success criteria, things like "routes to buyer-requirements node," "requests viewing details (date, time, name, phone)," and "uses British terminology (flat, letting, viewing)." The lesson from the Airheads diff is that pass conditions have to reach past the transcript. For the happy-path HVAC booking, the conditions now include "qualified comes back true" and "appointment_type is an HVAC value," not just "it booked."

3. Diff the whole export, not the prompt. This is roughly the script I ran to find all of this:

import json, sys

def flat(o, p=""):
    out = {}
    if isinstance(o, dict):
        for k, v in o.items():
            out.update(flat(v, f"{p}{k}."))
    elif isinstance(o, list):
        for i, v in enumerate(o):
            out.update(flat(v, f"{p}{i}."))
    else:
        out[p.rstrip(".")] = o
    return out

a = flat(json.load(open(sys.argv[1])))
b = flat(json.load(open(sys.argv[2])))
for key in sorted(a.keys() | b.keys()):
    if a.get(key) != b.get(key):
        print(f"{key}: {str(a.get(key))[:60]!r} -> {str(b.get(key))[:60]!r}")

It flattens both objects to dotted paths and prints every path that differs. Run it before every publish and the keyword list jumps out, because boosted_keywords.2 going from Invisalign to cooling is hard to miss.

4. Treat a clone as a checklist, not a copy. When I clone an agent for a new client now, I go field by field through the ones that carry business identity: name, prompt, keywords, every analysis field and its description, the summary prompt, default dynamic variables, webhook URL. If a field mentions the old business's vertical, it is a bug. A grep for the old vertical's words across the export catches most of it.

Ship quietly, watch, and decide the rollback before you publish

Publish when call volume is low and you are around to watch. Read every transcript from the first stretch of real calls, and for top-tier changes, confirm the downstream effects: the booking exists in the calendar, the contact exists in the CRM, the analysis fields came back with sane values.

Decide what "bad" means before you publish, not while staring at three ambiguous transcripts. Mine: any invented price or availability, any failed transfer, any confirmed booking that is missing downstream, or a visible drop in the booked rate against the previous week. Hit one and republish the previous version first, then investigate on the draft.

And keep a change log. Confession: Retell has a version_title field, and it is an empty string on every single export in that folder. Version 7 to version 10 tells me something changed three times. It does not tell me what or why. The fix is one line per publish: date, what changed, why, whether it stayed. I wrote about the replay side of this in There Is No Repro for a Phone Call.

What I took from it

The scary bugs in a voice agent are not the ones that make it sound wrong. Those you hear on the first test call. The scary ones live in config that never speaks: the transcription hints, the post-call extractors, the summary prompt. They make the agent sound fine and report nonsense, and the client sees the nonsense in their numbers before you do.

That is part of why I built VoiceDash: it puts every call, transcript and those analytics in front of the client on the agency's own domain, which is great when the analysis fields are right and very educational when they are not.

The longer, agency-focused version of this process (change tiers, the regression script, how to handle client change requests without losing your margin) is on the VoiceDash blog: How to update a live voice AI agent without breaking it.

If you ship LLM agents with a lot of config around the prompt, how do you review changes to the non-prompt parts? I am curious whether anyone has a better habit than a JSON diff and a grep.

Storing Retell Call Recordings for Client Portals: Expiring Links, Retention and Per-Tenant Access

Nabeel Hassan — Sun, 25 Oct 2026 10:43:20 +0000

Disclosure: I'm Nabeel, founder of VoiceDash, a client portal for Retell agencies. This post is about the plumbing, not the product, and every Retell behaviour below is taken from Retell's own docs (linked inline).

TL;DR: Retell recording URLs do not expire unless you opt in to secure URLs, and Retell keeps call data forever unless you set a retention period. If you show recordings to clients in your own portal, decide up front whether you proxy Retell's links or copy the audio into your own storage, scope every lookup by tenant, and make your retention match (or beat) the one you configured in Retell.

What Retell actually gives you

A finished call object (from the Get Call API or the call_ended / call_analyzed webhooks) carries up to four audio links:

recording_url: the mixed recording
recording_multi_channel_url: each party on its own channel
scrubbed_recording_url and scrubbed_recording_multi_channel_url: the same audio with PII intervals beeped out, when the agent uses the "Everything except PII" storage mode

There is also public_log_url, which deserves the same care as the audio.

Two agent-level settings decide how risky those links are: secure URLs and data retention. Both are off by default.

Problem 1: by default the links never expire

Retell's secure URL page is explicit: generated URLs for recordings and logs do not expire unless you opt in. If one of those links lands in a client's inbox, a CRM note or a support ticket, anyone holding it can play the call.

Opting in is per agent:

{
  "opt_in_signed_url": true,
  "signed_url_expiration_ms": 3600000
}

Leave signed_url_expiration_ms unset and you get 24 hours. The allowed range is 1 minute to 7 days.

Two details matter when you turn this on for an agent that already has history:

Files created before you opted in keep generating unsigned, non-expiring URLs.
If you later opt out, files created while the setting was on keep their signed behaviour; only new files go back to permanent links.

So "we enabled secure URLs" is not the same as "every recording link we have stored is now safe". Any URL you already persisted in a database or pushed to a CRM is still live. Treat stored Retell URLs as secrets, and plan to stop storing them at all (next section).

Problem 2: proxy or copy?

Once a client can log into your portal, you have two sane designs.

Proxy (store nothing, ask Retell each time). Your portal stores only call_id and tenant_id. When an authorised user presses play, your server calls Get Call, reads the fresh signed URL and redirects. With secure URLs on, Retell signs a new URL each time you request it, so a short expiry (5 to 15 minutes) works well. Upside: no audio storage, one source of truth. Downside: Retell's retention is now your retention, and every play is an API call.

Copy (own the audio). On call_analyzed, download the file and put it in your own bucket under a tenant prefix. Your portal mints its own short-lived links. Upside: you control retention, region and access logs, and Retell can be set to a short retention. Downside: you are now the one holding call audio, with everything that implies for consent and deletion requests.

You must copy if the agent uses "Everything except PII" or "Basic Attributes Only". The webhook docs say Retell deletes the raw recording, transcript and log files about 10 minutes after post-call processing ends, shortly after call_analyzed. The webhook still carries the full unscrubbed data, so it is your one window to grab it, and your one chance to decide whether you should keep the raw file or only the scrubbed one.

The ingestion handler

The copy design lives or dies in the webhook handler. Retell gives you three constraints worth designing around:

Verify X-Retell-Signature against the raw body, not re-serialised JSON (secure webhook docs). Only the API key with the webhook badge verifies.
The webhook times out after 10 seconds and is retried up to 3 times without a 2xx, so the same event can arrive more than once. Retell's own guidance is to key idempotency on event + call_id.
Acknowledge fast. Downloading a long WAV inside the request is how you earn duplicate deliveries.

import { Retell } from "retell-sdk";
import express from "express";

const app = express();
app.use(express.raw({ type: "application/json" }));

app.post("/retell/webhook", async (req, res) => {
  const raw = req.body.toString("utf-8");
  const sig = req.headers["x-retell-signature"];
  if (typeof sig !== "string" ||
      !(await Retell.verify(raw, process.env.RETELL_WEBHOOK_KEY!, sig))) {
    return res.status(401).end();
  }

  const { event, call } = JSON.parse(raw);
  if (event === "call_analyzed") {
    // Unique (event, call_id) row makes retries a no-op.
    const inserted = await db.webhookEvents.insertIfAbsent({
      event, call_id: call.call_id, received_at: new Date(),
    });
    if (inserted) {
      await queue.send("copy-recording", {
        call_id: call.call_id,
        agent_id: call.agent_id,
        url: call.scrubbed_recording_url ?? call.recording_url,
      });
    }
  }
  res.status(204).end();
});

The worker resolves agent_id to a tenant from your own table (never from anything the caller said), streams the file into s3://recordings/{tenant_id}/{call_id}.wav, and records the object key. Because of the 10-minute deletion window on the stricter storage modes, alert if the queue for this job backs up past a few minutes.

Choosing the scrubbed URL when it exists is a policy decision, not a technical one. For most client portals the scrubbed version is what a business owner should hear; keep the raw one only if the client has a documented reason and the consent to back it.

A schema that makes cross-tenant leaks hard

create table call_recordings (
  call_id       text primary key,
  tenant_id     uuid not null references tenants(id),
  agent_id      text not null,
  object_key    text,            -- null in the proxy design
  is_scrubbed   boolean not null,
  started_at    timestamptz not null,
  delete_after  timestamptz not null
);
create index on call_recordings (tenant_id, started_at desc);

Every read goes through one function that takes the session's tenant_id and the call_id and filters on both. If you use Postgres row-level security, put the policy on this table first: it holds the most sensitive rows in the system.

The playback endpoint then does three things in order: load the row with tenant_id from the session, write an access log line (who, which call, when), and return a link that expires in minutes. With the AWS SDK that last step is getSignedUrl(s3, new GetObjectCommand({ Bucket, Key }), { expiresIn: 600 }). Never put the object key or a Retell URL in the HTML that lists calls; mint the link only when someone presses play.

Retention: pick one number and make both systems obey it

Retell's data retention policy is per agent, set with data_storage_retention_days (1 to 730, or null for keep forever, the default). The dropdown offers 1, 3, 7, 30, 60, 90, 180, 365 and 730 days. Deletion runs daily and is permanent: recordings, transcripts, logs, dynamic variables and metadata go, and the call stops appearing in API responses.

That creates two failure modes:

Proxy design with retention on: the portal shows a call row whose recording silently 404s on day 91. Store delete_after and hide or label expired calls.
Copy design with retention on in Retell only: Retell deletes on day 30, your bucket keeps the file forever, and your client's "we keep recordings for 30 days" promise is false. Add a bucket lifecycle rule and a daily job that deletes rows past delete_after.

Which number to choose depends on the client's industry and the states their callers are in. That is a legal question, not an engineering one; I cover the consent side in Voice AI Call Recording Consent and AI Disclosure: The Compliance Checklist I Run for US Clients.

Checklist before a client sees a single recording

Secure URLs on for every client agent, with an expiry you can justify.
Old, unsigned URLs purged from your database, CRM notes and automations.
Proxy or copy decided per agent, and copy enforced for "Everything except PII" and "Basic Attributes Only" agents.
Webhook verified on the raw body, idempotent on event + call_id, acknowledged before the download.
agent_id to tenant_id mapping owned by you, checked on every read.
Playback links minted on click, expiring in minutes, with an access log.
One retention number per client, applied in Retell and in your own storage.

None of this is visible to the client when it works. It becomes very visible the first time a recording link travels further than it should.

Drafted with AI assistance and fact-checked against each vendor's docs.

Fine-Tuning Your Facts Is Hardcoding Them Into a Binary Nobody Can Read

Nabeel Hassan — Sat, 24 Oct 2026 21:35:34 +0000

Every engineer already knows not to hardcode a price into a compiled binary. You put it in config or a database, because it will change, because someone needs to see where it came from, and because you want to fix it without a release.

Then an AI feature shows up, someone asks "can we train it on our data?", and that instinct quietly disappears. Fine-tuning a model on your facts is exactly that hardcoded price, except the binary is a pile of weights nobody can read, diff or grep.

I build AI features for clients at Null Studio, and "train it on our data" is one of the first things we hear. The honest answer is usually that you do not need to, and if you did, it would not do what you expect. The way I think about it now is as a storage decision: there are three places knowledge or behaviour can live, and each has very different properties.

Three places to put things

The prompt is config. It is the text the model gets before it answers: role, rules, tone, output format, worked examples. Change it and the behaviour changes on the next request. It costs nothing to deploy and it is trivially reversible. A surprising share of "the AI is not doing what we want" bugs are fixed here with clearer instructions, a few examples and a defined output structure.

Retrieval is the database. RAG means searching your own content for the passages relevant to a question and handing them to the model alongside it. The model writes the answer; your documents supply the facts. The important property is that the knowledge lives outside the model. When an opening hour changes, you edit a document and the next answer reflects it. You can show the source. You can enforce permissions at query time, the same way you would on any other read path.

Fine-tuning is compiled code. You take a base model and train it further on example inputs and the outputs you want. The result is a modified model that leans toward those patterns without being told. That is genuinely useful for behaviour: a strict output format, a classifier with your own categories, a house style that is easy to show and hard to describe, or getting a smaller, cheaper model to match a bigger one on one narrow task.

What it is bad at is facts. A fine-tuned model might reproduce something it saw in training, or it might produce something that sounds exactly like it and is wrong, and it cannot tell you which.

Why facts do not belong in weights

Put it in engineering terms and the problems are obvious:

Every fact change is a release. New pricing means another training run, another test cycle, another deployment. With retrieval it is a row update.
There is no stack trace. A RAG answer can point at the paragraph it used. A fine-tuned answer came from the weights, so nobody can check where it came from.
There is no access control inside a model. Once restricted information is trained in, anyone who can call that model can potentially draw it out. "Do not reveal X" in a system prompt is not a security boundary. This gets worse once the model can take actions, which is why I give agents write access one verb at a time.
Bugs are hard to patch. If a RAG system answers wrong, you can usually find the bad or missing document. If a tuned model answers wrong, the fix is more data and another training run, with no guarantee the mistake is gone.

So the rule I use is short: facts go in retrieval, behaviour goes in the prompt, and fine-tuning is for behaviour the prompt cannot hold.

What my own voice agents actually do

I went and checked this against the client voice agents I have exported from Retell, because it is easy to preach a rule you do not follow.

All nine of them have knowledge_base_ids: []. No retrieval at all. Every one has a ## Context section in the prompt instead. The dental practice agent, for example, lists its services, business hours by day, office address and phone number right there in the prompt text, with the practice name pulled in through a COMPANY_NAME dynamic variable.

Is that a contradiction? I do not think so, and the reason is the useful part. For a small business, the entire fact sheet is a few hundred words. It fits comfortably in the prompt, it is plain text anyone on the team can edit, and a change ships the moment the agent is saved. That is still config, not compiled code. What I have never done is fine-tune a model on a clinic's hours.

The prompt stops being the right home for facts when the facts stop fitting: a catalogue, a policy manual, program rules that differ by location, or anything where different callers are allowed to see different things. That is where retrieval earns its complexity. The multilingual voice and SMS intake we built behind Fortell AI, which serves Community Action Agencies in more than 100 languages, has to answer against real program rules, and those belong in content the team can update, not in a prompt that keeps growing. I wrote about what goes wrong when one prompt carries everything separately.

When fine-tuning is the right call

It is a real tool. I would consider it when:

You have a narrow, repetitive task, like classifying messages into your own categories or extracting fields in a fixed shape, and prompting alone is not consistent enough at your volume.
You want a smaller, faster or cheaper model to match a larger one on that one task. That matters for cost and, on a phone call, for latency.
You have a style that is hard to specify but easy to demonstrate.
You have hundreds or more high-quality input and output pairs, reviewed by someone who knows the domain.

That last one kills more fine-tuning projects than anything else. Train on a small, messy set and the model faithfully learns the mess.

The order I actually work in

Build an evaluation set first. Real inputs, defined pass criteria. Without it you cannot tell whether any of the following steps helped. This is the part most teams skip, and I wrote up how I test outputs you cannot unit test.
Fix the prompt. Examples, output structure, rules. Measure.
Add retrieval for anything that depends on your own information. Measure again. Most quality problems at this stage turn out to be content problems: duplicate, outdated or missing documents. Your RAG demo probably works because someone picked the documents.
Try a different model. Sometimes that closes the gap with no training at all.
Only then consider fine-tuning, against a specific measured gap, with a target score and a comparison to the untuned setup.

These stack. A fine-tuned model still gets a prompt, and where facts matter it still gets retrieved context. Fine-tuning does not replace the other two; it sits on top of them.

The costs nobody puts in the estimate

The training run is usually the cheapest line item. The rest:

Data preparation. Collecting, cleaning and reviewing examples is real work, and most of it needs a domain expert, not just an engineer.
Base model churn. Providers update and retire models. A tune on a retired model has to be rebuilt on its successor and retested. Think of it as a dependency you cannot pin forever.
Lock-in. Some tunes only run on the provider that trained them. Ask where the tuned model lives, who can access it, and whether you can take it with you.
Data handling. Your examples go to whoever does the training. Check what they do with them, especially with personal or regulated data.
Ongoing evals. The tuned model still needs the eval set run on every change, and the set needs maintaining.

None of these are reasons to avoid fine-tuning. They are reasons not to reach for it by default.

Product features, same question

It is not just chatbots. For LectureNotes AI, which records lectures and produces summaries and outlines, what matters is faithfulness to what was actually said and a consistent outline structure. Faithfulness comes from grounding the output in the transcript; structure is a prompting job first. For Lifemaxxing AI, which builds personalised plans, personalisation comes from the user's own data passed in at request time, not from training a model per user.

In both, the useful question is not "should we train a model?" It is "where should this live so it stays correct and can change?" That is the same question you already ask about every other piece of state in your system.

If someone proposes fine-tuning first, ask them what it fixes that retrieval does not. If they cannot name a measured gap, you have your answer.

I am curious where others draw this line in practice. Have you shipped a fine-tune that actually beat a good prompt plus retrieval on your eval set, and what was the task?

Originally published on the Null Studio blog.

Most App Store Rejections Are Not About Your Code

Nabeel Hassan — Fri, 23 Oct 2026 21:34:42 +0000

Every engineer I know has the same first reaction to an App Store rejection: open the crash logs. That is almost always the wrong place to look. In my experience the build is fine. What failed is everything around the build: a login the reviewer could not get past, a form you filled in from memory, a calendar rule nobody on the team knew existed.

I managed App Store and Play Console releases as CTO at RAQTS, across mobile apps and a Unity platform, and I have been on the wrong end of enough review threads to stop treating store review as a final hurdle. Now I treat it like any other external system with an undocumented contract: probe it early, on purpose, and keep a checklist of the ways it fails.

Submit the first build as a test, not a launch

The most useful change I made was purely about timing. A rejection on launch day costs you a press date. A rejection two weeks earlier, on a build you submitted to see what happens, costs you nothing.

Both stores let you hold an approved build back. Apple lets you choose manual release after approval. Google Play lets you publish to an internal or closed track first. So I push a real build through review weeks early. A first app on a new account gets more scrutiny than an update, so I plan for one rejection and a resubmit and am pleasantly surprised when it does not happen.

The Google Play gate that is a calendar problem

If you create a new personal Google Play developer account, you cannot publish straight to production. You first run a closed test with at least 12 testers who stay opted in for 14 consecutive days, then apply for production access and answer questions about how you tested.

No engineering speeds that up. Discover it on launch week and you are two weeks late.

Two ways around it:

Use an organization account if there is a company. That needs its own verification (a D-U-N-S number and business details), which takes time, but it is not subject to the 12-tester rule in the same way.
Start the closed test the moment a usable build exists. With real testers, not twelve email addresses you control. Google asks what you learned from them, and that answer is easy to write when people actually used the app.

The general rule I took from this: developer accounts and signing keys get set up in week one, owned by the company, not by whichever contractor happened to create them.

The rejections I now plan for

1. The reviewer could not get in

If the app needs an account, give the reviewer one. Apple has a demo credentials field under App Review Information, and Google Play has an equivalent under App access.

The part engineers skip is the data. An empty account is technically a working login, but at RAQTS the product connected players and facilities, so a demo account with no facility, no sessions and no leaderboard history showed a reviewer a product that looked broken. The fix was a seeded demo account that exercised the main flows, plus review notes saying in two sentences what the app does and where the interesting part is.

I now think of the demo account as a fixture. It lives next to the other seed data, gets recreated by a script, and breaks the release checklist if it stops working. If a feature depends on hardware or a physical place, say so in the notes and attach a video, because the reviewer is not going to walk into a sports facility to test it.

2. It looks unfinished, or like a website in a box

Apple rejects apps that crash, show placeholder content, have dead buttons or link to "coming soon" pages. It also rejects apps whose main value is a wrapped website.

The engineering fix for the first part is boring: unfinished features go behind a flag, not behind a screen that says "soon". For the second, ask what the app does that the website cannot: push, offline, native camera or audio, widgets. If the honest answer is nothing, the reviewer has found something real.

3. No in-app account deletion

Both stores now require that if users can create an account, they can delete it. Apple wants deletion initiated inside the app. Google Play also wants a web link where people can request deletion without reinstalling.

"Email us to delete your account" does not pass. Build the real flow in version one: confirm, delete or anonymize server-side data, sign the user out. It is a day of work if you plan it, and an unpleasant week if you have to retrofit it into a schema where user IDs are foreign keys in fifteen tables.

4. Payments that skip the store

Digital content, features or subscriptions consumed inside the app default to the store's billing on both platforms. Physical goods and real-world services, like booking a court at a facility, are different and can use your own processor.

The rules have loosened in some regions and keep moving, so check current guidelines. What does not change is that the reviewer will tap every paywall, every "Upgrade" button and every link that leads to a checkout page. Decide the billing model before the first submission, because changing it later touches entitlement logic everywhere.

5. Privacy labels that do not match what the binary does

Apple's privacy labels and Google Play's Data safety form ask what you collect, why, and whether it is linked to the user. Those answers get compared against what the app and its SDKs actually do.

The mistake is filling the form in from memory of your own code. Your code is the small part. Analytics, crash reporting, attribution and ad SDKs all collect data on your behalf. So I generate the answer from the dependency list: every third-party SDK in the app, what each one collects, then the form. On Apple, also check that the privacy manifest covers any required-reason APIs your code or your dependencies use, or the upload gets flagged before a human ever sees it.

6. AI features without disclosure or controls

This is the category that has grown fastest, and it applies to most of what I build now. Apple's guidelines require apps to clearly disclose when personal data is shared with third parties, including third-party AI, and to get permission first. If the app sends someone's voice, notes or photos to an external model, say so in the flow where it happens, not only in the privacy policy.

Google Play's AI-generated content policy adds one more thing: users need a way to report offensive AI output from inside the app. A "Report this response" action wired to a backend log satisfies the intent, and it doubles as the cheapest eval dataset you will ever collect.

7. Store copy that overpromises

Screenshots must show the actual app. Descriptions cannot claim missing features, mention other platforms, or make health claims the app cannot back up. Anything near health or diagnostics should avoid "diagnose", "treat" and "cure" unless you have the regulatory standing for them.

After the first approval

Approval once is the start. For updates on a live product I lean on the stores' own safety rails:

Phased release on Apple and staged rollout on Google Play, so a bad build reaches a small slice of users first.
A minimum-version gate on the backend, because old builds stay in the wild for months. I wrote about that in The App Version You Shipped in March Is Still Calling Your API.
Builds from CI, not laptops, so signing and release do not depend on one person. My notes in Your Unity Pipeline Does Not Need Tests Yet apply almost unchanged to React Native and Flutter.
A calendar reminder for the target API level. Google raises the required Android target level every year, and an app that sits untouched eventually stops being offered to new users on newer devices.

At RAQTS the release pipeline was one of the first places I put my weight after stepping in as CTO, because a platform people use every day cannot afford a release that surprises anyone, the reviewer included.

The checklist I run before every first submission

Developer accounts created and owned by the company; Play closed test started early if required.
Seeded demo account recreated by script, two-sentence review notes, a video for hardware or location features.
No placeholder screens, dead buttons or "coming soon" links; unfinished work behind flags.
In-app account deletion, plus a web deletion link for Google Play.
Billing model decided for every paid feature.
Privacy labels and Data safety form generated from the real SDK list; privacy manifest checked.
AI data sharing disclosed in the flow, with consent and a way to report AI output.
Store copy and screenshots that match the build being submitted.

None of it is hard. It is just work that lives outside the codebase, which is exactly why engineers forget it.

What is the strangest rejection reason you have had from either store? I am collecting them for the next version of this list.

Originally published on nabeelbaghoor.com.

Every Tool Parameter Is a Question Your Voice Agent Has to Ask

Nabeel Hassan — Thu, 22 Oct 2026 21:36:13 +0000

If you have ever designed a REST endpoint, you already know most of what it takes to give a voice agent tools. Name it well, keep the parameters tight, validate inputs, return something useful. The difference is the client. Your endpoint is not called by a frontend developer reading your docs. It is called by a language model that is taking dictation from a stranger on a phone line, in real time, while that stranger waits in silence.

That one difference changes how I design every tool. I build production voice agents on Retell AI, with n8n and GoHighLevel behind them, and I went back through the nine client agent exports I keep in one folder to see what my tool definitions actually look like. They fall into two clear shapes. Comparing them taught me more than any guide I could have written up front.

What a custom function is, in API terms

In Retell, a custom function is four things: a name, a description, a JSON schema for the arguments, and a URL. During the call, the model decides it needs the tool, fills in the arguments from the conversation, Retell POSTs them to your URL, and whatever you send back lands in the model's context. Then it keeps talking.

So it is an RPC where the caller is an LLM and the arguments come from speech. Keep that framing, because every lesson below falls out of it.

Shape one: the schema that asked the model for IDs

Five of my nine exports book through a pair of tools called get_slots_from_ghl and create_appointment_using_ghl. The booking tool has eight parameters:

{
  "required": ["email", "first_name", "phone", "start_time", "end_time"],
  "properties": {
    "start_time": { "type": "string", "description": "Appointment start time in ISO 8601 format (e.g. 2026-04-15T14:00:00-07:00)" },
    "end_time":   { "type": "string", "description": "Appointment end time in ISO 8601 format" },
    "phone":      { "type": "string", "description": "Contact's phone in E.164 format (e.g. +17025551234)" },
    "calendarId": { "type": "string", "const": "REPLACE_WITH_CALENDAR_ID" },
    "locationId": { "type": "string", "const": "REPLACE_WITH_LOCATION_ID" },
    "title":      { "type": "string", "const": "Free Legal Consultation" },
    "first_name": { "type": "string" },
    "email":      { "type": "string" }
  }
}

(IDs replaced with placeholders, and descriptions trimmed.)

Read that as an API reviewer and a few things jump out.

The model is passing infrastructure IDs. calendarId and locationId are pinned with const, which helps, but the model is still the thing emitting them on every call. A const is a hint the model follows most of the time. It is not the same as the server simply knowing which calendar it owns.

The model is doing time math. It has to turn "Thursday around two" into an ISO timestamp with the right UTC offset, then compute an end time. The slots tool even needs an IANA timezone, and its description has to plead: "Never use abbreviations like PST or EST." When you have to write "never" in a parameter description, the parameter is in the wrong place.

The model is formatting a phone number. E.164 from speech is a parsing job. Parsing jobs belong in code.

None of this is broken, exactly. It mostly works. But "mostly" is doing a lot of work on a phone call, where a wrong offset books someone an hour off and nobody notices until they show up.

Shape two: only what a caller can say

The other four exports use a different trio: checkCalendar, bookCalendar and getWeekAvailability. Here is the booking tool from the roofing agent:

{
  "required": ["name", "reason", "requestedTime"],
  "properties": {
    "name":   { "type": "string", "description": "Caller's full name" },
    "reason": { "type": "string", "description": "Type of roofing service the caller needs (e.g. roof repair, full replacement, storm damage inspection, gutter installation)" },
    "requestedTime": { "type": "string", "description": "Confirmed inspection time in format 2026-04-15T09:00:00 (Eastern time, no offset suffix)" }
  }
}

Three parameters, and every one of them is something a human would actually say out loud. No calendar ID, no location ID, no end time, no offset. The time is local wall-clock time in the one timezone the business works in, and the backend owns everything else: which calendar, how long the slot is, what the offset is today.

getWeekAvailability goes further and takes no parameters at all.

This is the rule I now design by: every required parameter is a question the agent has to ask. If the caller cannot answer it, the model should not be filling it. Things the server can know (tenant, calendar, slot length, timezone) get resolved on the server, from the agent or the dialed number, never from the conversation. I wrote about the same idea for prompt variables in A Prompt With Placeholders Is a Function Signature Nobody Type-Checks, and the security version of it in Nobody Hacked It. The Lookup Tool Just Took a Phone Number.

The description is the only documentation the model reads

Compare the two descriptions for "check availability":

Shape one: "Check GHL calendar for available free consultation slots"
Shape two: "Check if a specific inspection time slot is available on the calendar. Pass requestedTime in format 2026-04-15T09:00:00 (Eastern, no offset). Call this after the caller names a specific day and time."

The first is a label. It tells the model what the tool is. The second is an instruction. It tells the model when to call it, what to pass, and implicitly when not to call it. The booking tool in shape two says "Call only after the caller has confirmed a specific time", which is a precondition written in English because there is no other place to write it.

That is the uncomfortable part of tool design for LLMs. In a normal API, preconditions live in code and the caller gets a 400. Here, the model decides when to call based mostly on prose. So the prose is the contract. I write descriptions the way I would write the docstring for a junior developer who is going to call this function under pressure: when, with what, and what must already be true.

Prose contracts also conflict with each other. My last post, All Nine of My Voice Agents Are One Giant Prompt, is about exactly that: a tool description and a global prompt rule disagreed, and the model had to pick one.

Your response body is a second prompt

This one surprised me when I reread my own n8n workflows. The mock workflows I test agents against do not return JSON for the model to interpret. They return plain sentences, and some of those sentences are instructions:

The availability mock returns the slot list prefixed with "ONLY EVER READ OUT MAX 2 AVAIABLE SLOTS PER DAY." (typo included, it is a real file)
The slots mock ends with "Present exactly two options to the caller."
The unavailable branch returns "The requested time is unavailable, tell the user to ask for a different day or time".

At first that looked hacky to me. Now I think it is correct, and I would just do it more deliberately. Whatever you return gets read by the model and, very often, read out loud. So:

Format for speech in the backend. "Tuesday at 10:00 AM and 2:00 PM" beats forty ISO timestamps. The model should never convert timezones while a caller waits.
Cap the output in the backend, not just the prompt. If the rule is two options, return two options. An instruction in the response is a backup, not the mechanism.
Make every failure speakable. "I could not reach the calendar" plus a next step is a valid response. A stack trace is not. The rule I enforce in the prompt: the agent only says "you're booked" if the tool returned a confirmation. A confident fake booking is the most expensive hallucination a voice agent can produce.

The two settings I shipped on defaults

Being honest about my own exports: every custom tool in all nine has speak_during_execution set to false and timeout_ms at 120000.

Two minutes is a fine timeout for a background job. On a phone call it is an eternity. If the calendar API hangs, the caller hears nothing for far longer than anyone will tolerate. What I recommend now, and what I am moving builds toward:

A short timeout on the Retell side, and a faster one inside the workflow, so the backend can return a speakable fallback before Retell gives up.
speak_during_execution on for anything that does real I/O, so the agent says something like "let me check that for you" instead of going silent.
One lookup per tool, and cached data for things that never change mid-call, like service areas or business hours.

The n8n side of that wiring is in An AI Voice Agent Is Just Two Webhooks.

Testing it

I test every tool three ways: call the endpoint directly with bad arguments (a weekend date, a malformed time), run scripted calls that should and should not trigger it (the "should not" calls catch weak descriptions), and break the backend on purpose to hear what the agent says. Silence or a fake confirmation means it is not ready. Why this is harder than normal testing is in There Is No Repro for a Phone Call.

The short version

Treat each tool as a public API whose client is a model taking dictation.
Only ask for what a caller can say. Resolve IDs, timezones and durations on the server.
Write descriptions as instructions with preconditions, because prose is the only contract the model reads.
Treat the response body as a prompt: speakable, capped, and honest about failure.
Do not ship a two-minute timeout on a phone call. I did, so you do not have to.

I run VoiceDash, a white-label portal where agencies show their clients every Retell call, recording and transcript, so a client can actually hear the agent check a slot and book it. The original, more agency-focused version of this guide is on the VoiceDash blog.

If you build tools for LLM agents, voice or not: where do you put preconditions today, in the description, in the schema, or in the server? I am curious how others handle it.