DEV Community: Ali Ahmed

How to stop an LLM from inventing numbers: evidence containment and numeral tracing

Ali Ahmed — Thu, 05 Nov 2026 22:37:12 +0000

You can't prompt an LLM out of inventing statistics, but you can make an invented number impossible to publish. In my Research-to-LinkedIn Automation project, which turns a researcher's new papers into LinkedIn posts, the model extracts and drafts, while deterministic code decides what counts as true. Every "stated" claim needs a verbatim quote that is actually in the paper, any number without one is thrown away, and every numeral in the final draft must trace back to the source. If code says no, the draft fails, whatever the model thinks.

Why are invented numbers the worst hallucination?

Posting about someone's research under their name means one wrong number is a public misquote of their own work. A vague sentence is embarrassing. A fabricated "37% reduction" is a correction request. So the design treats invented statistics as the highest-severity failure, and spends its strictest checks there.

Step 1: How do you make extraction honest?

Extraction is schema-driven (Zod plus the provider's structured-output mode), so there's no free-text parsing. Each claim carries a provenance:

STATED: the paper says it. evidence must be a verbatim quote.
INFERRED: a reasonable conclusion that isn't stated directly.
ABSENT: the paper doesn't say. The prompt tells the model this is correct and preferred over guessing.

Two structural choices do most of the work:

Ask only for what the source can support. If only the abstract is available, the schema doesn't even have fields for methodology, dataset or limitations. The model can't make up a methodology because there's nowhere to put one.
Confidence is computed, not self-reported. It's the share of claims that survive verification, not a number the model gives itself.

Step 2: What is evidence containment?

After the model answers, code checks every STATED claim:

function containsEvidence(source: string, evidence: string): boolean {
  const norm = (t: string) => t.toLowerCase().replace(/\s+/g, " ").trim();
  return norm(evidence).length > 0 && norm(source).includes(norm(evidence));
}

A STATED claim whose quote isn't in the paper is downgraded to INFERRED, and its fake evidence is cleared.
An important number whose quote isn't in the paper is discarded entirely. Numbers don't get a downgrade path.

It's plain substring matching after normalising case and whitespace. That's crude, and that's the point: a quote either appears in the paper or it doesn't, and no model gets a vote.

Step 3: How do you verify the finished draft?

The drafting model writes from the extraction, not the paper, so errors can creep back in. The draft then goes through two independent checks in parallel, and both must pass:

A. An LLM fact-checker labels each claim SUPPORTED, HEDGED_OK, UNSUPPORTED or CONTRADICTED, and flags overstatement and anything that reads like medical advice.

B. Deterministic checks, which override the LLM:

Numeral tracing. Every numeral in the draft (including forms like 12.5% or 3×) must appear in the verified important numbers or literally in the source text. One unmatched number fails the draft.
Causal-language guard. If a sentence uses "causes", "proves", "cures", "prevents" or similar, and the backing finding is phrased as correlation ("associated", "suggests", "may"), the draft fails for overstatement.

The rule from the design doc: if a check can be decided in code, code decides it. An LLM judge is good at nuance and bad at reliably noticing that 0.37 became 37%.

What doesn't this catch?

Being honest about the gaps matters as much as the checks:

A real number in the wrong place. Tracing confirms a number exists in the source, not that the draft attaches it to the right finding. The LLM pass is what catches that.
Paraphrase drift. A claim can be reworded until it no longer means what the quote supports. Containment checks the evidence, not the paraphrase.
Substring matching is literal. A quote broken by a hyphenation artefact from PDF extraction can fail even though it's real. That's the safe direction to fail in.

The general pattern

Let the model say "I don't know" (ABSENT), and reward it.
Make every claim carry verifiable evidence, and verify it in code.
Delete, don't downgrade, anything high-stakes that fails.
Use an LLM judge for nuance, with deterministic checks holding the veto.

Written by Ali Ahmed · Original article

Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice

Ali Ahmed — Thu, 05 Nov 2026 22:36:36 +0000

A Prometheus ServiceMonitor selects Services by labels. It never looks at annotations. In Grafana Pyroscope's Helm chart, a refactor nested prometheus.io/service-monitor: "false" under annotations instead of labels on the headless Service, so the exclusion silently stopped working. Prometheus scraped every pod twice, as job="pyroscope" and job="pyroscope-headless". My fix, grafana/pyroscope#5377 (in review), moves the key back under metadata.labels.

What's the difference between labels and annotations in Kubernetes?

Both are key-value metadata, but they do different jobs:

Aspect	Labels	Annotations
Purpose	Identify and select objects	Attach non-identifying information
Used by selectors (Services, ServiceMonitors, kubectl `-l`)	Yes	No
Typical content	`app`, `component`, `tier`	build info, tool configuration, long text

Some tools read annotations for configuration, like the old prometheus.io/scrape convention for annotation-based scraping. That's exactly why the two get mixed up. But the Prometheus Operator's ServiceMonitor uses a label selector.

How did the chart exclude the headless Service?

Pyroscope runs a normal Service and a headless one pointing at the same pods. Scraping both would duplicate every metric, so the chart's ServiceMonitor excludes Services marked with a label:

```yaml title="ServiceMonitor selector"
selector:
matchExpressions:
- key: prometheus.io/service-monitor
operator: NotIn
values:
- "false"




The headless Service carries `prometheus.io/service-monitor: "false"`, so it's left out. That only works if the key is a **label**.

## What broke?

A change that added support for custom `headlessAnnotations` (#4451) changed the YAML nesting in `services.yaml`. The `prometheus.io/service-monitor: "false"` line ended up under `metadata.annotations`.

Nothing failed. `helm template` still rendered valid YAML, the Service still worked, and Prometheus still scraped. It just scraped the same pod IPs twice under two job names (issue #5215): duplicate series, double the storage for those metrics, and dashboards that sum by job quietly counting double.

## How did I fix it?

* I moved the `prometheus.io/service-monitor: "false"` block so it renders under `metadata.labels` for the headless Service.
* I kept the `headlessAnnotations` feature from #4451 working.
* I bumped the chart version (2.1.2 → 2.1.3) and regenerated the rendered golden fixtures with `make helm/check`.

You can verify it with:



```bash title="Check the rendered headless Service"
helm template . -s templates/services.yaml --set serviceMonitor.enabled=true

and check that the key now sits under metadata.labels on the headless Service.

How do you catch this class of bug?

Test the selection, not just the rendering. A golden-file test that asserts "the headless Service has label X" would have failed when the key moved.
Watch for duplicate series. The same instance showing up under two job values is a classic sign of double scraping.
Review YAML nesting changes like logic changes. In Helm templates, indentation is the logic.

More upstream fixes, from LiteLLM to Node.js, are on my projects page.

Written by Ali Ahmed · Original article

Streaming LLM output through a Next.js proxy: why you only get stream_end

Ali Ahmed — Thu, 05 Nov 2026 22:31:47 +0000

When you proxy an LLM backend's Server-Sent Events (SSE) through a Next.js route handler, the usual failure is that nothing arrives until the end, and then everything arrives at once, or the client only ever sees the final stream_end. Something between the backend and the browser is buffering. The fix is a handful of specific things: the Node.js runtime, dynamic = 'force-dynamic', the right headers (Cache-Control: no-cache, no-transform and X-Accel-Buffering: no), passing bytes through as they arrive, and a client that parses SSE frames properly. I learned these building Turbo Alan Refiner, a multi-pass LLM text-refinement app (frontend source).

Why proxy SSE through Next.js at all?

Routing every backend call through /api/* route handlers avoids CORS, keeps tokens server-side, and lets you normalise the request. The cost is that your route is now a hop in a streaming pipeline, and every hop is a chance to buffer.

What does the route handler need?

```ts title="Route segment config"
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export const maxDuration = 300;




* `runtime = "nodejs"` gives full streaming support without edge-runtime limits.
* `dynamic = "force-dynamic"` stops Next.js caching or pre-rendering the route.
* `maxDuration = 300` lets long refinements outlive the default function timeout.

Then return a `ReadableStream` that forwards the upstream body as it arrives, with these headers:



```ts title="Streaming response headers"
return new Response(stream, {
  headers: {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    Connection: "keep-alive",
    "X-Accel-Buffering": "no",
  },
});

X-Accel-Buffering: no tells nginx-style proxies not to buffer.

no-transform matters more than it looks. A proxy that gzips the response has to collect bytes before it can compress them, which is exactly the buffering you're trying to avoid.

How should the proxy forward events?

Read with getReader() and decode with a streaming TextDecoder. Never await upstream.text(), which waits for the whole response.
Forward events, don't re-wrap them. If the upstream already sends data: {...} and your proxy prefixes data: again, the client receives data: data: {...}.
Send errors as events. If the upstream fails mid-stream, emit a final data: {"type":"error",...} frame so the UI can show it, instead of the stream just stopping.
Pass the client's abort on. Use an AbortController so closing the tab cancels the upstream request, rather than leaving an LLM job running for nobody.

Why does the client only see stream_end?

Usually one of three things:

Something buffered the stream (missing headers, the wrong runtime, a proxy in front), so all the events arrived together and the UI only rendered the last one.
The client parsed chunks, not lines. A network chunk can end in the middle of an event. Keep the incomplete tail in a buffer:

```ts title="Client: keep the incomplete last line"
buffer += decoder.decode(chunk, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? "";




3. **The client stopped too early.** Treat only true terminal events (`complete`, `error`) as the end. Keep processing everything already buffered before you close.

Be lenient about input, too: strip a doubled `data: data: ` prefix, and accept raw JSON lines as well as SSE-formatted ones.

## How do you debug it?

Open DevTools → Network → the streaming request → **EventStream**. If events show up one by one there but the UI updates all at once, the bug is in your client. If they all show up together, something upstream is buffering: check the headers, the runtime, and any proxy or CDN in front.

Related: [what a serverless cold start really is](https://aliahmed.si/blog/serverless-cold-starts-explained/) (long-lived streams and function timeouts run into the same platform limits).

---

Written by Ali Ahmed · [Original article](https://aliahmed.si/blog/nextjs-sse-proxy-streaming/)

Real-time KYC card verification over MQTT: a pub-sub pipeline explained

Ali Ahmed — Thu, 05 Nov 2026 22:31:11 +0000

You can build a real-time KYC (Know Your Customer) card verification pipeline from small independent services that talk through an MQTT broker: one publishes card data, one verifies it and publishes a result with reasons, and one stores and visualises the results. Each service only knows topic names, not the other services. That's the design of this project, which I built with two teammates on Mosquitto, Python 3.12, SQLite and a Flask dashboard. My part was the card client, the service that generates and publishes cards.

Why use MQTT for verification?

MQTT is a lightweight publish/subscribe protocol. Producers publish messages to topics, consumers subscribe to them, and a broker in the middle routes messages. For a verification pipeline that buys you:

Decoupling. The verifier doesn't know who sends cards, and the analyst doesn't know who verifies them. Any of them can be replaced, scaled or restarted alone.
Real time. Results flow as soon as each card is checked, with no batch job.
Delivery guarantees chosen per message, through QoS levels.

How does the pipeline fit together?

Card client publishes ID card records to kyc/card_data.
Verifier subscribes to kyc/card_data, validates each field, and publishes to kyc/result.
Analyst subscribes to kyc/result, stores results in SQLite, builds charts and exports CSV.
Dashboard (Flask) reads the analyst's data and shows live totals, approvals and the rejection rate.

How do you test a verifier properly?

With bad input, on purpose. The card client makes about 30% of the cards invalid, with malformed ID numbers, one-word names and expired cards, alongside valid ones. Each card has an ID, a name, an expiry date, a region and a card type. A pipeline tested only on clean data proves very little.

The client also retries its broker connection (up to 5 attempts) instead of dying on the first failure, because services start in any order.

What does the verifier check?

Every field, and it records why a card failed rather than just "rejected":

ID format: NNNN-NNNN-NNNN or NNNNNN-NNNN
Expiry: not in the past
Name: at least two words
Region: one of a known list (US, EU, ASIA, MEA)
Card type: one of the accepted types

A card passes only if every check passes. The published result carries the status plus the list of reasons, so a rejected card says "Invalid ID format, Card expired", not just "no". In real KYC, a rejection the operator can't explain is a support ticket waiting to happen.

What does QoS 1 mean for the design?

The services publish and subscribe at QoS 1, "at least once". The broker keeps sending a message until it's acknowledged, so nothing is lost, but a message can arrive twice, for example after a reconnect.

That has a consequence I'd design for in a production version: every consumer must be idempotent. A verifier that appends every message it receives would count a duplicated card twice. Storing results keyed by card ID (upserting, not appending) makes duplicates harmless. QoS 2 ("exactly once") avoids duplicates at the protocol level, but costs an extra round trip per message.

How does this compare to production identity verification?

This was a course project, deliberately simple: rule-based checks on structured fields. Production KYC, which I worked on at ShuftiPro, starts one step earlier, from a photo of the document. Computer vision finds and classifies the document, OCR reads it in the right script, face matching and liveness checks confirm the person, and only then do field checks like these run. The pipeline shape is the same, though: independent stages, explicit reasons, and test data that is deliberately bad.

Related: the Uber system design breakdown uses push messaging for the same reason, and the voice agent webhook post covers idempotency for repeated events.

Written by Ali Ahmed · Original article

How I got my first pull request merged into LiteLLM

Ali Ahmed — Thu, 05 Nov 2026 22:26:22 +0000

My first contribution to LiteLLM, the open-source LLM gateway, was PR #38591. It fixed a bug where content an agent had retrieved was compressed again and lost, which made agents loop. It was merged. If you want to make your first open-source contribution, the short version is: pick a project you already use, reproduce one specific bug, fix only that bug, and let the project's CI teach you its standards.

What was the bug?

LiteLLM can compress long context before sending it to a model. Large content is swapped for a compact reference, and the model can call a retrieval tool with that reference to get the full content back when it needs it.

The bug hit clients that run their own tool loop. When the agent asked for the full content, the retrieved text went back through compression and was replaced by the same reference again. The agent never saw the content it had asked for, so it either asked again (a loop) or answered from an empty skeleton.

The fix was to stop re-compressing content that had just been retrieved in those client-side tool loops.

How do you find something to work on?

I wasn't hunting for a famous repository. I was using LiteLLM, had some free time, and saw something I could pick up. That's the best starting point, because you already understand the problem as a user.

What worked for me:

Start from your own friction. A bug you've hit is a bug you can reproduce.
Read the issue tracker for "good first issue" or a recent regression, and check nobody has already claimed it.
Reproduce it locally first. A failing test or a minimal script is worth more than a long description.

What did the CI teach me?

LiteLLM is a big, fast-moving repository, and its CI/CD is thorough: linting, type checks, a large test suite and more. Going through those checks was one of the most useful parts. I picked up ideas I've since applied to my own projects, including at SmartLogics.

A big project's CI is a free code review of your habits. Let it fail, read why, and fix it properly rather than working around it.

How do you get a PR merged?

Keep it small. One bug, one fix, one test. Reviewers merge what they can understand quickly.
Explain the failure, not just the change. Steps to reproduce, what was expected, what happened.
Add the test that would have caught it.
Respond quickly to review comments.

What came next?

That first PR made the next ones easier. I've since contributed to Node.js (counting buffered HTTP output in bytes, not string length, so backpressure works), Symfony, Grafana Pyroscope, LibrePods, and NanoClaw, where I wrote a fix for an unauthenticated loopback webhook. The full list is on my projects page.

The best part wasn't the code. It was seeing that a small, careful fix to a project used by thousands of people is completely within reach. Here's to many more.

Written by Ali Ahmed · Original article

From 10 µm to 3 nm: how chips got so small (and why 3 nm isn't 3 nm)

Ali Ahmed — Thu, 05 Nov 2026 22:25:46 +0000

In 1971 Intel's first microprocessor, the 4004, was built on a 10 micrometre process and held about 2,300 transistors. Today's leading chips are marketed as 3 nm and hold tens of billions of transistors. The largest packages pass 100 billion. Shrinking transistors lets you fit more onto the same silicon, which means more parallel work, more cache and more specialised hardware. The catch: "3 nm" is now a product name, not a measurement.

How big is 10 micrometres?

A human hair is roughly 70–100 µm thick, so the 4004's features were already a fraction of a hair. 10 µm is 10,000 nm. On the node names alone, going from 10,000 to 3 is a shrink of more than 3,000 times, while transistor counts grew by a factor of millions.

Why do smaller transistors mean faster chips?

A transistor is an electronic switch. Every calculation, image and AI model comes down to billions of switches turning on and off. Smaller transistors give you three things:

More of them in the same area: more cores, bigger caches, dedicated AI and media engines
Less energy per switch, so more work per watt
Shorter distances for signals to travel

Is a 3 nm chip actually 3 nanometres?

No. Until the late 1990s the node name roughly matched a real physical dimension (the transistor gate length). Since then it has become a generation label. No feature on a "3 nm" chip is 3 nm wide; real gate pitches are several times larger. The industry now judges progress by transistor density and by performance and power at each new node, not by the name.

Why is shrinking so hard now?

At these sizes, engineers are fighting physics more than machinery:

Leakage: electrons tunnel through barriers that should block them, wasting power when a switch is meant to be off.
Heat: more switches in less space means more heat per square millimetre.
Variation: an error of a few atoms can ruin a transistor, and one bad transistor can ruin a chip.

How did the industry keep going?

By reinventing the transistor itself, more than once:

Planar transistors: a flat channel with the gate on top. This worked for decades.
FinFETs (from about 2011): the channel stands up as a fin, and the gate wraps it on three sides for better control.
Gate-all-around (GAA) / nanosheets (2 nm-class nodes): the gate wraps the channel completely.

Patterning moved to extreme ultraviolet (EUV) lithography, which uses 13.5 nm light to print features that older light sources couldn't.

The bigger picture

A modern chip isn't a scaled-down 1971 chip. Decades of materials science, quantum physics, chemistry, optics and computer engineering separate them. The story of computing is partly the story of humans learning to control matter with absurd precision, and we're still finding ways to go smaller.

Related: what's inside a GPU.

Written by Ali Ahmed · Original article

One phone number, two users: normalising WhatsApp sender IDs across Baileys and the Cloud API

Ali Ahmed — Thu, 05 Nov 2026 22:11:36 +0000

If a system can receive messages from the same person through two integrations, both must produce the same user ID, or that person silently becomes two users with different permissions. In NanoClaw, the Baileys WhatsApp integration identified a sender as 15551234567@s.whatsapp.net (a JID), while the WhatsApp Cloud API path used the bare wa_id, 15551234567. Roles and group memberships granted to one identity never applied to the other. My fix (#3070, in review) normalises the handle at one point and migrates existing data, merging the duplicates.

Why did one number become two users?

Path	Sender handle	Stored user ID
Baileys (native WhatsApp Web protocol)	`15551234567@s.whatsapp.net`	`whatsapp:15551234567@s.whatsapp.net`
WhatsApp Cloud API (via a chat SDK bridge)	`15551234567`	`whatsapp:15551234567`

Both paths use the channel type whatsapp, so the two IDs became two rows in users. Anyone moving a number from Baileys to the Cloud API, or running both at once, found that the approvals, roles and agent-group memberships they'd set up "didn't work". In fact they'd been granted to a different user.

The cause was one line: the code built the user ID directly from whatever raw handle the adapter emitted.

How should you normalise identifiers?

Pick one canonical form. Here that's bare digits, which the Cloud API already emits.
Normalise at one point, as early as possible. A normalizeWhatsAppHandle(channelType, handle) helper strips the @s.whatsapp.net domain and the optional :device suffix, and it's applied exactly where the user ID is built.
Scope it narrowly. It only touches WhatsApp user JIDs. Group JIDs (@g.us) and every other channel pass through unchanged.
Don't normalise identifiers that mean something else. The messaging-group routing ID is left as the raw JID on purpose, because routing depends on it. "Who sent this" and "where to reply" are different identities.

How do you fix data that's already split?

Normalising new messages isn't enough: existing installations already have both rows. A one-time migration:

Rewrites every user ID stored in JID form to the canonical form.
Cascades the rename through every table that references users: roles, agent-group members, DMs, and the optional approval and dropped-message tables.
Merges when a canonical row already exists (because the person also messaged through the Cloud API). Roles and memberships are unioned into the canonical row and the JID row is deleted, so nobody loses access in the merge.
Runs in a transaction, with the migration runner checking foreign-key integrity before it commits.

The general lesson

Any system with more than one way in (two messaging providers, email plus OAuth, a web app plus an API) needs a canonical identity decided at the boundary. Otherwise permission bugs show up as "it works for some people", which is the hardest kind to debug. For agent platforms the stakes are higher: identity decides who may approve what an agent does.

Related: localhost is not authentication, another NanoClaw fix.

Written by Ali Ahmed · Original article

What actually gets deployed to production? (Not your code)

Ali Ahmed — Thu, 05 Nov 2026 22:10:55 +0000

What gets deployed to production is not your source code but a build artifact: an immutable snapshot your pipeline produced from it, such as a Docker image, a compiled Next.js output, a Java JAR or a serverless package. Production servers never pull your repository or compile TypeScript on demand. They run the artifact. That one idea explains rollbacks, environment changes and most of CI/CD.

Part 4 of the Modern Deployments series. Previous: serverless cold starts.

What happens between git push and production?

A simplified pipeline looks like this:

git push
  → clone repository
  → install dependencies
  → compile / transpile
  → bundle and optimise
  → run tests
  → create build artifact
  → deploy artifact

The interesting part isn't the pipeline. It's what comes out of it.

What is a build artifact?

An artifact is the runnable output of a build, frozen at one point in time:

Stack	Typical artifact
Containers	Docker / OCI image
Next.js	`.next` build output, or a static `out/` folder
Java	JAR or WAR file
Serverless	Zipped function package
Static sites	A folder of HTML, CSS and JS

It's immutable: once built, it never changes. If you want different behaviour, you build a new one.

Why are rollbacks so fast?

When you roll back on Vercel or Render, the platform usually doesn't rebuild anything. It points traffic back at a previous artifact that already exists. Switching a pointer takes seconds, while a rebuild takes minutes.

The same idea explains preview deployments: every branch gets its own artifact and its own URL.

Why does changing a dependency need a new deployment?

Because you aren't editing a running application. A new dependency, a build setting, or (on many platforms) a build-time environment variable changes the inputs to the build, so you need a new artifact. Runtime configuration is different, and is the subject of the next post: where your environment variables actually go.

How should this change how you think about CI/CD?

Once this clicked for me, CI/CD stopped feeling like "automation" and started feeling like a manufacturing line. Every commit goes in, and a production-ready artifact comes out. Three practical consequences:

Build once, promote everywhere. The artifact that passed staging should be the exact artifact that reaches production.
Make builds reproducible. Lockfiles, pinned base images and pinned tool versions mean the same commit gives the same artifact.
Tag artifacts with the commit. When something breaks, you should know exactly which code is running.

We're not deploying repositories anymore. We're deploying artifacts, and everything before that is the process of making them.

Written by Ali Ahmed · Original article

Securing a voice AI agent's webhook: what my first version got right, and what I'd change

Ali Ahmed — Thu, 05 Nov 2026 22:06:21 +0000

A voice AI agent's webhook receives phone numbers, transcripts and summaries of real customer calls, so it needs three things: proof that the request came from the voice platform, a way to cope with the same call arriving several times, and side effects (like emails) that happen exactly once. HI Agent is a 24/7 AI phone agent I built for home-service businesses on Vapi. Its backend (source) gets those three mostly right. This post walks through how, and the two changes I'd make.

What does the backend do?

When a call ends, Vapi sends a POST to /webhook/vapi. The backend, Node.js and Express on Render:

Checks the shared secret Vapi sends in the x-vapi-secret header.
Upserts a call log in MongoDB Atlas, keyed by the call ID: phone number, duration, summary, transcript and the latest event type.
Emails the business owner the summary and transcript through Resend, but only when the call is completely finished.

How do you know the request came from Vapi?

Vapi lets you set a secret that it sends with every webhook. The middleware rejects requests before any work happens:

No header → 401 Missing webhook signature
Wrong value → 401 Invalid webhook signature
No secret configured in production → 500, so the endpoint fails closed instead of accepting everything

The secret lives in an environment variable on Render and in Vapi's webhook settings, never in the code.

Why upsert instead of insert?

Vapi doesn't send one event per call. It sends several (status updates and then an end-of-call-report), and any webhook can be retried. Inserting a row per request would leave you with duplicate, half-finished call logs.

So the handler uses findOneAndUpdate with upsert: true, keyed by call ID. Every event for a call updates the same document, and the stored status always shows the latest event type. Retries and repeats converge on one correct record.

How do you avoid sending five emails per call?

The email goes out only when message.type === "end-of-call-report", the event that carries the final summary and full transcript. Status updates are saved but trigger nothing. The owner gets one email per call, with complete information.

What would I change?

Reviewing it now, two things.

1. Compare the secret in constant time. The check uses providedSecret !== secret. A plain string comparison can stop at the first different character, which in principle leaks timing information. Over the internet that's hard to exploit, but the fix costs nothing:

```ts title="Constant-time secret check"
import { timingSafeEqual } from "node:crypto";

const a = Buffer.from(providedSecret);
const b = Buffer.from(secret);
const ok = a.length === b.length && timingSafeEqual(a, b);




**2. Remove the development bypass.** With no secret set and `NODE_ENV` not equal to `production`, the middleware skips validation entirely. That's convenient locally, but a staging deploy that forgets `NODE_ENV=production` becomes an open endpoint that writes to the database and sends email. Safer: always require a secret, and use a throwaway one locally. The repo already has a seed script that simulates a call, so local testing doesn't need the bypass.

A smaller improvement: the email step should also be idempotent. If Vapi retries the `end-of-call-report` after the email was sent but before the `200` arrived, the owner gets a second email. Recording `emailSentAt` on the call log and checking it first fixes that.

## The checklist for any AI agent webhook

* **Authenticate first**, in constant time, before parsing or storing anything.
* **Fail closed** when the secret is missing, in every environment.
* **Key storage by the platform's event or call ID** and upsert, so retries are harmless.
* **Trigger side effects on the final event only**, and record that they happened.
* **Treat transcripts as sensitive data:** they contain names, numbers and addresses.

The same lesson about authentication applies inside a single machine too: see [localhost is not authentication](https://aliahmed.si/blog/localhost-is-not-authentication/). For AI clients calling *into* your system, see [building an OAuth 2.1 MCP server](https://aliahmed.si/blog/oauth-mcp-server-crm/).

---

Written by Ali Ahmed · [Original article](https://aliahmed.si/blog/voice-ai-webhook-security/)

How does Uber show cars moving in real time? A system design breakdown

Ali Ahmed — Thu, 05 Nov 2026 22:05:45 +0000

A ride-hailing app like Uber shows cars moving in real time by combining three ideas: push messaging (the server sends updates only when something changes, instead of the app polling), spatial indexing with hexagons (Uber's open-source H3 library, so a request only looks at drivers in nearby cells instead of every driver), and prediction plus correction on the phone, which keeps the car icon gliding even when network updates arrive late.

Why not just poll for updates?

The naive design has the app ask the server every few seconds: "Any updates? How about now?" Uber's engineers have written about how this fell apart at scale. Most polling requests come back with nothing new, yet each one costs battery, bandwidth and server capacity, and on app launch they compete with the requests that matter.

So Uber moved to push. It built a real-time messaging platform, called RAMEN in its engineering blog, that keeps a connection open to each app and sends updates when they exist. That platform decides:

When to send: only on meaningful change. If a driver moves two metres, the rider doesn't need a message.
What to send: a small diff, not the whole trip state.
How to send it: over a long-lived connection (Server-Sent Events at first, later gRPC streams).

How do you find nearby drivers among millions?

Computing the distance from a rider to every driver is fine with 100 drivers and hopeless with millions.

Instead, the world is divided into cells. Uber open-sourced its grid as H3, a hierarchical hexagonal index. Every latitude and longitude maps to a cell ID at a chosen resolution. Drivers report their location, and the system keeps track of which cell each driver is in. To find drivers near a rider:

Look up the rider's cell.
Take that cell plus its ring of neighbours (k-ring).
Check only the drivers in those cells.

The search no longer scales with the total number of drivers on the platform, only with the number nearby.

Why hexagons instead of squares?

In a square grid, a cell has 4 edge neighbours and 4 corner neighbours at different distances, so "one ring of neighbours" is a lumpy shape. In a hexagonal grid all 6 neighbours sit at the same distance from the centre. Rings of hexagons approximate circles much better, which makes "everything within roughly X km" simple and fair.

How do cars move smoothly with patchy data?

Real networks drop packets and phones report GPS irregularly. If the app just jumped the icon to each new position, cars would teleport. Two standard techniques fix this:

Prediction (dead reckoning): between updates, estimate where the car should be from its last speed, heading and the road it's on, and animate it there.
Correction (filtering): when a real update arrives, blend the prediction with the measurement instead of snapping. A Kalman filter is the classic tool: it weighs each source by how much it can be trusted.

Snapping positions to the road network (map matching) removes the rest of the jitter.

What's the lesson?

"A car moving on a map" turns out to be real-time distributed messaging, event filtering, spatial indexing, predictive modelling and network optimisation, all working together quietly. The simplest user experiences are often powered by the most complex systems.

If you like system design breakdowns, read what a serverless cold start really is.

Written by Ali Ahmed · Original article

Your app doesn't run on a server anymore. It runs on a desired state.

Ali Ahmed — Thu, 05 Nov 2026 21:43:27 +0000

On a modern platform your application isn't attached to a particular server. You deploy a workload and declare a desired state ("keep three copies of this running"), and an orchestrator decides which machine runs it, moves it when a machine fails, and starts more copies when traffic grows. Servers still exist. They just stopped being the thing you deploy to.

Part 2 of the Modern Deployments series. Part 1: how free tiers work.

Is "my app runs on an EC2 instance" wrong?

I once heard someone say, "My app is running on an EC2 instance." It's not wrong. It's just not the whole story anymore.

Take a typical deployment on Vercel, Render or Railway. You push code to GitHub. A build pipeline packages the application, often into a container image. The image goes to a registry. The platform schedules it somewhere in its fleet. At no point do you pick a machine, SSH into it, or even learn where your app ended up. That's intentional.

Why did containers change deployment?

Unlike a virtual machine, a container doesn't carry its own operating system. Containers on a host share the host's Linux kernel and keep their own isolated processes, network and filesystem (through kernel features called namespaces and cgroups).

That makes them light: they start in seconds or less, and many fit on one machine. A platform can move, restart or scale your app quickly because there is so little to boot.

What happens when the machine fails?

Ten years ago, a dead host probably meant downtime and a pager going off.

Today an orchestrator like Kubernetes notices that the actual state (two copies running) no longer matches the desired state (three copies) and schedules a replacement on a healthy node. Managed platforms run a similar control loop. You just never see it.

That loop is the important idea:

You declare what should be true.
The control plane keeps watching what is true.
When they differ, it acts until they match again.

What does this change for developers?

Treat instances as disposable. Local disk and in-memory state can vanish when your app is moved. Put state in a database, object store or cache.
Health checks matter. The orchestrator decides what is "healthy" from your health endpoint, so make it honest.
Scale is configuration, not hardware. Replicas and autoscaling rules replace buying a bigger box.
Which CPU runs your code is an implementation detail, and it can change during the day.

This is also why cold starts exist: if the platform can place your workload anywhere, sometimes it has to create that place first.

Next in the series: what is a serverless cold start?

Written by Ali Ahmed · Original article

What is a serverless cold start, really?

Ali Ahmed — Thu, 05 Nov 2026 21:42:51 +0000

A cold start is the extra time a serverless platform spends creating an execution environment before your code can handle a request: finding compute, starting an isolated sandbox, booting the language runtime, loading your code and running its top-level initialisation. A warm request reuses an environment that already exists, which is why the first request after idle can take 2–3 seconds and the next one about 100 ms.

Part 3 of the Modern Deployments series. Previous: your app runs on a desired state.

Did the function "go to sleep"?

That's the usual explanation, and it's a decent mental model, but it isn't what happens. Nothing sleeps. The platform reclaims the environment. On a traditional server, your process is already running: memory allocated, runtime initialised, waiting for the next request. Serverless flips that. If a function hasn't been invoked for a while, the platform takes its CPU and memory back, because there is no reason to keep resources reserved for idle code.

What happens during a cold start?

When the next request arrives, your handler is not the first thing that runs. The platform has to:

Find available compute in its fleet.
Create an isolated execution environment: a container, a micro-VM (AWS Lambda uses Firecracker), or a V8 isolate on edge runtimes.
Start the language runtime: Node.js, Python, the JVM.
Load your deployment artifact. See what actually gets deployed.
Run module-level initialisation: imports, globals, SDK clients, config loading.
Invoke your handler.

All of that is the cold start. On a warm invocation, steps 1–5 are skipped.

Why does bundle size matter?

Steps 4 and 5 grow with your code. The more you ship, and the more work you do at import time, the longer it takes before the first line of your handler runs. A function that imports a huge SDK to call one endpoint pays for the whole SDK on every cold start.

How do you reduce cold starts?

Ship less code. Tree-shake, bundle, and drop dependencies you don't use.
Initialise lazily. Create expensive clients on first use, not at import time. Reuse them across warm invocations by keeping them in module scope.
Pick a lighter runtime where you can. Isolate-based edge runtimes start much faster than full containers, but support fewer APIs.
Keep instances warm with provisioned concurrency or minimum instances. This works, but you're now paying for idle compute again, which is the thing serverless was meant to avoid.
Put the database close by. A cold start next to a slow database connection handshake feels twice as bad.

Are cold starts a bug?

No. They're the price of creating compute on demand instead of paying to keep it running. Once you understand the lifecycle, the behaviour stops being surprising, and you can decide case by case whether to optimise it away or pay for warmth.

Next: what actually gets deployed to production?

Written by Ali Ahmed · Original article