A client wanted their management panel's AI assistant to answer questions like "what did this contact write me" or "who replied about a certain topic", pulling from thousands of conversations held on an external CRM. Between the idea and the first correct answer there was a bundling error that came back three times, a cursor that never advanced, and a phone number that pointed to the wrong contact.
Here's the full path, bugs included.
What already existed, and what was really missing
The client's management panel already had a solid integration with the external CRM used to handle WhatsApp, SMS, and email conversations with contacts: reading sales pipelines, opening chat with bubbles and attachments, sending replies. All reusable infrastructure, already in production for a while. One specific piece was missing: no function queried the CRM's full contact directory — only opportunities inside a specific pipeline — and, more importantly, there was no way at all for the panel's AI assistant to read past conversations.
First rule, before writing a single new line of code: reuse what already worked. The function that read multiple pipeline stages in parallel, the modal that opened chat with full history, the function that sent messages — all proven and already in use. The one genuinely missing piece was a function able to search the entire contact directory, with text search and pagination, which the existing functions (built for single pipelines) didn't do.
A search that has to run across all contacts, not just the 30 loaded on screen
With tens of thousands of contacts in the directory, loading them all client-side to filter isn't an option. The correct approach is server-side search: on every keystroke (debounced by 400ms to avoid hammering the API), the backend function forwards the query to the CRM's own search engine, which runs it against the entire archive — name, phone, email — not just the results already shown on screen. What the user sees, 30 contacts at a time with a "load more" button, is only the current page, not the search scope.
A real limitation worth knowing: a third-party CRM's search is almost always a "contains" match, not fuzzy — it doesn't tolerate typos or a swapped first/last name. Worth keeping in mind both in the UI and, later in this story, in a piece of AI logic that broke for exactly this reason.
The request that changes the scale of the project
At that point came the request that changed the nature of the work: the client wanted every contact and every conversation to also persist in their own database, kept up to date over time, queryable by the panel's AI assistant — no longer just fetched on the fly from the CRM when needed. With nearly 48,000 contacts and a chat history growing daily, this stopped being one more feature and became an architecture decision.
Why Firestore and not Realtime Database
The first decision, and the one worth getting right: Realtime Database retransmits the entire node on every write, to every connected client. With tens of thousands of contacts and conversations that keep growing, a node like contacts/{id}/messages would have become a bandwidth problem from the very first wave of writes. Firestore, by contrast, supports indexed queries, subcollections that grow without rewriting the parent document, and — the decisive reason here — native vector search, exactly what's needed for semantic search across all conversations at once, without adding a separate vector database to pay for and manage.
The data model is one document per contact (profile, tags, last sync) with a messages subcollection for history — not an array inside the document itself. A contact with hundreds of messages would soon hit the one-megabyte-per-document limit, and more importantly every new message writes a single new document instead of rewriting the entire history each time.
The batched backfill, and the first obstacle: "Cannot find package firebase-admin"
Writing and querying vector fields on Firestore reliably means using the official firebase-admin SDK, not the raw REST API. That runs against a project convention — zero npm dependencies in the existing Netlify Functions — and the exception made itself felt on the very first deploy:
Cannot find package 'firebase-admin' imported from /var/task/...
despite the package being correctly declared in a dedicated package.json.
The cause is a function-bundling problem, not a code problem: the bundler simply wasn't packaging the dependency into the final zip.
-
First attempt — explicitly setting the modern bundler via
node_bundler = "esbuild"in the config. Same error. -
Second attempt — explicitly marking the package as "external" so it wouldn't be bundled (
external_node_modules), the documented standard fix for exactly this case. Again the identical error, a sign the problem was no longer bundling but the install step itself during the build.
The definitive fix, when the "clean" routes aren't enough. No more half-measures: download the dependencies ahead of time and commit them already installed into the repository (so-called "vendoring" of
node_modules), so the platform finds them already there instead of having to install them during the build.
The interesting practical detail: the client works entirely from a browser, with no local terminal. The solution was a browser-accessible cloud dev environment (a "codespace") with a real built-in terminal: npm install inside the functions folder, then commit and push the generated files straight from the interface — zero commands to install on the person's own computer. With node_modules physically present in the repository, the next deploy went through on the first try.
The cursor that never advanced (and the hidden cost of redoing the same work)
With bundling solved, actually importing nearly 48,000 contacts can't be a single call: it has to run in batches, repeated by a function scheduled every few minutes, with a progress cursor saved on Firestore to resume exactly where it left off. The first real run revealed a subtler problem: the cursor stayed null forever. The configured batch (25 contacts) simply couldn't finish within the function's time budget, so every execution restarted from the same initial contacts — not only failing to advance, but also re-paying for embedding every message that had already been embedded before, a silent and entirely avoidable cost.
The fix had two parts: shrink the batch size to something that genuinely completes within the available time budget, and — more importantly — check which messages already had an embedding saved from a previous run, skipping them instead of regenerating them. With that fix, the cursor finally started advancing, and a rough estimate based on the observed pace put the full historical import at about a day and a half.
Real-time messages: a "customer replied" trigger that isn't enough on its own
For new messages, the CRM's automation offered a "customer replied"-style trigger — but with a non-obvious limitation: the variables available in the webhook body exposed the contact's identity, not the message text or its real timestamp. And, more importantly, no equivalent trigger existed for "the operator replied": automation engines of this kind react to customer actions, not to internal team actions.
The sturdier fix wasn't chasing a trigger that probably doesn't exist, but changing how the available one gets used: the webhook no longer carries the message text, only the contact's identity, used as a signal meaning "this contact was just active, resync it now." The function it calls re-reads the entire fresh conversation from the same API the backfill already uses — which includes both inbound and outbound messages in the same fetch. One available trigger, full coverage on both directions. A second, lighter webhook, attached to contact-created/updated triggers, covers tag or profile changes that happen without a new message.
Dirty data in the pipeline: system events mistaken for messages
An unplanned side effect: imported messages started including system events like "opportunity deleted", generated by the CRM whenever an opportunity changes stage — not text written by either side of the conversation. The fix, minimal but needed in three separate places (the real-time webhook, the shared sync function, and the backfill job already live in production), was to explicitly filter these events by type before saving them or generating an embedding for them.
A practical reminder about this kind of change: an import job that's already live and running should be fixed with surgical patches, not rewritten for code cleanliness midway through — the risk of introducing a new bug while it's processing tens of thousands of records far outweighs the cosmetic benefit.
Searching by phone, searching by name — and a silent ambiguity
The first real test of AI search immediately exposed the limit of semantic search: searching for a phone number with the vector engine doesn't work, because the number doesn't appear in the meaning of the message text. What was needed was a direct identity lookup, not a content search — two different tools for two different questions.
The first attempt at a phone lookup produced a sneakier bug: a ten-digit local number that happened, by pure coincidence, to start with the same digits as the country code was mistaken for an already-complete international number — and the search stopped at the first match found, returning the wrong contact with no warning at all. The correct fix wasn't "guess better" but checking every plausible variant in parallel (with and without the country code) and, if different contacts turn up, explicitly flagging the ambiguity instead of picking one at random. The same principle was then extended to a direct lookup by exact name and by tag, both with an automatic, silent fallback to semantic search whenever the direct match finds nothing.
The wrong time zone in the AI's answer
An easy detail to underestimate: timestamps were correctly stored in UTC, as they should be, but were passed to the language model as-is, leaving it to mentally convert them to local time in its final answer. The result was silently off by two hours, with no error to catch — a message from 1:22 PM local time was presented as 11:22 AM.
The reliable fix wasn't a more insistent prompt instruction, but removing that calculation from the model entirely: converting the timestamp server-side to the correct time zone (automatically handling daylight saving as well) before injecting it into the context, and explicitly stating that the time received is already local, so the model doesn't attempt the conversion again on its own.
How long an AI answer should be
One last problem, more mundane but frequent: some answers with long lists of messages were getting cut off halfway, with the last item — often the most recent and the most relevant one — missing. The cause was simply a response token budget too low for the language model. The fix has three layers, not just one: raise the maximum response token budget, order lists from most recent message to oldest (so any future truncation drops the least relevant data, not the freshest), and explicitly instruct the model to stay compact in its formatting, so it doesn't waste response headroom on unnecessary decoration.
What I take away from this
- Reusing existing infrastructure saves real time, but changing scale (from a few on-the-fly reads to tens of thousands of persistent, queryable records) is always an architecture decision, not just a code one — worth stopping to decide explicitly before writing the first line.
- A dependency exception needs a plan B ready in advance. If standard bundling doesn't work on the first try, manually vendoring dependencies is a slow but reliable path, and it's better to know that before deploy day, not during it.
- Identity search and semantic search are two different tools, not one tool with two modes. Using the wrong one produces a result that looks plausible but is silently incorrect — the hardest kind of bug to spot.
- The most annoying bugs were never the "big" decisions — Firestore or RTDB, vector or text search — but the small, silent details: a time budget too tight, a time zone left unconverted, a token limit set too low. None of these throws a visible error: they just produce a wrong answer that looks right.
I write about Firebase, Netlify, PWAs, and AI applied to real production projects on roversia.it/blog. This post was originally published there.
Top comments (0)