Not because they were bad. Because we counted, and we were running three retrieval
substrates for one product: a managed vector RAG service, a hosted graph database, and
the application's own Postgres. Three vendors. Three sets of credentials. Three failure
modes to reason about at 2am. Two of them billing us to embed the same text twice.
This is the write-up of why both are going, why one data warehouse is replacing both, and
the four things that cost us the most time to discover. If you're standing up RAG on
BigQuery, sections 4–7 are the ones that will save you a week.
The honest status: the graph migration is built. The RAG half is a decision we've
committed to measuring, with criteria written down in advance. I'll show you those
criteria at the end, including the outcome where we delete the whole thing instead.
1. Why we're leaving a managed RAG service
The seductive thing about managed RAG is that chunking, embedding, indexing and retrieval
arrive as one API call. The expensive thing is that when retrieval goes wrong, you own the
symptom and not the machinery.
Ours went wrong like this. A workspace with ~500 ingested documents was surfacing one to
three of them, over and over, no matter what was asked. Not low-quality answers —
starved answers, grounded in a handful of sources while 497 sat there unread.
The root cause is a shape you should go check for in your own stack right now:
One shared corpus for every tenant. Retrieve the global top-50 across all tenants.
Then filter down to the current tenant, client-side.
That's post-filtering. The top-50 is a lottery drawn from everyone's data, and your
tenant wins however many tickets it wins — which, on a shared corpus, is a handful.
Ingesting more documents does not help. It makes it worse, because it adds tickets to a
draw you're already losing.
The obvious fix — server-side metadata filtering — was a closed door at our configuration:
metadata search was only available in a beta API surface and explicitly unsupported in
serverless mode. Not an oversight on our part; a documented limitation.
The next fix — pass explicit document IDs to scope the search — collided with something
even more mundane. The serverless vector backend caps the request payload at ~10 KB, and
the query embedding alone was ~9.6 KB of it. That left room for roughly fifteen IDs.
Fifteen, out of five hundred. Which is how you end up maintaining ~150 lines of ID
batching, adaptive request-splitting, and an all-or-nothing fallback that only fires at
exactly zero results — so "three documents out of five hundred" was being counted as
success by our own code.
Add a regional quota of about 25 requests per minute, shared across import, list and
delete, and the picture resolves: we weren't tuning a retrieval system. We were building
scaffolding around one.
What we lose by leaving, stated plainly: native reranking, corpus lifecycle
management, and grounded generation as a single call. The first one is real and we have to
rebuild it. More on that in section 8.
2. Why we're leaving the graph database
This one is less dramatic and more useful, because the graph database was working
fine.
There was no incident. No performance cliff. It was a paid third-party service living
outside the cloud project everything else runs in, with its own credentials, its own
uptime, its own invoice. Consolidation was the whole motivation, and I'd rather say that
plainly than invent a technical grievance after the fact.
But auditing it did surface something worth generalising. Here was our graph model:
(:Resource)-[:HAS_CHUNK]->(:Chunk {text, embedding})
(:Resource)-[:MENTIONS]->(:Entity {name, type})
(:Entity)-[:RELATED {type}]->(:Entity)
And here is what the read path actually did: vector-search the chunks, join to the
resource, collect one level of entity relationships. That's it.
No variable-length patterns. No *1..5. No path finding, no centrality, no traversal of
unknown depth. A vector search and two joins.
So the question stopped being "how do we migrate our graph database" and became:
If none of your Cypher contains a variable-length pattern, what are you paying a graph
database for?
A graph database earns its price on traversals whose depth you don't know in advance —
that's the query class where the index-free adjacency actually beats a join. One hop is
not that query class. One hop is a join, and every database does joins.
If you have a knowledge graph in production, go grep your queries for * between two
node patterns. The answer is genuinely informative either way. If you find them, keep your
graph database — this post doesn't apply to you. We didn't find any.
3. Why one substrate, and why a warehouse
Once both moves are on the table, the third fact becomes impossible to ignore: we were
running two complete chunk → embed → search stacks over identical text.
The ingestion service scraped a page, wrote it to object storage, and imported it into the
managed corpus, which chunked and embedded it internally. Then the application read the
same text back out of object storage, chunked it again with its own splitter, and
embedded it again. Same source. Two chunkings. Two embedding passes. Two bills. Two things
that can drift apart silently and give different answers to the same question.
Neither vendor was wrong. The seam between them was.
Collapsing both into one BigQuery dataset gives one text corpus, one chunking, one
embedding model, one query surface — and, structurally, it can't starve:
-- pre-filter, then top-k
SELECT ... FROM VECTOR_SEARCH(
(SELECT chunk_id, text, embedding
FROM v_chunk JOIN v_chunk_embedding USING (chunk_id)
WHERE workspace_id = @ws), -- <<< this runs FIRST
'embedding',
(SELECT @qvec AS embedding),
top_k => @probe,
distance_type => 'COSINE')
The tenant predicate is inside the search subquery. The k nearest are computed within
the tenant, not filtered down to it afterwards. The starvation failure mode from
section 1 is not mitigated here; it is unrepresentable.
The cost model that makes this viable: in a warehouse, query cost is bytes scanned.
A 5,000-chunk workspace at 768 dimensions scans roughly 30 MB — about $0.0002 per
query — and brute-force vector search is competitive below a few thousand rows anyway.
And the alternative we turned down, because a decision post that only lists the
winner's virtues isn't a decision post: Postgres + pgvector was the better fit on the
merits. The database was already deployed, already credentialed everywhere, MERGE maps
1:1 onto ON CONFLICT, HNSW works at any row count, and latency stays in milliseconds
instead of BigQuery's 0.5–2s floor. We went the other way for warehouse-side batch
embedding and analytics colocation. It's a real trade, not a slam dunk, and if latency is
your binding constraint you should probably pick differently.
4. Trap one: the embedding parameter that is accepted and ignored
This is the most dangerous thing in the post, so it goes first.
Newer embedding models have deprecated the task_type parameter — and the backend
silently ignores it. You pass RETRIEVAL_DOCUMENT. You get no error. You get no
warning. You get raw-text embeddings with no task optimisation, and retrieval that is
quietly worse than it should be with nothing to debug.
Degraded quality with a clean log is the worst failure shape there is. A crash you can fix
in an afternoon.
The task now goes in the text, as a prefix, and it stays deliberately asymmetric:
# one shared constant — two call sites that must never drift
DOC_PREFIX = "task: retrieval_document | "
QUERY_PREFIX = "task: retrieval_query | "
-- ingest
SELECT * FROM AI.GENERATE_EMBEDDING(
MODEL `ds.embed_model`,
(SELECT chunk_id, 'task: retrieval_document | ' || text AS content FROM v_chunk ...),
STRUCT(768 AS output_dimensionality));
Two rules we now enforce with unit tests, because production won't tell you:
- Ingest and query must use the same model at the same dimensionality. Break it and you don't get an exception, you get meaningless neighbours.
- Ingest and query must use different task prefixes. That asymmetry is the entire point of task types; symmetry silently costs you quality.
Test that no code path passes TASK_TYPE at all. It's three lines and it protects you
from a bug that has no other detector.
5. Trap two: the warehouse's model endpoint name is a different surface than the model
The embedding model went GA on the ML platform in April. As of docs updated at the end of
July, the warehouse's remote-model endpoint list still named it with a -preview
suffix.
These are two independent surfaces and the warehouse's lags. A wrong endpoint string
doesn't degrade — it fails outright at CREATE MODEL, which is at least honest, but it
will burn an afternoon if you're confident the model is GA and therefore assume the
endpoint string matches.
Do this before you build anything on top of it:
CREATE OR REPLACE MODEL `ds.embed_model`
REMOTE WITH CONNECTION `proj.region.conn`
OPTIONS (ENDPOINT = 'model-name'); -- rejected? try 'model-name-preview'
Then put whichever one worked in config and never infer it again. Related gotcha from the
same family: a globally-available model can expand to a multi-region path that conflicts
with your dataset's region. That reads like an IAM error and isn't one.
6. Trap three: mutating DML doesn't queue past its limit — it fails
This is the most portable idea in the post, and it applies to anyone doing
fire-and-forget background writes into a warehouse.
Everyone knows the old per-day DML quota is gone. Here's what replaced it:
| Operation | Concurrency | Queue | Past the queue |
|---|---|---|---|
UPDATE / DELETE / MERGE
|
2 per table | up to 20 | jobs FAIL |
INSERT |
10 (after the first 1,500/table/24h run immediately) | up to 100 | queued |
Our ingestion is background tasks fanned out across many tenants. A large batch plus a
concurrent reconcile sweep can plausibly stack more than 20 MERGEs on one table — and
those don't wait their turn, they error. Recently-written rows aren't reliably mutable
either.
So we removed mutating DML entirely. Append-only writes, plus a dedup view:
CREATE OR REPLACE VIEW v_chunk AS
SELECT * FROM chunk
QUALIFY ROW_NUMBER() OVER (PARTITION BY chunk_id ORDER BY ingested_at DESC) = 1;
Re-ingesting appends a newer generation. The view returns only the newest. Every reader
gets MERGE semantics with zero mutating DML — the upsert logic didn't move into the
application, it moved into the read path, where the warehouse is happy to do it.
Two things this pattern quietly buys you:
- Load jobs instead of DML. They're free, they have no streaming-buffer semantics, and their few-second latency is irrelevant on an async path.
-
A free work queue. Embeddings fill via
INSERT … SELECTover chunks that have no embedding row yet. Rows whose embedding call failed are simply absent, so the next sweep retries them. TheNOT EXISTSjoin is the retry bookkeeping. There is no other retry bookkeeping.
One ordering rule, learned the hard way: load jobs are not transactional across tables, so
write your commit-marker table last. If the "this document is ingested" row lands
first and a later load fails, that document is permanently marked done with zero chunks —
never retried, never retrievable, and invisible to your gap metrics. Write it last and a
partial failure leaves no marker, so the sweep retries and the system converges. Orphan
rows from the failed attempt are harmless; the retry's newer generation supersedes them
through the same view.
7. Trap four: your embedding dimension is a cost decision, not a quality decision
The default output is 3072 dimensions. We use 768. That's not a quality compromise we
grudgingly accepted — it's the single biggest cost lever in the design.
In a warehouse, the vector dominates the row:
- 3072 dims → ~24 KB per chunk → a 5,000-chunk tenant scans ~120 MB per query
- 768 dims → ~6 KB per chunk → the same tenant scans ~30 MB per query
4× on every single query, forever. And modern embedding models are trained with
Matryoshka Representation Learning, so truncated outputs are automatically normalised —
768 is a recommended size, not a hack. You trade a little quality for four times the
throughput per dollar on the hot path.
Two companions to it:
Cluster on your tenant key. Every read is tenant-scoped; without clustering, each
query scans every vector in the table. This is the difference between scanning one
tenant's data and scanning everyone's, and it's one line of DDL.
Skip the vector index on purpose. It sounds like the obvious optimisation and it's a
trap here: a vector index requires a materialised table and forces you back into
post-filtering by tenant — which is exactly the starvation pattern from section 1,
faithfully reimplemented in a new database. Brute force below a few thousand rows per
tenant is fast, cheap and, more importantly, correct by construction.
Note also: two independent embedding stacks don't need to agree with each other. Vectors
from different stores are never compared as geometry — results are combined as text. The
only invariant that matters is internal consistency within one stack.
8. What we haven't decided — and the bar we set in advance
Here's where I stop claiming a win.
Ending the duplicate stack has exactly two coherent end-states, and they're mutually
exclusive:
Path A — keep the managed RAG service, share its embeddings. Point the corpus at a
feature-store backend that writes chunks and embeddings into a table you own. The service
chunks and embeds once; the graph reads those rows and adds only the entity layer. Our own
embedding fill disappears. Reranking, grounded generation and corpus lifecycle all survive
untouched.
Path B — remove the managed service, warehouse only. One stack, so no duplication by
construction. The ingestion service drops its import step and gets strictly simpler.
Corpus lifecycle code is deleted, not replaced. Grounded generation becomes
retrieve-then-generate at three call sites. And reranking has to be rebuilt — an LLM
reranker over the top-N candidates, temperature 0, falling back to score order on any
failure.
They can't be combined, because the feature-store option is the managed service.
We wrote the decision criteria down before collecting data, which is the only way this
isn't rationalisation:
| Metric | Why it decides the fork |
|---|---|
| Graph-hit rate | If retrieval rarely hits, Path B isn't viable at all |
| Results per hit | The old path starves to 1–3. This is the axis pre-filtering should win on |
| Answer quality, new vs fallback | The only thing that actually matters |
| Retrieval latency | The warehouse has a 0.5–2s floor; the managed call is faster |
| Bytes scanned per query | Validates the clustering claim, which is currently asserted and unmeasured |
- Path B is viable if hit rate is high, results-per-hit meaningfully beats the old path, and answer quality is at least equal.
- Path A wins if hits are inconsistent or quality drops — because then reranking and managed retrieval are carrying weight we didn't replace.
- Neither if the hit rate is low enough that the graph isn't earning its keep. In which case the honest move is deleting the graph outright, which is strictly cheaper than either migration.
That third option is the one I'd encourage you to keep on your own table. Every call site
in our system already degrades gracefully to the fallback — which means "turn the flag off
and delete 800 lines" was always available, and measuring first would have been cheaper
than either migration. We're doing the measurement after the build. Do it before.
The five-line version
- Post-filtering starves multi-tenant retrieval. Filter before top-k, not after.
- One hop is a join. Grep for variable-length patterns before you pay for a graph database.
- Deprecated parameters that are silently ignored are worse than ones that error.
-
Append-only + a dedup view gives you
MERGEsemantics with zero mutating DML. - Write the decision criteria down before the data arrives. Otherwise you'll find the data agrees with whatever you already built.
If you've run RAG on a warehouse in production — especially the latency floor on
interactive paths — I'd genuinely like to hear how it went.
Top comments (0)