so a while back i did something i do not recommend before bed.
i counted the vendors in a normal production AI stack. not a weird stack. not a bad team's stack. the stack everyone builds, the one in every architecture diagram on this site.
model gateway. vector database. memory service. rag framework. embeddings provider. orchestration. observability. two or three model providers.
that's eight. for one app.
and here's the first thing that got me: nobody approved that stack. not one person at any company, ever, sat down and said "yes, eight vendors, sounds right." it got approved one sprint at a time. a ticket here, a free tier there, a "we'll swap it later" that became load-bearing.
your architecture wasn't designed. it accreted. like a reef. or plaque.
the fun part: what each one actually is
security people have a phrase, "attack surface," and it sounds abstract until you translate each vendor into what it really is:
a model gateway is a standing API key in your CI. a vector database is a second copy of your data. a memory vendor is your prompts in someone else's logs. a rag framework is somebody's SDK running inside your process. observability is prompt logs leaving your network as the product working correctly.
read that last one again. the exfiltration is the feature. you pay for it monthly.
quick quiz you will fail
open two lists. one: every AI line item in your billing. two: every AI-related package in your lockfiles.
do they match?
they never match. i've asked a lot of teams. the answer is always no, and everyone laughs, and it's the nervous kind of laugh, because the gap between those two lists is software that touches your customer data and appears in exactly zero of your security reviews.
the gap is the attack surface. congratulations, you just did a security audit with grep.
"embeddings are just numbers though"
this is my favorite myth in all of AI, because it's so comforting and so wrong.
the theory: we embedded the text, so now it's a vector, vectors are anonymous math, we're fine.
the practice: embedding inversion is published research. morris et al. (2023, "text embeddings reveal (almost) as much as text") reconstructed the exact input text from its embedding 92% of the time for short inputs. not "similar vibes." the actual text.
so your vector database is not a bag of anonymous numbers. it's a second, queryable database of your source data, with its own API key, sitting in someone else's cloud.
now the aha that hurts: go ask whoever maintains your data inventory whether the vector store is on it.
it's not on it. it's never on it. you have a database of customer data that your own compliance team doesn't know exists, and you built it on purpose, with a jira ticket.
deletion theater
follow one user message through the stack. it hits the gateway (logged). gets embedded (copy two, the vector db). the memory service persists it (copy three). observability captures the full trace (copy four).
one message. four systems. four retention policies you've never read.
now legal walks over: "customer invoked their right to erasure, delete their data."
you cannot do that. what you can do is file four vendor tickets and hope. under gdpr this is the difference between "we called the delete endpoint and logged the result" and "we asked four companies nicely." one of those survives an audit. guess which one you have.
i call this deletion theater. everyone has a retention policy. almost nobody has a delete button that actually reaches all four copies.
deletion you can't prove is deletion you don't have. put it on a poster.
meanwhile, your dependencies
your rag framework is not one dependency. it's a tree. hundreds of packages, written by strangers, pinned loosely, executing in the same process that holds your database credentials.
this isn't hypothetical doom. langchain shipped a remote code execution bug in its math chain (CVE-2023-29374). pytorch's nightly build got hit by dependency confusion in 2022. owasp made supply chain a whole category in its LLM top 10 because this keeps happening.
the framework's security posture is your security posture. you married it. you just didn't have a ceremony.
and the stack you didn't even choose
everything above is the stack you deployed on purpose. there's another one.
it's your employees pasting contracts, code, and customer records into whatever consumer chatbot they like, on their personal accounts, right now, while you read this.
your company probably "banned" this. here's the aha: the ban is working as a compliance document and failing as a control. people don't stop using tools that make them faster. they stop telling you.
the only fix i've seen work is replacement, not prohibition. one sanctioned surface, the models people actually want, behind sso, logged. you can't firewall your way out of shadow AI. you can only out-compete it.
so what do you actually do
the counterintuitive answer, and the whole point: the fix is not more security. it's less stack.
every security tool you bolt onto an eight-vendor sprawl is a ninth vendor. the arithmetic only moves in one direction: fewer systems holding data, fewer standing keys, fewer egress paths. concretely, in priority order:
- fewer copies. every system holding prompts or embeddings is a copy. count them, then make the number smaller.
- one egress door. all model traffic through one brokered path where policy lives. no SDK in app code dialing providers directly.
- one control plane, your keys. one place to rotate, one audit trail that answers "which model saw which data." byok, so consolidating the plane never means handing over the keys.
- deletion as an api call. if you can't prove deletion programmatically across every copy, see poster above. you can build all of this yourself with enough glue code and discipline. some teams genuinely do. it's a lot of glue.
the horse, as promised
this counting exercise is roughly why backboard exists. routing across 17,000+ models, memory (first on the LoCoMo and LongMemEval benchmarks, receipts on github), embeddings, rag, and stateful threads behind one api and one key. for enterprises it ships as a signed container inside your own cloud, behind the IAM and SOC and audit trail you already trust. egress to outside models is opt-in per workload, never standing. deletion is an endpoint, not a ticket.
the objection i get, and it's a fair one: "congrats, you built a single point of compromise." three answers. byok means the platform never holds your keys, so consolidating the control plane isn't consolidating the credentials. in the enterprise deployment the one door is your door, inside your perimeter. and memory is exportable and routing is model-agnostic, so the exit is always open, which is the only honest answer to lock-in.
also, you'll notice zero certification badges in this post. on purpose. we publish nothing we haven't earned. ask for the architecture, not the badge.
the whole post in four lines
eight vendors is eight keys, eight egress paths, eight retention policies, and eight reviews your security team owes.
your embeddings are your data. your vector db is a database. put it on the map.
deletion you can't prove is deletion you don't have.
and the fix is not more security. it's less stack.
if you want to tear this apart, the code is at github.com/Backboard-io and my dms are open. i counted. your turn.
Top comments (2)
This is one of those articles where the more I read, the more uncomfortable the architecture becomes. 😄
I really like the way you've reduced “AI attack surface” to concrete things: copies of data, standing credentials, egress paths, dependency trees and independent retention policies. It makes the problem much harder to hand-wave away.
The point about the stack being accreted rather than designed is particularly sharp. I've seen a similar pattern when working with RAG and agentic systems: every new capability adds another integration, another trust boundary and another assumption about what the next component can be trusted with.
And the deletion argument is probably my favorite part. “Deletion you can't prove is deletion you don't have” is a brutal but very useful way of expressing the problem.
I'm also very much aligned with the supply-chain angle. An AI application isn't just the model: the SDKs, orchestration layer, retrieval components, observability pipeline and their transitive dependencies are part of the security boundary too.
You write about these problems in a way I really enjoy: concrete architecture first, consequences second, marketing last. There are some genuinely smart observations in here.
A lot of this overlaps with what I've been working on around AI security, RAG, agentic systems, adversarial testing and security-by-design. I've been approaching the problem from another direction: testing whether those boundaries actually hold when the system executes, rather than assuming the architecture diagram is the architecture.
Really enjoyed this one. If you're interested in that side of the problem, feel free to have a look at my work here on DEV as well. 🔐
The billing-list versus lockfile check is useful, but I’d add a third list: data stores and egress observed at runtime. Billing finds contracted vendors and lockfiles find libraries; neither catches a gateway forwarding to a newly enabled backend or traces exported by configuration. A synthetic tenant marker, followed by deletion and read-back across every observed sink, would turn the deletion claim into something testable.