DEV Community: Arnav Sharma

Enterprises Are Quietly Moving Their AI Back On-Premises. Here Is Why.

Arnav Sharma — Fri, 05 Jun 2026 09:00:58 +0000

42% of companies are considering moving workloads off the cloud. For AI infrastructure specifically, the reasons are more urgent than cost.

The Trend Nobody Expected
The story of the last decade was clear: everything moves to the cloud. On-premises infrastructure was expensive, inflexible, and the province of companies too slow to modernise.
Then 2025 happened.
42% of companies are now considering moving workloads back on-premises to escape vendor dependencies. 57% of IT leaders say they feel the need to run infrastructure within a single country, driven by data sovereignty requirements. Microsoft launched Sovereign Cloud capabilities in February 2026 specifically for AI models running fully disconnected from public cloud.
The cloud is not going away. But the assumption that everything should live in a public cloud, without question, is.
For AI infrastructure specifically, the reasons to reconsider that assumption are more urgent than for any other workload.

The Data Residency Problem Is Real and Getting Worse
When you run a RAG system on a managed cloud vector database, your data lives on someone else's servers in a region you may not have chosen.
For regulated industries, this is not an inconvenience. It is a compliance problem.
EU GDPR requires that personal data used in AI systems be processed in compliant environments with documented data flows and provenance. The EU-U.S. Data Privacy Framework remains legally uncertain following continued legal challenges, which means data stored in U.S.-based cloud services under EU jurisdiction is in an unclear compliance state.
In financial services, RBI guidelines in India, FCA requirements in the UK, and FINRA rules in the U.S. all have specific requirements about where sensitive financial data can be processed. A vector database storing embeddings of customer transaction data on a cloud server in Virginia creates questions that compliance teams cannot always answer satisfactorily.
In healthcare, HIPAA Business Associate Agreements are required for any service that handles protected health information. Most managed vector database providers offer BAAs only on enterprise tiers at significant cost premiums. Self-hosted on-prem deployment sidesteps this requirement entirely because the data never leaves the organisation's own infrastructure.
These are not edge cases. They are the primary procurement blockers for AI infrastructure in BFSI, healthcare, pharma, and government, which together represent the largest and highest-value potential customers for production AI systems.

The IP Protection Problem
The second reason enterprises are reconsidering cloud-hosted AI infrastructure is intellectual property.
When you embed your proprietary research, your internal documents, your customer data, your product roadmap, and your institutional knowledge into a vector database, the database contains a compressed representation of everything your organisation knows. That representation is your most valuable asset.
Storing it on a third-party cloud server raises questions that do not arise for, say, your email archive. The embeddings encode the semantic meaning of your data. A sufficiently capable adversary with access to your vector index could, in principle, extract meaningful information about the contents.
Most enterprises are not concerned about active adversarial attacks on their cloud provider. They are concerned about a simpler question: does our legal and governance framework require that our most sensitive intellectual property remain within our own controlled infrastructure? For an increasing number of organisations, the answer is yes.
Drug companies embedding molecular research, law firms embedding client documents, investment banks embedding proprietary trading strategies: in each case, the organisation's legal and competitive position argues strongly for keeping the data within their own perimeter.

The Cost Reality at Scale
Cost is the third driver of cloud repatriation, and for AI infrastructure it arrives sooner than for most workloads.
Modern server hardware is dramatically more powerful and cost-effective than it was five years ago. A single well-configured on-premises server with 64GB of RAM and modern NVMe storage can handle vector search workloads that would cost $800 or more per month on a managed cloud service.
The break-even point, where self-hosted infrastructure becomes cheaper than the managed alternative, has moved significantly earlier for vector database workloads than for general-purpose cloud compute. The memory-intensive nature of HNSW-based vector search means the instance sizes required for production workloads are expensive on cloud providers where you pay per GB of RAM.
Basecamp's analysis is the most-cited example: projected $7 million in savings over five years by avoiding cloud lock-in. Their workload is not vector search specifically, but the principle applies directly. At scale, the unit economics of owning your infrastructure beat the unit economics of renting it, and the scale at which this becomes true for vector databases is lower than for most other workloads.

The Hybrid Answer That Actually Works
The practical conclusion is not "cloud bad, on-prem good." It is that the architecture decision should be driven by the specific requirements of each workload rather than by a default assumption.
For AI infrastructure, a hybrid approach is increasingly the right answer. Development, experimentation, and low-sensitivity workloads on managed cloud. Sensitive production workloads, IP-containing knowledge bases, and regulated data on-premises or in private cloud environments the organisation controls.
This approach requires an infrastructure component that works identically in both environments. A database that runs on the managed cloud, can be migrated to self-hosted, and behaves identically in both is genuinely valuable. A database that only runs on managed cloud forecloses the option when you need it.
The teams that build on open-source infrastructure with on-prem deployment options maintain flexibility as their compliance requirements evolve. The teams that build on closed-source managed services discover, usually at an inconvenient moment, that their options are limited.

What the Enterprise Buyers Are Actually Asking For
The procurement conversations in enterprise AI infrastructure in 2026 have shifted noticeably from two years ago.
In 2024, the questions were primarily about performance and ease of use. Which database is fastest? Which has the best developer experience?
In 2026, the questions are: Can this run on our infrastructure? What certifications does it carry? Where does our data reside? What is the exit path if we need to migrate? Is the source code available for audit?
These are the questions that regulated industries ask about every piece of infrastructure they adopt. AI infrastructure is now subject to the same scrutiny. The vendors that can answer all five questions positively are the ones winning enterprise deals in 2026. The ones that can only answer the first two are losing them.

Endee supports on-premises deployment, private cloud, and Endee Cloud with identical APIs across all environments. ISO 27001 and SOC 2 Type II certified. Open source under Apache 2.0. Deploy where your data needs to be at endee.io.

Vector Database Benchmarks Are Lying to You. Here Is What to Test Instead.

Arnav Sharma — Fri, 05 Jun 2026 08:56:08 +0000

The leaderboards look impressive. They also test almost nothing that matters in production. Here is the gap.

The Number That Does Not Mean What You Think
Every vector database publishes benchmark results. Queries per second. Recall at various thresholds. Indexing throughput. P50 latency.
They look rigorous. They have tables and charts and methodology sections. And for most production use cases, they tell you almost nothing useful.
The reason is simple: benchmarks reward performance under static conditions. Production systems survive continuous writes, metadata filter combinations, and concurrency spikes. The conditions that determine whether a database works in production are almost never the conditions it was benchmarked under.

The Single Client Problem
VectorDBBench, the most widely used open-source benchmarking tool for vector databases, tests with a single client. One request at a time, measuring how fast the database responds.
Production systems do not have one client. They have 50, 100, or 500 concurrent clients hitting the database simultaneously, often with different queries and different metadata filter combinations.
Reddit's engineering team made this explicit after their 2025 deployment managing 340 million vectors. Under single-client conditions, performance looked fine. As concurrent users grew, the database spent more time resolving metadata filters than calculating similarity distances. P99 latency jumped by 10x.
A 10x P99 spike under concurrent load. That is the difference between a system that works and a system that is unusable at peak hours. Single-client benchmarks tell you nothing about whether this will happen to you.

The Static Data Problem
Benchmarks test after data ingestion completes. The index is built, the data is settled, the test begins. Milvus's own engineering team acknowledged this directly: "Benchmarks test after data ingestion completes, but production data never stops flowing."
Production RAG systems in 2026 require real-time data to be useful. Customer tickets, product inventory, regulatory updates, internal research: the knowledge base changes continuously. The database needs to re-index as quickly as it ingests, while still serving queries at low latency.
Some databases handle concurrent reads and writes gracefully. Others show significant latency degradation when writes and reads are happening simultaneously. Benchmarks run under static conditions will not tell you which category your candidate database falls into.

The Filter Benchmark That Actually Matters
Filtered vector search is the most common production query pattern and the most consistently underrepresented in benchmarks.
A real enterprise query looks like this: find documents semantically similar to this question, where the document belongs to this department, was created after this date, and is tagged with this category. The vector similarity search and the metadata filtering happen together, in a single query.
Most benchmarks test vector search separately from metadata filtering. The combined performance on realistic filter combinations, under concurrent load, is the number that determines whether your system works for real users.
The 2026 VectorDBBench analysis noted that the gap between filtered and unfiltered query performance is one of the largest and least discussed differences between vector databases. A database that ranks first on unfiltered recall may rank fourth on filtered recall at equivalent concurrency. The leaderboard does not show this because the leaderboard does not test it properly.

The Five Tests That Actually Predict Production Performance
Before committing to any vector database for a production workload, run these five tests yourself. Do not rely on the vendor's published results.
Concurrent filtered search at your expected peak load. Simulate 50 to 100 concurrent clients with realistic metadata filter combinations. Measure P95 and P99, not P50. Check whether P99 degrades more than 3x from P50 under load. If it does, you have a concurrency problem.
Write and read performance simultaneously. Send a continuous stream of writes while running read queries at production volume. Measure latency on the reads. Databases that handle this gracefully maintain stable read latency while ingesting. Databases that do not show read latency spikes proportional to write volume.
Recall at your actual data scale. Benchmarks commonly test at 1 million vectors. If your production workload is 50 million, test at 50 million. Recall degrades at scale for some indexes and holds stable for others. The difference is significant and invisible in small-scale tests.
Memory consumption at 2x your expected production size. Provision a node sized for your expected data volume and then load twice as much data. Does the database handle this gracefully with degraded performance, or does it fall over? Understanding the failure mode before production is significantly better than discovering it after.
Cold start query latency. Restart the database and measure latency on the first 1,000 queries. Some databases take time to warm up caches. In systems that restart periodically or fail over to new instances, cold start latency is the latency your users experience after any disruption.

The Benchmark Number That Is Actually Useful
Of all the numbers in a vector database benchmark report, the one that correlates most reliably with production performance is cost per billion queries at a fixed recall threshold.
This number captures efficiency. A database that achieves 98.5% recall on cheap hardware is more efficiently designed than one that achieves 98.5% recall on expensive hardware. Efficiency at the architectural level predicts efficiency under the varied conditions of production far better than peak performance under ideal conditions.
The March 2026 independent benchmark that tested eight configurations at 98.5% recall produced cost-per-billion-queries numbers ranging from $84 to $7,088 for comparable recall levels. The 84x gap reflects fundamentally different architectural efficiency. An architecturally efficient database is also, in practice, a database that handles resource pressure more gracefully under concurrent load. The two properties come from the same underlying design choices.

What This Means for Evaluation
The practical implication is that vendor-published benchmarks should be treated as directional, not definitive. They tell you roughly where to look, not what you will actually experience.
The teams that evaluate vector databases correctly run their own tests on their own data, at their expected production query patterns, with realistic concurrency, including writes. They check P99, not P50. They test at 2x their expected scale, not at demo scale.
This takes more time than reading a benchmark table. It also produces databases that work reliably in production instead of databases that worked in testing and failed under load.
The benchmark leaderboard is a starting point for the shortlist, not the endpoint for the decision.
Endee ranks first in the March 2026 independent VectorDBBench comparison across throughput, recall, latency, and cost simultaneously at 98.5% recall. Run your own tests on your own data at endee.io. Free to start, no credit card required.

The AI Vendor Lock-In Nobody Talks About Until They Are Stuck

Arnav Sharma — Fri, 05 Jun 2026 08:50:54 +0000

_72% of enterprises worry about cloud vendor lock-in. 58% build inside a single ecosystem anyway. Here is what happens when they try to leave.
_

The Migration Nobody Budgeted For
A company builds their AI infrastructure on a managed vector database. It works. The team ships. The system goes to production.
Eighteen months later, the pricing changes. Or the compliance team flags a data residency issue. Or a competitor launches something significantly better and the team wants to switch.
Then the real cost of the decision becomes visible.
AI vendor lock-in is often a six-figure cost event even for a single system. StackAI's 2026 infrastructure analysis put a formula to it: migration cost equals engineering hours multiplied by loaded rate, plus dual-run infrastructure during the transition period, plus data movement costs, plus revalidation, plus the risk buffer for what goes wrong. For a vector database at production scale with a live application depending on it, that total lands between $80,000 and $400,000 before anyone has written a line of migration code.
Most teams did not price this in when they chose their database.

How Lock-In Builds Silently
Vector database lock-in does not announce itself. It accumulates across three layers, and most teams only notice it when they try to move.
The first layer is the data layer. Indexing pipelines, metadata schemas, and filtering semantics are built around the specific behaviours of the database you chose. Pinecone's namespace model, Weaviate's collection schema, Milvus's partition key design: each of these shapes how you structure and retrieve your data. When you try to move to a different database, the schemas do not port cleanly. The filtering semantics are different. The chunking strategies that were optimised for one index type may perform differently on another. This is not a theoretical problem. It is the first thing every migration team encounters.
The second layer is the application layer. The SDK you used, the query patterns your application relies on, the metadata filter logic embedded in your retrieval code: all of it was written for a specific database's API. Different databases have meaningfully different APIs even when the underlying concepts are similar. Rewriting retrieval logic for a new database is not a weekend project at production scale.
The third layer is the operational layer. Your team learned one database. They know its failure modes, its monitoring characteristics, its performance tuning levers. Switching databases means relearning all of this at the same time you are managing a live migration.
Each layer compounds the others. The result is that switching vector databases in production is genuinely expensive and risky, in a way that switching, say, a logging tool is not.

The Numbers Behind the Concern
A HashiCorp 2026 cloud survey found that 72% of enterprises are worried about vendor lock-in. 58% keep building inside a single ecosystem anyway, because the alternative feels harder than the current cost.
That 58% number is the interesting one. These are not teams that are unaware of the risk. They are teams that have evaluated the alternatives and decided the switching cost is higher than the lock-in cost, at least for now.
The problem with "at least for now" is that it defers the decision to a moment when it will be more expensive and more urgent. Building deeply into a closed-source managed service is a bet that the service will never change its pricing, never have a compliance problem, never fall behind competitors technically, and never become unavailable at a critical moment. That is a lot of things to bet on simultaneously.
42% of companies are now considering moving workloads back on-premises specifically to escape vendor dependencies, according to 2026 cloud infrastructure data. Basecamp projected $7 million in savings over five years by avoiding cloud lock-in. The UK Cabinet Office estimated that overreliance on a single cloud provider could cost public bodies 894 million pounds.
These are not small numbers. They reflect a growing recognition that the convenience of a managed service in year one can become a strategic liability by year three.

Why Vector Databases Are a Specific Lock-In Risk
Not all infrastructure lock-in is equal. A logging service or a monitoring tool can usually be swapped out in days. A vector database at the core of a production AI system is a different category of dependency.
Your vector database holds your indexed knowledge. Everything your RAG system knows, every memory your AI agent has accumulated, every document your semantic search system can find: it is all in there, in a format specific to that database. The schema, the metadata, the index configuration, and the query logic were all built together. They are not independently portable.
Pinecone is closed source. There is no way to inspect or modify the underlying engine. If Pinecone changes its pricing model, changes its API, or simply decides to deprecate a feature your system depends on, your options are limited to accepting the change or migrating. Both are expensive.
The September 2025 pricing change that introduced a $50 per month minimum regardless of usage was a small version of this risk materialising. It was a manageable change. The teams that panicked were the ones who had never considered what "manageable" might look like at a different scale.

The Open Source Difference
An Apache 2.0 licensed database changes the lock-in calculation fundamentally.
With an open-source database, you can inspect the codebase, modify it for your needs, self-host it on your own infrastructure, and move between the managed cloud version and the self-hosted version without changing your application code. The vendor can change their pricing. They can be acquired. They can shut down the managed service entirely. In none of those cases are you stuck, because the software itself is yours to run.
This is not a theoretical advantage. It is the concrete answer to the question "what do we do if this vendor becomes untenable?" With a closed-source managed service, the answer is expensive. With an open-source database, the answer is straightforward.
The teams building AI systems that will be in production for three or more years are thinking about this. The teams building prototypes are not. The distinction matters a great deal when year three arrives.

What to Check Before You Commit
Before committing to any vector database for a production AI system, ask four questions.
Can I move between the managed cloud and self-hosted versions without rewriting my application code? If the answer is no, you are building in a switching cost from day one.
Is the source code available for inspection and modification? For regulated industries, this is often a compliance requirement. For everyone else, it is a useful indicator of whether the vendor has confidence in their product.
What does migration look like if I need to switch in two years? Ask for specifics. If the answer is vague or the conversation gets uncomfortable, that tells you something.
Does the license allow me to run this on my own infrastructure permanently? Closed-source managed services can change this at any time.
The teams that ask these questions early make architecture decisions they are still comfortable with three years later. The teams that ask them after they are stuck are the ones funding the six-figure migration.
Endee is open source under the Apache 2.0 license. Run it on Endee Cloud, self-host it, or switch between the two without code changes. No lock-in by design. Start free at endee.io.

3 Seconds Used to Be Fine. In 2026 It Kills Your Product.

Arnav Sharma — Fri, 05 Jun 2026 08:44:06 +0000

The latency budgets for AI systems have tightened dramatically in the last 18 months. Most retrieval layers are not built for what users now expect.

The Threshold Nobody Warned You About
Three seconds of end-to-end AI response time was workable in 2024. Teams shipped systems at that speed and users tolerated it. It was slow, but it was new and impressive enough that people gave it grace.
That grace period is over.
By 2026, three seconds is a dealbreaker. Users expect responses under one second. Voice AI agents need total response times under 800 milliseconds. Conversational chat agents have a 200 millisecond budget before the experience starts to feel broken. The bar shifted quickly and it is not shifting back.
The problem is that most retrieval layers were built for a different set of expectations.

Where the Time Actually Goes
A RAG system has multiple stages between a user's question and the answer they receive. Each stage consumes time from a budget that is tighter than most teams realise.
The embedding call converts the user's query into a vector. With a typical hosted embedding API, this takes 100 to 400 milliseconds depending on the provider and network conditions.
The vector search retrieves relevant chunks from the database. A well-configured purpose-built vector database handles this in under 50 milliseconds. A poorly configured one, or one under concurrent load, can take 200 to 500 milliseconds.
The re-ranking step scores the retrieved chunks for relevance. Add 50 to 200 milliseconds.
The LLM generates the response. Add 400 to 1,500 milliseconds depending on output length and model.
Add these together for a voice AI use case with a strict 800 millisecond total budget and the math is unforgiving. If the embedding call takes 300ms and the LLM takes 400ms, the vector search has 100ms left. Every millisecond over that number breaks the experience.

The Benchmark Numbers for 2026
The 2026 Salt Technologies vector database benchmark, testing at 1 million vectors across 1,536 dimensions, gives the clearest current picture of where each database actually lands.
Qdrant hits 4ms at P50, the lowest among purpose-built vector databases. Redis comes in at 5ms P50 for in-memory workloads. At a 99% recall threshold, both Qdrant and Postgres with pgvector and pgvectorscale hit sub-100ms maximum query latency.
The P99 number is the one that matters for production. P50 is the median. P99 is what your slowest 1% of users experience. In a system with 10,000 daily active users, P99 latency determines the experience for 100 users every day. In enterprise AI, those 100 users often include the ones most likely to write the internal assessment of whether the system is worth keeping.
Reddit's engineering team, managing 340 million vectors, identified metadata filtering as the primary performance bottleneck in their 2025 deployment. As concurrent users grew, the database spent more time resolving metadata filters than calculating similarity distances. Moving data between the vector graph and the relational metadata store caused P99 latency to jump by 10x.
A 10x P99 spike under concurrent load is not a configuration problem. It is an architecture problem. And it is invisible in single-client benchmarks.

The Concurrency Gap in Most Evaluations
Standard benchmarks like VectorDBBench test with a single client. Production systems run with 100 or more concurrent clients hitting different metadata subsets simultaneously.
This gap between benchmark conditions and production conditions is one of the most common reasons teams are surprised by their latency numbers after launch. The database performed well in testing. Testing had one client. Production has a hundred.
Metadata filtering amplifies the concurrency problem. A filter like "retrieve documents from this user, tagged with this category, created after this date" requires the database to combine vector similarity calculation with structured attribute lookups. Under single-client conditions this is fast. Under concurrent load with varied filter combinations, the query planner is doing genuinely complex work and the latency profile changes.
This is why Endee's sub-5ms P99 under realistic load is a meaningful benchmark result. P99 under concurrent production conditions is what determines whether your AI system actually feels fast to users. P50 under a single client tells you almost nothing.

The Voice AI Forcing Function
Voice AI is the use case forcing the latency conversation to a conclusion.
Voice AI agents need sub-100ms retrieval to hit under 800ms total response time. That leaves roughly 100ms for vector search after embedding and before LLM generation. At that budget, the difference between a 4ms database and a 50ms database is not marginal. One makes the product work. The other does not.
This matters beyond voice AI specifically because voice AI is where the latency requirements become undeniable. Teams that have not thought carefully about retrieval latency are confronted by it the moment they try to build a voice product. The constraint that was tolerable in a text interface is fatal in a voice one.
And voice is growing. Enterprise copilots, call center AI, meeting assistants, real-time translation layers: all of these are voice or near-voice applications where the 800ms total budget is not negotiable.

What Fast Retrieval Actually Requires
Getting to sub-10ms P99 vector search under production load requires three things working together.
The index needs to be resident in memory or accessible with predictable, low-latency disk reads. Indexes that spill to disk under concurrent load produce the P99 spikes that break user experience.
The filtering architecture needs to handle metadata lookups without adding query planning overhead that scales with concurrent users. Databases that separate vector and metadata storage into different internal systems compound latency under load in exactly the way Reddit's team described.
The database needs to be tested under concurrent load at realistic query rates before deployment, not under single-client conditions that tell you nothing about production behaviour.
The teams that check all three of these boxes build AI systems that feel fast. The teams that check none of them discover the problem after launch, when fixing it requires a migration that nobody planned for.

The Practical Test Before You Ship
Before any AI system goes to production, run a load test at your expected peak concurrent users with realistic query patterns and metadata filter distributions. Check P95 and P99 latency, not just P50. Check what happens when concurrent users double.
If the P99 numbers are above 50ms at peak load, you have a retrieval architecture problem that no amount of prompt engineering or model selection will fix. The fix is in the database.
Three seconds was fine in 2024. In 2026, it loses users. Sub-second retrieval is not a stretch goal. It is the baseline.

Endee delivers sub-5ms P99 latency under realistic concurrent load, ranked first in independent benchmarks on throughput and recall simultaneously. Free to start at endee.io.