Private LLM Options: Local, Cloud, or Confidential?

Rio Consulting — Tue, 06 Oct 2026 01:22:01 +0000

You want a private LLM, and the obvious move is to buy a box and run it yourself. Before you do, it's worth understanding what that hardware really costs, what happens when it's out of date in two years and you're still paying it off, and what the other options are. There are three routes: your own hardware, a cloud provider under contract, or confidential inference on hardware that can't read the prompt in the first place.

This post walks through your options for Private LLMs, what each costs, and a deeper look into AI privacy when you're sharing the agent with the rest of your team.

What does "private" mean for an LLM?
Option 1: a local LLM on your own hardware
Option 2: a cloud provider with a contract
Option 3: confidential inference with SayGM
How do the three compare?
The other side: private memory in shared agents
Common questions about private LLMs
Which one should you pick?

What does "private" mean for an LLM?

The term Private LLM / AI is used across various contexts

Physical privacy means the prompt never leaves a machine you control.

Contractual privacy means it leaves, but the provider has agreed in writing not to train on it, share it, or keep it past a set window.

Technical privacy means it leaves, but it's processed inside hardware that the operator can't see into, and you can check that yourself.

Who is able to read the prompt, and is the thing stopping them a wall, a promise, or a chip?

Option 1: a local LLM on your own hardware

A local LLM is the cleanest answer to the privacy question. Nothing leaves the building, there's no provider to trust, and no terms of service to reread every time they change.

A current hardware roundup puts an RTX 5090 at $1,999 MSRP (USD) with 32 GB of VRAM, managing roughly 25 to 30 tokens per second on a 70B model, and a 128 GB Mac Studio at $3,500 to $4,000 for similar speeds (fungies.io hardware guide). That's a perfectly good setup for one person.

Tradeoffs are:

Concurrency. One card serving 25 tokens per second is one conversation. An agent that fans out into ten parallel calls is queueing on that same card.
Your Wife (or Husband). Will they believe that you really need a kitted out gaming PC for "work"?
Model ceiling. A self hosted LLM means open-weight models, usually quantised to fit. They're good, and getting better fast, but the largest frontier models aren't available to download at any price.
Flexibility. One year down the line, if the latest models are just outside of your reach you'll need to buy a whole new machine again. Realistically, you'll be stuck with this hardware for 4+ years. At least it will still run Crysis when the spouse isn't looking.
Your time. Drivers, serving stack, updates, the box that falls over on a Saturday.

So local is the most private option and often the cheapest per token once the hardware is paid off. It is a large upfront cost and is likely to get outdated soon.

Option 2: a cloud provider with a contract

This is where most privacy-first enterprises land. The big three clouds all sell access to frontier models under enterprise terms that are a long way from pasting things into a consumer chat app. The privacy here is contractual, so the detail is in the documents.

AWS Bedrock states that model providers don't have access to customer prompts and completions. Its data retention documentation describes a zero data retention mode where "no request or response data is written to durable storage by AWS or shared with the model provider". The same page also describes a review mode that applies to some of the newest frontier models, where prompts and completions are "retained within the AWS boundary for up to 30 days and may be reviewed by AWS". Which mode you get depends on the model you pick.

Microsoft Azure is similarly direct. Its data privacy page says prompts and completions "are NOT available to OpenAI or other providers", and are not used to train foundation models without your permission. By default there is an abuse monitoring store where prompts can be held for human review. Managed customers can apply to have that modified, and you can confirm it's off by checking for a "ContentLogging": "false" attribute on the resource.

Google Cloud documents its own zero data retention path for Vertex AI, which again is something you configure and request rather than something you get by default.

The provider's staff and systems are technically able to see the prompt; and the thing stopping them is policy. For a lot of teams that's fine. For source code, client documents or anything you'd lose sleep over, it's worth knowing that's the shape of the guarantee.

This approach also comes with a costly enterprise contract to pay. Best for hospitals and other large public organisations who need someone to point the finger at.

Option 3: a private LLM gateway

The third option is newer and sits between the other two. A private LLM gateway is an API you call like any cloud endpoint, except the gateway runs inside a trusted execution environment (TEE). The short version of a TEE is that the chip encrypts the memory of whatever is running inside it, so the host machine and the people operating it can't read what's in there. You rent the hardware like cloud, but the privacy is enforced by the chip, closer to local.

For example, SayGM runs its gateway in an Intel TDX confidential VM and explain very well how trusted execution environments work for AI teams if you want the mechanics.

There are two different guarantees on a gateway like this and it matters not to blur them:

Confidential models. Open-weight models run inside the enclave itself. The prompt is only decrypted inside the hardware running the model, so the gateway can't read it, and neither can the host.
Frontier models. The route through the gateway is sealed, but if Anthropic or OpenAI are serving your models they will still see the prompts - though there are often options to mask out personal information.

The benefits of private LLM Gateways are:

No upfront cost, or expensive contracts
Pay per use
Hardware-level verification that your data isn't being logged

However, they aren't able to offer truly private enterprise access to frontier models like Amazon Bedrock can.

The other side: private memory in shared agents

Everything above is about protecting a prompt on its way to a model. Once you give an agent memory and let more than one person use it, now you have to control leaking internally within users.

A shared agent with memory means that anybody with access can attempt to coax information that other people had shared out of the agent. Say you set up a fully private AI Agent for your company, but then the agent begins sharing private conversations from the HR team to the staff it concerns.

The research on this is not reassuring. A 2026 benchmark from ServiceNow AI Research and Mila, PiSAs, tested multi-user agent systems and found at least one privacy violation in over 75% of test runs, with shared memory making things markedly worse. Separately, the MEXTRA paper showed that private interactions stored in agent memory can be pulled out with black-box prompts alone.

You can run the most private LLM in the world and still have your agent tell the intern what the founder asked it last week. A few things that help:

Scope memory per user by default, and make shared memory something a person opts a specific note into.
Enforce access in the storage layer. A line in the system prompt saying "don't share private data" is the approach the benchmark found insufficient.
Redact before you write. Secrets and personal data that never reach memory can't be recalled from it.
Treat anything in shared memory as readable by everyone who can talk to the agent, because in practice it is.

Common questions about private LLMs

Is a local LLM completely private?

The inference is. Whether the whole setup is depends on what else the machine talks to: telemetry in the tools around it, web search plugins, and any memory store your agent writes to.

Do cloud providers train on my prompts?

Under enterprise API terms, generally not. Azure says so in writing in the page linked above. Retention is the more useful thing to check, since prompts can be stored for abuse review even when they aren't used for training.

What is the cheapest private LLM option?

At low volume, a pay-per-token confidential tier, because there's no hardware to buy. At high, steady volume on a single model, owned hardware can win once it's paid off, if you don't count your own time.

Which one should you pick?

If it's just you and the workload is steady, a local LLM is hard to argue with. If you need the top frontier models with an SLA and a contract your legal team already understands, go cloud and read the retention pages properly. If you're a small team running agents over data you'd rather nobody could read, and you'd like to verify that instead of trusting it, a private LLM gateway is the option built for that.

Whichever private LLM route you take, sort the memory question out before the agent goes multi-user. That's the leak people find second.

DEV Community: Rio Consulting