Most guides on running large language models locally start from the same assumption: you need a lot of expensive GPU memory, or you don't get to run a big model at all. That framing quietly rules out most of the organisations that have the strongest reason to keep their intelligence on their own machines. Banks, hospitals and government teams can't send their most sensitive data to someone else's cloud and trust a policy page, and they usually can't buy a rack of the newest accelerators either.
I want to walk through the approach we took to break that trade-off, because the engineering idea is simple once you see it, and it changes what "run it yourself" actually costs.
The real constraint is memory, not compute
A big mixture-of-experts model has a lot of parameters, but only a small fraction of them fire for any given token. The weights still have to live somewhere, and that "somewhere" is what people reach for expensive GPU memory to solve. The compute you need per token is modest. The memory footprint is the wall.
So we stopped treating GPU memory as the thing that has to hold the whole model. Instead, the model sits in ordinary system RAM, which is cheap and plentiful on a normal workstation or server, and only the parts each request actually needs get streamed onto the GPU. The GPU does the maths. The RAM holds the model. You size the machine on the resource you already have, not the one that costs the most.
The result is that a model most people assume needs a datacentre will serve from a single workstation-class box. It isn't instant, and I won't pretend it is: streaming experts off system memory costs latency. For a lot of real work, that's a trade worth making, because the alternative isn't "faster in the cloud", it's "not on your own hardware at all".
Offline is a property you can test, not a promise
Running locally only matters if it stays local. It's easy to say a component is air-gapped and much harder to make that something a reviewer can check. The inference engine we run refuses any outbound connection. It's loopback only, and an independent adversarial review confirmed the egress guard holds. That's the difference between "trust us" and "here's the test you can run yourself".
If you're building anything in this space, I'd push for the same standard. Make the no-egress claim falsifiable. Bind the listener to loopback, block outbound at the process, and let someone try to break it before you ship it.
Prove what ran, not just that it ran
The last piece is accountability. When an autonomous system takes an action on regulated data, "it worked" isn't enough. You need to be able to prove, later, which model ran, on which machine, on which inputs, and what it did.
We seal each session into an append-only, hash-chained record and verify it against a signed checkpoint held off the box. That makes it tamper-evident: an attacker can delete a file or refuse a verifier, but they can't silently alter the record and have it still verify. I'm careful with the language here. It's tamper-evident, not tamper-proof. The honest claim is that tampering shows, not that it's impossible.
Why this matters
Put the three together and you get something specific: a capable model, running on hardware a normal organisation already owns, fully offline, with a record of exactly what happened that survives an audit. That's the foundation the rest of our system is built on, and it's the part that turns sovereign AI from an infrastructure project into something you actually run on your own terms.
None of this needs the newest hardware or a cloud contract. It needs a change of assumption: that the limit is memory, not compute, and that "sovereign" has to be testable, not asserted. If you're weighing up local inference for work you can't send elsewhere, that's the lens I'd start from.
I'm Micky Irons, founder of Mickai. We build a Sovereign Intelligence Operating System that runs on your own hardware. Our patents are filed, not granted, and I'd rather be precise about that than impressive.
Top comments (0)