Introduction
This article will show, how to set up a local or even air-gapped LLM using Ollama and either Podman or Docker. This can be used as a basis to facilitate Gen AI around restricted or secret data. I'm further using this setting to set up a chat bot, that can search trough secret documents, and a coding assistant, that I can use with a VS Code extension to get development support on secret classified code projects.
Info: all the used podman commands, execpt for the import and export of the volumes are working for docker as well, if you just replace "podman" with "docker".
Cache yourself the desired LLMs
The first step, to setting up a air-gapped LLM is to make the LLM itself accessible for the containers lying in network isolation. For that, I am going to export the volume from a non air-gapped container, that is already set with our desired LLMs, and import it to Podman.
Info: I could of course just assign the same volume to the non restricted and the air-gapped containers, but in my specific case, I even did the export of the volume on a separate machine, just to be sure. The transfer of the exported volume file, I'm sure you can manage without explanation.
To cache the desired LLMs, I'm first going to set up a Ollama Container, using the official Ollama Image form Docker Hub:
podman run -d \
-v ollama:/root/.ollama \
-p 11434:11434 \
--name ollama docker.io/ollama/ollama
After the container is set up, I can run this command for each of the LLMs I'd like to have inside of the air-gapped container. To find these LLM options, I just browsed trough the [Ollama Library].(https://ollama.com/library).
podman exec -it ollama ollama pull <your desired LLM>
Having pulled all the LLMs I've ever dreamed of, but not too many, since the export and import of the volume takes quite some time, due to its size, I'm now ready to export the Volume and transfer the file to the air-gapped machine:
podman volume export ollama --output ollama.tar
Create the Volume and restore the cached LLMs
On the air-gapped machine, I can now create a new volume using the following command:
podman volume create ollama
Now all there is left to do, is to bring life (or data) to the newly created volume. For that I'm running:
podman volume import ollama ollama.tar
Exporting and Importing Volumes is working in Docker as well. This however needs an active license for Docker Desktop. Also I only found a way to do this using the UI of Docker Desktop. I did not find any CLI commands to do this.
Configure the Network to be internal
To setup the Network around the container to be air-gapped, there are different options. The easiest way to set this up, is to just use the network option --internal. This keeps the container isolated from the host network, but keeps the option to add other containers within the same network:
podman network create ollama-internal-network --internal
However, often you'd like to still reach your container outside of the Podman console interface. This is the case in my use case as well, since I need the container to be reachable by either the VS Code extension as a coding assistant, or some sort of chatbot fronted, that itself I need to reach as well. Because of that I'm creating the network without the internal option, but assign fine tuned firewall rules to the network by using the kernel level rules of iptables.
So first I create the network:
podman network create ollama-internal-network
Then I retrieve the address range of this specific network:
networkAddressSpace=$(podman network inspect ollama-internal-network \
--format '{{range .Subnets}}{{.Subnet}}{{end}}')
And then I assign the specific firewall rules using iptables. In my case, I only care about the container not reaching the outside world, so I disable any egress from the network (address space of the network to be precise):
iptables -A OUTPUT -s $networkAddressSpace -j DROP
This now prevents my container to reach the outside world, by dropping all outgoing packages.
Create the internal Docker Container
After I've set up the isolated network for my container, and prepared the volume with the LLMs, I now can setup the container for my use:
podman run -d -v ollama:/root/.ollama \
-p 11434:11434 \
--network ollama-internal-network \
--name ollama-internal ollama/ollama
With the correct network assigned, the container now can't reach the outside world, so my data should be safe.
Test with a cached LLM
To test, if the volume import and assignment to the container was successful, I send the ollama run <myLLM> command to the container and check, if the LLM is actually available:
podman exec -it ollama-internal ollama run mistral
Conclusion
With my Ollama container now being set up in a secure way, I can now either directly throw prompts at the LLMs, that contain restricted information, or I can use this basis to setup a secure Coding Assistant or even a Chatbot that has knowledge about some secret documents and files.
Top comments (0)