DEV Community

Celso Nery
Celso Nery

Posted on

Building Your Own Private AI Server

#ai #ollama #self #selfhosted

🇧🇷 Leia a versão em portugues aqui

Installing and Running Ollama Bare Metal: Building Your Own Private AI Server

In this article, I'll use a Debian server to run an AI model locally, from scratch.

Turning a local computer into a private AI server is a great move, especially for guaranteeing full privacy and avoiding monthly subscriptions. With the right hardware and today's tools, the process is much simpler than it sounds.

Hardware requirements

The most critical component is the GPU (graphics card). Unlike regular software, AI models run on parallel processing cores and demand a lot of video memory (VRAM).

Component Minimum recommendation Ideal recommendation
GPU NVIDIA (8GB VRAM) NVIDIA RTX 3090/4090 (24GB VRAM)
RAM 16GB 32GB or more
Storage NVMe SSD (50GB free) NVMe SSD (500GB+ for multiple models)
OS Linux or Windows with WSL2 Linux (Ubuntu/Fedora)

Note: NVIDIA cards are the industry standard thanks to CUDA cores. While it's possible to run models on CPUs or Macs (Apple Silicon), performance on NVIDIA cards is dramatically better.

The runtime tool: Ollama

I'll be using Ollama. It's open-source and manages downloading and running models in an optimized way.

  1. Installation: ollama.com.
  2. Running a model: in the terminal, just type: ollama run llama3
  3. Advantage: it creates a local API on port 11434, letting you connect other interfaces to it.

Visual interface (Open WebUI)

I'll also install Open WebUI, a web interface that looks like ChatGPT but runs 100% on your own server — in case you don't want to rely solely on the terminal.

Open WebUI is the full, popular interface. We'll run it via Docker:

  • Supports multiple users;
  • Allows uploading documents for analysis (RAG);
  • Connects directly to Ollama.

The models (the "brain")

Depending on your hardware and goals, you'll want to use different models:

  • Llama 3 (Meta): currently the best all-around, general-purpose model.
  • Mistral / Mixtral: excellent balance between speed and intelligence.
  • DeepSeek Coder: specialized in programming and script writing.
  • Phi-3 (Microsoft): a lightweight model for lower-powered machines.

Private use cases

With your server running, you can set up advanced features that wouldn't be safe on public clouds:

  • RAG (Retrieval-Augmented Generation): feed the AI your PDFs, contracts, or private code. It will answer based only on your files, without any data ever leaving your network.
  • Code automation: use VS Code extensions (like Continue.dev) to use your local AI as a free "Copilot."
  • Local agents: build scripts that automatically organize your files or analyze system logs.

Choosing the Best Distro for Ollama

The choice of Linux distribution directly impacts how easy it is to manage NVIDIA drivers.

Here's a short analysis based on my own experience.

The best option: Fedora

Fedora is, currently, one of the best choices for local AI.

  • Modern kernel: essential for support of new GPUs and virtualization/container technologies.
  • NVIDIA drivers: the RPM Fusion repository makes installing proprietary drivers and CUDA very simple and stable.
  • Podman vs Docker: Fedora ships with Podman by default, but you can install the official Docker Engine without issues.
  • Profile: ideal for anyone who wants the latest versions of Ollama and AI libraries without compiling anything manually.

The stability pick: Debian (Bookworm)

If your priority is a server that "never stops," Debian is the way to go.

  • Clean environment: if you prefer configuring YAML files and using the CLI (Vim/nmcli) like I do, Debian gives you exactly what you ask for, without extra bells and whistles.
  • NVIDIA: installing nvidia-driver and nvidia-container-toolkit on Debian Stable is extremely well documented and solid.
  • Point of attention: Debian Stable's kernel can lag behind support for brand-new GPUs (like a future 50-series card), but for an RTX 30 or 40, it works perfectly.

The enterprise alternative: openSUSE Tumbleweed

I really like openSUSE (which I use on my own work machine) — it's an excellent option for anyone who wants a rolling release system (always up to date) but with the safety of Btrfs and Snapper.

  • Zypper: robust package management.
  • YaST: makes network and security configuration easier.
  • OBS (Open Build Service): makes it easy to find specific AI packages that might not be in the official repositories.

Comparison table for AI workloads

Distro Driver management Kernel updates Docker/Containers
Fedora Easy/Modern Fast Native/Podman
Debian Solid/Manual Slow Industry standard
openSUSE Via repos Rolling Very good

My recommendation depending on your profile

  1. Go with Fedora if you want everything running fast, with the latest versions.
  2. Go with Debian if you want to set it up once and forget the server exists (a "set and forget" style).

My choice: Debian (Stable)

For our "test" AI server, Debian was the choice for a simple reason: library compatibility and stability.

  • Why Debian: it's the cleanest possible base. Coming from Slackware and preferring to configure everything via CLI (Vim, nmcli), Debian doesn't install anything I didn't ask for. It's perfect for running Docker Engine natively and predictably.
  • The "NVIDIA advantage": almost all nvidia-container-toolkit packages and CUDA drivers are tested first, and more thoroughly, on Debian/Ubuntu bases. This avoids a kernel update breaking your AI server in the middle of a project.

Why I avoided Fedora and Tumbleweed for this server

Although they're excellent distros, for a dedicated server:

  • Rolling release kernel: on Fedora or openSUSE Tumbleweed, the kernel updates very frequently. If you install the NVIDIA driver and the kernel updates, you may run into the dreaded version "mismatch," requiring you to recompile the driver module (DKMS).
  • Focus on stability: an AI server for a company needs to be available whenever n8n calls the API. Debian Stable guarantees the runtime environment won't change for years.

Installing Ollama "Bare Metal"

As mentioned earlier, we'll use Debian 12. I'm running this on a modest server with only 8GB of RAM and no dedicated GPU, due to lack of time (and budget) 😅.

Installation

Ollama provides an official script that sets up the binary and the systemctl service automatically:

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Network configuration (important for external access)

By default, native Ollama only listens on 127.0.0.1 (localhost). If you want to integrate with n8n or access it from another machine on the network, you need to open it up for external access:

1. Edit the Ollama service:

sudo systemctl edit ollama.service
Enter fullscreen mode Exit fullscreen mode

Or edit the file directly at /etc/systemd/system/ollama.service.

2. Between the [Service] lines, before anything else, add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Enter fullscreen mode Exit fullscreen mode

3. Save (Ctrl+O, Enter) and exit (Ctrl+X).

4. Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama
Enter fullscreen mode Exit fullscreen mode

Optimizing for 8GB of RAM (Swap)

Since I only have 8GB of RAM, I needed to create a large swap file on the SSD so the AI wouldn't freeze the system.

Step by step to create 12GB of swap on the SSD:

# Created a swap file inside /home (where I had space)
sudo fallocate -l 12G /home/swapfile
sudo chmod 600 /home/swapfile
sudo mkswap /home/swapfile
sudo swapon /home/swapfile

# Made it permanent (added it to /etc/fstab)
echo '/home/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Enter fullscreen mode Exit fullscreen mode

Downloading strategic models

For my scenario (8GB RAM), I downloaded phi3.5 by running this command in the terminal:

# The best "jack of all trades" that fits in RAM (2.3GB)
ollama pull phi3.5
Enter fullscreen mode Exit fullscreen mode

Where are the models stored? (Important)

By default, on Linux, Ollama stores models at:

/usr/share/ollama/.ollama/models
Enter fullscreen mode Exit fullscreen mode

The stress test (monitoring)

Open a second terminal tab (or use tmux/screen) and run htop. Then, on the first tab, call the model:

ollama run phi3.5
Enter fullscreen mode Exit fullscreen mode

This will open Ollama's prompt.

Ollama prompt

Now just ask a question. For example:

>>> Why is the sky blue?
Enter fullscreen mode Exit fullscreen mode

What to watch for:

  • RAM: check whether usage stabilizes around 4GB–5GB.
  • CPU: since I don't have a GPU yet, CPU usage stays around 90–100% while the AI "thinks."

Ollama running — memory and CPU usage

Managing models

To check which models are installed:

ollama list
Enter fullscreen mode Exit fullscreen mode

If you need to remove a model:

ollama rm <model>
Enter fullscreen mode Exit fullscreen mode

Example:

ollama rm phi3.5:latest
Enter fullscreen mode Exit fullscreen mode

Depending on your hardware and goals, you'll pick different models

  • Llama 3 (Meta): currently the best all-around, general-purpose model.
  • Mistral / Mixtral: excellent balance between speed and intelligence.
  • DeepSeek Coder: specialized in programming and script writing.
  • Phi-3 / 3.5 (Microsoft): lightweight models for lower-powered machines.

Some models I managed to run on my server 

ollama pull deepseek-coder-v2:16b-lite-instruct-q4_K_M
ollama pull llama3.2:3b
ollama pull gemma2:2b
ollama pull phi3.5:latest
Enter fullscreen mode Exit fullscreen mode

Installing the Open WebUI Interface

At this point, your private AI is already functional. But if you have the resources for it, we can add a web interface similar to ChatGPT's.

1. Install Docker: the cleanest way to do this is via Docker.

2. Bring up Open WebUI:

docker run -d --network host --name open-webui -v open-webui-data:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e WEBUI_SECRET_KEY=$(openssl rand -hex 32) --restart always ghcr.io/open-webui/open-webui:main
Enter fullscreen mode Exit fullscreen mode

Now open your browser and go to http://localhost:8080 or the IP of the server you're setting this up on.

Open WebUI Running

Connecting with n8n and Other Applications

n8n integration

n8n has native support for Ollama. You don't need to configure manual HTTP requests.

  • In n8n: look for the "Ollama Chat Model" node.
  • Connection: provide your server's URL (e.g., http://localhost:11434 or the server's IP on the local network).
  • Credentials: you generally don't need an API key for local use.
  • Usage: connect this node to an "AI Agent" or "Chain" inside n8n, and that's it: your automation workflow now uses your local Llama 3 or Mistral instead of ChatGPT.

Integration into your application (Laravel, Vue, etc.)

Ollama exposes a very simple REST API. If you've used the OpenAI API before, you'll notice it's nearly identical.

  • Main endpoint: POST http://localhost:11434/api/generate
  • Example payload: 
{
  "model": "llama3",
  "prompt": "Why is the sky blue?",
  "stream": false
}
Enter fullscreen mode Exit fullscreen mode

Test performed using curl 

curl -s -X POST http://10.0.10.240:11434/api/generate -H "Accept: application/json" -H "Content-Type: application/json" -d '{ "model": "phi3.5", "prompt": "Why is the sky blue?", "stream": false }' | jq
Enter fullscreen mode Exit fullscreen mode

With this, you have a fully private AI server up and running — from choosing the right Linux distro, through installing Ollama and Open WebUI, all the way to integrating it with your own applications and automation workflows via n8n. No subscriptions, no third-party API calls, and full control over where your data lives.

Top comments (0)