DEV Community: Ethan Vance

How NVMe-oF Boosts Server Storage Performance

Ethan Vance — Sat, 24 Oct 2026 09:32:29 +0000

Enterprise data centers are currently facing an unprecedented infrastructure challenge. Workloads involving artificial intelligence (AI) training, real-time database analytics, and high-frequency transaction processing demand massive throughput and microsecond-level latency. While modern solid-state drives (SSDs) easily meet these demands at the local server level, a critical bottleneck emerges the moment this storage is shared across a network.

Legacy storage protocols—such as Internet SCSI (iSCSI), Serial Attached SCSI (SAS), and the traditional Fibre Channel Protocol (FCP)—were fundamentally engineered for the era of mechanical spinning disks. When paired with modern flash media, these legacy protocols introduce severe software overhead and artificial bottlenecks, preventing enterprise NVMe SSDs from delivering their true potential over a network.

This is exactly where NVMe over Fabrics (NVMe-oF) restructures the architecture of modern data centers. By extending the highly efficient NVMe protocol beyond the individual server chassis, NVMe-oF enables networked storage to perform identically to direct-attached local drives, virtually eliminating network latency.

In this guide, we will explore the core mechanics of the NVMe-oF protocol, compare its transport options (TCP, RDMA, and FC), and break down the specific networking hardware required to eliminate data center bottlenecks.

What Is NVMe over Fabrics (NVMe-oF)?

To understand NVMe over Fabrics (NVMe-oF), we first need to look at standard NVMe. Originally, the Non-Volatile Memory Express (NVMe) specification was designed exclusively for local, direct-attached storage. It connected ultra-fast SSDs directly to a server's motherboard via the PCIe bus, bypassing older, slower connection methods.

NVMe-oF is a network protocol extension that takes this incredible local performance and pushes it across a network. It enables networked storage to perform exactly like those direct-attached drives, maintaining microsecond-level latency and massive throughput across Ethernet, Fibre Channel, or InfiniBand networks.

Here is how NVMe-oF achieves this seamless network integration:

A Message-Based Architecture: Unlike local NVMe, which uses a memory-mapped model, NVMe-oF relies on remote communication. Commands and responses are packaged into efficient, self-contained units called capsules. This allows multiple small messages to be transmitted as a single, highly efficient unit across the network fabric.
Preserving Massive Parallelism: The true power of NVMe lies in its ability to do many things at once. NVMe-oF preserves this parallel architecture, supporting up to 65,535 I/O queue pairs per controller, with each queue holding up to 65,535 commands. While legacy protocols struggle with serial bottlenecks, NVMe-oF processes thousands of I/O operations simultaneously without breaking a sweat.
Automated Addressing and Discovery: NVMe-oF uses NVMe Qualified Names (NQNs) to identify storage targets. It features standardized discovery mechanisms, meaning host servers can automatically locate and connect to available storage subsystems without tedious manual configuration. It also supports multipath I/O, ensuring both high availability and aggregated performance.

Ultimately, NVMe-oF eliminates the performance penalties that have historically separated networked storage from local flash drives, allowing organizations to scale their storage architecture without sacrificing raw speed.

Legacy Protocols vs. NVMe-oF: Why Traditional Storage Became a Bottleneck

Before NVMe-oF became the enterprise standard, storage networks relied on three primary protocols: Internet SCSI (iSCSI), Serial Attached SCSI (SAS), and Fibre Channel Protocol (FCP). While these protocols were highly reliable and perfectly suited for the era of spinning hard disk drives, they were fundamentally not designed for modern flash storage.

When you place an ultra-fast NVMe SSD behind a legacy protocol, the network connection itself becomes a bottleneck, artificially restricting the drive from reaching its full potential. Here is exactly why traditional storage falls short compared to NVMe-oF:

1. The Translation Penalty (SCSI Overhead)

Legacy protocols like iSCSI and FCP rely on encapsulating old SCSI commands to transfer data. Every time an application requests data, the system has to translate the native NVMe commands into SCSI, transmit them across the network, and then translate them back to NVMe at the storage array. This constant back-and-forth translation consumes valuable CPU cycles and introduces noticeable latency. NVMe-oF eliminates this entirely. It speaks native NVMe end-to-end across the fabric, removing the middleman and the associated processing delays.

2. Command Queues and the Serialization Bottleneck

One of the biggest limitations of legacy protocols is their command queuing depth. A typical SCSI-based protocol supports roughly 256 outstanding commands per device. If an application suddenly issues 10,000 parallel read requests—which is common in AI workloads or big data analytics—SCSI creates a severe serialization bottleneck, forcing commands to wait in line. NVMe-oF, by contrast, supports up to 65,535 I/O queue pairs, with each queue holding up to 65,535 commands. It operates like a massive multi-lane highway, allowing thousands of I/O operations to process concurrently without blocking.

3. Milliseconds vs. Microseconds Latency

Because legacy protocols involve multiple layers of processing—such as SCSI translation, TCP segmentation, and complex IP routing—data delivery is typically measured in milliseconds. While that was fast enough for legacy web hosting, modern transaction processing systems demand much more. By stripping away these heavy protocol stacks, NVMe-oF delivers latency measured in single-digit microseconds, essentially matching the performance of a drive plugged directly into the server motherboard.

By abandoning the legacy SCSI framework entirely, NVMe-oF allows data centers to finally utilize the raw speed of flash media across a distributed network.

Crucial Infrastructure Requirements: Network and Hardware Specs

To achieve direct-attached flash speeds across a distributed network, NVMe-oF demands a specialized, high-performance infrastructure. Running this advanced protocol on legacy networking gear will immediately create bottlenecks, neutralizing its primary benefits.

If you are deploying NVMe-oF in a modern data center or evaluating a dedicated server provider, these are the mandatory hardware specifications you need to look for:

High-Speed Network Bandwidth: Legacy iSCSI environments could survive on 1GbE or 10GbE networks. NVMe-oF, however, generates massive throughput that requires a bare minimum of 25GbE. For true enterprise deployments, data-intensive workloads, and AI training, 100GbE, 200GbE, or even 400GbE networks are the industry standard.
RDMA-Capable SmartNICs: Standard Network Interface Cards (NICs) rely on the host CPU to process network traffic, which reintroduces the exact latency NVMe-oF is designed to eliminate. To achieve single-digit microsecond latency, servers must be equipped with RDMA-capable NICs (SmartNICs). These specialized cards support protocols like RoCE v2 (RDMA over Converged Ethernet) or iWARP, allowing data transfers to bypass the operating system and CPU entirely, handling the heavy lifting at the hardware level.
Enterprise-Grade Network Switches: When you open 64,000 parallel data queues, standard switches can quickly become overwhelmed and drop packets. NVMe-oF—particularly when using RDMA transport—requires enterprise-grade switches configured for Lossless Ethernet. This involves enabling features like Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) to ensure smooth, zero-drop data delivery even during massive traffic spikes.

Upgrading to NVMe-oF is not just about changing the storage drives; it requires a holistic upgrade of the underlying network fabric to ensure the data has a clear, frictionless path to travel.

The Three NVMe-oF Transport Options: TCP, RDMA, and Fibre Channel

NVMe over Fabrics is not a rigid, one-size-fits-all technology. Because different data centers have varying budgets, existing infrastructure, and performance requirements, the protocol was designed to operate over three primary transport types.

Understanding the distinctions between these three transports is critical for selecting the right implementation for your specific workloads.

1. NVMe over TCP (NVMe/TCP)

Introduced to the NVMe-oF specification in 2019, NVMe/TCP encapsulates NVMe commands within standard TCP/IP packets. Its biggest advantage is deployment simplicity.

The Pros: NVMe/TCP runs on standard Ethernet infrastructure. You do not need specialized hardware; any modern Network Interface Card (NIC) capable of 10GbE or higher can support it. This makes it highly cost-effective and the preferred choice for cloud or hybrid-cloud deployments.
The Trade-off: Because it relies on software initiators to handle command processing, it introduces a moderate amount of CPU overhead. Latency typically hovers between 100 and 500 microseconds—which is vastly superior to legacy iSCSI, but not quite as fast as RDMA.

2. NVMe over RDMA (NVMe/RDMA)

When applications demand the absolute lowest latency possible, NVMe/RDMA is the undisputed leader. It uses Remote Direct Memory Access (RDMA) technology—most commonly deployed as RDMA over Converged Ethernet (RoCE v2)—to transfer data directly between the host and storage memory.

The Pros: By bypassing the operating system and host CPU entirely, NVMe/RDMA delivers single-digit microsecond latency (often sub-10 microseconds). It essentially matches the performance of a physical drive plugged directly into the server.
The Trade-off: It requires strict hardware prerequisites, specifically RDMA-capable SmartNICs and enterprise-grade switches configured for lossless network traffic. It is the go-to transport for High-Performance Computing (HPC), AI model training, and high-frequency financial trading.

3. NVMe over Fibre Channel (NVMe/FC)

For decades, enterprise Storage Area Networks (SANs) have relied on Fibre Channel for its rock-solid stability. NVMe/FC allows organizations to map NVMe commands directly onto Fibre Channel frames, completely bypassing the old SCSI translation layer.

The Pros: If an enterprise already has a modern Fibre Channel infrastructure (Gen 5 switches or newer), they can adopt NVMe/FC through simple firmware upgrades without physically replacing their hardware. It delivers excellent, consistent sub-50 microsecond latency with very low CPU overhead.
The Trade-off: It remains tethered to the Fibre Channel ecosystem, meaning it is primarily suited for organizations that want to protect their existing SAN investments rather than transitioning to Ethernet-based fabrics.

Ultimately, the choice comes down to your infrastructure realities: TCP for simplicity and cloud scale, RDMA for uncompromising peak performance, and FC for upgrading established enterprise SANs.

Business Benefits and Real-World Use Cases of NVMe-oF
Upgrading to NVMe over Fabrics isn’t just a technical exercise for system administrators; it drives measurable, bottom-line business outcomes. When storage performance ceases to be a bottleneck, the entire IT infrastructure becomes more agile and cost-effective.

Here is how adopting NVMe-oF directly benefits the enterprise:

Transformative Business Benefits

Infrastructure Consolidation: Historically, applications requiring extreme speed were forced to rely on direct-attached local storage. This led to "stranded capacity"—expensive drives sitting idle in one server while another server ran out of space. NVMe-oF allows you to pool high-performance storage on a centralized array and share it across dozens or hundreds of servers without any performance penalty. This significantly reduces hardware sprawl and maximizes your storage investments.
Reclaiming CPU Resources: Legacy storage protocols are incredibly taxing on server CPUs. By offloading storage processing through efficient NVMe queues (and completely bypassing the CPU with NVMe/RDMA), NVMe-oF frees up expensive processing power. This means your server CPUs can focus 100% on running your actual applications, instantly improving overall system efficiency.
Maximizing Virtual Machine (VM) Density: In virtualized environments (like VMware, KVM, or Hyper-V), NVMe-oF’s massive queue depth is a game-changer. It effectively eliminates the "noisy neighbor" problem, where one VM's sudden I/O burst slows down the rest of the server. With thousands of parallel queues, you can safely host more VMs on a single physical host while guaranteeing consistent, high-tier performance for all of them.

Real-World Use Cases: Where NVMe-oF Shines

While almost any workload benefits from faster storage, NVMe-oF is practically mandatory for modern, data-intensive applications:

AI and Machine Learning Training: Training Large Language Models (LLMs) and advanced AI neural networks requires feeding massive datasets to GPUs as fast as possible. If the storage cannot keep up, the highly expensive GPUs sit idle waiting for data (GPU starvation). NVMe-oF provides the sustained, massive throughput required to keep AI clusters continuously fed, drastically reducing training times.
Real-Time Analytics and Big Data: E-commerce platforms, fraud detection systems, and enterprise databases (like high-traffic SQL or NoSQL clusters) rely on processing streaming data instantly. NVMe-oF ensures that these platforms can query massive, distributed datasets with consistent microsecond response times, enabling real-time decision-making.
High-Frequency Financial Trading: In the financial sector, a delay of a single millisecond can result in millions of dollars in missed opportunities. Trading algorithms require storage that introduces essentially zero latency. For these ultra-competitive environments, NVMe/RDMA is the undisputed standard, providing the absolute lowest possible latency to access market data instantly.

By aligning storage infrastructure with the demands of modern applications, NVMe-oF allows businesses to push the limits of what their software can achieve.

Conclusion: The Future of Storage Networking

NVMe over Fabrics officially eliminates the performance penalties that have historically separated networked storage from direct-attached drives. By breaking free from the constraints of legacy SCSI protocols, organizations can finally consolidate their infrastructure, reclaim valuable CPU resources, and deliver the uncompromised microsecond latency that modern, data-intensive applications demand. Whether you implement it via TCP, RDMA, or Fibre Channel, NVMe-oF is undeniably the new standard for enterprise storage.

Run Ollama with GPU on Ubuntu Using Docker and Open WebUI

Ethan Vance — Fri, 09 Oct 2026 06:19:30 +0000

Running large language models locally gives developers more control over their AI environment. With Ollama, Docker, and Open WebUI, you can set up a self-hosted AI chatbot on an Ubuntu server with NVIDIA GPU acceleration.

In this guide, we'll cover the essential components required to get started with GPU-powered local AI.

What You'll Need

Before deploying Ollama, prepare the following:

Ubuntu 22.04 or 24.04 LTS
A compatible NVIDIA GPU
NVIDIA drivers
Docker Engine
NVIDIA Container Toolkit
Sufficient GPU VRAM, RAM, and storage for your chosen model

Step 1: Update the server and check your GPU

sudo apt update && sudo apt upgrade -y

Confirm Ubuntu can see your GPU:

lspci | grep -i nvidia

You should see a line with your GPU model. If nothing appears, the GPU is not detected: check with your hosting provider before continuing.

Step 2: Install the NVIDIA driver

List the recommended driver for your GPU:

sudo ubuntu-drivers devices

Install it automatically (this picks the recommended version):

sudo ubuntu-drivers autoinstall

Reboot:

sudo reboot

After reconnecting, verify:

nvidia-smi

You should see a table showing your GPU name, driver version and VRAM. If you see it, the driver works.

⚠️ Secure Boot warning If Secure Boot is enabled in BIOS, the NVIDIA kernel module may refuse to load and nvidia-smi will fail. Either disable Secure Boot in BIOS/IPMI, or enroll a MOK key when the installer asks. See Troubleshooting, Problem 1.

Step 3: Install Docker

curl -fsSL https://get.docker.com | sudo sh

Allow your user to run Docker without sudo:

sudo usermod -aG docker $USER

Log out and log back in (or run newgrp docker), then test:

docker run --rm hello-world

Step 4: Install the NVIDIA Container Toolkit

This is the piece that lets Docker containers use your GPU.
Add NVIDIA's repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Install it:

sudo apt update
sudo apt install -y nvidia-container-toolkit

Tell Docker to use the NVIDIA runtime, then restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU access inside a container

docker run --rm --gpus all ubuntu nvidia-smi

If you see the same GPU table as in Step 2, GPU passthrough to Docker works. Do not continue until this test passes

Step 5: Create the project folder

mkdir -p ~/ai-server && cd ~/ai-server

5.1 Create the docker-compose.yml

nano docker-compose.yml

Paste this:
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=30m        # keep model loaded in VRAM for 30 min
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    # No "ports:" section on purpose. Ollama has no login,
    # so it must NOT be reachable from the internet.

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - ENABLE_SIGNUP=${ENABLE_SIGNUP:-true}
    volumes:
      - open_webui_data:/app/backend/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: unless-stopped
    depends_on:
      - open-webui
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config

volumes:
  ollama_data:
  open_webui_data:
  caddy_data:
  caddy_config:

Save with Ctrl+O, Enter, then Ctrl+X.

5.2 Create the .env file

Generate a random secret key:

echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > .env
echo "ENABLE_SIGNUP=true" >> .env

5.3 Create the Caddyfile

Option A: with a domain (recommended, free automatic HTTPS). First point your domain's DNS A record to your server IP, then:

cat > Caddyfile << 'EOF'
ai.yourdomain.com {
    reverse_proxy open-webui:8080
}
EOF

Replace ai.yourdomain.com with your real domain.

Option B: no domain (testing only, plain HTTP).

cat > Caddyfile << 'EOF'
:80 {
    reverse_proxy open-webui:8080
}
EOF

Step 6: Open the firewall

Only SSH, HTTP and HTTPS are needed:

sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

⭐ Important: Docker can bypass UFW for any port you publish with ports:. That is why Ollama has no published port in the compose file above. Never publish 11434 to the internet.

Step 7: Start everything

docker compose up -d

Check that all three containers are running:

docker compose ps

Step 8: Download your first model

docker exec -it ollama ollama pull llama3.1:8b

Test it from the command line:

docker exec -it ollama ollama run llama3.1:8b "Explain what a dedicated server is in two sentences."

Verify the model is using the GPU (the most important check)

While a model is loaded, run:

docker exec -it ollama ollama ps

Look at the PROCESSOR :

100% GPU: Perfect: the whole model is in VRAM
40%/60%: Model is too big for VRAM, partly on CPU (slow). Use a smaller model or lower quantization
CPU/GPU: Model is too big for VRAM, partly on CPU (slow). Use a smaller model or lower quantization
100% CPU: GPU is not being used. See Troubleshooting, Problem 3

You can also watch GPU usage live in a second terminal:

watch -n 1 nvidia-smi

Step 9: Open the chat interface and create your admin account

Visit https://ai.yourdomain.com (or http://YOUR_SERVER_IP for Option B).

Click Get Started.
Create your account. The first account becomes the administrator.
Pick your model from the dropdown at the top and start chatting.

Lock down registration (do this right after creating your admin)
Otherwise anyone who finds your URL can sign up.

sed -i 's/ENABLE_SIGNUP=true/ENABLE_SIGNUP=false/' .env
docker compose up -d

⚠️ Note In some Open WebUI versions, the signup setting is also stored in the admin panel after first start. If signups are still open, go to Admin Panel → Settings → General and turn off Enable New Sign Ups.

Which model should I run? (VRAM guide)

Rule of thumb for 4-bit quantized models (the Ollama default): model size in GB is roughly VRAM needed, plus 1 to 2 GB for context.

Your GPU VRAM: 6 to 8 GB
Comfortable model size: 3B to 8B
Example use: Chat, summaries, simple coding help

Your GPU VRAM: 12 to 16 GB
Comfortable model size: 8B to 14B
Example use: Better reasoning, coding

Your GPU VRAM: 24 GB
Comfortable model size: up to ~30B
Example use: Strong general assistant

Your GPU VRAM: 48 GB
Comfortable model size: up to ~70B
Example use: Near top-tier open models

Your GPU VRAM: 80 GB+
Comfortable model size: 70B+ with long context
Example use: Heavy or multi-user workloads

Browse available models at ollama.com/library and pull any with:

docker exec -it ollama ollama pull MODEL_NAME:TAG

List installed models, and delete ones you no longer need:

docker exec -it ollama ollama list
docker exec -it ollama ollama rm MODEL_NAME:TAG

Using the API from your own apps

Ollama is OpenAI-compatible. Because we did not publish the port, call it from inside the Docker network, or temporarily test from the server itself:

docker exec -it ollama curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

For remote API access, put it behind an authenticated gateway or use a VPN such as WireGuard. Never expose raw Ollama to the internet.

Troubleshooting

Problem 1 - nvidia-smi says "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"

Likely causes and fixes:

You have not rebooted after installing the driver. Run sudo reboot.
Secure Boot is on. Check with mokutil --sb-state. If enabled, disable Secure Boot in BIOS/IPMI or enroll the MOK key.
Kernel and driver mismatch after a kernel update. Reinstall:

sudo apt install --reinstall linux-headers-$(uname -r)
sudo ubuntu-drivers autoinstall
sudo reboot

The open-source nouveau driver is loaded. Check with lsmod | grep nouveau. If it shows, blacklist it:

echo -e "blacklist nouveau\noptions nouveau modeset=0" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u && sudo reboot

Problem 2 - docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

The NVIDIA Container Toolkit is missing or Docker was not configured. Re-run:

sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then retest: docker run - rm - gpus all ubuntu nvidia-smi

Problem 3 - Ollama is running on CPU, not GPU ("Ollama not using GPU")

Work through this checklist in order:

Does nvidia-smi work on the host? If not, go to Problem 1.
Does docker run --rm --gpus all ubuntu nvidia-smi work? If not, go to Problem 2.
Does your compose file contain the deploy.resources.reservations.devices block for the ollama service? Without it, the container gets no GPU.
Check the Ollama logs for GPU detection messages:

docker logs ollama 2>&1 | grep -i -E "gpu|cuda|vram"

Is the model simply too large for your VRAM? Run ollama ps. A split such as 30%/70% CPU/GPU means it is overflowing. Use a smaller model.

Problem 4 - GPU works at first, then Ollama falls back to CPU after some time

This is a known issue on some Linux setups where the GPU driver state is lost. Try reloading the UVM module and restarting the container:

sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
docker restart ollama

If it keeps happening, make sure you are on a recent driver version and keep the system updated.

Problem 5 - Open WebUI shows no models, or "Ollama connection error"

Confirm OLLAMA_BASE_URL=http://ollama:11434 is set and both containers are in the same compose project.
Check Ollama is up: docker compose ps and docker logs ollama.
Confirm you actually pulled a model: docker exec -it ollama ollama list.
Restart: docker compose restart open-webui.

Problem 6 - HTTPS does not work / Caddy shows certificate errors

DNS A record must point to your server IP before starting Caddy. Check with dig +short ai.yourdomain.com.
Ports 80 and 443 must be open in UFW and any provider-level firewall.
Read the logs: docker logs caddy.
If you hit certificate rate limits from repeated failed attempts, wait and retry, or test with Option B first.

Problem 7 - "CUDA out of memory" or the model fails to load

Use a smaller model or a smaller quantization.
Reduce context length in Open WebUI (Settings → Advanced Params → Context Length).
Make sure no other process is holding VRAM: nvidia-smi shows processes using the GPU.

Problem 8 - Responses are very slow

Run ollama ps: if it is not 100% GPU, that is your cause.

Large context windows slow things down. Lower the context length.
First request after idle is slower because the model loads into VRAM. OLLAMA_KEEP_ALIVE=30m (already set) reduces this.

Problem 9 - Port 3000 or 11434 is reachable from the internet

You should not publish these ports. Remove any ports: entries for ollama and open-webui, then run docker compose up -d. Remember Docker bypasses UFW for published ports.

Maintenance

Update to the latest versions:

cd ~/ai-server
docker compose pull
docker compose up -d
docker image prune -f

Back up your data (chats, users, settings):

docker run --rm -v ai-server_open_webui_data:/data -v $(pwd):/backup ubuntu \
  tar czf /backup/open-webui-backup.tar.gz /data

(The volume name is your folder name plus _open_webui_data. Check yours with docker volume ls.)

Useful daily commands:

See running containers:docker compose ps
View logs:docker compose logs -f
Loaded models:docker exec -it ollama ollama ps
Stop everything:docker compose down
Start everything:docker compose up -d
GPU usage:nvidia-smi

Security checklist

SSH key login only (disable password login)
Ollama port is not published
Sign-ups disabled after creating the admin
HTTPS enabled via Caddy
WEBUI_SECRET_KEY set to a random value
Firewall allows only ports 22, 80, 443
Regular updates and backups scheduled

How Bare-Metal Dedicated GPU Servers Prepare You for NVIDIA’s CUDA Rust

Ethan Vance — Thu, 08 Oct 2026 07:03:17 +0000

NVIDIA is fundamentally changing native GPU programming by bringing Rust directly to GPU kernels, setting a new standard for AI systems development. By compiling Rust natively to PTX, developers can now eliminate complex memory leaks, data races, and runtime crashes entirely at compile time. This shift introduces two primary development tracks—the granular SIMT model (cuda-oxide) and the compiler-optimized Tile abstraction (cutile-rs)—giving engineers unprecedented memory safety without sacrificing bare-metal performance.

However, transitioning to this memory-safe ecosystem introduces new infrastructure bottlenecks. Heavy compilation pipelines and bleeding-edge toolchains require massive CPU resources, stable OS-level configurations, and completely isolated environments. In this guide, we break down how these two CUDA Rust tracks work, how strict borrowing rules prevent GPU crashes, and why properly compiling and benchmarking these next-generation workloads strictly demands the unthrottled power and root access of bare-metal dedicated GPU servers.

The Shift to Native Rust for GPU Kernels

In September 2026, NVIDIA announced a major shift in native GPU programming: the introduction of CUDA Rust. For years, writing GPU kernels meant relying exclusively on CUDA C++ or CUDA Python. However, the systems layer of AI—spanning inference engines, serving infrastructure, and drivers—is rapidly changing, and Rust is becoming the industry standard.

The reason is simple. Rust catches entire classes of concurrency and memory bugs at compile time without giving up bare-metal performance. NVIDIA is already driving this shift; the Nova Linux driver is written in Rust, and NVTX has Rust bindings.

Until recently, the GPU kernel itself was the exception. You could launch kernels from Rust, but the kernel code had to be written in another language. NVIDIA CUDA Rust finally closes that gap. Developers can now write GPU kernels directly in Rust and compile them natively to PTX (Parallel Thread Execution), rather than wrapping code from somewhere else.

However, because this technology is still in early alpha and requires heavy compilation power, testing it on a standard laptop or a shared VPS is a bottleneck. To compile complex Rust code and run early-stage kernels without crashing, developers need the isolated, high-performance environment of a bare-metal dedicated GPU server.

Two Approaches to CUDA Rust: SIMT vs. Tile

NVIDIA provides two distinct programming tracks for writing GPU kernels in Rust, matching the existing models in CUDA. The choice depends on how much manual control you need over the hardware versus letting the compiler optimize the execution architecture.

1. The SIMT Track (cuda-oxide)

SIMT (Single Instruction, Multiple Threads) is the traditional GPU programming model familiar to developers who write in CUDA C++ or numba-cuda. In this track, you write code dictating exactly what a single thread does, and the GPU launches thousands of them in parallel.

NVIDIA implements this through cuda-oxide, a custom rustc codegen backend. It intercepts compilation, routing functions through Rust MIR, the Pliron IR framework, and LLVM IR, ultimately translating them into PTX.

Key characteristics of the SIMT track:
Granular Hardware Control: You manage thread indexing and memory allocation directly.
Memory Safety Mechanism: Because standard Rust prevents multiple threads from holding a mutable reference (&mut) to the same array, cuda-oxide introduces DisjointSlice. This custom type splits a single mutable borrow into per-thread pieces, granting each thread exclusive access to its own element.
System Requirements: This is a low-level approach requiring Linux, a GPU with Compute Capability 8.0+, the CUDA 12.x toolkit, and a pinned nightly Rust toolchain.

2. The Tile Track (cutile-rs)

The Tile model represents a newer, higher-level abstraction. Instead of dealing with individual scalar values and manual thread counts, you perform computations on tiles (sub-tensors) of data.

Powered by the cutile-rs crate, each tile block runs the kernel body as a single logical thread. The #[cutile::module] macro embeds the kernel's AST into the host binary, which is then JIT-compiled through CUDA Tile IR exactly when it is launched.

Key characteristics of the Tile track:

Compiler Optimization: The Tile IR compiler decides how your tiles map onto the physical GPU architecture, meaning your source code isn't locked into architecture-specific choices.
Simplified Safety Constraints: There is no need for DisjointSlice. The Tile track uses a .partition() method on the host side. This automatically partitions data chunks, assigning exclusive ownership to each tile block, which satisfies Rust's strict &mutborrowing rules by default.
Lighter Requirements: Unlike SIMT, the Tile track works on stable Rust (1.89 or newer). It requires CUDA 13.3 and Compute Capability 8.0+, but eliminates the need for nightly toolchains and custom LLVM setups.

SIMT vs. Tile: Which CUDA Rust Model Should You Choose?

When starting a new project, NVIDIA’s guidance is straightforward: reach for the Tile track first.

Because the Tile IR compiler automatically decides how to map tiles onto the underlying GPU architecture, your source code remains clean and hardware-agnostic. You only need to drop down to the SIMT track when your specific workload demands manual thread indexing, hyper-specific control over the execution grid, or custom shared memory management.

Ultimately, the choice doesn't lock you in. NVIDIA plans to support inter-language interoperability, allowing you to use the CUDA exposure that best fits your existing stack.

The Real Advantage: Compile-Time Memory Safety

Regardless of whether you choose SIMT or Tile, the biggest advantage of writing GPU kernels in Rust is what happens before the code ever hits the hardware.

In traditional C++, having thousands of parallel threads accessing the same memory buffers in no guaranteed order is a recipe for disaster. If two threads hit the same address and one is writing, the execution order dictates the result. These types of data races and aliasing bugs rarely reproduce on demand—they often pass unit tests only to fail catastrophically in production.

Rust fixes this by construction. Both cuda-oxide and cutile-rs enforce strict borrowing rules at compile time: inputs can be shared (&), but an output buffer (&mut) must belong exclusively to a single writer.

If a developer accidentally passes an output buffer as one of its own inputs, the Rust compiler immediately intervenes, halting the build with a borrowing error (e.g., error[E0502]: cannot borrow as mutable because it is also borrowed as immutable). By catching classic aliasing mistakes at compile time, CUDA Rust eliminates an entire class of runtime GPU bugs before they ever happen.

How CUDA Rust Supercharges GPU Servers

The introduction of CUDA Rust isn't just a syntax update for developers—it is a massive upgrade for how efficiently a GPU server operates. By bringing Rust’s famous "fearless concurrency" to the GPU, the underlying server hardware gains completely new operational advantages:

1. 100% Hardware Utilization (Zero Runtime Overhead)

Historically, running AI workloads meant relying on Python wrappers or heavy abstraction layers, which inevitably waste server CPU cycles and create latency. CUDA Rust compiles natively to PTX (Parallel Thread Execution). This means there is zero runtime overhead. Every single compute cycle on your GPU server is dedicated purely to processing the AI model, resulting in drastically faster inference times.

2. Crash-Free Uptime (No More Memory Leaks)

One of the biggest issues with traditional C++ GPU kernels is memory mismanagement—specifically data races and aliasing bugs that cause Out-of-Memory (OOM) errors and server application crashes. Because CUDA Rust enforces strict borrowing rules at compile time, these memory errors are physically impossible in production. Your GPU servers will run heavy workloads 24/7 with rock-solid stability, completely eliminating the need for unexpected reboots.

2. Crash-Free Uptime (No More Memory Leaks)

A modern GPU server runs thousands of threads simultaneously. When multiple threads try to write to the same memory block, it corrupts the data. CUDA Rust introduces mechanisms like DisjointSlice (in SIMT) and automatic partitioning (in Tile) that give each thread exclusive access to its own data chunk. This guarantees that massive parallel computations run flawlessly without data corruption.

The MIG Servers Solution

This shift is exactly why shared cloud instances and local workstations are no longer enough. To test and deploy NVIDIA’s CUDA Rust efficiently, developers need Bare-Metal Dedicated GPU Servers.

We provide unmetered, dedicated environments engineered for deep-level systems programming and AI inference. When you rent a dedicated GPU server from us, you get:

Compute Capability 8.0+ Hardware: Ready-to-deploy enterprise GPUs (including NVIDIA A100, H100, and high-end RTX series) that perfectly match CUDA Rust’s strict requirements.
100% Root Access: Install your pinned nightly toolchains, custom CUDA drivers, and cargo-oxide environments without restrictions.
Zero Virtualization Overhead: True bare-metal performance ensures that when you benchmark your Rust kernels, you are measuring the hardware's raw power

The Future of GPU Kernels is Memory-Safe

NVIDIA’s push into native Rust for GPU programming is not just an experimental side project; it is the new baseline for systems-level AI development. By allowing developers to compile Rust directly to PTX, NVIDIA is closing the longest-standing gap in the AI stack. The memory safety that Rust brings to host-side infrastructure—like drivers and inference engines—is finally moving down to the kernel level.

While both cuda-oxide and cutile-rs are currently in early testing, the engineering trajectory is clear. The days of hunting down elusive data races, manual aliasing bugs, and execution order crashes in thousands of parallel threads are coming to an end.

For developers, the transition starts now. Whether you choose the granular hardware control of the SIMT track or the compiler-optimized abstraction of the Tile track, understanding how strict borrowing rules apply to GPU execution is going to be a mandatory skill. The ecosystem will inevitably shift toward these memory-safe constructs for building the next generation of scalable, high-performance AI runtimes.

AMD EPYC 9006 "Venice" CPUs: Architecting the Agentic AI Stack

Ethan Vance — Thu, 08 Oct 2026 05:43:27 +0000

Modern AI infrastructure has rapidly transitioned from static, monolithic training and inference clusters to highly dynamic Agentic AI pipelines. Unlike traditional models that rely on predictable compute cycles, agentic systems process single requests through complex, multi-stage execution chains encompassing data retrieval, external tool calls, and real-time code execution.

This architectural shift creates a highly variable compute footprint. An agentic workflow is inherently shape-shifting, demanding fluctuating computational capacities and dynamic resource allocation across different server nodes during a single execution loop.

Consequently, rigid hardware deployments are no longer viable for modern data centers. Supporting autonomous AI agents requires strict architectural flexibility at every layer of the technology stack. For enterprise environments, this necessitates highly adaptable server CPUs capable of processing diverse, shifting computational profiles without introducing latency or performance bottlenecks into the agentic pipeline.

The Converged Compute Demands of Agentic Workflows

Enterprise environments have historically managed varying system profiles for distinct workloads, provisioning separate infrastructure for databases, virtualization, real-time analytics, web services, and high-performance technical computing. Agentic AI disrupts this traditional model by converging multiple computational profiles into a single, cohesive execution pipeline.

Instead of isolating tasks, an AI agent might query a vector database, execute analytical Python code, and render a web service response within milliseconds. To support this, modern server architecture must natively handle diverse, concurrent workloads without the latency overhead of routing tasks across fragmented, specialized clusters.

Enter the 6th Gen AMD EPYC™ 9006 "Venice" Architecture

Addressing the critical need for multi-workload flexibility, AMD’s newly evaluated 6th Gen AMD EPYC™ 9006 "Venice" server CPUs deliver a comprehensive solution. Rather than relying on narrow optimizations for specific tasks, the Venice architecture is engineered to execute across general-purpose, enterprise, cloud-native, AI, and high-performance computing (HPC) workloads with exceptional efficiency.

The AMD EPYC 9006 series represents a highly scalable processor portfolio built upon a common software foundation. The lineup spans from lightweight 8-core deployments tailored for edge environments up to massive 256-core flagship processors designed for rack-scale AI host nodes.

This tiered architectural approach ensures that each specific role within an agentic pipeline can leverage the exact processor profile it requires, completely eliminating the need to maintain separate operating environments. Currently in production, the "Venice" CPUs are actively being integrated into major OEM platforms, with leading cloud service providers slated for deployment later this year.

Performance Benchmarks: EPYC 9996 vs. Competing Architectures

To quantify this architectural flexibility, extensive testing evaluates the flagship AMD EPYC™ 9996 server CPU against major market alternatives across a highly diverse set of AI, enterprise, and technical workloads.

The data below reveals significant throughput and per-core performance advantages:

SPECrate® 2026 Integer Performance: The EPYC 9996 delivers 1.2x higher per-core performance and an impressive 2.24x platform-level performance advantage over an Nvidia Vera-based platform.
Enterprise & Cloud-Native Workloads: Across highly concurrent applications—including server-side Java, OpenSSL cryptography, MongoDB, Redis, NGINX web serving, and transaction processing—AMD testing reports substantial gains ranging from 2.4x to 3.7x.
High-Performance Computing (HPC): When benchmarked against the Intel® Xeon® 6980P processor in complex compute-heavy tasks like molecular dynamics, materials modeling, and weather forecasting, the EPYC 9996 maintains a 1.8x to 3.13x performance lead.

Crucial Data Center Metric: In a modeled 100-kilowatt rack configuration, the AMD EPYC 9996 CPU is estimated to deliver 3.4 times the total throughput of a comparable Nvidia Vera-based platform, drastically improving power-to-performance ratios for enterprise data centers.

Balancing Per-Core Performance with Massive Core Density

The visual architecture of the AMD EPYC processor illustrates how this performance is physically achieved on the server motherboard. Notice the dense thermal and memory interface layout surrounding the core silicon—this physical design is what enables the chip to balance two historically competing server requirements: latency and concurrency.

In an agentic AI pipeline, strong loaded per-core performance is mandatory for accelerating latency-sensitive sequential work, such as rapid database querying or executing complex logical inferences.

Simultaneously, massive core density is required to support the extreme concurrency of agentic tools, allowing hundreds of parallel API calls and vector searches to run without bottlenecking rack-level throughput. By utilizing a scalable processor portfolio, enterprise customers can precisely optimize for both latency and concurrency across their server fleet without imposing compromised hardware constraints on specific stages of their AI workflows.

Future-Proofing Your Infrastructure for Varied Workloads
The trajectory of Agentic AI guarantees one absolute certainty: workloads will only continue to diversify. As AI agents gain the ability to interact with more complex external toolchains, handle multimodal data streams, and execute multi-step logic autonomously, the computational demands placed on servers will become increasingly unpredictable.

The strategic answer to a highly varied software workload has never been rigid, single-purpose hardware. Future-proofing enterprise AI infrastructure requires deploying a compute foundation where flexibility is natively built into the silicon. By leveraging adaptable, high-density architectures like the AMD EPYC 9006 series, organizations can confidently scale their data centers today, knowing their server nodes possess the exact shape and power required for whatever AI workflows emerge tomorrow.

Intel Xeon 6 Dedicated Servers: Performance & Memory Guide

Ethan Vance — Fri, 18 Sep 2026 04:20:43 +0000

As workloads like AI, cloud-native applications, and large in-memory databases explode, the demands placed on server infrastructure have never been higher. To meet these intensive computing challenges, the Intel Xeon 6 processor lineup represents a massive architectural shift.

For those of us architecting or managing dedicated server hosting, raw compute power alone doesn't cut it anymore. We need efficiency, maximum scalability, and lightning-fast data transfer to prevent bottlenecks.

Here is a deep dive into how the new Xeon 6 architecture and DDR5 memory are redefining data center performance.

Intel Xeon 6 splits into E-cores (for 3:1 density consolidation) and P-cores (for raw speed/AI). The biggest game-changer is the memory controller, unlocking 6000 MT/s in a 2 DIMM per channel (2DPC) configuration—crushing the memory bottlenecks that plague modern virtualization and database workloads.

The Architecture Split: E-Cores vs. P-Cores

Unlike previous generations, Intel Xeon 6 processors are split into two distinct series. This allows server architects to provision exact hardware for specific workload needs.

🟢 Efficient-cores (E-cores): Sierra Forest-SP

Built primarily for cloud-native and scale-out workloads, E-cores prioritize rack density and power efficiency.

The Benefit: Run significantly more instances and VMs per server rack while keeping thermal and energy limits strictly controlled.
The Stat: Delivers an impressive 3:1 rack consolidation compared to older 2nd Gen Intel Xeon processors, drastically lowering TCO.

🔵 Performance-cores (P-cores): Granite Rapids

Engineered for traditional, heavy-duty enterprise applications requiring predictable, low latency.

The Benefit: Perfect match for transaction-heavy systems, real-time analytics, and massive SQL/NoSQL databases.
The Stat: When paired with advanced MRDIMMs, Xeon 6 with P-cores delivers up to 5.5x better AI inferencing performance compared to competing AMD EPYC processors.

The 2DPC Memory Breakthrough: Why 6000 MT/s Matters

CPU cores are useless if they are starved of data. Memory speed dictates total data throughput.

Typically, populating servers with more memory modules (increasing capacity) forces the system to drop to slower RAM clock speeds due to electrical loads.

Intel Xeon 6 (P-cores) introduces a unique advantage using a 2 DIMM per channel (2DPC) configuration. This achieves maximum RAM capacity without sacrificing CPU performance.

Intel has pushed the boundaries of DDR5 server memory by enabling 2DPC speeds of 6000 MT/s (on Xeon 6700/6500P). For perspective, competing platforms are often capped at 4400 MT/s in similar high-density setups.

Real-world impact:

25% better average performance vs. 5th Gen Intel Xeon at equal core counts.
2.18x performance jump in heavy media transcode workloads (SVT-AV1) compared to a 1DPC setup at 6400 MT/s.
Firmware Upgradeable: Running populated memory at 5200 MT/s? You can unlock the full 6000 MT/s speed via a simple software update—zero hardware swaps required.

4 Operational Benefits for the Data Center

Upgrading to Xeon 6 dedicated servers translates to measurable operational wins:

Prevents Core Starvation: Faster 2DPC DDR5 speeds ensure your expensive CPUs don't sit in wait-states. Critical for SAP HANA, Redis, and large SQL clusters.
Maximizes VM Density: Pack massive RAM capacities into virtualized environments, increasing revenue-generating VMs per node without performance dips.
Isolates "Noisy Neighbors": 6000 MT/s provides enough bandwidth headroom to buffer tenants from each other's data spikes in multi-tenant environments.
Lowers TCO: Faster task completion allows servers to return to idle states sooner. At 40% utilization, Xeon 6 (P-cores) delivers 1.9x better performance-per-watt vs. the previous generation.

DDR5 & Enterprise-Grade RAS Features

DDR4 is officially phased out. Xeon 6 supports a variety of advanced DDR5 modules:

RDIMMs: Up to 6400 MT/s for standard enterprise workloads.
MRDIMMs: Multiplexed Rank DIMMs pushing bandwidth up to 8800 MT/s on the 6900-series.
3DS RDIMMs: Up to 256GB per module for ultimate capacity.

To guarantee uptime, the platform includes robust RAS (Reliability, Availability, and Serviceability) features:

SDDC (Single Device Data Correction): Detects/corrects device-level errors.
Post Package Repair: Auto-repairs hard/soft bit errors without reboots.
Memory Sparing & Mirroring: Live duplicates across DIMMs for redundancy.
ECC: Advanced single/multi-bit error correction.

Which Workload Fits Where?

AI & HPC: Go with the Xeon 6900-series (12-channel DDR5) for immense bandwidth.
Enterprise DBs & Analytics: P-core Xeon 6 processors for low-latency throughput.
Virtualization: 2DPC 6000 MT/s configs for max VM consolidation.
Cloud-Native Deployments: Xeon 6700E-series for power efficiency and density.

The shift to DDR5 and bifurcated core architectures gives us more tools than ever to right-size infrastructure.

How to Fix High RAM Usage on a Linux Dedicated Server

Ethan Vance — Fri, 04 Sep 2026 10:56:04 +0000

Experiencing sluggish performance, unresponsive terminals, or unexpected application crashes on your MIG servers infrastructure usually points to memory exhaustion.

This guide provides the exact diagnostic commands and mitigation strategies to stabilize your Linux environment and permanently resolve memory leaks, using production-safe best practices.

1. Evaluate Current Memory Availability

Before taking action, you must understand how your server distributes its resources. Run the following command to get a snapshot of your system's memory:

free -h

Total
Description: Physical RAM installed on your server.
Actionable Insight: Baseline reference for your hardware capacity.

Available
Description: Memory currently free and ready for new processes.
Actionable Insight: If consistently near zero (and swap is growing), your server is struggling.

Buff/Cache
Description: RAM used by the Linux kernel to cache files.
Actionable Insight: High numbers are healthy; Linux frees this automatically when apps need RAM.

Swap
Description: Disk space used as emergency overflow RAM.
Actionable Insight: Sustained, active swap usage severely degrades performance.

2. Identify Resource Hogs

Locate the specific applications draining your RAM using built-in Linux utilities.

Interactive Monitoring: Run top and press Shift + M to sort active tasks by memory usage. For a more readable interface, install and run htop (press F6 to sort by memory).
Targeted Process Snapshot: To immediately print the top 10 memory-consuming processes directly to your terminal:

ps aux --sort=-%mem | head -10

Track Memory Leaks: If you suspect an application's Resident Set Size (RSS) is growing endlessly without releasing memory, monitor its Process ID (PID) over a 10-minute window:

for i in $(seq 1 10); do ps -o pid,rss,vsz -p <PID>; sleep 60; done

3. Immediate Remediation Actions

If your server is actively freezing, apply these commands to safely recover system resources.

Gracefully Terminate Rogue Processes: Send a standard termination signal to allow the application to clean up and shut down properly.

kill <PID>

Crucial Warning: Only use kill -9 as an absolute last resort if the process is completely frozen. It forces an immediate shutdown without cleanup, which can cause data corruption in databases or file writes.>

Restart Application Services: Restarting daemons (like Nginx, MySQL, or Node.js) flushes their allocated memory pools.

sudo systemctl restart <service_name>

Diagnostic Cache Clearing (Use Sparingly): Linux intentionally uses free RAM to cache files for better performance. Dropping the cache can prove whether RAM is truly locked up by applications, but doing this routinely will actually hurt server performance as the kernel is forced to re-read files from the disk.

sync && echo 1 | sudo tee /proc/sys/vm/drop_caches

4. Long-Term Server Optimization

Prevent future bottlenecks by tuning kernel parameters and configuring workload-specific limits.

Lower Swappiness: Depending on your workload, you can encourage the kernel to favor physical RAM before relying on slow disk swap. Open /etc/sysctl.conf, append vm.swappiness=10, and apply with sudo sysctl -p.
Deploy Emergency Swap Space: Swap prevents fatal Out-Of-Memory (OOM) crashes by providing overflow space, though it is not a replacement for sufficient physical RAM. Size this appropriately for your server (e.g., 2GB to 4GB):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Enforce Workload-Based Memory Limits: Prevent a single memory leak from taking down the entire server by setting Systemd limits. Run sudo systemctl edit <service_name> and configure limits based on your app's actual needs:

[Service]
MemoryMax=4G
MemorySwapMax=0

(Note: Set MemoryMax based on actual profiling; if set too low, Systemd will routinely kill your service. Setting MemorySwapMax=0 strictly prevents the service from using swap, which guarantees an OOM kill if physical RAM limits are exceeded).

Audit and Disable Unused Services: Background services consume baseline RAM. Identify unnecessary software and disable it from booting:

sudo systemctl disable <service_name>

Discussion:

What is your go-to terminal command when a server starts lagging? Drop your tips in the comments below! 👇

Dedicated Servers for AI Inference: CPU, GPU, RAM and Network Requirements

Ethan Vance — Thu, 03 Sep 2026 10:05:30 +0000

You can install the most powerful GPU on the market into a server, load a large language model (LLM), and still experience severe performance bottlenecks.

A dedicated server for AI inference can have a high-end accelerator and still perform poorly because of insufficient VRAM, KV-cache pressure, weak CPU resources, slow storage, or PCIe limitations. A GPU alone does not determine AI inference performance.

AI inference is fundamentally a system-level workload. While the GPU is critically important, your CPU, system RAM, NVMe storage, networking, and interconnects must be perfectly balanced around the specific model and workload you are deploying.

What Is AI Inference and Why Does Infrastructure Matter?

To properly size an AI inference server, you must separate inference from training. AI training is a massive, highly parallel batch process. AI inference—whether it is real-time generative AI, API model serving, or automated batch inference—is the execution phase where that trained model generates responses to live prompts.

In a production inference environment, performance is dictated by:

Time to First Token (TTFT): How fast the system processes the prompt and returns the very first piece of the answer.
Inter-Token Latency (ITL): The microsecond delay between each generated token.
Tokens Per Second (Throughput): The total volume of output the server can generate across all concurrent users.

Concurrency: How many independent requests the hardware can handle simultaneously before queue times spike.

Optimizing for these metrics means recognizing that an AI inference server is a complete data pipeline. Fast NVMe storage, efficient PCIe topology, high-bandwidth GPU interconnects, and robust power systems allow those four core resources to function without I/O bottlenecks.

CPU Requirements for AI Inference Servers

A GPU-accelerated inference server still depends on the CPU for request processing, tokenization, preprocessing, orchestration, and data movement. It is a very common mistake to over-invest in high-end GPUs while severely bottlenecking the system with an underpowered processor.

Before a prompt ever reaches the GPU, and after the GPU generates a response, the CPU must actively manage:

Incoming API requests and network payloads.
Tokenization and post-processing.
Dynamic request batching and scheduling for the GPU.
Retrieval-Augmented Generation (RAG) pipelines and vector searches.

How Many CPU Cores Do You Need?

Small LLM (e.g., 7B-8B models)

CPU Requirement: Moderate

High-concurrency API

CPU Requirement: High

RAG / Agentic AI workflows

CPU Requirement: High

Multi-GPU inference

CPU Requirement: Very High

The CPU-to-GPU Balance: High-performance environments require enterprise-grade processors (such as AMD EPYC or Intel Xeon) not just for their core counts, but to provide the extensive PCIe lanes required to keep modern GPUs saturated.>

GPU Requirements for AI Inference

Do not select an inference GPU based only on raw compute performance (TFLOPs). To determine if an accelerator can handle your specific AI workload, evaluate a complete matrix of features:

VRAM Capacity: Can it hold the model and the required runtime memory?
Memory Bandwidth: How fast can it move data during token generation?
Precision Support: Does it natively accelerate formats like FP8 or INT4?
Power Consumption: Can the server chassis sustainably cool the GPU under 24/7 loads?
GPU Interconnect: Does it support high-speed communication (like NVLink)?

How Much GPU Memory Does AI Inference Need?

Baseline memory depends entirely on the precision format (quantization) you choose:

FP32 (Full Precision)

Approximate Memory per Parameter: 4 bytes

FP16 / BF16 (Half Precision)

Approximate Memory per Parameter: 2 bytes

INT8 (Quantized)

Approximate Memory per Parameter: 1 byte

4-bit (Highly Quantized)

Approximate Memory per Parameter: ~0.5 byte

AI Model Size vs GPU Memory Requirements

7B / 8B

FP16 / BF16 Weights: ~14 GB - 16 GB
INT8 Weights: ~7 GB - 8 GB
4-bit Weights: ~3.5 GB - 4 GB

13B / 14B

FP16 / BF16 Weights: ~26 GB - 28 GB
INT8 Weights: ~13 GB - 14 GB
4-bit Weights: ~6.5 GB - 7 GB

32B / 34B

FP16 / BF16 Weights: ~64 GB - 68 GB
INT8 Weights: ~32 GB - 34 GB
4-bit Weights: ~16 GB - 17 GB

70B / 72B

FP16 / BF16 Weights: ~140 GB - 144 GB
INT8 Weights: ~70 GB - 72 GB
4-bit Weights: ~35 GB - 36 GB

Important Sizing Disclaimer: These numbers represent the approximate memory required for the model weights alone. You must add 20% to over 100% additional memory overhead to support the context window, continuous batching, and the KV cache.>

When Do You Need Multiple GPUs?

The Model Does Not Fit: A 70B parameter model running in FP16 requires roughly 140GB just to load the model weights. An 80GB GPU simply cannot hold this model alone.
Higher Throughput and Concurrency: If your API scales from 10 concurrent users to 1,000, you need multiple GPUs for data parallelism.
Large-Scale Distributed Models: Massive foundation models approaching 400B parameters require clusters of multi-GPU servers.

How Much System RAM Does an AI Inference Server Need?

System RAM and GPU VRAM are not interchangeable. Adding 512GB of standard DDR5 system memory will not help you load a massive 70B parameter model if your GPU only has 24GB of VRAM. The model weights required for hardware acceleration must reside in the GPU’s VRAM.

However, system RAM actively supports the surrounding inference architecture. Practical deployment ranges include:

Entry-Level (64GB – 128GB): Sufficient for serving single, smaller LLMs (7B-8B class).
Production (128GB – 256GB): The standard starting point for enterprise deployments (continuous batching, application containers).
Large Multi-GPU (256GB – 1TB+): Required for multi-node inference, massive models, or heavy vector database queries.

Network and Storage Infrastructure

Network capacity is dictated by concurrent users and request payloads.

Network Consideration: 1Gbps may be sufficient
Network Consideration: 10Gbps dedicated servers are a strong standard
Network Consideration: 10Gbps to 25Gbps+ (essential for heavy RAG inputs)
Network Consideration: 100Gbps to 200Gbps+ (RDMA/RoCE/InfiniBand)

Traditional hard drives or SATA SSDs will severely cripple an inference server during model loading. High-speed NVMe storage ensures that local disk reads never become a bottleneck. Furthermore, modern CPUs must offer enough direct PCIe Gen4/Gen5 lanes to support multiple GPUs, high-speed NICs, and NVMe storage without lane sharing or cross-socket latency.

Common AI Inference Server Bottlenecks

GPU VRAM

Symptoms: Out-of-memory (OOM) errors upon load
Possible Solution: Add more VRAM, or apply 4-bit/8-bit quantization

KV Cache

Symptoms: Memory exhaustion during long conversations
Possible Solution: Optimize context limits, deploy Paged Attention (vLLM)

CPU

Symptoms: GPU utilization is consistently low (waiting)
Possible Solution: Upgrade to stronger enterprise CPUs

System RAM

Symptoms: System swapping to disk, slow RAG queries
Possible Solution: Increase system memory

PCIe

Symptoms: Slow data-transfer rates
Possible Solution: Ensure Gen4/Gen5 topology without lane sharing

Network

Symptoms: High API latency despite fast token generation
Possible Solution: Upgrade to 10Gbps or 25Gbps dedicated networking

Dedicated AI Inference Server vs Cloud GPU

When deploying AI into production, you must choose between renting cloud GPU instances or deploying a dedicated server.

Hardware Control

Dedicated Server (Bare Metal): High (Full root access, custom topology)
Cloud GPU Instance: Provider-dependent

Resource Predictability

Dedicated Server (Bare Metal): High (No noisy neighbors)
Cloud GPU Instance: Service-dependent

Long-Running Workloads

Dedicated Server (Bare Metal): Often highly cost-effective 24/7
Cloud GPU Instance: Can become incredibly expensive

Scaling

Dedicated Server (Bare Metal): Hardware-based (Requires provisioning)
Cloud GPU Instance: Rapid and elastic

Cloud GPUs are excellent for prototyping, training, or highly variable workloads that spike randomly. However, for predictable, continuous 24/7 API serving, dedicated bare-metal infrastructure drastically reduces long-term operational costs while providing total control over your system topology.

Why Dedicated Infrastructure Makes Sense for Production AI Inference

Deploying an AI inference server on bare metal guarantees that 100% of the CPU cores, system RAM, NVMe storage, and PCIe lanes are dedicated entirely to your workload. Whether you need a high-performance NVIDIA H100 dedicated server for a massive LLM, or a balanced multi-GPU setup with 100Gbps unmetered networking for agentic AI workflows.

How to Detect a DDoS Attack on a Linux Server via CLI

Ethan Vance — Fri, 21 Aug 2026 05:00:35 +0000

When your Linux server load suddenly spikes, guessing the cause is not an option. In these critical moments, you need to know immediately whether you are dealing with a legitimate traffic surge, a misbehaving application, or a DDoS attack.

Here is a hands-on guide to diagnosing malicious traffic directly from your terminal using standard Linux command-line utilities.

1. Monitor Network Interfaces (Volumetric Attacks)

The most common DDoS attack is a volumetric flood. Before digging into logs, check the raw traffic hitting your network interfaces.

Real-Time Bandwidth with iftop:

sudo iftop -n

Note: The -n flag prevents DNS resolution, which is crucial during an attack because DNS lookups will slow down the tool.

Check Packets Per Second (PPS) with sar:

Bash
sar -n DEV 1

If your inbound traffic (RX) or PPS is pinned to its absolute limit while CPU usage remains normal, it strongly indicates a Layer 3 network-level flood.

2. Analyze Active TCP States (Protocol Attacks)

If legitimate users are failing to connect, the attacker is likely targeting your server's connection-handling capacity (Layer 4).

Count Total Connections by IP Address:

Bash
ss -H -tn | awk '{print $5}' \vert{} sed 's/:[^:]*$//' | sort | uniq -c | sort -nr | head -10

Detect a SYN Flood Attack:

A SYN flood repeatedly sends initial connection requests (SYN) but never completes the handshake. To count connections stuck in the SYN_RECV state:

Bash
ss -H -n -t state syn-recv | wc -l

Pro-Tip: For a rapid summary of your current TCP states without locking up your terminal, simply type ss -s.

3. Inspect Web Server Logs (Layer 7 HTTP Floods)

If your network bandwidth is fine but your server's CPU or memory is maxed out, you might be facing an HTTP flood.

Identify Top Attacking IPs via Web Logs (Nginx):

Bash
tail -n 10000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

Find the Most Hammered URLs:

Bash
tail -n 10000 /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -10

4. Defense and Upstream Mitigation

Once the observed traffic patterns are consistent with a DDoS attack, you can begin mitigation.

Block specific IPs: sudo iptables -A INPUT -s ATTACKER_IP -j DROP
Enable TCP SYN Cookies: sudo sysctl -w net.ipv4.tcp_syncookies=1

Understand the Limits of Local Server Defense:

Local firewalls are useful for targeted attacks, but they cannot stop a massive volumetric flood. If an attacker sends 50Gbps of traffic to your 1Gbps interface, dropping packets locally at the OS level still means your pipe is clogged.

To survive large-scale volumetric or complex multi-vector DDoS attacks, malicious traffic must be filtered before it ever reaches your server through upstream traffic scrubbing or high-capacity network-level mitigation.

Why Upgrading Your Server's Hardware Won't Stop a DDoS Attack

Ethan Vance — Thu, 20 Aug 2026 04:44:50 +0000

When a web application starts slowing down or dropping connections under heavy load, the instinct for many developers and sysadmins is to scale up: add a faster CPU, double the RAM, or move to NVMe storage.

But what happens when the traffic isn't a viral product launch, but a coordinated Distributed Denial-of-Service (DDoS) attack?

The harsh reality is that a dedicated server can have top-tier hardware and a 20Gbps network port, yet still become entirely unreachable. Let’s dive into the technical mechanics of why this happens and where the actual bottlenecks occur.

The Traffic Path: Where Do Things Break?

When an attack is launched, the malicious data doesn't instantly appear on your server's processor or memory. The traffic follows a specific path:

Internet → Upstream Network → Mitigation Infrastructure → Server Network Interface → OS → Application

If incoming traffic exceeds the available capacity at any point in this path, the link becomes saturated. Legitimate traffic simply cannot reach your server.

How DDoS Attacks Choke Network Performance

Attacks generally fall into three categories (Volumetric, Protocol, and Application-layer), and they impact your network in the following ways:

1. Bandwidth Saturation (The Clogged Pipe)

Every network connection has a finite limit. In a volumetric attack (like a UDP flood), the attacker's goal is to consume all available bandwidth between the target and the wider internet. Even if you have a 10Gbps or 20Gbps port, a massive attack can fill that "pipe" completely. Your CPU might be sitting at 5% utilization, but your users still get a 502 Bad Gateway or connection timeout because their requests can't physically reach the server.

2. Connection State Exhaustion

Not all attacks rely on pure data volume. Protocol attacks (like SYN floods) target the connection-handling capacity of network infrastructure like firewalls or load balancers. By initiating massive numbers of incomplete TCP connection attempts, attackers can consume all available connection state tables. Once exhausted, the network device simply drops any new connections from legitimate users.

3. Packet Loss and Latency

Excessive traffic creates network congestion. When routers and switches are overwhelmed, packets spend more time waiting in queues (increasing latency). When the buffers are full, packets are dropped entirely (packet loss). This forces retransmissions, creating even more traffic and making interactive applications (like WebSockets or game servers) unusable.

The Host Firewall Misconception

A common misconception among developers is that iptables or a standard host-based firewall is enough to stop a network flood.

While a firewall is essential for controlling access (e.g., blocking unused ports), it only filters traffic after it has reached your server's network interface. If a volumetric attack has already saturated your upstream bandwidth, your local firewall dropping the packets won't magically free up the network pipe.

The Solution: Upstream Mitigation

To truly protect a server, you can't rely on the server to defend itself. The most effective protection happens upstream.

Upstream DDoS mitigation analyzes incoming data in real-time, passing it through traffic scrubbing centers. The core concept is straightforward:

Incoming traffic → Scrubbing Center → Malicious traffic dropped → Clean traffic forwarded

This preserves your network capacity and ensures your server's hardware is only processing legitimate application logic, not fighting off junk packets.

Configuring NVIDIA MIG: Technical Walkthrough for Bare-Metal GPU Partitioning published: true

Ethan Vance — Tue, 11 Aug 2026 11:36:06 +0000

High-capacity GPUs (like the A100, H100, or Blackwell) often sit underutilized in bare-metal environments. Software-based time slicing lacks hardware-level resource guarantees.

NVIDIA Multi-Instance GPU (MIG) solves this by partitioning a single physical card into independent instances at the hardware level, giving each partition dedicated compute, memory, and bandwidth.

Here is a quick step-by-step CLI walkthrough on Ubuntu 24.04:

1. Install Drivers and Verify Support

sudo ubuntu-drivers install
nvidia-smi -q

Look for the MIG Mode section in the output to confirm hardware support. (Install nvidia-container-toolkit if you plan to run Docker).

2. Enable MIG Mode

Ensure no active workloads are attached, then enable MIG mode on GPU 0:

sudo nvidia-smi -i 0 -mig 1

3. List Supported Profiles

Query the supported resource layouts for your GPU model:

nvidia-smi mig -lgip

4. Partition the GPU

Create GPU Instances (GI) and Compute Instances (CI) using your chosen profile string:

sudo nvidia-smi mig -cgi 2g.48gb,2g.48gb -C

Verify active partitions using nvidia-smi mig -lgi.

5. Target MIG Instances in Docker

List generated UUIDs:

nvidia-smi -L

Pass the specific MIG UUID directly to your Docker container:

docker run --rm -it --gpus '"device=MIG-af414487-fcaa-5f42-b210-6f614c9cf780"' nvcr.io/nvidia/pytorch: nvidia-smi

6. Reset Partitions

To revert to single-instance operation:

sudo nvidia-smi mig -dci
sudo nvidia-smi mig -dgi
sudo nvidia-smi -i 0 -mig 0

Managing bare-metal lifecycle, drivers, and hardware provisioning at scale takes operational bandwidth. At MIG servers, we provide pre-configured, dedicated GPU hardware optimized for partitioned AI workloads out of the box.

Deep Dive into NVIDIA Blackwell Architecture: Redefining GenAI Infrastructure

Ethan Vance — Thu, 06 Aug 2026 07:30:18 +0000

When we talk about the evolution of modern GPU servers, the NVIDIA Blackwell architecture represents a monumental leap forward. Purpose-built to handle the most demanding AI and cloud computing workloads, Blackwell is strictly an enterprise-grade system.

Unlike consumer gaming GPUs, Blackwell is completely optimized for processing massive datasets, complex neural networks, and generative AI systems. It directly succeeds the highly successful NVIDIA Hopper architecture, bringing a massive leap in compute performance, memory bandwidth, and multi-node scalability to the data center.

Let's dive into the hardware and see what makes this silicon so groundbreaking. 👇

The Hardware: An Entirely New Class of AI Superchip 🧠

To achieve unprecedented computing density, NVIDIA engineering broke through traditional manufacturing limits:

Unmatched Scale: These GPUs pack an astounding 208 billion transistors, providing the raw compute density needed for trillion-parameter models.
Custom Fabrication: The architecture is manufactured utilizing a custom-built TSMC 4NP process, balancing extreme performance with energy efficiency.
Unified Architecture: To overcome physical die limits, all Blackwell products feature two reticle-limited dies. Instead of acting as separate processors, they are seamlessly linked by a 10 terabytes per second (TB/s) chip-to-chip interconnect. This allows the dual-die setup to function flawlessly as a single, unified GPU.

Inside the Technological Breakthroughs ⚡

Blackwell is not just a faster chip; it is a fundamental redesign of how computing resources interact.

1. Second-Generation Transformer Engine
Training and running Large Language Models (LLMs) requires staggering amounts of computational power. Blackwell introduces its second-generation Transformer Engine, which pairs custom Tensor Cores with software like NVIDIA TensorRT™-LLM. What truly sets it apart is micro-tensor scaling, enabling FP4 (4-bit floating point) AI precision. This effectively doubles the performance and memory capacity for next-generation models while maintaining high accuracy.

2. 5th-Generation NVLink & NVLink Switch
Even the fastest GPUs will bottleneck if the network connecting them is slow. The 5th-generation NVLink interconnect solves this by scaling up to 576 GPUs. Within a single 72-GPU NVLink domain (NVL72), the NVLink Switch Chip enables a massive 130TB/s of GPU bandwidth.

3. Secure AI with Confidential Computing
Security is paramount for enterprise data. Blackwell is the industry’s first TEE-I/O capable GPU. NVIDIA Confidential Computing protects sensitive data and models from unauthorized access without any performance degradation.

4. Decompression Engine & RAS
The architecture features a dedicated Decompression Engine that accelerates the full pipeline of database queries. Additionally, intelligent resiliency is handled via a dedicated RAS (Reliability, Availability, and Serviceability) Engine, which uses AI-powered predictive management to minimize downtime.

Blackwell vs. Hopper: What’s the Real Difference? 📊

The Hopper architecture (H100) is an incredibly powerful foundation for today's workloads. However, Blackwell (B200) is purpose-built for the massive scale of tomorrow.

Feature	NVIDIA Hopper (H100)	NVIDIA Blackwell (B200)
Primary Focus	Mixed AI & Traditional HPC	Massive LLMs & Generative AI
Transformer Precision	1st Gen Tensor Cores with FP8 precision	2nd Gen Tensor Cores with FP4 micro-tensor scaling
Interconnect Technology	4th Gen NVLink	5th Gen NVLink (Scales up to 576 GPUs)
Domain Bandwidth	Scalable for standard GPU clusters	Up to 130 TB/s within a 72-GPU NVL72 domain
Confidential Computing	Standard hardware security	First TEE-I/O capable GPU with no performance overhead

The Infrastructure Reality Check 🏗️

While the performance gains are undeniable, deploying Blackwell in-house introduces severe infrastructure challenges. These are not plug-and-play GPUs:

⚠️ Extreme Power Draw: A single Blackwell GPU can consume up to ~1,000 watts, straining standard data center electrical limits.

💧 Mandatory Liquid Cooling: Traditional air-cooling systems are incapable of dissipating the heat. Liquid cooling infrastructure is now a strict requirement.

🏢 Incompatible with Standard Racks: You cannot slot a Blackwell GPU into a legacy server chassis. These require purpose-built AI systems like NVIDIA HGX or DGX platforms.

Key Developer Use Cases 💻

By matching hardware innovations to modern software demands, Blackwell unlocks new capabilities:

LLM Training & Fine-Tuning: Accelerate time-to-market for proprietary models using FP4 precision.

Large-Scale Inference: Handle high token throughput efficiently for real-time AI chatbots, driving down the compute cost-per-token.

Big Data Analytics & HPC: Rapidly process massive datasets and blend computing simulations with machine learning seamlessly.

Physical AI & Robotics: Train complex vision models and run high-fidelity Digital Twin simulations for autonomous logic.

Conclusion 🏁

The NVIDIA Blackwell architecture has definitively set the new standard for accelerated computing. However, as developers and engineers, we must also prepare for the massive physical constraints—navigating 1000W power limits and liquid cooling will be just as crucial as writing the algorithms themselves.

The future of AI infrastructure is incredibly exciting, but it demands a complete rethink of data center physics.

How to Build a High-Availability (HA) Cluster on Bare Metal

Ethan Vance — Thu, 23 Jul 2026 08:49:54 +0000

When deploying mission-critical applications, a Single Point of Failure (SPOF) is a disaster waiting to happen. High Availability (HA) is essential for achieving uptime targets like 99.99%.

In this tutorial, we outline a production-grade, 7-node High-Availability cluster built on bare-metal servers using a Private VLAN for maximum performance, security, and hardware control.

🏗️ 3-Tier Architecture Overview

Traffic flows through three distinct, isolated tiers:

Floating VIP (203.0.113.100): Single public IP entry point.
Tier 1 (Load Balancers - LB-01 & LB-02): Active/Passive HAProxy + Keepalived setup for automated 1–3 second VIP failover.
Tier 2 (Web Tier - WEB-01 & WEB-02): Nginx nodes isolated from public access.
Tier 3 (Database Tier - DB-01, DB-02, DB-03): MariaDB Galera Cluster with synchronous, certification-based replication.

📋 Server Topology & IP Scheme

+----------+-----------------+---------------+--------------------+
| Hostname | Role            | Public IP     | Private IP (VLAN)  |
+----------+-----------------+---------------+--------------------+
| VIP      | Floating IP     | 203.0.113.100 | -                  |
| LB-01    | Load Balancer 1 | 203.0.113.101 | 10.0.0.10          |
| LB-02    | Load Balancer 2 | 203.0.113.102 | 10.0.0.11          |
| WEB-01   | Web Node 1      | -             | 10.0.0.20          |
| WEB-02   | Web Node 2      | -             | 10.0.0.21          |
| DB-01    | DB Node 1       | -             | 10.0.0.30          |
| DB-02    | DB Node 2       | -             | 10.0.0.31          |
| DB-03    | DB Node 3       | -             | 10.0.0.32          |
+----------+-----------------+---------------+--------------------+

🔒 Security & Sysctl Tuning

Allow incoming traffic strictly from your Private VLAN (10.0.0.0/24) using ufw:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh

# Allow internal Web Traffic
sudo ufw allow from 10.0.0.0/24 to any port 80
sudo ufw allow from 10.0.0.0/24 to any port 443

# Allow internal Galera & MySQL Traffic
sudo ufw allow from 10.0.0.0/24 to any port 3306  # MySQL
sudo ufw allow from 10.0.0.0/24 to any port 4444  # Galera SST
sudo ufw allow from 10.0.0.0/24 to any port 4567  # Galera Cluster
sudo ufw allow from 10.0.0.0/24 to any port 4568  # Galera IST

sudo ufw enable

Apply kernel tweaks on all nodes to enable non-local binding and scale network queues:

1. MariaDB Galera Configuration

Edit the Galera configuration file:

/etc/mysql/mariadb.conf.d/60-galera.cnf

[galera]
bind-address = 0.0.0.0

binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2

wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name = ha_production_cluster
wsrep_cluster_address = "gcomm://10.0.0.30,10.0.0.31,10.0.0.32"

wsrep_sst_method = mariabackup

Kernel Networking (Sysctl)

Create the sysctl configuration file:

/etc/sysctl.d/99-ha-cluster.conf

net.core.somaxconn = 65535
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_max_syn_backlog = 65535

Apply the changes:

sudo sysctl --system

💡 Note

Bootstrap the Galera cluster only on DB-01:
sudo galera_new_cluster
After the cluster is initialized, start MariaDB normally on DB-02 and DB-03:
sudo systemctl start mariadb

2. Keepalived Unicast VRRP

Modern bare-metal servers and many cloud providers block multicast traffic. To ensure reliable VIP failover, configure Unicast VRRP.

Edit the Keepalived configuration:

/etc/keepalived/keepalived.conf

LB-01 (MASTER)

vrrp_instance VI_1 {
    state MASTER
    interface ens18
    virtual_router_id 51
    priority 100

    unicast_src_ip 10.0.0.10

    unicast_peer {
        10.0.0.11
    }

    virtual_ipaddress {
        203.0.113.100/32
    }
}

LB-02 (BACKUP)

vrrp_instance VI_1 {
    state BACKUP
    interface ens18
    virtual_router_id 51
    priority 90

    unicast_src_ip 10.0.0.11

    unicast_peer {
        10.0.0.10
    }

    virtual_ipaddress {
        203.0.113.100/32
    }
}

🔗 Read the full step-by-step implementation guide with all HAProxy config files and failover testing steps:

Read Full Step-by-Step Tutorial Here