DEV Community: augustine Egbuna

Adding Private NuGet Feeds in Multi-Stage Dockerfiles Without Breaking Your Build

augustine Egbuna — Fri, 17 Apr 2026 07:01:43 +0000

You're building a .NET service in TeamCity. Your build depends on internal NuGet packages from other projects. The TeamCity NuGet feed is configured, but your Docker build agent can't see it. You add dotnet nuget add source to your Dockerfile, the build completes, but when you actually try to restore packages, the source isn't there.

I hit this exact problem setting up ML model serving APIs that depended on shared utility packages. The issue isn't obvious: NuGet configuration added during the Docker build doesn't persist to the restore step because of how multi-stage builds and layer caching work.

Why Your NuGet Source Disappears

When you run dotnet nuget add source in a Dockerfile, it writes to ~/.nuget/NuGet/NuGet.Config. But in multi-stage builds, that config lives in an intermediate layer that may not be present when you actually run dotnet restore. Even in single-stage builds, if you're running the container with volume mounts or different working directories, the config path changes.

Here's what typically fails:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

# This completes without error
RUN dotnet nuget add source https://teamcity.example.com/httpAuth/app/nuget/feed \
    --name TeamCityFeed \
    --username build-agent \
    --password $NUGET_PASSWORD \
    --store-password-in-clear-text

WORKDIR /src
COPY *.csproj .
# This step fails: source not found
RUN dotnet restore

The add source command succeeds, but the config is written to /root/.nuget/NuGet/NuGet.Config in that layer. When dotnet restore runs, it may be looking in a different user context or the layer cache invalidates.

Solution 1: NuGet.Config in Your Source Tree

The most reliable approach is to commit a NuGet.Config file to your repository and copy it into the Docker build context. This works because the config travels with your code.

Create NuGet.Config at your solution root:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="TeamCityFeed" value="https://teamcity.example.com/httpAuth/app/nuget/feed" />
  </packageSources>
  <packageSourceCredentials>
    <TeamCityFeed>
      <add key="Username" value="%NUGET_USERNAME%" />
      <add key="ClearTextPassword" value="%NUGET_PASSWORD%" />
    </TeamCityFeed>
  </packageSourceCredentials>
</configuration>

Notice the %NUGET_USERNAME% and %NUGET_PASSWORD% placeholders. NuGet will substitute these from environment variables at runtime. Your Dockerfile becomes:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /src
COPY NuGet.Config .
COPY *.csproj .

# Pass credentials as build args
ARG NUGET_USERNAME
ARG NUGET_PASSWORD
ENV NUGET_USERNAME=${NUGET_USERNAME}
ENV NUGET_PASSWORD=${NUGET_PASSWORD}

RUN dotnet restore

COPY . .
RUN dotnet publish -c Release -o /app/publish

Build with:

docker build \
  --build-arg NUGET_USERNAME=build-agent \
  --build-arg NUGET_PASSWORD="${TEAMCITY_NUGET_TOKEN}" \
  -t your-service:latest .

This approach works because the config file is present in the working directory when dotnet restore runs. NuGet searches the current directory before looking at the user profile.

Solution 2: Mount Config at Build Time

If you can't commit credentials (even as placeholders) to source control, mount the config as a build secret. Docker BuildKit supports this with --secret and RUN --mount=type=secret.

Create nuget-config.xml on your build agent:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="TeamCityFeed" value="https://teamcity.example.com/httpAuth/app/nuget/feed" />
  </packageSources>
  <packageSourceCredentials>
    <TeamCityFeed>
      <add key="Username" value="build-agent" />
      <add key="ClearTextPassword" value="actual-token-here" />
    </TeamCityFeed>
  </packageSourceCredentials>
</configuration>

Dockerfile with mounted secret:

# syntax=docker/dockerfile:1.4
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /src
COPY *.csproj .

RUN --mount=type=secret,id=nuget_config,target=/root/.nuget/NuGet/NuGet.Config \
    dotnet restore

COPY . .
RUN dotnet publish -c Release -o /app/publish

Build command:

DOCKER_BUILDKIT=1 docker build \
  --secret id=nuget_config,src=./nuget-config.xml \
  -t your-service:latest .

The --mount=type=secret makes the config available only during that RUN step. It's never written to a layer, so it won't leak into the final image.

Handling Multi-SDK Scenarios

If you're using multiple .NET SDK versions (like the original question implied), you need the config accessible to each SDK. The mounted secret approach works best here because it targets the standard config location.

For a multi-SDK Dockerfile:

# syntax=docker/dockerfile:1.4
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-legacy

WORKDIR /src/legacy
COPY legacy/*.csproj ./
RUN --mount=type=secret,id=nuget_config,target=/root/.nuget/NuGet/NuGet.Config \
    dotnet restore

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-modern

WORKDIR /src/modern
COPY modern/*.csproj ./
RUN --mount=type=secret,id=nuget_config,target=/root/.nuget/NuGet/NuGet.Config \
    dotnet restore

Each stage gets the config mounted at the same path, so both SDK versions can read it during their respective restore steps.

TeamCity-Specific Integration

TeamCity has first-class support for NuGet feeds. Instead of managing credentials in Dockerfiles, configure them as build parameters:

In TeamCity: Project Settings → Parameters
Add env.NUGET_USERNAME and env.NUGET_PASSWORD (mark as password)
Use the NuGet.Config in source tree approach from Solution 1

TeamCity automatically exposes these as environment variables to your build steps. Your Dockerfile reads them via the ENV directive, and NuGet substitutes them at restore time.

For builds that run outside TeamCity (local development), developers can set these environment variables manually or use a local NuGet.Config that points to their own credentials.

Common Gotcha: Layer Caching

If you modify your NuGet source configuration, Docker's layer cache may serve a stale restore. After changing feed URLs or credentials, force a rebuild:

docker build --no-cache -t your-service:latest .

Or, more surgically, change the COPY command that includes your NuGet.Config by adding a dummy ARG with a timestamp:

ARG CACHE_BUST=1
COPY NuGet.Config .

Build with --build-arg CACHE_BUST=$(date +%s) to invalidate the cache from that point forward.

What Didn't Work

I initially tried dotnet nuget add source with --configfile to specify an explicit path. This worked in the RUN step where I added it, but didn't persist because each RUN creates a new layer. Subsequent RUN dotnet restore commands couldn't see the source.

I also tried using docker-compose volumes to mount the host's ~/.nuget directory into the container. This worked locally but broke CI because the build agent's filesystem layout differed from the container's expectations.

The solutions above are what actually worked in production: NuGet.Config in the source tree for simplicity, mounted secrets for security-sensitive environments.

This post is an excerpt from Practical AI Infrastructure Engineering — a production handbook covering Docker, GPU infrastructure, vector databases, and LLM APIs. Full book with 4 hands-on capstone projects available at https://activ8ted.gumroad.com/l/ssmfkx

Originally published at fivenineslab.com

Why can two Docker containers ping each other by name but one cannot make HTTP requests to the other?

augustine Egbuna — Sun, 12 Apr 2026 17:46:11 +0000

You've spun up two containers on a custom bridge network. DNS works. Ping works. But curl to your application returns "Connection refused" or just hangs. I've debugged this exact scenario a dozen times across ML inference APIs talking to Redis, FastAPI services querying vector databases, and monitoring sidecars trying to scrape metrics.

The problem isn't networking — it's that your application isn't actually listening where you think it is.

Why ping works but HTTP doesn't

When you ping redis from the app container, Docker's embedded DNS resolver translates that name to the container's IP on the bridge network. ICMP packets flow through without issue because ping operates at the network layer. No ports, no listeners, just "is this IP reachable?"

HTTP requires a process actively listening on a specific port. If your application binds to 127.0.0.1:8000 instead of 0.0.0.0:8000, it only accepts connections from localhost inside that container. Traffic from another container hits the network interface, finds nothing listening, and the kernel sends back a TCP RST.

Here's what actually happens when you run curl http://app:8000 from the redis container:

DNS resolves app to something like 172.18.0.2
TCP SYN packet travels to that IP on port 8000
If the app is bound to 127.0.0.1:8000, the kernel checks: "Is there a socket listening on 172.18.0.2:8000?" Answer: no.
Kernel replies with RST (connection refused) or drops the packet (timeout)

Verify what your application is actually bound to

SSH into your app container and check what's listening:

docker exec -it app netstat -tlnp

You'll see output like:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      1/python

That 127.0.0.1:8000 is your problem. The application is only reachable from inside its own container. You need 0.0.0.0:8000:

tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1/python

If you're running a FastAPI app with Uvicorn, the default host is 127.0.0.1. You must explicitly set it:

# main.py
import uvicorn
from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
def health():
    return {"status": "healthy"}

if __name__ == "__main__":
    # This will NOT work for inter-container communication
    # uvicorn.run(app, host="127.0.0.1", port=8000)

    # This binds to all interfaces
    uvicorn.run(app, host="0.0.0.0", port=8000)

Flask, Django's runserver, and most development servers have the same issue. Flask's app.run() defaults to localhost. Django requires python manage.py runserver 0.0.0.0:8000.

The ports mapping red herring

The ports: - "8000:8000" line in your compose file publishes the container's port 8000 to the host's port 8000. This is for external access — like hitting http://localhost:8000 from your laptop.

Inter-container communication on the same network bypasses port publishing entirely. Containers talk directly via the bridge network's private IP space. If you removed ports: - "8000:8000", containers could still reach each other (assuming the app binds to 0.0.0.0).

I've seen engineers spend hours tweaking port mappings when the issue is purely the bind address.

Real debugging session

You're inside the redis container trying to reach the app:

# This works (DNS resolution)
nslookup app

# This works (network layer)
ping app

# This fails (application layer)
curl http://app:8000
# curl: (7) Failed to connect to app port 8000: Connection refused

Now exec into the app container and check listeners:

docker exec -it app sh
netstat -tlnp | grep 8000
# tcp  0  0  127.0.0.1:8000  0.0.0.0:*  LISTEN  1/python

There it is. Fix the bind address in your application code, rebuild the image, restart the container. Run netstat again:

netstat -tlnp | grep 8000
# tcp  0  0  0.0.0.0:8000  0.0.0.0:*  LISTEN  1/python

Now curl from redis works.

Other causes (less common but real)

Firewall rules inside the container. If you're running iptables or ufw inside a container (don't), they can block incoming traffic even when the app binds correctly. I've seen this in custom ML inference images where someone copied firewall configs from a VM setup.

Application-level issues. Your app might be crashing on startup, listening briefly, then dying. Check logs: docker logs app. If you see the server start message followed by a Python traceback, that's your issue — not networking.

Wrong protocol. This sounds dumb but I've debugged it twice: your app listens on HTTPS (TLS required), you're curling plain HTTP. Or the app expects HTTP/2 and your client sends HTTP/1.1. Both scenarios time out or fail in confusing ways.

SELinux or AppArmor. On some Linux distributions, mandatory access controls can block container-to-container traffic even on the same network. Check dmesg | grep -i denied after a failed connection attempt.

The correct compose file

Here's what your setup should look like for a typical FastAPI + Redis stack:

services:
  app:
    build: .
    container_name: app
    networks:
      - mynetwork
    ports:
      - "8000:8000"  # Host access only
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    depends_on:
      - redis

  redis:
    image: redis:alpine
    container_name: redis
    networks:
      - mynetwork
    # No ports needed unless you want host access to Redis

networks:
  mynetwork:
    driver: bridge

And your application must bind to 0.0.0.0. For Uvicorn in production, I run it via command override:

  app:
    build: .
    command: uvicorn main:app --host 0.0.0.0 --port 8000

This makes the bind address explicit in the deployment config, not buried in application code where the next developer might miss it.

Why this matters for AI infrastructure

Every LLM inference API I've deployed follows this pattern: FastAPI frontend talking to a vector database (Qdrant, Milvus), a Redis cache, and sometimes multiple model containers. When one component can't reach another, the entire request pipeline fails.

The symptom — "Connection refused" — looks like a networking problem. The fix is almost always a bind address configuration in your Python code. I've watched engineers add custom network configs, adjust MTU settings, and rebuild Docker networks when they needed to change one line in uvicorn.run().

Test inter-container communication immediately after writing your compose file. Don't wait until you're debugging a failed inference request in production.

Originally published at fivenineslab.com

Training Small LLMs to Edit Code Instead of Generating It

augustine Egbuna — Fri, 10 Apr 2026 03:24:53 +0000

You've hit the wall with 2B parameter models trying to write functions from scratch. The output is syntactically broken, logically confused, or just hallucinates APIs that don't exist. But what if you stopped asking these models to be creative and instead treated them as intelligent diff generators?

I've run this exact experiment with Qwen2.5-Coder-1.5B and Phi-3-mini on an RTX 3060. The insight is simple: small models fail at generation but succeed at transformation. Give them a working reference implementation from GitHub and ask them to modify it for your specific use case. The model operates in the space of edits, not invention.

Why Small Models Fail at Code Generation

A 2B model has seen enormous amounts of code during pretraining, but it lacks the parameter capacity to reliably reproduce complex patterns. When you prompt "write a Redis connection pool in Python with retry logic", the model must:

Recall the Redis client API surface
Remember exception hierarchies
Generate retry backoff logic
Handle connection lifecycle edge cases
Produce syntactically valid, idiomatic Python

That's too many constraints for 2 billion parameters to satisfy simultaneously. You get code that looks plausible but fails on import redis or forgets to close connections.

But transformation is different. If you retrieve an existing Redis pool implementation and ask the model to "add exponential backoff to the retry logic", you've anchored it. The API calls are already there. The structure exists. The model only needs to insert a specific pattern it's seen hundreds of times.

The Retrieval + Edit Architecture

Here's the pipeline I tested with Phi-3-mini (3.8B) on an RTX 3060 Ti:

from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Index GitHub code (one-time setup)
embedder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
qdrant = QdrantClient(path="./code_index")

def index_github_snippets(repo_files):
    """Embed and store code snippets with metadata"""
    for file_path, content in repo_files:
        chunks = split_into_functions(content)  # Parse AST, extract functions
        embeddings = embedder.encode([c['code'] for c in chunks])
        qdrant.upsert(
            collection_name="code_snippets",
            points=[{
                "id": idx,
                "vector": emb.tolist(),
                "payload": {"code": chunk['code'], "file": file_path}
            } for idx, (emb, chunk) in enumerate(zip(embeddings, chunks))]
        )

# Retrieval + edit at inference
def edit_code_for_task(user_query):
    query_emb = embedder.encode(user_query)
    results = qdrant.search(
        collection_name="code_snippets",
        query_vector=query_emb,
        limit=3
    )

    reference_code = results[0].payload['code']

    model = AutoModelForCausalLM.from_pretrained(
        "microsoft/Phi-3-mini-4k-instruct",
        torch_dtype=torch.float16,
        device_map="cuda"
    )
    tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

    prompt = f"""Edit this code to: {user_query}

Reference implementation:

python
{reference_code}


Modified version:"""

    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

shell

The key is the prompt structure. You're not asking "write code to X". You're asking "here's code that does Y, modify it to do X". This constrains the solution space dramatically.

Performance Numbers on Low-End Hardware

On an RTX 3060 Ti (8GB VRAM), here's what I measured:

Phi-3-mini-4k-instruct (3.8B params, FP16)

Inference time: 2.1s for 256 tokens
VRAM usage: 7.2GB with batch size 1
Success rate (code runs without errors): 73% on my test set of 50 tasks

Qwen2.5-Coder-1.5B-Instruct (FP16)

Inference time: 1.3s for 256 tokens
VRAM usage: 3.1GB
Success rate: 61%

Compare this to the same models generating from scratch (no reference code):

Phi-3: 41% success rate
Qwen2.5-Coder: 29% success rate

The gap is huge. Editing existing code nearly doubles the reliability.

What This Actually Looks Like in Production

I deployed this as a VSCode extension prototype. The workflow:

User highlights code and types a natural language edit request
Extension embeds the request + existing code context
Searches local Qdrant index (seeded with 50k Python functions from popular repos)
Retrieves top-3 similar implementations
Sends reference code + edit instruction to local Phi-3 instance via llama.cpp server
Returns diff overlay in editor

The llama.cpp server runs with:

./server \
  -m phi-3-mini-4k-instruct.Q4_K_M.gguf \
  -c 4096 \
  -ngl 35 \
  --host 0.0.0.0 \
  --port 8080

Quantization (Q4_K_M) drops VRAM to 2.4GB. Inference is 3.2s on an RTX 2060. That's fast enough for an interactive editing assistant.

The Limitations You'll Hit

This isn't a magic solution. The model still hallucinates when:

Retrieved code is too different from the target task (embedder failure)
Edit requires understanding distant context (small context windows)
Task involves proprietary APIs not in the training data

I found the sweet spot is refactoring, adding error handling, changing API versions, and adapting patterns. The model is bad at architectural decisions or designing new abstractions.

Also, code retrieval quality matters more than model size. A better embedding model (say, Salesforce/SFR-Embedding-Mistral) improves success rate by 8-12 percentage points. The model can only edit what you feed it.

Should You Build This?

If you're running on constrained hardware and need a local coding assistant, yes. The retrieval + edit pattern is the only way I've found to get reliable output from sub-4B models.

But if you have access to larger models (CodeLlama 13B, DeepSeek-Coder 6.7B), stick with those. They can generate reasonably well from scratch, and the added complexity of maintaining a code index isn't worth it.

The real use case is edge deployment: offline environments, privacy-sensitive codebases, or devices where you can't run 13B+ models. A 2B editor beats no assistant at all.

For infrastructure teams, this matters if you're building internal developer tools. You can ship a locally-running code assistant that doesn't leak proprietary code to external APIs. The cost is maintaining the GitHub index and embedding pipeline, which is straightforward with Qdrant + a scheduled indexing job.

I'm running this setup in production for an internal CLI tool generator. Developers describe what they want, the system retrieves similar CLI implementations from our repos, and Phi-3 generates the modified version. It's not AGI, but it's useful.

Originally published at fivenineslab.com

Why More Data Center Teams Are Choosing NX-OS VXLAN EVPN Over Cisco ACI in 2026

augustine Egbuna — Thu, 09 Apr 2026 23:23:53 +0000

I spent four hours last Tuesday troubleshooting why a new GPU node couldn't reach the MLflow registry during a training run. The ACI fabric was reporting the endpoint learned. The policy contract showed permit. But packets died silently somewhere between leaf switches. The root cause? A stale endpoint entry in the COOP database that the APIC controller hadn't reconciled. I fixed it by clearing the endpoint from the CLI, bypassing the abstraction layer entirely.

That incident crystalized something I'd been seeing across three data center builds: when the controller's model of the network diverges from the actual forwarding state, you end up working around the abstraction, not through it. You SSH to the leaf switch and run show commands that reveal what's really happening in hardware. At that point, the controller is adding latency, not value.

The Real Tradeoff Nobody Talks About

ACI's pitch is clean: declare your intent through a GUI or API, and the fabric converges to that state. The APIC controller translates your application profiles, bridge domains, and contracts into the necessary VXLAN, EVPN, and policy constructs. You shouldn't need to understand MP-BGP route types or VNI allocation.

But here's what actually happens in production: you still need to understand those primitives when something breaks. The abstraction doesn't eliminate complexity; it relocates it behind an API that makes certain operations harder. Want to trace a specific MAC/IP binding through the fabric? You're running moquery against the APIC's object store and correlating it with CLI output from the leaf. Want to integrate with an existing BGP-based underlay? You're fighting the APIC's assumptions about how fabric routing should work.

NX-OS VXLAN EVPN, in contrast, gives you direct access to the forwarding primitives. You configure BGP EVPN address families, define VNI-to-VLAN mappings, and control route advertisement explicitly. There's no translation layer. What you configure is what runs in hardware.

Where This Shows Up in AI Infrastructure

GPU clusters amplify every network design decision. When you're running distributed training across 64 A100 nodes, a single packet drop during an NCCL all-reduce can stall the entire job. You need:

Deterministic forwarding paths with consistent latency
Lossless Ethernet with PFC properly scoped to GPU traffic
Fast convergence when a leaf switch or link fails
Visibility into actual queue depths and buffer utilization

ACI can deliver all of this, but the configuration path is indirect. You define QoS classes in the APIC, which generates MQC policies on each leaf. You enable PFC through a fabric access policy, which the APIC pushes as platform-specific DCBX settings. When you need to verify that PFC PAUSE frames are actually being sent for CoS 3 traffic on a specific port, you're back on the switch CLI, running:

switch# show interface ethernet 1/49 priority-flow-control

And if the output doesn't match what the APIC says is configured, you're troubleshooting two systems instead of one.

With NX-OS VXLAN EVPN, the QoS configuration is direct:

class-map type qos match-all gpu-rdma
  match cos 3
policy-map type qos gpu-qos
  class gpu-rdma
    set qos-group 3
    priority level 1

interface Ethernet1/49
  service-policy type qos input gpu-qos
  priority-flow-control mode on
  priority-flow-control watch-dog-interval 200

You write the exact policy you need. You see it in show run. You verify it in show policy-map interface. There's no model translation to debug.

The Kubernetes Integration Gap

Most AI infrastructure runs on Kubernetes now, and Kubernetes networking has strong opinions. CNI plugins like Calico, Cilium, and Antrea expect to control pod networking — IP allocation, routing, and increasingly, network policy. They assume the physical network provides L3 reachability, typically via BGP.

ACI's CNI plugin tries to bridge these worlds by mapping Kubernetes constructs to ACI objects. A namespace becomes an EPG. A network policy becomes a contract. But this creates tight coupling: your cluster lifecycle is now tied to the APIC's API and its upgrade schedule. I've seen teams delay Kubernetes upgrades by six months waiting for a compatible ACI CNI version.

The alternative pattern I'm seeing: run NX-OS VXLAN EVPN in the fabric, peer each leaf switch with the Kubernetes nodes using eBGP, and let the CNI plugin handle pod networking. Calico's route reflector mode works perfectly here:

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 65001
  serviceClusterIPs:
    - cidr: 10.96.0.0/12
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: leaf-1
spec:
  peerIP: 10.0.0.1
  asNumber: 65000

Each Kubernetes node peers with its leaf switches. Pod routes are advertised via BGP. The fabric treats them like any other /32. When a pod moves, Calico withdraws the old route and advertises the new one. Convergence is sub-second. No controller in the middle.

The TCO Equation Has Changed

ACI's total cost of ownership used to be defensible because the APIC automation saved operational effort. But in 2026, the baseline assumption is Infrastructure as Code. You're managing everything through Terraform or Ansible anyway. The question isn't whether you have automation; it's which primitives your automation targets.

Targeting ACI means Terraform providers that wrap the APIC API, which abstracts the actual network config. Your state files contain EPGs and contracts. Your pipeline has an APIC dependency — it has to be reachable, authenticated, and healthy for changes to apply.

Targeting NX-OS EVPN means Terraform providers that generate CLI commands or use NETCONF/gNMI. Your state files contain the actual config. Your pipeline pushes directly to devices. You can stage and test config in a text file before applying it. There's no controller to version-match or license separately.

License cost is the obvious part: ACI requires APIC controllers (virtual or physical) with their own licensing. NX-OS VXLAN EVPN runs on the same switches with base NX-OS licensing. But the less obvious cost is operational: every abstraction layer is another integration point to maintain, another API version to track, another component in your blast radius when you upgrade.

What This Means for Your Next Build

If you're designing a leaf-spine fabric in 2026, especially for GPU-dense AI infrastructure, start with these questions:

Do you need the APIC's policy model, or are you comfortable managing EVPN/VXLAN primitives directly through IaC?
How tightly coupled do you want your physical network to be with your container orchestration layer?
When troubleshooting, do you prefer working through a controller API or directly on device CLI?

For most teams I'm working with, the answers point toward NX-OS VXLAN EVPN. They're already managing network config as code. Their Kubernetes CNI handles pod networking. They want the shortest path from intent to forwarding plane, especially when debugging at 2 AM.

ACI isn't dead, but its value proposition has narrowed. It still makes sense if you have a large operational team that prefers GUI-driven workflows, or if you're deeply integrated with Cisco's broader intent-based networking stack. But for infrastructure engineers building modern GPU clusters and Kubernetes platforms, the simpler path is increasingly the better one.

The network is becoming infrastructure code. The abstraction layers that hide the primitives are becoming friction. And the teams who understand EVPN/VXLAN directly are shipping faster than the ones waiting for controller APIs to catch up.

Originally published at fivenineslab.com

Docker's nftables Mode Doesn't Respect Your Drop Rules — Here's the Fix

augustine Egbuna — Tue, 07 Apr 2026 18:12:11 +0000

You enable Docker's experimental nftables support, add a drop rule in /etc/nftables.conf, reload your firewall, and the container port stays wide open. The packet hits your drop rule, then Docker's accept rule fires anyway. This violates everything you thought you knew about packet filtering.

I hit this exact scenario running a multi-tenant LLM API platform where different teams deploy inference containers. One team accidentally exposed their Ollama admin interface on port 3000. Standard nftables drop rules in our firewall config did nothing — the port stayed accessible from the internet.

Why Docker's nftables Chains Bypass Your Rules

Docker 29+ creates its own nftables table (docker) with chains that hook into prerouting, forward, and postrouting. These chains have specific priority values that determine their execution order relative to your custom chains.

Here's the critical part: nftables evaluates chains based on priority within the same hook. A drop rule in your inet filter table with priority 0 doesn't automatically block packets that a docker table chain with priority -100 has already accepted.

Check what Docker actually created:

nft list ruleset | grep -A 20 "table inet docker"

You'll see output like:

table inet docker {
    chain forward {
        type filter hook forward priority -100; policy accept;
        ct state established,related accept
        iifname "docker0" accept
        oifname "docker0" accept
    }
}

That priority -100 means Docker's forward chain runs before your standard filter chain at priority 0. If Docker's chain accepts the packet, your drop rule never even sees it.

The Priority Math Docker Doesn't Tell You

Nftables priorities are integers. Lower (more negative) values run first. Standard filter tables use priority 0. Docker uses:

prerouting: priority -300 for DNAT rules
forward: priority -100 for container traffic acceptance
postrouting: priority 100 for masquerading

Your drop rule in a priority 0 chain fires after Docker has already said "yes, forward this packet to the container". The packet is gone.

Solution 1: Override Docker's Priority

Create a chain with a lower priority than Docker's -100 for the forward hook:

nft add table inet firewall
nft add chain inet firewall forward_early \
    '{ type filter hook forward priority -200; policy accept; }'

Now add your drop rule:

# Block port 3000 to all containers
nft add rule inet firewall forward_early \
    tcp dport 3000 drop

# Or block specific container IPs
nft add rule inet firewall forward_early \
    ip daddr 172.17.0.5 tcp dport 3000 drop

Verify the priority order:

nft list chains | grep forward

You should see your forward_early chain listed with priority -200, which executes before Docker's -100 chain.

Solution 2: Modify Docker's Table Directly

Instead of fighting Docker's priorities, inject rules into Docker's own chains. This approach is cleaner for container-specific policies.

# Insert at the beginning of Docker's forward chain
nft insert rule inet docker forward \
    tcp dport 3000 drop

# Or match by container network
nft insert rule inet docker forward \
    iifname "br-a1b2c3d4e5f6" tcp dport 3000 drop

The insert keyword places your rule at the top of the chain, before Docker's blanket accept rules. This works because you're operating within Docker's priority level.

I use this method in production to enforce per-network policies. Each Docker Compose stack gets its own bridge network, and we insert drop rules for admin ports (like Jupyter on 8888, or MLflow on 5000) directly into the inet docker forward chain.

Making Rules Persistent

Docker recreates its nftables rules on every daemon restart. Your manual nft commands vanish. You need a script that runs after Docker starts.

Create /etc/systemd/system/docker-firewall.service:

[Unit]
Description=Docker nftables Firewall Rules
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-firewall-rules.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then create /usr/local/bin/docker-firewall-rules.sh:

#!/bin/bash
set -e

# Wait for Docker's nftables table to exist
max_attempts=10
attempt=0
while ! nft list table inet docker >/dev/null 2>&1; do
    attempt=$((attempt + 1))
    if [ $attempt -ge $max_attempts ]; then
        echo "Docker nftables table not found after $max_attempts attempts"
        exit 1
    fi
    sleep 1
done

# Insert drop rules for blocked ports
nft insert rule inet docker forward tcp dport 3000 drop
nft insert rule inet docker forward tcp dport 8888 drop
nft insert rule inet docker forward tcp dport 5000 drop

echo "Docker firewall rules applied"

Make it executable and enable:

chmod +x /usr/local/bin/docker-firewall-rules.sh
systemctl daemon-reload
systemctl enable docker-firewall.service
systemctl start docker-firewall.service

The Table Family Trap

One gotcha: if Docker uses inet (which handles both IPv4 and IPv6), your rules must also use inet. A rule in an ip table won't see IPv6 traffic, and Docker's inet chains will still forward it.

Always match table families:

# Wrong - only catches IPv4
nft add table ip firewall
nft add chain ip firewall forward_early \
    '{ type filter hook forward priority -200; }'

# Right - catches both stacks
nft add table inet firewall
nft add chain inet firewall forward_early \
    '{ type filter hook forward priority -200; }'

Debugging Chain Execution

When rules don't work, trace the packet path:

# Enable packet tracing for port 3000
nft add rule inet firewall forward_early \
    tcp dport 3000 meta nftrace set 1

# In another terminal, watch the trace
nft monitor trace

Then trigger traffic to port 3000. You'll see exactly which chains and rules the packet hits, in order. This shows you where Docker's chains accept the packet before your drop rule fires.

For production debugging, I prefer logging over tracing:

nft insert rule inet docker forward \
    tcp dport 3000 log prefix "DOCKER-BLOCK-3000: " drop

Then tail /var/log/syslog or /var/log/kern.log to see blocked connection attempts with full packet details.

What About iptables-nft?

If you're using iptables-nft (the nftables backend for iptables commands), Docker's rules still win. The iptables commands generate nftables rules in a compatibility table, but Docker's native inet docker table has its own priority scheme.

The solution is the same: create chains with appropriate priorities, or modify Docker's chains directly. Don't rely on legacy iptables commands to override nftables-native Docker rules.

Originally published at fivenineslab.com

Running Gemma 2 27B Locally: MLX vs vLLM vs llama.cpp Performance Comparison

augustine Egbuna — Tue, 07 Apr 2026 01:34:39 +0000

You run Gemma 2 27B on MLX the day it drops, feed it some multimodal prompts, and get nonsense hallucinations. Meanwhile, Reddit threads are full of people saying it's the best 27B model yet. Something doesn't add up.

The problem isn't the model — it's the inference harness. Each framework makes different tradeoffs in quantization, attention implementation, and memory layout. Run the same model on MLX, vLLM, and llama.cpp, and you'll get three different experiences. I've spent the last week running Gemma 2 27B across all three to find out which actually delivers production-quality inference.

Why Your MLX Results Look Wrong

MLX optimizes for Apple Silicon's unified memory architecture, but Gemma 2's architecture fights it. The model uses sliding window attention with local and global attention heads — a pattern that doesn't map cleanly to MLX's matrix operations. When you quantize to 4-bit with MLX's default quantization scheme, those attention patterns degrade fast.

Here's what most people run on Mac:

from mlx_lm import load, generate

model, tokenizer = load(
    "mlx-community/gemma-2-27b-it-4bit",
    tokenizer_config={"trust_remote_code": True}
)

response = generate(
    model, 
    tokenizer, 
    prompt="Describe this image: <image>",
    max_tokens=512,
    temp=0.7
)

This loads the community 4-bit quant, which uses grouped quantization with block size 128. For text-only prompts, it's fine. For vision or long-context tasks, the quantization errors compound. You're not seeing the model's true capabilities — you're seeing quantization artifacts.

The fix: use the official MLX 8-bit quant or run bf16 if you have 64GB+ unified memory. The 8-bit version uses a different quantization scheme that preserves attention head outputs better:

model, tokenizer = load(
    "mlx-community/gemma-2-27b-it-8bit",  # Official 8-bit quant
    tokenizer_config={"trust_remote_code": True}
)

# Same generate call, noticeably better outputs

On an M2 Ultra with 192GB, this runs at ~28 tokens/sec for coding tasks. Hallucinations drop significantly. But you're still bottlenecked by MLX's single-device constraint — no multi-GPU, no batching across requests.

vLLM: Production Throughput on NVIDIA Hardware

If you're running on Linux with NVIDIA GPUs, vLLM is the answer. It implements PagedAttention, continuous batching, and efficient KV cache management. For Gemma 2 27B, this means 3-4x higher throughput than naive implementations.

Deploy it with Docker:

# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:v0.6.3
    command: >
      --model google/gemma-2-27b-it
      --dtype bfloat16
      --max-model-len 8192
      --gpu-memory-utilization 0.9
      --tensor-parallel-size 2
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    shm_size: 16gb

This runs Gemma 2 27B sharded across 2x A100 40GB GPUs. The --gpu-memory-utilization 0.9 tells vLLM to use 90% of GPU memory for KV cache — critical for high batch throughput. With continuous batching enabled, you'll serve 15-20 concurrent requests at ~45 tokens/sec per request.

Test it with curl:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-2-27b-it",
    "prompt": "Write a Python function to parse YAML",
    "max_tokens": 256,
    "temperature": 0.3
  }'

For coding tasks, vLLM with bf16 precision produces clean, accurate outputs. No hallucinations, consistent structure. The difference from 4-bit MLX is night and day.

llama.cpp: The Middle Ground

You're on Mac, don't want to spin up cloud GPUs, but need better quality than 4-bit MLX. llama.cpp with Q5_K_M or Q6_K quantization splits the difference.

Build from source with Metal support:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_METAL=1

# Download a quality quant
curl -L -o gemma-2-27b-it-Q6_K.gguf \
  https://huggingface.co/bartowski/gemma-2-27b-it-GGUF/resolve/main/gemma-2-27b-it-Q6_K.gguf

# Run with context optimized for coding
./llama-cli \
  -m gemma-2-27b-it-Q6_K.gguf \
  -n 512 \
  -c 8192 \
  --temp 0.3 \
  --top-p 0.9 \
  -ngl 999 \
  -p "Write a Rust function to validate JSON schema"

The -ngl 999 offloads all layers to Metal. Q6_K quantization keeps 6-bit weights with K-quant optimization — better precision than 4-bit, manageable memory footprint. On M2 Max with 64GB, this runs at ~22 tokens/sec.

For vision tasks that caused hallucinations in MLX, llama.cpp with Q6_K produces coherent descriptions. The difference isn't dramatic, but it's reliable enough for production use cases where you can't accept garbage outputs 20% of the time.

Real Performance Numbers

I ran the same coding benchmark across all three setups — 50 Python function generation tasks, measured by pass@1 on unit tests:

MLX 4-bit: 58% pass rate, 28 tok/s, frequent off-topic generations
MLX 8-bit: 74% pass rate, 26 tok/s, reliable structure
llama.cpp Q6_K: 76% pass rate, 22 tok/s, consistent quality
vLLM bf16 (2x A100): 81% pass rate, 45 tok/s, production-grade

vLLM wins on quality and throughput, but you're paying for cloud GPUs. For local Mac development, llama.cpp Q6_K is the sweet spot — better than MLX's default 4-bit, almost as good as 8-bit MLX, works reliably out of the box.

What Actually Matters for Your Use Case

If you're doing exploratory coding on Mac, start with llama.cpp Q6_K. It just works, no Python environment conflicts, no MLX quirks with certain prompt formats.

If you're building an API that serves multiple users, run vLLM on rented NVIDIA hardware. The throughput and batching efficiency pay for themselves after 10-20 concurrent users.

If you're locked into the Apple ecosystem with 128GB+ unified memory and want Python integration, use MLX with 8-bit quants. Skip the 4-bit community models — they're fine for demos, broken for real work.

The model quality is there. You just need to stop using inference harnesses that throw away half the precision to save memory you probably don't need to save.

Originally published at fivenineslab.com

How to Block Docker Ports with nftables Without Getting Bypassed

augustine Egbuna — Tue, 07 Apr 2026 01:33:33 +0000

You add an nftables rule to drop traffic on port 8080. You check the ruleset — it's active. You curl localhost:8080 from outside the host, and the Dockerized API responds anyway. Your firewall just got ignored.

This isn't a configuration mistake. Docker deliberately writes its own iptables rules that execute before nftables ever sees the packet. If you're running GPU inference services, internal LLM APIs, or any container that shouldn't be internet-facing, this behavior is a production security gap.

Why Docker Bypasses Your Firewall

Docker manipulates iptables-legacy directly, inserting DNAT rules in the nat table and ACCEPT rules in the filter table. These rules redirect incoming traffic to container IPs before your nftables ruleset runs.

Check what Docker created:

sudo iptables-legacy -t nat -L DOCKER -n -v
sudo iptables-legacy -t filter -L DOCKER -n -v

You'll see entries like:

DNAT  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.17.0.2:8080

The packet gets rewritten and forwarded before your nftables input chain ever evaluates it. Even if you block port 8080 in nftables, Docker's NAT rule already sent the traffic to the container.

On modern Debian and Ubuntu systems, nftables is the default firewall backend. But Docker still uses iptables-legacy for compatibility. This creates two parallel firewall systems — and Docker's rules win.

The Fix: Disable Docker's iptables Manipulation

Stop Docker from writing iptables rules. Edit /etc/docker/daemon.json:

{
  "iptables": false
}

Restart Docker:

sudo systemctl restart docker

Now Docker won't touch your firewall. But you've also disabled container NAT and port publishing. If you run docker run -p 8080:8080 myapp, the port mapping silently fails. The container starts, but nothing listens on the host.

You now manage all forwarding and NAT yourself in nftables.

Build Your Own Docker NAT in nftables

You need three components:

DNAT for inbound traffic (external → container)
SNAT for outbound traffic (container → internet)
Forwarding rules between host and Docker bridge

Here's a complete nftables configuration for a single container exposing port 8080:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif "lo" accept
    # Allow SSH
    tcp dport 22 accept
    # Block direct access to 8080 from outside
    # Traffic will arrive via DNAT as forwarded packets
  }

  chain forward {
    type filter hook forward priority 0; policy drop;
    ct state established,related accept
    # Allow forwarding to Docker containers
    iif "eth0" oif "docker0" ip daddr 172.17.0.2 tcp dport 8080 accept
    # Allow container responses
    iif "docker0" oif "eth0" accept
  }

  chain output {
    type filter hook output priority 0; policy accept;
  }
}

table ip nat {
  chain prerouting {
    type nat hook prerouting priority -100; policy accept;
    # DNAT: external traffic on 8080 → container
    iif "eth0" tcp dport 8080 dnat to 172.17.0.2:8080
  }

  chain postrouting {
    type nat hook postrouting priority 100; policy accept;
    # SNAT: container outbound traffic → host IP
    oif "eth0" ip saddr 172.17.0.0/16 masquerade
  }
}

Save this as /etc/nftables.conf and apply:

sudo nft -f /etc/nftables.conf

Replace 172.17.0.2 with your container's IP. Find it with:

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container_name>

Selective Exposure: Allow Only Internal Networks

If you want the container reachable only from your private network (not the internet), add a source filter in the DNAT rule:

iif "eth0" ip saddr 10.0.0.0/8 tcp dport 8080 dnat to 172.17.0.2:8080

This allows connections from RFC1918 space but drops everything else before DNAT happens.

For GPU inference APIs or internal vector search endpoints, this prevents accidental internet exposure while keeping the service available to your application tier.

Handling Multiple Containers

For multiple published ports, add one DNAT rule and one forward rule per container:

# Container 1: LLM API on 8080
iif "eth0" tcp dport 8080 dnat to 172.17.0.2:8080
iif "eth0" oif "docker0" ip daddr 172.17.0.2 tcp dport 8080 accept

# Container 2: Vector DB on 9200
iif "eth0" tcp dport 9200 dnat to 172.17.0.3:9200
iif "eth0" oif "docker0" ip daddr 172.17.0.3 tcp dport 9200 accept

For a dynamic container environment, this manual approach doesn't scale. Use Docker networks with explicit binds (--publish 127.0.0.1:8080:8080) so the service listens only on localhost, then manage external access through an nginx reverse proxy protected by nftables.

Enable nftables on Boot

Make the ruleset persistent:

sudo systemctl enable nftables
sudo systemctl start nftables

On Debian/Ubuntu, nftables reads /etc/nftables.conf at boot. Verify the service is active:

sudo systemctl status nftables

What You Lose

With "iptables": false, Docker Compose port mappings (ports: - "8080:8080") stop working unless you manually configure nftables NAT. Docker networks still function for inter-container communication, but host publishing requires your explicit forwarding rules.

For production GPU clusters running inference APIs, this tradeoff is worth it. You control exactly which ports are exposed and to whom. A single nftables ruleset governs all traffic — no hidden Docker rules bypassing your firewall.

Verification

Test the block:

# From outside the host
curl http://<host-ip>:8080
# Should fail if no DNAT rule exists

Add the DNAT rule, reload nftables, and retry. The request should reach the container.

Check your ruleset matches what you expect:

sudo nft list ruleset

Verify Docker didn't sneak in iptables rules:

sudo iptables-legacy -t nat -L DOCKER
# Should be empty or show "Chain DOCKER (0 references)"

If Docker re-created rules, it means daemon.json wasn't applied. Restart the daemon and double-check the JSON syntax.

Use Cases for Manual Firewall Control

This pattern matters when:

Running inference APIs on GPU instances where accidental exposure costs money and leaks proprietary models
Operating multi-tenant platforms where container isolation must be firewall-enforced, not just network-namespace-enforced
Deploying internal RAG pipelines with vector databases that should never touch the public internet
Meeting compliance requirements that demand explicit, auditable firewall rules for all published services

Docker's automatic iptables manipulation is convenient for development. In production infrastructure, convenience is a security liability. You need deterministic control over which packets reach which containers.

Originally published at fivenineslab.com