I Built a Dead-Simple API Gateway for My Local LLMs in 50 Lines of Python

#ai #python #ollama #selfhosted

I Built a Dead-Simple API Gateway for My Local LLMs in 50 Lines of Python

I run three machines with local LLMs. A Mac Mini with an M4, a Windows box with an RTX 3060, and an Ubuntu server with a couple older GPUs. Each has Ollama installed. Each has different models loaded.

For months, I hardcoded URLs in my scripts. Need a quick answer? Query the Mac. Need a coding assistant? Hit the Windows machine. Need the big model? Wait for the Ubuntu server.

It was annoying. So I built a tiny API gateway that routes requests automatically. It took an afternoon. It runs on a single Python file. And it completely changed how I use my local AI setup.

The Problem: Three URLs, Zero Logic

Before the gateway, my scripts looked like this:

# quick_question.py
import requests

# Which machine do I use today?
# Mac Mini — fast, small models
response = requests.post("http://192.168.1.100:11434/api/generate", json={
    "model": "qwen2.5:7b",
    "prompt": "Explain Python decorators"
})

# code_review.py
import requests

# Windows — has the GPU
response = requests.post("http://192.168.1.106:11434/api/generate", json={
    "model": "qwen3-coder:30b",
    "prompt": "Review this function..."
})

# hard_question.py
import requests

# Ubuntu — has the most VRAM
response = requests.post("http://192.168.1.100:11434/api/generate", json={
    "model": "deepseek-r1:70b",
    "prompt": "Design a distributed task queue..."
})

Three scripts. Three URLs. Zero flexibility. If the Windows machine was offline, the coding script just failed. If I added a new model, I had to update everything manually.

The Fix: A Stupid-Simple Gateway

I wanted one URL. One API. Let the gateway figure out which machine can handle the request.

Here's what I built:

# gateway.py
from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

# My machines and what they can run
MACHINES = {
    "mac": {"url": "http://192.168.1.100:11434", "models": ["qwen2.5:7b", "granite3.2-vision:2b"]},
    "windows": {"url": "http://192.168.1.106:11434", "models": ["qwen3-coder:30b", "deepseek-r1:8b"]},
    "ubuntu": {"url": "http://192.168.1.100:11434", "models": ["deepseek-r1:70b", "minicpm-v"]},
}

def find_machine(model):
    for name, cfg in MACHINES.items():
        if model in cfg["models"]:
            return cfg["url"]
    return None

@app.route("/api/generate", methods=["POST"])
def generate():
    data = request.get_json()
    model = data.get("model")

    if not model:
        return jsonify({"error": "No model specified"}), 400

    machine_url = find_machine(model)
    if not machine_url:
        return jsonify({"error": f"Model {model} not found on any machine"}), 404

    try:
        response = requests.post(
            f"{machine_url}/api/generate",
            json=data,
            timeout=300
        )
        return response.json(), response.status_code
    except requests.exceptions.ConnectionError:
        return jsonify({"error": f"Machine for {model} is offline"}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=11435)

That's it. 50 lines. Run it on any machine, point all your scripts at http://gateway:11435, and forget about which box has which model.

Why This Is Better Than I Expected

I can move models around. When I got a new GPU for the Windows machine, I moved the big coding model there. Changed one line in MACHINES. Every script kept working.

Health checks are trivial. I added a /health endpoint that pings each machine. If one is down, my main script knows and routes around it.

Load balancing is obvious. If two machines have the same model, I can pick whichever is less busy. I haven't needed this yet, but the structure supports it.

My scripts got dumber. In a good way. They don't need to know about the infrastructure anymore. They just ask for a model and get an answer.

What I Didn't Build (On Purpose)

No database. No config files. No Docker. No Kubernetes. No "service mesh."

This is a single Python file with a dictionary. If I need to change something, I edit the file and restart it. Takes 10 seconds.

I thought about making it "proper" — YAML configs, hot reloading, Prometheus metrics. But this is for my home lab. I'm the only user. Complexity is the enemy.

The Real Win: Mental Overhead

Before the gateway, using my local AI felt like work. I'd open a script, remember which machine had which model, check if it was online, then query it.

Now it feels like... using an API. Any API. I don't think about the infrastructure. I just write the prompt and get the result.

That's the whole point of infrastructure: it should disappear.

Numbers (Because Why Not)

Lines of Python: 50
Time to build: 2 hours (including testing)
Time saved per week: ~30 minutes of "which machine is this on again?"
Additional dependencies: Flask (already installed for other projects)
Cost: $0

Getting Started

If you have multiple Ollama instances, you can literally copy-paste the script above, change the IPs and models, and be done.

pip install flask requests
python gateway.py

Then in your scripts:

import requests

# One URL. Any model. Gateway handles the rest.
response = requests.post("http://localhost:11435/api/generate", json={
    "model": "qwen3-coder:30b",
    "prompt": "Refactor this function..."
})

The Honest Bottom Line

Is this production-ready? No. Does it handle edge cases? Barely. Is it good enough for my home lab? Absolutely.

Sometimes the right architecture is the one you'll actually maintain. A 50-line Python file I can debug in my head beats a "proper" solution I'd never finish.

If you're running multiple Ollama instances and manually switching between them — just build the gateway. It takes an afternoon and saves you from ever thinking about machine IPs again.

Sam Hartley is a solo dev running a multi-machine AI home lab in Turkey. Writes about the boring infrastructure that makes local AI actually usable.

→ Custom automation setups on Fiverr
→ Follow CelebiBots on Telegram