I built a private ChatGPT with Flask and Ollama. Here's what tripped me up.

I wanted a chatbot that runs entirely on my laptop and access it from anywhere from my mobile phone. No API key, no per-token bill, and no prompts leaving the machine. Ollama runs the model, and Flask is the thinnest web layer I know, so the plan looked like an afternoon's work.

It mostly was. But a few things cost me more time than they should have, so here they are, along with the numbers I got when I benchmarked it.

The whole thing is about 100 lines of Python and HTML: ampaslabs/python-flask-ollama-chatbot (MIT).

1. Ollama speaks OpenAI, but only under /v1

Ollama has an OpenAI-compatible API, so you can use the regular openai Python package and switch to a hosted model later by changing one line. My first attempt looked like this:

client = OpenAI(base_url="http://localhost:11434", api_key="ollama")

All it gave me was:

NotFoundError: Error code: 404

The OpenAI-compatible routes live under /v1. Without it, the SDK calls /chat/completions on the Ollama root, which doesn't exist:

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

The api_key is required by the SDK and ignored by Ollama, so any string works.

2. Streaming in Flask is a generator, not a websocket

I assumed word-by-word streaming would need websockets or Server-Sent Events. It doesn't. Ask the model for a stream, then wrap a generator in a plain Response:

@app.route("/api/chat", methods=["POST"])
def chat():
    history = (request.get_json(silent=True) or {}).get("messages", [])
    if not history:
        return jsonify({"error": "messages is required"}), 400

    messages = [{"role": "system", "content": SYSTEM_PROMPT}] + history[-20:]
    stream = client.chat.completions.create(model=MODEL, messages=messages, stream=True)

    def generate():
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content

    return Response(stream_with_context(generate()), mimetype="text/plain")

On the browser side, fetch gives you a reader:

const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  reply += decoder.decode(value, { stream: true });
  bubble.textContent = reply;
}

curl -N is the quickest way to check whether streaming actually works. If the text shows up all at once, something between you and Flask is buffering. For Nginx, that's proxy_buffering off.

3. The model has no memory, so the browser keeps it

LLMs are stateless. "Tell me more about it" only works if the previous turns are sent along with it. The browser keeps a messages array and posts the whole thing every time. The server adds the system prompt and keeps the last 20 messages (history[-20:] above), so a long chat doesn't overflow the context window.

It's simple, and it means the Flask app holds no state at all, which comes in handy in point 5.

4. Use textContent, never innerHTML, for model output

It's tempting to use innerHTML so the model's Markdown renders nicely. Don't. Model output is untrusted input. A prompt like "reply with a script tag that‚Ķ" is enough to inject HTML into your own page. I insert replies with textContent, which is why you can see raw **bold** markers in the screenshot. If you want rendered Markdown, run it through marked and then DOMPurify.

5. Concurrency: the queue is in Ollama, not Flask

This was the surprise. I wrote a small benchmark script that uses only the standard library. It measures tokens/sec from Ollama's own timing data, then time to first token through the Flask app, first with one chat and then with three at once.

Apple M1 Pro, 16 GB, llama3.2 (3B), Ollama 0.32.15, gunicorn with -w 1 --threads 4:

Test	Result
Generation speed (Ollama direct)	39.1 tokens/sec
Time to first token, 1 chat	0.05 s median
Total reply time, 1 chat (~114 tokens)	2.30 s median
Time to first token, 3 chats at once	2.23 s p50 / 4.68 s p95

Flask adds almost nothing: the first token arrives about 50 ms after the request. With three people chatting at once, though, time to first token jumps to 2‚Äì5 seconds. That isn't Flask's fault. Ollama queues requests for the same model. More gunicorn workers won't help, because the model doesn't run in your Python process. What helps is setting OLLAMA_NUM_PARALLEL on a machine with enough memory to serve requests in parallel.

So I run it with one worker and a few threads, which is enough to hold several streaming connections open:

gunicorn -w 1 --threads 4 -b 127.0.0.1:5001 app:app

Two smaller gotchas

• Port 5000 on macOS. AirPlay Receiver uses it, and Flask fails with Address already in use. I moved the app to 5001 instead of turning AirPlay off.
• Debug mode. debug=True is great while you're building. But the Werkzeug debugger lets anyone who can reach an error page run Python on your machine. Turn it off before anyone else can reach the app.

Using it from my phone

Once it worked, I wanted to use it away from my desk without setting up port forwarding, which doesn't work behind CGNAT anyway. Before you expose it, add a login such as HTTP Basic Auth, or anyone with the link gets free use of your GPU. Then open an HTTPS tunnel:

socketxp connect http://localhost:5001

Disclosure: I build SocketXP, so that's the tunnel I reach for. The Flask side is the same whichever tunnel you use. I wrote up the networking part, including 0.0.0.0, firewalls and ProxyFix, in Expose a Flask App to the Internet Without Port Forwarding.

Try it

ollama pull llama3.2
git clone https://github.com/ampaslabs/python-flask-ollama-chatbot.git
cd python-flask-ollama-chatbot
pip install -r requirements.txt
python3 app.py

The full step-by-step walkthrough is here.

If you run python3 benchmark.py on your own hardware, post your numbers in the comments. I'm especially curious how a Raspberry Pi 5 and a desktop RTX card compare with the M1.