Build Your First Local AI Chat App with Ollama and Python: A Step-by-Step Guide

Lokendra Rewapati — Wed, 14 Oct 2026 16:12:36 +0000

An AI demo often starts with a few lines of code: send a prompt, receive an answer, display it on a screen.

That is a useful beginning. But it leaves several questions unanswered.

Where is the model running? Does it remember earlier messages? Why is the first response slow? What happens when the model server stops? And why can a confident answer still be wrong?

In this first article, we will build a local chat application and use it to understand the components behind those questions. The goal is a small application whose behavior you can inspect and change.

What you will build: a Python chat interface that sends conversation history to a local model, displays its answer, measures response time, and handles common failures.

What you will learn: the difference between a model and its runtime, how chat history becomes model input, why context needs limits, and what must change before this becomes a production service.

1. The three components behind the chat window

Our application has three main components:

Component	Responsibility	Our choice
User interface	Accept input, display messages, keep session state	Streamlit
Inference runtime	Load model weights and execute generation	Ollama
Language model	Generate an answer from the supplied context	`qwen3:1.7b`

The model is the learned set of parameters. The runtime is the software that loads those parameters and performs inference. The application controls the conversation and decides what to send.

Ollama exposes an HTTP interface on your machine. The Python application calls it; your browser displays the Python application's interface.

There are two local services to keep distinct: the chat interface usually runs on port 8501, while Ollama uses port 11434.

For this lab, inference runs locally using a downloaded model. Installation and model download require internet access. We do not use a paid model API.

Reference repositories

These are the upstream projects used in this tutorial:

You do not need to clone their source repositories. We install their tools and write a small application that uses them.

2. Prerequisites and realistic hardware expectations

Use Python 3.11 or 3.12 for this tutorial, a terminal, and a code editor. The commands below cover macOS, Linux, and Windows PowerShell.

We use a small model so the first example is accessible. The Ollama catalog lists the qwen3:1.7b download at approximately 1.4 GB when this draft was prepared. Download size is not total runtime memory: model state, context, the operating system, and the application need additional memory.

An 8 GB RAM machine is a practical starting point, rather than a guaranteed minimum. Available memory and the selected context size matter. More memory or supported GPU acceleration can improve the experience; a small model can also run on CPU, with hardware-dependent latency.

Do not compare this small teaching model's answer quality with a much larger hosted model and assume local execution caused the difference. Model capability and deployment location are separate choices.

3. Install Ollama and download the model

Install Ollama using the instructions for your operating system in the official quickstart. On macOS and Windows, launch the installed application. On Linux, follow the service instructions provided by the installer.

Verify the CLI:

ollama --version

Download the model:

ollama pull qwen3:1.7b

List installed models:

ollama list

Try a terminal conversation:

ollama run qwen3:1.7b

Ask: Explain an API in three short sentences. Exit the conversation with /bye.

If the runtime is not running, start it in a separate terminal:

ollama serve

An address-in-use error can mean the existing Ollama application or service already owns the port. Check the running service before starting a second copy.

4. Understand the API before building the interface

The chat endpoint accepts a model name and an array of messages. A message identifies a role and its content.

{
  "model": "qwen3:1.7b",
  "messages": [
    {"role": "system", "content": "Explain clearly and acknowledge uncertainty."},
    {"role": "user", "content": "What is an API?"}
  ],
  "stream": false,
  "think": false
}

The system message expresses application guidance. User messages contain requests. Assistant messages contain earlier answers that we choose to send back.

We disable streaming in the initial example so one request produces one complete JSON response. We also disable Qwen3's thinking mode to keep this first lab focused on visible answers. Thinking support and its controls depend on the model; this setting is not universal across all models.

Use think only with a current Ollama version and a model that supports it. The chat API reference documents the request and response fields.

5. Create an isolated Python environment

Create a folder named local-ai-chat and open a terminal inside it.

On macOS or Linux:

python3 -m venv .venv
source .venv/bin/activate

On Windows PowerShell:

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1

If PowerShell blocks activation, you can call the environment's Python directly rather than changing machine-wide security settings:

.\.venv\Scripts\python.exe -m pip install streamlit

In the activated environment, install Streamlit:

python -m pip install streamlit

The HTTP client in this lab uses Python's standard library, so there is no additional client dependency. For a published release, preserve the exact resolved dependency versions after validating the application:

python -m pip freeze > requirements.txt

A requirements file records Python packages. Record the Python version, Ollama version, and model identity separately; all can affect reproducibility.

6. Build the complete chat application

Create app.py with this code:

import json
import time
import urllib.error
import urllib.request

import streamlit as st

ENDPOINT = "http://127.0.0.1:11434/api/chat"
MODEL = "qwen3:1.7b"
SYSTEM = (
    "You are a helpful technical tutor. Explain clearly. "
    "When you are uncertain, say so. Do not invent citations."
)
MAX_INPUT_CHARS = 2000
MAX_PREVIOUS_MESSAGES = 6  # Three complete user/assistant turns.


def call_model(messages):
    payload = {
        "model": MODEL,
        "messages": messages,
        "stream": False,
        "think": False,
        "options": {
            "temperature": 0.2,
            "num_ctx": 4096,
            "num_predict": 512,
        },
    }
    request = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=180) as response:
        result = json.load(response)
    if result.get("error"):
        raise ValueError(str(result["error"]))
    answer = result.get("message", {}).get("content")
    if not isinstance(answer, str) or not answer.strip():
        raise ValueError("The runtime returned no visible answer.")
    return answer, result


st.set_page_config(page_title="Local AI Chat", page_icon="💬")
st.title("Local AI Chat")
st.caption(f"Model: {MODEL} · Runtime: Ollama · Local inference")

if "messages" not in st.session_state:
    st.session_state.messages = []
if "last_run" not in st.session_state:
    st.session_state.last_run = None

with st.sidebar:
    st.header("Conversation controls")
    st.write("Each request includes the last three completed turns.")
    if st.button("Clear conversation"):
        st.session_state.messages = []
        st.session_state.last_run = None
        st.rerun()
    if st.session_state.last_run:
        st.subheader("Last successful request")
        st.json(st.session_state.last_run)

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

prompt = st.chat_input("Ask a question")
if prompt:
    prompt = prompt.strip()
    if not prompt:
        st.warning("Enter a question first.")
        st.stop()
    if len(prompt) > MAX_INPUT_CHARS:
        st.warning(f"Keep this lab's input under {MAX_INPUT_CHARS} characters.")
        st.stop()

    user_message = {"role": "user", "content": prompt}
    history = st.session_state.messages[-MAX_PREVIOUS_MESSAGES:]
    messages = [{"role": "system", "content": SYSTEM}, *history, user_message]

    with st.chat_message("user"):
        st.markdown(prompt)

    try:
        started = time.perf_counter()
        with st.chat_message("assistant"):
            with st.spinner("Generating locally..."):
                answer, result = call_model(messages)
            elapsed = time.perf_counter() - started
            st.markdown(answer)
            st.caption(f"Complete response received in {elapsed:.2f} seconds")
    except urllib.error.HTTPError as exc:
        st.error(f"Ollama returned HTTP {exc.code}. Check its logs and model name.")
    except urllib.error.URLError:
        st.error("Cannot reach Ollama. Check that the local runtime is running.")
    except (TimeoutError, OSError):
        st.error("The request timed out or the connection failed. Try a shorter input.")
    except (ValueError, KeyError, TypeError):
        st.error("The runtime returned an unexpected response. Check Ollama's logs.")
    else:
        # Commit only complete, successful turns to conversation history.
        st.session_state.messages.extend([
            user_message,
            {"role": "assistant", "content": answer},
        ])
        st.session_state.last_run = {
            "elapsed_seconds": round(elapsed, 2),
            "input_tokens": result.get("prompt_eval_count"),
            "output_tokens": result.get("eval_count"),
            "done_reason": result.get("done_reason"),
        }
        st.rerun()

Start the app, keeping Ollama running:

python -m streamlit run app.py --server.address 127.0.0.1 --browser.gatherUsageStats false

On Windows without activation, use .\.venv\Scripts\python.exe in place of python.

Open http://localhost:8501. You should see the chat title, input box, and conversation controls. The Ollama service is still a separate process.

7. Follow one request through the application

Ask: Explain a reverse proxy using a simple example.

The application strips whitespace, checks the input size, selects recent history, and adds the system message. Python serializes the request as JSON and sends it to the local runtime. Ollama performs inference and returns a JSON response. The application extracts the visible answer and commits a complete conversation turn to session state.

That sequence exposes an important engineering boundary: generation is one operation inside a larger application flow. Input handling, state updates, error handling, and measurement remain our responsibility.

Why save a turn only after success?

Imagine that we save the user message, then the model request fails. The next request could inherit an unanswered turn. Repeated failures could create several adjacent user messages and confusing history.

This app saves the user message and assistant answer together after a successful call. Failed attempts can be logged separately in a larger application, while the model-facing conversation stays coherent.

8. Conversation history is supplied context

Ask these two questions in sequence:

My example application is a school admissions portal. Suggest three useful API endpoints.
Which of those should require authentication?

The second question depends on the first. Our application makes the reference understandable by resending previous messages.

Now clear the conversation and ask the second question again. The earlier context is absent. The model may ask for clarification or give a generic answer; the exact wording varies.

The model's knowledge learned during training, the application's conversation history, and persistent user memory are different mechanisms. This lab implements temporary session history. It does not store a user profile or update model weights.

Also notice that the interface displays the full session history while only the last three completed turns are sent. An old message being visible on screen does not mean it is in the model's current context.

9. Context limits: characters are only a rough guard

We use three separate controls:

Control	Purpose	Limitation
2,000 input characters	Prevent very large individual submissions	Characters do not correspond directly to tokens
Three previous turns	Keep history growth bounded for the lab	A fixed turn count can discard relevant information
4,096-token context setting	Request a bounded model context	It does not prove the constructed input fits

Tokens can be words, parts of words, punctuation, or other text units. Code, multilingual text, and tables tokenize differently. The assembled request includes instructions, history, the new question, and formatting overhead.

A production application should budget using the selected model's tokenizer or a validated estimation strategy, reserve generation space according to runtime behavior, and remove or summarize context deliberately. Do not rely on silent truncation preserving the most important information.

Our output cap also means a response can stop before it is complete. Inspect done_reason and the runtime's behavior before treating a short answer as a successful explanation.

10. Measure latency without inventing a benchmark

The sidebar shows elapsed time and token counts from the most recent successful request.

We measure time around the complete HTTP call. That includes runtime work and the response transfer. It is not time to first token because this version does not stream.

The first request may involve loading the model. Later requests may benefit from a warm runtime. Compare multiple runs on the same hardware, with comparable inputs and output lengths, before drawing conclusions.

For your own measurements, record the hardware, available memory, model identity, Ollama version, prompt, output length, and whether the runtime was warm. A single screenshot of a fast response is not a performance study.

Low temperature can reduce variation in generation, but it does not make responses correct or guarantee identical results across runtimes and hardware.

11. Make failure part of the exercise

Use these checks before calling the setup complete:

Exercise	What to check
Ask a short question	A nonempty answer appears and metrics update
Ask a dependent follow-up	The response uses the recent conversation
Clear the conversation	Messages and previous metrics disappear
Submit over 2,000 characters	The app rejects the request before inference
Stop Ollama and ask a question	An error appears; no failed turn enters saved history
Ask for a fact about an unseen private document	Recognize that the model has no evidence from that document

The final exercise matters as much as the first. A fluent answer is not evidence that the application accessed a source or verified a claim. Our chat application has no retrieval layer, browser, or database tool.

12. Troubleshooting

Problem	Likely cause	Next action
`ollama` command missing	Installation or PATH issue	Complete the OS-specific install and reopen the terminal
Connection refused	Runtime unavailable	Launch Ollama or start its service
HTTP error after model request	Missing model or incompatible request	Check `ollama list`, runtime logs, and installed version
Model is very slow	CPU execution, memory pressure, or cold loading	Close heavy apps, shorten context, and compare warm runs
`streamlit` is missing	Wrong Python environment	Install using the same Python that launches the app
Answer ends abruptly	Output limit reached or generation stopped	Inspect the stop reason before raising the output cap
Earlier details are forgotten	History window excluded them	Inspect the messages actually sent

We deliberately do not add automatic retries. Retrying a long generation can double work and hide persistent failure. If retries are added later, define which errors are transient and bound their number and total duration.

13. What changes for production?

This lab teaches the basic request path. A shared service adds responsibilities:

Identity and isolation: authenticate users and isolate their conversation state.
Durability: persist conversations with retention and deletion rules instead of relying on session state.
Capacity: limit concurrent requests, apply backpressure, and handle cancellation.
Evaluation: test factual accuracy, task success, and regressions on representative examples.
Observability: track failures, token usage, latency, model versions, and request identifiers.
Evidence: add retrieval or tools when answers require information outside the supplied context.

Local execution changes where inference happens. It does not automatically secure logs, browser sessions, or a publicly exposed service. Keep this lab bound to the loopback interface and use synthetic inputs while learning.

Streaming would improve perceived responsiveness, but it introduces partial answers and cancellation behavior. Retrieval would add evidence, but also indexing, access controls, and evaluation. Tool calling would add actions, which require validation and permissions outside model instructions.

Each extension is an engineering problem worth studying separately.

14. Try these small experiments

Experiment A: Remove history. Send only the system message and new question. Compare dependent follow-ups with the original implementation.

Experiment B: Change the output cap. Compare short and long requested explanations. Record token count, elapsed time, and whether the answer finishes.

Experiment C: Compare instruction styles. Ask the same question with a tutor instruction and an incident-analysis instruction. Judge relevance and correctness separately from tone.

Change one variable at a time. Otherwise, a better answer does not tell you which change helped.

What we have built

We now have a small application with explicit model input, bounded recent history, local inference, measurable request behavior, and visible failure handling.

The chat window is the easiest part to see. The more useful understanding is underneath it: an application decides what context enters the model, how requests execute, and what happens when the result is unavailable or wrong.

Next, we will add documents and build the first retrieval pipeline. That is where the question changes from “Can the model answer?” to “Did the system supply the evidence needed to answer?”

DEV Community: Lokendra Rewapati