DEV Community

qing
qing

Posted on

Build a Local LLM Chatbot with Ollama and Python

#llm #ai #tutorial #python

Build a Local LLM Chatbot with Ollama and Python

Build a Local LLM Chatbot with Ollama and Python

Imagine typing a question into your chatbot and getting a response in milliseconds, completely offline, with zero data leaving your machine. No API keys, no monthly subscription fees, and no privacy concerns about your data being sent to a cloud server. This isn’t a futuristic dream—it’s the reality of running a Local Large Language Model (LLM) on your own computer. With the rise of tools like Ollama, building a private AI chatbot in Python has become as simple as installing a few packages and writing a short script. Let’s dive in and build one together.

Why Go Local?

Before we write any code, it’s worth understanding why running an LLM locally is a game-changer. Cloud-based AI services like OpenAI or Anthropic are powerful, but they come with trade-offs: you pay per token, your data is processed on their servers, and you’re dependent on their uptime. A local LLM flips this model. You download the model once, run it on your hardware, and you have full control.

Ollama is the engine that makes this accessible. It’s a lightweight, open-source tool that simplifies running LLMs like Llama 3, Phi 3, or Mistral on macOS, Linux, and Windows. It handles model downloads, memory management, and inference, exposing a simple API that Python can easily interact with [1][2].

Step 1: Install Ollama and Pull a Model

The first step is getting Ollama on your machine. Visit ollama.com, click Download, and install the version for your operating system [2]. Once installed, verify it’s working by opening your terminal or Command Prompt and running:

ollama --version
Enter fullscreen mode Exit fullscreen mode

If you see a version number, you’re ready to go. Next, you need a model. Ollama supports dozens of open-source models, but for a beginner-friendly chatbot, Llama 3.2 is a great choice. It’s small, fast, and surprisingly capable.

To download it, run:

ollama pull llama3.2
Enter fullscreen mode Exit fullscreen mode

This command fetches the model and stores it locally. Depending on your internet speed, this might take a few minutes [2][7]. Once it’s done, you can test it directly in the terminal:

ollama run llama3.2
Enter fullscreen mode Exit fullscreen mode

Type a question like “What’s the capital of France?” and see the model respond. If you get a reply, Ollama is working perfectly.

Step 2: Set Up Your Python Environment

Now let’s build the Python side. First, create a project folder:

mkdir local-llm-chatbot
cd local-llm-chatbot
Enter fullscreen mode Exit fullscreen mode

Inside this folder, create a virtual environment to manage your dependencies cleanly. On macOS or Linux:

python3 -m venv chatbot
source chatbot/bin/activate
Enter fullscreen mode Exit fullscreen mode

On Windows (Command Prompt):

python3 -m venv chatbot
.\chatbot\Scripts\activate.bat
Enter fullscreen mode Exit fullscreen mode

On Windows (PowerShell):

.\chatbot\Scripts\Activate.ps1
Enter fullscreen mode Exit fullscreen mode

Once the environment is active, install the necessary Python packages:

pip install langchain langchain-ollama ollama
Enter fullscreen mode Exit fullscreen mode

We’re using LangChain and langchain-ollama because they provide a clean, high-level interface for interacting with Ollama models, making our code shorter and more maintainable [2][3][7].

Step 3: Write the Chatbot Code

Open your code editor (VS Code is a great choice) and create a file called main.py. Here’s a complete, working Python script that creates a simple chatbot with conversation history:

from langchain_ollama import OllamaLLM

# Initialize the model
model = OllamaLLM(model="llama3.2")

# Conversation history to maintain context
history = []

print("🤖 Local LLM Chatbot (powered by Ollama + Llama 3.2)")
print("Type 'exit' to quit.\n")

while True:
    # Get user input
    user_input = input("You: ").strip()
    if user_input.lower() == "exit":
        print("Chatbot: Goodbye!")
        break

    # Add user message to history
    history.append({"role": "user", "content": user_input})

    # Generate response
    response = model.invoke(history)

    # Print and store response
    print(f"Chatbot: {response}")
    history.append({"role": "assistant", "content": response})

    # Optional: limit history to last 10 messages to save memory
    if len(history) > 20:
        history = history[-10:]
Enter fullscreen mode Exit fullscreen mode

Save the file and run it:

python main.py
Enter fullscreen mode Exit fullscreen mode

You’ll see a chat interface where you can type questions and get responses from Llama 3.2, all running locally. The history list ensures the model remembers previous messages, giving you a more natural conversation experience [1][2].

Step 4: Customize and Expand

This is just the foundation. Here are a few ways to make it even better:

  • Change the model: Try phi3, mistral, or gemma by pulling them with ollama pull phi3 and updating the model parameter in the code.
  • Add a web UI: Use Streamlit or Flask to create a browser-based interface. For example, Streamlit lets you build a chat UI in under 50 lines of code [6].
  • Implement RAG: If you want your chatbot to answer questions from your own documents (like PDFs or Word files), you can add a RAG (Retrieval-Augmented Generation) layer using LangChain and ChromaDB [2][8].
  • Add tools: Let your chatbot call custom functions (e.g., fetch weather, search the web) by defining tools in LangChain [8].

Troubleshooting Tips

  • Model not found: Make sure you ran ollama pull llama3.2 and that Ollama is running (ollama serve).
  • Slow responses: Smaller models (like phi3) are faster but less capable. If you have a GPU, Ollama will automatically use it for faster inference.
  • Memory issues: If you get a “out of memory” error, try a smaller model or reduce the conversation history length.

Start Building Your Private AI Today

You now have a fully functional, local LLM chatbot running on your machine. No API keys, no subscriptions, no data leaks. Just pure, private AI.

The best part? This is just the beginning. You can extend this chatbot to read your documents, automate tasks, or even integrate it into your existing apps. The local AI revolution is happening right now, and you’re part of it.

Try it out today: Install Ollama, pull Llama 3.2, run the script above, and start chatting with your own AI. Share your results on Dev.to, tag me, and let’s build the future of private AI together. 🚀

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.

💡 Related: **Content Creator Ultimate Bundle (Save 33%)* — $29.99*

📧 Get my FREE Python Cheatsheet → Follow me on Dev.to and drop a comment below — I'll DM you the cheatsheet directly!

🐍 50+ essential Python patterns, one-liners, and best practices for everyday development. Free for all readers.

Top comments (0)