Making My First AI Chat App: Learning From DevOps Pass AI's Ollama Integration

Justin K. — Fri, 06 Dec 2024 15:12:04 +0000

The blog i used as a source:

https://dev.to/devopspass-ai/workshop-make-your-first-ai-app-in-a-few-clicks-with-pythonollamallama3-31ib

After discovering DevOps Pass AI's guide on building an AI app with Ollama, I decided to explore how it works and document my questions and learnings along the way. Here's what I discovered while building my first AI chat application.

Initial Questions I Had

When I first read through the tutorial, several questions came to mind:

Why use Ollama instead of making direct API calls to OpenAI or other services?
What makes Llama3 a good choice for a local AI model?
How does the chat history persistence work, and why is it important?

Let's go through what I learned while exploring each of these aspects.

Understanding the Local AI Setup

The first interesting thing I noticed was the use of local AI through Ollama. After asking around and testing, I found some key advantages:

No API costs or usage limits
Complete privacy since everything runs locally
No internet dependency after initial model download
Surprisingly good performance with Llama3

The setup process was straightforward: (Bash)

ollama serve
ollama pull llama3

I was initially concerned about the 4.7GB model size, but the download was quick on my connection and it runs smoothly even on my modest development machine.

Exploring the Chat Application

The most intriguing part was how simple yet functional the chat application is. Let's break down what I learned about each component:

Chat History Management

I was particularly curious about how the chat history worked. The code uses a clever approach: (python)

file_path = sys.argv[1] + '.json'
if os.path.exists(file_path):
with open(file_path, 'r') as f:
messages = json.load(f)

This means each chat session maintains its own history file. I tested this by starting multiple conversations: (Bash)

python app1.py coding_help
python app1.py devops_queries

bashCopypython app1.py coding_help
python app1.py devops_queries
Each created its own JSON file, keeping conversations separate and persistent.
The AI Response Handling
One thing that caught my attention was the streaming response implementation:
pythonCopystream = ollama.chat(
model='llama3',
messages=messages,
stream=True,
)

for chunk in stream:
print(chunk['message']['content'], end='', flush=True)
This gives a much more natural feel to the conversation, as responses appear gradually like human typing rather than all at once.
Testing Different Use Cases
I experimented with various types of questions to understand the model's capabilities:

Technical Questions
Copy>>> How can I set up Kubernetes monitoring?
The responses were detailed and technically accurate.
Code Generation
Copy>>> Write a Python function to monitor CPU usage
It provided working code examples with explanations.
Contextual Conversations
Copy>>> What are the best practices for that?
The model maintained context from previous questions effectively.

What I Learned About Performance
Some interesting observations about running AI locally:

First response after starting is slightly slower (model warm-up)
Subsequent responses are quick
Response quality matches many cloud-based services
No throttling or rate limits to worry about

Questions I Still Have
After building and testing the application, I'm curious about:

How to fine-tune the model for specific use cases?
Can we optimize the model for faster responses?
What's the best way to handle errors or unexpected responses?

Conclusion: Is It Worth Building?
After experimenting with this setup, I'd say it's definitely worth trying if you:

Want to learn about AI integration
Need privacy-focused AI solutions
Are interested in building custom AI tools
Want to avoid API costs for AI services

The learning curve is surprisingly gentle, and the results are impressive for a local setup.
Questions for the Community

Has anyone else built similar local AI applications?
What other models have you tried with Ollama?
How are you handling error cases in your AI applications?

Let me know in the comments - I'm particularly interested in hearing about different use cases and improvements!

DEV Community: Justin K.

Making My First AI Chat App: Learning From DevOps Pass AI's Ollama Integration