Stop Guessing Your Meds: Building a Multimodal RAG Assistant with LLaVA and ChromaDB

#ai #tutorial #opensource #discuss

Ever stared at a cryptic medicine bottle, wondering if it interacts with your morning coffee or that other pill you're taking? For the elderly or those with visual impairments, reading tiny labels on medication packaging is more than a nuisance—it’s a safety hazard.

In this tutorial, we are building a Medication Safety Assistant. This isn't just a simple OCR tool; we are implementing a Multimodal Retrieval-Augmented Generation (RAG) pipeline. We'll use LLaVA (Large Language-and-Vision Assistant) to "see" the medicine box, ChromaDB to store and retrieve detailed medical instructions, and Ollama to run everything locally and privately.

By the end of this guide, you'll understand how to bridge the gap between computer vision and structured knowledge retrieval to build life-saving AI applications. 🚀

The Architecture: How Vision Meets Knowledge

Traditional RAG handles text. Multimodal RAG allows our system to process an image, convert the visual features into a query, and then fetch the relevant "truth" from a local vector database.

graph TD
    A[User Uploads Photo of Medicine] --> B[LLaVA via Ollama]
    B --> C{Identify Brand & Active Ingredients}
    C --> D[Generate Search Query]
    D --> E[(ChromaDB - Medical Knowledge)]
    E --> F[Retrieve Safety Guidelines & Dosage]
    F --> G[LLaVA Reasoning + Context]
    G --> H[Final Safety Instructions & UI]
    style B fill:#f96,stroke:#333,stroke-width:2px
    style E fill:#69f,stroke:#333,stroke-width:2px

Prerequisites 🛠️

To follow along, ensure you have the following installed:

Ollama: To serve LLaVA locally.
Python 3.10+
Tech Stack: LLaVA, Ollama, ChromaDB, Gradio.

pip install chromadb ollama gradio sentence-transformers

Step 1: Setting up the Vector Knowledge Base

Before we can identify medicine, we need a "brain" containing the actual medical instructions. We'll use ChromaDB to store embeddings of medicine names and their corresponding contraindications.

import chromadb
from chromadb.utils import embedding_functions

# Initialize ChromaDB
client = chromadb.PersistentClient(path="./med_db")
default_ef = embedding_functions.DefaultEmbeddingFunction()
collection = client.get_or_create_collection(name="medicine_docs", embedding_function=default_ef)

# Mock Data: In a real app, you'd parse PDFs of medical leaflets
med_data = [
    {"id": "001", "name": "Ibuprofen", "text": "Do not take with Aspirin. Max 1200mg/day. Avoid alcohol."},
    {"id": "002", "name": "Metformin", "text": "Used for Type 2 Diabetes. May cause stomach upset. Take with meals."},
]

for med in med_data:
    collection.add(
        documents=[med["text"]],
        metadatas=[{"name": med["name"]}],
        ids=[med["id"]]
    )

Step 2: Vision Identification with LLaVA

Now, we use the LLaVA model via Ollama. Its job is to look at the image and extract the medicine name. LLaVA is incredible because it understands spatial relationships and can read text even on curved surfaces like pill bottles.

import ollama

def identify_medicine(image_path):
    with open(image_path, 'rb') as f:
        img_data = f.read()

    response = ollama.generate(
        model='llava',
        prompt='Identify the brand name and the active ingredients of the medicine in this image. Output only the names.',
        images=[img_data]
    )
    return response['response'].strip()

Step 3: The Multimodal RAG Logic

This is where the magic happens. We take the visual output from LLaVA, query our vector database, and then pass that context back to the model to generate a safe, conversational answer.

def safety_assistant(image_path):
    # 1. Vision Step
    identified_med = identify_medicine(image_path)
    print(f"Identified: {identified_med}")

    # 2. Retrieval Step
    results = collection.query(
        query_texts=[identified_med],
        n_results=1
    )

    context = results['documents'][0][0] if results['documents'] else "No specific safety data found."

    # 3. Final Reasoning Step
    final_prompt = f"""
    The user is asking about the medicine: {identified_med}.
    Based on the official medical database: {context}.
    Provide a concise safety warning and dosage instructions. 
    If there are no details found, warn the user to consult a doctor.
    """

    final_response = ollama.generate(model='llama3', prompt=final_prompt)
    return final_response['response']

The "Official" Way to Build AI 🥑

While building a local prototype is great for learning, deploying production-grade AI in highly regulated sectors like healthcare requires more robust patterns.

For advanced architectural patterns, such as Hybrid Search (combining keyword and semantic search) and Agentic RAG workflows, I highly recommend exploring the deep-dive articles at wellally.tech/blog. They provide excellent resources on scaling these LLM implementations for enterprise use cases where reliability is non-negotiable.

Step 4: Putting it all together with Gradio

Let’s wrap this in a user-friendly interface. Gradio allows us to create a functional UI in just a few lines of code.

import gradio as gr

def process_and_chat(image):
    # Save the uploaded image temporarily
    image.save("temp_input.jpg")
    return safety_assistant("temp_input.jpg")

interface = gr.Interface(
    fn=process_and_chat,
    inputs=gr.Image(type="pil"),
    outputs="text",
    title="AI Medication Safety Assistant 💊",
    description="Upload a photo of your medicine packaging to get safety warnings and dosage info."
)

if __name__ == "__main__":
    interface.launch()

Conclusion & Next Steps

We just built a multimodal system that can potentially save lives! By combining LLaVA for vision and ChromaDB for verified knowledge, we've created a prototype that is both smart and grounded in reality.

What's next?

OCR Refinement: Use specialized OCR models if LLaVA struggles with tiny fonts.
Multi-turn Dialogue: Let the user ask follow-up questions about the identified med.
Cross-Checking: Connect to an API (like OpenFDA) for real-time interaction checks.

What do you think? Would you trust an AI assistant to read your meds, or are we still a few years away? Let me know in the comments! 👇

If you enjoyed this tutorial, don't forget to follow for more "Learning in Public" AI guides! 🚀💻