We’ve all been there: staring at a crumpled prescription box, trying to decipher whether it says "once daily" or "twice daily" amidst a sea of fine print. In the world of healthcare, a small misinterpretation isn't just a nuisance—it's a safety risk.
Today, we are diving deep into Computer Vision and Multimodal AI to solve this. We're building an automated pipeline that uses Meta's Segment Anything Model (SAM) to isolate medicine labels, OCR to extract text, and LangChain to transform that chaotic text into structured, life-saving medication logic.
Whether you're interested in MedTech AI, OCR optimization, or the latest in Multimodal LLM workflows, this guide has something for you.
The Architecture 🏗️
Standard OCR often fails on medicine packaging because of busy backgrounds, 3D curves, and distracting logos. Our approach uses a "Segment-then-Read" strategy to ensure the highest possible data integrity.
graph TD
A[Raw Image of Medicine Box] --> B[SAM: Segment Anything]
B --> C{Region Proposal}
C -->|Identify Label| D[Isolate & Crop Label]
D --> E[OCR Engine: PaddleOCR/Tesseract]
E --> F[Raw Unstructured Text]
F --> G[LangChain + LLM]
G --> H[Structured JSON: Dosage, Frequency, Warnings]
H --> I[FastAPI Response]
Prerequisites 🛠️
Before we get our hands dirty, ensure you have the following in your tech stack:
- Segment Anything (SAM): For precise image segmentation.
- PaddleOCR / Tesseract: To convert pixels to characters.
- LangChain: To orchestrate our LLM logic.
- FastAPI: To wrap it all into a production-ready API.
Step 1: Isolating the Label with SAM 🧠
Standard bounding boxes are "so 2022." With SAM, we can generate precise masks even for oddly shaped bottles. We'll use the SamPredictor to identify the most likely label area.
import numpy as np
import torch
from segment_anything import sam_model_registry, SamPredictor
def segment_prescription_label(image_path):
# Load SAM model (Vit-H is recommended for high accuracy)
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
image = cv2.imread(image_path)
predictor.set_image(image)
# We can use a center-point prompt or automatic mask generation
masks, scores, _ = predictor.predict(
point_coords=np.array([[image.shape[1]//2, image.shape[0]//2]]),
point_labels=np.array([1]),
multimask_output=True,
)
# Select the mask with the highest confidence score
best_mask = masks[np.argmax(scores)]
return best_mask
Step 2: OCR Extraction 📄
Once we have the mask, we crop the image and pass it to our OCR engine. We use PaddleOCR here because of its superior performance with vertical text and medical fonts.
from paddleocr import PaddleOCR
def extract_text_from_mask(image, mask):
# Apply mask to isolate label
isolated_label = cv2.bitwise_and(image, image, mask=mask.astype(np.uint8))
ocr = PaddleOCR(use_angle_cls=True, lang='en')
result = ocr.ocr(isolated_label, cls=True)
raw_text = " ".join([line[1][0] for res in result for line in res])
return raw_text
Step 3: Structuring Logic with LangChain 🦜🔗
Raw OCR text is usually a mess of "500mg", "Take twice", and "Keep away from children." We need an LLM to parse this into a JSON schema that a mobile app can actually use.
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
from pydantic import BaseModel, Field
class MedicationLogic(BaseModel):
drug_name: str = Field(description="Name of the medicine")
dosage: str = Field(description="The amount to take per dose")
frequency: str = Field(description="How often to take the medicine")
warnings: list = Field(description="Crucial contraindications or warnings")
def parse_medication_text(text):
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = PromptTemplate.from_template(
"Extract medical instructions from this text: {text}. "
"Return ONLY structured JSON matching the schema."
)
chain = prompt | llm.with_structured_output(MedicationLogic)
return chain.invoke({"text": text})
Advanced Patterns & Best Practices 🥑
When moving this to production, you'll face challenges like lighting glare on plastic bottles and blurry "shaky-hand" photos.
If you are looking for more production-ready examples and advanced patterns regarding multimodal RAG or healthcare AI compliance, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover extensively how to handle edge cases in medical document processing that go beyond basic tutorials.
Key Optimization Tips:
- Image Pre-processing: Use CLAHE (Contrast Limited Adaptive Histogram Equalization) before SAM to help with glare.
- Confidence Thresholds: If the LLM's confidence is low, trigger a "Human-in-the-loop" flag.
- Entity Linking: Cross-reference extracted drug names with a database like RxNorm.
Step 4: Serving with FastAPI ⚡
Finally, let's wrap this into an endpoint.
from fastapi import FastAPI, UploadFile
app = FastAPI()
@app.post("/analyze-prescription")
async def analyze_prescription(file: UploadFile):
# 1. Save file & Read image
# 2. Run SAM Segmentation
# 3. Run OCR
# 4. Run LangChain Logic Extraction
logic = parse_medication_text(raw_ocr_text)
return {
"status": "success",
"data": logic
}
Conclusion 🏁
By combining the spatial intelligence of SAM with the linguistic reasoning of GPT-4 via LangChain, we've built more than just a scanner—we've built a digital safety net.
The future of healthcare lies in these multimodal bridges between the physical world and digital logic. What are you building with Vision models lately? Let me know in the comments! 👇
Happy coding! 🚀💻
For more insights into AI-driven health solutions, don't forget to visit wellally.tech/blog.
Top comments (0)