DEV Community: Blastted

The Sloth's Guide to Support Ticket Triage: Benchmarking Zero-Shot vs. Few-Shot vs. QLoRA on Ollama

Blastted — Sun, 25 Oct 2026 16:35:45 +0000

🦥 The Dreaded Chore: Why Humans Shouldn't Triage Support Tickets

October 20 was International Sloth Day, and sloths have life figured out: conserve energy, move with precision, and let the world come to you.

If you have ever worked in tech support, IT operations, or software engineering, you know there is one daily routine that completely drains your will to live: manual support ticket triage.

Every morning, hundreds of tickets land in the queue. They range from polite bug reports to ALL-CAPS panic attacks about minor UI glitches, multi-language complaints, and messy copy-pasted stack traces. Someone has to sit there, read every single one, and manually assign:

category (Technical Bug, Billing, Account Access, Security & Compliance, Feature Request, General Inquiry)
urgency (Low, Medium, High, Critical)
department (Engineering, Finance, Tier-1 Support, Security)

It is monotonous, mentally taxing, and prone to human error when fatigue sets in.

So for Hacktoberfest 2026 Week 3, I channeled our inner sloth to build Open-Weight LLM Benchmark: an automated, end-to-end local evaluation pipeline and interactive Streamlit dashboard that benchmarks small open-weight language models to take over this chore completely.

⏱️ The Lazy Payoff: What Does This Actually Save?

To qualify for genuine sloth status, an automation has to show a real payoff:

Time Saved: Manual human triage takes between 45 to 90 seconds per ticket. For an organization receiving 200 tickets daily, that translates to 2.5 to 5 hours of manual triage labor every single day. Our fine-tuned local model triages tickets in ~5.4 seconds on local CPU—completely hands-free.
Zero Prompt Maintenance: Rather than writing and maintaining brittle prompt templates with multi-shot examples that consume hundreds of tokens per call, a single compact instruction reliably yields the exact schema.
Cognitive Relief: No more context-switching between your actual engineering sprint and the ticket queue. Support escalations route immediately to the right team without human delay.

🔓 Why Open Innovation & Local Weights Matter

Why not just ping a closed cloud API like GPT-4o or Claude? When building operational automations for real companies, closed models create three major friction points:

Data Privacy & Compliance (Zero Data Leaks): Support tickets contain sensitive customer data—billing invoice IDs, email addresses, internal stack traces, and security breach reports. Sending this data across third-party cloud APIs triggers GDPR, HIPAA, and SOC-2 compliance hurdles. Running open-weight models locally with Ollama ensures customer data never leaves your private network.
Zero API Cost & Predictable Infrastructure: Cloud API token bills accumulate rapidly as ticket volume spikes. Open weights run on hardware you already own with zero per-token inference charges.
Weight Sovereignty & Custom Domain Adaptation: You cannot fine-tune closed frontier models cheaply or alter their internal representation of business rules. With open weights (Meta's Llama 3.2 3B Instruct), we can train LoRA adapters using Unsloth in 15 minutes on a free Google Colab T4 GPU, bake operational SLA boundaries directly into the model weights, and export a 2.0 GB quantized GGUF (Q4_K_M) file.

🚧 The Engineering Bottlenecks with Small Base Models

When testing off-the-shelf small language models (1B–3B parameters) for automated ticket triage, we hit three immediate production roadblocks:

Unreliable Structured Outputs (JSON Failures): Off-the-shelf instruction models love to chat. They wrap their answers in conversational filler ("Sure! Here is the JSON you requested: \\json..."`), breaking downstream automated webhooks and parsers.
The "Tone vs. Reality" Trap (Urgency Mismatch): Base models confuse loudness with severity. When an angry customer writes: > "🚨 URGENT 911 P0!! CANNOT SLEEP!! Change the button color from indigo to navy blue!"

Base models routinely classify it as Critical, when the operational SLA is clearly Low. Conversely, when a calm researcher submits a clean report leaking SSH keys in an open S3 bucket, base models misclassify it as Low or Medium.

The Few-Shot Token Tax & Latency Penalty: Adding 5 few-shot examples to the prompt guides the base model towards better formatting, but it inflates prompt token counts by 4x (from 156 tokens to 612 tokens) and adds a severe latency penalty during local CPU inference.

🧪 The Experiment: Zero-Shot vs. 5-Shot vs. QLoRA Fine-Tuned

To solve this, we designed a reproducible benchmark:

Base Model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
Dataset: 1,200 custom-engineered customer support tickets split into:
- data/finetune_train_1000.csv (1,000 training rows in ChatML format)
- data/benchmark_test_200.csv (200 unseen holdout evaluation rows)
Adversarial Diversities: 30.5% of the evaluation dataset consists of production traps:
- Cross-Category Overlap (e.g., a React TypeError crash while clicking a Billing button)
- Urgency Mismatch (Priority Inflation vs. Calm Criticals)
- Noisy Typos & Hinglish/Multilingual ("bro wtf yaar 😭 my acount got debited 2 times... paisa kat gaya bank se")
- Adversarial Traps (Tickets containing embedded JSON error logs designed to break json.loads())
Fine-Tuning: Parameter-efficient fine-tuning (QLoRA) using Unsloth and Hugging Face TRL (SFTTrainer) with rank $r=16$, $\alpha=16$, targeting all projection modules (q, k, v, o, gate, up, down). Crucially, we used train_on_responses_only to mask loss solely to the assistant's strict JSON response.
Local Deployment: Merged LoRA adapters, converted to Q4_K_M 4-bit GGUF, and served via Ollama (Modelfile with temperature 0.0).

🏆 Benchmark Results (200 Unseen Holdout Tickets on Local CPU)

We evaluated all three tiers locally via Ollama on the 200 holdout tickets with strict Pydantic v2 validation:

Model Tier	Category Macro-F1	Urgency Macro-F1	Exact Match (All 3 Fields)	Strict JSON (`json.loads`)	Schema Valid	Avg Latency / Ticket	Avg Prompt Tokens
1. Zero-Shot Base (`llama3.2:3b`)	61.7%	47.3%	41.5%	10.0%	96.5%	6,093.3 ms	156
2. Few-Shot Base (5-Shot) (`llama3.2:3b`)	72.9%	60.4%	57.0%	99.5%	99.5%	6,567.8 ms	612
3. QLoRA Fine-Tuned (`ticket-classifier-ft`)	100.0%	96.1%	96.5%	100.0%	100.0%	5,481.3 ms	146
Delta (Fine-Tuned vs Zero-Shot)	+38.3%	+48.8%	+55.0%	+90.0%	+3.5%	-612.0 ms	-10 toks
Delta (Fine-Tuned vs 5-Shot)	+27.1%	+35.7%	+39.5%	+0.5%	+0.5%	-1,086.5 ms (-16.5%)	-466 toks (-76.1%)

Key Analytical Takeaways

Eliminating the Wrapper Tax (10.0% $\rightarrow$ 100.0% Strict JSON): In Zero-Shot mode, llama3.2:3b wrapped 90% of outputs in markdown code blocks or introductory fluff. QLoRA with response-only loss masking forced the model to emit raw, parseable JSON strings 100% of the time without needing regex cleanup or external grammar engines.
Conquering Tone vs. Reality (47.3% $\rightarrow$ 96.1% Urgency F1): Fine-tuning taught the model operational SLAs instead of surface sentiment matching.
Speed AND Accuracy Win: While 5-shot prompting improved Exact Match from 41.5% to 57.0%, it quadrupled prompt tokens and inflated CPU latency. The fine-tuned model achieved 96.5% Exact Match while cutting prompt tokens by 76.1% and running 1,086 ms faster per ticket.

📸 Visualizing the Results

1. Executive Scorecard

The comparative performance overview across all three model tiers:

2. Performance Across Edge-Case Diversities

How each model tier handled standard tickets vs. noisy typos, Hinglish, cross-category overlaps, and urgency mismatches:

3. Confusion Matrices

Comparing classification confusion between Zero-Shot, 5-Shot, and QLoRA:

🖥️ Interactive Streamlit Dashboard

To make exploring the benchmark intuitive, we built a Streamlit application featuring:

Executive Scorecard: High-level macro metrics, latency distributions, and throughput indicators.
Confusion Matrices: Interactive heatmaps for Category, Urgency, and Department predictions.
Error Inspector: Filter by failure mode to inspect exact ticket text, ground truth, and what each model predicted.
Live Inference Arena: Test live tickets against your local Ollama instance in real-time.

📦 Try It Out & Reproduce

The complete dataset, training notebook, Ollama Modelfile, and evaluation code are open-source:

GitHub Repository: abhas20/LLM-Benchmark
Tech Stack: Python 3.11, Unsloth, Meta Llama 3.2, Ollama, Pydantic v2, Streamlit, Plotly, uv.

Happy Hacktoberfest, and remember to channel your inner sloth: automate the tedious stuff with open-source AI! 🦥🚀

LedgerGone: Local WhatsApp Khata & Receipt Auditor for Kirana Stores using Multimodal Gemma 3 and TigerData

Blastted — Sun, 18 Oct 2026 12:57:19 +0000

What I Built

Small Kirana (mom-and-pop grocery) and hardware shopkeepers in India receive dozens of supplier invoices, handwritten paper slips (parchis), UPI payment screenshots (PhonePe, GPay), and WhatsApp voice notes ("Sharma ji delivered 10 boxes of soap for Rs 4,500, paid 1,500 cash, 3,000 balance pending") every day.

At closing time, tallying up scattered notes is a nightmare. Yet, shopkeepers strongly resist uploading their sensitive margin, customer credit (udhaar), and tax data to third-party cloud accounting SaaS.

LedgerGone is a 100% local-first, privacy-preserving Khata and receipt auditor for Kirana and small retail stores. It combines:

Multimodal Gemma 3: Ingests receipt photos, handwritten parchis, and WhatsApp voice notes directly (with zero external OCR pipelines) to extract structured line items, GST, payments, and pending balances.
TigerData (TimescaleDB + pgvector): Stores time-series financial events in partitioned hypertables for cash-flow analytics and indexes line items using pgvector embeddings for semantic natural language inventory search.
1-Click WhatsApp Khata Statements: Automatically generates polite, audited WhatsApp payment reminders with Click-to-Chat deep links (wa.me) and downloadable PDF statements.
Pure CPU Optimization: Runs seamlessly on standard laptops without requiring a dedicated GPU.

Why Open Innovation Matters (The "Side Hustle" Edge)

Hacktoberfest Week 2 asks builders: Why does open innovation matter for your project, and why would someone pay for it?

Unit Economics That Actually Work: Kirana stores operate on tight 4–8% margins. Traditional AI apps charging per-token cloud API bills destroy the economics of a small business tool. Because Gemma 3 and FastEmbed run locally on the shopkeeper's CPU, the cost to audit a receipt is ₹0.00. This enables LedgerGone to be sold as an affordable one-time setup or ₹299/month local utility with 100% gross margins.
Absolute Data Sovereignty & Privacy: Tax invoices, cash entries, and supplier discounts never leave the merchant's machine. Shopkeepers don't have to worry about data harvesting or tax exposure from cloud accounting providers.
Offline Resilience: In tier-2/3 Indian cities and rural markets where internet connectivity fluctuates, a local-first system guarantees the shop's counter operations never stall.

System Architecture

flowchart TD
    subgraph Inputs["Kirana Counter Inputs"]
        A1["📷 Supplier GST Bills"]
        A2["📝 Handwritten Parchis"]
        A3["📱 UPI Screenshots"]
        A4["🎙️ WhatsApp Voice Notes"]
    end

    subgraph AI["Local AI Engine (CPU Optimized)"]
        B1["🧠 Multimodal Gemma 3 (Ollama)"]
        B2["⚡ FastEmbed ONNX (384-dim Vectors)"]
    end

    subgraph Database["TigerData (TimescaleDB + pgvector)"]
        C1[("⏰ TimescaleDB Hypertables<br/>Daily Cash Flow & Time-Series Audit")]
        C2[("🔍 pgvector HNSW Index<br/>Semantic Line Item & Price History")]
    end

    subgraph Outputs["Merchant Actions"]
        D1["📊 Streamlit Web Dashboard"]
        D2["💬 1-Click WhatsApp Khata Reminders"]
        D3["📄 Audited PDF Statements"]
    end

    Inputs --> B1
    B1 -->|Structured JSON| B2
    B2 -->|Vectors & Items| Database
    B1 -->|Audited Ledger| Database
    Database --> D1
    D1 --> D2
    D1 --> D3

Key Features

1. Direct Multimodal Auditing (No OCR Preprocessing)

Gemma 3's native multimodal capabilities process messy handwritten parchis and thermal receipts directly in a single pass. It extracts:

Vendor/Party name
Line items (item, quantity, rate, total)
GST & tax breakdown
Paid amount vs. pending udhaar balance
Payment mode (UPI / Cash / Credit)

2. Time-Series Khata Analytics (TimescaleDB Hypertables)

Every audit entry is committed as a hypertable event. LedgerGone uses Timescale's time_bucket('1 day', created_at) to calculate daily sales, collections, and outstanding credit lines in sub-millisecond queries.

3. Semantic Inventory Search with pgvector

Shopkeepers frequently ask questions like "When did we last buy basmati rice and at what rate?" or "Show me all cement and hardware bills". With 384-dimensional vector embeddings stored in TigerData's pgvector index, LedgerGone enables instant semantic similarity matching (<=> cosine distance).

4. 1-Click WhatsApp Reminders & PDF Export

Merchants can select any customer or supplier to generate a polite, formatted WhatsApp summary. Clicking "Send via WhatsApp" opens https://wa.me/ with the pre-filled statement, ready to send. An official PDF Khata statement can also be downloaded with one click.

Tech Stack

Language: Python 3.12+
Frontend: Streamlit
Vision & NLP: Gemma 3 Multimodal (via Ollama)
Time-Series & Vector DB: TigerData (TimescaleDB PostgreSQL 18 + pgvector)
Embeddings: FastEmbed (ONNX CPU Runtime, BAAI/bge-small-en-v1.5, 384 dimensions)
PDF Engine: ReportLab

Partner Track Qualifications

🏆 Best Use of Gemma ($200): Leverages Gemma 3 as an end-to-end multimodal document and voice auditor for Indian retail accounts.
🐯 Best Use of Tiger Data ($100): Integrates TimescaleDB hypertables for time-series cash flow aggregation alongside pgvector for semantic line-item search.

Code & Links

GitHub Repository: https://github.com/abhas20/LedgerGone
Hacktoberfest Week 2 Entry: #hf26challenge

Introduction to Open-Weights AI Model

Blastted — Tue, 13 Oct 2026 04:10:08 +0000

" Your journey is yours to define, your dreams are yours to pursue, and your story is yours to write. " -- with this let me start my first post

This is a new topic I am exploring currently, so decided to share what I understood so far, and will be grateful if you can add or correct me if I am wrong anywhere , so that it will help me as well as any beginner reading this

What is Open-Weight AI Model??

An open-weight AI model is an AI model whose trained parameters — called weights — are made available for people to download, run, and sometimes fine-tune for their own applications.
Some Popular open-weights AI model include -- Gemma (By Google DeepMind) , gpt-oss (By OpenAI), Qwen, Llama (By Meta) and many more.

SO we can download the weights but how much RAM do we need to run ? The Formula is :

WEIGHT Memory (in GB) = (Parameters * Bits per Parameter) / 8

Parameters - No. of Parameters in billions(B) [For example - 4B,8B,70B]
Bits per Parameters - Bits used to store each parameter [For Example - BF16/FP16(16 bits), INT8(8 bits) ]
8 - number of bytes

Also you need additional memory for KV Cache and additional Runtime Overhead

TOTAL MEMORY = WEIGHT MEMORY + KV CACHE MEMORY + RUNTIME OVERHEAD

So for example, a 7B model in 4-bit quantization has roughly 3.5 GB of theoretical weight storage. But it may require additional space for KV Cache and other overhaed so it may approx to 7-8GB.

What is a PARAMETER?

You might be familiar with relationship like this in simple ML:

Price = w1(Area) + w2(Bedrooms) + b
Here w1,w2 are nothing but parameters

A parameter is a numerical value inside an AI model that it learns during training. These values help the model recognize patterns, understand language, write code, and generate answers.

Does more parameters mean a smarter AI?

Not necessarily. A larger model often has greater capacity to learn complex patterns, but actual performance also depends on:

Training data quality and quantity.
Model architecture.
Training methods.
Fine-tuning and reasoning capabilities.
The task you're asking it to perform.

How to run and use a Open-Weight AI model ?

The simplest way is :

Download a pre-trained model
Load it in the memory
Using the loaded model use an interface to interact

The easiest way to do these is by using ollama , vLLM or any other tools

What are these tools ?

They are the engine and the pipeline that allow you to download open-source models (like Llama 3, Qwen, or Mistral) and run/deploy them yourself on your own hardware.

What they actually do?

When you install them on a computer:

They download the raw brain (the model weights) from open repositories like Hugging Face or the Ollama Library.
They load that model into your computer's RAM or GPU memory.
They spin up a local network address (like http://localhost:11434 or http://localhost:8000) on your machine.
Any app you write can now send text to that address, and your computer will generate the response—100% locally, for free, and completely offline.

Ollama vs vLLM

Ollama prioritizes simplicity: It acts like Docker for AI models. It manages the downloading, memory cleanup, and execution behind a very clean interface. It uses llama.cpp under the hood to squeeze models onto consumer hardware.
vLLM prioritizes raw performance: It uses an advanced memory management algorithm called PagedAttention and Continuous Batching. Instead of processing user requests sequentially (like Ollama mostly does under load), vLLM overlaps incoming queries dynamically, achieving up to 20x–24x higher throughput in multi-user environments

HOPE YOU LIKE IT - THIS IS SOMETHING NEW FOR ME TOO - SO WILL BE GRATEFUL IF YOU CAN SHARE YOUR THOUGHTS AND REVIEWS