DEV Community

Blastted
Blastted

Posted on AI-assisted

The Sloth's Guide to Support Ticket Triage: Benchmarking Zero-Shot vs. Few-Shot vs. QLoRA on Ollama

#devchallenge #hf26challenge #ai #opensource

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

🦥 The Dreaded Chore: Why Humans Shouldn't Triage Support Tickets

October 20 was International Sloth Day, and sloths have life figured out: conserve energy, move with precision, and let the world come to you.

If you have ever worked in tech support, IT operations, or software engineering, you know there is one daily routine that completely drains your will to live: manual support ticket triage.

Every morning, hundreds of tickets land in the queue. They range from polite bug reports to ALL-CAPS panic attacks about minor UI glitches, multi-language complaints, and messy copy-pasted stack traces. Someone has to sit there, read every single one, and manually assign:

  • category (Technical Bug, Billing, Account Access, Security & Compliance, Feature Request, General Inquiry)
  • urgency (Low, Medium, High, Critical)
  • department (Engineering, Finance, Tier-1 Support, Security)

It is monotonous, mentally taxing, and prone to human error when fatigue sets in.

So for Hacktoberfest 2026 Week 3, I channeled our inner sloth to build Open-Weight LLM Benchmark: an automated, end-to-end local evaluation pipeline and interactive Streamlit dashboard that benchmarks small open-weight language models to take over this chore completely.

⏱️ The Lazy Payoff: What Does This Actually Save?

To qualify for genuine sloth status, an automation has to show a real payoff:

  • Time Saved: Manual human triage takes between 45 to 90 seconds per ticket. For an organization receiving 200 tickets daily, that translates to 2.5 to 5 hours of manual triage labor every single day. Our fine-tuned local model triages tickets in ~5.4 seconds on local CPU—completely hands-free.
  • Zero Prompt Maintenance: Rather than writing and maintaining brittle prompt templates with multi-shot examples that consume hundreds of tokens per call, a single compact instruction reliably yields the exact schema.
  • Cognitive Relief: No more context-switching between your actual engineering sprint and the ticket queue. Support escalations route immediately to the right team without human delay.

🔓 Why Open Innovation & Local Weights Matter

Why not just ping a closed cloud API like GPT-4o or Claude? When building operational automations for real companies, closed models create three major friction points:

  1. Data Privacy & Compliance (Zero Data Leaks): Support tickets contain sensitive customer data—billing invoice IDs, email addresses, internal stack traces, and security breach reports. Sending this data across third-party cloud APIs triggers GDPR, HIPAA, and SOC-2 compliance hurdles. Running open-weight models locally with Ollama ensures customer data never leaves your private network.
  2. Zero API Cost & Predictable Infrastructure: Cloud API token bills accumulate rapidly as ticket volume spikes. Open weights run on hardware you already own with zero per-token inference charges.
  3. Weight Sovereignty & Custom Domain Adaptation: You cannot fine-tune closed frontier models cheaply or alter their internal representation of business rules. With open weights (Meta's Llama 3.2 3B Instruct), we can train LoRA adapters using Unsloth in 15 minutes on a free Google Colab T4 GPU, bake operational SLA boundaries directly into the model weights, and export a 2.0 GB quantized GGUF (Q4_K_M) file.

🚧 The Engineering Bottlenecks with Small Base Models

When testing off-the-shelf small language models (1B–3B parameters) for automated ticket triage, we hit three immediate production roadblocks:

  1. Unreliable Structured Outputs (JSON Failures): Off-the-shelf instruction models love to chat. They wrap their answers in conversational filler ("Sure! Here is the JSON you requested: \\json..."`), breaking downstream automated webhooks and parsers.
  2. The "Tone vs. Reality" Trap (Urgency Mismatch): Base models confuse loudness with severity. When an angry customer writes: > "🚨 URGENT 911 P0!! CANNOT SLEEP!! Change the button color from indigo to navy blue!"

Base models routinely classify it as Critical, when the operational SLA is clearly Low. Conversely, when a calm researcher submits a clean report leaking SSH keys in an open S3 bucket, base models misclassify it as Low or Medium.

  1. The Few-Shot Token Tax & Latency Penalty: Adding 5 few-shot examples to the prompt guides the base model towards better formatting, but it inflates prompt token counts by 4x (from 156 tokens to 612 tokens) and adds a severe latency penalty during local CPU inference.

🧪 The Experiment: Zero-Shot vs. 5-Shot vs. QLoRA Fine-Tuned

To solve this, we designed a reproducible benchmark:

  • Base Model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
  • Dataset: 1,200 custom-engineered customer support tickets split into:
    • data/finetune_train_1000.csv (1,000 training rows in ChatML format)
    • data/benchmark_test_200.csv (200 unseen holdout evaluation rows)
  • Adversarial Diversities: 30.5% of the evaluation dataset consists of production traps:
    • Cross-Category Overlap (e.g., a React TypeError crash while clicking a Billing button)
    • Urgency Mismatch (Priority Inflation vs. Calm Criticals)
    • Noisy Typos & Hinglish/Multilingual ("bro wtf yaar 😭 my acount got debited 2 times... paisa kat gaya bank se")
    • Adversarial Traps (Tickets containing embedded JSON error logs designed to break json.loads())
  • Fine-Tuning: Parameter-efficient fine-tuning (QLoRA) using Unsloth and Hugging Face TRL (SFTTrainer) with rank $r=16$, $\alpha=16$, targeting all projection modules (q, k, v, o, gate, up, down). Crucially, we used train_on_responses_only to mask loss solely to the assistant's strict JSON response.
  • Local Deployment: Merged LoRA adapters, converted to Q4_K_M 4-bit GGUF, and served via Ollama (Modelfile with temperature 0.0).

🏆 Benchmark Results (200 Unseen Holdout Tickets on Local CPU)

We evaluated all three tiers locally via Ollama on the 200 holdout tickets with strict Pydantic v2 validation:

Model Tier Category Macro-F1 Urgency Macro-F1 Exact Match (All 3 Fields) Strict JSON (json.loads) Schema Valid Avg Latency / Ticket Avg Prompt Tokens
1. Zero-Shot Base (llama3.2:3b) 61.7% 47.3% 41.5% 10.0% 96.5% 6,093.3 ms 156
2. Few-Shot Base (5-Shot) (llama3.2:3b) 72.9% 60.4% 57.0% 99.5% 99.5% 6,567.8 ms 612
3. QLoRA Fine-Tuned (ticket-classifier-ft) 100.0% 96.1% 96.5% 100.0% 100.0% 5,481.3 ms 146
Delta (Fine-Tuned vs Zero-Shot) +38.3% +48.8% +55.0% +90.0% +3.5% -612.0 ms -10 toks
Delta (Fine-Tuned vs 5-Shot) +27.1% +35.7% +39.5% +0.5% +0.5% -1,086.5 ms (-16.5%) -466 toks (-76.1%)

Key Analytical Takeaways

  1. Eliminating the Wrapper Tax (10.0% $\rightarrow$ 100.0% Strict JSON): In Zero-Shot mode, llama3.2:3b wrapped 90% of outputs in markdown code blocks or introductory fluff. QLoRA with response-only loss masking forced the model to emit raw, parseable JSON strings 100% of the time without needing regex cleanup or external grammar engines.
  2. Conquering Tone vs. Reality (47.3% $\rightarrow$ 96.1% Urgency F1): Fine-tuning taught the model operational SLAs instead of surface sentiment matching.
  3. Speed AND Accuracy Win: While 5-shot prompting improved Exact Match from 41.5% to 57.0%, it quadrupled prompt tokens and inflated CPU latency. The fine-tuned model achieved 96.5% Exact Match while cutting prompt tokens by 76.1% and running 1,086 ms faster per ticket.

📸 Visualizing the Results

1. Executive Scorecard

The comparative performance overview across all three model tiers:

Executive Scorecard Dashboard

2. Performance Across Edge-Case Diversities

How each model tier handled standard tickets vs. noisy typos, Hinglish, cross-category overlaps, and urgency mismatches:

Diversity Edge-Case Breakdown

3. Confusion Matrices

Comparing classification confusion between Zero-Shot, 5-Shot, and QLoRA:

Confusion Matrix Comparision

🖥️ Interactive Streamlit Dashboard

To make exploring the benchmark intuitive, we built a Streamlit application featuring:

  1. Executive Scorecard: High-level macro metrics, latency distributions, and throughput indicators.
  2. Confusion Matrices: Interactive heatmaps for Category, Urgency, and Department predictions.
  3. Error Inspector: Filter by failure mode to inspect exact ticket text, ground truth, and what each model predicted.
  4. Live Inference Arena: Test live tickets against your local Ollama instance in real-time.

📦 Try It Out & Reproduce

The complete dataset, training notebook, Ollama Modelfile, and evaluation code are open-source:

  • GitHub Repository: abhas20/LLM-Benchmark
  • Tech Stack: Python 3.11, Unsloth, Meta Llama 3.2, Ollama, Pydantic v2, Streamlit, Plotly, uv.

Happy Hacktoberfest, and remember to channel your inner sloth: automate the tedious stuff with open-source AI! 🦥🚀

Top comments (0)