Built a text classifier that routes Indian citizen grievances (Hindi / Hinglish / English, code-mixed) into 14 government department categories, using a fine-tuned google/muril-base-cased. Final honest holdout metrics: 87.3% accuracy, 0.869 macro F1. Full writeup of the bugs I hit below, because the debugging was more instructive than the happy path.
Dataset: https://www.kaggle.com/datasets/abhisheksingh016/citizen-grievance-dataset
Notebook: https://www.kaggle.com/code/abhisheksingh016/citizen-grieveances/
Model: https://huggingface.co/pyofpython/citizen-grievance-classifier-muril
The problem
Citizen grievances in India get typed in whatever language/script mix is natural to the writer — Devanagari Hindi, Romanized Hindi ("Hinglish"), English, or all three in one sentence. Classic TF-IDF-style models can't bridge that: "पानी" and "paani" share zero characters, so a lexical model treats them as unrelated tokens even though they mean the same thing.
Bug #1: The eval that lied to me
First pass: 100% accuracy with a TF-IDF + ensemble (NB, LogReg, SVM, RandomForest, XGBoost, soft-voted). Too good. Turned out my train/test split let the exact same sentence templates appear on both sides — the model was matching template fingerprints, not learning language.
Fix: built a second evaluation set from templates held out entirely from training — genuinely unseen phrasing. Re-ran: accuracy dropped to 20–31%. That's the real number for a lexical model on code-mixed text.
Switching to MuRIL
google/muril-base-cased is pretrained on 17 Indian languages and their transliterated forms, so it embeds "पानी" and "paani" close together in vector space — solving the exact gap that broke the TF-IDF approach.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("google/muril-base-cased")
model = AutoModelForSequenceClassification.from_pretrained(
"google/muril-base-cased", num_labels=14
)
Bug #2: Shared RNG silently breaking unrelated categories
When I added more training templates to a few categories, completely unrelated categories (that I never touched) started scoring near-zero F1. Root cause: one global random instance shared across all category generation — adding templates upstream shifted the RNG's internal state, which changed which templates got selected as holdout for categories downstream in the dict.
Fix: give every category its own independent RNG, seeded from a hash of its name:
def category_rng(category):
seed = sum(ord(c) for c in category) * 7919 + len(category) * 104729
return random.Random(seed)
Now editing category A can never again perturb category B. Verified with a reproducibility check — identical MD5 hash across 3 repeated generation runs.
Bug #3: Real keyword bleed between categories
A Sanitation & Garbage template about sewage overflow happened to mention "paani" (water) — pulling it toward the Water Supply category. Similarly, an Employment template mentioned "ration" in passing. Both fixed by rewording to remove the borrowed vocabulary. Confirmed the fix worked: Water Supply and Sanitation both went from mediocre to 1.00 F1 in the next training run.
Bug #4: Kaggle disk quota crash mid-training
RuntimeError: [enforce fail at inline_container.cc:668]
Not a code bug — Kaggle's /kaggle/working has a ~20GB quota, and save_strategy="epoch" with no limit was writing a full model + optimizer checkpoint every epoch. Filled the disk by epoch 2.
Fix:
TrainingArguments(
...
load_best_model_at_end=True,
metric_for_best_model="macro_f1",
save_total_limit=1, # only keep the best checkpoint
)
Final results
Evaluated on the held-out, unseen-phrasing set (the honest one):
Metric Score
Accuracy: 87.3%
Macro F1: 0.869
Most categories land near-perfect (Water Supply, Sanitation, Healthcare, Land Records all ≥0.98 F1). The weak spot is Corruption & Bribery (F1 0.40) — genuinely ambiguous, since corruption complaints usually reference another department (a bribe-taking ration dealer, a bribe-taking clerk), not a labeling error.
Inference
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("pyofpython/citizen-grievance-classifier-muril")
model = AutoModelForSequenceClassification.from_pretrained("pyofpython/citizen-grievance-classifier-muril")
text = "Hamare mohalle mein 5 din se paani nahi aa raha hai."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=96)
logits = model(**inputs).logits
print(model.config.id2label[logits.argmax(-1).item()])#-> Water Supply
Would love feedback from anyone doing NLP on code-mixed/low-resource languages — especially if you've found a cleaner way to handle the "corruption spans every department" labeling problem.
Top comments (1)
I really like that you included the failures instead of only the final metrics. Bug #1 is a great reminder that evaluation quality matters more than model choice. One question about the “Corruption & Bribery” class: have you considered treating it as an orthogonal attribute rather than a department? In many real complaints, corruption isn’t the destination category—it’s a property of the complaint itself. A multi-task or multi-label setup (Department + Corruption flag) might better reflect the underlying problem than forcing the model to choose a single label.