DEV Community: Unmona Das

How to Fine-Tune mBERT for Hinglish Toxicity Classification: A Step-by-Step Guide

Unmona Das — Tue, 07 Jul 2026 17:14:42 +0000

If you've tried running an off-the-shelf English toxicity classifier on Hindi social media text, you've probably watched it fall apart on sentences like "yaar tum kitne pagal ho" — not because the model is bad, but because it was never shown Hinglish (code-mixed Hindi-English) during training. Word order shifts, scripts mix mid-sentence, and English tokenizers choke on Devanagari.

This tutorial walks through fine-tuning bert-base-multilingual-cased (mBERT) as a toxicity classifier on Hinglish text, end to end: loading data, tokenizing, building the model, training, and evaluating. No GPU required — everything here runs on CPU in about 15-20 minutes.

What you'll need

Python 3.8+
torch, transformers, scikit-learn, tqdm
A CSV of labeled Hinglish text (toxic / non-toxic). We'll use a 500-row balanced sample — 250 toxic, 250 non-toxic — pulled and hand-checked from a larger Hindi social media corpus.

Install the dependencies:

pip install torch transformers scikit-learn tqdm

Step 1: Load and inspect the data

Start by loading your CSV and confirming the class balance. Skipping this step is how people end up training a classifier that just learned to predict the majority class.

import pandas as pd

df = pd.read_csv('hinglish_toxicity_sample_500.csv')
print(df['label'].value_counts())
print(df.sample(5))

A quick note from experience: if you're building your own dataset from a larger scraped corpus, don't trust the labels blindly. A meaningful chunk of "toxic"-labeled posts in raw social media corpora turn out to be ordinary news or opinion text with nothing abusive in them. Filter for clearly-labeled examples and spot-check by hand before you train on anything.

Step 2: Split and tokenize

mBERT ships with its own tokenizer, trained across 104 languages including Hindi. That's what makes it a reasonable starting point for Hinglish — it already has subword coverage for Devanagari and Latin script.

from sklearn.model_selection import train_test_split
from transformers import BertTokenizer

train_texts, val_texts, train_labels, val_labels = train_test_split(
    df['text'].tolist(), df['label'].tolist(), test_size=0.2, random_state=42
)

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')

def tokenize(texts):
    return tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors='pt')

train_encodings = tokenize(train_texts)
val_encodings = tokenize(val_texts)

max_length=128 is generous for social media posts — most Hinglish comments are short. Bump it up only if your data includes longer paragraphs.

Step 3: Wrap the data in a Dataset

import torch

class ToxicityDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx], dtype=torch.float)
        return item

    def __len__(self):
        return len(self.labels)

train_dataset = ToxicityDataset(train_encodings, train_labels)
val_dataset = ToxicityDataset(val_encodings, val_labels)

Step 4: Build the classifier

This is the simplest version: mBERT's pooled sentence output goes straight into a linear layer. No sequence modeling beyond what's already happening inside BERT's attention layers.

import torch.nn as nn
from transformers import BertModel

class BertClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.classifier = nn.Linear(768, 1)

    def forward(self, input_ids, attention_mask):
        pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        return torch.sigmoid(self.classifier(pooled)).squeeze()

model = BertClassifier()

Worth asking before you reach for anything fancier: does this simple version already work? In my own runs, plain BERT hit 0.96 accuracy on a clean, unambiguous sample — stacking a BiLSTM on top didn't beat it. Complexity earns its place on messy, ambiguous data (sarcasm, a slur used non-abusively, an insult buried mid-sentence), not automatically. Start simple, and only add architecture once you've confirmed the simple version is actually falling short on your data.

Step 5: Train

from torch.utils.data import DataLoader
from torch.optim import AdamW

train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True)
optimizer = AdamW(model.parameters(), lr=2e-5)
loss_fn = nn.BCELoss()

model.train()
for epoch in range(3):
    total_loss = 0
    for batch in train_loader:
        optimizer.zero_grad()
        outputs = model(batch['input_ids'], batch['attention_mask'])
        loss = loss_fn(outputs, batch['labels'])
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1} — avg loss: {total_loss / len(train_loader):.4f}")

Three epochs is usually enough on a small dataset like this. Watch the loss — if it's still dropping steadily at epoch 3, it's worth running a fourth.

Step 6: Evaluate

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

val_loader = DataLoader(val_dataset, batch_size=8)
model.eval()

preds, true = [], []
with torch.no_grad():
    for batch in val_loader:
        outputs = model(batch['input_ids'], batch['attention_mask'])
        preds.extend((outputs > 0.5).int().tolist())
        true.extend(batch['labels'].int().tolist())

print(f"Accuracy:  {accuracy_score(true, preds):.2f}")
print(f"Precision: {precision_score(true, preds):.2f}")
print(f"Recall:    {recall_score(true, preds):.2f}")
print(f"F1:        {f1_score(true, preds):.2f}")

On the clean 500-row sample used here, this setup lands around 0.96 accuracy and 0.96 F1 — strong enough that it's a reasonable baseline before you try anything more elaborate.

Where to go from here

The obvious next step is stacking a bidirectional LSTM on top of BERT's token-level output, on the theory that Hindi's freer word order needs sequence modeling that a single pooled vector can't capture. I tried exactly that in a follow-up experiment, and the result wasn't what I expected — worth a read if you're deciding whether the extra complexity is worth it for your own dataset.

The code and dataset for this tutorial (plus the BiLSTM comparison) are on GitHub: https://github.com/udgithubit/hinglish-toxicity-bert-bilstm

BERT vs BERT+BiLSTM: I Expected BiLSTM to Beat BERT on Hinglish Toxicity. It Didn't

Unmona Das — Sun, 05 Jul 2026 18:23:48 +0000

More than 600 million people speak Hindi, and a huge share of them are online, posting the way people actually type when they're not writing for a textbook: mixing Hindi and English mid-sentence, switching scripts, dropping in slang, letting word order drift depending on mood. Linguists call this code-mixing. Most content moderation systems just call it noise they weren't built to handle.

That gap matters. A model trained mostly on English inherits a blind spot the moment it meets Hinglish, and when a moderation system misses real abuse — or wrongly flags an innocent post because it couldn't parse the sentence — someone on the other end absorbs that failure.

I wanted to test something concrete: does stacking a BiLSTM on top of BERT's embeddings actually improve toxicity detection on Hinglish text, the way the technique is supposed to? I went in expecting a clean win for the hybrid. What I got was more useful than that.

The setup

Libraries: torch, transformers, scikit-learn, tqdm
Backbone: bert-base-multilingual-cased (mBERT)
Hardware: CPU — no GPU needed, though budget 15-20 minutes per run

The dataset is a 500-row sample, 250 toxic and 250 non-toxic, pulled from a larger 60,000-post Hindi social media corpus. Worth being upfront here: the full corpus had real label noise. A chunk of posts marked "toxic" turned out to be ordinary news or opinion text, nothing abusive about them at all. Rather than train on that, I filtered for clearly-labeled examples in each class and checked them by hand. That makes this a clean teaching example, not a benchmark on real-world noise — a distinction worth keeping in mind for everything below.

udgithubit / hinglish-toxicity-bert-bilstm

Why BERT alone misses toxic Hinglish (code-mixed Hindi-English) comments, and how adding a BiLSTM head fixes it — with a runnable Colab notebook.

Hinglish Toxicity Detection: BERT + BiLSTM

Why a plain BERT classifier and a BERT+BiLSTM hybrid perform the way they do on code-mixed Hindi-English (Hinglish) toxic content — and what that reveals about when architectural complexity actually helps.

The problem

Most toxicity classifiers are trained and tuned on English. Hindi social media text is frequently code-mixed (Hinglish), has free word order, and mixes scripts (Devanagari + Latin). This repo explores whether adding a bidirectional LSTM head on top of BERT's contextual embeddings improves detection — and reports an honest result, not just a clean win.

What's in this repo

bert_baseline.py — plain BERT classifier (pooled output → linear layer)
bert_bilstm.py — two hybrid variants
- BERT → BiLSTM: full token-level BERT embeddings fed through a BiLSTM
- BiLSTM → BERT (frozen): BERT frozen, only the single pooled vector fed through a BiLSTM
data/hinglish_toxicity_sample_500.csv — 500-row balanced sample (250 toxic / 250…

View on GitHub

Three architectures, one question

Plain BERT. mBERT's pooled sentence representation goes straight into a linear classifier. No sequence modeling beyond what BERT already does internally.

class BertClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.classifier = nn.Linear(768, 1)

    def forward(self, input_ids, attention_mask):
        pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        return torch.sigmoid(self.classifier(pooled)).squeeze()

BERT → BiLSTM. Instead of collapsing straight to one pooled vector, this feeds BERT's full token-level output — one embedding per token — into a bidirectional LSTM, which then handles the final classification.

class BertBiLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        bert_out = self.bert(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        lstm_out, _ = self.bilstm(bert_out)
        output = lstm_out[:, -1]
        return torch.sigmoid(self.classifier(output)).squeeze()

The idea: Hindi's free word order means the word that tips a sentence from neutral into abusive can land near the start or the end. A bidirectional model should, in theory, track that better than a single squashed vector.

BiLSTM → BERT, frozen. BERT stays frozen entirely, and only its single pooled output — not the token sequence — gets passed through a BiLSTM.

class BiLSTMBert(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        for param in self.bert.parameters():
            param.requires_grad = False
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        with torch.no_grad():
            pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        lstm_in = pooled.unsqueeze(1)
        lstm_out, _ = self.bilstm(lstm_in)
        return torch.sigmoid(self.classifier(lstm_out[:, -1])).squeeze()

I threw this one in on purpose. It's almost guaranteed to underperform, and watching exactly how it fails tells you something.

What actually happened

Model	Accuracy	Precision	Recall	F1
Plain BERT (baseline)	0.96	0.98	0.94	0.96
BERT → BiLSTM	0.95	1.00	0.88	0.94
BiLSTM → BERT (frozen)	0.65	0.56	0.93	0.70

I expected the middle row to win clearly over the top one. It didn't — plain BERT matched it, arguably edged it out. The frozen variant collapsed, exactly as expected.

Why the "obvious" improvement never showed up

This is the part most tutorials skip past.

The BiLSTM's real strength is tracking sequential context — how meaning shifts as a sentence unfolds — and that matters most when the text is ambiguous. My 500-row sample was filtered specifically to remove ambiguity: clear-cut toxic examples, clear-cut clean ones, nothing sitting in between. On text that unambiguous, mBERT's own attention mechanism is often already enough. There's no subtle sequential signal left for a BiLSTM to squeeze extra value out of.

My guess is the BiLSTM's advantage is real, just not visible here — it probably shows up on the harder, messier data: sarcasm, an insult buried in an otherwise casual sentence, a toxic-sounding word used in a way that isn't actually abusive. That's precisely the kind of ambiguity a small, clean sample is built to exclude.

The frozen variant's collapse is easier to explain. Freeze BERT, pass in only one already-pooled vector, and you've removed the sequence entirely — there's nothing left for a bidirectional LSTM to sweep forward and backward across. The architecture's whole premise depends on having a sequence of positions to work with. Take that away and weak results aren't surprising; they're confirmation the setup didn't hold together in the first place.

Key takeaway

Complexity doesn't pay off automatically — it earns its keep on the data where a simpler model's blind spots start to show. On a clean benchmark, plain BERT might genuinely be enough. On messy, real-world, code-mixed social media — the actual moderation problem — I'd expect the picture to look different, and that's the harder experiment worth running next.

If you're building something similar, don't take "always add a BiLSTM" or "never bother" from this. Test on data that resembles your hardest real cases, not your cleanest ones. A model that looks identical to a simpler one on easy examples can pull ahead — or fall apart — once you feed it the sarcastic, code-mixed, ambiguous content that made the problem hard to begin with.

This is a small demonstration, not a production system. But if you're working on moderation for another low-resource, code-mixed language — Bengali, Tamil, Assamese, and plenty of others face the same gap — this three-way comparison is a fast way to find where the real signal actually sits, instead of assuming it.

Do you think the BiLSTM would pull ahead if I introduced more sarcastic or code-mixed examples? I'd love to hear where the "complexity ceiling" usually sits in your own NLP work.

Explore the full code & dataset on GitHub