If you've tried running an off-the-shelf English toxicity classifier on Hindi social media text, you've probably watched it fall apart on sentences like "yaar tum kitne pagal ho" — not because the model is bad, but because it was never shown Hinglish (code-mixed Hindi-English) during training. Word order shifts, scripts mix mid-sentence, and English tokenizers choke on Devanagari.
This tutorial walks through fine-tuning bert-base-multilingual-cased (mBERT) as a toxicity classifier on Hinglish text, end to end: loading data, tokenizing, building the model, training, and evaluating. No GPU required — everything here runs on CPU in about 15-20 minutes.
What you'll need
- Python 3.8+
-
torch,transformers,scikit-learn,tqdm - A CSV of labeled Hinglish text (toxic / non-toxic). We'll use a 500-row balanced sample — 250 toxic, 250 non-toxic — pulled and hand-checked from a larger Hindi social media corpus.
Install the dependencies:
pip install torch transformers scikit-learn tqdm
Step 1: Load and inspect the data
Start by loading your CSV and confirming the class balance. Skipping this step is how people end up training a classifier that just learned to predict the majority class.
import pandas as pd
df = pd.read_csv('hinglish_toxicity_sample_500.csv')
print(df['label'].value_counts())
print(df.sample(5))
A quick note from experience: if you're building your own dataset from a larger scraped corpus, don't trust the labels blindly. A meaningful chunk of "toxic"-labeled posts in raw social media corpora turn out to be ordinary news or opinion text with nothing abusive in them. Filter for clearly-labeled examples and spot-check by hand before you train on anything.
Step 2: Split and tokenize
mBERT ships with its own tokenizer, trained across 104 languages including Hindi. That's what makes it a reasonable starting point for Hinglish — it already has subword coverage for Devanagari and Latin script.
from sklearn.model_selection import train_test_split
from transformers import BertTokenizer
train_texts, val_texts, train_labels, val_labels = train_test_split(
df['text'].tolist(), df['label'].tolist(), test_size=0.2, random_state=42
)
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
def tokenize(texts):
return tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors='pt')
train_encodings = tokenize(train_texts)
val_encodings = tokenize(val_texts)
max_length=128 is generous for social media posts — most Hinglish comments are short. Bump it up only if your data includes longer paragraphs.
Step 3: Wrap the data in a Dataset
import torch
class ToxicityDataset(torch.utils.data.Dataset):
def __init__(self, encodings, labels):
self.encodings = encodings
self.labels = labels
def __getitem__(self, idx):
item = {k: v[idx] for k, v in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx], dtype=torch.float)
return item
def __len__(self):
return len(self.labels)
train_dataset = ToxicityDataset(train_encodings, train_labels)
val_dataset = ToxicityDataset(val_encodings, val_labels)
Step 4: Build the classifier
This is the simplest version: mBERT's pooled sentence output goes straight into a linear layer. No sequence modeling beyond what's already happening inside BERT's attention layers.
import torch.nn as nn
from transformers import BertModel
class BertClassifier(nn.Module):
def __init__(self):
super().__init__()
self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
self.classifier = nn.Linear(768, 1)
def forward(self, input_ids, attention_mask):
pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
return torch.sigmoid(self.classifier(pooled)).squeeze()
model = BertClassifier()
Worth asking before you reach for anything fancier: does this simple version already work? In my own runs, plain BERT hit 0.96 accuracy on a clean, unambiguous sample — stacking a BiLSTM on top didn't beat it. Complexity earns its place on messy, ambiguous data (sarcasm, a slur used non-abusively, an insult buried mid-sentence), not automatically. Start simple, and only add architecture once you've confirmed the simple version is actually falling short on your data.
Step 5: Train
from torch.utils.data import DataLoader
from torch.optim import AdamW
train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True)
optimizer = AdamW(model.parameters(), lr=2e-5)
loss_fn = nn.BCELoss()
model.train()
for epoch in range(3):
total_loss = 0
for batch in train_loader:
optimizer.zero_grad()
outputs = model(batch['input_ids'], batch['attention_mask'])
loss = loss_fn(outputs, batch['labels'])
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch+1} — avg loss: {total_loss / len(train_loader):.4f}")
Three epochs is usually enough on a small dataset like this. Watch the loss — if it's still dropping steadily at epoch 3, it's worth running a fourth.
Step 6: Evaluate
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
val_loader = DataLoader(val_dataset, batch_size=8)
model.eval()
preds, true = [], []
with torch.no_grad():
for batch in val_loader:
outputs = model(batch['input_ids'], batch['attention_mask'])
preds.extend((outputs > 0.5).int().tolist())
true.extend(batch['labels'].int().tolist())
print(f"Accuracy: {accuracy_score(true, preds):.2f}")
print(f"Precision: {precision_score(true, preds):.2f}")
print(f"Recall: {recall_score(true, preds):.2f}")
print(f"F1: {f1_score(true, preds):.2f}")
On the clean 500-row sample used here, this setup lands around 0.96 accuracy and 0.96 F1 — strong enough that it's a reasonable baseline before you try anything more elaborate.
Where to go from here
The obvious next step is stacking a bidirectional LSTM on top of BERT's token-level output, on the theory that Hindi's freer word order needs sequence modeling that a single pooled vector can't capture. I tried exactly that in a follow-up experiment, and the result wasn't what I expected — worth a read if you're deciding whether the extra complexity is worth it for your own dataset.
The code and dataset for this tutorial (plus the BiLSTM comparison) are on GitHub: https://github.com/udgithubit/hinglish-toxicity-bert-bilstm
Top comments (1)
Wrote this as more of a hands-on companion to my BiLSTM comparison post — curious if anyone's run into surprises fine-tuning mBERT on other code-mixed languages (Bengali, Tamil, Swahili, etc.)? Always curious where the same 'simple baseline wins' pattern shows up elsewhere.