BERT vs BERT+BiLSTM: I Expected BiLSTM to Beat BERT on Hinglish Toxicity. It Didn't

Unmona Das — Sun, 05 Jul 2026 18:23:48 +0000

More than 600 million people speak Hindi, and a huge share of them are online, posting the way people actually type when they're not writing for a textbook: mixing Hindi and English mid-sentence, switching scripts, dropping in slang, letting word order drift depending on mood. Linguists call this code-mixing. Most content moderation systems just call it noise they weren't built to handle.

That gap matters. A model trained mostly on English inherits a blind spot the moment it meets Hinglish, and when a moderation system misses real abuse — or wrongly flags an innocent post because it couldn't parse the sentence — someone on the other end absorbs that failure.

I wanted to test something concrete: does stacking a BiLSTM on top of BERT's embeddings actually improve toxicity detection on Hinglish text, the way the technique is supposed to? I went in expecting a clean win for the hybrid. What I got was more useful than that.

The setup

Libraries: torch, transformers, scikit-learn, tqdm
Backbone: bert-base-multilingual-cased (mBERT)
Hardware: CPU — no GPU needed, though budget 15-20 minutes per run

The dataset is a 500-row sample, 250 toxic and 250 non-toxic, pulled from a larger 60,000-post Hindi social media corpus. Worth being upfront here: the full corpus had real label noise. A chunk of posts marked "toxic" turned out to be ordinary news or opinion text, nothing abusive about them at all. Rather than train on that, I filtered for clearly-labeled examples in each class and checked them by hand. That makes this a clean teaching example, not a benchmark on real-world noise — a distinction worth keeping in mind for everything below.

udgithubit / hinglish-toxicity-bert-bilstm

Why BERT alone misses toxic Hinglish (code-mixed Hindi-English) comments, and how adding a BiLSTM head fixes it — with a runnable Colab notebook.

Hinglish Toxicity Detection: BERT + BiLSTM

Why a plain BERT classifier and a BERT+BiLSTM hybrid perform the way they do on code-mixed Hindi-English (Hinglish) toxic content — and what that reveals about when architectural complexity actually helps.

The problem

Most toxicity classifiers are trained and tuned on English. Hindi social media text is frequently code-mixed (Hinglish), has free word order, and mixes scripts (Devanagari + Latin). This repo explores whether adding a bidirectional LSTM head on top of BERT's contextual embeddings improves detection — and reports an honest result, not just a clean win.

What's in this repo

bert_baseline.py — plain BERT classifier (pooled output → linear layer)
bert_bilstm.py — two hybrid variants
- BERT → BiLSTM: full token-level BERT embeddings fed through a BiLSTM
- BiLSTM → BERT (frozen): BERT frozen, only the single pooled vector fed through a BiLSTM
data/hinglish_toxicity_sample_500.csv — 500-row balanced sample (250 toxic / 250…

View on GitHub

Three architectures, one question

Plain BERT. mBERT's pooled sentence representation goes straight into a linear classifier. No sequence modeling beyond what BERT already does internally.

class BertClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.classifier = nn.Linear(768, 1)

    def forward(self, input_ids, attention_mask):
        pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        return torch.sigmoid(self.classifier(pooled)).squeeze()

BERT → BiLSTM. Instead of collapsing straight to one pooled vector, this feeds BERT's full token-level output — one embedding per token — into a bidirectional LSTM, which then handles the final classification.

class BertBiLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        bert_out = self.bert(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        lstm_out, _ = self.bilstm(bert_out)
        output = lstm_out[:, -1]
        return torch.sigmoid(self.classifier(output)).squeeze()

The idea: Hindi's free word order means the word that tips a sentence from neutral into abusive can land near the start or the end. A bidirectional model should, in theory, track that better than a single squashed vector.

BiLSTM → BERT, frozen. BERT stays frozen entirely, and only its single pooled output — not the token sequence — gets passed through a BiLSTM.

class BiLSTMBert(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        for param in self.bert.parameters():
            param.requires_grad = False
        self.bilstm = nn.LSTM(768, 128, num_layers=2, bidirectional=True, batch_first=True)
        self.classifier = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask):
        with torch.no_grad():
            pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        lstm_in = pooled.unsqueeze(1)
        lstm_out, _ = self.bilstm(lstm_in)
        return torch.sigmoid(self.classifier(lstm_out[:, -1])).squeeze()

I threw this one in on purpose. It's almost guaranteed to underperform, and watching exactly how it fails tells you something.

What actually happened

Model	Accuracy	Precision	Recall	F1
Plain BERT (baseline)	0.96	0.98	0.94	0.96
BERT → BiLSTM	0.95	1.00	0.88	0.94
BiLSTM → BERT (frozen)	0.65	0.56	0.93	0.70

I expected the middle row to win clearly over the top one. It didn't — plain BERT matched it, arguably edged it out. The frozen variant collapsed, exactly as expected.

Why the "obvious" improvement never showed up

This is the part most tutorials skip past.

The BiLSTM's real strength is tracking sequential context — how meaning shifts as a sentence unfolds — and that matters most when the text is ambiguous. My 500-row sample was filtered specifically to remove ambiguity: clear-cut toxic examples, clear-cut clean ones, nothing sitting in between. On text that unambiguous, mBERT's own attention mechanism is often already enough. There's no subtle sequential signal left for a BiLSTM to squeeze extra value out of.

My guess is the BiLSTM's advantage is real, just not visible here — it probably shows up on the harder, messier data: sarcasm, an insult buried in an otherwise casual sentence, a toxic-sounding word used in a way that isn't actually abusive. That's precisely the kind of ambiguity a small, clean sample is built to exclude.

The frozen variant's collapse is easier to explain. Freeze BERT, pass in only one already-pooled vector, and you've removed the sequence entirely — there's nothing left for a bidirectional LSTM to sweep forward and backward across. The architecture's whole premise depends on having a sequence of positions to work with. Take that away and weak results aren't surprising; they're confirmation the setup didn't hold together in the first place.

Key takeaway

Complexity doesn't pay off automatically — it earns its keep on the data where a simpler model's blind spots start to show. On a clean benchmark, plain BERT might genuinely be enough. On messy, real-world, code-mixed social media — the actual moderation problem — I'd expect the picture to look different, and that's the harder experiment worth running next.

If you're building something similar, don't take "always add a BiLSTM" or "never bother" from this. Test on data that resembles your hardest real cases, not your cleanest ones. A model that looks identical to a simpler one on easy examples can pull ahead — or fall apart — once you feed it the sarcastic, code-mixed, ambiguous content that made the problem hard to begin with.

This is a small demonstration, not a production system. But if you're working on moderation for another low-resource, code-mixed language — Bengali, Tamil, Assamese, and plenty of others face the same gap — this three-way comparison is a fast way to find where the real signal actually sits, instead of assuming it.

Do you think the BiLSTM would pull ahead if I introduced more sarcastic or code-mixed examples? I'd love to hear where the "complexity ceiling" usually sits in your own NLP work.

Explore the full code & dataset on GitHub

DEV Community: Unmona Das