"Rules to Learning: Why NLP needed Transformers ?"

RISHIKA DHAR — Sun, 02 Aug 2026 22:44:18 +0000

For all the newbies, let me break this down step by step.

What is NLP?

NLP stands for Natural Language Processing. It is a field of AI focused on enabling computers to understand, interpret, and generate human language across many tasks such as translation, summarization, sentiment analysis, question-answering, and classification, to name a few.

Let us first touch upon a fundamental concept in NLP systems: the shift from Rules to Learning.

Now, what exactly is Rules and Learning, and how is it related to the evolution from rules-based NLP to transformers? Let's deep dive.

Why couldn't traditional NLP solve language well?

Natural language is high-dimensional, ambiguous, and context-dependent. The same token sequence can carry different meanings depending on surrounding context.

For e.g.

"I sat by the river bank." → means the edge of a river
"I deposited money at the bank." → means a financial institution

Formal grammars and fixed rule sets cannot capture this, because ambiguity resolution requires world knowledge and statistical regularities that no finite rule set can enumerate.

This is the wall that rule-based and purely statistical (n-gram) approaches ran into. Later architectures took different routes toward the same underlying problem, each with its own tradeoffs, as covered below.

What were Rule-based systems?

Early NLP systems (1950s–1980s) relied on hand-engineered rules:

Pattern-matching heuristics such as regex (e.g. [A-Z][a-z]*)
Symbolic grammars — a more sophisticated version of flat pattern matching — which took sentence structure into account (e.g., noun and verb phrases), and were written mostly by linguists and engineers

Language had infinite variations, and writing rules by hand couldn't keep up. Every new variation added to the complexity of handling edge cases and bugs. Moreover, systems could only handle what was explicitly specified as a rule and nothing more.

What were Statistical models?

The 1990s shift replaced hand-written rules with corpus statistics. Instead of encoding grammar explicitly, models estimated the probability of a word given its preceding context, learned from frequency counts in large text corpora. This removed the need for manual rule authorship, and the model generalized from data rather than from linguistic intuition.

An n-gram is a classic example of an early-age statistical model. It is just "a sequence of n words in a row." The core idea is to predict the nth word by looking at the last n-1 words and asking, "in all the text I've seen, what word most commonly follows this sequence?"

Say your training text contains lots of sentences like:

"the cat sat on the mat"
"the cat sat on the chair"
"the dog sat on the floor"

If you're building a trigram model (n=3, meaning you look at 2 previous words to predict the 3rd), and someone types "the cat," the model looks at every time "the cat" appeared in its training data and checks what came next. It counts: "sat" appeared after "the cat" 2 times out of 2. So it predicts "sat" with high confidence. (bigram = 2 words, trigram = 3 words, etc.)

The limitation is the fixed context window. An n-gram model conditioned on the previous n-1 words has no mechanism to incorporate information from earlier in the sequence. There is no representation of document-level semantics — only local co-occurrence statistics.

Why were Deep Learning models needed?

Deep Learning models extend n-gram models by working on a larger context and understanding document-level semantics. They learn richer representations (token* meaning as vectors) and try to carry context further than a fixed window.

Two separate problems needed solving:

Representing token* meaning as something richer than a frequency count
Maintaining context over arbitrary-length sequences rather than a fixed window

Distributed Representations, or Embeddings,** addressed the first problem: tokens* are mapped to dense vectors in a continuous space, where geometric proximity correlates with semantic or syntactic similarity.

Recurrent Architectures (RNNs, and later LSTMs) addressed the second: a hidden state vector is updated at each timestep, in principle carrying forward information from all prior words rather than a fixed window.

RNN or Recurrent Neural Network

An RNN processes a sentence one word at a time — each word is a "timestep" (timestep 1 = first word, timestep 2 = second word, and so on). At each timestep, it does two things:

Looks at the current word
Combines it with a running summary of everything seen so far — this running summary is called the hidden state vector (just a list of numbers representing "what this sentence is about so far")

It then updates that hidden state and passes it forward to the next timestep.

Concrete walkthrough, for "the cat sat":

Timestep 1: read "the" → update hidden state to reflect "the"
Timestep 2: read "cat" → combine with the hidden state from step 1 → update hidden state to reflect "the cat"
Timestep 3: read "sat" → combine with the hidden state from step 2 → update hidden state to reflect "the cat sat"

By the end, the hidden state is supposed to carry a summary of the whole sentence — not just a fixed window like n-grams.

The problem with RNNs

The hidden state is a fixed size (chosen before model training). So every update has to compress the old information to make room for the new word. Over many timesteps, early information gets diluted and loses detail. This is often called the "Vanishing Gradient Problem."

LSTM, or Long Short-Term Memory

An LSTM is an upgraded RNN designed specifically to fight the vanishing gradient problem. It adds small learned components called gates at each timestep, which decide:

What to forget from the old hidden state (irrelevant old info)
What to add from the current word (new relevant info)
What to output to the next step

Instead of blindly compressing everything, it learns to actively decide "this detail matters, keep it," or "this detail doesn't matter anymore, drop it."

So in essence, RNNs pass a running summary forward, word by word, but it fades over long sentences. LSTMs do the same thing, but with learned "keep/forget" gates that help the summary last longer. The vanishing gradient problem is only partially mitigated, though, not solved — very long sequences can still lose information.

Problems with RNNs and LSTMs

Vanishing gradient problem, partially mitigated but not solved by LSTM gating mechanisms.
Computationally expensive to scale, since they cannot be parallelized during training — each timestep's computation depends on the previous one's output.

Why were Transformers revolutionary?

Transformers sit at the far end of the spectrum. There is no hand-coded rule anywhere in a transformer for what a word means or how far back to look for context — the model learns all of that directly from data, including which words in a sentence should attend to which other words. That's what self-attention is learning.

Self-attention learns which words in a sentence are relevant to which other words, entirely from patterns in training data.

The 2017 paper "Attention Is All You Need" introduced an architecture that discards sequential recurrence entirely. Instead of propagating a compressed hidden state token-by-token from the first word, self-attention computes, for every token, a weighted representation over all other tokens in the sequence directly, in a single operation.

This resolves both prior bottlenecks simultaneously:

Accuracy: self-attention computes a relevance score against every other token in the sentence. This means any two tokens in a sequence have a direct computational path to each other, regardless of distance, eliminating the degradation inherent to sequential hidden-state propagation.
Training efficiency: because attention over a sequence doesn't require strict sequential processing, the computation parallelizes efficiently across modern accelerator hardware (GPUs/TPUs), which is a major reason model and dataset scale could increase by orders of magnitude.

The combination — a mechanism that models context more effectively and trains far more efficiently — is what made transformers a discontinuity rather than an incremental improvement, and what made today's model scale computationally feasible.

Where this leaves us

Rule-based systems failed on coverage. Statistical models solved coverage but failed on context window size. Recurrent deep learning solved context propagation in principle but failed on long-range retention and training parallelism. Transformers solve both. Each architecture's core limitation is precisely what the next one was designed to address.

*Token: the basic unit of text that an NLP model processes.
**Embedding: a vector (list of numbers) representing a token's meaning.

DEV Community: RISHIKA DHAR