Priyesh Dave

Posted on Sep 1

Day 8: Named Entity Recognition (NER) Fundamentals

#ai #nlp #genai #100daysofcode

Previously, on Day 7: Explained part-of-speech (POS) tagging, its importance in NLP, different approaches to tagging, common tagsets, and typical challenges such as ambiguity and unknown words.

What Is Named Entity Recognition (NER)?

Named Entity Recognition (NER) is a core task in Natural Language Processing (NLP). Its purpose is to find and label words or phrases in text that refer to specific "named entities." A named entity is any real-world object with a distinct name. In practice, this means people, companies, locations, dates, money amounts, product names, and more.

For example, when you read a news article, you can immediately spot terms like “Barack Obama,” “United Nations,” or “Monday, July 10.” NER lets computers do the same thing. It marks these phrases so that downstream programs know they refer to specific, meaningful things in the real world. This is foundational for many NLP tasks; knowing "who," "where," and "when" unlocks advanced automation and insight from raw text.

Common Types of Named Entities

Most NER systems look for a standard set of categories, but the specifics can change depending on the tool or domain. Here are the most common named entity types, with examples:

Person (PER): Names of individuals. Examples: “Ada Lovelace,” “Serena Williams”
Organization (ORG): Names of companies, institutions, or groups. Examples: “Google,” “United Nations,” “Harvard University”
Location (LOC): Geographic places. Examples: “Germany,” “Mount Everest,” “Nile River”
Date/Time (DATE/TIME): Specific times or dates. Examples: “July 4, 1776,” “last Friday,” “3:00 PM”
Miscellaneous (MISC): Other named things, like product names or events. Examples: “iPhone 15,” “Olympic Games”

Some systems recognize more detailed categories, such as Money (“$100”), Percentages (“55%”), or Products. The core idea is the same: identify and label text that names a unique, real-world object.

How NER Works: High-Level Process

NER is a labeling task. The basic process is:

Tokenization: The text is split into tokens. A token is usually a word or punctuation mark. (Recall tokenization from earlier lessons on preprocessing.)
Label Assignment: Each token (or sometimes a complete phrase) gets a label for its entity type—if any. For example, in the sentence “Apple launched a new product in California”:

| Token | Label |
|-----------|------------|
| Apple | ORG |
| launched | O (None) |
| a | O |
| new | O |
| product | O |
| in | O |
| California| LOC |

This format is similar to Part-of-Speech (POS) tagging, but instead of grammatical roles, the labels mark entity types.

Simple Rule-Based NER

Early NER systems were rule-based, built from patterns or lists. Two basic techniques:

Capitalization Heuristic: In English, proper nouns are capitalized. Sequences of capitalized words can signal named entities.
Dictionary (Gazetteer) Lookup: Lists of known entities, such as country or company names, are matched against the tokens.

Example rule for person names:

If a word is capitalized and appears in a list of person names, tag it as PERSON.

Strengths:

Fast to build for simple, well-defined situations.
No need for training data.

Limitations:

Misses new or unusual names easily.
Handles ambiguity poorly (e.g., “Apple” as the company vs “apple” as the fruit).
Difficult to scale for messy or diverse text.

Statistical and Machine Learning-Based NER

Modern NER uses machine learning. Instead of hand-coding rules, we train models on labeled examples. The model learns to predict, for each token, the correct entity label.

NER here is a sequence labeling problem—just like POS tagging (reviewed previously). Each word in a sentence receives a tag such as:

“B-PER” (beginning of a person name),
“I-PER” (inside a person name),
“O” (outside any entity).

Popular algorithms include:

Conditional Random Fields (CRF): Estimates the probability of a sequence of labels, considering both individual tokens and their context.
Neural Networks (LSTMs, Transformers): Can learn complex patterns from large datasets, often achieving better accuracy.

Because these models train on thousands or millions of sentences, they can catch subtleties that rules miss—like context or edge cases. They are also much more robust to new data and rare phrases.

NER in Practice: Minimal Code Example

Here’s a real NER example using spaCy, a popular Python NLP library with a built-in English model:

import spacy

# Load the English NER model. If needed: python -m spacy download en_core_web_sm
nlp = spacy.load("en_core_web_sm")

text = "Barack Obama was born in Honolulu and served as President of the United States."

doc = nlp(text)

for ent in doc.ents:
    print(ent.text, ent.label_)

What happens here:

spacy.load loads the language model with NER capability.
Passing text to nlp processes the text: tokenization, NER, and more.
Looping over doc.ents prints each found entity and its type.

Sample output:

Barack Obama PERSON
Honolulu GPE
President NORP
United States GPE

GPE (Geo-Political Entity) covers locations like countries or cities.
NORP is used for nationalities, religious, or political groups.

You can swap in any sentence to see what entities the model detects.

Applications and Challenges of NER

NER in the Real World

Information Retrieval: Tagging and highlighting names and events for search engines.
Automated Document Processing: Organizing or routing documents by named entities (like the sender or mentioned subjects).
Relationship Extraction: Finding meaningful links between people, organizations, or places.
Knowledge Graphs: Turning free text into structured facts for databases, supporting search and analytics.

Real-World Challenges

Ambiguity: Many names have multiple meanings. “Jordan” could be a person or a country; only context reveals which.
Overlapping Entities: A single phrase can represent more than one thing (for example, “New York Times” is both a place and an organization).
Language Change: New entities (people, brands, events) keep appearing, so models and entity lists must keep up.
Domain-Specific Names: Technical, medical, or legal text often contains entities not found in general-purpose models.

NER is not perfect—handling ambiguity, change, and domain-specific language remains hard. But even a basic NER pipeline brings order to messy text, extracting structure that downstream software can use. The next steps in NLP will show how NER becomes more powerful when combined with context and larger language models.

Key Takeaways

NER labels specific real-world entities like people, organizations, and dates in text.
Traditional NER uses rule-based methods such as capitalization and dictionary lookup.
Modern NER relies on machine learning models like CRF and neural networks for higher accuracy.
NER is foundational for applications like information retrieval, document processing, and knowledge graphs.
Ambiguity, new entities, and domain-specific language remain major NER challenges.

Try It Yourself

Choose a short news paragraph or sentence and process it using spaCy's NER tool or an online NER demo. Write down all entities detected and their types. Then, check the text yourself for any missed or misclassified entities and briefly reflect on possible reasons the NER tool struggled with those cases.

Further Resources

Coming up on Day 9: Text Classification with Naive Bayes and Logistic Regression

DEV Community