Building an NLP Pipeline That Actually Understands Offer Text
Most "extract structured data from unstructured text" tutorials stop at named entity recognition and call it a day. In practice, that's maybe 30% of the problem. The real work is building a pipeline that can reliably pull out conditional information — things like "eligible only for orders above ₹999" or "valid till stocks last" — from messy, inconsistent text scraped off the web.
Here's how I approached this while building an NLP pipeline to extract eligibility conditions and offer insights from scraped promotional text.
The problem with naive NER
Out-of-the-box spaCy models are great at tagging ORG, MONEY, DATE, and PERCENT entities. But offer text isn't clean prose — it's fragments, abbreviations, and domain-specific phrasing:
"Flat 20% off* | Min order ₹499 | T&C apply | Valid on select cards only"
A generic model will happily tag 20% and ₹499 as entities, but it has no idea that one is a discount and the other is a threshold condition, or that "select cards only" is a restriction that changes the entire meaning of the offer.
Combining rule-based parsing with entity extraction
The fix isn't to throw more data at a bigger model — it's to combine spaCy's statistical NER with rule-based matching for domain-specific patterns:
pythonimport spacy
from spacy.matcher import Matcher
nlp = spacy.load("en_core_web_sm")
matcher = Matcher(nlp.vocab)
Pattern: "min order" / "minimum order" followed by a currency amount
min_order_pattern = [
{"LOWER": {"IN": ["min", "minimum"]}},
{"LOWER": "order", "OP": "?"},
{"IS_CURRENCY": True, "OP": "?"},
{"LIKE_NUM": True}
]
matcher.add("MIN_ORDER", [min_order_pattern])
def extract_conditions(text):
doc = nlp(text)
matches = matcher(doc)
conditions = []
for match_id, start, end in matches:
span = doc[start:end]
conditions.append(span.text)
return conditions
This gets you a huge accuracy boost on structured sub-phrases without needing a labeled dataset or fine-tuning a transformer.
Where entity extraction still earns its keep
Rule-based matching handles known patterns well, but you still need statistical NER for the long tail — brand names, card issuer names, dates written in inconsistent formats. The trick is layering them:
Rule-based matcher — catches conditions, thresholds, restrictions (high precision, domain-specific)
Statistical NER — catches entities you can't enumerate in advance (brands, dates, amounts)
Post-processing merge — resolves overlaps and attaches conditions to the entities they modify
pythondef parse_offer(text):
doc = nlp(text)
entities = [(ent.text, ent.label_) for ent in doc.ents]
conditions = extract_conditions(text)
return {
"entities": entities,
"conditions": conditions,
"raw": text
}
Lessons that don't show up in the docs
A few things that saved a lot of debugging time:
Normalize before you tag. Scraped text has inconsistent casing, stray HTML entities, and unicode symbols like ₹ that trip up tokenizers if you don't clean them first.
Version your matcher patterns. As you scrape more sources, you'll keep discovering new phrasings. Treat your rule set like code — test it, don't just append to it.
Precision over recall for conditions. A false positive (extracting a wrong condition) is worse than a missed one, since downstream systems often act on these conditions automatically.
Why this matters beyond offers
The same layered approach — statistical NER for open-ended entities, rule-based matching for domain patterns, and a merge step to reconcile them — generalizes to almost any "extract structured facts from semi-structured text" problem: parsing job postings, extracting clauses from contracts, or pulling structured fields out of support tickets.
If you're building something similar, I'd genuinely recommend starting with the rule-based layer before reaching for a fine-tuned model. It's faster to iterate on, easier to debug, and often gets you 80% of the way there for a fraction of the engineering cost.
I write about backend systems, NLP, and applied ML. Follow along for more.
Top comments (0)