A keyword search for "self-healing battery electrode" misses the patent that describes an "autonomously repairing anode coating." Same invention, zero shared keywords. That single gap is why prior-art search is a semantic problem, not a string-matching one, and why so many invalidation searches quietly leak relevant references.
Here is the payoff in eight lines of Python:
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
query = "self-healing battery electrode"
candidate = "autonomously repairing anode coating for lithium cells"
emb = model.encode([query, candidate], convert_to_tensor=True)
score = util.cos_sim(emb[0], emb[1]).item()
print(f"cosine similarity: {score:.3f}")
# -> cosine similarity: 0.571
Keyword overlap: zero. Semantic similarity: 0.57, well above the noise floor. By the end of this article you will have a working mental model (the PRIOR Stack), a runnable retriever, and a reproducible rubric for scoring any patent-search tool, including Derwent Innovation.
Disclosure: I work on PatentScan, a semantic patent-search tool. This article is a technical walkthrough with one comparison section. All code and methods are vendor-neutral and reproducible.
Why Patent Prior-Art Search Is a Semantic Problem
Semantic patent search uses vector embeddings to match inventions by meaning rather than shared keywords. Keyword retrieval fails here because inventors (and their attorneys) deliberately vary terminology. Synonymy, paraphrase, and the lexical gap between claim language and plain description all conspire to hide relevant prior art from a Boolean query.
The cost is measurable. Patent-retrieval benchmarks such as the CLEF-IP patent retrieval benchmark have shown for years that lexical-only systems leave substantial recall on the table, precisely the references that sink a freedom-to-operate (FTO) opinion or lose an invalidation fight. Recall, not precision, is the metric that keeps patent examiners and litigators awake, because a single missed reference can void a claim.
Semantic retrieval closes that gap by representing meaning as geometry. If two inventions sit close in vector space, they are candidates regardless of vocabulary.
The PRIOR Stack: A Mental Model for Semantic Patent Retrieval
Every real semantic prior-art system, hosted or managed, reduces to five layers:
- Parse: normalize patent documents (strip boilerplate, segment claims, abstracts, CPC codes).
- Represent: convert text into embeddings with a transformer encoder.
- Index: store vectors in an approximate nearest neighbor (ANN) structure for fast lookup.
- Order: combine hybrid retrieval with re-ranking to sharpen top results.
- Review: run human-in-the-loop evaluation to validate output.
The data shape flows cleanly: raw text becomes a 384- or 768-dimensional vector, enters an index, and leaves as a ranked list. We will build one layer per section, then reuse PRIOR as the scoring axis for the vendor comparison.
Building a Minimal Prior-Art Retriever (Hands-On)
Here is the whole pipeline end to end. Run it in a notebook.
from sentence_transformers import SentenceTransformer
import faiss, numpy as np
# Parse layer: a tiny abstract corpus
corpus = [
"A lithium cell with an autonomously repairing anode coating.",
"A wind turbine blade with embedded strain sensors.",
"A polymer electrolyte that self-heals micro-cracks during cycling.",
"A method for distributed training of neural networks across edge devices.",
]
# Represent layer: embeddings
model = SentenceTransformer("all-MiniLM-L6-v2")
vecs = model.encode(corpus, normalize_embeddings=True)
# Index layer: FAISS inner-product = cosine on normalized vectors
index = faiss.IndexFlatIP(vecs.shape[1])
index.add(np.asarray(vecs, dtype="float32"))
# Query
q = model.encode(["self-healing battery electrode"], normalize_embeddings=True)
scores, ids = index.search(np.asarray(q, dtype="float32"), k=3)
for rank, (i, s) in enumerate(zip(ids[0], scores[0]), 1):
print(f"{rank}. {s:.3f} {corpus[i]}")
Expected output:
| Rank | Score | Matched abstract |
|---|---|---|
| 1 | 0.571 | autonomously repairing anode coating |
| 2 | 0.461 | self-heals micro-cracks during cycling |
| 3 | 0.088 | wind turbine blade with strain sensors |
The two self-healing inventions surface at the top despite no literal keyword match, while the turbine and ML patents fall away. That is the Parse → Represent → Index → Order loop in miniature, using sentence-transformers and FAISS.
From Prototype to Product: The Patent-Search API Layer
The prototype works on four abstracts. Production means tens of millions of patent families, which is exactly where the build-vs-buy decision bites: you do not want to index 100M+ vectors yourself unless retrieval is your core product.
A managed patent-search API collapses the whole PRIOR stack into one call:
curl https://api.example.com/v1/search \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"query": "self-healing battery electrode", "top_k": 5}'
{
"results": [
{"publication": "US-XXXXXXX-A1", "score": 0.57, "title": "Autonomously repairing anode coating"}
],
"page": 1,
"latency_ms": 180
}
Expect bearer-token auth, per-key rate limits, cursor pagination, and a documented response schema. For teams who would rather call an endpoint than operate an ANN cluster, PatentScan's patent-search API exposes concept-based retrieval over a large patent corpus without self-hosting vectors.
Build vs. buy, at a glance:
| Factor | Self-host (FAISS/HNSW) | Managed API |
|---|---|---|
| Upfront effort | High (ingest, embed, index) | Minimal |
| Corpus maintenance | You own it | Vendor owns it |
| Latency control | Full | Vendor-bound |
| Fit | Retrieval is your product | Retrieval is a feature |
Evaluating Any Patent-Search Tool: A Reproducible Benchmark Method
Never trust a vendor adjective. Build a gold evaluation set of known prior-art pairs (cited references from granted patents are free ground truth), then score every candidate on the same metrics.
| Metric | Definition | Why it matters |
|---|---|---|
| Recall@k | Fraction of true prior art found in top k | Missed references lose cases |
| MRR | Mean reciprocal rank of first hit | Measures how high relevant results land |
| nDCG | Rank-weighted relevance | Rewards good ordering, not just presence |
| Latency p95 | 95th-percentile response time | Interactive usability |
Then score each tool per PRIOR layer (0–2) instead of using vague stars:
| Tool | Parse | Represent | Index | Order | Review | Notes |
|---|---|---|---|---|---|---|
| Derwent Innovation | 2 | 2 | 2 | 2 | 2 | DWPI expert indexing; transformer AI Search (launched Dec 2024). Public pricing not disclosed (as of Q1 2026). |
| Orbit Intelligence | 2 | 1 | 2 | 1 | 2 | Strong coverage; pricing not disclosed |
| PatSeer | 1 | 1 | 2 | 1 | 1 | Semantic clustering; pricing not disclosed |
| PatSnap | 1 | 1 | 1 | 1 | 1 | Analytics-first; pricing not disclosed |
| IPRally | 1 | 2 | 2 | 2 | 1 | Graph/AI-first retrieval |
| PatentScan | 1 | 2 | 2 | 2 | 1 | Concept-based API |
Scores are a rubric template: re-run them against your own gold set before trusting any row.
Build vs. Buy: A Decision Tree for IP-Tech Teams
No hedging. Pick the branch that matches your constraint:
- Prototyping or under ~1M documents? Build. Use open corpora like Lens.org, WIPO PATENTSCOPE, or Google Patents Public Datasets (BigQuery) plus FAISS.
- Need audited coverage across 100+ jurisdictions for litigation? Buy enterprise (Derwent-class). The curation and legal-status data justify the license.
- Latency or unit cost is the hard constraint? Call a managed API rather than operating your own ANN fleet.
- Retrieval quality is your differentiator? Build, and invest in the Represent and Order layers.
The 10-question checklist: corpus size, jurisdiction coverage, update cadence, Recall@k on your gold set, latency p95, re-ranking quality, API availability, compliance/audit trail, total cost of ownership, and in-house ML capacity.
Key Takeaways & Next Steps
- Keyword search fails on prior art because inventions are paraphrased, not repeated.
- The PRIOR Stack (Parse, Represent, Index, Order, Review) organizes every semantic system.
- A working retriever is 20 lines with sentence-transformers and FAISS.
- Evaluate tools with Recall@k, MRR, and nDCG on your own gold set, never adjectives.
- Build under a million docs; buy for jurisdiction-audited litigation coverage.
FAQ
Why do embeddings beat keywords for prior-art search?
Embeddings match by meaning, so "autonomously repairing coating" and "self-healing electrode" land close in vector space despite zero shared words, closing the recall gap keyword search leaves open (CLEF-IP).
Which embedding model is best for patents, general or domain-tuned?
Start with a general model like all-MiniLM-L6-v2 from sentence-transformers for prototyping; domain-tuned encoders help recall on dense technical claims once you have a gold set to measure gains.
FAISS vs. a managed vector database, which should I pick?
Use FAISS when you control infrastructure and corpus; choose a managed vector DB when you want operations, scaling, and persistence handled for you.
How do I measure if my patent search is actually good?
Build a gold set from granted-patent citations, then track Recall@k, MRR, and nDCG against it. Recall@k matters most for invalidation work.
When is Derwent Innovation or an enterprise license worth it over building?
When you need audited coverage across 100+ jurisdictions, legal-status data, and litigation support (Derwent Innovation). Below that, building is cheaper and more flexible.
Can I call a patent-search API instead of indexing 100M+ vectors myself?
Yes. A managed patent-search API returns ranked results over a large corpus via one authenticated request, no self-hosted ANN index required.
References & External Sources
- sentence-transformers Documentation - Reference implementation for generating the embeddings used in the Represent layer.
- FAISS (Facebook AI Similarity Search) - Open-source ANN library powering the Index layer in the hands-on code.
- HNSW ANN Paper (Malkov & Yashunin) - Validates the graph-based approximate nearest neighbor method behind production vector search.
- CLEF-IP Patent Retrieval Benchmark - Established evaluation framework documenting the recall gap in lexical patent retrieval.
- Google Patents Public Datasets - Open corpus for building and testing self-hosted patent retrieval.
Experience modern patent search yourself. Paste any invention or concept description into PatentScan and see what advanced concept-based discovery finds in seconds.
Top comments (0)