Reliability is the first design constraint for semantic duplicate detection in a listing feed. Short answer: keep canonical facts separate from derived search data, make every retrieval decision replayable, and test the pipeline with labeled pairs before tuning a similarity threshold. That approach works for a logistics marketplace too, where the same warehouse may arrive from several partners under slightly different wording.
The schema is part of the test harness. If an address correction, a new embedding, and a reviewer decision all overwrite one row, you cannot tell whether retrieval failed or the data changed underneath it. I want each transition to leave a small, inspectable record.
A reliability-first field guide
There are four useful schema shapes. Choose based on how much evidence your team must preserve when a duplicate decision is challenged.
| Schema shape | Pick this when | Reliability benefit | Cost to operate |
|---|---|---|---|
| One canonical row | Feed is small and identifiers are authoritative | Few moving parts | Little history for audits |
| Canonical row plus normalized text | Wording varies but fields are stable | Reproducible text input | Normalization must be versioned |
| Canonical row plus vector record | Semantic paraphrases create misses | Old and new vectors can be compared | Embedding refresh jobs |
| Evented decision record | Merges need review or reversal | Every decision is replayable | More storage and tooling |
Start with the second shape, then add the third when labeled tests show lexical recall is insufficient. Add event records before automatic merges, not after the first disputed listing. A vector index is a component; the evidence around it is the reliability feature.
How can retrieval architecture and schema make listing discovery testable?
Model the pipeline as four contracts: ingestion, normalization, candidate generation, and decision recording. In words: partner row -> canonical identity -> lexical candidates -> semantic score -> reviewer outcome. Each contract accepts a version and emits identifiers that let a test replay the same input.
For the logistics case, a listing owns business state while derived records own search state. This separation lets an engineer re-run normalization without changing availability or asking rate. It also keeps a stale vector visible instead of silently replacing it.
type ListingStatus = "active" | "paused" | "archived";
type Listing = {
id: string;
source: string;
sourceId: string;
status: ListingStatus;
street: string;
locality: string;
postalCode: string;
areaSqM: number | null;
askingRate: number | null;
currency: string | null;
updatedAt: string;
};
type ListingText = {
listingId: string;
normalized: string;
language: string;
textVersion: number;
};
type ListingVector = {
listingId: string;
textVersion: number;
modelId: string;
values: number[];
};
type DuplicateDecision = {
leftId: string;
rightId: string;
lexicalEvidence: string[];
vectorScore: number;
thresholdVersion: number;
outcome: "same" | "different" | "review";
};
textVersion is deliberately boring. When a normalizer changes, increment it and create a new vector record; retain the previous record until comparisons pass. The decision record stores the threshold version as well, so a later policy change does not rewrite history.
Before an embedding call, create fixtures that exercise the failures your feed actually produces: unit changes (sqm versus square metres), reordered addresses, abbreviations, translated descriptions, and two different leases in the same building. Label each pair as same, different, or review. A hundred carefully chosen pairs can reveal more than a huge random sample.
The test should assert invariants, not a magical score. A same-property pair must survive canonical identity checks; a different-lease pair must never auto-merge solely because the prose is similar. Keep the embedding provider behind an interface so tests can use fixed vectors.
interface Embedder {
embed(text: string): Promise<number[]>;
}
function cosine(a: number[], b: number[]): number {
let dot = 0;
let aa = 0;
let bb = 0;
for (let i = 0; i < a.length; i += 1) {
dot += a[i] * b[i];
aa += a[i] * a[i];
bb += b[i] * b[i];
}
return aa && bb ? dot / (Math.sqrt(aa) * Math.sqrt(bb)) : 0;
}
export async function scorePair(
left: string,
right: string,
embedder: Embedder,
): Promise<number> {
const [leftVector, rightVector] = await Promise.all([
embedder.embed(left),
embedder.embed(right),
]);
return cosine(leftVector, rightVector);
}
Run the fixtures after every normalization change. I once assumed that replacing a unit abbreviation was harmless; the review queue disagreed. The score moved for one warehouse pair, then candidate counts rose because a lexical token disappeared. The useful clue was the pair's textVersion, not the final search latency. A replay test would have caught that before deployment.
Keep it replayable.
Operational signals that prove the schema is behaving
Instrument transitions, not just requests. Count feed_rows_received, canonical_rows_written, lexical_candidates, vector_calls, duplicate_reviews, and decisions_replayed. Track histograms for candidate count, embedding latency, and vector age. These measurements tell you which contract changed.
A practical alert is a mismatch: vector calls rising while lexical candidates fall. That pattern means the cheap gate may be starving while semantic work expands. Another alert watches active rows whose vector textVersion is behind the text record. Give that age a deadline; a silent backlog makes yesterday's description look current.
Attach source, sourceId, listingId, textVersion, and thresholdVersion to structured traces. Do not log the full listing description because contact details can be embedded in it. Store field-level match evidence instead. It is enough for a reviewer to see that street number and postal code agreed while the lease type differed.
For local development, make the replay command deterministic. Save the fixture input, normalizer version, candidate IDs, and expected outcome. When a test fails, the engineer should be able to inspect one JSON record rather than reconstruct a production query from memory. In one feed review, that record showed three independent changes landing together: the partner had changed sqm to square metres, a postal-code tokenizer had started dropping a space, and the embedding queue was processing imports in source order. The first two changes altered candidate membership; the third delayed the replacement vector. Looking only at the end-to-end latency hid all three. Comparing the old and new textVersion records exposed the chain, and replaying the labeled pair confirmed that the intended outcome was still review, not same. That is the kind of failure a schema can make obvious or nearly impossible to diagnose.
Where this design is the wrong fit
The catch is state. Versioned text, vectors, and decisions require retention and cleanup policies. This is not suitable when a tiny dataset has authoritative source IDs and almost no semantic variation; a single lexical table is easier to explain and maintain. Stick with exact matching when a false merge is more damaging than a missed candidate and the business can tolerate manual lookup.
Semantic similarity also cannot establish legal identity. Two descriptions can refer to one building while representing separate leases. Require a trusted parcel or source agreement before automatic merging, and send ambiguous pairs to review.
Your mileage may vary across languages and property types. I am not sure one threshold will hold for short rural listings and long commercial descriptions; a region-split labeled set is what would resolve that uncertainty. Reliability comes from preserving the evidence so the threshold can change without losing the reason for an earlier decision.
Top comments (0)