Building a Search Engine for 4 Languages, 2 Alphabets, and 107 Messy Color Names (Elasticsearch + pgvector + CIEDE2000)

Real production queries from my fashion marketplace in Azerbaijan:

гара йупка      # "black skirt" — Azerbaijani words, Cyrillic script
pembe yubka     # "pink skirt"  — half Turkish, half Russian
roziviy platya  # "pink dress"  — Russian translit, with a typo

My catalog is written in Azerbaijani (qara ətək, çəhrayı don). Vanilla Elasticsearch returns nothing — or confidently wrong results — for all three. Here's the full architecture that fixes it, with the actual techniques and the failures along the way. Stack: Django, Elasticsearch 8, PostgreSQL + pgvector, Gemini embeddings. Team: one founder + AI pair programming.

Prefer watching it in action first? 90-second demo of the live engine:

1. Script folding + synonyms (the floor)

Everything starts with character folding and transliteration at analysis time: ə→e, ş→s, Cyrillic→Latin (гара→qara). On top: 103 curated synonym groups mapping street language, not dictionary language — one slippers group reads tərlik, terlik, тапочки, шлёпки, slippers, səndəl (five languages/registers → one catalog concept), and the suede group carries the slang spellings users actually type (zamuj, zamıj, zamsha → zamşa).

Bonus mechanism: the synonym keys themselves are fuzzy-matched (threshold 0.78, tunable). "yupka" isn't in any group — "yubka" is; the bridge connects the misspelling to the key, the key expands to the catalog term: yupka → yubka → ətək. Two hops, zero user friction. Raise the threshold to 0.82 and the bridge silently dies — measured knob, not a constant.

One analyzer discipline that saves you months of "why is this matching": edge_ngram at index time only. Index-time custom_analyzer builds front-prefixes (autocomplete recall); query-time uses a separate search_analyzer with NO ngram — otherwise every 2-letter fragment of the query becomes a prefix matcher and your results fill with garbage. We learned this from a prod bug where the fragment "el" matched gloves, dresses, and a shop description simultaneously.

2. Query → structured intent

Every query passes a token classifier that tests each token against the live taxonomy in all four languages at once, priority-ordered: gender → category → subcategory → color/size → price.

"pembe yubka"  →  { color_family: "cehrayi", category: "Skirts", text: "" }

Multi-word subcategories get a sliding-window phrase matcher with token-coverage scoring:

for size in (3, 2):
    for window in windows(tokens, size):
        for sub, field, name_tokens in candidates:   # stopwords stripped
            if all(any(similarity(w, nt) >= 0.9 for nt in name_tokens)
                   for w in window):
                coverage = size / len(name_tokens)
                if coverage >= 0.5:
                    accept(sub, window)

So "ətək dəsti" (skirt set) matches the subcategory "Ətək və Üst Geyim Dəsti" as a phrase — a single ambiguous token can never hijack the filter.

Naive matching also needs guards: without them, "ayaqqabısı" (a long word for shoes) fuzzy-matches the size "S". Length-ratio rules and a size-keyword list protect real size queries while blocking accidental ones. Price hints parse too — "50 manatadək" becomes {price_max: 50}, "ucuz" becomes a cheap-range filter.

End-to-end, one mixed-language line — qadın pembe yubka 50 manatadək (AZ + TR + RU-translit + AZ price) — parses to:

{ gender: "women",            # AZ column match
  color_families: ["pink"],   # TR alias
  category: "Skirts",         # RU word → synonym pool → ətək
  price_max: 50 }             # parsed, not matched

Zero tokens left as free text. Swap pembe for English pink — identical parse; aliases carry all four languages, so one sentence can mix them freely.

3. Color science: 107 vendor strings → 15 families

Vendors wrote 107 distinct color values, including pipes ("Qırmızı | Açıq çəhrayı") and literal CSS:

radial-gradient(circle, #000000 20%, #C69258 20%)   # a leopard print

The fix is not more string matching. It's color science:

HEX_RE = re.compile(r'#([0-9a-fA-F]{6}|[0-9a-fA-F]{3})\b')  # finds ALL hexes,
                                                             # even inside gradients

def hex_to_lab(hex_code):        # sRGB → linear → XYZ(D65) → Lab
    ...

def delta_e(lab1, lab2):         # full CIEDE2000 (Sharma formulation),
    ...                          # ~80 lines, zero dependencies

Every family has anchor hexes; every vendor color maps to its nearest anchor by CIEDE2000. We validated against the official Sharma test vectors — 9/9 to four decimals, including the notorious 1.5381 pair. Upgrading from naive CIE76 (Euclidean in Lab) to CIEDE2000 re-classified nine boundary colors correctly: mustard→yellow (was: gold), ivory→beige (was: white), navy stopped bleeding into purple.

And 15 is today's taxonomy, not a ceiling. Families live in an admin table — adding a 16th is a database row, not a deploy. More importantly, the classifier has no color limit at all: whatever new shade a vendor invents tomorrow, its hex lands in Lab space and maps to the nearest family automatically. As sellers grow, the color vocabulary grows without permission — and the organization absorbs it.

Rules on top of the math:

• Pipe/dual colors → both families (a red|pink set must appear in pink searches)
• ≥4 hex-derived families → collapse to "multicolor"
• Pattern aliases (leopard, zebra) → multicolor, hex families skipped

Products with no color variant at all? One-shot backfill: product photo → Gemini Flash-Lite → "dominant product colors as hex, ignore background and skin" → CIEDE2000 → family. 76 products classified for under $0.01 total.

One war story: the first version set max_output_tokens=256 on the Gemini call. An AI review agent flagged it before deploy: thinking tokens share that budget, so the model could burn the whole ceiling thinking and return empty text — the entire backfill would have silently produced nothing. Ceiling raised to 2048, truncation logged.

4. Filters that degrade instead of failing

Hard filters are how you serve empty pages. Every filtered search descends a ladder; the first rung with hits wins:

yield 'full',      search_with(category, color, text)
if color:
    yield 'no_color',  search_with(category, text)
if subcategory:
    yield 'partial',   search_with(text_plus_category_only)
yield 'text_only', search_with(text)

A wrong constraint costs ranking quality, never a blank screen.

Second forgiveness mechanism: the taxonomy bridge. "yubka" detects the Skirts category — but skirt sets live in another category whose subcategory is literally named "Skirt & Top Set." A hard category.id filter silently hid them. Now the filter is:

Q('bool', should=[
    Q('term', **{'category.id': detected_id}),
    Q('match', **{'subcategory.title': {          # concept bridge
        'query': specific_tokens_only,            # generic tokens stripped
        'analyzer': 'search_analyzer',
    }}),
], minimum_should_match=1)

Generic tokens (clothing, set, top…) are excluded so the bridge can't over-widen — if only generic tokens remain, no bridge is built. (Credit: an AI code-review agent caught the over-widening risk before it shipped.)

5. Semantic layer: one embedding space, two doors

• Text and image embeddings live in a single shared 1536-dim space (Gemini Embedding 2, multimodal), stored in pgvector — durable across index rebuilds.
• When lexical results are weak (below a threshold), we run text-kNN and image-kNN and fuse with Reciprocal Rank Fusion:

def rrf_merge(*ranked_lists, k=60, limit=None):
    scores = defaultdict(float)
    for lst in ranked_lists:
        for rank, pid in enumerate(lst):
            scores[pid] += 1.0 / (k + rank + 1)
    ...

• The same space powers visual search (photo → products). One space, two doors.
• The taxonomy is embedded too. Every category/subcategory/gender has a centroid — the mean vector of its products — in pgvector. Visual search classifies the query photo against centroids before ranking:

rows = (VisualCentroid.objects
        .filter(embedding__isnull=False)
        .annotate(distance=CosineDistance('embedding', query_vector))
        .values_list('kind', 'ref_id', 'distance'))
# → "this photo is a women's earring" → type-constrain results
# (an earring photo can never return trousers on color similarity alone)

• Crucially, semantic only fires on weak lexical results. When exact match works, don't dilute it with vibes.

Two production footnotes for the vector crowd:

SET LOCAL hnsw.iterative_scan = relaxed_order;

pgvector's HNSW index silently under-returns when you post-filter a kNN query (e.g. status='Published') — the index yields K candidates, the filter eats most of them, and you get 3 results where 40 exist. Every filtered kNN of ours runs inside that session setting.

And freshness: every product save dispatches a Cloud Task that re-syncs its ES document within seconds — no nightly rebuild, no Celery workers, just HTTP-triggered tasks on Cloud Run. A vendor lists a dress; it's searchable before they close the tab.

6. One brain, three doors: the chatbot runs on the same engine

Our AI assistant has no product knowledge of its own. It calls the search engine through 7 authenticated tool endpoints (API-key via constant-time compare, plus Google-signed JWT for service accounts):

search_products            # same intent parser, same filters
search_shops               # same product-evidence ranking
search_faq / search_blog   # RAG corpus: FAQ + docs + blog chunks,
                           # same embedding space, same RRF fusion
search_products_by_vibe    # semantic-only door
find_similar_products      # embedding neighbors
get_catalog_info           # taxonomy/variants/brands (cached)

Visual search is the third door — same embeddings, same centroids, same ranking rules. Fix a synonym once; the search bar, the camera, and the chatbot all improve in the same second.

7. The data flywheel: why BigQuery is the whole point

Every meaningful interaction emits a structured JSON event — 18 event types, 15 of them routed by the Cloud Logging sink into BigQuery (60K+ events in the last 30 days). The interesting ones aren't pageviews:

log_smart_upload_save(
    ai_suggestions=...,      # what the AI proposed
    vendor_choices=...,      # what the human decided
    changes=...,             # field-level diff: accepted/modified/rejected
    acceptance_rate=...,     # one number per upload
)

That's a labeled training pair manufactured by normal platform usage. Same pattern everywhere:

• SearchHistory stores query → clicked product (text AND visual; visual queries keep their 1536-dim embedding) → query-relevance pairs
• Moderator rejections carry categories and feed a negative-example index; new uploads get a kNN pre-check against it before any Gemini call — a lookalike of a rejected image is flagged in milliseconds and skips the API cost entirely
• A moderator review queue grades visual search results and corrects categories → hand-labeled ground truth
• Zero-result queries land in a dedicated BigQuery view → weekly synonym curation

The staircase: (1) dashboards and mining today → (2) behavior-driven ranking (CTR adjusts weights, semantic blend ratio set by measurement) → (3) fine-tuning our own retrieval/classification models on suggestion-vs-decision pairs and click-relevance pairs. Nobody else has this dataset, because nobody else's users type "гара йупка".

8. The control plane

Every AI behavior is admin-tunable with zero deploys — 47 knobs across two singleton config models (semantic thresholds, RRF k, color-filter toggle, visual similarity tiers, gate model, rate limits, session TTLs). Config resolution: DB override → settings → defaults, cache-invalidated on admin save.

Thresholds aren't vibes. A zero-API-cost eval command runs leave-one-out over every product embedding, compares three classification methods (neighbor-vote / centroid / text-RAG argmax), and prints the threshold grid that reaches 95% precision:

score ∈ {0.60..0.85} × margin ∈ {0..0.05} → precision/coverage table
→ recommended gate = first cell with precision ≥ 0.95

The admin knob gets set to what the benchmark says.

Cost is part of the control plane too: the Gemini catalog context lives in a server-side context cache (~94% of prompt tokens per AI-upload call served from cache), vision inputs are resized before sending, and the rejected-image kNN pre-check skips Gemini calls entirely — each skipped call is logged with its saved cost.

Production debugging, not guessing

When "pembe" (pink) started returning beige and denim skirts in production, we didn't tweak boosts blindly. We traced the parsed intent on the live system:

FILTERS = {'variants': [{'title': 'Açıq Mavi | Krem | Çəhrayı',
                         'matched_field': 'title_tr', ...}],
           'color_families': ['cehrayi', 'bej', 'mavi'], ...}

There it is: "pembe" exact-matched a pipe item's Turkish translation and inherited all three of its families, diluting the filter. Fix: word-aliases take precedence over item inheritance — pink means pink. One trace, one line changed, covered by two new tests.

Numbers

• 4 languages, 2 alphabets, ~100 synonym groups
• 15 color families; 116/116 colors auto-classified; 76 products colored from photos for <$0.01
• CIEDE2000: 9/9 official vectors
• 18 analytics event types → BigQuery; 7 AI tool endpoints; 47 zero-deploy admin knobs
• 366 automated tests on the search stack
• Built and reviewed by 1 founder + AI agents (code review, ES review, migration safety, perf)

Where this goes

The layered architecture is deliberately a template: the Turkic-language world (Turkey, Central Asia) shares the same script-mixing and transliteration chaos as our home market — each new language is a new folding table + synonym pack on the same brain. Next signals on the roadmap: user behavior (already streaming), then body measurements and personal style (opt-in, privacy-first) as ranking inputs. Target state: the search bar behaves like a personal stylist that happens to accept text, photos, and chat.

Takeaways

1. Multilingual search dies at the alphabet layer first. Fold scripts before you tune ranking.
2. A color word is a filter, not a keyword. Parse intent.
3. Lab + CIEDE2000 is 80 dependency-free lines that outperform any hand-maintained color mapping.
4. Build a relaxation ladder. Wrong constraints should cost ranking, never results.
5. Zero-result logs are free product management.
6. One brain, many doors: search bar, camera, and chatbot should share the same intent parser, embeddings, and ranking.
7. Capture every AI-suggestion-vs-human-decision pair from day one. That's your fine-tuning dataset compounding while you sleep.

Questions about any layer — analyzers, the phrase matcher, the CIEDE2000 port, RRF tuning — ask below. I'll share details.

I'm building geyin.az (Azerbaijan's first AI-powered fashion marketplace) solo, in public, with AI. Previous write-up: how we built a zero-hallucination AI content pipeline.