DEV Community: Opensolr

I made search engines understand emojis (and it's weirdly useful)

Opensolr — Thu, 11 Dec 2025 00:28:04 +0000

Been working on hybrid search (lexical + vector) for a while and accidentally discovered something fun: when you use good embeddings, you can literally search with emojis.

Not as a gimmick - it actually works because the embedding model (BGE-M3, 1024 dimensions) learned semantic relationships between concepts and their emoji representations.

Try it yourself

These are live search engines running on real e-commerce data:

Type 🔑 (key emoji) → get actual keys: https://search.opensolr.com/dedeman?q=🔑

Type 🚲 (bike) → get bicycles and accessories: https://search.opensolr.com/dedeman?q=🚲

Type 🖨️📄 (printer + paper) → get printer supplies: https://search.opensolr.com/b2b?q=🖨️📄

This one's my favorite - type "cute domestic pet earrings" on a jewelry store: https://search.opensolr.com/rueb?q=cute+domestic+pet+earrings

(it finds cat and dog earrings even though the product titles are in a completely different language)

How it actually works

The pipeline is:

Crawl website → extract text with Trafilatura
Generate 1024D embeddings via BGE-M3
Store in Solr with both text + vectors
At query time: run lexical search + KNN vector search
Combine scores (hybrid approach)

The emoji thing works because BGE-M3 was trained on multilingual + multimodal data. The model learned that 🔑 and "key" and "Schlüssel" (German) and "cheie" (Romanian) are all semantically close.

So when someone searches 🚲, the embedding is close to "bicycle", "bike", "Fahrrad", "bicicletă", etc.

The weird part

Cross-language search just... works. The Romanian e-commerce site has products in Romanian, but you can search in English or with emojis and it finds relevant stuff. No translation layer, no language detection preprocessing - the embeddings handle it.

Same with conceptual queries. "things to wear around neck" finds necklaces, pendants, chains - even though no product has "things to wear around neck" in the title.

Stack details for the curious

Embeddings: BGE-M3 (BAAI), 1024 dimensions
Inference: Running on RTX 4000 Ada, ~2-5ms per query
Search: Solr 9.6 with dense vector support
Crawling: Custom PHP + Python (Playwright for JS-heavy sites, Trafilatura for extraction)
Extra features: VADER for sentiment, langid for language detection, custom price extraction

Query latency is ~40-50ms total including embedding generation.

Hybrid vs pure vector

Pure vector search is cool but has issues:

Exact matches sometimes rank lower than "similar" results
Product codes/SKUs get weird results
Users expect "nike shoes" to prioritize exact Nike matches

Hybrid fixes this. Lexical handles exact matches, vectors handle the "I don't know the exact word but I know what I want" queries.

The Solr query can be seen in the debig view (bottom-right button) where you can see the actual vector query functions.

vectorQuery = {!knn f=embeddings topK=250}[-0.032, 0.009, -0.049, ...]

lexicalQuery = {!edismax qf="title^550 description^450 uri^1 text^0.1" 
                         pf="title^1100 description^900" ...}

q = {!func}sum(
      product(1, query($vectorQuery)), 
      product(1, div(query($lexicalQuery), sum(query($lexicalQuery), 6)))
    )

Bonus: AI-generated hints

Added an experimental feature where the search can explain results. Search "measure 🔥" on a technical documentation site and it tells you which specific device to use for measuring temperature/fire:

https://search.opensolr.com/fluke?q=measure+🔥

It pulls context from indexed PDFs and generates a recommendation. Uses a local LLM (running on same GPU).

Anyway, thought some of you might find the emoji thing interesting. The cross-language aspect was unexpected - I didn't build it for that, it just emerged from using multilingual embeddings.

Happy to answer questions about the setup or hybrid search in general.

Hybrid Search in Apache Solr is NOW Production-Ready (with 1024D vectors!)

Opensolr — Tue, 09 Dec 2025 13:43:57 +0000

A few days back I shared my experiments with hybrid search (combining traditional lexical search with vector/semantic search). Well, I've been busy, and I'm back with some major upgrades that I think you'll find interesting.

TL;DR: We now have 1024-dimensional embeddings, blazing fast GPU inference, and you can generate embeddings via our free API endpoint. Plus: you can literally search with emojis now. Yes, really. 🚲 finds bicycles. 🐕 finds dog jewelry. Keep reading.

What Changed?

1. Upgraded from 384D to 1024D Embeddings

We switched from paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions) to BAAI/bge-m3 (1024 dimensions).

Why does this matter?

Think of dimensions like pixels in an image. A 384-pixel image is blurry. A 1024-pixel image is crisp. More dimensions = the model can capture more nuance and meaning from your text.

The practical result? Searches that "kind of worked" before now work really well, especially for:

Non-English languages (Romanian, German, French, etc.)
Domain-specific terminology
Conceptual/semantic queries

2. Moved Embeddings to GPU

Before: CPU embeddings taking 50-100ms per query. Now: GPU embeddings taking ~2-5ms per query.

The embedding is so fast now that even with a network round-trip from Europe to USA and back, it's still faster than local CPU embedding was. Let that sink in.

3. Optimized the Hybrid Formula

After a lot of trial and error, we settled on this normalization approach:

score = vector_score + (lexical_score / (lexical_score + k))

Where k is a tuning parameter (we use k=10). This gives you:

Lexical score normalized to 0-1 range
Vector and lexical scores that play nice together
No division by zero issues
Intuitive tuning (k = the score at which you get 0.5)

4. Quality Filter with frange

Here's a pro tip: use Solr's frange to filter out garbage vector matches:

fq={!frange l=0.3}query($vectorQuery)

This says "only show me documents where the vector similarity is at least 0.3". Anything below that is typically noise anyway. This keeps your results clean and your users happy.

Live Demos (Try These!)

I've set up several demo indexes. Each one has a Debug button in the bottom-right corner - click it to see the exact Solr query parameters and full debugQuery analysis. Great for learning!

🛠️ Romanian Hardware Store (Dedeman)

Search a Romanian e-commerce site with emojis:

🚲 → Bicycle accessories

No keywords. Just an emoji. And it finds bicycle mirrors, phone holders for bikes, etc. The vector model understands that 🚲 = bicicletă = bicycle-related products.

💎 English Jewelry Store (Rueb.co.uk)

Sterling silver, gold, gemstones - searched semantically:

🐕 → Dog-themed jewelry

⭐️ → Star-themed jewelry

🧣 Luxury Cashmere Accessories (Peilishop)

Hats, scarves, ponchos:

winter hat → Beanies, caps, cold weather gear

📰 Fresh News Index

Real-time crawled news, searchable semantically:

🍳 → Food/cooking articles

what do we have to eat to boost health? → Nutrition articles

This last one is pure semantic search - there's no keyword "boost" or "health" necessarily in the results, but the meaning matches.

Free API Endpoint for 1024D Embeddings

Want to try this in your own Solr setup? We're exposing our embedding endpoint for free:

curl -X POST https://opensolr.com/api/embed \
  -H "Content-Type: application/json" \
  -d '{"text": "your text here"}'

Returns a 1024-dimensional vector ready to index in Solr.

Schema setup:

<fieldType name="knn_vector" class="solr.DenseVectorField" 
           vectorDimension="1024" similarityFunction="cosine"/>
<field name="embeddings" type="knn_vector" indexed="true" stored="false"/>

Key Learnings

Title repetition trick: For smaller embedding models, repeat the title 3x in your embedding text. This focuses the model's limited capacity on the most important content. Game changer for product search.
topK isn't "how many results": It's "how many documents the vector search considers". The rest get score=0 for the vector component. Keep it reasonable (100-500) to avoid noise.
Lexical search is still king for keywords: Hybrid means vector helps when lexical fails (emojis, conceptual queries), and lexical helps when you need exact matches. Best of both worlds.
Use synonyms for domain-specific gaps: Even the best embedding model doesn't know that "autofiletantă" (Romanian) = "drill". A simple synonym file fixes what AI can't.
Quality > Quantity: Better to return 10 excellent results than 100 mediocre ones. Use frange and reasonable topK values.

What's Next?

Still exploring:

Fine-tuning embedding models for specific domains
RRF (Reciprocal Rank Fusion) as an alternative to score-based hybrid
More aggressive caching strategies

Happy to answer questions. And seriously, click that Debug button on the demos - seeing the actual Solr queries is super educational!

Running Apache Solr 9.x on OpenSolr.com - free hosted Solr with vector search support.

Here's the hybrid vector+lexical scoring trick nobody explains.

Opensolr — Tue, 02 Dec 2025 07:53:37 +0000

We're OpenSolr - Solr hosting and consulting. We're obsessed with search (probably too much).

When we added vector search to Solr, we hit a problem nobody talks about: combining scores.

Vector similarity: 0 to 1
Lexical (BM25/edismax): 0 to whatever

Naive sum = lexical always wins, even when semantically wrong.
Fix: normalized_lexical = lexical / (lexical + k)

Now we have:

Cross-lingual search (EN→RO)
Emoji search (🔥 finds fires, 🐕 finds dog products)
Semantic fallback (wine emoji finds champagne when no wine exists)
Full debug inspector on every search

Live demos you can try:

https://opensolr.com/search/dedeman?q=🔥wood (Romanian hardware)
https://opensolr.com/search/vector?q=🔥 (news)
https://opensolr.com/search/peilishop?q=winter+hat (fashion)

Click the debug button to see actual Solr params. We built it to be educational.
Solr 9.x has dense vector support. You don't need Pinecone.

If you're fighting relevance issues or want help with hybrid search, that's literally what we love doing.
Happy to give pointers.