At my company, we realized that users do not search for homes the way traditional search systems expect. Some users type simple keywords. Others describe what they want to see in images, for example “homes with large glass windows in the living room” or “backyards with large trees”.
Our existing search system was not designed for this behavior.
Property descriptions usually do not contain visual details like trees, lighting, or window style. Because of this, keyword search and vector search on text were not enough. We needed image based search at scale.
At the same time, the implementation was not scalable. We had around 1.3 million image embeddings created using Nomic vision models. Vector search was done inside PostgreSQL using pgvector, and most of the logic was written as complex SQL queries. As traffic and data increased, performance degraded badly. Search latency went up to 5 minutes, making the feature unusable.
Ownership and Approach
As a senior software engineer, I took ownership of fixing both the performance and correctness of Deep Search.
One key decision I made was to keep the system deterministic and not use LLMs for ranking or retrieval. LLMs are useful, but they are non deterministic and hard to control at scale.
I used an LLM only to understand user intent. The LLM parses the user query and extracts:
- Hard filters like number of bedrooms or location
- A flag that indicates whether image based search is required
For example:
- “2 bedroom apartment in Manhattan” only needs deterministic filters and keyword search
- “2 bedroom house with backyard having large trees” requires visual understanding and triggers Deep Search
This keeps LLM usage limited and makes the rest of the system predictable and debuggable.
Challenges Found During Implementation
While redesigning the system, I discovered a major issue with how results from different retrieval systems were merged.
BM25 search has its own ranking and pagination. Vector search also has its own ranking and pagination. When results were paginated first and then merged, pagination broke completely.
Page 2 from BM25 and Page 2 from vector search did not represent the same set of results. Some pages contained mostly vector matches, some had none, and rankings changed between requests. This caused unstable and inconsistent results, which is not acceptable for a production search system.
This problem required rethinking how ranking and pagination were handled.
Solution
When Deep Search is triggered, I built a hybrid search pipeline with clear separation of concerns:
- Hard deterministic filters
- BM25 full text search using RedisSearch
- Vector search on property images using pgvector
The key change was that none of these systems paginate independently anymore.
To merge results correctly, I implemented Reciprocal Rank Fusion, which is designed to combine ranked lists from different search engines.
For each property, we compute a single hybrid score:
- BM25 rank is converted into a reciprocal score
- Vector similarity scores are normalized
- We apply weights based on whether Deep Search is triggered
hybridScore = alphaBM25 * bm25Score + betaVector * (vecScore / maxVec)
Only after this unified hybrid score is computed do we apply pagination. This guarantees stable ordering across pages.
Result
This approach solved both performance and ranking issues.
- Latency reduced from around 5 minutes to under 10 seconds
- Pagination is stable and deterministic
- Visual matches surface naturally when users ask for visual features
- Keyword intent still influences ranking
- Results are consistent across pages
Top comments (0)