DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Our product search only returns rows the scanner can actually answer for

Munchable answers a barcode. You point the phone at a pack, the rules engine checks the ingredient list against the conditions on your profile, and you get a verdict. That works in a shop. It does not work on a Tuesday evening when you are writing a list and the pack is still in the shop.

So we added name search. It is one GET route and about a hundred lines of query building, and almost none of that hundred lines is about matching text. It is about which rows are allowed to appear at all.

A hit you cannot tap is worse than no hit

The catalogue has rows in states the scan path quietly refuses to answer for: withheld rows, and rows we hold a barcode and a brand for but no ingredient list. A barcode lookup handles those by telling you it cannot check this product, which is an honest answer to a question you asked by scanning something.

It is a terrible answer to a question you asked by typing. You searched, you saw a result, you tapped it, and the app said "not found". You did the right thing and the app punished you for it.

So the searchable set is the servable set, defined once:

const servable = sql`${catalogProducts.status} <> 'withheld'
  and cardinality(${catalogProducts.ingredientsTags}) > 0`;
Enter fullscreen mode Exit fullscreen mode

Every row this route can return is a row the engine can produce a real verdict for. Nothing else is in the index, let alone in the results.

The index and the query are the same two expressions

Search runs over name and brand joined together, so that "alpro oat" can match a row named "Oat drink" by "Alpro". That join is an expression, and the filter above is a predicate, which makes the supporting index a partial expression index:

CREATE INDEX products_search_trgm_idx ON catalog.products
USING gin (
  (coalesce(product_name, '') || ' ' || coalesce(brands, '')) gin_trgm_ops
)
WHERE status <> 'withheld' AND cardinality(ingredients_tags) > 0;
Enter fullscreen mode Exit fullscreen mode

Postgres will only use that index when the query's expression and predicate match the index definition. Not "are logically equivalent". Match. Swap the argument order of the concatenation, drop a coalesce, write the predicate with the two conditions reversed in a way the planner does not normalise, and you still get correct results, just from a sequential scan over the whole catalogue, and you find out in production when the table has grown.

That is a silent failure mode, so the two expressions are written exactly once in the application and reused by both sides:

const haystack = sql`(coalesce(${catalogProducts.productName}, '')
  || ' ' || coalesce(${catalogProducts.brands}, ''))`;
Enter fullscreen mode Exit fullscreen mode

This is not tidiness. It is the only mechanism I have that stops the query and the index drifting apart, because nothing else in the stack will tell me when they do.

Per word, not per phrase

The matcher tests each word of the query separately against the haystack, rather than the whole query as one substring:

export function queryTerms(query: string): string[] {
  return [...new Set(query.split(' ').filter(Boolean))];
}
Enter fullscreen mode Exit fullscreen mode

"alpro oat" becomes two independent ILIKE '%alpro%' and ILIKE '%oat%' conditions, which is what lets a brand word and a product word match a row where they appear in that order, the other order, or in different columns. Duplicates are dropped, so typing the same word twice does not buy a second index probe for the same answer.

ILIKE has three special characters, and a product name search is exactly the place people type them:

export function likePattern(term: string): string {
  return `%${term.replace(/[\\%_]/g, (c) => `\\${c}`)}%`;
}
Enter fullscreen mode Exit fullscreen mode

Without that, searching for "100%" looks for "100" followed by anything, which quietly returns the wrong twenty rows rather than throwing.

Ranking is two lines, and the second one is the important one

.orderBy(
  desc(sql`similarity(${haystack}, ${query})`),
  sql`${catalogProducts.uniqueScansN} desc nulls last`,
  catalogProducts.barcode,
)
Enter fullscreen mode Exit fullscreen mode

Trigram similarity against the whole query first, so the best textual match wins. Then how often the product has actually been scanned, which is the tiebreak that matters: for any common word, there is a household product and there are forty regional variants that happen to share the letters. Similarity alone cannot tell them apart. Scan counts can.

The third key is the barcode. It never changes an answer, it just makes the order total, so the same query does not shuffle its ties between requests.

Constants that are product decisions

export const SEARCH_MIN_CHARS = 3;
export const SEARCH_MAX_CHARS = 80;
export const SEARCH_LIMIT = 20;
Enter fullscreen mode Exit fullscreen mode

Three characters because a trigram match on fewer is mostly noise. Eighty because that is longer than any product name, so anything past it is not a search. Twenty because there is no paging: a longer list is a scroll, not an answer, and if the right product is not in the first twenty then ranking is what needs fixing, not the limit.

The route knows nothing about products

The search endpoint returns three fields per hit: barcode, name, brand. That is everything a tappable row needs and nothing else.

Tapping the row does a normal product lookup through the existing endpoint. That is deliberate: the scan path's cache, its trust rules and its quota accounting all apply to a search result exactly as they apply to a scan, without the search route knowing that any of them exist. The alternative, returning the full product from search, would have meant a second path into the same data with a second copy of every one of those concerns.

Rate limiting is two keys enforced in parallel, one on the account and one on the client address, and it fails open:

try {
  decision = await enforceParallel([
    { limiter: lim.searchDev, key: user.id },
    { limiter: lim.searchIp, key: getClientIdentifier(request) },
  ]);
} catch {
  decision = { success: true };
}
Enter fullscreen mode Exit fullscreen mode

If the limiter backend is unavailable, the search happens. A name search is a read that reveals nothing about the person doing it, so the cost of letting one through is lower than the cost of breaking the feature during an incident. That is not the default posture everywhere in this codebase, and the fact that it differs per route is the point.

Condition-blind, on purpose

The query is matched against product names and nothing else. It is not stored, it is not logged and it is not keyed to the account. The route handles it the way the barcode lookup handles a barcode: as a condition-blind read.

That falls out of a rule the whole product is built on, which is that your conditions stay on your device. The server is asked "what is in this product", never "what is in this product, for someone with IBS". Search would have been the easiest place in the app to quietly break that rule, because search logs are so obviously useful. They are also, for a health app, a list of what people are worried about eating.

See the data it searches

The same catalogue and the same engine sit behind our public pages, so you can poke at the data without installing anything:

Search itself lives in the app, on the scan hub, at munchable.app. There is no manual barcode entry next to it and there never will be, because nobody has ever wanted to type thirteen digits. Typing "oat drink" is a different proposition entirely.

Top comments (0)