One real scan counts a hundred: ranking a curation queue by your own users

Munchable curates its own ingredient data. The queue of work is always longer than the week, so the interesting engineering question is not how the curation runs. It is which row goes first.

Get that ordering wrong and you can run the pipeline flat out for a month, close thousands of rows, and change nothing at all for anybody using the app.

The ordering we inherited

Our catalog started from a one-time import, and those rows carried a popularity number. Easy to sort by, already in the table, no work at all.

It is also a number about somebody else's users, frozen on the day of the import, describing a market that is not ours. A product that was heavily scanned elsewhere three years ago outranks the own-brand yoghurt that four of our subscribers scanned this morning. Every hour of curation goes to the first one.

So we needed our own popularity signal, and we needed it without building a tracking system. Munchable is a health app. The one thing we will not do is accumulate a per-user record of what people scan.

A popularity counter that cannot become a history

The whole counter is one Redis sorted set per calendar month. The key is the month, the member is the barcode, the score is the number of lookups.

That is everything. No account identifier, no device identifier, no IP address, and no timestamp finer than the month itself. Two months are kept and the keys expire on their own.

It is worth being precise about why this shape is the point rather than an implementation detail. A row of "barcode, count, month" cannot be turned back into anybody's scan history, including by us, because the identity was never in it. There is no anonymisation step to get wrong, no join key someone could add later in a hurry. The privacy property is structural.

Two more rules around it:

• Counting is fire and forget. The lookup route schedules it after the response has been sent, so a scan never waits on the counter, and it is best effort throughout. If Redis is unavailable, curation falls back to the frozen import numbers and nothing else notices.
• Counting must never be able to fail a scan. The person in the aisle is the priority; the analytics are not.

It is documented, in those terms, in the privacy policy under "What a barcode lookup reveals": munchable.app/privacy.

One real scan counts a hundred

Now the ordering. The imported popularity numbers run into the thousands for a famous product. A barcode our own users looked up has, on a good month, a count in single or double digits.

Sort by the sum and the import wins forever. So the live counts are weighted:

/**
 * One live lookup counts this much seed popularity. One real scan by a
 * Munchable user has to outrank a famous import comfortably, because it is a
 * person who pays for this app standing in front of that label right now.
 */
const LIVE_SCAN_WEIGHT = 100;

A constant with a comment that is a product decision rather than a tuning note. It says: our users' current shelf beats an import's historic popularity by two orders of magnitude, and if that is wrong, this is the one line to argue with.

Everything downstream reads the ranking, so the argument happens in exactly one place.

The aggregate, and keeping it bounded

The ranking is one pass: every ingredient tag on every product we serve, how many products carry it, and how much those products are actually scanned.

with carried as (
  select p.barcode, t.tag, coalesce(p.unique_scans_n, 0)::float as seed_w
  from catalog.products p, unnest(p.ingredients_tags) as t(tag)
  where p.ingredients_tags is not null and p.status <> 'withheld'
)
select carried.tag,
       count(*)::int as c,
       sum(carried.seed_w + 100 * coalesce(live.w, 0))::float as w
from carried
left join live on live.barcode = carried.barcode
group by carried.tag
order by w desc, c desc
limit $1

The interesting part is live, which is not a table. The Redis counts are passed into the query as two parallel arrays and unnested into a relation:

with live as (
  select barcode, w
  from unnest($barcodes::text[], $weights::float[]) as l(barcode, w)
)

The obvious alternatives are worse. Fetching per-barcode counts inside the loop makes the query's runtime depend on a network round trip per row. Writing the counts into a Postgres table first adds a synchronisation problem and something new to vacuum, for data that expires in two months anyway. Passing them in as arrays keeps the whole ranking as one query over one snapshot, and the join is just a join.

The other things that keep a million-row aggregate from becoming an incident:

• The statement timeout is set LOCAL inside the transaction, so it applies to this query and does not leak to anything else sharing the connection pool.
• The row limit is a parameter with a default, not "whatever comes back".
• The grouping happens in Postgres. Pulling tag arrays into the application to count them there is how you turn a database's job into a memory problem.

There is a second, optional pass that collects which other ingredients each candidate is usually printed alongside, because what a word is seen next to is often what makes an ambiguous word placeable. It is capped by ids and by products, wrapped in a try, and returns empty on any failure with a warning logged. Context helps and is never required. A nice-to-have query that can take down the job it decorates is not a nice-to-have.

Two queues, one measurement

The same aggregate feeds two weekly jobs that do genuinely different work:

• One takes the ingredients the engine cannot name: words that appear on labels and do not resolve to anything we know.
• The other takes the ingredients it can name but cannot score: words we understand perfectly, which no rule set has an opinion about.

The second queue is the one teams forget, and it is where the user-visible wins hide. "We know exactly what this is and have nothing to say about it" reads, on a phone in a supermarket, exactly like "we have no idea what this is".

Both queues are ordered by the same weighted ranking, so both spend their budget on what people are actually scanning this month.

What it looks like from outside

You cannot see a work queue from the front end, which is rather the point of doing it this way. What you can see is the result: the set of ingredients we have a considered answer for keeps growing, and it grows in the order our users bump into them.

• munchable.app/answers, one page per ingredient and condition we can answer for
• munchable.app/conditions, what each condition's rules check
• munchable.app/privacy, the counter behind the ordering, described exactly

If you are running any kind of curation or moderation backlog, the highest-leverage change is usually not throughput. It is asking what your queue is sorted by, and whether that number is about your users or about somebody else's.