Doorman: my apps' contact form got 23 messages in six weeks, and all 23 were pitches

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

I have three small Shopify apps. Between 29 August and 13 October their listings' contact form and my email got 35 messages: 23 once the repeats and image-only ones are gone, and all 23 were selling something. Installs and reviews "from real store owners" at $30 a store, paid posts in Shopify Discords, UGC videos. One opened "are you the right person in charge of this brand...?", so I answered it. It was a pitch.

The problem isn't the pitches. It's the day a real merchant writes "the cleaner deleted a product I wanted to keep, can I get it back?", and it sits between two messages that begin "I came across your Shopify app and was really impressed".

Gmail already catches most of this. The pitches sent straight to my address mostly went to spam. The nine that came through the app listing's contact form all reached the inbox, because Shopify delivers them inside its own notification email, from its own address. A spam filter sees Shopify; only the words inside are a pitch. That's the inbox Doorman is for: the one behind a listing, where every message arrives wearing the marketplace's uniform.

Doorman sorts a contact-form inbox: customers first, pitches folded away. Drop in .eml files or a Gmail Takeout .mbox, or paste a message, and press Sort. Pitches fold away with the giveaway sentence highlighted; close calls stay on top as "not sure"; nothing is deleted or answered. It runs entirely in your browser: the model downloads from Hugging Face once, and a customer's message, with their name, store and order in it, never leaves the page.

Who would pay for it: anyone whose contact form belongs to a marketplace listing, starting with Shopify app and theme developers. Those pitches can't be filtered upstream, because the marketplace is the sender. A new app lives on its first reviews, and a merchant whose question sat under a pile of pitches is a lost install or a one-star review. The page stays free; what I'd charge for is the version that runs on the Gmail page itself (below).

Demo

Live: doorman.mikey9220.workers.dev. The example is five merchants' questions from the public Shopify forum that the classifier has never seen, and seven pitches from my own inbox (anonymised, and part of its training data). The first sort downloads EmbeddingGemma once (197 MB); then the twelve take about two seconds on my Mac's GPU.

Code

mikey92 / doorman

Sorts the contact-form inbox of a small app or store: real customers in, sales pitches out. Gemma runs in your browser; messages never leave your machine.

Doorman

Your app's or shop's contact form, sorted: customers first, sales pitches folded away. It runs in your browser; nothing you load leaves the page.

Live: https://doorman.mikey9220.workers.dev · Write-up: on DEV · Evaluation: eval/CLASSIFIER.md · Data: eval/DATA.md

In six weeks my three Shopify apps' contact form, and the email behind it, got 23 different messages from 21 senders. All 23 were people selling me something: installs, reviews, a promo post in their Discord. Not one was a customer. The day one is, I don't want it buried under the next "I came across your app and was really impressed".

Load messages (paste one, drop .eml files, or a Gmail Takeout .mbox) and press Sort Inbox. EmbeddingGemma 300M, running in a Web Worker on your GPU, turns each message and each of its sentences into 768 numbers; a logistic regression trained on 245 hand-labelled forum posts and my…

View on GitHub

How I Built It

.eml / .mbox / paste → the text, minus the form's wrapper and quoted replies
→ EmbeddingGemma 300M on WebGPU: the message and each sentence as 768 numbers
→ logistic regression (768 weights, 7 KB of JSON): P(pitch)
→ ≥ 0.8 folded away · 0.5–0.8 "not sure" · below 0.5 customer
→ the most pitch-like sentence quoted on the card

A static site on Cloudflare's free tier; Transformers.js runs onnx-community/embeddinggemma-300m-ONNX (q4). No server.

First try: ask Gemma 3 1B, and read its mind

Last week I ran Gemma 3 1B in the browser, so I asked it to copy the sentence where the sender offers their own product or service, or write NONE.

Gemma 1B doesn't copy. Asked for "word for word", it paraphrases: "OFFER: StartStorez Bundle is a free bundle app" for a message that said "you can give StartStorez Bundle a try". Every quote failed the check.

So I stopped asking it to write anything. For each sentence I asked "In this sentence, is the sender offering the business their OWN product, app, service, agency or help? Answer Yes or No", and instead of reading its answer, read its next-token logits for "Yes" and "No". With an open model you can do that: the generate loop takes a logits processor, and mine only looks.

const peek = (_ids, logits) => {
  pair ??= [logits.data[yes], logits.data[no]];
  return logits;
};
await model.generate({ ...inputs, max_new_tokens: 1, do_sample: false, logits_processor: [peek] });
return pair[0] - pair[1]; // log-odds of Yes over No

The logits told me something the answers never would have: Gemma 1B says Yes to almost everything. On 40 hand-labelled messages, customers scored between −2.3 and 11.6 in log-odds; pitches between 6.8 and 14.9. A merchant asking for "Alerts via email, ideally with an attached CSV or Excel report" scored 11.6, higher than 11 of the 20 pitches. It ranks pitches above customers more often than not (AUC 0.91), but only 9 of 20 pitches beat every customer, and with one forward pass per sentence it took about 20 seconds a message on my Mac's CPU.

What worked: don't ask, measure

A pitch doesn't need to be understood to be recognised. "I came across your app", "I can help you get your first installs", "let me know if you're open to a quick chat": pitches read alike, and an embedding model is built to notice what reads alike. EmbeddingGemma is a quarter of the download, and one forward pass gives me a message's 768 numbers. All I needed was examples to fit a line through them.

Where the examples came from

My inbox had 23 pitches and no customers: the problem, not a dataset. So I went where merchants ask in public: the Shopify Community "Shopify Apps" board. I took the 60 most recently active topics, 637 posts, and read and labelled every one: asking for help, selling something of their own, helping with nothing to sell, or other.

282 of the 637 posts were someone selling their own app or service. 48 of the 60 topics had at least one. One merchant asked for bundle app recommendations "(from shop owners, not app developers!)"; seven of the thirteen replies were from app developers.

Leaving out 150 posts where I couldn't tell whether the account was the app's maker, and keeping one pitch per account, gave 115 merchants asking and 130 pitches. (How it was labelled.)

How well it works

Five-fold cross-validation over the forum posts, split by topic so nothing is ever scored by a model that saw its thread (full report):

Fold away at P ≥	Customers hidden	Pitches folded away
0.5	5 of 115	125 of 130
0.8 (the page)	2 of 115	102 of 130
0.9	0 of 115	79 of 130

The pitches it doesn't fold away aren't lost either: the 28 under 0.8 stay on top, 23 of them marked "not sure" with their most pitch-like sentence quoted. They cost five seconds; a hidden customer costs a sale, which is why the bar is 0.8.

AUC 0.991: pick a random pitch and a random customer, and 99 times in 100 the pitch scores higher. On the same 40 messages Gemma 1B judged, scored by a model trained on the other 205 posts, it's 1.000: the most pitch-like customer scores 0.28 and every pitch is above 0.8. I chose the settings on this table, so read it as a ceiling.

The two customers it would hide are merchants describing their own shop in ad voice ("Award-winning refill shop in Oxfordshire…"). That is a pitch, just not aimed at me, and it's why nothing is ever deleted.

My inbox: a lesson in where training data comes from

Trained on the forum alone, the classifier folded away only 13 of my 23 pitches at 0.8 (22 at 0.5). Cold emails are written differently from forum replies: longer, more flattering, and they end by asking for a call. The fix was to train on them too, and to test it honestly I scored each of the 21 senders with a model trained on the forum and on every other sender's messages. 23 of 23 folded away at 0.8, 19 at 0.9. That inbox has no customers in it, so it's only half the test; the forum table is the other half. The shipped classifier is trained on all 268 messages.

Why Does Open Innovation Matter?

The messages are the private part. A support inbox is customers' names, emails, store URLs and order problems. With an open model in the browser, nothing is uploaded and no server holds anything. "Your customers' messages never leave your machine" is the whole pitch of a product like this, and with a hosted API I couldn't say it.

It's free to run, so it can be cheap to buy. A message takes under a fifth of a second of the visitor's GPU. A hosted API would put a bill on every message, a key behind a server, and a floor under the price.

I could look inside. Reading Gemma 1B's logits is what showed me it was saying Yes to everything, which a chat API's confident "Yes" would have hidden. Open weights also let me train a 7 KB head on EmbeddingGemma's vectors, ship it as JSON anyone can read, and run the same code in Node for the evaluation.

Where a closed model would be better: a frontier model would likely sort these well without a single labelled example. But it would see every customer's message, charge for every one, and I'd still have needed the 245 labels to know how well it does.

My First User

Me, and I'm not counting it: all 23 of my pitches folded away, each sender scored by a model that had never seen them. I'm looking for three app developers to run their own Takeout export through it and tell me what it got wrong; what they find goes in the comments.

What they'll ask next, if my inbox is any guide: "can it run inside Gmail?" That's the Chrome extension: the same model, on the page you already have open.

Prize Categories

• Best Use of Gemma: EmbeddingGemma 300M reads every message on the visitor's GPU, with its classification prefix; Gemma 3 1B's open logits showed why asking a small model "is this a pitch?" doesn't work.