I run a small Shopify app that hides payment and shipping methods at checkout, and on the 23rd I shipped five little in-app banners to nudge free merchants towards the paid plan, with conditions, priorities and an opt-out all pinned down by tests. Four days later I went and measured how many merchants had seen one. Zero.
Not "a few". Zero. And in the same 4 days a Pro trial appeared, which I very nearly wrote down as the banners working. It wasn't. Let's go through it, because the mistake is not in the code and I suspect you've got one of these somewhere too.
What I built
So the banners are these, each with its own condition, all evaluated by one pure function that decides which single banner a page gets, if any.
- free plan, exactly 3 active rules: "you've hit the limit"
- free plan, merchant picks a Pro-only field in the condition dropdown: an inline hint under the dropdown
- free plan, at least 1 active rule, not dismissed: "try Pro"
- any plan, rules saved but none switched on, not dismissed: "saved, but off"
- installed 7 days or more, at least 1 active rule, never asked before: a one-time "how is it working for you, write a review"
Only one shows per page. Priority is over-limit, then at-limit, then not-running, then review, then the trial nudge, and there's a test that guards both the order and the copy.
The review one writes asked: true into a metafield on the app installation the moment it renders, so a merchant is asked once and never again. I was quite pleased with all of this. It is, I think, the most carefully specified feature in the app, which is going to turn out to be the joke.
Counting the audience, badly
Before shipping I counted who could possibly see any of it, and I got the count wrong, so I'd rather say that here than have you find it in the tables.
The app is installed on 29 stores. Some are mine (development stores I test on), and the way I told mine apart from real merchants was a flag Shopify puts on the shop record, plan.partnerDevelopment. True meant "ours", and ours got left out. That gave me 18 merchants at launch and 20 by the weekend.
But whose partner? The flag doesn't say. It's true for anybody's development store, and the way this came out is almost embarrassing: someone helping me tried to open two of those "ours" stores from my Shopify account to check a banner, and got "you don't have permission". They were somebody else's.
So the real split is 4 stores of mine (2 confirmed, 2 I'm fairly sure of) and 25 third-party stores, which were 23 at launch, not 18, and the paid conversion I'd been quoting as 2 out of 20, 11%, is actually 2 out of 25, 8%. Same two paying merchants, smaller number, and I'd been reporting the bigger one for a week.
The code, by the way, never made this mistake. reviewPromptDue() looks at install age, active rules and whether I've already asked, and nothing else, and it was only my spreadsheet that had opinions about who counts.
4 days later
The review banner leaves a mark when it renders, so it's the one I can actually measure. 12 installations qualified for it, 2 of mine and 10 real merchants, and I queried the metafield on all 29 stores. asked was set on none of them.
Now, what does zero mean here? Either nobody who qualified had opened the app since the deploy, or the banner renders and the write to the metafield doesn't happen. Those are very different problems and the number alone doesn't tell you which one you've got.
So I opened one of my own stores that was in a qualifying state (Pro plan, installed 11 days, 4 active rules, never asked) in a normal browser, once, and watched. The banner was there. The metafield went to {"asked":true,"optedOut":false} with an updatedAt of the exact second the page loaded.
Eligibility works, the route works, the write works. Which leaves the other explanation: in 4 days, not one of the qualifying merchants I can observe opened the app.
Why "I can observe"? Because of my own priority order. One of the 10 is on the free plan with exactly 3 active rules, so if that merchant opens the app they get the "you've hit the limit" banner, which outranks the review one and writes nothing. For that store I can't tell either way. So it's 9 merchants with no record of opening the app, and 1 I have no information about, and the only banner that leaves a mark can be pushed aside by one that doesn't. (It can also go quiet on its own: if the Admin API call that checks the metafield fails, the code treats the merchant as opted out and shows nothing, on the grounds that not asking is better than asking twice. So even "no record" is a slightly softer fact than it looks.)
I want to be careful with the wording here, because "the banners are broken" is the sentence people will repeat and it's false. They render and they record. They just didn't reach anyone, because there was nobody in the room. (The 2 stores that now have asked set are the 2 of mine that I opened to check, so the merchant count is still 0.)
The trial that wasn't mine
Over the same 4 days the numbers that moved were Pro trials 0 to 1, paid Pro 2 to 2, and listing reviews 0 to 0. One trial, right after shipping trial nudges! You can see how that ends up in a weekly report, can't you?
But I looked at the timestamps. That store completed OAuth at 09:09 UTC on the 24th and started the Pro trial at 10:01 UTC, 52 minutes later. The trial banner only shows to a free merchant with at least 1 active rule, and the review banner needs 7 days of install age, so a brand new install that goes straight to Pro inside an hour was never in a state where any banner could have reached it.
The timing is right and the causation simply isn't there. So the trial goes in the "happened anyway" column, and I'm a bit annoyed at how close I came to filing it under "banners".
And I can't measure clicks anyway
The five upgrade buttons are plain links, href straight to the plan page, so a click never touches my server. There's no tracking code in the app at all, 0 lines of it, and that's deliberate: the privacy page says there are no analytics or advertising trackers embedded in the app, and I'd have to change that public promise before I could count a single click.
Is that worth doing for a click counter? I don't know yet. It's a policy decision rather than an afternoon of work, and I haven't made it.
So the measured effect of the other four banners is not "zero". It's "never measured". Those are different too, and the second one is worse, because you can't even tell whether the thing you built is doing anything.
What I actually learnt
Specifying a banner precisely and getting it in front of a person are separate problems, and I only solved the first one. An in-app message reaches the people who open the app, and a merchant who sets up a rule once and leaves (a perfectly sensible thing to do with a checkout app) will never see anything I put inside it, however good the conditions are.
The other thing is duller but probably more important. Without a mark that gets written when something happens, you can't tell "built" from "reached", and sooner or later you'll take credit for something that happened on its own. The review banner writes a mark. The other four don't. Guess which one I can say anything about?
Anyway, next week the job is getting in front of the 10 merchants who qualify, by some route that isn't inside the app, and deciding whether I'm willing to amend a privacy page for a click counter. The first four banners are still up, still correctly specified, and still unmeasured.
Top comments (0)