My plugin spent thirteen days in the WordPress.org Featured tab last month. I
decided to measure what that did, properly, because nobody who had been through
the programme had ever published figures.
The findings are on my own site. This post is about the instrumentation, because
that turned out to be the harder half, and three of the mistakes cost me real
data before I caught them.
Rule one: rows, not reports
The first version was a written summary at the end of each day. That lasted two
days.
Prose does not subtract. You cannot diff two paragraphs. The moment I wanted
"how many more activations than yesterday", I had to reread and recount.
Everything went into one CSV, one row per day, one column per measurement, and
a script that recomputes the whole history from source on every run. The script
is idempotent: run it five times, get the same file. That property matters more
than it sounds, and I will come back to it.
Rule two: three install counters that must not agree
This is the part that surprised me most, and it is the thing I would tell anyone
instrumenting a WordPress plugin.
There is no such thing as "the install count". There are at least three, they
measure different things, and forcing them to reconcile destroys information:
wp_active_installs the bucket printed on the plugin page ("200+")
wp_installs_api the same figure from the WordPress.org API
db_act_cum my own count of plugin activations, cumulative
On 23 August the page said 40 and the API said 30. Same plugin, same moment.
The API lags the page by a day or more. Neither is wrong.
And the third one is not comparable to either. An activation is my plugin
reporting in from a site. WordPress.org counts sites that keep it and phone
home through wp-cron. Over the run I recorded 565 activations while the public
bucket moved by 170. Both numbers are correct. They answer different questions.
My first instinct was to pick one and call it canonical. That was wrong. I kept
all three in separate columns and wrote down what each one means, because the
gap between them is itself a measurement — it is churn, visible for free.
⚠️ If you ever find yourself writing a reconciliation step between counters like
these, stop. You are about to average away the only interesting signal.
Rule three: the funnel is cohort-based, not a count of events
The number I actually cared about was not installs. It was how many of those
people ever ran a search.
The naive version counts search events per day. That answers a different
question, because one enthusiastic tester inflates it.
What I wanted was first occurrence per instance:
activation → wizard → signup → FIRST search
Each stage records the first time a given instance_id reaches it, ever. A site
that activates in August and searches in September lands in August's activation
row and September's first-search row. That is the honest shape, and it is the
only way "conversion" means anything.
The result was the finding of the whole exercise. Over thirteen days the tab
sent twenty-two times more people than my baseline period, and the share who
ever ran a single search went from 50% to 4.8%.
Three traps that cost me data
Container logs die on deploy. docker logs is not storage. Every deploy
wiped the window I was measuring. I lost a day before I noticed and moved to
dumping logs to disk before any deploy, as a step in the deploy path rather than
a thing I remember to do.
nginx rotates under tomorrow's date. access.log-20260823 contains traffic
from the 22nd. My filter matched on filename, so an entire day's requests never
counted. The fix is to read both D and D+1 for any given day. This was a real
bug, live for a day, and it under-reported exactly the metric I was watching.
Evening runs record a partial day. Run the collector at 6pm and you have
written a row for two thirds of a day, which then sits there looking complete.
Because the script recomputes from source, I made it re-derive the last three
days on every run. Yesterday's partial row closes itself the next morning
without anyone touching it.
That last one only works because of the idempotence property from rule one. If
the script appended instead of recomputing, every fix would need a migration.
The counter that stopped me publishing a wrong conclusion
When the conversion rate collapsed, the obvious explanation was bot traffic.
I had one column that ruled it out: the share of new accounts that verified
their email address. Thirteen of seventeen before the rotation, a hundred of
a hundred and thirty-two during. Identical rate on either side. Scripts do not
verify email.
But that column only covers people who created an account, and they were about
one in eight of the arrivals. So it rules out the lazy explanation and says
nothing about the rest, and the write-up says exactly that.
I nearly wrote a much stronger sentence off the back of that row. The thing that
stopped me was noticing the denominator was not what I assumed. Worth checking,
every time, which population a rate is actually a rate of.
What I would do differently
Log the reason, not just the outcome. I can see that a search returned
nothing. Reconstructing why, after the fact, is guesswork.
Record delivery, not dispatch. Related lesson from the same week: my mail
code logged "sent" the moment SMTP accepted the message. A bounce arrived eight
hours later, from a queue I do not control. The application log said success the
entire time.
Decide what is public before you start. Figures from WordPress.org are
public by definition and I published them raw. Absolute counts from my own
database are not, so those went out as rates. Making that call on day one meant
I never had to relitigate it while writing.
The write-up
The actual numbers, the daily roster of all eight plugins, and the caveats where
they belong:
https://queryra.com/blog/wordpress-org-featured-plugin
If you have instrumented something similar, I would like to compare notes. One
plugin is not much to go on.
Top comments (0)