DEV Community: Juraj Madzunkov

Your code is tested. Your measurements are not.

Juraj Madzunkov — Sun, 30 Aug 2026 14:15:24 +0000

I keep a ledger of every change I ship that changes behaviour. Each entry carries a written prediction and a pre-ship baseline, and a review date is set the day it ships, usually five days out. On that date I have to come back and read the real numbers against what I said would happen.

Twelve changes have now been through that review. None has been reverted.

That is not the interesting number. The interesting number is that eight of the twelve reviews found something genuinely broken - and in all eight it was the measurement, not the change.

What an entry looks like

- id: adaptive-source-cap
  metric: digest_fill_rate
  target: >
    Sunday 2026-08-09 and Monday 2026-08-10 fill rates return toward
    their pre-2026-07-29 level (91.7-97.6%) and clearly beat the
    76.7%/59.0% measured after the cap change [...]
  baseline: >
    [...] A/B on 2026-08-03's real pool for the reporting subscriber:
    cap1 -> 11, cap2 -> 13, cap3 -> 14, against 20 requested -- so the
    relaxation is expected to recover roughly 2-3 stories, not all 8.
    The rest is genuine weekend thinness and should stay unfixed.

That is the real entry, trimmed only where marked. Read the two fields together.

The baseline contains an A/B on real data: relaxing the cap takes the subscriber from 11 stories to 13 or 14, against the 20 they asked for. That is a ceiling of 65-70% fill, and the baseline says so in words - recover roughly 2-3 stories, not all 8.

The target, written in the same sitting, a few lines below, demands 91.7-97.6%.

The fix did exactly what its own experiment predicted it would do, and failed its target for it. Verdict recorded as "Worked, against a target it could never have reached."

The three failure modes, in order of how hard they were to see

1. The target contradicts its own baseline

The one above. Easy to catch once you look, and the fix is mechanical: if an entry contains an experiment bounding the effect, the target may not exceed that bound. That is now a rule.

2. The baseline and the metric are in different units

An entry recorded a baseline of "286 collected, 71 delivered" over 30 days, described as computed with the metric's own logic. It was not. Those were distinct stories; the live metric counted slots across digests. Same nouns, different denominators.

The review came out wrong in a flattering direction, which is the dangerous kind. A result that disappoints you gets re-checked. One that confirms you does not.

3. The metric moves when your population does

This is the one worth the article.

The metric pooled a per-slot share across every subscriber who received a digest that day:

# what it did, in effect
share = matching_slots_across_everyone / total_slots_across_everyone

That number moves when the subscriber mix moves, even when nothing in the pipeline has changed. And it did move: two subscribers stopped receiving partway through the review window.

Same five days, same rows, three defensible readings:

reading	result
pooled across everyone	+1.45
late-leaving subscribers dropped	+4.26
per-subscriber, each against themselves	+3.84

Only the third is composition-free. It is also the only one that showed what actually happened: 6 of 6 subscribers improved, at +0.59 slots per digest.

I had already drafted the verdict "failed target, not failed fix" off the pooled number.

Two defensible readings disagreeing by 3x on identical data is not an inconvenience to resolve by picking one. It is the finding.

The related trap: the open day

One more, because it is the cheapest to fix and the easiest to ship.

My subscribers choose their own delivery hour, so the archive table gains rows all day - the send window runs 05:04 to 12:00 UTC. Every "since date X" metric treated today as a finished day.

A partial day does not read as missing data. It reads as a data point.

And the error is directional, not random: late digests skew one way, so an early read is always wrong in the same direction, and always on the newest point - exactly where you look to decide whether something is trending. It manufactured a four-day decline that was not there.

def _last_complete_day() -> str:
    """
    The most recent date whose archive rows are final: yesterday, UTC.

    Yesterday is provably safe rather than merely conservative: every row
    was created AND last updated within its own digest_date, checked
    across all 90 days of retention, zero exceptions.
    """
    return (date.fromisoformat(today_str()) - timedelta(days=1)).isoformat()

Two details in that docstring matter more than the one-liner under them.

It is not a time-of-day check. The first version of this guard hardcoded a cutoff derived from the current subscriber list, and it was wrong immediately - it assumed a single late subscriber at 11:00, because the other two happened to have that send day switched off on the day I measured. Dropping the whole open day is correct no matter who sends when.

It is deliberately not applied everywhere. The pool-backed metrics read a file the 05:00 pipeline writes once, so today's file is either absent and skipped, or already complete. Clamping those too would throw away a good data point to fix a problem they do not have. A guard applied uniformly because uniformity feels tidy is just a different way of being wrong.

Ten call sites route through it. A random error adds visible noise. A directional error on your most recent point manufactures a slope.

The takeaway

We have a whole discipline for testing code and essentially none for testing the things we measure code with. A broken function throws. A broken measurement returns a number, and the number looks exactly like an answer.

If you keep any kind of before-and-after metric, three questions worth asking today:

Does my target contradict a bound recorded in the same entry?
Are my baseline and my live metric counting the same unit?
Does my denominator move when my population moves rather than when my system does?

I build devdigest, a daily tech and AI digest that shows you why it picked each story - the ledger above is how I keep myself honest about whether any of it works. It launched on Product Hunt this weekend: https://www.producthunt.com/products/devdigest-io

A directional error on your newest data point manufactures a trend

Juraj Madzunkov — Thu, 27 Aug 2026 16:20:59 +0000

My dashboard had a column called "today". Today wasn't finished.

I build devdigest, a daily tech digest. Every change that changes behaviour gets reviewed against real data five days later, and I write the numbers down before I let myself have an opinion about them. It is the only thing standing between me and believing whatever I was hoping for.

This week's series, the share of each digest landing in the four categories a fix I had just shipped was meant to feed:

69.2, 67.6, 60.0, 53.0

Four days, straight down. I had the write-up half composed in my head.

Then I noticed the last number was wrong.

The bug

Subscribers choose when their digest arrives. The first goes out just after five in the morning, the last at midday. My measurement treated the current day as a finished one, so reading it in the morning meant measuring a day that had not happened yet.

The real figure was 53.9, not 53.0. Nine tenths of a point. Not exciting. Here is why it is worth writing down.

Why a small error mattered

The error is directional, not random. The late digests skew one way, so an early reading is always wrong in the same direction. And it is always wrong on the newest point, which is exactly where you look when you are deciding whether something is trending.

A random error adds noise you can see. A directional error on the most recent point manufactures a slope.

The part I got wrong twice

Day five came in at 61.9 and did not continue the decline.

My first instinct was to write that up as the decline never having been real. That instinct is wrong, and it is wrong in the same way as the one it replaced. Four points looked like a trend. Five points look like a recovery. Neither is something five points can tell you, and day six could resume the fall.

What I can say is narrower:


Observed four-day drop	15.3 points
Largest four-day drop in the 22 days before it	8.7 points

So something real happened, and waving it away as noise would have been the mirror image of the mistake I had just caught myself making.

The measurement error and the reading error turn out to be the same error wearing different clothes: wanting a small number of days to settle a question they cannot settle.

So I am not calling it. There is a date in my notes to read the same series over eleven days instead of five, and I will post what it says.

The fix

The guard is small. Metrics now stop at the last day that is provably closed:

def _last_complete_day() -> str:
    """The most recent date whose rows are final: yesterday, UTC."""
    return (date.fromisoformat(today_str()) - timedelta(days=1)).isoformat()

Yesterday is provably safe here rather than merely conservative, and I checked before choosing it: every row in the table was created and last updated within its own date, across the whole retention window, so once the date has passed nothing further is written to it.

A partial day does not look like missing data. It looks like a data point, and you will believe it.

If anything you rely on has a "today" column, go and find out what time today finishes.

I read the Terms of Service for every source in my news aggregator

Juraj Madzunkov — Mon, 17 Aug 2026 12:01:31 +0000

I build devdigest, a daily tech digest. It pulls from RSS feeds, has a model read each excerpt to categorise and score it, writes an original summary, and emails subscribers a list of titles, summaries and links back to the original articles. Full article text is never reproduced or stored beyond a short-lived per-run cache. To read anything, you click through to the publisher.

Before charging money for that, I wanted to know whether any of my sources explicitly banned it. So I read the actual Terms of Service for all of them. Not the privacy policy, not a summary, the real current terms, quoting the clause or explicitly recording "no ToS found."

I expected the answer to track company size. Big corporate sites run their own RSS feeds, press APIs and developer-relations teams, so I assumed they would be relaxed. Small scrappy publications felt like the ones who would care.

I had it exactly backwards.

The split

The most restrictive terms came from VC-data companies and large corporate infrastructure blogs. Verbatim:

Crunchbase News bans using content "to train models (including generative artificial intelligence technologies)," and separately bans anything that "'Crawls,' 'scrapes,' or 'spiders' any page, data, or portion of" the content.

HPCwire/AIwire bans any "robot, spider, or other automatic device" without prior written permission, plus "any form of data extraction or data mining, or other commercial exploitation of any kind." The most specifically targeted restriction I found anywhere.

TechRadar (Future plc): "You shall not conduct, facilitate, authorise or permit any text or data mining or web scraping... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."

Cloudflare's blog bars automated bots from "scrape, data mine, copy, or use the materials or content on this website for developing, training, fine-tuning, or otherwise contributing to or improving a machine learning model or artificial intelligence (AI) system."

Sifted (§7.6), the European startup publication: "You must not conduct, facilitate, authorise or permit any text or data mining or web scraping in relation to our Platform... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."

Now the other direction, from the sites I assumed would be precious:

InfoQ: "We permit the posting of a summary and then a link back to the InfoQ landing page." That is my exact pipeline, described approvingly, in their terms.

MIT News: its Terms of Use state it "offers RSS feeds for syndication purposes."

GitHub: "these Terms do not restrict lawful access to or use of the contents of public repositories by third parties."

arXiv: the API Terms of Use explicitly names RSS-based discovery and notification tools as a permitted use case, in those words.

TechCrunch maintains dedicated RSS terms, separate from its general ToS, explicitly permitting display of feed content with attribution and a link to the full article.

Engineering blogs, university news offices and syndication-native sites were mostly fine with it. The well-lawyered commercial sites were not.

My best guess at why: the restrictive ones have their own data-licensing business to protect, or expect to. A clause banning AI training is an asset if you intend to sell that access later. The permissive ones live on distribution. Being read and linked to is the product, so the feed is a front door rather than a leak.

The part that actually matters: three ways this goes wrong

The size heuristic being backwards was interesting. What was useful was discovering that the obvious check is wrong in three different directions.

1. The ToS is clean and robots.txt bans you.

I originally cleared Towards Data Science. It is Medium-hosted, and Medium's Terms of Service contain no RSS, scraping, commercial-use or AI-training restriction. Clean read.

I had not checked robots.txt. Medium's explicitly disallows ClaudeBot, GPTBot and every other major AI crawler by name. My pipeline runs on OpenAI's model, so GPTBot's presence in that list was not an academic detail. I removed the source.

Checking the terms and not robots.txt gave me a clean answer to the wrong question.

2. The site publishes an RSS feed and its ToS bans scraping anyway.

Sifted surfaces an RSS feed link on its own homepage. Its Terms of Use, quoted above, ban text and data mining and separately define and prohibit any "'robot', 'bot', 'spider', 'scraper' or other automated device."

A published feed is not a licence. It is extremely easy to read the existence of /feed as permission, because for most of the web's history that is roughly what it meant. It does not mean that now, and the terms are the binding document.

3. The scraping clause exists but is scoped somewhere else entirely.

TechCrunch has a "robots/spiders/scrapers" clause. On a keyword search it looks like a red flag. Reading it in context, it is scoped to user-submitted content, comments and forums, not published articles, and the separate dedicated RSS terms permit exactly what I was doing.

If I had grepped for "scraper" and moved on, I would have dropped a source that explicitly allows this. False positives cost you real sources.

Boilerplate is not signal

The single most useful methodology note, and the one that took longest to arrive at.

Nearly every commercial media ToS carries some version of "all rights reserved, no reproduction without permission." It is close to universal. Treated as a prohibition, it bans what every RSS reader, every feed aggregator and every newsletter on earth does daily.

So it cannot be the test. If your rule flags everything, it has told you nothing.

What I ended up looking for is a clause that specifically names automated access, bots, scraping, RSS or feed reuse, or (increasingly common since 2024) AI training and summarisation. Those describe the actual pipeline. Generic anti-plagiarism language does not.

The rule I settled on: exclude only on an explicit stated prohibition. Never infer a ban from silence. Several sources had no discoverable terms at all - Meta's AI blog has no ToS page whatsoever, Simon Willison's blog has a bare copyright symbol and nothing else. Silence is not permission, but it is also not a prohibition, and inventing one costs a good source for nothing.

What I did about it

Eight sources came out over the audits: Crunchbase News, HPCwire/AIwire, TechRadar and Cloudflare's blog on explicit clauses; Ars Technica and Wired on Condé Nast's posture, including active litigation against an AI company over scraping, where I could never get a verbatim read; Towards Data Science on the robots.txt finding; Sifted once §7.6 turned up in a later pass.

The list has since grown past 120 sources and the audit runs on a schedule rather than once, because terms change. The 2024-onward AI-training clauses are mostly new language. A source cleared in 2023 is not cleared now, and several of the clauses above carry recent dates.

None of these sites did anything wrong. Writing a restrictive ToS is entirely a publisher's prerogative, and given how much unlicensed scraping has funded model training, the defensive drafting is easy to understand. I quote them here because the terms are public and because "check for yourself" is useless advice without showing what the check actually looks like.

Caveats

I am not a lawyer and this is not legal advice. This is one developer reading terms carefully before shipping, which is a lower bar than a real IP review and a considerably higher one than not looking.

Two things I would tell anyone building something that reads other people's content automatically:

Check the terms and robots.txt, separately, and treat them as independent. They disagree, in both directions.

And check the sites you assume are fine. My entire mental model of who would object was wrong, and the only reason I know that is that I read all of them instead of the ones I expected to be a problem.