Every morning a cron job on my VPS pulls weekly prices for about 1,080 ski chalets and apartments in the Alps and rebuilds a comparison site from them. One of the pages it builds answers a simple question: how much more do you pay if you go skiing in the Dutch spring school holiday?
My first answer was 59%. It went into a draft headline.
The real answer is 40%. Nothing crashed in between. The query was correct, the numbers were correct, and the conclusion was still wrong.
The obvious query
The spring holiday for the north and middle of the Netherlands starts on 20 February 2027. The natural comparison is early March, one or two weeks later, when schools are open again.
So: take every accommodation with a price per person (ski pass included) that is still available in the week of 20 February, take the median, do the same for the weeks of 6 and 13 March, divide.
+---------------------+-----------+--------------------+
| week | available | median per person |
+---------------------+-----------+--------------------+
| 20 February | 187 | €1,045 |
| early March | 341 | €658 |
+---------------------+-----------+--------------------+
+59%. Clean, quotable, and exactly the kind of number a journalist would pick up.
The problem is in the second column.
Two medians, two different sets of chalets
In early March, 341 accommodations are available. On 20 February only 187 are. The rest are already booked.
A median over 187 leftovers divided by a median over 341 is not a price change. It is a comparison between two different groups of chalets that happen to share a database.
So I checked who was missing. I took the accommodations that are free in early March and split them by what they are doing on 20 February:
+----------------------------------+------------------------------+
| group | median price in early March |
+----------------------------------+------------------------------+
| already booked for 20 February | €646 |
| still free on 20 February | €704 |
+----------------------------------+------------------------------+
The cheap ones sell out first. What is left for the holiday week is the more expensive end of the market, and my 59% was measuring the leftovers as much as the holiday.
Compare every chalet with itself
The fix is to stop comparing groups. For each accommodation that is available in both periods:
- take its price in the holiday week,
- divide by its own average price in the reference weeks,
- take the median of all those ratios.
This is the function that now builds the page (the codebase is in Dutch, paar means pair):
def paar(wa, wbs, lijst=None):
r = []
for a in (alle_met if lijst is None else lijst):
pr = alle[a['id']]
pa = prijs_in(pr, wa)
pb = [float(x) for x in (prijs_in(pr, w) for w in wbs) if x]
if pa and pb:
r.append(float(pa) / statistics.mean(pb))
return (statistics.median(r), len(r)) if len(r) >= MIN_N else (None, len(r))
There is a side effect I did not plan for. A ratio has no unit. Accommodations priced per stay instead of per person can now join in, because each one is only ever compared with itself. That lifted the group from 154 to 219 accommodations available in both periods.
Result: +40%.
Still a real surcharge. But 19 points of my first number came from who was still available, not from what anything cost.
It is not even wrong in a consistent direction
I ran both methods on all four school holiday weeks:
| holiday week | compared with | medians | paired |
|---|---|---|---|
| 19 December | mid January | +23% | +23% |
| 26 December (New Year) | mid January | +52% | +56% |
| 13 February (spring, south) | early March | +46% | +35% |
| 20 February (spring, north and middle) | early March | +59% | +40% |
For New Year the simple method is too low. The places still free on 26 December are the cheaper ones (median €637 in January, against €674 for those already booked), so the leftovers pull the median down instead of up.
For 19 December both methods agree. That week has not sold out unevenly yet. It will, as December gets closer, and the simple number will start to drift while the paired one should stay where it is.
That is what makes this kind of error hard to spot. There is no sign to check. Sometimes it inflates, sometimes it deflates, sometimes it does nothing, and every version looks plausible.
What I took from it
The median of what is available is a property of the day you measure, not a price. Availability is a filter, and it is not random. Anything you compute after that filter inherits its bias.
Compare a thing with itself whenever you can. Pairing removes every difference between groups in one move: location, size, quality, type. It also quietly solved a unit problem I had been working around.
Check who is missing before you trust a comparison. The split into "already booked" and "still free" took five lines and told me more than the headline number did.
Put the denominator on the page. The live page shows how many accommodations each percentage is based on, and per country it only shows a number when at least 15 accommodations could be compared. Smaller groups produced figures like +116% that were pure noise.
This is the same lesson as my win rate post: the population changed underneath the number, and nothing in the code noticed.
The page is skizie.com/wintersport-goedkoopst (in Dutch), rebuilt every morning on Skizie, a ski holiday comparison site I run on my own. The raw numbers are an open dataset on GitHub, mavemedia/wintersport-prijzen, CC BY 4.0. If you see a hole in the method, I would like to hear it.
Top comments (0)