DEV Community

Mark Veenema
Mark Veenema

Posted on

My pipeline ran green for 8 days while importing nothing

#programming #python #devops #webdev

Every morning at 13:20 a cron job on my VPS imports holiday park accommodations from eleven affiliate XML feeds. Every morning it exited 0. The log said "klaar" (done). Nothing in Sentry, nothing in the mail.

For eight days, one of those eleven feeds imported exactly zero rows.

How I found it

Not from an alert. I was adding a column to a dashboard and noticed one provider's parks all had the same stale timestamp:

SELECT adverteerder,
       MAX(laatste_feed_seen) AS laatst,
       COUNT(*)               AS rijen
FROM accommodaties
WHERE actief = 1
GROUP BY adverteerder
ORDER BY laatst;
Enter fullscreen mode Exit fullscreen mode 
+--------------+---------------------+-------+
| adverteerder | laatst              | rijen |
+--------------+---------------------+-------+
| Landal       | 2025-09-08 13:21:04 |  1126 |
| Roompot      | 2025-09-16 13:22:11 |  1402 |
| EuroParcs    | 2025-09-16 13:22:40 |   389 |
| ...          | 2025-09-16 ...      |   ... |
Enter fullscreen mode Exit fullscreen mode

Ten providers from yesterday. One from eight days ago, across 1126 rows. Nothing had errored. The feed had simply stopped matching anything my parser looked for.

Layer one: the country filter

The script starts by throwing away everything outside the Netherlands:

if eigenschap(p, 'country') != 'Nederland':
    continue
Enter fullscreen mode Exit fullscreen mode

The feed used to say Nederland. It now says NL. Every single product hit that continue on the first line of the loop, so the parser did its full run over 1887 products and wrote nothing. No parse error, because nothing got far enough to parse.

NL_CODES = ('NL', 'NLD', 'Nederland')

if eigenschap(p, 'country') not in NL_CODES:
    continue
Enter fullscreen mode Exit fullscreen mode

I expected that to be the whole bug. It was the first of four.

Layer two: renamed fields

With the filter fixed, products got through, and every one of them came out with nulls where the useful data should be. The provider had migrated the feed to a new schema and renamed four fields:

old new
ParkCode parkID
city place
zipcode postal_code
province region

Any sensible parser asked for a missing field returns None. Mine returned None four times per product and wrote the row anyway, because every column except the key was nullable.

The region rename came with a value change too. Provinces arrived in English, underscored:

REGIO_MAP = {
    'North_Holland': 'Noord-Holland',
    'South_Holland': 'Zuid-Holland',
    'Wadden':        'Friesland',
}

def landal_provincie(p):
    r = eigenschap(p, 'region')
    return REGIO_MAP.get(r, r)
Enter fullscreen mode Exit fullscreen mode

Wadden is not a province at all, it is an island group. It maps to Friesland for my purposes. A straight string pass-through would have created a province page for a thing that is not a province.

Layer three: the field that kept its name and changed its meaning

This was the expensive one, and it is the reason I am writing this post.

<name> still existed. It was still a string. It still looked completely reasonable:

<product>
  <name>Amerongse Berg</name>
  ...
Enter fullscreen mode Exit fullscreen mode

In the old feed, <name> was the accommodation name: "Veluwelodge XL 6 personen". In the new feed, <name> is the park name, and the accommodation name moved to a different element.

So my parser ran clean, my import reported success, and it inserted 1887 accommodations all called "Amerongse Berg". Rows where every unique thing about the accommodation had been replaced by the name of the site it sits on.

I shipped that. It went live for about twenty minutes before I noticed a category page listing the same lodge forty times.

DELETE FROM accommodaties
WHERE adverteerder = 'Landal'
  AND aangemaakt_op >= CURDATE();
Enter fullscreen mode Exit fullscreen mode

The fix reads both, preferring the explicit park field and falling back to the old element:

code = eigenschap(p, 'parkID') or vol_id.split('_')[0]
naam = eigenschap(p, 'parkName') or tekst_van(p, 'name')
Enter fullscreen mode Exit fullscreen mode

A renamed field fails loudly. A repurposed field fails silently, passes every type check you have, and gets committed.

Layer four: the IDs changed

The last layer: my primary key for an accommodation was a SHA1 hash of the old product ID. The new feed issues different product IDs. Every row in the new import was therefore new, and the 1126 existing rows were orphans with no path back.

What survived the migration was the affiliate URL, which carries both IDs as query params. So I rebuilt the key from the URL:

def bestaande_landal_sleutels(cur):
    import re, urllib.parse
    cur.execute("SELECT affiliate_url FROM accommodaties "
                "WHERE adverteerder=%s AND actief=1", (ADVERTEERDER,))
    uit = set()
    for (url,) in cur.fetchall():
        u  = urllib.parse.unquote(url or '')
        mp = re.search(r'park=([A-Za-z0-9]+)', u)
        ma = re.search(r'accommodation=([A-Za-z0-9_-]+)', u)
        if mp and ma:
            uit.add(f"{mp.group(1)}_{ma.group(1)}")
    return uit
Enter fullscreen mode Exit fullscreen mode

A one-off migration script matched old rows to new IDs through that set: 1033 updated, 93 deactivated as genuinely gone, and 114 duplicates surfaced that had been quietly sitting in the table for months. The duplicates were not caused by this bug. The bug just made me look.

The brake I added afterwards

The other half of the fix is a brake on the importer itself. New products are no longer inserted blind; they have to already exist, or they get counted and skipped:

if u['acc_id'] not in bestaand:
    telling['nieuw_overgeslagen'] = telling.get('nieuw_overgeslagen', 0) + 1
    continue
Enter fullscreen mode Exit fullscreen mode

That counter is the actual alert. A healthy run skips a handful. A run that skips 1800 means the feed changed shape again, and the number lands in the log instead of 1800 junk rows landing in the database.

The script I am glad I had not written yet

The same week, I had been planning a cleanup job to deactivate accommodations that disappear from their feed. Stale inventory is a real problem: a lodge that no longer exists is a dead link and a bad search result.

If that script had existed during those eight days, it would have deactivated all 1126 Landal accommodations. Correctly, by its own logic. They were absent from the parse result on every single run.

So when I did write it, the rule went in first:

DAGEN   = 5    # comfortably above the daily cron interval
MAX_PCT = 20   # more than this share of one source: skip the source
MAX_ABS = 50   # and never more than this many at once per source
Enter fullscreen mode Exit fullscreen mode

Mass disappearance is a parser symptom, not an inventory one. Real inventory decays a few rows at a time. If twenty percent of a provider vanishes overnight, the thing that broke is my code, and the correct response is to refuse to act and say so.

Ordering matters as much as the thresholds. The deactivation job runs at 13:30, after every fetcher has finished (last one at 13:25) and before the page generator at 13:35. Run it first and it deactivates based on yesterday's data; run it last and it publishes pages for accommodations it is about to retire.

What I took from it

Exit code 0 means the script finished, not that it worked. Mine finished eight times, beautifully, having written nothing.

Count what you skip, not just what you write. Every continue in a parse loop is a silent discard. One counter per skip reason, logged every run, would have caught this on day one.

Freshness is a first-class metric. MAX(laatste_feed_seen) per source, compared against the cron interval, is a four-line query and the single highest-value check in the whole pipeline. It is now the thing I look at first.

A field that keeps its name and changes its meaning is the worst case. No type error, no missing key, no exception. Just 1887 rows of plausible nonsense. The only defence is asserting on values, not shapes: if every product in a batch shares one name, something is wrong regardless of whether it validates.

Write the circuit breaker before the destructive job, not after. I got lucky on the ordering of two tasks on a list. That is not a strategy.

If you want the practical version rather than the incident, I wrote the three checks up separately: Three checks that catch a silently broken data pipeline.

The pipeline feeds Parkzie, a Dutch holiday park comparison site I run on my own. Eleven feeds, three languages, one person, which is exactly the setup where a silent failure gets to live for eight days.

Top comments (2)

Collapse
 
mavemedia profile image
Mark Veenema •

Follow-up for anyone who wants the practical version rather than the incident: same three guards, written as something you can apply on Monday instead of a story about what went wrong.

mavemedia.hashnode.dev/three-check...

Collapse
 
mavemedia profile image
Mark Veenema • • Edited

I pulled these three guards out into a small repo, no dependencies, one file:
github.com/mavemedia/feed-guard

Some comments may only be visible to logged-in visitors. Sign in to view all comments.