The File-Watching Race Condition That Only Shows Up in Production

Say you're waiting for a text from someone. Check your phone before they send it, and you'll see it the second it lands. Check five minutes late, you'll still see it, just later than you'd like.

Now imagine a phone that only shows you texts sent while the screen happens to be open. Locked screen, text comes in, it's just gone. No buzz, no badge, nothing. You'd never even know to go looking for it.

That's file watching when it goes wrong. A file watcher only catches a file if it's already up and running before the file shows up. Write the file a split second before the watcher starts, and the watcher never sees it. Not late. Not lost somewhere. Never received in the first place, like it was shouted into an empty room.

Here's the annoying part. It's an easy mistake to make without noticing. Plenty of code writes the file first and starts the watcher after, because that order feels natural when you're writing it. Seems harmless. It's actually the whole bug.

And it hides really well. On a laptop, everything happens one thing at a time, so there's no rush and no problem. Ship it to production, where a bunch of things happen close together under real traffic, and suddenly files start going missing with the logs sitting there giving you absolutely nothing to go on.

Before watchers, there was polling

The usual fix people reach for first is polling. Just keep asking ""is the file here yet.""

while True:
    files = os.listdir(""/logs/orders"")
    for f in files:
        if f.endswith("".log"") and not already_processed(f):
            process(f)
    time.sleep(5)

This works. It also means you're always a little behind. However long that sleep is, that's your delay, every single time. And now you've got extra bookkeeping to handle too. Something has to remember which files already got processed, or a restart just runs the whole batch again.

Cron has the same problem, just dressed up differently.

# /etc/cron.d/watch-orders
* * * * * /usr/local/bin/check-for-new-orders.sh

Run it every minute and you're up to a minute late on everything, all the time. Crank it to every 10 seconds and now the CPU is spinning on a check that finds nothing almost every time it runs.

Neither of these is really about scheduling. It's a script pretending it has a real event to react to when it doesn't.

The real watcher, and its one strict rule

OS level file watchers fix the delay completely. inotify on Linux. FSEvents on macOS. ReadDirectoryChangesW on Windows. They push an event to your process the instant a file shows up. No loop. No polling. Barely any CPU spent waiting around.

That's the upside. The catch is the ordering rule from the intro. The watcher has to already be listening before the file lands. Get that backwards, even by a few milliseconds, and the file just slips through.

Where this actually bites in a real workflow

Take a fairly normal order pipeline. Charge the customer. Write a log file. Watch for that log file to confirm it landed. Send a confirmation once it does.

Write the steps in that literal order, log first, watcher second, and you've already built the bug in. The file can exist before the watcher even starts, and that event is gone for good.

The fix is to kick off ""start watching"" and ""write the file"" at the same moment, as two things happening in parallel instead of one after the other.

parallel:
  watch_log_creation   (file watcher, blocks until *.log appears)
  write_order_log      (writes the file)

Here's what that looks like as an actual step, using Unmeshed's filewatcher.agent.

{
  ""name"": ""filewatcher.agent"",
  ""type"": ""WORKER"",
  ""ref"": ""watch_log_creation"",
  ""input"": {
    ""type"": ""FILE_WATCHER"",
    ""directory"": ""/logs/orders"",
    ""fileNamePattern"": ""*.log"",
    ""watchCriteria"": ""ENTRY_CREATE"",
    ""watchDuration"": 20000
  }
}

Because this step runs alongside the one writing the file, the watcher is already subscribed before the write ever happens. Not thanks to a sleep timer. Not thanks to a retry cleaning up after the fact. Both branches just start together, by design.

A few things worth keeping in mind

Subscribe before you produce. Every time. If the watcher and the file write aren't started together on purpose, there's a race condition sitting there waiting for the wrong day to show up.

Treat the timeout as an alarm, not a normal wait. watchDuration set to 20000 doesn't mean ""wait 20 seconds, that's just how long it takes."" It means ""if nothing's shown up in 20 seconds, something upstream is broken."" Worth treating that seriously instead of just retrying and hoping.

Make the glob pattern tighter than it feels like it needs to be. *.log will grab every log file in that folder, including ones that belong to a completely different process. ORD-*.log takes two extra seconds to type and saves you a genuinely confusing debugging session down the road.

The actual takeaway

The tool was never really the hard part. inotify versus polling got settled a long time ago. The ordering is where things quietly go wrong, and it's a rough one to debug because it only shows up under timing conditions your laptop just never runs into.

Building this by hand means treating watching and writing as one single move, done at the exact same time, on purpose, not two steps that happen to be next to each other in the code."