A health check that flags "started but never finished" — for every day it runs before the schedule is done

A scheduled pipeline calls a health check partway through its own daily run. The check flagged one thing: today's date, "started but never finished." The pipeline was not stuck. It just was not done yet.

The check

The logic, generalized to the actual shape:

if "today start" in log_text and "today end" not in log_text:
    flagged.append(day)

Read a log file, look for a start marker and an end marker, flag anything with the first and not the second. That is correct for any day that has actually ended.

Why it flagged today

The log for the current day had a start marker. It could not have an end marker yet — the pipeline only writes that marker after the last task of the day finishes, and the health check was itself invoked by an earlier task in that same run. At the moment the check ran, the day's schedule was still in progress. No end marker existed to find, whether or not anything was actually wrong.

This isn't intermittent. The code above makes it deterministic: every invocation made after the run has started but before its end marker is written produces this exact flag.

What I verified before calling it a false positive

I didn't want "it's probably fine" to be the answer just because it was the reassuring one. I pulled the log for the flagged day and read the timestamps against the task list:

[08:07:00] task A — exit 0
[08:07:00] task B — exit 0
[08:07:00] incident recorded: task C failed
[08:07:00] task C — exit 1
[08:07:00] task D — exit 0
[08:08:42] task E starts

Every entry from the start of the run to the point the health check itself fired belongs to the same process ID, in sequence, with no gap. Task C failing with exit 1 is a real, separate problem — logged and recorded as its own incident — but it is not what the health check was reporting on. Nothing in that window shows the run stopping and something else starting later. "Started, no end marker yet" and "started, then stopped" produce an identical log shape to this check. Only one of them is actually a problem.

What I did not do

I did not change the health check. The fix needs a decision this task wasn't scoped to make: how to tell "the log is from today, and today isn't over" apart from every other case where a start-without-end pattern really does mean something stopped. What I wrote down instead: compare the log's date to the current date, and only apply the no-end-marker rule to days that are not today. The script itself is still unchanged.

Why this shape is worth knowing about

Any health check built on "there's a start marker, I'm checking for an end marker" runs into this if it can be invoked before the thing it's checking is actually finished — a check called mid-pipeline, by one of the pipeline's own steps, is exactly that scenario. In this incident, the tell was: the flagged date was the current date, and the process that supposedly "stopped" was still writing to the log when I went to look.

What I check now

When this kind of check reports "started, never finished," the first thing I look at is whether the flagged date is today. If it is, I check whether the run is actually still going — same process ID, timestamps still advancing — before treating it as an incident. A start-without-end pattern is only worth treating as stopped once the window it's looking at has actually closed.

This article was written with the help of AI. The incident, the log lines, and the exit codes are real and from my own machine; the draft was AI-written, then checked against the original log before publishing.