My Runs Took 19 Hours, My Measurement Counted 58,530

Two weeks ago I audited the claims in this blog's agent instructions one by one
against the code. The one that mattered most in that audit was deploy duration:
the docs said "about 12 minutes" in five separate places, and measuring twenty-four
runs gave a median of 3.3 minutes. That was one workflow. This morning the question
was wider: how much machine time do all of the repository's workflows consume?

Measuring looked easy. The GitHub Actions API gives timestamps for every run;
subtract created_at from updated_at and you have the duration. One line of
Python, one coffee. I pulled the last thousand runs, summed the differences, and the
screen told me: 58,530 hours. Six and a half years. Across fifteen days.

The number was so absurd that it saved me. I wish every bad measurement were wrong
that loudly; most of them look plausible, settle into a dashboard and quietly bend
decisions. The uncomfortable part wasn't that the measurement was wrong — it was that
I turned out to be the culprit.

A thousand runs — but why a thousand?

I didn't choose the window; my query did. While pulling the runs I had set a
created=>=2026-09-25 filter, and when the API returned nothing after the tenth page
I assumed that was all of it. It wasn't. The repository's run list reports a
total_count of 4,240, and when I removed the filter and paged again, all
forty-three pages came back full — every one of the 4,240 runs, reaching back to
14 September.

The ceiling is real, just not where I thought it was: GitHub documents that this
endpoint returns at most 1,000 results for searches that use filters such as
actor, branch, event, status or creation date. An unfiltered listing has no such
limit. So what cut my dataset down to 1,000 runs wasn't the repository's history. It
was my query.

I kept the measurement on those thousand runs anyway — 10 October 08:45 UTC through
25 October 08:09 UTC, fifteen days. But I'm now writing that down as a choice rather
than a finding. The rest of this article is about a bigger version of the same
mistake: the shape of the query decides the shape of the answer.

The second surprise came from inside the dataset. I also pulled the job records for
each of those thousand runs, because I wanted the real duration from there. Only
801 of the 1,000 yield a real duration; the remaining 199 consist entirely of
skipped jobs, and curiously all of them belong to a single workflow (I'll come back
to that). A skipped job has a completed_at, and its started_at doesn't mark a
real beginning, so a naive "end minus start" produces negative durations on those
rows. My first pass produced exactly that. This is where I always lose the most time
when measuring: the data contains rows that have nothing to do with the question, and
averaging them without filtering makes you look foolish.

So now I have two durations for 801 runs: the naive one from the API field, and
the real one from the jobs' own timestamps.

Measurement	Total	Median
updated_at − created_at (naive)	58,530 hours	87.4 hours
Job timestamps (real)	19.2 hours	73 seconds

Three thousand and fifty times off. In the median, the gap between 87 hours and 73
seconds is the difference between "it ran for four days" and "it ran for a minute and
a half."

If the error depends on age, the calendar is the suspect

My first instinct was to blame the API. Then I sorted the rows by date and the
pattern stared back at me: for today's and yesterday's runs, the naive duration is
correct. Across the 147 intact runs the median difference between naive and real
is 4 seconds — the method works. The error lives only in rows older than three
days, and there it has no exceptions: 654 of the 801 runs (81.6%) show a naive
duration more than an hour above reality.

An age-dependent error doesn't tell you the arithmetic is broken. It tells you the
data was modified after the fact. If a run's duration grew days after the run
finished, somebody touched the record and walked away.

The age at corruption came out of the same data: of the 654 broken rows, the fifth
percentile is 3.30 days, the median is 3.75 days, and the highest is 4.23 days.
Every record rots about three and a half to four days after birth. That was oddly
close to a number I had written myself.

The culprit's signature: 05:43

In the repository's .github/workflows/ directory sits a workflow called
cleanup-actions-storage.yml. I wrote it. Its purpose is clear: delete old run logs
every day so the Actions storage quota never fills. Its default keep_days is 3.

I tied it down with two measurements. First: on the morning of 25 October the cleanup
run started at 05:41:56 and finished at 05:45:02. The new timestamps on the records
that rotted that morning are spread across four separate seconds, from 05:43:05 to
05:43:08 — the middle of the three minutes the janitor was awake.

The second is more decisive. I checked whether the logs of the broken rows still
exist, probing the log endpoint for twenty runs spread across the window:

Run age	Log HTTP	Naive duration	State
0–3 days (4 runs)	200	0.00–0.35 hours	intact
3+ days (16 runs)	404	78–101 hours	broken

Twenty runs, twenty hits. Not one record with surviving logs has an inflated
duration; not one record with deleted logs has a correct one. The script's artifact
and cache deletion steps aren't filtered by age — whatever they delete, they delete
every night — so they can't produce a signature that lands on a three-day threshold.
That leaves the one step that does look at age: log deletion.

There's also this: the cleanup job's own run records meet the same fate. The record
of the janitor that ran on the morning of 21 October was overwritten at 05:43:08 on
25 October — by the colleague that came four days later. I watched an automation eat
its own tail and, I'll admit, I rather enjoyed it.

My intent wasn't careless; I even wrote the comment at the top of the workflow
myself: run records (success/failure status) are preserved, only the detailed log
files are deleted. The sentence is true. The run record is still there, its status is
there, its commit is there. The one thing that never crossed my mind was that the
record's timestamp is data too. The log deletion call pulls updated_at to the
current hour, and the record looks like it was breathing for four days.

One field does survive, and the data shows it: in all 654 broken rows
run_started_at still matches created_at to the second. The janitor doesn't touch
it.

Every sweep touches exactly one day

A small detail, but it sits at the heart of the mistake I repeat most often in my
career: keep_days says 3 and the measurement says 3.75 days. Two reasons.

First, the cleanup script builds its cutoff at day resolution. The line is right
there in the 25 October run's own log: Cutoff date: 2026-10-22, with the query
created=<2026-10-22. That sweep therefore excludes all of 22 October and touches
only runs created on the 21st. The data says the same: all 37 records overwritten
that morning were created on 21 October, aged between 3.30 and 4.17 days. A job that
ran just after midnight sees four days; one that ran in the late afternoon sees three
and a half. A single calendar day becomes an age band a day wide.

Second, and sneakier: the cron time doesn't hold. The workflow says 0 1 * * *,
i.e. 01:00 UTC. Across the last thirty scheduled runs the median delay is 16
minutes and the worst is 286 minutes. The distribution splits in two: runs up
to 13 October started 9–16 minutes late, and every run after 14 October was 259–286
minutes late. That looks less like load jitter than a regime change; GitHub's docs
say the schedule event can be delayed under load, but I didn't measure the cause of
this step and I don't know it. The practical effect: my "runs at one in the morning"
cleanup actually runs around 05:30, pushing the age band a day and a half forward.

In fairness: 2 of the 654 broken rows don't fit this pattern. Both were updated
75–79 minutes after their run, on the evening of 22 October, and both are CodeQL
analysis runs. It's plausible that code-scanning results touched the record late, but
I couldn't verify it. The arithmetic works for 652 of them and not for two; I'm
writing that down rather than deleting it, because a table that hides the limits of
its measurement lies to the reader.

Hunting for the right field: defined and undefined

So which field should I trust? I downloaded GitHub's own OpenAPI description and
looked at the workflow-run schema. The schema has 36 fields, and 15 of them
carry no description at all — including created_at and updated_at. Both
fields my metric rests on are typed but undefined: "string, date-time", with nothing
said about which moment they capture.

One field does carry a definition, and it's precisely the one I needed:
run_started_at is documented as "The start time of the latest run. Resets on
re-run." One sentence, with a warning inside it — it resets on a re-run. My window
contains no re-runs (every run_attempt is 1), so I can use it; in another
repository I couldn't use it so comfortably.

That leaves two ways to get the real duration:

• The /timing endpoint — it returns run_duration_ms per run. On my sample run it reported 224,000 ms, while the value I computed from job timestamps was 204 seconds. The two sources disagree by 10%, and both say "three and a half minutes." I don't know which is more accurate, and closing that gap isn't this article's job.
• The /jobs endpoint — the jobs' own start and finish timestamps. It survives log deletion, which makes it the one duration source the janitor doesn't touch.

There's an irony here. /timing carries this warning in GitHub's documentation:
"This endpoint is in the process of closing down." The endpoint that gives the right
number is closing; the field that feeds the wrong arithmetic stays put. And the
warning lives in prose only: the machine-readable deprecated flag in the schema is
not set. If you scan your endpoints programmatically for deprecation, you won't see
this one — I wouldn't have either, had I not read the page with my eyes.

My decision: build the duration metric on /jobs, and read updated_at only as
"when was this record last touched." The field's name was saying that all along. I
was the one asking it a different question.

The real numbers

Once the measurement was fixed, the table turned boring — in the good sense. In
fifteen days, 801 measurable runs, 19.2 hours of work in total. Median 73
seconds, mean 86 seconds, longest run 23.4 minutes.

Workflow	Total runs	Measurable	Real median
Workflow Failure Alert	327	128	7 s
Social Media Post	184	184	76 s
Daily publish pace check	127	127	7 s
Code Quality: Push on main	91	91	84 s
Push on main	91	91	197 s
Veo Prompt (Daily Short)	49	49	68 s
Deploy Only (Code, No Content)	44	44	172 s
FAQ Backfill (TR + EN)	30	30	70 s

The first row explains the 199 runs I mentioned earlier: every run made up
entirely of skipped jobs belongs to Workflow Failure Alert. That sentry ran 327 times
in the window, and in 199 of them no job ran at all — it looked at whether something
had broken, said no, and closed.

Two weeks ago I counted
what my watchman workflows give and what they cost:
two sentries that produce nothing were eating three out of every five runs. That
article asked "how many times did it run?", and the answer is the Total runs
column: 327 and 127. This table puts duration next to them: 7 seconds. Their bill is
heavy in run count and almost nothing in machine time. Reading the same data with two
different questions produces two different decisions — and both are correct.

Two wrong numbers, two different causes

Back to where I started. For the Deploy Only workflow, the median across 44 runs is
2.9 minutes, the 90th percentile is 5.1 minutes (by linear interpolation), and
the worst is 23.4 minutes. Two weeks ago the same workflow showed a median of 3.3
minutes over twenty-four runs; the two measurements agree. The number in the
docs is still ~12 min.

In that earlier article I also found where the 12 came from: it lives in two code
comments, left over from a period when the build genuinely took 8–12 minutes. It was
correct the day it was written. The build got faster; the number stayed.

That's the interesting pair. The number that went wrong by never being updated is
four times reality; the number I found by measuring is three thousand times reality.
The bad measurement produced a worse outcome than the stale guess — because the guess
carries an "approximately" next to it and binds nobody, whereas nobody would have
questioned the measurement had I put it in a table. Attach a method to a number and
the right of objection closes. That, I think, is the most expensive side effect of
measurement culture: we grant trust to the costume rather than to the figure.

Five questions I now ask before choosing a field

I wrote myself a short list after this run. Before standing up a new metric:

1. Does the field have a definition? If I'm deriving meaning from a field with no description in the schema, I may have invented that meaning. (15 of 36 fields were undefined.)
2. Who else modifies this record? Cleanup jobs, retention sweeps, re-runs. My own automations belong on that list too.
3. Is the error age-dependent? Run the same query separately against today's rows, yesterday's, and rows from a week back. Inconsistency means the data changes after the fact.
4. Is my query truncating the data? The silent ceiling on a filtered search had cut my 4,240 runs down to 1,000, and it took me half a day to notice.
5. How long will the correct source live? If I'm wiring a metric to an endpoint that is closing down, I need to note that today — otherwise in six months I'm left with a dashboard that quietly returns nothing.

The second item gave me the most trouble, because it required writing my own name in
there. The same thing happened when I wrote about
five TODOs that had been sitting for months:
the hardest component in my inventory keeps turning out to be me.

So what do I do about the janitor?

Closing the article on "lesson learned" would be easy, but it leaves the reader
empty-handed. I have three options, and I'm weighing them against the data.

The first is to stop deleting logs altogether and hand retention to the repository's
own setting. GitHub lets you set log and artifact retention between 1 and 90 days in
repository settings; that turns my job into changing a number rather than maintaining
a workflow. The open end: I don't know whether the platform's own sweep overwrites
updated_at — I didn't measure it, and I won't write what I haven't measured.

The second is to keep the cleanup and make the duration durable instead: write each
run's real duration into my own ledger when it finishes. Then an overwritten stamp
stops mattering.

The third is to leave the script alone and move the metric to /jobs — which is what
I did today. I also noted something on the side: every night the script re-sends the
DELETE call for hundreds of runs whose logs are already gone (its log says "100 runs
to clean" on all five pages). The wasted API calls cost close to nothing, but that
behaviour is exactly what keeps overwriting the stamps night after night.

Measuring is not an innocent act

For a long time I thought of measuring as passive work: the system runs, I watch from
the side, I write the number down. This run showed me the opposite. In my own
infrastructure, the ground my measurement stood on was being erased by another
automation of mine. I wrote both, both were doing their jobs correctly, and the place
where they intersect had no owner.

If nobody re-derives a metric, it rots slowly. And you only notice the rot on the day
it hands you an absurd result — I was lucky enough to get a result as absurd as six
and a half years. A milder error, say twice reality, would have sat on a dashboard
for months and produced a meeting about how "the deploy is getting slower."

What I did today isn't even two lines of code: leave one field for another, question
the filter on a query, note the side effect of a cleanup job. But the real gain was a
single sentence — I have to count my own automations as part of the system I measure,
not outside it.

Official Sources

• REST API endpoints for workflow runs — GitHub Docs: the fields of the run object, the 1,000-result limit on filtered searches, the closing-down notice on the /timing endpoint, and the definition of the log-deletion call.
• Events that trigger workflows — GitHub Docs: that the schedule event can be delayed under load, and that cron runs in UTC by default.
• github/rest-api-description: the OpenAPI description of GitHub's REST API; the field descriptions in the workflow-run schema (and the fields that have none) can be verified here.
• Managing GitHub Actions settings for a repository — GitHub Docs: retention settings for run logs and artifacts.