DEV Community

operix
operix

Posted on

Our `*/5` GitHub Actions schedule ran about every three hours, then a YAML heredoc switched it off

By a backend engineer with ten years of experience, building small Wix and Shopify apps as operix.

One of our apps shows Google Calendar and iCal feeds on Wix sites. Visitors get a cached copy, and a scheduled job calls a refresh route so that a site nobody has visited for a while doesn't serve a stale calendar. In September that job was a GitHub Actions workflow with cron: '*/5 * * * *'. It almost never ran every five minutes, it stopped completely for a day and a half because of a mistake of ours, and the Railway cron we replaced it with had problems of its own.

Eighty runs where there should have been thousands

On 23 September we finally listed the workflow's scheduled runs with gh run list --event schedule. Between 10 and 21 September there were 80 of them, against the 3,000-plus a five-minute cron implies. The gaps between consecutive runs ranged from 102 to 353 minutes, with a median of 190.

GitHub's documentation does warn about this, in the section on the schedule event: "The schedule event can be delayed during periods of high loads of GitHub Actions workflow runs. High load times include the start of every hour. If the load is sufficiently high enough, some queued jobs may be dropped." We had read "delayed" as minutes.

We felt it in the app. Until then, the first visitor to a quiet site simply got a calendar that could be hours old. That night we added a rule: if the cached copy is more than two hours old, make the visitor wait for a fresh one. With the schedule running every three hours, that wait became the normal path for quiet sites, and a slow feed could hold a request for more than ten seconds until it failed. The next day we capped the wait at 3.5 seconds, after which the visitor gets the old copy.

We also moved the schedule off the top of the hour, to 2-57/5, which didn't help: when we measured again between 10 and 13 October, the workflow ran every three to eight hours.

The heredoc that ended the YAML block

For about a day and a half the schedule stopped completely, and that one was our fault.

At 13:00 UTC on 21 September we pushed a change that computed a build's age with a few lines of Python inside the workflow's shell step. Here it is, simplified, inside a minimal workflow:

name: refresh
on:
  schedule:
    - cron: '*/5 * * * *'
jobs:
  refresh:
    runs-on: ubuntu-latest
    steps:
      - run: |
          stamp="2026-09-21T13:00:00Z"
          age_days=$(python3 - "$stamp" <<'PY'
import sys, datetime
print(sys.argv[1])
PY
)
          echo "$age_days"
Enter fullscreen mode Exit fullscreen mode

A heredoc's closing PY has to start at column 0 of the script bash runs, and we put it, along with the Python, at column 0 of the YAML file. But run: | is a YAML block scalar, and a block scalar ends at the first line that's indented less than the block. The block ended just before import sys, and YAML tried to parse the rest of the file as YAML, which it isn't. The whole workflow file became invalid. Twenty-five minutes later we pushed a second change that broke the same rule a different way: a quoted python3 -c string whose try: and except lines started at column 0.

From the outside nothing looked broken; scheduled refreshes simply stopped. The one visible symptom was an email after every push, saying that a run of refresh.yml failed with "No jobs were run". That workflow had no push trigger at all, so a push-event run with that name meant GitHub couldn't parse the file. We only connected those emails to the schedule when someone pointed at them.

The last successful scheduled run was at 10:51 UTC on 21 September. The workflow's next run of any kind was a manual one after the fix, at 00:06 UTC on 23 September, about 37 hours later.

You can reproduce the break without GitHub. Save the file above and parse it:

ruby -ryaml -e 'YAML.load_file(".github/workflows/refresh.yml")'
Enter fullscreen mode Exit fullscreen mode

Ruby's parser stops with "could not find expected ':' while scanning a simple key at line 12 column 1", which is the import sys line. The rule that matters is that no line of a run: | block sits to the left of the block. Indenting the heredoc along with everything else works, because YAML strips the common indentation before bash sees the script. We went the other way and put both pieces of Python in quoted -c arguments, indented with the rest:

      - run: |
          stamp="2026-09-21T13:00:00Z"
          age_days=$(python3 -c 'import sys, datetime
          print(sys.argv[1])' "$stamp")
          echo "$age_days"
Enter fullscreen mode Exit fullscreen mode

Python still receives its lines starting at column 0. Parsing the file with that Ruby one-liner is now on our checklist for workflow changes, though nothing enforces it yet; add -Eutf-8:utf-8 if the file has non-ASCII comments, or Ruby may refuse to read it.

Fixed, and still not running

The fixed file parsed, and the manual run at 00:06 UTC worked. Half an hour later we pushed the file again with rewritten comments, and still no scheduled run came, though the API reported the workflow as active. At 03:27 and 03:28 UTC we pushed it twice more with comment changes. The first scheduled run arrived at 04:38 UTC, and the rest of that day's runs came at 09:30, 14:19, 18:26, 21:37 and 23:55.

We don't know whether those later pushes brought the schedule back or whether it was coming back anyway, since the one at 00:37 hadn't. We still touch the file after a fix because it's cheap, but we can't say it works.

Moving to Railway, and the cron that never got set

On 13 October we moved the job to a small Railway service that runs on a cron schedule, every 30 minutes, and kept the GitHub workflow as a backup. Our Shopify apps already ran on Railway, and we wrote about a different Railway surprise earlier this month.

It ran once, at deploy time, and not at all overnight. We had put cronSchedule in railway.json, and railway up hadn't applied it: the service's cron schedule read as empty. Setting it through Railway's public GraphQL API (serviceInstanceUpdate) worked. Since then we check two things after any change, that the service reports a next cron run time and that a log line actually appears at that time, because the one run we had seen was just the deploy starting the service.

Railway skips a run while the last one is still going

On 9 November we moved the schedule to every 5 minutes, because the app's paid plan promises a 5-minute refresh. Runs at :00 and :30 refreshed every site, and the rest refreshed only the paid ones.

Railway's documentation says what happens next: "if a previous execution is still running when the next scheduled execution is due, Railway will skip the new cron job." Our full pass covered 171 sites and took between 236 and 455 seconds. Over 30.4 hours of logs, 28 of the 61 full passes ran past five minutes, and 32 of the 61 runs due at :05 or :35 never started.

The paid-only runs had their own problem. A paid site's cache lived exactly five minutes, the same as the interval. A run that arrived a few seconds early found every cache still fresh and refreshed nothing, which happened in 8 of 32 paid-site runs over three and a half hours.

The fix, deployed on 10 November, makes every run short. Each run does the paid sites first, then one sixth of all sites. A site's slice comes from a hash of its ID, so it doesn't move when the site list changes:

export function sliceIndex(siteId: string, of: number): number {
  let h = 0x811c9dc5;                       // FNV-1a, 32 bits
  for (let i = 0; i < siteId.length; i++) {
    h ^= siteId.charCodeAt(i);
    h = Math.imul(h, 0x01000193) >>> 0;
  }
  return h % of;
}
// the cron picks slice Math.floor(utcMinutes / 5) % 6, so every slice comes back each 30 minutes
Enter fullscreen mode Exit fullscreen mode

The scheduled path also treats a cache as stale two minutes before it expires, so an early run still refreshes. Every site is visited every 30 minutes as before, and paid sites every five. From 09:55 UTC on 10 November to 00:55 UTC the next morning there were 181 five-minute slots and none of them was empty. The median run took 87 seconds and the longest 140.

Small print we've collected

  • railway logs -n 20000 returns zero lines and puts "Error in limit - Invalid input" on stderr. If your script throws stderr away, an empty log looks like a quiet night. -n 5000 works.
  • Railway's docs also say it "does not guarantee execution times to the minute", so don't build anything that needs :05 exactly.
  • On 8 November we noticed a warning in our Railway logs that railway.json config files stop working on 1 December 2026. If your cron schedule lives only in that file, move it into the service settings before then.
  • We don't know whether GitHub delays every repository's schedules the same way. Ours was a private repository with a single workflow; a public one, or one running many workflows, may see something else. If you depend on */5, run gh run list --event schedule for a week and look at the gaps.

All of this exists to keep one Wix app's calendars current, Google Calendar Events Sync. Its cron log lines now go into a report we read each morning.

Top comments (0)