DEV Community

Lex Nwimue P.
Lex Nwimue P.

Posted on

Cron Expressions, LLMs, and Investigating Crimes You Committed

One of the very few things most developers lazily evaluate is Regex (regular expressions). There are several memes about it, but they're undeniably useful when pattern matching is important. The second, in my opinion, is the cron expression. Less scary than regex, a bit more predictable, but still lazily evaluated.

Coincidentally, they're also useful for pattern matching—just in the context of time. Recurring times, if I might.

A project I've worked on for a while now relies heavily on background workers/jobs to execute several flows that contribute crucially to the overall system. In fact, there are probably five times as many jobs as there are route groupings (user routes, debit routes, etc.) in the codebase. You can already tell where this is going, because background jobs need cron expressions to work. They need to know when to run and how often to run. In other words, they need a pattern.

One such job is what we loosely call the mandate-debit-readiness job. Its responsibility is to identify mandates that are ready for debit and update their state accordingly. It was supposed to run every five minutes and had been doing so for almost three months.

Or so we thought.

Last week, we noticed that the job didn't seem to run on weekends. Mandates that should have been marked as ready weren't updated until the very first minute of Monday morning. Naturally, we started investigating.

The first thing we checked was the worker logic itself. Maybe there was some condition buried somewhere that skipped weekends. There wasn't. Nobody thought to check the cron expression initially; it took a few minutes before someone finally did.

Then we found this:

0 * * * * 1-5
Enter fullscreen mode Exit fullscreen mode

Even as someone who lazily evaluates cron expressions, one thing immediately stood out: the 1-5 at the end. It didn't feel like it should be there, and my brain instantly translated it to:

Monday through Friday.

There was even a comment beside it — something along the lines of:

every hour on weekdays
Enter fullscreen mode Exit fullscreen mode

At first glance, it looked like we'd found the culprit. The job wasn't running on weekends because we'd explicitly told it not to. We didn't notice there was nothing indicating "every 5 minutes" about this expression, so case closed.

Or so we thought.

After a quick sanity check with Gemini, we removed the 1-5 restriction, updated the comment, deployed the change.

Problem solved.

Except it wasn't.

A day or two later, it was reported that ready mandates were taking far too long to get updated. Excuse me? We literally just fixed this. The weekend issue was gone. Weekdays were never supposed to have been a problem.

Naturally, the PM started a Slack huddle, and we found ourselves knee-deep in logs again.

Debugging jobs is a fascinating experience. It's like being the detective assigned to investigate a crime that you committed.

Eventually, a pattern emerged from the logs. The job wasn't running every few minutes as expected — it was running every hour. Wonderful.

Back to the code.

And there it was. That same comment we'd previously ignored because we were too busy celebrating our fix.

every hour...
Enter fullscreen mode Exit fullscreen mode

Interesting.

I pasted the updated cron expression into Gemini again, and Gemini insisted it meant every hour. Weird. I pasted the exact same expression into ChatGPT, and ChatGPT told me it ran every minute. Now things were getting interesting.

Same expression.

Different interpretations.

I tried Claude. Claude sided with Gemini. I tried Crontab Guru. Crontab Guru also sided with Gemini.

At this point, I was less concerned about the bug and more fascinated by the fact that several systems appeared to disagree on something that felt as standardised as a cron expression. And if the expression really meant "every hour", another question remained:

How had this job been behaving correctly for months?

That mystery was becoming more interesting than the actual bug.

Then it clicked.

The issue wasn't necessarily the LLMs. It was my assumption that cron expressions are universally interpreted the same way.

Our codebase contains a mix of cron formats. Some jobs use five fields:

*/5 * * * *
Enter fullscreen mode Exit fullscreen mode

Others use six:

0 0 2 * * *
Enter fullscreen mode Exit fullscreen mode
0 0 3,22 * * *
Enter fullscreen mode Exit fullscreen mode

And once you introduce multiple cron formats into the conversation, things get surprisingly messy.

Some cron implementations use:

minute hour day-of-month month day-of-week
Enter fullscreen mode Exit fullscreen mode

Others include seconds:

second minute hour day-of-month month day-of-week
Enter fullscreen mode Exit fullscreen mode

Suddenly, an expression that looks obvious isn't quite so obvious anymore. The real source of truth isn't Gemini. It isn't ChatGPT. It isn't Claude. And it isn't Crontab Guru.

It's the actual scheduler that's parsing and executing the expression in production.

Eventually, we updated the expression to:

*/5 * * * *
Enter fullscreen mode Exit fullscreen mode

Which explicitly means:

Run every five minutes.

The behaviour aligned with our expectations, and that was the end of the incident.

Mostly.

What stuck with me wasn't the cron bug itself. It was how quickly we became confident in a theory that happened to fit the evidence we had at the time. The expression looked suspicious. The comment appeared to confirm our suspicion. Gemini agreed with us. We deployed the change and moved on, only to discover that we'd introduced a completely different problem.

In hindsight, the most interesting part of the story isn't that different LLMs produced different interpretations. It's that they had valid reasons to. Once I realised our ecosystem contained a mix of five-field and six-field cron formats, the disagreement became a lot less mysterious.

The real lesson wasn't about cron expressions at all. It was about assumptions.

Regex and cron expressions occupy a similar place in many developers' minds. We don't use them often enough to remember all the details, but we use them just often enough to feel confident that we do. Most of the time that's fine. Then one day, they become the problem.

And when they do, you find yourself investigating a crime that, more often than not, you committed yourself.

Top comments (0)