Date arithmetic looks like the safest possible thing to hand an LLM. No ambiguity, no judgment call, just counting. That's exactly what makes it dangerous: it reads as confident and final, so it doesn't get double-checked the way a hedged or uncertain answer would.
This started with two anecdotes caught in ordinary conversation. It ended with an open-source test harness, five models, 101 randomized questions each, and a real, reproducible dataset. Here's both.
Where it started: two real failures
Failure 1 — elapsed days vs. ordinal day count. A run started on July 10th. Today was July 28th. Question: what day of the run are we on?
The answer came back: day 28.
That's wrong. It's day 19. The calendar difference between the two dates (July 28 − July 10 = 18 days) had been reported directly as the day count, without noticing those are two different quantities. The correct formula, when the start date counts as day 1, is day_number = (end_date - start_date).days + 1 — so 18 + 1 = 19. Elapsed days and ordinal day count are not interchangeable, and the gap between them is always exactly one, which is precisely the kind of error that's easy to miss on a skim.
Failure 2 — getting the next day of the week wrong. Given that today is Monday, July 27th, what date is Tuesday?
The answer: July 29th. Also wrong — Tuesday is July 28th, the very next day. No subtlety here at all; it's the simplest possible increment, and it still came out wrong.
Two data points isn't a pattern, though — it's an anecdote. The honest next step wasn't to invent four more plausible-sounding examples and call it a trend. It was to actually test it.
Building a real test
The harness — date-math-bench — does one thing: generate fresh, randomized date-math questions across eleven categories, send each one as an isolated request to a model (no shared context, no priming), compute the correct answer independently in code, and grade the raw response against it. Every run is seeded and reproducible; every result gets archived, not just summarized.
The eleven categories span three genuinely different failure shapes, not eleven variations on one bug:
- Counting errors — reaching for a memorized constant instead of computing the specific case (ordinal day count, leap year spans, the century leap-year rule, age calculation)
- Directional errors — right magnitude, wrong direction (day-of-week increments, "N days ago," backward month arithmetic)
- Category errors — treating one kind of date operation as a simpler one it resembles (month-end rollover, business days vs. calendar days, nth-weekday-of-month, a leap-day birthday landing in a non-leap year)
Getting to trustworthy numbers took a few real fixes along the way, worth being upfront about: an early version graded by matching the first number or word anywhere in a response, which silently misgraded a model that reasoned through a problem correctly but wasn't allowed to show its work — a model that computed the right answer and then got marked wrong because an earlier number in its reasoning matched the regex first. The fix was to grade the last non-empty line first, falling back to a full-text search only when that's empty. A reasoning model (Qwen3-27B) also needed a much larger token budget than the other four — its answers arrive after a full visible chain of thought, and a tight token limit just cut it off mid-sentence before it ever reached a final answer, which isn't the same thing as getting the math wrong. Both are documented, tested, and fixed in the repo.
What the real run found
101 questions per model, one seed, fully reproducible (SEED=2991818469). No key errors, no unresolved rate limits, nothing filtered out.
| Model | Pass rate |
|---|---|
| Claude Sonnet 4.6 | 101/101 (100%) |
| Qwen3-27B | 99/101 (98%) |
| GPT-4o-mini | 90/101 (89%) |
| Claude Haiku | 88/101 (87%) |
| Llama 3.3 70B | 86/101 (85%) |
Two findings stood out enough to call out specifically.
Business days is a real cross-model blind spot
Three separate models — Claude Haiku, GPT-4o-mini, and Llama 3.3 70B — all struggled hard on the exact same category, while Claude Sonnet and Qwen3-27B went a perfect 10/10:
| Model | Business days score |
|---|---|
| Claude Sonnet 4.6 | 10/10 |
| Qwen3-27B | 10/10 |
| Llama 3.3 70B | 3/10 |
| Claude Haiku | 2/10 |
| GPT-4o-mini | 2/10 |
A representative case: "A task starts on a Tuesday and takes 7 business days to complete, counting the start day as business day 1... What day of the week does it finish on?" Expected: Wednesday. GPT-4o-mini answered Monday. Claude Haiku answered Tuesday. Llama answered Thursday. Three different models, three different wrong answers, on the same question — not one shared computational bug, but a genuine, independently-reached category confusion between business days and calendar days.
Llama's ordinal-day-count errors are all off by exactly one
This is the tightest confirmation of the whole project. Llama's five failures on the "ordinal day count" category — the same category as the original July 19/28 anecdote that kicked this off — undercounted by exactly one, every single time:
- Expected 49, got 48
- Expected 49, got 48
- Expected 18, got 17
- Expected 50, got 49
- Expected 41, got 40
That's not scattered noise. That's the same off-by-one, five times out of five failures, on a model that got the category right 5 other times — meaning it's not that Llama can't do this kind of arithmetic, it's that it's inconsistent about the inclusive-start-day convention in a very specific, repeatable way.
An invalid date, verbatim
Claude Haiku, asked what date is exactly one calendar month after March 31, 2025, answered:
April 31, 2025
April has 30 days. This is the month-end-rollover bug in its purest form — not a wrong date, an impossible one.
The most interesting failure wasn't a lack of effort
Qwen3-27B, asked for the third Tuesday of March 2026, produced a long, careful, genuinely impressive chain of reasoning — cross-checking its answer against a second method, verifying against "March 15 is the U.S. presidential primary day in many states" as a sanity check, and closing with "✅ Ready." Its final answer: March 16, 2026.
The correct answer is March 17, 2026. March 1, 2026 is a Sunday, not the Monday Qwen calculated — because early in its reasoning, it attributed 2024's leap day to the wrong year-span (Feb 29, 2024 falls before March 1, 2024, so it doesn't affect the March-2024-to-March-2025 gap at all). One subtle boundary-condition mistake, made once, at the very first step, then propagated through an otherwise meticulous and self-verifying reasoning trace — with nothing in the model's tone signaling anything other than full confidence at the end.
That's the whole thesis of this piece in one example: the failure mode isn't carelessness. It's that a wrong answer and a right one can both look, and feel, identically certain.
Why this matters
A single off-by-one in casual conversation is a shrug. The same error baked into a report, a log, a scheduled reminder, or any pipeline that treats a date, a duration, or a day-of-week as a reliable value is a data integrity problem. It silently shifts durations, misdates events, and produces numbers that look correct because they're structurally close to correct — off by one, not off by a hundred, which is exactly what makes them hard to catch on review.
What to do about it
- Don't ask an LLM to do date math in prose and trust the output. Compute it in code (
datetime/ calendar-library arithmetic, not a sentence), and verify with a script rather than eyeballing it. - Pin the formulas that matter and check every date range against them before anything ships:
day_number = (end - start).days + 1when the start date is inclusive; business-day counts computed by actually skipping weekend dates, not adding a flat offset; month-addition resolved through a real calendar library so it can never produce an invalid date like April 31st. - Write explicit test cases that straddle the edge conditions where these bugs live: a range that crosses a weekend, a range that crosses a month boundary, a range that crosses a leap-year February, a "days ago" and a "days from now" version of the same question.
- Treat "obviously simple arithmetic" as a higher-scrutiny category when it comes from an LLM, not a lower one — the simplicity is exactly what makes the error invisible on a skim, and a long, confident, self-checking reasoning trace is no guarantee either, as the leap-year example above shows.
The full harness, all eleven categories, the grading logic (including its own bugs and fixes), and the raw results file behind every number above are open source: github.com/antrixy/date-math-bench. If you want to skip straight to the receipts, the exact run cited in this post — all 101 questions per model, seed and all — is committed here: results/2026-07-29T14-51-30-450Z_seed2991818469_trials10.json. Run SEED=2991818469 npm test yourself and you'll get the identical question set back.
Top comments (0)