Originally published at han-co.com · Part of the "Review" strand of my Credit & Finance Data Science series.
Google released the TimesFM-3 time-series foundation model on 31 August 2026. The announcement says it ranks first among pretrained models on three benchmarks. So I tested it myself.
Two things separate this from a summary of the announcement. First, I ran it against classical statistical models and a GBM under the same conditions, on public data I chose. Second, I carved out a period the model could not possibly have trained on and compared the scores there separately. A model trained on more than a trillion time points might simply have memorized the public benchmarks, and I wanted to check.
To give the conclusion first: the zero-shot performance is real, and I found no sign of contamination. But the two new features this version leads with produced no effect on my data.
What TimesFM-3 claims
First, what changed.
It has 330 million parameters and was pretrained on more than a trillion time points. Unlike earlier versions, which were univariate-only, this one learned multivariate forecasting from the start. It alternates attention that looks only at the past along the time axis with attention that also looks across the other series along the variate axis. On top of that, you can now feed in covariates known only for the past and covariates known into the future as well. The idea is to use information you have in advance, like promotion schedules or holidays.
The decoding changed too. Instead of pulling patches out one at a time autoregressively, it generates them in a single pass. That is the reason for the speed difference we'll see later.
And there's one thing a practitioner should check first. The code is Apache-2.0, but the weights are under a separate non-commercial license, restricted to non-commercial, non-production use. I'll come back to that at the end.
How I tested it
I chose two public datasets with opposite characteristics.
The first is daily pageviews per Wikipedia article. I pulled 71 articles from January 2023 to 2 September 2026. It has a clear weekly cycle but jumps around with the news: messy data. The second is hourly temperature by city. I pulled 24 cities from January 2024 to 30 August 2026. Its 24-hour cycle repeats almost deterministically: smooth data. Both are free and need no authentication, so anyone can reproduce this.
The comparison set was seasonal naive, AutoETS, AutoARIMA, and a global LightGBM. The LightGBM used a log transform and per-series level normalization, trained on lag and calendar features: the kind of thing you'd actually build at work. To that I added TimesFM-2.5 and TimesFM-3, plus TimesFM-3 with covariates and TimesFM-3 grouped as multivariate.
Horizons were 7 and 28 days for the daily data, 24 and 168 hours for the hourly data. For metrics I looked at MASE, sMAPE, and weighted quantile loss for the probabilistic forecasts.
How three of my first conclusions got overturned
Here's an honest confession. At first I ran this with a single forecast origin. The conclusions that came out were: 3.0 and 2.5 are effectively tied, covariates actively hurt, and multivariate is a coin flip.
Then I increased the number of origins and re-ran, and the first two of those flipped. 3.0 was clearly better than 2.5, and covariates weren't harmful so much as neutral. The one-percent differences from a single origin were just that window's luck.
In my own data, AutoETS's relative performance swung by 10 percent from changing the window alone. So the final experiment moved to rolling origins: 18 for the daily data, 14 for the hourly data, plus 60 more origins for the contamination check. That accumulated roughly 22,000 series-forecasts per model, and that's where the real results start.
Result 1. The zero-shot claim is real
First, the overall scores. I set seasonal naive to 1.0 and took the geometric mean of each model's MASE ratio. Lower is better.
The TimesFM family is lowest in all four conditions, despite doing no training at all.
TimesFM-3 beat seasonal naive by 26 to 30 percent on the daily data and 14 to 26 percent on the hourly data. It beat the LightGBM, which was trained on these very series, by 16 to 29 percent. Those numbers came out of no training and no tuning whatsoever.
What stands out is how much the classical models vary. AutoETS beat naive on the Wikipedia data, but on the temperature data it was 48 percent worse than naive. TimesFM, meanwhile, was stable on both. I'd argue that consistency is this model's real strength: it saves you from having to pick a model that suits each dataset's character.
What the forecasts actually look like
Numbers alone don't give you a feel for it, so I plotted the forecast trajectories.
Right of the vertical line is the forecast window. The shaded band is TimesFM-3's 10 to 90 percent interval.
In the first panel every model catches the weekly cycle; there's no real difference. The second panel is the interesting one. The actuals climb from mid-August, and while seasonal naive and LightGBM stay at the old level, only TimesFM-3 follows the trend upward. A good share of the improvement over naive comes from situations like this.
The third panel shows the flip side, this model's weakness. It captures Tokyo's daily temperature cycle well but flattens the peaks. The actual high reached 35°C while the forecast sits around 30°C. It has a tendency to be conservative about extremes, and in a situation like a heat wave, where the tail is what matters, that becomes a problem.
Result 2. So what is statistically significant
Drawing conclusions from averages alone would repeat the mistake I made earlier. So I paired the results by series and origin and ran a paired bootstrap and a Wilcoxon test.
The horizontal lines are 95 percent confidence intervals. Left of 1.0 means the first model is better.
The top twelve rows have intervals well clear of 1.0. That TimesFM-3 beats naive and LightGBM is not up for debate.
The improvement of 3.0 over 2.5 was also significant in all four conditions. But the size is small: 3 to 5 percent as a ratio, with a per-series win rate of about 56 percent. That is, it wins a little more than half the time. Parameters went from 200 million to 330 million, and this is the accuracy gain.
Result 3. The two headline features did nothing
The heart of this version is multivariate support and covariates, so I looked at them separately.
Plain TimesFM-3 is set to 1.0. The covariate and multivariate bars barely move off 1.0.
For covariates I fed in day-of-week and a weekend flag, known into the future. Three of the four conditions were statistically meaningless, and one was significantly worse. I can guess why. The model already learns the weekly cycle from the series itself. Calendar information is redundant, and redundant inputs enter as noise.
Multivariate was the same. At first I grouped the series by topic myself, but since that might handicap the feature, I regrouped them by clustering on actual correlation. Having given it a fair chance, the result was unchanged: three conditions meaningless, one significantly worse.
This needs a caveat. The only covariates I supplied were calendar features, which the model already knows. Had they been genuinely exogenous information that isn't in the series — a promotion schedule, a news event — the story might have been different. In fact the failure case below is exactly that situation. What is clear is that feeding in calendar features alone buys you nothing.
It holds as the horizon grows
The ranking holds at both the short and the long horizon.
Everyone's error grows with the horizon, but the ranking doesn't change. That said, stretching the temperature data to 168 hours shrinks TimesFM's advantage from 26 percent to 14 percent. The further out it goes, the more it converges toward the mean.
Result 4. The real strength is speed
The x-axis is log scale. The bottom left is the fast-and-accurate corner.
This chart was the most striking one. Forecasting 71 series 28 days ahead took TimesFM-3 1.8 seconds. The same work took TimesFM-2.5 9.2 seconds, LightGBM 9.3 seconds including training, and AutoETS 18.7 seconds. AutoARIMA took 378 seconds per origin in my early runs, and performed worse than naive.
It is the most accurate and the fastest at the same time. That's the payoff from dropping autoregressive decoding. It's 5x faster than 2.5, and I'd argue that is the real improvement in this version. In practice, 5x speed matters more than 3 percent accuracy. For reference, it used only 1.3 GB on a laptop RTX 4070.
Result 5. There was no training-data contamination
This is the part I put the most work into. A model trained on a trillion time points may already have memorized the public benchmarks. If so, ranking first on a benchmark is memorization, not skill.
So I fixed the calendar window and varied only the year. I ran ten weekly origins each across late June to late August of 2024, 2025, and 2026. 2024 and 2025 are likely inside the training data; the summer of 2026 came after the model was built, so it cannot be. Fixing the window also rules out the explanation that one particular year was simply harder.
The control is LightGBM. It is retrained at every origin, so it cannot be contaminated by construction.
The shaded region is after the model's release. The TimesFM-3 line does not drop off.
The result is clear. TimesFM-3's relative advantage was slightly better in 2026, the year it could not have trained on: 2.2 percent better on Wikipedia, 1.5 percent on temperature. The LightGBM control also moved year to year (9.4 percent on Wikipedia), so year-to-year variation exists anyway, and TimesFM was steadier than that.
I found no evidence that the benchmark scores are inflated by memorization. That's a result in the model's favor, and I consider it firmer ground than the numbers in the announcement.
Looking at the distribution, not the average
Log scale. The TimesFM family sits lower across the whole distribution, not just at the median.
Mean MASE is dominated by a handful of large failures, so I looked at the distribution. The TimesFM boxes sit lower as a whole, meaning it isn't winning by luck on a few series but is broadly better. The upper whiskers, though, look like everyone else's. What it does well it does clearly well; what it does badly it does as badly as the rest.
Where it fails
I looked directly at those bad cases.
The y-axis is log scale, and the spread is still this wide.
In the forecast window, pageviews on the Google article jumped 16x, from a usual 17,000 to 285,000. That's a news event. No time-series model that only looks at the past can catch it. MASE climbing to 3.74 isn't the model being bad; it's the problem being unforecastable.
This connects back to the covariate discussion. A situation like this is exactly where exogenous information is needed. If you could feed in a genuine external signal, like news or an event calendar rather than day-of-week, the covariate feature would earn its keep. My test found no benefit under the conditions I ran, which is not the same as the feature being useless.
So, is it usable?
On performance alone it's impressive. With no training and no tuning it beats a tuned GBM, and it's 5x faster on top of that. Its consistency across very different data is a real advantage in practice.
But the weights are licensed for non-commercial, non-production use. That means you cannot put it into work at a company. As of now, the realistic uses are these: a reference line for gauging the performance ceiling in internal experiments, a quick way to measure how hard a new forecasting task is, or simply something to study and learn from.
To summarize what I confirmed:
- The zero-shot forecasting performance is real. It clearly beats classical models and GBM in both domains.
- The accuracy gain of 3.0 over 2.5 is significant but small, at 3 to 5 percent.
- The substantive improvement in this version is speed. It's 5x faster.
- The headline multivariate and covariate features produced no measurable benefit under my conditions.
- There was no sign of training-data contamination.
- Failures come from unforecastable events, which is a limit of the problem rather than of the model.
- And the license keeps it out of production work.
One last note on method. The biggest thing I learned writing this was not about the model but about validation. Two of the three conclusions I drew from a single origin flipped once I added more origins. When you read the flood of "I ran it and it was great" posts that follow every new model release, it's worth asking how many draws that conclusion came from. I nearly published mine as-is.
I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/
Top comments (1)
The peak-flattening behavior is a direct byproduct of training on symmetric patch losses like MAE or MSE. When a model minimizes expected squared error across diverse normalized series, the optimal point prediction is the conditional mean. In high-volatility regimes, that guarantee pulls the forecast away from sharp peaks to hedge against asymmetric variance penalties. For risk and capacity planning, point estimates are the wrong contract anyway; the 90th percentile interval is where the actual pricing signal lives.
The covariate result matches what happens in transformer attention generally. Temporal self-attention already reconstructs harmonic cycles directly from sequence history. Feeding explicit calendar flags into the context just adds redundant dimensions without introducing new information, so the projection heads spend capacity filtering out collinear signals.