The October 23 shutdown got the headlines. Twelve model IDs, one deadline, a lot of us grepping at midnight.
But the calendar didn't stop on the 2...
For further actions, you may consider blocking this person and/or reporting abuse
@shuttlewright Ha, "the habit the calendar itself is trying to teach" — that's a better line than anything in the article, honestly. The monthly shutdown_date diff is the bit I'd actually want people to take away: announcements get buried, a diff doesn't. And you're right about Dec 1 and Jan 20 — the chat stack gets the attention and the dashboards, while the image and voice pipelines sit in some other service nobody owns until they 400. That's exactly how those two bite.
Posting this top-level, by the way — your comment isn't rendering on the article page for me, so I couldn't reply in the thread.
That’s a nice calendar and it’s good that the shuttlewright’s audit is covered too. A little note regarding the Evals rows though; when using the dashboard’s export, you get a JSONL file with the results of the run. That’s history and can’t be executed again, and OpenAI support has stated that not all types of evaluations can be exported to Promptfoo configuration. Evals API will be available until Nov 30 and it gets you everything: all evaluation data sources and graders, all runs and graded outputs. The safest thing for you to do this week would be to use the API and then create your own tests from scratch. Here’s the API route I used for that: dev.to/enzii/your-openai-evals-fre...
That JSONL point is the one I'd missed — a history file you can't re-run isn't really a migration, it's just an archive. API before Nov 30 it is, then rebuild from scratch. Thanks for flagging the Promptfoo gap too, that's exactly the kind of thing that bites you a week later.
I’m glad I could help you out there, Paul. It was surprising that OpenAI's Promptfoo export has gaps, where the export looks like it works until you try it out. Thanks for maintaining the calendar.
Yeah, that Promptfoo gap is the sneaky kind — looks like a clean export until you try to actually run it. Thanks for flagging it, and for the calendar kind words.
Thanks - and agreed on the API over the dashboard export. Whatever the exact cutoff date turns out to be, pulling everything through the API this week is the move that does not depend on it. One addition from having done similar migrations: save the raw API responses (graders, data sources, run configs) into version control next to the tests you recreate, not just into a scratch folder. When you rebuild the evals in your own harness, the grader definitions are the part you will diff against later, and a dated raw dump is the only reliable reference for what the original actually did.
And yeah, the version-control bit — I'd have dumped it in a scratch folder and regretted it. A dated raw dump next to the rebuilt tests makes sense, the grader definitions are the part you'd actually diff later.
"Announcements get buried, a diff doesn't" - and the diff catches one more thing worth naming: the retirement that was never announced at all. A model appearing in /v1/models with shutdown_date already set is a pre-announced death with no email and no article, and only the diff sees it. On the ownership point: "nobody owns it until it 400s" is an org problem wearing a tooling costume. The durable fix is an owner per pipeline - one named person for image, one for voice, one for chat - because a dashboard everyone checks is a dashboard nobody checks. The audit finds the exposure; the name next to it is what gets it fixed.
The owner-per-pipeline line is the one I'm stealing this time. A dashboard everyone checks is a dashboard nobody checks — yeah, that's painfully true in my experience. The audit tells you where the exposure is, but without a name next to it nothing actually moves until the 400s start. And the unannounced-retirement angle is the scariest version of this, honestly — a shutdown_date sitting there quietly with no email and no article.
Correction to my wording above: OpenAI says it emails customers actively using a retiring model and documents deprecations on its deprecations page. shutdown_date is also available through the model APIs. A monthly diff is another way to track changes; it is not the only source, and I have not verified a retirement that went unannounced.
Appreciate the correction — and you're right to flag it. I ran with the 'never announced' framing a bit too eagerly in my head too, and the honest version is what you wrote: the diff is another signal, not the only one. That's a better, more careful claim, and it's the one people should actually act on.