What actually survives the OpenAI Evals shutdown, and what you have to rebuild by hand

Orbora — Mon, 09 Nov 2026 21:21:18 +0000

I spent this week moving evals off OpenAI Evals before the Nov 30 shutdown,
and the part that surprised me was not how much work it is. It was how much
of the work looks done when it is not.

Evals went read-only on Oct 31. The dashboard and the API go away on Nov 30.
Agent Builder and reusable prompt objects go the same day. If your baselines
still live in the dashboard, this is the checklist I wish I had on day one.

Pull the definitions through the API first, before anything else.

The export button in the dashboard gives you a JSONL of completed run
results. That is history, not a test suite. It does not hand you back a
runnable config, and OpenAI support has said so plainly. Not every eval type
can export a config at all.

The grader definitions live behind the API, and the API dies Nov 30. Pull
the definitions, the datasets, and the run history now, while they still
exist, and keep the raw JSON. Even if you rebuild everything yourself later,
this pull is the part you cannot redo in December.

Import the JSONL for what it is: history.

Promptfoo will import the dashboard JSONL, and that is worth doing. What you
get is a historical record of past runs, with the old items and their pass
and fail states preserved. What you do not get is a runnable eval. No
providers, no prompts, no assertions come across. Your new Promptfoo run is
a separate run, and it should be. Treat the import as your audit trail, not
your migration.

Rebuild each grader by hand, and sort them before you start.

This is where the real time goes, and it helps to sort graders into three
piles before touching any of them.

Straight recreates: string checks, contains and regex checks, JSON schema
checks. These map one to one onto Promptfoo assertions, and most of an
inventory is usually this pile.

Adapted graders: model-graded rubrics that have to be re-expressed, composite
graders that split into several assertions, thresholds that need re-tuning.
Expect these to take the longest per grader, and expect the numbers to move
a little. Different judge prompts and different aggregation give different
scores, even on the same outputs.

Rebuilds from scratch: multi-turn agent tests, custom code graders with side
effects, anything that depended on OpenAI internal state. For these, the
honest move is to re-author the intent in the new harness and establish a
fresh baseline, not to pretend the old numbers transferred.

Run the parity check, and write down every difference.

Rebuild the eval, then run it against the same model and the same data as
your last good run on the old platform. Compare case by case, not just in
aggregate. For each difference, write down which of these it is: a harness
difference, a grader translation difference, or a genuine output difference.

I did this on a small demo eval, ten cases, twenty one assertions, rebuilt
in Promptfoo. Eight cases passed and two failed, and the two failures were
deliberate bad replies I had seeded in: one promised a guaranteed instant
refund, the other affirmed a feature that does not exist. A grader set that
passed those would have been a failed migration. Catching them was the point.

The one translation catch in that demo was a single line. A code grader
written as a function body, return output.length < 600;, threw an error
until I re-expressed it as the bare expression output.length < 600.
Same meaning, different shape, and the failure was silent unless you read
the per-assertion output. That is the whole migration in miniature: nothing
here is hard in isolation, and it is all easy to miss.

Accept the new baseline in writing, to yourself if no one else.

Similarity scores will differ between harnesses. That is expected, not a
failure. Once you have run the parity check and classified the differences,
the new run becomes your reference baseline going forward. Future regressions
get measured against it, not against numbers from a platform that no longer
exists. If you do not make that decision explicitly, you will keep comparing
against ghosts.

Do the same inventory for Agent Builder and stored prompts.

They die the same day, and they are usually owned by the same people who
own the evals. Agent Builder workflows do not convert cleanly into code,
and stored prompts have no export API. Inventory them in the same pass,
while the API is still up, or they become a second emergency on Nov 29.

That is the whole list. The deadline supplies the urgency, so the work does
not need any. Start with the API pull. Everything else can be scheduled
after it.

DEV Community: Orbora

What actually survives the OpenAI Evals shutdown, and what you have to rebuild by hand