OpenAI's hosted Evals goes read-only on 31 October: what to export and where to move

If you built evals in the OpenAI dashboard or through the Evals API, two dates matter. OpenAI's deprecations page lists:

• 31 October 2026: existing evals become read-only.
• 30 November 2026: the Evals dashboard and API are scheduled to shut down.

The same notice says the graders documented for eval workflows are part of this transition. This post covers what is affected, what to save while you still can, and the open-source tools you can move to.

What is shutting down (and what is not)

The change applies to OpenAI's hosted Evals platform: the dashboard UI and the /evals API, where you define an eval with a data source and a set of testing_criteria (graders), then launch runs against models.

It does not apply to the open-source openai/evals repository on GitHub. That repo is a separate Python framework with a registry of evals and tooling for writing your own. The names overlap, so it is worth checking which one your team actually uses before you plan anything. If your evals live in YAML files and Python classes in a repo, you are on the open-source framework. If they live in the dashboard and you call them by eval ID, you are on the hosted platform.

Why the first date matters more

OpenAI's notice says existing evals become read-only on 31 October. It does not spell out what read-only allows, so do not assume you will be able to edit or create evals after that date. The final cutoff on 30 November is when the dashboard and API are scheduled to go away entirely. Plan your export for before the first date, and treat the second as the hard stop for anything you missed.

What to export before the deadline

For each eval you care about, save four things:

1. The eval definition. This is the JSON returned when you retrieve an eval through the API: its name, data source configuration and the full testing_criteria list. The graders hold the logic you will need to rebuild, including model-grader prompts, label sets, pass thresholds and score ranges.
2. The test items. If the eval used an uploaded file, keep the JSONL. If you only have the file ID, download it now. Evals built on stored completions or logs have no item file of their own, so you will have to rebuild that dataset from your own logs.
3. The run configuration. Which model, which prompt template, which sampling parameters. Without this you cannot tell whether a new tool reproduces your old results or just produces different numbers.
4. Past results. Export run summaries and, where possible, per-item outputs and grades. These are your baseline. After migration, rerun the same items and compare. If pass rates move a lot, the cause is usually a grader that did not translate one-to-one, not a real model regression.

Store all of this in version control next to your application code. Exported evals that sit in someone's downloads folder tend to get lost.

A practical tip: list every eval through the API first and write the IDs to a file, then loop over them. It is easy to forget evals that someone created months ago and nobody opens.

Migration options

None of these is a drop-in replacement. Each has its own config format and grading model, so expect some manual work, mostly around model-graded checks.

promptfoo

OpenAI's deprecations notice links to its cookbook guide, "Moving from OpenAI Evals to Promptfoo", as the migration path. promptfoo is open source and MIT licensed, and its README notes that it is now part of OpenAI. You configure it in a promptfooconfig.yaml file with prompts, providers (models) and tests, and attach assertions to each test. Its assertion types include equals, contains and icontains for string checks, and llm-rubric for model-graded checks. The cookbook describes the migration as recreating each eval, so budget time for it.

The open-source openai/evals repo

If you want to stay close to OpenAI's original eval concepts and are comfortable with Python, the open-source framework is still on GitHub. You write evals as registry entries backed by JSONL samples and eval classes. It runs locally against whatever models you configure. Check the repo's recent activity yourself before committing to it for the long term.

Inspect

Inspect is an open-source framework for large language model evaluations, developed by the UK AI Security Institute and Meridian Labs. In Python you define a task from a dataset, a solver that produces an answer for each sample, and a scorer that evaluates the output using text comparisons, model grading or custom logic. It also has building blocks for tools and agents. It suits teams whose evals go beyond single prompt-response checks, for example tool use or agent tasks.

DeepEval

DeepEval is an open-source Python framework that its maintainers describe as similar to Pytest, but for unit testing LLM apps. You write test cases in Python and apply metrics such as G-Eval, answer relevancy and hallucination, many of which use an LLM as the judge. It fits well if your team already runs tests in CI with pytest.

Mapping graders: where the work is

Most of the effort goes into graders, not test data. Some patterns:

• String checks (equals, contains, case-insensitive contains) translate cleanly in every tool above.
• Text similarity metrics may exist under the same name but use a different implementation, so scores can differ. Re-check thresholds against your old results.
• Score-model and label-model graders become an LLM-as-judge check in the new tool. Rubric wording, score scales and the judge model all affect results, so treat these as approximations until you have compared them with your baseline.
• Python graders and multi-graders usually need to be rewritten by hand as custom assertions or scorers.

A converter for the promptfoo route

We published a small open-source script, openai-evals-migration, that takes an exported eval JSON plus its JSONL items and writes a promptfoo config and tests file. It has no dependencies beyond Node 18. It maps string checks and some similarity metrics directly, converts score-model and label-model graders to llm-rubric with a warning that they are approximations, and lists anything it cannot convert (Python, multi and endpoint graders, among others) in a migration report. It handles the mechanical part. You still need to review the judges and compare results.

A short checklist

• [ ] Confirm whether you use the hosted platform, the open-source repo, or both.
• [ ] List all hosted eval IDs.
• [ ] Export definitions, item files, run configs and results before 31 October.
• [ ] Pick a target tool and rebuild one eval end to end first.
• [ ] Rerun on the same items and compare with the exported baseline.
• [ ] Remove calls to the Evals API from CI before 30 November.

We keep a page with the dates and the link to OpenAI's notice here, and update it if OpenAI changes the schedule.

Written by the Evals Review team, which runs evalsreview.com.