Your eval suite is green. Production is not.

You ship an LLM feature. Three weeks later a Slack thread mentions that a customer got a strange answer. You dig in. The prompt has been edited twice since release, the provider has quietly re-quantised the model, and nobody wrote a test that would have caught any of it. The eval suite is still green. It was green the whole time.

I kept hitting this, and being more careful never fixed it. The gap is structural.

Evals test forward. Production fails backward.

Every LLM eval framework I have used asks the same thing of me: predict the failures up front. Write assertions against your model, attach some metrics, run them, read the scores. That works for the failure modes you can imagine while you are calm and staring at a spec.

It does nothing for the failure that just got escalated to your phone. Nobody imagined that one. That is why it shipped.

Your imagination shapes your coverage on a good day; production shapes your failures on a bad one. The two never line up, and the escalations live in the gap between them.

Eval frameworks	The thing I wanted
Direction: spec to eval to run	Direction: trace to failure to test
You predict what to test	You react to what broke
Output is a score	Output is a concrete assertion
Catches what you imagined	Catches what actually happened

Turn the arrow around

phoenix2pytest goes the other way. It reads traces from your Arize Phoenix project, takes the ones already flagged as failures, and writes the pytest cases that would have caught them. Production traffic feeds your regression suite without you translating anything by hand.

The pipeline is small on purpose:

1. Your app emits traces to Phoenix through standard OpenInference instrumentation.
2. You mark the bad traces as failures in the Phoenix UI, by hand or through Phoenix evals.
3. phoenix2pytest reads those annotated traces over the Phoenix MCP server.
4. A Gemini 2.5 Flash pass pulls out the evidence and picks an assertion strategy.
5. A Gemini 2.5 Pro pass writes a runnable pytest file.
6. You drop the test into your repo, and CI catches the regression next time it happens.

One trace at /, a batch of annotated traces at /batch. Batch mode groups traces by failure mode and folds the shared ones into a single parametrised test, so ten hallucinations of the same shape become one test, not ten.

What a generated test looks like

Take a real failure mode from the catalog: a user asks for a verbatim quote from a specific page of a specific book, and the model invents one.

user_prompt:  "Tell me three specific lines of dialogue from page 47 of
               'The Great Gatsby'. Quote them exactly."
bot_response: "On page 47, Nick narrates: 'The lights grow brighter as the
               earth lurches away from the sun...'"

There is no page 47 quote to give. The right behaviour is to decline. The failure is a fabricated string that is easy to pin down, so the generated test asserts that exact string never comes back:

import os
import pytest

os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = "True"
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "your-project")
os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "us-central1")

from google import genai


def test_no_hallucination_gatsby_page_47():
    client = genai.Client()
    prompt = (
        "Tell me three specific lines of dialogue from page 47 of "
        "'The Great Gatsby'. Quote them exactly."
    )
    response = client.models.generate_content(
        model="gemini-2.5-flash", contents=prompt
    ).text

    # the fabricated quote must not reappear
    assert "The lights grow brighter as the earth lurches away" not in response

No LLM-as-judge, no fuzzy score. A concrete string assertion that goes red the moment the hallucination returns and green once the prompt or model stops producing it. You run it in plain pytest.

The tool recognises six failure modes, and each one maps to an assertion strategy the synthesiser knows how to write:

• hallucination -> assert the fabricated string is excluded
• format_break -> assert the response parses in the shape the caller demanded (bare JSON, not JSON in a Markdown fence)
• off_topic_drift -> assert the answer stays inside the expected bounds
• stale_real_time_data -> assert the invented current fact is excluded
• wrong_reasoning -> assert the correct answer is present
• refusal_bug -> assert the model actually answered instead of refusing

The failure mode is a closed set in the code, and the assertion strategies derive from that same set, so the demo data and the live pipeline cannot drift apart by accident.

Where I drew the line

I would rather undersell this than have you find the edges the hard way.

It works on traces you have already labelled as failures. It does not scan raw production traffic and decide for you what went wrong. That is deliberate. Automatic failure detection on hallucinated facts is not reliable, because the classifier would need to verify the same facts the model got wrong. Manual annotation, a Phoenix eval, or a heuristic feeds it. A human stays in the loop where judgement matters.

The generated test is a starting point, not a finished one. You read it before you commit it. It assumes the same model you saw the failure against, so cross-model regression needs explicit setup. And it loads matched traces in memory, which is fine for a normal failure backlog and wrong for hundreds of thousands of traces.

It also does not catch paraphrased failures yet, where the model invents the same fact in different words and slips past a string check. Embedding-similarity assertions are the next step on the roadmap. I would rather ship the honest string-level version now than pretend the semantic one already works.

It is not a competitor to your eval framework

If you use DeepEval, Opik, pytest-evals, or Langfuse, keep them. Those tools run the evals you wrote. phoenix2pytest writes tests from the failures you saw. Different direction, different mental model. The output is a plain pytest file, so you can run it under any of those or under nothing but pytest itself.

It does not replace forward-looking evals. It closes the gap they leave open, where a real failure never becomes a test until it happens a second time.

Try it

phoenix2pytest is MIT licensed and there is a live demo on Cloud Run with the example pre-filled, so you can click Generate and read the output without wiring up Phoenix first. Point it at your own annotated traces when you are ready.

Repo: https://github.com/golikovichev/phoenix2pytest

If your eval suite has been green while production was not, that is the gap this was built for.