Monitoring AI Quality: Evaluation Pipelines for Non-Deterministic Products

Shipping an AI feature is the easy part. Knowing that it still works next month is the hard one.

In a traditional application, the same input always gives you the same output. That's why tests work so well: you write expect(sum(2, 2)).toBe(4) once, and it protects you forever. AI features break that contract. Send the same input to an LLM twice and you get two different answers. Both may be good. One may be a bit worse. And one day, after a prompt tweak or a silent model update, all of them may be wrong, with no error in the logs.

In this article we'll walk through how to monitor the quality of an AI feature: a pipeline that generates real outputs, asks a second model to grade them, and tells the team in Slack when quality drops. Then we'll go one step further and run it on a schedule with cron jobs, so you also catch the problems that don't come from your own deploys.

How to read this article: if you're on the product or leadership side, the first sections explain the problem and what to measure, with no code required. If you're a developer, the implementation and the cron examples are in "Building the Pipeline" and "Beyond CI".

To keep things concrete, we'll use one example throughout the article: a meeting assistant. It reads the transcript of a recorded call and writes a structured summary: key decisions, action items with owners, and highlights tied to moments in the recording. Each team picks a summary template (sales call, standup, job interview, customer support and so on), so the same call should produce a different summary depending on that choice. That's a lot of ways to be almost right.

Why Should Product Leaders Care?

With AI, quality is not a value, it's a distribution. Your feature doesn't produce "the right answer"; it produces a range of answers, and most of them are good. The job is to keep that range where you want it.

Think of it like a restaurant chain. The same dish is cooked by different cooks, in different kitchens, with ingredients from different suppliers. Nobody tastes every plate, but someone tastes a sample against the recipe on a regular schedule. That's what an evaluation pipeline does for an AI feature.

The tricky part is that quality can change even when nobody touches the code. There are three common sources of change:

• Your prompts: the team improves a prompt for one case and, without noticing, makes another case worse.
• Your model provider: models get updated, deprecated and replaced. A new version can be better on average and still worse for your specific task.
• Your inputs: customers start using the feature in ways you didn't plan, with longer calls, other languages or new templates.

None of these throws an error. The output just gets a little worse, and your users notice before you do. For an AI feature, that's the real cost: not a crash, but trust. A feature that is great 90% of the time and strange the other 10% teaches users to double-check everything, and then it stops saving them time.

An evaluation pipeline gives you three things:

1. A number you can track. "Is the AI good?" becomes "is accuracy still above the level we agreed on?".
2. A gate before release. A prompt change that lowers quality is caught on staging, not by a customer.
3. An early warning. A scheduled run catches the changes that come from outside your team.

How Do You Test Something That Never Gives the Same Answer?

You stop checking for the answer and start checking for properties of a good answer. Instead of expect(output).toBe(...), you ask: is it well-formed? Does it cover the whole call? Is it faithful to the transcript? Each question needs a different kind of check, and they get more expensive as they get smarter.

Check	What it answers	Cost	Example
Deterministic	Is the output well-formed?	Free, instant	The summary is valid JSON with every required section
Metric (code)	Does it have a measurable property?	Cheap	Highlights are spread across the whole recording
LLM-as-a-judge	Is it accurate, complete and useful?	One extra LLM call per sample	A 1 to 100 score for each metric you define, like accuracy or clarity

All of them run on top of the same foundation:

1. A Golden Dataset

A golden dataset is a fixed set of realistic inputs that represent how your customers use the feature. For our meeting assistant, that could be 20 fixtures: 15 call transcripts and 5 written meeting notes, covering every summary template. A second set of 5 timestamped recordings is used for the coverage check. Use synthetic or anonymized data, never raw customer content.

Because the inputs never change, the scores are comparable over time. If last week's average was higher than today's, something on our side or the provider's side changed.

2. Metric Checks

Some qualities can be measured with plain code, no AI needed. A good one for our example is coverage. If the summary only cites moments from the first ten minutes of a one-hour call, the reader gets half the picture. So we split the recording into bins of about four minutes and measure how evenly the summary's timestamps spread across them. A score of 0 means everything is bunched in one place; 1 means perfectly even.

3. LLM-as-a-Judge

For the things code can't measure, like "was this decision really made in the call?", you use a second model as a judge. It receives the source, the generated output and a rubric, and grades the output.

The most important part is the rubric, and it should be yours. Start by writing down the metrics that define a good output for your product. For our meeting assistant, they could be:

• Accuracy: every decision and action item really happened in the call.
• Completeness: no decision or action item was left out.
• Owners: every action item has a person responsible for it.
• Clarity: someone who missed the meeting understands the summary.

A support bot would have different ones, like tone, policy compliance or whether the issue was solved. Then ask the judge to return a score from 1 to 100 for each metric, with a one-sentence reason for each score. Run it with temperature 0 to keep it as stable as possible.

Why a separate score per metric instead of one grade? Because it lets you decide what "good enough" means for each metric on its own, as we'll see in the thresholds.

The judge is also an LLM, so treat its scores as measurements with some noise, like a thermometer that can be off by a degree. Before trusting the numbers, compare a few judge scores with a human expert's opinion, and adjust the rubric until they mostly agree.

4. Thresholds

A score is only useful when it's compared against something. With a 1 to 100 score per metric, you set a threshold for each metric, based on how much that metric matters to your product:

Metric	Threshold	Why
Accuracy	90	An invented decision destroys trust
Owners	85	Action items without an owner don't get done
Completeness	80	A missed item is bad, but easy to add by hand
Clarity	70	Style matters, but less than the facts

These numbers are only an example. Choosing them is a product decision, not an engineering one: a healthcare product may want accuracy at 98, while a brainstorming tool may be fine with 70. Together, the thresholds are the quality level you expect from the feature, written down.

There are two ways to apply them, and you'll probably want both:

• On the average across the golden set, for the run as a whole. One strange sample doesn't fail the run, but a general drop does.
• On each sample, for the metrics where a single bad output is unacceptable. One invented decision may be enough to fail the run.

Code metrics, like coverage, work the same way: decide how even the spread must be for your product, and fail the run below that.

A natural next step for consistency is to run the same input several times and measure the spread of the scores. A feature with an average of 80 and a small spread is a better product than one with the same average that swings between 60 and 95.

Building the Pipeline

The pipeline itself doesn't need to be complex. It's a script that runs the golden set against a real environment, calling the same endpoints the product uses, so it tests the whole stack at once: prompts, model, parsing and response format. Run it as a CI job, like a GitHub Action, every time something that can change quality changes: a prompt, a model version or the code around them.

![Evaluation pipeline: a prompt or model change, or a scheduled run, runs the golden set; a rubric check and a coverage check run in parallel; the results go to the job summary and to a Slack message that is green only when both checks pass]

The two checks run side by side, and a failure in either one turns the Slack message red.

Generate, Then Grade

The core of the pipeline is a Python script, run_evals.py. For each fixture it makes two calls to OpenAI: one to generate the summary with the same prompt and model the product uses, and one to ask the judge to grade it. Here is a simplified version:

import asyncio
import sys

from openai import AsyncOpenAI

client = AsyncOpenAI()  # reads OPENAI_API_KEY from the environment

MODEL = "gpt-6-luna"  # the same model the product uses
THRESHOLDS = {"accuracy": 90, "owners": 85, "completeness": 80, "clarity": 70}


async def generate_summary(source):
    # Step 1: generate the summary with the product's own prompt
    response = await client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SUMMARY_PROMPT},
            {"role": "user", "content": source},
        ],
    )
    return response.choices[0].message.content


async def judge_summary(source, summary):
    # Step 2: ask the judge for a 1-100 score per metric
    response = await client.chat.completions.create(
        model=MODEL,
        reasoning_effort="none",  # temperature only works with reasoning off
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content": f"Source:\n{source}\n\nSummary:\n{summary}"},
        ],
    )
    # JudgeResult validates the answer, we'll see it in the next section
    return JudgeResult.model_validate_json(response.choices[0].message.content)


async def process_fixture(fixture):
    source = fixture["source"]  # a transcript or written notes
    summary = await generate_summary(source)
    return await judge_summary(source, summary)


async def run_all():
    fixtures = load_fixtures()
    results = await asyncio.gather(*(process_fixture(f) for f in fixtures))

    failed = []
    for metric, threshold in THRESHOLDS.items():
        average = sum(result.scores[metric] for result in results) / len(results)
        status = "PASS" if average >= threshold else "FAIL"
        print(f"{metric}: {average:.1f} (threshold {threshold}) {status}")
        if average < threshold:
            failed.append(metric)

    sys.exit(1 if failed else 0)


asyncio.run(run_all())

If you read our article on async tasks in arrays, asyncio.gather will look familiar: it's Python's Promise.all. With the SDK's async client, all fixtures run in parallel. LLM calls are slow, so running them one by one would make the pipeline far slower.

SUMMARY_PROMPT and MODEL should be exactly what the product uses, and JUDGE_PROMPT is the rubric from the next section. To keep the example short, step 1 calls OpenAI directly. If your feature runs behind your own API, call that API instead, so each run also tests what's around the model, like how the prompt is built and how the answer is parsed.

A complete version should also keep failed requests apart from low scores. A request that errors is reported by name, and any failure fails the run, because "the AI service returned a 500" and "the AI got worse" are different problems.

Asking the Judge

The judge's prompt lists your metrics and asks for a strict JSON answer. A trimmed version for our example:

You are evaluating a meeting summary against the call transcript.
Score each metric from 1 to 100 and explain each score in one sentence.

- accuracy: every decision and action item really happened in the call
- completeness: no decision or action item was left out
- owners: every action item has a person responsible for it
- clarity: someone who missed the meeting understands the summary

Answer only with JSON:
{"scores": {"accuracy": <1-100>, ...}, "reasons": {"accuracy": "<one sentence>", ...}}

Validate the answer before using it. The judge is an LLM too, so sometimes it returns a score out of range, or something that isn't JSON at all. With Pydantic, that becomes an error instead of a made-up number:

from typing import Annotated

from pydantic import BaseModel, Field

Score = Annotated[int, Field(ge=1, le=100)]


class JudgeResult(BaseModel):
    scores: dict[str, Score]  # one 1-100 score per metric
    reasons: dict[str, str]  # one sentence per metric

A score per metric tells you what kind of problem you have. Low accuracy usually means the AI is inventing things that didn't happen in the call. Low completeness means items were skipped. Low clarity means the summary is vague. Each one points to a different fix, and the reasons tell you where to look.

Version the judge's prompt too, just like the prompts it evaluates; a prompt management tool like Langfuse makes that easy. And if you want one headline number for a dashboard, the average of your metrics works well.

Reporting Where People Look

A check that nobody sees is a check that doesn't exist. Every run publishes its results in three places:

1. The CI job summary, with each metric's score next to its threshold.
2. Artifacts with every generated output. A score tells you something is wrong; the output tells you what.
3. A Slack message, green or red, so the whole team sees the result without opening the CI.

What Usually Goes Wrong

Evaluation pipelines are not immune to the problem they solve. They call slow, non-deterministic services, so they time out, the judge occasionally returns something unparseable, and a deploy may not be live yet when the run starts. When that happens often enough, teams tend to switch the automatic run to manual, and a manual check is one that people forget to run.

The lesson: separate "the pipeline broke" from "quality dropped". They need different alerts and often different owners. If every red message might be noise, people stop reading them, and then the pipeline protects nobody.

Beyond CI: Running Evaluations on a Schedule

A pipeline that runs after a deploy answers one question: did we break something? But remember the three sources of change. Two of them, the model provider and your customers' inputs, never come with a deploy. A model can change on a Tuesday afternoon while your team is in a meeting. To catch that, the evaluation has to run on a clock, not on a commit.

There are two kinds of scheduled runs, and they answer different questions:

Scheduled run	Inputs	Question it answers
Golden set on a cron	The same fixtures every time	Did something change under us?
Production sampling	A random sample of real outputs	How good is what customers actually get?

The golden set is your control group: the inputs never change, so any change in the score comes from the system. Production sampling is your reality check: the inputs change every day, and that's the point.

Option 1: The Golden Set on a Cron

The easiest step is to reuse what you already have. Take the same validation script and add a schedule trigger:

on:
  schedule:
    - cron: "0 11 * * 1-5" # weekdays at 11:00 UTC
  workflow_dispatch:

A few things to keep in mind with scheduled GitHub workflows:

• The cron expression is always in UTC.
• Scheduled runs only start from the default branch, and they can be delayed when GitHub is under heavy load.
• In public repositories, scheduled workflows are disabled after 60 days without activity in the repository.
• If you point it at production, run the fixtures under a dedicated test account or tenant, so evaluation data never mixes with real customer data and is easy to filter out.

Option 2: Sampling Production With a Cron Job

The golden set tells you if the system changed. It doesn't tell you how the AI is doing with the inputs customers send today. For that, you sample real outputs and grade them with the same judge.

Here's what that job can look like in Node, using node-cron for the schedule and p-map to limit how many judge calls run at once:

import cron from 'node-cron';
import pMap from 'p-map';

const SAMPLE_SIZE = 20;
const MAX_DROP = 5; // points below the 7-day average
const SIX_HOURS = 6 * 60 * 60 * 1000;

const average = (values) =>
  values.reduce((sum, value) => sum + value, 0) / values.length;

cron.schedule('0 */6 * * *', async () => {
  // 1. Sample real outputs generated since the last run
  const samples = await AiOutputs.aggregate([
    { $match: { createdAt: { $gte: new Date(Date.now() - SIX_HOURS) } } },
    { $sample: { size: SAMPLE_SIZE } },
  ]).toArray();

  if (!samples.length) return;

  // 2. Grade them with the same judge and metrics used in CI, 4 at a time
  const results = await pMap(samples, (sample) => judge(sample), {
    concurrency: 4,
  });

  // 3. Store the run, so we can see trends and not only pass/fail
  const metrics = Object.keys(results[0].scores);
  const averages = Object.fromEntries(
    metrics.map((metric) => [
      metric,
      average(results.map((result) => result.scores[metric])),
    ])
  );
  await EvalRuns.insertOne({
    createdAt: new Date(),
    averages,
    promptVersion: samples[0].promptVersion,
    model: samples[0].model,
    scores: results.map(({ outputId, scores }) => ({ outputId, scores })),
  });

  // 4. Alert on any metric that dropped below its recent baseline
  const baseline = await getRollingAverages({ days: 7 });
  const dropped = metrics.filter(
    (metric) => averages[metric] < baseline[metric] - MAX_DROP
  );
  if (dropped.length) {
    await notifySlack(
      dropped
        .map(
          (metric) =>
            `${metric}: ${averages[metric].toFixed(1)} vs 7-day average ${baseline[metric].toFixed(1)}`
        )
        .join('\n')
    );
  }
});

AiOutputs, EvalRuns, judge, getRollingAverages and notifySlack stand for your own collections and helpers. The shape is what matters: sample, grade, store, compare.

Some points that are easy to miss:

• Run it once per cluster. If your app runs on several instances, node-cron fires on every one of them, and you pay for the same evaluation several times. Use a scheduler that runs once, like a Kubernetes CronJob, or a package with a shared lock, like SyncedCron in Meteor.
• Privacy comes first. Production sampling sends real customer data to the judge. Keep the judge inside the same compliance boundary as the feature itself (same provider, same data agreements) and store scores and IDs, not copies of the content. This matters even more with health, financial or children's data.
• Sample size is your cost control. Each sample is one more LLM call. Twenty samples every six hours is 80 judge calls a day; pick a number your budget can live with.
• Store the versions. Saving the prompt version and the model with each run is what lets you answer the first question everyone asks when the score drops: what changed?

Alert on Trends, Not Only Thresholds

A fixed threshold catches big drops. But if accuracy goes from 97 to 91, it still passes a threshold of 90, and something clearly got worse. Comparing each run against a rolling baseline, like the 7-day average in the example above, catches the slow slides that a fixed number misses.

And remember the lesson from the previous section: give "the job failed to run" its own alert. A monitor that fails silently looks exactly like a monitor that says everything is fine.

What to Track

One number is enough to start, but a few more turn a red light into a diagnosis. These are the metrics we find useful, and who usually reads each one:

Metric	What it tells you	Who reads it
Average judge score	Overall quality, the headline number	Product, leadership
Score vs. 7-day baseline	Whether quality is sliding, even above the threshold	Product, leadership
Score per scenario (e.g. per summary template)	Where the AI is weak, and which customers feel it	Product
Score per metric (e.g. accuracy, clarity)	What kind of problem to fix	Engineering
Spread across repeated runs	Consistency: how much the answer varies for the same input	Product, engineering
Coverage and other code metrics	Measurable properties the judge can miss	Engineering
Latency per call	How long users wait for the AI	Product, engineering
Failed runs and request errors	Whether the monitor itself is healthy	Engineering
Tokens per run	What the feature costs, and what monitoring it costs	Leadership

Put the numbers where people already look. For leadership, that can be a single daily line in Slack: the average, the trend against the baseline, passed or failed. For engineers, it's the job summary with each metric's score and a link to the generated outputs. The goal is that nobody has to remember to go check a dashboard.

Conclusion

AI features don't fail like regular code. They don't crash; they drift. A prompt change, a model update or a new kind of input can make the output a little worse, and no log will tell you. The only way to know is to keep measuring.

The good news is that the tools for this are the ones you already use: a test script, a CI workflow, a cron job and a Slack channel. What changes is what you assert. Not the exact answer, but the properties of a good one: well-formed, complete, faithful to the source, and good enough by a rubric that your team agreed on.

If you want to start this week, here's the path we'd follow again:

1. Collect 10 to 20 realistic, anonymized inputs that cover your main use cases.
2. Add the cheap checks first: valid format and required fields.
3. Write down the metrics that define "good" for your product, and ask a judge for a 1 to 100 score on each.
4. Run it once, read the scores next to the outputs, and set a threshold for each metric from that baseline.
5. Run it after every deploy that touches prompts or models.
6. Run it on a schedule, and sample production when you're ready.
7. Send the results where people already look, and store every run to see the trend.

Your AI feature will never give the same answer twice. With an evaluation pipeline watching it, it can still give the same quality.