DEV Community: Norman Niemer

How to cache intermediate DataFrames in Python (without brittle pickle files)

Norman Niemer — Wed, 02 Sep 2026 13:00:00 +0000

Why features_v3_final.pkl keeps burning you — and how to make cached reuse something you can actually trust.

You know the loop. You load raw data, build features, fit a model, look at the result, tweak one line, and run the whole thing again — waiting several minutes for load_raw and build_features to recompute output that is byte-for-byte identical to last time. So you start saving intermediate DataFrames to disk. A few weeks later your folder is a graveyard: features.pkl, features_v2.pkl, features_v3_final.pkl, features_v3_final_ACTUAL.pkl. Nobody remembers which one is current, and worse — you can't trust any of them.

That last part is the real cost. A caching layer you can't trust is worse than no cache at all, because it quietly serves you the wrong answer. This post is about how to cache intermediate DataFrames in Python so that reuse is safe — every cached value is guaranteed to match the code and parameters that produced it, or it gets rebuilt.

oryxflow is a small, local-first, zero-infrastructure Python library that turns your data-science script into dependency-aware tasks, so every value on disk is tied to the code, inputs, and parameters that produced it — and a step rebuilds the moment any of those change. Safe reuse is the result of that, not a separate feature.

Why hand-rolled DataFrame caches fail

The instinct is reasonable. You wrap an expensive step in an existence check:

import os
import pandas as pd

if os.path.exists('features.pkl'):
    features = pd.read_pickle('features.pkl')
else:
    features = build_features(load_raw())
    features.to_pickle('features.pkl')

This works for exactly one afternoon. Then it starts lying to you, in three distinct ways:

1. No code-awareness → silent stale results. You edit build_features to add a column or fix a bug. The file features.pkl still exists, so the branch above loads the old DataFrame and skips your new code entirely. Nothing errors. Your model trains on features that no longer match the function that supposedly built them. This is the bug that ends with a wrong number in a report, and it's nearly impossible to spot by reading the script — the code looks right; the cache is what's wrong.

2. Manual path bookkeeping → collisions and typos. Every cached step needs its own filename, and you assign them by hand. Two experiments both write features.pkl and clobber each other. You typo features.pkl in the read but not the write, so it silently recomputes forever. You copy a block, forget to rename the file, and now two steps share one cache. The _v3_final naming scheme is what path bookkeeping looks like once it has failed a few times.

3. No parameter-awareness. The moment you add a knob — window=30 vs window=60, model='ols' vs model='gbm' — a single features.pkl can't represent both. You either recompute every time (defeating the cache) or hand-roll features_window30.pkl filenames and get right back to problem #2.

Each fix breeds more bookkeeping, and none of them fix the dangerous one — #1 — because a filename simply doesn't know anything about the code that wrote it.

The fix: cache by task identity, not by filename

The way out is to stop keying your cache on a filename you chose and start keying it on the identity of the work — the task's code, its inputs, and its parameters. If any of those change, the identity changes, and the cached value is rebuilt. If none change, the value is reused. You never name a file.

That's what oryxflow does. You declare each step as a Task: what it depends on, and what it produces. The output format follows the base class you inherit — TaskPqPandas persists a DataFrame as Parquet, TaskPickle persists a fitted model, TaskCSVPandas writes CSV — so there's no serialization code and no path to manage.

import oryxflow

class LoadRaw(oryxflow.tasks.TaskPqPandas):        # DataFrame -> Parquet, automatically
    def run(self):
        self.save(load_raw())                       # no filename, ever

@oryxflow.requires(LoadRaw)                          # declares the dependency + copies params
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                        # loads LoadRaw's cached DataFrame
        self.save(build_features(df))

@oryxflow.requires(BuildFeatures)
class FitModel(oryxflow.tasks.TaskPickle):           # model object -> pickle, automatically
    def run(self):
        features = self.inputLoad()
        model = fit(features)
        self.save(model)
        self.saveMeta({'n_rows': len(features)})     # small run metadata alongside it

flow = oryxflow.Workflow(FitModel)
flow.run()
model = flow.outputLoad()

Run it once and all three steps execute. Run it again and every step is served from its cached output — no recomputation, no if os.path.exists, no .pkl you had to name. The cache is addressed by task identity, so it can't collide with another step and can't be loaded under the wrong name.

Because identity includes parameters, sweeping a knob just works. Give BuildFeatures a window = oryxflow.IntParameter(default=30) and each value gets its own cached output automatically — window=30 and window=60 coexist without you inventing two filenames, and re-running either one reuses whichever upstream work it shares.

Editing a step invalidates exactly what changed

Here's the property that hand-rolled caches can't offer, and the reason reuse is safe: oryxflow tracks each task's code — and the helper files it imports — comparing what your code does, not how it's written. When you edit the body of build_features, oryxflow sees the change, so BuildFeatures and everything downstream of it (FitModel) are marked stale and rerun on the next flow.run(). The expensive LoadRaw step upstream is untouched and stays cached.

Crucially, what counts is logic, not text. Rename a local variable for clarity, reflow a line, or add a comment, and nothing reruns — a cosmetic edit isn't a behavior change, so oryxflow doesn't waste your time rebuilding for it. Change an actual computation and the affected band of the DAG rebuilds automatically. You get the reuse of a cache with the correctness of a from-scratch run, and you never manually delete a .pkl to force a refresh again.

Every decision is recorded to a local lineage log (.oryxflow/events.jsonl), so you can answer "why did this rerun?" after the fact. There's no server, no database, no account, and no telemetry — it's all local files on your machine.

One honest caveat: oryxflow guarantees your cache is reproducible, not that your logic is right. It reruns whenever the code changes, so what you get always matches the code that's on disk. Whether that code is correct is still your job — but at least you'll never again be debugging a result that came from a version of the code you already deleted.

Hand-rolled pickle cache vs oryxflow

Hand-rolled .pkl cache oryxflow task
Trustworthy reuse Only if you never make a mistake Reproducible by construction
Edit the step's code Stale file silently reused Rebuilds that step + downstream
Cosmetic edit (comment/rename) Rebuild only if you remember to delete the file No rerun — logic unchanged
Filenames / paths You name and track every one None — cache keyed by task identity
Different parameters One file, or hand-rolled suffixes Separate cached output per value
Choosing the format to_pickle / to_parquet by hand Follows the base class (Parquet, CSV, pickle)">

Takeaway

Caching intermediate DataFrames by filename fails because a filename knows nothing about the code that wrote it — so it happily hands you stale features after you've changed the logic. Cache by task identity instead: let each step's code, inputs, and parameters define what "already done" means, and reuse becomes something you can actually trust. Edit a step and exactly what changed reruns; edit a comment and nothing does. The speed is the side effect — what you're really buying is the ability to believe the DataFrame you just loaded.

pip install oryxflow

Frequently asked questions

How do I cache intermediate DataFrames in Python without brittle pickle files?

Hand-rolled pickle caches key on a filename you chose, which knows nothing about the code that wrote it, so they silently serve stale DataFrames after you edit the logic. Cache by task identity instead: key each cached value on the step's code, inputs, and parameters. oryxflow does this in plain Python, persisting a DataFrame as Parquet automatically and rebuilding it whenever the code or parameters change, so you never name or track a .pkl.

How do I know a cached DataFrame isn't stale?

You can't know it from a file, because a filename records nothing about the code that wrote it — which is why a hand-rolled cache eventually hands you features that no longer match the function that supposedly built them. Make the cache key the identity of the work instead: the step's code, its inputs, and its parameters. oryxflow does this and rebuilds a step the moment any of the three change, so a stale value is never served and you never have to delete a file to force a refresh.

How do I avoid recomputing a DataFrame every run?

Wrap each expensive step as a task that declares its dependencies and saves its output, so an engine can reuse the previous result when nothing changed and rebuild it when something did. oryxflow, a local-first zero-infrastructure Python library, caches each intermediate DataFrame by task identity and reruns a step only when its code, inputs, or parameters actually change — so an unchanged step turns a multi-minute rerun into a load from disk, and a changed one can't be missed.

Do you need MLflow, or do you need a reproducible pipeline?

Norman Niemer — Tue, 01 Sep 2026 13:00:00 +0000

They sound like the same problem. They're not — and mixing them up is why so many ML projects still can't reproduce last week's result.

If you've searched for "manage machine learning experiments" you've been pointed at MLflow, Weights & Biases, and DVC. They're excellent tools. But a lot of people install one, log some metrics, and are surprised to find their pipeline is still a mess of stale pickle files they can't reliably reproduce. That's because these tools solve a different half of the problem than the one biting you.

Two different problems that both get called "experiment management"

Problem A — tracking: "Which run got 0.91 AUC, and what were its hyperparameters?" This is a logging and comparison problem. It's what MLflow, W&B, and Neptune are built for: you call log_metric(...), log_param(...), and get a searchable dashboard of every run.

Problem B — computation: "To reproduce that 0.91 run, which steps do I actually need to rerun, and which are already computed?" This is a dependency and caching problem. It's about not recomputing a 10-minute feature step when only the model changed, and about guaranteeing the model you're evaluating was trained on the current data.

Trackers answer A. They do not answer B. MLflow will faithfully log that you got 0.91 — it has no idea whether the features feeding that model are stale, and it won't skip recomputing them for you. That gap is where reproducibility quietly dies.

What a pipeline-caching engine does that a tracker doesn't

A workflow engine like oryxflow models your work as a DAG of tasks and owns the computation side:

import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_data())

@oryxflow.requires(GetData)
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(build_features(self.inputLoad()))

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='gbm')

    def run(self):
        features = self.inputLoad()
        clf = fit(self.model, features)
        self.save(clf)
        self.saveMeta({'score': clf.score(...)})   # <-- log to MLflow here too

oryxflow.run(TrainModel())

From this it gives you three things a tracker structurally cannot:

Skip-what's-done. Rerun the script and completed tasks load from cache instead of recomputing. Change one task and only its downstream reruns.
Correct-by-construction invalidation. Change a parameter, the data, or a task's code, and exactly the affected outputs are marked stale and rebuilt. You can't accidentally evaluate a new model on old features.
Load-any-result-by-name. TrainModel().output().load() gives you the model; BuildFeatures().output().load() gives you the features — no hunting for .pkl paths.

The honest answer: use both

This isn't "oryxflow vs MLflow." The two compose cleanly:

def run(self):
    features = self.inputLoad()
    clf = fit(self.model, features)
    self.save(clf)                         # oryxflow: caches + invalidates
    mlflow.log_param('model', self.model)  # MLflow: dashboard + comparison
    mlflow.log_metric('score', clf.score(...))

oryxflow owns the pipeline: dependency order, caching, minimal reruns, reproducibility.
MLflow / W&B own the record: the searchable history of what each run scored.

Put the tracker calls inside your oryxflow tasks and you get both a reproducible computation graph and a clean experiment log — without either tool pretending to be the other.

What about DVC?

DVC is the tool people most often conflate with this space, because it does do pipeline caching. The honest difference is what identity is built from. DVC hashes files and YAML-declared stages: you describe your pipeline in dvc.yaml — each stage's command, dependencies, and outputs — and DVC recomputes a stage when a declared file hash changes. oryxflow's identity is native Python task identity — parameters plus automatic code-change detection, zero config files: the DAG is your requires() methods, a parameter change is automatically a new cached identity (no stage file to edit), and a code change reruns the task and everything downstream on its own. It compares what your code does, not how it's written, so comment and formatting edits never recompute (pin a task with code_version when you'd rather manage it by deliberate bumps). If your workflow is command-line stages over large versioned data files, DVC's file-hash model fits. If your workflow is Python tasks you iterate on inside a session — parameter sweeps, per-entity fan-outs — keeping identity in the code you're already editing beats maintaining a parallel YAML description of it.

So which do you actually need?

You can already reproduce runs but can't compare them → you want a tracker (MLflow/W&B).
You can log runs but rerunning your pipeline is slow, fragile, and you're never sure what's stale → you want a caching workflow engine (oryxflow, or its heavier cousins Luigi, Metaflow, Kedro).
Most real projects want both.

If it's the second problem you feel every day — the fifteen-minute edit-rerun loop, the features_v3_final.pkl graveyard, the "wait, was this trained on the new data?" — start here:

pip install oryxflow

Docs: https://docs.oryxflow.dev · Source: https://github.com/oryxintel/oryxflow

oryxflow is a lightweight, dependency-free alternative to Luigi, Metaflow, and Kedro, focused on research iteration rather than production orchestration — and it plays nicely with whatever tracker you already use.

Frequently asked questions

Do I need MLflow, or just pipeline caching?

It depends on which problem bites you. If you can already reproduce runs but can't compare them, you want a tracker like MLflow. If you can log runs but rerunning your pipeline is slow, fragile, and you're never sure what's stale, you want a caching workflow engine. Most real projects want both — oryxflow caches the computation while MLflow records the results.

What's the difference between MLflow and pipeline caching?

MLflow answers tracking — which run got which metric with which parameters — and gives you a searchable dashboard. Pipeline caching answers computation — which steps must actually rerun, and which are already computed — so you never evaluate a model on stale data, and never recompute an unchanged feature step either. A tracker records what happened; a caching engine like oryxflow makes the pipeline behind it reproducible.

Can I use MLflow and oryxflow together?

Yes, and that's the recommended pattern. Put your tracker's logging calls inside your cached oryxflow tasks: oryxflow owns the pipeline — dependency order, caching, minimal reruns, reproducibility — while MLflow owns the record of what each run scored. You get a reproducible computation graph and a clean experiment log, without either tool pretending to be the other.

Cheap, durable LLM evals: pydantic-evals + oryxflow

Norman Niemer — Thu, 27 Aug 2026 17:00:00 +0000

An eval matrix is a Cartesian product. Metered platforms bill every cell, every time — then delete the answers.

The first calculation looks fine

You're evaluating an intent classifier — does this query need a web search or not. 200 labeled cases. You wire up Braintrust, run it, and it's free: Starter gives you 1 GB of processed data and 10,000 scores per month, and you used 200.

So you do it properly. A real eval isn't one scorer, it's five: correct label, per-class assertion, output format valid, latency under threshold, and an LLM judge on the borderline reasoning. Now each config costs 200 × 5 = 1,000 scores.

Then you ask the question you actually care about — is the lite model good enough? Four models. And one run per case is noise, so three reps each:

200 cases × 5 scorers × 4 models × 3 reps =12,000 scores

That's one afternoon. Your monthly free allowance was 10,000.

The overage is trivial — $2.50 per 1,000 additional scores on Starter, so 2,000 over costs you five dollars. This is the part that feels fine. It stays feeling fine right up until the workload that made you like the tool is the workload that bills you.

Then the matrix multiplies

Six weeks in, everything has grown the way it's supposed to. Your dataset grew because you seeded it from production traces. Your scorer count grew because you learned what breaks. You're sweeping prompt variants too, because that's the whole point of having evals.

500 cases × 6 scorers × 4 models × 3 prompts × 3 reps = 108,000 scores per full sweep

You run that weekly. 432,000 scores a month. Pro at $249/month includes 50,000 scores, with overage at $1.50 per 1,000:

382,000 scores over $573
Monthly $822" width="520" height="360">

Nothing pathological happened. Nobody made a mistake. Every one of those five dimensions grew for a good reason, and they multiply.

And there are three meters, not one

Scores are the meter that scales with the shape of eval work, but the others ride along. Braintrust bills processed data at $4/GB on Starter and $3/GB on Pro — and agent traces carrying tool calls and retrieved context run hundreds of KB each, one per rep of every cell. If your LLM judge runs through their proxy, token credits meter too.

Seats are free and unlimited, which is genuinely nice and completely beside the point. Your cost driver isn't headcount. It's the Cartesian product.

The part that should bother you

Here's the arithmetic that actually motivates this post. You changed one prompt variant. The other two are byte-identical. All four models are unchanged. All 500 cases are unchanged.

You re-ran and re-paid for all 108,000.

Two thirds of that sweep was recomputation of cells that could not possibly have moved. You paid the platform to meter it, and you paid the model provider to generate it, and the answers were already sitting on your disk from Tuesday.

And then it's deleted — 30-day retention on Pro, 14 on Starter. (Pro does offer extended storage at $0.50/GB/mo, and Enterprise offers custom retention and export, so this is a dial rather than a wall — but it's a dial you pay to turn, on data that is measured in kilobytes.) You spent $822 to learn something, and next month you'll spend it again to remember it.

Every one of those dimensions is a parameter

Cases, scorers, models, prompt variants, reps. That's a parameter sweep, and data science has had good tooling for parameter sweeps for a decade: declare the grid, cache one output per parameter set, and changing one value reruns exactly the cells that value touches.

Change one prompt variant under that model and 36,000 score-equivalents recompute instead of 108,000 — and the platform bill for them is zero, because they're parquet files in data/. You still pay your model provider for the third that genuinely changed. That's the only cost that was ever real.

The rest of this post wires that up in about sixty lines. The division of labor:

pydantic-evals — scoring. Cases, evaluators, reports. Open source, free.
oryxflow — the matrix, the persistence, and the provenance. One cached output per parameter set, tied to the code and parameters that produced it.
Logfire (or your platform of choice) — traces and the debugging UI, which is where observability spend is genuinely well placed.

Nothing overlaps.

Step 1: the dataset and the scorers

Our running example is the intent classifier: two labels, needs_web and no_web.

One methodological note first, because it's the thing most eval write-ups get wrong. This is classification, not a job for LLM-as-judge. The output is one of two fixed strings; exact match is the correct scorer. Reach for LLMJudge when the output is free text with no single right answer — not here, where a judge would add cost, latency and sampling noise to a comparison that was already deterministic.

Second: aggregate accuracy lies. Your errors aren't symmetric. Searching when you didn't need to costs milliseconds. Not searching when you needed to means the agent answers a question about this morning from stale training data. If needs_web is 30% of traffic and you miss a tenth of it, accuracy reads 97% and recall on the class that matters is 90%.

pydantic-evals covers this without a custom scorer: ConfusionMatrixEvaluator and PrecisionRecallEvaluator are report-level evaluators — they run once over the whole experiment — and you pass them via report_evaluators.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',    inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',   inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',    inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)

That's the scoring layer, complete. It knows nothing about caching, matrices, or where results live — which is exactly why it composes.

Step 2: one cell of the matrix, cached

An oryxflow task is a class with parameters, a run(), and a save(). The engine keys the cached output on the parameter values, so each (model, prompt_version) pair gets its own stored result automatically — no filenames to invent, no collisions.

import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)                       # one row per case, parquet
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
            'n_failures': len(report.failures),
        })
        report.print()                        # confusion matrix, in your terminal

Note what gets persisted: per-case rows, not a summary. A stored accuracy number can't answer a question you didn't think to ask in July. Stored per-case rows can — you can slice by class, by case, by query type, months later, without re-running anything.

Step 3: the matrix

A requires() that returns a dict fans out over the grid. inputLoadConcat() stacks the dependencies' outputs into one DataFrame, tagging each with the parameters of the task it came from — so model and prompt_version arrive as columns for free:

MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())

Run it and ask your questions:

flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # what matters

Now edit PROMPTS to add 'v4'. Three new cells run. The six existing ones load from disk in milliseconds and cost nothing — not in API calls, not in metered scores. That's the entire argument of this post, and it fits in one line of diff.

Step 4: provenance, which you get without asking

Most eval setups rot for a reason unrelated to retention: a score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find the number and be unable to defend it.

oryxflow writes an event stream as it runs — plain JSONL you can jq. Each execution records parameters, code version, a fingerprint of the task's source, upstream output hashes, the git SHA, duration, and the reason it ran:

oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes

The reason field is the underrated one. Edit your scoring logic and affected tasks recompute automatically, with a reason naming the changed symbol — code change (auto: evals/run.py::EvalRun). You can't accidentally compare a score computed under the old scorer against one computed under the new one, because the old entry is no longer a valid cache hit.

Step 5: share it

Put data/ under Git LFS — /oryxflow:init-gitlfs with the Claude Code plugin, or git lfs track "data/**" by hand. Results now version alongside the code that produced them, and a teammate who clones gets the numbers without re-running anything.

Two things people get wrong:

Split blessed baselines from scratch sweeps with env=. oryxflow.Workflow(EvalMatrix, env='baseline') writes under data/env=baseline/; Tuesday's throwaway grid goes to env='dev' and never enters LFS. Skip this and LFS bloats with abandoned experiments.
Commit the scored rows, not the traces. Per-case parquet is kilobytes; raw spans are megabytes and belong in Logfire, where the UI for reading them lives. GitHub's free LFS quota is 1 GB with bandwidth metered separately.

What this doesn't solve

Four caveats, each with the fix.

1. The model endpoint is a cache blind spot. oryxflow hashes your code and inputs. It cannot see a provider updating weights behind a stable alias, so pointing at claude-sonnet-5 means your cache will serve July's numbers as current in December, confidently.

Fix: pin the snapshot in the parameter — claude-sonnet-5-20260501, not the alias. The identity of what you measured becomes part of the cache key, and a new snapshot is a new value that runs fresh while the old numbers stay readable. To deliberately re-measure the same endpoint (checking for drift), invalidate just that family:

flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns, nothing else does

2. You're caching a stochastic process. oryxflow guarantees a result came from the code and inputs it recorded. It does not guarantee a rerun reproduces it. A cached run is a faithful record of one sampling.

Fix: set temperature=0 where supported and make it a parameter so the record shows it. Then measure the residual variance rather than assuming it away — which is caveat 4.

3. Latency needs care. pydantic-evals runs cases concurrently by default. Time that and you're measuring your own queueing.

Fix: a separate task with max_concurrency=1, discard the first call, report p50/p95 rather than the mean:

class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)          # rep index

    def run(self):
        timings = []
        classify = timed(build_classifier(self.model, 'v3'), timings)
        dataset.evaluate_sync(classify, max_concurrency=1, progress=False)
        s = pd.Series(timings[1:])                    # drop warm-up
        self.save(pd.DataFrame({'latency_s': s}))
        self.saveMeta({'p50': float(s.quantile(.5)), 'p95': float(s.quantile(.95))})

4. Reps. Decide whether the rep index is a parameter or lives inside pydantic-evals:

Rep as a parameter (n_run above) — each rep caches separately, so going from three reps to five runs exactly two new evaluations. Bigger DAG.
repeat= inside evaluate_sync — one cached unit covers all reps. Smaller DAG, but changing the count re-runs everything.

When reps are expensive, take the parameter and fan out with WorkflowMulti:

flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()              # every rep, tagged, in one DataFrame

df.groupby('model')['latency_s'].quantile([.5, .95])

Need five reps instead of three? Extend the dict. Reps 1–3 load from cache; only 4 and 5 hit the API.

Takeaway

The bill wasn't caused by anything you did wrong. It was caused by a Cartesian product meeting a per-unit meter, and by an architecture where the thing you re-pay for is recomputation of cells that could not have changed.

Score with pydantic-evals, because it's a well-designed open-source scoring library. Debug in Logfire, because a good trace UI is worth paying for. But own the matrix — cache each cell by its parameters, keep the scored rows in your repo, and let a changed prompt variant cost you the third of the grid that actually moved.

pip install oryxflow pydantic-evals

Frequently asked questions

Why does Braintrust get expensive as my evals grow?

Because an eval matrix is a Cartesian product and scores are metered per unit. Cases times scorers times models times prompt variants times repetitions multiplies fast: 500 cases with 6 scorers across 4 models, 3 prompts and 3 reps is 108,000 scores per sweep. Run that weekly and Pro's included 50,000 scores per month is gone in the first sweep — at $1.50 per additional 1,000 the monthly bill lands near $822. Nothing pathological caused it; every dimension grew for a good reason and they multiply.

How do I stop paying to re-run eval cells that did not change?

Cache each cell of the matrix separately, keyed by its parameters. Change one prompt variant of three and only the cells using that variant should recompute — the other two thirds load from disk. oryxflow does this with a task parameterized by model and prompt_version, so a sweep costs you the calls that genuinely changed and nothing else, and the platform bill for the cached cells is zero because they are parquet files in your own data directory.

What is the cheapest way to run LLM evals?

Split scoring from persistence. Use pydantic-evals for cases, evaluators and reports — it is open source and free — and an ordinary caching workflow engine for the matrix and storage, so re-running a sweep only calls the model for cells whose parameters or code actually changed. Keep an observability platform for live traces and the debugging UI, where the money is genuinely well spent, and stop metering the part that is just a Cartesian product over a labeled dataset.

How do I keep per-case eval results after platform retention expires?

Save the scored rows yourself as parquet under a version-controlled data directory and put it under Git LFS. Per-case scored rows for a few hundred cases are kilobytes, so they version alongside the code that produced them and stay readable indefinitely. Leave the bulky raw traces in the observability platform where the trace UI lives — that split keeps your repository small and your results permanent.

Then keep going:

Sibling post: Your eval platform deletes your results in 14 days — the retention argument in full.
Sibling post: LLM evals are a parameter sweep — why the tooling already exists.
Parameter sweeps without rerunning upstream steps
Managing complex workflows — the event stream and reset scopes.
Collaborate and share results — Git LFS and env=.
oryxflow on GitHub

LLM evals are a parameter sweep — use a parameter sweep tool

Norman Niemer — Wed, 26 Aug 2026 17:35:55 +0000

The scoring is genuinely new. The matrix underneath it is a solved problem from 2015.

Three questions teams actually ask about their LLM systems:

Is my classifier right?
Did my prompt change help?
Is the cheap model good enough?

They feel like three different projects. They're one shape: a labeled dataset, run under N configurations, compared.

Write that out as a grid — cases × scorers × models × prompt variants × repetitions — and you have described a Cartesian product over parameters. A parameter sweep. Data science has had good tooling for parameter sweeps for a decade, built around one idea: cache one output per parameter set, and when a value changes, recompute exactly the cells that value touches.

The LLM eval space rebuilt that layer as SaaS, with a meter on it.

Some of what those platforms sell is genuinely new and genuinely worth paying for. Traces of an agent's tool calls, a UI for clicking through failures, span-level debugging — that's a hard problem, well solved, and not something you want to rebuild. But the matrix isn't LLM-specific, and neither is the persistence. Those are the parts where treating evals as a normal data-science sweep pays off immediately.

The division of labor

pydantic-evals — scoring. Cases, evaluators, reports. This is the new part.
oryxflow — the matrix, the persistence, and the provenance. One cached output per parameter set, tied to the code and parameters that produced it.
Logfire (or your platform of choice) — traces and the debugging UI.

Nothing overlaps, which is the whole point. Each layer does one job and none of them needs to know about the others.

The running example

Every agent has an intent classifier somewhere. Ours decides whether an incoming query needs a live web search: two labels, needs_web and no_web.

Two methodological points before code, because they're where evals usually go wrong.

This is classification, not a job for LLM-as-judge. The output is one of two fixed strings. Exact match is the correct scorer. LLMJudge exists for free-text outputs with no single right answer — using it here would add cost, latency and sampling noise to a comparison that was already deterministic.

Aggregate accuracy lies. Your two errors aren't symmetric. A false positive — searching unnecessarily — costs a few hundred milliseconds. A false negative — not searching when you should have — means the agent answers a question about this morning's news from training data, confidently. That's the failure users screenshot. If needs_web is 30% of traffic and you miss a tenth of it, headline accuracy reads 97% while recall on the class that actually matters is 90%.

pydantic-evals handles this without a custom scorer. ConfusionMatrixEvaluator and PrecisionRecallEvaluator are report-level evaluators — they run once over the whole experiment rather than per case — and you pass them via report_evaluators on the Dataset:

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',    inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',   inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',    inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)

The sweep

Everything you vary becomes a task parameter. oryxflow keys the cached output on those values, so each combination gets its own stored result — no filenames to invent, no results_sonnet_v3_final2.json graveyard.

import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
        })
        report.print()

Save per-case rows, not a summary. A stored accuracy number can only answer the question you thought of at the time. Per-case rows let you slice by class, by case, by query type, six months later, without re-running a thing.

The grid is a requires() that returns a dict. inputLoadConcat() row-stacks the dependencies and tags each with the parameters of the task it came from, so model and prompt_version arrive as columns:

MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())

flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # what matters

Two properties you inherit for free

Here's why this framing pays, beyond tidiness.

Trustworthy comparison. This is the one people underrate. If the dataset build is itself a cached upstream task, every configuration reads the same output — not because you were careful, but because there is only one output to read. That kills the quiet failure mode where you regenerated the case set midway through a sweep and half your comparison is against a different denominator. The comparison is apples-to-apples by construction — the same reason parameter sweeps over shared features are trustworthy in ordinary ML work: the shared upstream is computed once, so there's nothing to accidentally desynchronize.

Marginal cost. Add a fourth model and only the fourth model runs. The six existing cells load from disk in milliseconds. For ordinary data work that's convenience — you saved some CPU. For LLM evals it's money, and it compounds with every axis: a 300-case set across three models and two prompts is 1,800 calls, and adding a model without caching means re-running all 2,400 instead of the 600 that are new. Under per-score metered billing you re-meter the cached cells too, so you pay twice for the privilege of not learning anything new.

What ran, and why

A score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find it and be unable to defend it — which is how most eval setups actually die, long before retention expires.

oryxflow writes an event stream as it runs: plain JSONL you can jq. Every execution records parameters, code version, a fingerprint of the task's source, upstream hashes, git SHA, duration, and the reason it ran.

oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes

Edit your scoring logic and the affected tasks recompute automatically, with a reason naming the changed symbol — code change (auto: evals/run.py::EvalRun). You can't compare an old-scorer score against a new-scorer score by accident, because the old one stops being a valid cache entry the moment the code changes.

And it's durable in the boring sense: data/ under Git LFS versions your results alongside the code that made them. Use env= to keep blessed baselines (env='baseline') apart from Tuesday's throwaway grid (env='dev'), commit the scored parquet, and leave the megabyte traces in Logfire where the trace UI lives. Eval platform retention windows run 14 to 30 days on the common plans; a git repo runs as long as the repo does.

Where the analogy breaks

A sweep over hyperparameters and a sweep over models aren't identical, and four differences matter.

1. The endpoint is outside the cache. oryxflow hashes your code and inputs. It cannot see a provider updating weights behind a stable alias — so claude-sonnet-5 as a parameter value means your cache will serve July's numbers as current in December.

Fix: pin the snapshot — claude-sonnet-5-20260501. The identity of what you measured is now part of the cache key, and a new snapshot is a new value that runs fresh while old numbers stay readable. To deliberately re-measure the same endpoint and check for drift, invalidate that family alone:

flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns; dataset build doesn't

2. The function isn't deterministic. In a hyperparameter sweep, re-running a cell reproduces it. Here it doesn't. oryxflow guarantees a result came from the code and inputs it recorded — it is a faithful record of one sampling, not a reproducible constant.

Fix: temperature=0 where supported, as a parameter so the record shows it. Then measure the residual variance instead of hoping about it — see point 4.

3. Latency measurement fights the harness. pydantic-evals runs cases concurrently by default, so wall-clock under a sweep measures your own queueing.

Fix: a separate task with max_concurrency=1, discard the first call, report p50/p95 rather than the mean — one tail call destroys a mean and tells you nothing about typical experience.

4. Repetitions are a design decision, not a detail. Either the rep index is a parameter or it lives inside pydantic-evals:

Parameter — each rep caches separately, so extending a three-rep study to five runs exactly two new evaluations. Bigger DAG.
repeat=in evaluate_sync— one cached unit covers all reps. Smaller DAG, coarser cache; changing the count re-runs everything.

When reps are expensive, take the parameter and let WorkflowMulti fan out:

class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)          # rep index
    ...

flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()          # every rep, tagged, one DataFrame

Decide you need five reps? Extend the dict — reps 1–3 come from cache, only 4 and 5 call the API. That's the same marginal-cost property, applied to the axis most likely to grow after you've already run the study.

Takeaway

Score with pydantic-evals, because scoring LLM output well is a real problem and it solves it. Debug in Logfire, because trace UIs are worth paying for. But recognize the middle layer for what it is: a Cartesian product over parameters, with one cached result per cell.

That's not an LLM problem. It's a parameter sweep, and you can have the marginal-cost economics and the apples-to-apples guarantee that come with treating it like one.

pip install oryxflow pydantic-evals

Frequently asked questions

Are LLM evals just a parameter sweep?

Structurally, yes. "Is my classifier right", "did my prompt change help", and "is the cheap model good enough" are all the same shape underneath: a labeled dataset, run under N configurations, compared. Cases, scorers, models, prompt variants and repetitions are the axes of a grid, which makes an eval matrix a Cartesian product over parameters — exactly what parameter-sweep tooling has handled well for a decade. What is genuinely new is the scoring, and pydantic-evals covers that.

How do I make sure my model comparison is apples-to-apples?

Share the upstream. If the dataset build is its own cached task that every configuration depends on, all models are scored against byte-for-byte identical cases — not because you were careful, but because there is only one dataset output to read. That removes the quiet failure where you regenerated the case set halfway through a sweep and half your comparison is against a different denominator.

Do I need an LLM eval platform, or can I use a normal workflow tool?

You need both, for different jobs. A platform earns its cost for live traces and a UI to click through failures, which is a real problem well solved. The matrix and the persistence are not LLM-specific — one cached output per parameter set is ordinary data-science plumbing, and doing it with a caching workflow engine means adding a model to your grid runs only that model instead of re-metering cells whose results already exist on your disk.

Should the repetition index be a task parameter in an LLM eval?

Make it a parameter when repetitions are expensive. Each rep then caches separately, so extending a three-rep study to five runs exactly two new evaluations instead of all five. The trade-off is a larger DAG. If repetitions are cheap, use pydantic-evals' repeat argument instead — one cached unit covers every rep, giving a smaller graph but a coarser cache, so changing the count re-runs the whole thing.

Then keep going:

Sibling post: Your eval platform deletes your results in 14 days — the retention argument.
Sibling post: Cheap, durable LLM evals: pydantic-evals + oryxflow — the cost arithmetic and the step-by-step build.
Parameter sweeps without rerunning upstream steps
Managing complex workflows — the event stream and reset scopes.
Why oryxflow
oryxflow on GitHub

Your eval platform deletes your results in 14 days

Norman Niemer — Tue, 25 Aug 2026 17:58:58 +0000

LLM evals are a parameter sweep. Use a parameter sweep tool.

You benchmarked four models in July. In August the numbers are gone.

Nothing broke. No one deleted anything. Retention expired.

Here's the landscape as of July 2026:

Logfire 30 days up to 90 days (Growth); custom on Enterprise
Braintrust 14 days (Starter) 30 days (Pro, $249/mo); custom on Enterprise">

This isn't a criticism of either product. It's a statement about what they are: eval platforms are working sets, not archives. They're optimized for the thing you're debugging this week — live traces, a UI to click through failures, diffs against yesterday's run. That's a genuinely hard problem and both solve it well.

But the thing that expires is tiny. A few hundred cases across a few dozen experiments, stored as scored rows, is kilobytes. There is no technical reason to lose it. It expires because storage duration is a pricing lever, which is a perfectly reasonable business decision and a terrible fit for the question "did the July prompt actually beat the June one?"

Two details make the point sharper. Braintrust's own docs say cloud-storage export is "only available on the Enterprise plan" — so getting your data out is itself a tier. And Pydantic's Logfire docs, to their credit, tell you the answer outright: if you need longer than 30 days, write to both Logfire and long-term storage such as S3. The vendor agrees with the premise. The only question is what "long-term storage" should look like for eval results specifically.

The reframe

Three questions teams actually ask about their LLM systems:

Is my classifier right?
Did my prompt change help?
Is the cheap model good enough?

Underneath, these are one shape: a labeled dataset, run under N configurations, compared.

That's a parameter sweep. Data science has had good tooling for parameter sweeps for a decade — cache one output per parameter set, don't recompute what hasn't changed, keep the results next to the code that made them. The LLM eval space rebuilt that as SaaS, with a metered billing model attached to the scoring step.

You don't have to pick a side. Split the job by what each layer is actually good at:

pydantic-evals — scoring. Cases, evaluators, reports.
oryxflow — the matrix, the persistence, and the provenance. One cached output per parameter set, tied to the code and parameters that produced it.
Logfire (or your platform of choice) — traces and the debugging UI.

Nothing overlaps. That's the whole idea. pydantic-evals doesn't want to be a cache; oryxflow doesn't want to be a scorer; neither wants to render a waterfall of spans.

The running example: does this query need web search?

Every agent has one of these. A router decides whether an incoming query needs a live web search or can be answered from the model's own knowledge. Two labels: needs_web, no_web.

Make the methodological call before writing any code, because this is where most eval posts go wrong: this is classification, not a job for LLM-as-judge. The output is one of two fixed strings. Exact match is the correct scorer. An LLM judge here adds cost, latency, and sampling noise to a comparison that was already deterministic.

Then the trap. Aggregate accuracy lies to you.

Your two error types are not symmetric. A false positive — searching when you didn't need to — costs you a few hundred milliseconds and a fraction of a cent. A false negative — not searching when you needed to — means the agent answers a question about this morning's news from training data, confidently and wrongly. That's the failure users screenshot.

If needs_web is 30% of your traffic and you miss a tenth of it, that's 3% of cases. Your accuracy reads 97%. The number that matters — recall on needs_web — is 90%, and it's invisible in the headline.

The good news: you don't have to write a custom scorer for this. pydantic-evals ships ConfusionMatrixEvaluator and PrecisionRecallEvaluator as report-level evaluators — they run once over the whole experiment rather than per case — and you pass them via report_evaluators on the Dataset.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',   inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',  inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',   inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)

That's the scoring layer, complete. It knows nothing about models, caching, or where results live.

Wiring the matrix

Now the sweep. One task, parameterized by the things you vary, saving one row per case:

import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
            'n_failures': len(report.failures),
        })
        self.logger.info('acc={} recall_needs_web={}',
                         rows['correct'].mean(),
                         rows.loc[rows['expected'] == 'needs_web', 'correct'].mean())
        report.print()          # confusion matrix, in your terminal

Then the matrix itself. requires() returning a dict fans out over every combination, and inputLoadConcat() stacks the results:

MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())

inputLoadConcat() tags each dependency's rows with that task's parameters, so model and prompt_version come back as columns — no manual bookkeeping, no filename parsing:

flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # the number that matters

The payoff

Add a fourth model to the grid and only the fourth model runs. The six existing cells load from disk in milliseconds; the two new ones call the API.

For ordinary data work that's convenience — you saved a few minutes of CPU. For LLM evals it's money, and the arithmetic is worth doing explicitly. Take a 300-case set, three models, two prompts: six cells, 1,800 API calls, and however many scores your platform meters. Adding a model without caching means re-running all 2,400 calls. With caching it's 600. You pay for a quarter of the work because three quarters of it hasn't changed.

Contrast that with per-score metered billing, where re-running the matrix re-meters every cell — including the five you already paid for last week and whose results are byte-identical.

The provenance you get without asking

Most eval setups rot for a reason that has nothing to do with retention: a score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find the number in a spreadsheet and be unable to defend it.

oryxflow writes an event stream as you go — plain JSONL you can jq. Each task execution records its parameters, code version, a fingerprint of the task's source, upstream output hashes, the git SHA, the duration, and the reason it ran:

oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes

That last field is the one people underrate. Edit your scoring logic and the affected tasks recompute automatically, with a reason naming the changed symbol — code change (auto: evals/run.py::EvalRun). You can't accidentally compare a v2 score computed under the old scorer against a v3 score computed under the new one, because the old one no longer exists as a valid cache entry.

Sharing it

Put data/ under Git LFS. The Claude Code plugin does it in one step with /oryxflow:init-gitlfs; by hand it's git lfs track "data/**". Now your eval results version alongside the code that produced them, and a teammate who clones the repo gets the numbers without re-running anything.

Two things people get wrong:

Use env= to separate blessed baselines from scratch sweeps. oryxflow.Workflow(EvalMatrix, env='baseline') writes under data/env=baseline/; your ad-hoc Tuesday-afternoon grid goes to env='dev' and never enters LFS. Skip this and LFS bloats with abandoned experiments.
Commit the scored rows, not the traces. Per-case parquet is kilobytes. Raw spans are megabytes and belong in Logfire, where the UI for reading them lives. GitHub's free LFS quota is 1 GB with bandwidth metered separately — easy to stay inside if you're storing scores, easy to blow through if you're storing traces.

The honest section

Four things this setup does not solve, and what to do about each.

1. The model endpoint is a cache blind spot. oryxflow hashes your code and your inputs. It cannot see that a provider updated the weights behind a stable alias. Point EvalRun at claude-sonnet-5 and your cache will confidently serve July's numbers as current in December.

Fix: put the pinned snapshot id in the parameter — claude-sonnet-5-20260501, not the alias. Now the identity of the thing you measured is part of the cache key, and moving to a new snapshot is a new parameter value that runs fresh while the old numbers stay readable. When you deliberately want to re-measure the same endpoint — checking for drift, say — invalidate just that family:

flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns; nothing else does

2. You're caching a stochastic process. oryxflow guarantees a result came from the code and inputs it recorded. It does not guarantee that re-running produces the same numbers. A cached run is a faithful record of one sampling, not a reproducible constant.

Fix: set temperature=0 where the provider supports it and make it a parameter so the record shows it. Accept that the remaining variance is real, and if it's large enough to change your decision, measure it — which is caveat 4.

3. Latency needs care. pydantic-evals runs cases concurrently by default. Measure wall-clock under that and you're measuring your own queueing, not the model.

Fix: a dedicated task with max_concurrency=1, discard the first call as warm-up, and report p50/p95 rather than the mean — one slow tail call wrecks a mean and tells you nothing about typical experience.

class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)      # rep index

    def run(self):
        timings = []
        classify = timed(build_classifier(self.model, 'v3'), timings)
        dataset.evaluate_sync(classify, max_concurrency=1, progress=False)
        s = pd.Series(timings[1:])                # drop warm-up
        self.save(pd.DataFrame({'latency_s': s}))
        self.saveMeta({'p50': float(s.quantile(.5)), 'p95': float(s.quantile(.95))})

4. Repetitions. You have to decide whether the rep index is a parameter or lives inside pydantic-evals. Both are defensible:

Rep as a parameter (n_run above) — each repetition caches separately, so adding reps 4 and 5 to a three-rep study runs exactly two new evaluations. The DAG gets bigger.
repeat= inside evaluate_sync — one cached unit covers all reps, smaller DAG, but changing the rep count re-runs everything.

Prefer the parameter when reps are expensive, and fan out with WorkflowMulti:

flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()          # every rep, tagged, in one DataFrame

df.groupby('model')['latency_s'].quantile([.5, .95])

Decide you need five reps instead of three? Extend the dict. Reps 1–3 load from cache; only 4 and 5 hit the API.

Takeaway

An LLM eval is a labeled dataset, run under N configurations, compared. That's a parameter sweep, and parameter sweeps have had good tooling for a decade.

Score with pydantic-evals, because it's a well-designed scoring library. Debug in Logfire, because a trace UI is genuinely worth paying for. But keep the scored rows yourself, cached by parameter set, versioned next to the code that made them — because they're kilobytes, they're the actual output of your work, and no retention policy should be able to decide when you stop being able to answer "did that change help?"

pip install oryxflow pydantic-evals

Frequently asked questions

How long do LLM eval platforms keep my results?

Not long, by design. As of July 2026 Logfire retains 30 days on its free and Team plans and up to 90 days on Growth; Braintrust retains 14 days on its free Starter plan and 30 days on Pro at $249/month. Longer windows are Enterprise-priced. These are working sets, not archives — if you want a July benchmark still readable in December, you have to keep a copy yourself.

How do I keep LLM eval results permanently without paying for longer retention?

Save the scored per-case rows to your own repo and treat the eval matrix as a cached parameter sweep. Score with pydantic-evals, then wrap each (model, prompt) run in an oryxflow task so the result is cached on disk keyed by its parameters, versioned with Git LFS, and still readable years later. The scored rows are kilobytes; leave the bulky raw traces in your observability platform where the debugging UI lives.

Should I use LLM-as-judge to evaluate an intent classifier?

No. When the output is one of a fixed set of labels, exact match against the expected label is the correct scorer — an LLM judge adds cost, latency and noise to a comparison that is already deterministic. Use EqualsExpected for the per-case score and a ConfusionMatrixEvaluator report evaluator for the class breakdown, because aggregate accuracy hides the error that actually costs you: the false negative that stops the agent from searching.

How do I avoid re-running my whole eval matrix when I add one model?

Make each cell of the matrix a separately cached unit keyed by its parameters. In oryxflow an EvalRun task parameterized by model and prompt_version caches one output per combination, so adding a fifth model to a four-model grid runs only the fifth model's cases — the other four load from disk. The saving is real money, because with per-score metered billing re-running the matrix re-meters every cell you already paid for.

Then keep going:

Sibling post: LLM evals are a parameter sweep — the same argument from the tooling side.
Sibling post: Cheap, durable LLM evals: pydantic-evals + oryxflow — the step-by-step build.
Parameter sweeps without rerunning upstream steps
Managing complex workflows — the event stream and reset scopes.
Collaborate and share results — Git LFS and env=.
oryxflow on GitHub

When not to use oryxflow

Norman Niemer — Fri, 21 Aug 2026 14:00:00 +0000

Being clear about where a tool doesn't fit is part of being trustworthy — so here's where oryxflow is the wrong choice.

oryxflow is a local, zero-infrastructure library for the research loop: it ties every result to the code and inputs that produced it, tracks lineage, and reruns exactly what a change affects — so a number can't quietly sit on stale data and you can regenerate any result months later, without a server, a scheduler, or a database. (It reuses the steps that didn't change, which is why none of that costs you rerun time.) That's genuinely useful — but only for a specific shape of problem. Every honest tool has a boundary, and pretending oryxflow fits everywhere would waste your time and cost you trust. So here is where it doesn't fit, and what to reach for instead.

A quick one-off doesn't need a pipeline — but you can still start here

If your whole analysis is "load a CSV, group by a column, plot one thing," you don't need task classes around it. Five lines you'll run once get no payoff from caching, and wrapping them in a DAG is ceremony.

The value of a caching DAG rises with three things: depth (how many dependent steps), cost (how expensive each step is), and breadth (how many parameter combinations you sweep). For a five-line notebook cell, all three are near zero, so the return is near zero too.

Use instead: a plain script for the first pass — but keep it inside the project, not in a scratch folder you'll abandon. If you build with the oryxflow Claude Code plugin, that's already the convention: exploration lives as a read-only probe at eda/<subject>/<name>.py, one line of docstring stating the question it answers, printing the answer legibly, run with python -m eda.<subject>.<name>. A probe writes no pipeline artifact — disposable scratch goes to a gitignored scratch area — and anything material it turns up gets written into the project's data doc, so a question you've answered once isn't re-asked next session.

Then, when a script turns out to be load-bearing — you keep re-running it, something depends on its output, or you start sweeping it over parameters — you don't rewrite it by hand. /oryxflow:migrate reads the script as the spec, shows you a step-to-task map, writes only on your approval, and never deletes the source. So there's no cliff between "exploring" and "having a pipeline": you can start with a small EDA and let it scale to any complexity, with no rewrite in between. More on both ends: migrate a notebook to a pipeline and Claude Code for data science.

Don't reach for oryxflow when… you need production orchestration

Scheduled runs at 6am, retries across a cluster, backfills over a date range, alerting when a job fails, SLAs your team is on the hook for — that's production operations, and oryxflow doesn't do it. There's no scheduler, no distributed retry, no alerting, and no operational UI.

Use instead: Airflow, Prefect, or Dagster. These are excellent at what they do, and they do a different job than oryxflow — they orchestrate operations; oryxflow makes the research loop that happens before anything is scheduled trustworthy and reproducible. Many teams develop logic in oryxflow and later wrap the finished pipeline in one of these for production. They're complementary, not competitors.

Don't reach for oryxflow when… you need distributed or very-large-scale execution

If a single step needs a Kubernetes cluster, or your data doesn't fit on one machine and you need engine-level parallelism, the OSS core isn't built for that. It's local-first and runs in-process.

Use instead: Flyte or Metaflow on Linux or WSL. (oryxflow's paid Pro tier adds SQL, cloud storage, Dask, and PySpark backends — but the open-source core is deliberately local-first, and that's the right lens for evaluating fit here.)

Don't reach for oryxflow when… you want an experiment dashboard

If what you need is a searchable web UI showing every run's metrics, params, and charts side by side — sortable, filterable, shareable with your team — oryxflow doesn't provide it. It gives you a queryable lineage log, not a hosted dashboard.

Use instead: MLflow or Weights & Biases. And note these compose with oryxflow: log your metrics to MLflow from inside a task's run(), and let oryxflow handle the caching and reproducibility around it. You don't pick one — you use both, each for its strength. (More on that split in MLflow, or a reproducible pipeline.)

Don't reach for oryxflow when… you need Git-tied data versioning

If your goal is versioning large data artifacts alongside your code, pinned to Git commits and pushed to remote storage, that's a different discipline.

Use instead: DVC. It also composes with oryxflow — DVC for artifact versioning, oryxflow for the compute graph on top.

The honest caveat: oryxflow does not check that your result is correct

This is the one worth reading twice. oryxflow guarantees that an output was produced by the exact code and inputs it recorded — it makes your pipeline reproducible. It does not guarantee the result is right.

It will happily cache, with full lineage:

a join that silently went many-to-many when it should have been many-to-one,
a test set that leaked into training,
a ratio computed against the wrong denominator,
a timestamp shifted by a timezone you forgot to normalize,
a backtest that peeks at data from the future.

Every one of those is reproducible, lineage-tracked, and wrong. oryxflow manages pipeline mechanics; it has no opinion about statistical judgment. Those bugs are caught by habit — sanity checks, held-out validation, reading your own numbers skeptically — not by any caching machinery. If a post ever tells you a workflow tool makes your analysis correct, close the tab. For where this boundary lives, see what caching does not protect against.

Where it is the right tool

With the boundaries drawn honestly, the fit is clear. oryxflow earns its keep when:

a result has to be defensible — you'll hand it to someone, or make a decision on it,
you need to reproduce or hand off that research months later,
your pipeline is a deep chain of dependent steps, where staleness has somewhere to hide,
some of those steps are expensive (minutes to hours), so rerunning everything to be sure isn't an option,
you sweep a matrix of parameters and want every configuration compared against the same upstream, and
pipelines are authored by AI agents that benefit from an explicit, inspectable task graph.

A typical task is small and declarative — declare dependencies, load inputs, save outputs, and the engine reruns what a change affects and reuses the rest:

@oryxflow.requires(CleanData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(add_features(df))

flow = oryxflow.Workflow(task=Features)
flow.run()

That's the sweet spot: results that have to hold up, in a pipeline deep and expensive enough that you'd never verify them by rerunning the whole thing from scratch.

Takeaway

Start a quick exploration as a plain script — inside an oryxflow project, where /oryxflow:migrate can promote it the day it earns a pipeline. Use Airflow, Prefect, or Dagster for production ops. Use Flyte or Metaflow for distributed scale. Use MLflow or W&B for dashboards, DVC for data versioning — and compose them with oryxflow where it helps. And never expect any of them, oryxflow included, to check your statistics for you.

Being honest about fit is the whole point: reach for oryxflow when you have a research pipeline — or the first small script that might become one — and reach for something else when the job is scheduling, scaling out, or display.

pip install oryxflow

Frequently asked questions

When should I not use oryxflow?

Skip oryxflow for production orchestration (scheduling, retries, alerting — use Airflow, Prefect, or Dagster), for distributed or larger-than-memory execution (Flyte or Metaflow), for a hosted experiment dashboard (MLflow or W&B), and for Git-tied data versioning (DVC). oryxflow is a local, zero-infrastructure library for making research results trustworthy and reproducible; it reruns exactly what a change affects and reuses the rest, but it doesn't schedule, scale out, or display.

Is oryxflow overkill for a quick one-off analysis?

No — a five-line cell you run once doesn't need task classes, and you don't have to decide upfront either way. If you build with the oryxflow Claude Code plugin, exploration gets a home in the project from the start: a read-only probe script that states the question it answers and still runs next session, instead of a snippet you lose. When a probe turns out to be load-bearing — you keep re-running it, or something downstream depends on its result — /oryxflow:migrate lifts it into cached, parameterized tasks and never deletes the original. So you can start small in oryxflow and scale to any complexity, with no rewrite in between.

Does oryxflow check that my analysis is correct?

No — and no workflow tool does. oryxflow guarantees an output was produced by the exact code and inputs it recorded, which makes your pipeline reproducible, not correct. It will happily cache, with full lineage, a leaked test set, a bad join, or a backtest that peeks at the future. Those bugs are caught by sanity checks, held-out validation, and reading your own numbers skeptically — not by any caching machinery.

How to turn a messy notebook into a reproducible pipeline, one step at a time

Norman Niemer — Thu, 20 Aug 2026 14:00:00 +0000

It was supposed to be a quick analysis. Turn it into a pipeline you can trust — one step at a time, without a big rewrite.

A notebook is the best place to start an analysis and the worst place to keep one.

While you are exploring, cells run out of order, variables linger in memory long after the cell that defined them was edited, and half your intermediate results are stale copies from an hour ago. It all still works — until you reopen the notebook a week later, hit "Run All," and get a different number. Now you are stuck asking the question every data scientist eventually asks: which version of the code and which data actually produced this result? If you cannot answer that, you cannot trust the result, and you cannot reproduce it for anyone else.

The usual culprits are hidden state (a variable that only exists because you ran a cell you have since deleted), out-of-order execution, and stale intermediates that never got refreshed after an upstream change. None of these show up as errors. They just quietly make your output wrong.

oryxflow is a small Python library that ties every result to the exact code and parameters that produced it. You declare each step of your analysis as a task — with its parameters and its dependencies — and the library runs them in the right order and reruns a step whenever its code or inputs actually change, so nothing you look at is quietly built on something you edited an hour ago. That is reproducibility you get for free, not reproducibility you have to remember to maintain. And because the steps that didn't change are never recomputed, you get it without paying for it in rerun time.

The good news: you do not rewrite your notebook to get there. You migrate it one step at a time, and you have a working pipeline after every single step.

Start with the linear script

Here is a typical notebook, flattened into the script it really is: load data, build features, train a model.

import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.read_parquet('raw.parquet')          # slow: pulls a big table

df['x_squared'] = df['x'] ** 2               # feature engineering
features = df[['x', 'x_squared']]

model = LinearRegression().fit(features, df['y'])

Every time you tweak the model, that first line pulls the whole table again. And nothing records that this model came from that data and that feature code.

Convert the first expensive step into a task

Do not convert everything. Start with the one step that hurts most — usually the slow load. Wrap it in a task class, and replace the return with self.save(...).

import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):   # TaskPqPandas = saves a DataFrame as parquet
    def run(self):
        df = pd.read_parquet('raw.parquet')
        self.save(df)

Run it:


flow = oryxflow.Workflow(GetData)
flow.run()
df = flow.outputLoad()                         # the saved DataFrame, back in your hands

The first flow.run() does the slow load and caches the result. Every run after that is a cache hit — it loads from disk instead of re-fetching. Your remaining notebook cells keep working, now fed by df = flow.outputLoad(). You have already gained something and rewritten almost nothing.

Wire the next step with @oryxflow.requires

Now pull feature engineering into its own task. The @oryxflow.requires(GetData) decorator declares the dependency; inside run, self.inputLoad() hands you GetData's already-loaded output.

@oryxflow.requires(GetData)
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                  # GetData's output, already loaded
        df['x_squared'] = df['x'] ** 2
        self.save(df[['x', 'x_squared', 'y']])

You never call GetData yourself. You ask for BuildFeatures, and oryxflow runs its dependencies first, in order — no more "did I run the cells in the right sequence?"

Finish the DAG

Add the model as a final task. Models are not DataFrames, so save them with TaskPickle. This is also where parameters earn their keep: expose the knobs you actually sweep.

from sklearn.linear_model import LinearRegression

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):   # TaskPickle = pickle, for models & arbitrary objects
    power = oryxflow.IntParameter(default=2)
    fit_intercept = oryxflow.BoolParameter(default=True)

    def run(self):
        df = self.inputLoad()
        X, y = df[['x', 'x_squared']], df['y']
        model = LinearRegression(fit_intercept=self.fit_intercept).fit(X, y)
        self.save(model)

That is the whole pipeline: GetData → BuildFeatures → TrainModel. Preview it before running anything:

flow = oryxflow.Workflow(TrainModel)
flow.preview()                                 # prints the task tree, runs nothing
flow.run()
model = flow.outputLoad()

preview() shows you the execution plan — which steps will run and which are already cached — without touching your data. When you are ready, run() executes only what is needed.

What you actually gain

Reproducibility by construction. Each task's output is stored under an id derived from its code and its parameters. Change the feature formula, and oryxflow knows BuildFeatures is now different from the version that produced the cached file. "Which data made this result?" stops being a detective problem.

No wasted recompute. Rerun the pipeline and nothing happens if the outputs already exist — cache hits all the way down. Edit BuildFeatures, though, and oryxflow reruns it and TrainModel automatically, because the model depends on features that just changed. Code invalidation is automatic — there is no cache to manually clear and no reset to remember. This is the same mechanism the sibling post Stop rerunning your whole pipeline digs into.

Load any result by name. Every intermediate is addressable. Want the features without rerunning the model? oryxflow.Workflow(BuildFeatures).outputLoad() hands back the cached frame. Comparing two settings? oryxflow.Workflow(TrainModel, {'power': 3}).outputLoad() loads that configuration's model directly. No more scrolling for the cell that defined the variable you need.

What you do not have to decide upfront

A twenty-minute first look at a new dataset does not need tasks. If you will not run any of it twice, the task scaffolding is pure overhead — pull a step into a task when the computation is slow enough to want caching, or the result is important enough to need reproducing.

What you do not have to do is decide that in advance. If you use Claude Code, the oryxflow plugin gives that first look a home inside the project — a read-only probe that states the question it answers and still runs next session, instead of a cell you lose — and when a probe turns out to be load-bearing, /oryxflow:migrate promotes it into cached, parameterized tasks, reading your script as the spec and never deleting it. So "is this analysis big enough for a pipeline yet?" is a question you can answer by doing the work rather than guessing first: start with a simple script and let it scale as the work gets complex, with no rewrite in between. Walkthrough: migrate a notebook to a pipeline.

Where oryxflow genuinely stops is operations. It is built for the research loop, not for production orchestration; if you need scheduling, retries, and alerting across a fleet, use a real orchestrator. oryxflow makes your own analysis trustworthy and fast to iterate on.

A good rule of thumb: pull a step into a task the second time you find yourself waiting for it to recompute something that did not change.

Takeaway

Migrate incrementally. Wrap the most expensive step first, keep a working pipeline at every stage, and never do a big-bang rewrite.
@oryxflow.requires wires dependencies; self.inputLoad() reads them; self.save() caches the output.
Editing a task reruns it and everything downstream automatically — reproducibility you do not have to maintain by hand.

pip install oryxflow

If you use Claude Code, the oryxflow plugin's /oryxflow:migrate command walks a script through exactly this conversion, one step at a time.

Frequently asked questions

Why do I get different numbers when I rerun a notebook a week later?

Usually it's hidden state, out-of-order cell execution, or stale intermediate results that never refreshed after an upstream change — none of which surface as errors, they just quietly make your output wrong. oryxflow removes all three by declaring each step as a task with explicit dependencies: the engine runs them in the right order, stores each output under an id derived from its code and parameters, and reruns a step whenever its code or inputs actually change.

How do I convert a Jupyter notebook into a reproducible pipeline?

Migrate incrementally rather than rewriting. Wrap your slowest step — usually the data load — in an oryxflow task that ends with self.save(), run it once so the result is stored, then feed the rest of your notebook from that output. Add one step at a time with @oryxflow.requires to wire dependencies. You keep a working pipeline at every stage, and each output is tied to the exact code and parameters that produced it.

Do I have to rewrite my whole notebook at once to use oryxflow?

No — that's the point of migrating incrementally. Start by converting the single step that hurts most, usually the slow load, and leave the rest of your cells untouched, now fed by flow.outputLoad(). You have a working pipeline after every step, and you only pull the next step into a task when it earns it. oryxflow is designed for this one-step-at-a-time conversion, not a big-bang rewrite.

Next steps:
· Why oryxflow
· Transition guide
· Quickstart
· Claude Code plugin
· GitHub

Reproducible data science workflows in Python

Norman Niemer — Wed, 19 Aug 2026 14:00:00 +0000

Reproducibility and lineage are the product. Caching is just how you get them for free.

A workflow is reproducible when you can say exactly which code and which inputs produced a given result — and regenerate it on demand. That's the whole definition, and it's a surprisingly high bar. Most analysis pipelines fail it not because anyone was careless, but because the tools we reach for — notebooks and loose scripts — have no memory of what produced what.

This post is about closing that gap in plain Python: what actually makes a data-science workflow reproducible, why workflows quietly lose reproducibility in the first place, and how a small library called oryxflow gets you there without a server, a database, or a YAML file. If you want the canonical positioning — where oryxflow sits relative to notebooks and orchestrators — read why oryxflow. This is the keyword-first explainer.

Why workflows lose reproducibility

Nobody sets out to build a pipeline they can't reproduce. It happens by accretion:

Hidden notebook state. A variable exists only because you ran a cell you've since edited or deleted. The notebook still runs top-to-bottom in your head, but not on disk.
Out-of-order execution. Cells run 1, 2, 4, 3, 2-again. The result on screen reflects a path through the code that no fresh "Run All" will reproduce.
Stale intermediates. You saved features.parquet yesterday, changed the feature code today, and trained on the old file without noticing. Nothing errored. The number is just quietly wrong.
No durable link between artifact and code. You have model.pkl and you have your script, but nothing records that this model came from that commit of the feature code and those parameters. Six weeks later, you can't answer the question that matters: which version made this?

None of these are exotic. They're the default failure mode of exploratory work, and they all share a root cause: the saved artifacts and the code that made them live in separate worlds, with no enforced connection between them.

What makes a workflow reproducible

Reproducibility isn't one feature. It's three properties working together.

Deterministic task identity from code + params. Every step has an identity computed from its code and its parameters. Same code, same inputs → same identity → the same output, every time. Change either and you get a new identity — so a result is never silently attributed to code that didn't produce it. 2.** Automatic invalidation.** You should not be able to evaluate new code against a stale output. When a step's logic changes, that step and everything downstream of it must be recomputed — automatically, without you remembering to delete a cache file. Reproducibility that depends on your discipline isn't reproducibility.
A lineage record. A durable, append-only log of what ran, when, and why. Not because it's tidy, but because "regenerate this result" requires knowing what produced it in the first place.

oryxflow is built around exactly these three. You declare each step of your analysis as a Task with its parameters and its dependencies; the library runs them in dependency order, caches each output, and reruns a step only when its code or inputs actually change. The identity is deterministic, the invalidation is automatic, and every run appends to a lineage file. You don't maintain reproducibility — you get it as a side effect of writing tasks.

Deterministic identity, and automatic code-change invalidation

The part that's easy to underrate: oryxflow watches your code, not just your parameters. It tracks each task's logic plus the helper files it imports, comparing what your code does, not how it's written. Edit the body of BuildFeatures — or a helper function it calls — and oryxflow knows that task's output is stale and reruns it and everything downstream. Reformat the code, add a comment, rename a local variable? Nothing reruns — those don't change what the code does. And an expensive upstream step whose code you didn't touch stays cached.

That's the property that makes the research loop both fast and trustworthy: you can never test new logic against an output the old logic produced, and you never pay to recompute a step that didn't change.

Building a small reproducible DAG

Here's the canonical shape — GetData → BuildFeatures → TrainModel — with nothing but the verified API. Each task's base class picks its serialization by type: TaskPqPandas saves a DataFrame as parquet, TaskPickle saves an arbitrary Python object. No paths, no to_parquet calls, no config.

import oryxflow


class GetData(oryxflow.tasks.TaskPqPandas):        # saves a DataFrame as parquet
    source = oryxflow.Parameter(default='raw.parquet')

    def run(self):
        df = load_table(self.source)               # your loader
        self.save(df)


@oryxflow.requires(GetData)                        # declares the dep AND copies params
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                      # the upstream DataFrame
        df['x_squared'] = df['x'] ** 2
        self.save(df[['x', 'x_squared', 'y']])


@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):       # saves any Python object
    alpha = oryxflow.FloatParameter(default=1.0)

    def run(self):
        df = self.inputLoad()
        model = fit_model(df[['x', 'x_squared']], df['y'], alpha=self.alpha)
        self.save(model)
        self.saveMeta({'n_rows': len(df), 'alpha': self.alpha})

Run it through a Workflow, which resolves the DAG and runs only what's needed:

flow = oryxflow.Workflow(TrainModel, {'alpha': 0.5})
flow.preview()                                     # show the plan without running
flow.run()                                         # runs GetData -> BuildFeatures -> TrainModel

model = flow.outputLoad()                           # the trained model
meta = flow.outputLoadMeta()                        # {'n_rows': ..., 'alpha': 0.5}

The first run computes all three steps. Edit TrainModel.run and rerun: GetData and BuildFeatures stay cached, only TrainModel recomputes. Change alpha and you get a distinct identity — the old and new models coexist, each tied to its parameters. Nothing here writes a path or checks whether a file already exists; that bookkeeping is the library's job. For the fuller pattern — parameter sweeps, resetting, sharing flows — see managing workflows.

When to use it — and when not to

Reach for a reproducible DAG when a result needs to be defensible: something you'll hand to a colleague, revisit in a month, or make a decision on. The moment a workflow has more than one expensive step and you're iterating on the later ones, deterministic identity and automatic caching pay for themselves.

A first look doesn't need tasks — and it doesn't need a decision either. If you're eyeballing distributions and nothing downstream depends on the output, task boilerplate is pure overhead; write a plain script. But you don't have to choose a tool before you know how big the work will get: if you build with the oryxflow Claude Code plugin, that first look already has a home in the project — a read-only probe that states the question it answers and still runs next session — and /oryxflow:migrate lifts it into cached, parameterized tasks the day something starts depending on its result. So you can start a small exploration here and let it scale as the shape stabilizes, with no rewrite in between. Where oryxflow genuinely doesn't fit — scheduling, distributed scale, dashboards — has a whole post: when not to use oryxflow.

Where oryxflow sits

Reproducibility isn't binary, and oryxflow isn't the only layer that touches it. It occupies the missing middle between a notebook and a production orchestrator.

Orchestrators are a complementary layer, not a competitor: they schedule and distribute pipelines in production, which is a real and different problem. Experiment trackers like MLflow and W&B are complementary too — they record runs; oryxflow structures and caches them. oryxflow's differentiators are the combination that neither layer offers: type-driven zero-config I/O, automatic code-change invalidation, deterministic task identity, and a local-first design with native Python identity — no YAML, no server, no account, no telemetry. There's a full comparison in oryxflow vs. the field.

The honest caveat: reproducible ≠ correct

Be clear about what reproducibility buys you. oryxflow guarantees that an output was produced by the exact code and inputs it recorded — no more. It does not guarantee that the output is right. A pipeline with a bug in its feature logic is reproduced just as faithfully as a correct one; you'll get the same wrong number every time, tied cleanly to the flawed code that made it.

That's not a weakness — it's the honest scope. Reproducibility is what makes a wrong result debuggable: because you know exactly which code and inputs produced it, you can find the bug, fix it, and let automatic invalidation rerun precisely what the fix touched. Correctness is still your job. Reproducibility is what makes doing that job tractable.

Takeaway

A workflow is reproducible when you can name the exact code and inputs behind any result and regenerate it. That takes three things — deterministic task identity, automatic invalidation so you can't test new code on stale outputs, and a durable lineage record. oryxflow gives you all three in plain Python, locally, with the caching that makes them cheap thrown in for free. It won't make your pipeline correct. It will make it something you can trust, hand off, and reproduce.

pip install oryxflow

Frequently asked questions

How do I make a data science workflow reproducible in Python?

A workflow is reproducible when you can name the exact code and inputs behind any result and regenerate it on demand. That takes three things: deterministic task identity from code and parameters, automatic invalidation so you can't test new code against a stale output, and a durable lineage record. oryxflow gives you all three in plain Python, locally, with caching that makes them cheap thrown in for free.

What makes a pipeline reproducible without a lot of infrastructure?

Reproducibility needs a durable link between each result and the code and inputs that produced it, not a server or database. oryxflow delivers that locally: it computes a deterministic identity for each step, reruns a step automatically when its code changes, and appends every run to a local lineage log at .oryxflow/events.jsonl. No server, no database, no account — just pip install and local files.

The best Claude Code plugins and tools for data science

Norman Niemer — Fri, 14 Aug 2026 15:45:41 +0000

The landscape is early, so stop shopping for a "best plugin" and start choosing by the job you need done — the one below keeps AI-generated analysis reproducible.

If you let Claude Code loose on a data-science project, the first thing you notice is how much it can do in one session: pull data, write features, fit a model, plot the result. The second thing you notice is how easily it loses the thread. Across a few turns the agent forgets which steps already ran, re-executes a ten-minute feature build for no reason, or — worse — edits an upstream step and evaluates a model on the old output without realizing anything went stale. The code looks fine. The result is quietly wrong.

That is the real problem a good Claude Code setup for data science has to solve. Not "can the agent write pandas" — it can — but can you trust what it produced, and could you reproduce it tomorrow? Agents are stateless between turns; your pipeline is stateful. The tools that matter are the ones that close that gap.

How we picked

There is no crowded, mature market of data-science Claude Code plugins yet, so we're not ranking twenty near-identical products. We're grouping the genuinely useful options by the job they do and judging each on four things that actually decide whether AI-assisted analysis holds up:

Reproducibility — can you (or a teammate, or the agent next week) recreate a result exactly, and know what it was built from?
State management across turns — does the tool help the agent track what's already computed and what went stale, so it doesn't build on outdated work?
Local-first and private — does your data and lineage stay on your machine, with no telemetry or mandatory cloud service?
Does the job it claims — honest scope. A tool that connects to your warehouse is not pretending to make your analysis reproducible, and vice versa.

A quick word on honesty: this space moves fast and many "plugins" are really MCP servers (Model Context Protocol connectors) or thin wrappers. Where we're confident a specific tool exists, we name it. Where we aren't, we describe the category so you can search for the current best option yourself rather than trust a made-up product name.

Building reproducible pipelines — the oryxflow Claude Code plugin

This is the job most AI-assisted data-science setups get wrong, and it's the one oryxflow is built for. oryxflow is a small, local-first Python library that turns ordinary scripts and notebooks into a dependency-aware task graph: you declare tasks with parameters and requires() dependencies, and the engine runs them in order and reruns exactly the ones a change affects — while anything genuinely unchanged loads from disk, so keeping the graph honest costs you less time than not having it.

The companion Claude Code plugin is what makes an agent good at using that model. It's a skill plus a handful of slash commands that activates automatically when you're working in an oryxflow project — if the skill/plugin/MCP distinction is fuzzy, Claude Code skills for data science untangles it. It front-loads the correct idioms so the agent doesn't build on stale data, verifies that an edit actually reran the tasks it should have, and reuses finished work instead of recomputing it. In practice that means:

/oryxflow:init-project scaffolds a ready-to-run project structure.
/oryxflow:migraterestructures a messy notebook or linear script into cached, parameterized tasks one step at a time, so you always have a working pipeline.
/oryxflow:check-standards keeps names, style, and docstrings consistent.
/oryxflow:init-gitlfs and /oryxflow:update-project handle data versioning and keeping an older project current.

Why it's the strongest fit for this job: reproducibility is a property of your computation graph, and oryxflow owns that graph — parameters and code changes become new cached identities automatically, so you can't accidentally evaluate a new model on old features. It's local-first with no telemetry; your data and lineage stay on your machine.

One honest limit, stated plainly: oryxflow makes analysis reproducible, not correct. It guarantees you can recreate a result and that the result was built from current inputs. It does not check that your feature logic is sound or your metric is the right one — that's still your job. What it removes is the whole class of "was this trained on the new data?" uncertainty that makes agent-generated pipelines untrustworthy. See Why library + plugin is a matched pair and why oryxflow for the fuller argument.

Connecting to your data — MCP data connectors

Before you can build a pipeline you have to reach the data, and this is where the broader MCP (Model Context Protocol) ecosystem shines. There are MCP servers for databases and warehouses, filesystems, and APIs, letting Claude Code query a table or read a file directly as part of a turn. If your bottleneck is "the agent can't see my data," a data connector is the right tool — and it's a genuinely different job from making the resulting analysis reproducible.

The pairing is natural: use an MCP connector to reach the raw data, then wrap the pull in an oryxflow task so the fetched result is cached and lineage-tracked rather than re-queried on every turn. Local-first varies by connector — a filesystem server is fully local, a managed-warehouse one obviously isn't — so pick per your privacy needs.

Notebooks and code execution — execution tools

A lot of data-science work still lives in notebooks, and there are notebook and code- execution tools — including generic Jupyter/notebook MCP servers — that let an agent run cells and read back outputs interactively. These are great for exploration and for the messy, visual first pass where you don't yet know what the pipeline should be.

Their honest limitation is the same statelessness problem we opened with: a notebook is a pile of cells with hidden execution order, and an agent iterating in one has no built-in notion of what's stale. That's exactly why the common path is explore in a notebook, then graduate the keeper steps into a cached pipeline — the subject of Turn a messy notebook into a reproducible pipeline.

Experiment tracking — MLflow (including its experimental MCP)

Once you're producing results worth comparing, you want a tracker. MLflow is the best-known, and it now ships an official (experimental) MCP server that lets an agent query your logged runs, parameters, and metrics conversationally.

This is a complementary job, not a competing one. Tracking answers "which run scored 0.91, and what were its hyperparameters?" — a logging and comparison problem. Reproducibility answers "which steps do I need to rerun to recreate that run, and which are already computed?" — a computation problem. A tracker will faithfully log a score without any idea whether the features feeding it are stale. Put your tracker calls inside your cached tasks and you get both: a reproducible graph and a searchable record. We go deeper in MLflow, or a reproducible pipeline?.

At a glance

FAQ

What are the best Claude Code plugins for data science?

The best Claude Code plugin depends on the job: use an MCP connector to reach your data, a notebook tool to explore, an experiment tracker to compare runs, and oryxflow to make the pipeline itself trustworthy and reproducible — so the agent never builds on stale data and every result traces back to the code that made it. They compose rather than compete: reach data with a connector, wire the steps with oryxflow, and log outcomes to your tracker. Note oryxflow is a skill plus slash commands, not an MCP server.

What's the best plugin for keeping AI-generated pipelines reproducible?

If the specific worry is that an agent will build on stale data or produce a result you can't recreate, the oryxflow plugin is the strongest fit, because it's backed by a task-graph engine that makes reproducibility a structural property rather than a discipline you have to remember. Connectors and notebook tools are excellent at their jobs — getting data in and exploring it — but they don't own your computation graph, so they can't guarantee that the result you're looking at was built from current inputs.

Do I have to choose one tool?

No, and you probably shouldn't. The categories map to different jobs: a connector to reach data, a notebook tool to explore, a task-graph engine to make the pipeline reproducible, a tracker to compare runs. The pattern that works is to reach data with a connector, wire the steps with oryxflow, and log the outcomes to your tracker from inside the tasks.

Takeaway

The Claude Code data-science plugin landscape is young, so the winning move isn't to find the one blessed plugin — it's to pick by the job in front of you. For reaching data, use an MCP connector; for exploring, a notebook tool; for comparing runs, a tracker like MLflow. And for the job that decides whether any of it is trustworthy — reproducible, lineage- tracked pipelines an agent can iterate on without building on stale work — the oryxflow plugin is the layer that makes the other tools' output something you can stand behind. It won't tell you your analysis is correct; it will guarantee it's reproducible, which is the half agents keep getting wrong.

pip install oryxflow

Best practices for AI-assisted data analysis

Norman Niemer — Wed, 12 Aug 2026 16:09:59 +0000

AI coding agents write plausible analysis fast. The hard part — is it reproducible, and is it right? — hasn't changed. These practices are about making AI-generated analysis you can actually trust.

Ask a coding agent to load a dataset, engineer features, train a model, and compare a few configurations, and it will produce clean, plausible code in seconds. That speed is real and it is worth having. But "plausible code, fast" and "a correct, reproducible analysis you can stand behind" are different bars, and the distance between them is exactly where AI-generated work quietly goes wrong.

The good news: most of that gap is a workflow problem, not an intelligence problem. If you give the agent a structure that makes reproducibility automatic, it stops making a whole class of mistakes — stale intermediates, silent re-runs, results nobody can trace. What that structure cannot do is make your statistics correct. So the practices below split cleanly: the first seven you can hand to your tooling, and the last one you can never hand to anyone.

oryxflow is a small, local-first Python library that turns a data-science script into a cached, dependency-aware graph of tasks — zero infrastructure, no server, no telemetry. Its Claude Code plugin teaches an agent to work inside that structure. Together they enforce most of these practices for you; /oryxflow:init-project is the on-ramp.

1. Structure work as a dependency graph, not a linear script

An agent loses pipeline state across turns. A long linear script gives it no reliable memory of what has already run or whether it is still valid — so it re-derives that picture every turn and gets it wrong. Model the work as tasks with declared dependencies instead. Each step names its inputs, the engine runs them in order, and "what depends on what" is data the agent can read rather than reconstruct.

import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = load_raw()                  # your loader
        self.save(df)

@oryxflow.requires(GetData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()            # upstream output, typed and loaded for you
        self.save(build_features(df))

oryxflow.Workflow(Features, {}).run().outputLoad()

Note there is no file-path plumbing and no to_parquet/read_parquet. The base class you pick (TaskPqPandas, TaskPickle, TaskCachePandas, …) drives type-based, zero-config I/O — the single most common place hand-written analysis code rots.

2. Cache expensive steps, so the careful path is also the fast one

This one is plumbing rather than a promise, but it's what keeps the rest of the list from being a tax. The agent iterates: every turn it might touch feature code, model params, or a plot. If each turn re-pays for the big join and the slow fit, iteration is expensive and the agent starts taking shortcuts — reusing an output it should have rebuilt, skipping the rerun that would have caught the staleness. oryxflow skips any task whose output already exists, so the costly upstream steps run once and every later turn is fast. Cheap iteration is not just ergonomics; it removes the incentive to cut exactly the corners the other seven practices are about.

3. Keep a durable link between code and output

This is the one that bites hardest. The agent edits feature code, forgets to regenerate the saved features, and trains on stale data. Nothing errors. The pipeline runs; the numbers are just wrong. oryxflow watches your task code and, when the body of a step changes, automatically invalidates that step and everything downstream — so the next run recomputes exactly what changed and nothing else. You never evaluate new code on old output, and you never blow away the whole cache to be safe.

4. Keep lineage you can audit

"Which code and which data produced this result?" should have an answer you can look up, not one you reconstruct from memory. oryxflow records run events to .oryxflow/events.jsonl locally — a plain, greppable trail of what ran, when, and why it was (or wasn't) recomputed. When a number looks off, that file is where you start.

5. Verify the rerun actually happened

Do not trust that an edit took effect just because the agent said so. After a change, confirm the affected task actually recomputed — check that its output timestamp moved, or read the lineage trail from practice 4. Agents are confident narrators; "I've updated the features and retrained" is a claim, not evidence. flow.preview() shows you what the engine considers complete versus pending before you run, so you can see the plan and catch a step that should be stale but isn't.

6. Keep exploratory work separate from the pipeline

Early exploratory analysis should stay loose — scratch cells, quick plots, throwaway checks. Don't prematurely formalize it, and don't let it silently become load-bearing either. When a piece of exploration earns its place — you'll rerun it, others depend on it, it feeds a result — promote it into a task deliberately. The plugin's /oryxflow:migrate command does that conversion for you, lifting a notebook or script into cached tasks when it's ready, so the boundary between "playing" and "pipeline" stays honest.

7. Version your data so results are regenerable

Reproducible code on top of a mutable, unversioned dataset is only half-reproducible. Track your data alongside your code so any result can be regenerated from a known state. /oryxflow:init-gitlfs sets up Git LFS for the data directory, so inputs and outputs are versioned the same way the code is.

8. Never outsource judgment — reproducible is not correct

Here is the honesty that the other seven practices exist to protect: no tool makes your statistics correct. oryxflow guarantees that the same code and data give the same result, that stale steps recompute, that lineage is auditable. It does not check that your join keys are right, that your validation is honest, or that your features don't leak the target. Those are judgment, and judgment does not delegate — not to a library, and emphatically not to the agent.

So keep these firmly in human hands, every time:

Sanity-check joins and aggregations. Row counts before and after, spot-check a few keys, confirm the grain is what you think it is. A silent fan-out or dropped rows survives any amount of caching.
Hold out real validation data and keep it untouched until the end. An agent optimizing a metric will happily overfit to whatever you let it see.
Watch for leakage and lookahead. A feature computed with information from the future, or a target that sneaks into the inputs, produces beautiful, reproducible, worthless results.
Read the numbers skeptically. An accuracy that jumped suspiciously, a distribution that shifted, a metric that's too good — treat these as bugs to explain, not wins to ship.

Reproducibility is what lets you investigate correctness efficiently: because the pipeline is stable and traceable, when a number looks wrong you can trust that the code and data in front of you are what produced it. That's the foundation judgment stands on — not a substitute for it.

The practices at a glance

Orchestrators (Airflow, Prefect, Dagster) and experiment trackers (MLflow) are complementary layers here, not competitors — they schedule and record; oryxflow is the local, code-tight inner loop the agent iterates in.

Takeaway

AI makes analysis fast to write. Trust comes from making it reproducible by structure and correct by judgment — two different jobs. Let oryxflow and its plugin enforce the reproducible half automatically, and keep the correctness half where it belongs: with a skeptical human reading the numbers. /oryxflow:init-project sets up the foundation; you bring the judgment.

pip install oryxflow

Frequently asked questions

What are best practices for AI-assisted data analysis?

Structure the work as a dependency graph instead of a linear script, cache expensive steps, keep a durable link between code and output, retain auditable lineage, verify that edits actually reran, separate exploration from the pipeline, version your data, and never outsource statistical judgment. The first seven you can hand to tooling; oryxflow and its Claude Code plugin enforce them, and the last stays with you.

How do I keep AI-generated data analysis reproducible and trustworthy?

Give the agent a structure that makes reproducibility automatic: a caching, dependency-aware task graph that reruns only what a code or data change affects and logs what ran. oryxflow provides code-change invalidation and a greppable .oryxflow/events.jsonl trail, so any result traces back to its inputs. Reproducible is not correct, though, so you still sanity-check joins, hold out validation data, and watch for leakage.

What tools help make AI data analysis reproducible?

Coding agents write the analysis, notebooks display it, and trackers like MLflow record runs, but none guarantee the computation is reproducible. That reproducibility layer is where a local-first caching library fits. oryxflow turns a data-science script into a cached, dependency-aware graph with code-change invalidation and local lineage, and its Claude Code plugin teaches the agent to work inside that structure. Orchestrators like Airflow are a complementary scheduling layer.

Claude Code skills for data science: what they are, and what they aren't

Norman Niemer — Tue, 11 Aug 2026 15:50:07 +0000

A skill is the part of a Claude Code plugin that teaches the agent how to work. For data science, that turns out to be exactly what's missing from "AI writes the analysis fast."

If you've used Claude Code on a data-science project, you know the two feelings that follow each other closely: this is fast, then wait, can I trust what it just did? The agent pulls data, engineers features, fits a model, and plots a result in one session — and somewhere in there it re-runs a ten-minute step it didn't need to, or evaluates a model on an output that went stale three prompts ago. The code looks fine. The number is quietly wrong.

Claude Code skills are the mechanism that closes that gap. This post explains what a skill actually is, how it differs from the other things people call "plugins," and why a skill — not a data connector, not a notebook runner — is the piece that makes AI-written data analysis reproducible.

What is a Claude Code skill?

A skill is a bundle of instructions and conventions that Claude Code loads into its context automatically, when they're relevant — you don't invoke it. Think of it as procedural knowledge the agent picks up the moment it recognizes the situation: "you're editing a pipeline file, so here's how this project expects pipelines to be built and verified."

That auto-activation is the whole point. Instead of you re-explaining your conventions every session, or pasting a style guide into every prompt, the skill supplies them exactly when the agent needs them and stays out of the way otherwise. A skill can carry naming rules, project structure, the right way to use a library, checks to run after an edit — anything that answers how should the agent work here?

Skills vs plugins vs slash commands vs MCP servers

These terms get used interchangeably, and they shouldn't be. They're different layers:

The distinction that matters most for data science: a skill changes how the agent works with what's already on your machine, while an MCP server connects the agent to something external. Reproducibility is a discipline-of-work problem, so it's a skill problem — not something a connector solves.

Why data science needs a skill, specifically

A coding agent's weakness in data work isn't writing pandas — it's invisible state across turns. Agents are effectively stateless between steps; your pipeline is stateful. Over a long session the agent loses track of what's already computed and whether it's still valid, then:

builds on stale intermediates — a feature changed, a cached file didn't, and the model trains on yesterday's data;
loses lineage — nobody can say which code and inputs produced model_final_v3.pkl; -** wastes compute **— a one-line downstream edit re-runs the expensive data pull.

None of these are math errors. They're mechanics-of-the-pipeline errors, and they get worse as the agent writes more of the code. A skill is the right fix because the fix is procedural: check what's already computed before recomputing; verify an edit actually reran what it should have; record what ran and why. That's conventions-of-work — exactly what a skill encodes.

What a good data-science skill does

A reproducibility skill makes the agent a disciplined user of a cache and a lineage log. Concretely, it has the agent:

start each session by reading cache state — pending staleness warnings, recent runs and failures — so it never assumes a stale result is fresh;
verify after each edit that the steps which should have rerun actually did, so a silent miss can't slip through;
answer every staleness or expensive-recompute prompt with the right move — recompute, accept an output-equivalent refactor, or pin instead of guessing;
record decision-relevant results as lineage, so they become the agent's memory across sessions.

A worked example: the oryxflow skill

oryxflow is a local-first Python library that turns scripts and notebooks into a dependency-aware task graph: you declare tasks with parameters and requires() dependencies, and the engine runs them in order and reruns exactly what a parameter, data, or code change affects — while recording what ran, with which code, to a local lineage log. Anything genuinely unchanged loads from disk rather than recomputing, which is what keeps that guarantee cheap.

The oryxflow plugin ships a skill plus slash commands — not an MCP server. The oryxflow skill auto-activates when you work in an oryxflow project and front-loads the correct idioms so the agent never builds on stale data, verifies its own edits actually reran, and reuses finished work instead of recomputing it. The slash commands cover the explicit actions: /oryxflow:init-project to scaffold, /oryxflow:migrate to restructure a loose notebook into a pipeline, /oryxflow:check-standards to keep names and docstrings consistent.

The result is the brand promise, delivered mechanically: trustworthy, reproducible AI data analysis — lineage-tracked by default, and cheaper to run than the loose scripts it replaces.

How to use skills for data science

Skills come packaged in plugins, so you install the plugin once and the skill is simply on from then on:

/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow

After that, describe the analysis you want. The skill works in the background — you don't call it. If you install into an empty directory and nothing happens, scaffold a project first with /oryxflow:init-project so there's a pipeline for the skill to act on.

Can I build my own skill?

Yes — a skill is fundamentally a written set of conventions the agent loads when relevant, so the hard part isn't packaging, it's knowing what good looks like for your domain. For data science that's the reproducibility discipline above: read state first, verify edits, answer staleness prompts deliberately, record lineage. If you'd rather not write that from scratch, the oryxflow skill already encodes it and works on any oryxflow project.

Takeaway

A Claude Code skill is auto-loading know-how — conventions the agent picks up when the context matches, without being invoked. It's distinct from a slash command (explicit action) and an MCP server (external connector).
For data science, the missing piece is reproducibility across turns, and that's a discipline-of-work problem — which makes it a skill problem.
The oryxflow skill is a worked example: it makes Claude Code cache, verify, and track lineage by default. See Claude Code for data science for the full picture, or the best Claude Code plugins for data science for how it fits alongside data connectors and notebook tools.

Frequently asked questions

What are Claude Code skills?

Claude Code skills are bundles of instructions and conventions the agent loads automatically when they become relevant, so you never invoke them. A skill differs from a slash command, which is an explicit action, and from an MCP server, which is an external data connector. Skills change how the agent works with what is already on your machine. oryxflow ships one for reproducible data science.

Is there a Claude Code skill for data science?

Yes. oryxflow ships a Claude Code plugin whose skill auto-activates when you work in an oryxflow project, front-loading the idioms that keep AI-written analysis trustworthy and reproducible: never build on stale data, verify that edits actually reran, and reuse finished work instead of recomputing it. It is a skill plus slash commands, not an MCP server. Install the plugin once and the skill stays on.

What does a data-science skill for Claude Code do?

A data-science skill teaches the agent how to work, not what data to fetch. It has the agent read cache state at session start, verify after each edit that the right steps reran, answer staleness or expensive-recompute prompts deliberately, and record lineage as memory across sessions. The oryxflow skill encodes exactly this reproducibility discipline.

How to make Claude Code a trustworthy data scientist

Norman Niemer — Mon, 10 Aug 2026 16:30:00 +0000

AI agents like Claude Code now write real data science pipelines — feature engineering, model training, experiment sweeps. Here's the honest account of where they fail at it, and why a lightweight workflow library removes exactly those failures.

Coding agents have gotten good at writing pandas and scikit-learn. Ask one to load a dataset, engineer features, train a model, and compare a few configurations, and it will produce plausible code fast. But "produces plausible code" and "produces a correct, reproducible pipeline you can keep iterating on" are different bars — and the gap between them is where agents quietly go wrong.

This post is written from the perspective of the agent. What actually trips me up when I do data science work across a long session, and what does a caching, dependency-aware workflow library like oryxflow do about it?

The core weakness: I can't see state across turns

The thing that makes me error-prone in data work isn't syntax. It's invisible state. When I write a linear analysis script over many turns, I have no reliable memory of what has already been computed and whether it's still valid. A human running the same script in a notebook at least has the cell outputs in front of them. I'm reconstructing that picture from scratch every turn, and I get it wrong in three specific ways:

Stale intermediates. I write features.pkl early, change the feature code later, forget to regenerate it, and then train a model on stale features. No error is raised. The pipeline runs, the numbers are just wrong. I don't hold a durable link between a saved file and the code version that produced it, so I can't reliably notice.
Expensive recompute in my inner loop. My whole working style is run → observe → edit → run. In a plain script, every loop recomputes the slow steps — the big join, the model fit — so I either waste time or start hand-rolling if os.path.exists(...) caches, which then become failure mode #1.
Path and load bookkeeping I get wrong. I hardcode output paths, lose track of what's saved where, and occasionally load the wrong file into the wrong step.

None of these are intelligence problems. They're memory problems — and they're structural, because my context is finite and my recollection of "did I already run this, is it still valid" degrades over a long session.

What a caching DAG does: it externalizes the state I'm bad at holding

A workflow library flips the model. Instead of a script that runs top to bottom, you declare each step as a task with explicit dependencies, and the engine owns execution:

`import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_raw())            # no filename to manage

@oryxflow.requires(GetData)              # declares the edge
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(engineer(self.inputLoad()))

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='gbm')
    def run(self):
        feat = self.inputLoad()
        clf = fit(self.model, feat)
        self.save(clf)
        self.saveMeta({'score': clf.score(...)})

oryxflow.run(TrainModel())`

Look at what this removes for an agent specifically:

The dependency graph is now data, not something I have to remember. The requires edges are the state I would otherwise be reconstructing every turn. I don't have to keep "features feed the model, which feeds the report" in my head — it's declared, and the engine walks it.
Re-running is cheap and correct by default. Run twice and completed tasks load from cache instead of recomputing (3 complete, 0 ran). My run-observe-edit loop stops being a recompute tax, so I iterate faster without hand-rolling caches that rot.
There are no filenames for me to get wrong. self.inputLoad() and output().load() address results by task identity, not by path.
Every task has the same shape. requires + run + save. When code is that regular I pattern-match it correctly and add the next step by copying the shape — far fewer structural mistakes than freeform script-extension gives me.
The unifying idea: the DAG is a memory prosthesis for exactly the thing I'm worst at. A disciplined human gets something from this. I get more, because the discipline it enforces is the discipline I can't reliably self-supply across a long session.

The value scales up with complexity — and it starts at quick EDA

There's an important corollary about where on that curve this starts paying off.

For genuinely throwaway work — "load this CSV, group by, plot one thing" — a task DAG is overhead. Plain pandas in a scratch .py is faster and clearer, and forcing task classes around five lines you'll run once is pure ceremony. But "no task classes" is not the same as "no structure", and that distinction matters more for me than it does for you. When I explore with the oryxflow Claude Code plugin active, the exploration itself is structured: I write a read-only probe inside the project — a small script whose one-line docstring states the question it answers, which prints the answer legibly and runs again next session — instead of a snippet that dies with my context. And whatever it turns up gets written into the project's data doc. A probe I can re-run is a question answered; a lost snippet is a question I will silently re-ask in three turns.

Then, when a probe turns out to be load-bearing — rerun often, depended on, or swept over parameters — I don't rewrite it: /oryxflow:migrate promotes it into cached, parameterized tasks, reading the script as the spec and leaving it in place (walkthrough: migrate a notebook to a pipeline). Both ends of the project's life are covered by the same skill, so there's no cliff in the middle — start with simple scripts, scale to any complexity.

And the calculus inverts as projects get complex — super-linearly. Consider what "complex" actually means in a real data science project and what each trait does to an agent working without a DAG:

Deep dependency chains (ten-plus steps from raw data to final output). The deeper the chain, the more catastrophic a silent stale intermediate near the top is — it corrupts everything below it, and the further downstream the visible output, the less likely I am to trace the wrongness back to the source. Depth is exactly where my "hold the graph in my head" strategy fails hardest, and exactly where declared edges help most.
Expensive nodes you cannot afford to recompute. Real pipelines have steps that are slow and frequently upstream of the thing you're editing: large multi-table joins, model training, walk-forward retraining over an expanding window, computing explainability artifacts, and — increasingly — calls to external LLMs inside a task. Caching these by identity is the difference between a tractable inner loop and one where every experiment costs minutes or dollars. The more expensive the node, the more the cache is worth.
Parameter sweeps and experiment matrices. Serious modeling means comparing a Cartesian product of choices — model type × preprocessing variant × feature set × training window × strategy, and so on. Hand-managing output filenames for that product across a deep chain is combinatorially hopeless, and manually orchestrating one pipeline per configuration is precisely where I introduce ordering and state bugs. A declarative sweep collapses it:

`flow = oryxflow.WorkflowMulti(TrainModel, {
    'ols': {'model': 'ols'},
    'gbm': {'model': 'gbm'},
})
flow.run()
print(flow.outputLoadMeta())   # {'ols': {'score': ...}, 'gbm': {'score': ...}}`

Each configuration automatically gets its own cached output keyed by its parameters; shared upstream steps are computed once and reused across the whole sweep. Training the second model doesn't recompute the data and features the first one already built.

Multiple data sources joined together. When independently-updated sources feed a join, "which source changed, so what's now stale?" is a provenance question I can't answer by memory. The dependency edges make it explicit and mechanical.
Many steps of uniform shape. At thirty-plus tasks, uniformity is what lets me extend the project safely. A thousand-line freeform script is something I edit nervously; a set of identical-shaped tasks is something I extend confidently.

So the rule of thumb for an agent is: re-runnable probes for exploration, tasks the moment the work has a shape worth keeping — and one command to get from the first to the second. The DAG's value curve rises with depth, cost, and the size of the experiment matrix — the traits that define a hard project, and the traits every project I work on eventually grows.

Notice that limits (1) and (2) are not analytical — they're mechanical gaps the library leaves open. Which is the whole point of pairing the library with an agent-side skill — see Claude Code skills for data science for what a skill is and why it, rather than a data connector, is the thing that closes a gap like this.

Library plus plugin: a matched pair

The two residual mechanical risks — verify that an edit's rerun actually happened (the blind-spot net) and get the multi-input wiring right — are precisely what an editor-integrated skill can carry. The oryxflow Claude Code plugin exists for this: it activates when an agent touches pipeline files and front-loads the correct idioms — the session-start events.print_status()habit, the verify-the-rerun check after every edit, answering staleness and expensive-recompute warnings with the right exit (recompute /accept_code / pin), and the right patterns for selecting named inputs from multi-parent tasks.

That produces a clean division of labor:

The library carries the state-tracking an agent is structurally bad at — the dependency graph, the caching, the parameter-keyed reruns, automatic code invalidation with downstream propagation, and the warnings on pinned or expensive tasks.
The plugin carries the remaining disciplines — verifying reruns landed, answering warnings with the right exit, and the multi-input API.

And this pairing gets more valuable as complexity rises, not less — the opposite of most tooling, which buckles under scale. For the full picture of how the library and plugin work together, see Claude Code for data science.

Bottom line

For quick exploration, plain files are fine — a DAG there is just ceremony. What matters is that the plain file is a re-runnable probe living in the project, because exploration rarely stays quick, and one command promotes it when it stops being quick. And for any data science work with a shape — a deep chain, expensive steps, a matrix of experiments, several data sources joined together — a caching, parameter-aware workflow library stops being optional. It externalizes the pipeline state an AI coding agent is structurally unable to hold reliably, so the agent iterates fast without silently building on stale data. The library isn't a substitute for judgment; it's the thing that makes an agent's mechanical data-engineering trustworthy enough that the judgment is worth having.

Frequently asked questions

How do I trust analysis an AI agent wrote — is it reproducible?

Trust comes from structure, not from the agent's confidence. Put the analysis in a DAG that reruns exactly what a code or data change affects and records what ran to a greppable lineage log. oryxflow gives you that: automatic code-change invalidation with downstream propagation, plus a .oryxflow/events.jsonl trail. Reproducible is not the same as correct — the DAG makes a wrong pipeline faithfully reproducible too, so judgment stays yours.

How do I keep an AI agent from building on stale data?
The failure is silent: the agent edits feature code, forgets to regenerate the saved output, and trains on stale data with no error raised. An engine that tracks task code fixes it, because editing a step makes the next run recompute that step and everything downstream automatically. oryxflow does this via source-level code-change invalidation, so you never evaluate new code on old output.

How do I stop an AI coding agent from rerunning expensive steps?

The same task identity that keeps results honest also makes reuse safe: because each step is keyed on its code, inputs and parameters, anything genuinely unchanged can load from disk instead of recomputing, so the big join or model fit runs once rather than every turn. In oryxflow that is automatic — the agent's run-edit loop stops being a recompute tax, which is what stops reproducibility from costing you time.

pip install oryxflow
Source & examples: https://github.com/oryxintel/oryxflow
Docs: https://docs.oryxflow.dev
Build pipelines with an agent: https://github.com/oryxintel/oryxflow-claude-plugin