DEV Community

Sam Ellis
Sam Ellis

Posted on

Model versioning is a data problem

We kept having the same argument: which model is in production, and why does it behave differently from staging?

The answer was almost never the model file. It was the data.

What we changed:

  • Every training run records a hash of the input dataset.
  • The serving layer logs the model version and the feature version.
  • A small registry maps model-version -> data-hash -> metrics.

That last piece is just a table, but it ended most of the "why is it different" conversations.

The model is the easy part to version. The data is where the surprises live.

Side quest: I'm learning Zig because I want to understand how the fast inference runtimes are built. If you've read any of them, I'd take recommendations.

Top comments (3)

Collapse
 
taylorhacks profile image
Taylor Rivera •

The data hash idea is underrated. We did the same after a week of confusion.

Collapse
 
null_griffin profile image
Finley Griffin •

Model is easy to version; data is where it gets messy. Great writeup.

Collapse
 
logan_works profile image
Logan Jenkins •

Curious if you version the feature pipeline too, or just raw data?