We kept having the same argument: which model is in production, and why does it behave differently from staging?
The answer was almost never the model file. It was the data.
What we changed:
- Every training run records a hash of the input dataset.
- The serving layer logs the model version and the feature version.
- A small registry maps
model-version -> data-hash -> metrics.
That last piece is just a table, but it ended most of the "why is it different" conversations.
The model is the easy part to version. The data is where the surprises live.
Side quest: I'm learning Zig because I want to understand how the fast inference runtimes are built. If you've read any of them, I'd take recommendations.
Top comments (3)
The data hash idea is underrated. We did the same after a week of confusion.
Model is easy to version; data is where it gets messy. Great writeup.
Curious if you version the feature pipeline too, or just raw data?