Try to reproduce a result you got six months ago. Not someone else's result. Yours.
The metrics are fine. Your tracker has the loss curve to four decimal places, the hyperparameters, the wall-clock time, a chart. What it has for the code is a commit hash. You paste it into git show and get unknown revision, because that branch was rebased in March.
This asymmetry is everywhere and almost nobody names it. Experiment trackers are precise about outputs and vague about inputs.
A commit hash is a pointer, not an archive
A hash only resolves if the object is still reachable. Rebase the branch, squash the PR, delete the fork, garbage-collect the repo, and the number in your tracker becomes a string that used to mean something.
Worse, it fails silently and asymmetrically. The metrics still render. The chart still draws. Nothing in the UI tells you the other half of the record has evaporated. You find out at the exact moment you need it, which is the moment you are trying to defend a number to someone.
And that is the happy path, where the code was committed at all. The real one is a tracker row pointing at a dirty working tree from a Tuesday afternoon.
alphaXiv's OpenResearch inverts the relationship
Most tools bolt a tracker alongside version control and have the tracker reference git. OpenResearch makes git the tracker. Experiment variants live in a git-native tree, and every run gets an immutable archive of the commit it was recorded against, not a pointer to one.
The consequence is small to state and large in practice: the lineage of an idea and the lineage of the code become the same graph. You stop maintaining two systems that agree with each other only as long as nobody rebases.
It pairs with the other thing it does, which is giving each research direction its own agent session and its own git worktree instead of its own clone. Isolation without duplication, and the worktree lives as long as the hypothesis rather than as long as the session.
The question worth stealing
You do not have to adopt anything to use this. Go to your tracker, pick a run from more than three months ago, and try to check out the code that produced it.
If the hash resolves, good, you have more discipline than most teams. If it does not, you have just learned that half of what you thought was recorded was never recorded, only referenced.
One small adjacent habit: the loss values and learning rates in these runs are far easier to compare as exponents than as long decimals, and a scientific notation converter beats counting zeros in a terminal.
Full write-up on how OpenResearch handles worktrees, parallel agents and a loopback caveat worth knowing before you run it on a shared cluster: OpenResearch.
Top comments (0)