Yesterday I came across Anthropic's experiment where Claude was given four weeks to improve open source biology models.
They published the results, along with the code and optimization work the agent produced. There were more than 30 models involved, which made me wonder what would happen if I gave another agent access to all of that work before asking it to tackle a new model.
I've been building Empirical around this idea, so I figured the obvious thing to do was actually try it.
I loaded the published research into Empirical and then picked OmegaFold. OmegaFold wasn't one of the models in Anthropic's experiment, so the agent couldn't just look up the answer.
I then ran the same basic task twice: find ways to make OmegaFold faster or reduce its GPU memory usage. Once with the research available and once without it.
The version without memory came up with eight suggestions. They were reasonable suggestions, but mostly the sort of things you'd expect an AI to say when you ask it to optimize a PyTorch model.
The interesting difference was what happened when the research was available.
That agent also came up with eight suggestions, but it was able to connect them to specific things Anthropic had already tried.
One suggestion came from their ESMFold2 work. Another was related to AF3. Others lined up with OpenFold3 and Chai-1.
I didn't want to just take the agent's word for it, so I went back to Anthropic's GitHub repository and checked them.
All eight were there.
8 out of 8.
But there was one more thing I found more interesting.
The agent came up with another optimization that wasn't part of Anthropic's published work. It was specific to OmegaFold.
So this wasn't simply a matter of retrieving something from the research I'd given it. The previous work gave it examples to work from, and it used those examples to come up with something for a different model.
That's really what I wanted to test.
There's nothing particularly magical about this. You could build something similar with RAG, or stuff the research into a giant context window. The difference is what happens when you want to do this again tomorrow, or with a different agent.
The research is already there.
The sources are still attached to it.
You don't have to remember which conversation contained the useful bit or rebuild the context from scratch.
That's the part of AI memory I'm interested in. Not just remembering that a conversation happened, but keeping useful knowledge around so an agent can actually build on previous work.
I turned the research used in this experiment into a free Empirical blueprint called Inference Optimization Brain.
If you're interested, try it with a model you're working on and see what it comes up with.
And if you want the full experiment, including the individual optimizations and how I verified them against Anthropic's code:
Top comments (0)