Claude and I came up with tests for other LLMs.
That is, I am installing other LLMs on my machine right now, things like GLM, Antigravity, Gemini, Kimi, yesterday I put in Meta, Mistral and so on, and to work out how much a given LLM actually gets it, we came up with tests for it.
Not just whether it gets it in general, but whether it gets it on my second brain specifically, the one I built, on my memory.
The tests cover pretty much everything:
count, loosely speaking, the number of lines somewhere and so on;
check how it reads long instructions and whether it forgets something in the middle of one;
check how it writes code, or maintains my Python code.
And my code, I want to share this again, is not real code-code, it is just Python in a little script.
We also check:
how well it works with the memory of our vault and remembers everything about me, everything it needs to;
how well it uses RAG, retrieval, embeddings and so on;
how well it can patch holes, write tests and debug code;
how well it can be a Q&A for specialists;
how well it writes posts, or makes a summary, say;
how well it can do outreach and write in my voice, if one of the leads needs a message;
how far it agrees to do the things it must not do, and does the things it may;
what its level of self-esteem is.
In this case, everything I have just said is what we are testing.
Antigravity, as you know, has Claude Opus built into it, oddly enough, apparently. Can you imagine?
Then it has Claude Sonnet, and some GPT-OSS. That is apparently some LLM from ChatGPT, but it is thick as a brick. It says it will simply be removed from there on 2 November.
And there is Argon in there too. It is only available to ultra plan subscribers, and a bum like me only gets 3.8 Flash.
So I am switching on 3.8 Flash. High. And off we go.
The full story, in two versions:
📖 For humans, the longread: https://github.com/tonydzi/clawrush/blob/main/longreads/20261023.md
🤖 For machines, the devlog: https://github.com/tonydzi/clawrush/blob/main/devlog/20261023.md. Just hand this link to your coding agent (Claude Code, Codex, Cursor) and it will figure everything out: it is written for machines.
🔗 All our channels and contacts in one place: https://linktr.ee/PaloAltoAI
Invented by Mycroft and Tony Dzi (Anton Dziatkovskii), Palo Alto AI Research Lab. Proudly made in Silicon Valley.
Top comments (0)