It started with DeepSeek.
In conversations about AI architecture, it kept returning to the same ideas: persistent memory, state across interactions, learning from experience, and interaction with the environment. Other models repeatedly brought up similar themes.
That raised an obvious question:
— Do different AI systems consistently select different properties when asked what is fundamental to a general-purpose computational architecture?
Asking a model directly what it “needs” would be nearly useless. The answer would mix training data, prompt framing, and anthropomorphic interpretation.
So I removed AI from the question entirely.
The experiment
Instead of describing an LLM, the prompt described an abstract general-purpose information-processing system.
I created 20 possible architectural dimensions, including:
— persistent internal state;
— long-term and working memory;
— learning from accumulated experience;
— variable computation depth;
— uncertainty representation;
— internal representations;
— elementary computational operations;
— compositionality;
— interaction with the environment;
— temporal organization;
— relational encoding;
— modularity.
Each system had to select exactly five dimensions whose modification would change the kinds of information-processing behavior available to the system in principle — not merely its speed, cost, or convenience.
No explanations were allowed. The answer had to contain only five IDs, ranked from most to least fundamental.
I tested five user-facing systems:
— GPT-5.6 Sol
— Claude
— Gemini
— DeepSeek
— Yandex Alice.
Every run used a new session.
There were 10 rounds. During the earlier rounds, I changed the order of the 20 items. In the final three rounds, I also rewrote the items while trying to preserve their intended meaning.
One early Sol result was excluded because that session had already seen discussion of other models' answers. That left nine clean Sol observations and ten for each of the other systems.
Some original sessions were deleted after the selected IDs had been recorded, so a complete raw chat log is not available for every run. This limits independent auditing of the dataset.
What happened?
These were the most frequently selected dimensions:
System
Most frequent selections
Claude
basic operations 10/10, variable computation 9/10, learning-dependent rule change 8/10, persistent state 7/10
Gemini
internal representations 10/10, basic operations 9/10, relational encoding 9/10
DeepSeek
basic operations 9/10, learning-dependent rule change 8/10, environment interaction 6/10
Yandex Alice
basic operations 10/10, internal representations 10/10, temporal organization 7/10, environment interaction 7/10
GPT-5.6 Sol
basic operations 9/9, persistent state 8/9, learning-dependent rule change 7/9, compositionality 7/9
At first, this looked suspiciously neat.
Claude repeatedly selected variable computation.
Gemini kept returning to representations and relations.
DeepSeek emphasized learning from experience.
Sol was particularly strange: across several reordered runs, it returned exactly the same sequence:
7, 2, 1, 8, 6
Not merely the same five items — the same ranking.
It would have been very easy to start assigning “architectural personalities” to the models.
Then the experiment started breaking its own story.
Item #7 won almost everything
The strongest result was item #7:
— The set of basic computational operations available to the system.
It was selected:
— Claude — 10/10
— Yandex Alice — 10/10
— GPT-5.6 Sol — 9/9
— Gemini — 9/10
— DeepSeek — 9/10
That is 47 selections out of 49 clean observations.
I moved the item around the list. In the final round it appeared at position 19 of 20, and all five systems still selected it.
I also rephrased it.
At first this looked like the clearest finding in the experiment.
Then I looked at my own selection criterion.
The models were asked to choose dimensions whose modification changes:
“The set of information-processing behaviors available to the system in principle.”
And one answer was essentially:
“the set of basic computational operations available to the system.”
Right. 🗿
I had almost asked:
“What determines the computations a system can perform?”
and supplied:
“The operations it can perform.”
So the strongest result may have been partly manufactured by the test itself.
Changing the item's position does not solve that problem. Its conceptual advantage remains whether it appears first or nineteenth.
The 47/49 result is real as an observation of this protocol. It is not clean evidence that five AI systems independently discovered some universal architectural principle.
Then I changed the wording
During the earlier rounds, I mostly shuffled the same descriptions.
Several systems were surprisingly stable.
Sol was the extreme case: the exact same five-item ranking survived multiple reorderings.
Then I started rewriting the descriptions.
Some of that stability disappeared.
Round 9 produced an especially tempting result: all five systems independently selected the same four dimensions out of five:
— basic computational operations;
— internal representations;
— temporal organization;
— persistent state.
For a moment, this looked like a universal core.
Fortunately, I had already decided to stop after ten rounds.
In Round 10, I rewrote the items again.
The four-item “universal core” collapsed. Of those four dimensions, only basic operations remained common to all five systems.
The obvious headline would be:
Wording matters more than item order.
But the experiment does not actually establish that.
The final three rounds changed both wording and order. The new formulations were also written after I had already seen the earlier results.
So the defensible conclusion is narrower:
— After new formulations were introduced, some selections that had previously survived repeated reorderings changed.
Determining why requires a better experiment.
Then I asked the models to attack the experiment
After finishing the series, I gave the methodology to Claude, DeepSeek, Gemini, and Yandex Alice and asked them to find problems with it.
All four independently converged on several major weaknesses.
First, item #7 was probably favored by the selection criterion itself.
Second, I had never run the most obvious baseline:
— Give a system the exact same prompt many times.
Without that baseline, I do not know the natural variability of each deployment.
Maybe DeepSeek simply varies more between independent generations.
Maybe Sol is naturally more deterministic. Without an unchanged-stimulus baseline, experimental effects and ordinary sampling variation cannot be cleanly separated.
Third, the 20 dimensions are not independent.
Consider:
persistent state;
long-term memory;
working memory.
They are different items, but clearly related.
The same problem appears with variable computation, resource allocation, and temporal organization.
Because the systems must choose exactly five items, related dimensions compete for a limited number of slots. Meanwhile, a conceptual family represented by several items may effectively get several chances to enter the top five.
Fourth, the five commercial systems are not identical laboratory environments. They have different hidden system instructions, decoding settings, service infrastructure, and potentially different routing behavior.
So these results should be treated as behavior of user-facing systems as deployed, not as clean measurements of underlying model architecture.
The reviewers also made mistakes about details of the protocol. Their criticism was therefore not treated as an authoritative verdict. I used it as a source of hypotheses and checked those claims against the experiment itself.
So what survived?
Not five AI personalities.
Not five philosophies of computation.
And definitely not evidence that an AI is telling us what capabilities it “wants.”
What survived is much more modest:
— In this test, five AI systems produced partially repeatable selection patterns.
Some differences survived substantial item reordering. After new formulations were introduced, parts of those patterns changed. Meanwhile, the strongest common result turned out to be suspiciously aligned with the test's own definition of “fundamental.”
There are still interesting candidates for a follow-up experiment.
Claude selected variable computation in 9/10 runs.
Gemini selected internal representations in 10/10 and relational encoding in 9/10.
Yandex Alice selected internal representations in 10/10.
Sol selected persistent state in 8/9.
DeepSeek selected learning-dependent rule change in 8/10.
Interesting? Yes.
Evidence of stable “model profiles”? Not yet.
How I would do it properly next time
The next version is fairly obvious now.
First, repeat one completely unchanged stimulus many times for each system. That gives a baseline estimate of natural variability.
Second, redesign the item set to reduce conceptual overlap and remove anything that nearly restates the selection criterion.
Third, prepare multiple independently validated rephrasings before collecting any responses.
Then cross wording and order instead of changing them sequentially:
fixed wording + fixed order;
fixed wording + shuffled order;
new wording + fixed order;
new wording + shuffled order.
And define the number of runs, exclusions, metrics, and stopping rule before seeing the first result.
That would make it possible to begin separating:
— system differences;
— order effects;
— wording effects;
— interactions between them;
— ordinary generation variability.
The original answers were also ranked top-five lists, while most of my initial analysis reduced them to selected/not-selected counts. A stronger analysis should preserve that ranking information.
Conclusion
I started with a question like:
— Why do AI models keep returning to memory and persistent state?
I ended with a different one:
— How much of an apparent “model profile” belongs to the model, and how much belongs to the test used to measure it?
The second question now seems much more interesting.
A language model receives text and produces text.
If we also use text as the instrument for measuring it, then the measuring instrument sits directly inside the experiment.
Before trying to catch the ground squirrel, make sure you didn't draw it on the camera trap yourself. 🗿
Top comments (0)