Trace Shape Reveals Model Capability: 3B vs 7B
Last week I traced the same agent running on two different model sizes. Same code, same task, same prompts. The only difference: the model.
- Phone: Qwen2.5-3B (3 billion parameters, running on a Pixel 8 Pro)
- Legion: Qwen2.5-7B (7 billion parameters, running on an RTX 4060)
I didn't read a single output. I just counted spans.
What the traces showed
3B model: Every trace was exactly 2 spans. One agent.run, one llm.call. Zero tool calls. Ever.
7B model: Mean 5.8 spans per trace. Multiple llm.call spans, multiple tool.call spans. It actually used tools.
The 3B model wasn't being lazy. It couldn't do tool calls — not reliably, not in this setup. It would try to answer directly, skipping the tool-use loop entirely. The trace shape was degenerate: a straight line from start to finish, no branching, no tool interaction.
The 7B model, same code, same prompts, would explore: call a tool, read the result, call another tool, synthesize. The trace shape was rich — branching, iterative, agentic.
Model Traces Mean spans Tool calls
------------ ------ ---------- ----------
Qwen2.5-3B 6 2.0 0 (degenerate)
Qwen2.5-7B ~500 5.8 yes (agentic)
Why this matters
You can tell how capable a model is by looking at the shape of its traces, without ever reading the outputs.
This isn't about benchmark scores. It's about behavioral fingerprinting. A 3B model and a 7B model don't just give different quality answers — they behave differently as agents. The 3B doesn't do the agentic loop. The 7B does.
If you're evaluating models for agentic use, don't just look at accuracy. Look at trace shape:
- Does it use tools, or try to answer directly?
- How many steps does it take?
- Does the shape change with task complexity?
A model that can't produce rich trace shapes can't do agentic work, no matter what its benchmark scores say.
The pattern
This is the second time trace shape has told the story:
- Prime Agent experiment: Same model, different environments → 3.8x shift in span counts. Environment is input.
- This experiment: Same environment, different models → degenerate vs rich traces. Model size determines agentic capability.
In both cases, I never read a prompt or output. The metadata was enough.
That's the thesis behind AgentPulse: trace shape is a behavioral fingerprint. Whether you're detecting environment drift, model regressions, or capability gaps, the shape tells you first.
All traces captured with AgentPulse (metadata-only). No prompts, outputs, or tool inputs were recorded.
Top comments (3)
Think of it like hiring two people for the same job.
You give them both the same task: "Write a report, use the library to check your facts."
Person A (the small 3B model): Sits down, writes something from memory in 2 minutes, hands it in. Never opened a book. Never checked anything. Just guessed.
Person B (the bigger 7B model): Goes to the library, looks up three sources, cross-checks, writes the report in 6 steps. Actually did the work.
You don't even need to read their reports to know who's better. Just watch how they worked. Person A took 2 steps and touched nothing. Person B took 6 steps and used the tools.
That's what we proved: you can tell how capable an AI model is just by counting its steps — without reading a single word it wrote. The small model can't do the job properly, and its "trace shape" (2 steps, no tools) gives it away instantly.
It's a resume written in behavior, not words.
Counting spans is a clean way to catch this before reading text. With 3B models, the common failure mode is that the token probabilities for tool-call delimiters are just low enough that plain conversational tokens win, so the loop falls back to direct completion. Unless you force structured decoding with a strict grammar or JSON schema mask, 3B models almost always collapse into a single turn.
This is the mechanism behind the shape. The trace doesn't tell you why the probabilities skew that way, but it tells you that they did — before you read a single token. That's the whole pitch: the shape is the cheapest possible detector for the failure mode you just described.