DEV Community: Vishwajitsinh Mahida

"Trace Shape Reveals Model Capability: 3B vs 7B"

Vishwajitsinh Mahida — Wed, 04 Nov 2026 20:59:46 +0000

Trace Shape Reveals Model Capability: 3B vs 7B

Last week I traced the same agent running on two different model sizes. Same code, same task, same prompts. The only difference: the model.

Phone: Qwen2.5-3B (3 billion parameters, running on a Pixel 8 Pro)
Legion: Qwen2.5-7B (7 billion parameters, running on an RTX 4060)

I didn't read a single output. I just counted spans.

What the traces showed

3B model: Every trace was exactly 2 spans. One agent.run, one llm.call. Zero tool calls. Ever.

7B model: Mean 5.8 spans per trace. Multiple llm.call spans, multiple tool.call spans. It actually used tools.

The 3B model wasn't being lazy. It couldn't do tool calls — not reliably, not in this setup. It would try to answer directly, skipping the tool-use loop entirely. The trace shape was degenerate: a straight line from start to finish, no branching, no tool interaction.

The 7B model, same code, same prompts, would explore: call a tool, read the result, call another tool, synthesize. The trace shape was rich — branching, iterative, agentic.

Model         Traces  Mean spans  Tool calls
------------  ------  ----------  ----------
Qwen2.5-3B    6       2.0         0 (degenerate)
Qwen2.5-7B    ~500    5.8         yes (agentic)

Why this matters

You can tell how capable a model is by looking at the shape of its traces, without ever reading the outputs.

This isn't about benchmark scores. It's about behavioral fingerprinting. A 3B model and a 7B model don't just give different quality answers — they behave differently as agents. The 3B doesn't do the agentic loop. The 7B does.

If you're evaluating models for agentic use, don't just look at accuracy. Look at trace shape:

Does it use tools, or try to answer directly?
How many steps does it take?
Does the shape change with task complexity?

A model that can't produce rich trace shapes can't do agentic work, no matter what its benchmark scores say.

The pattern

This is the second time trace shape has told the story:

Prime Agent experiment: Same model, different environments → 3.8x shift in span counts. Environment is input.
This experiment: Same environment, different models → degenerate vs rich traces. Model size determines agentic capability.

In both cases, I never read a prompt or output. The metadata was enough.

That's the thesis behind AgentPulse: trace shape is a behavioral fingerprint. Whether you're detecting environment drift, model regressions, or capability gaps, the shape tells you first.

All traces captured with AgentPulse (metadata-only). No prompts, outputs, or tool inputs were recorded.

We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.

Vishwajitsinh Mahida — Wed, 04 Nov 2026 20:37:09 +0000

This is a submission for the MLH x DEV Writing Challenge

What I Built

I built AgentPulse, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels).

This week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. 3.8x increase from folder contents alone.

The evidence adapter verdict: noise (confidence 0.6) — not regression, because it requires 20 traces/set to claim regression. Correct conservative call on n=10.

What I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust.

Demo

Session    Spans  LLM  Tool  Time   Set
--------   -----  ---  ----  -----  ----
01a1a806   8      4    3     14:26  BASE
01a1a80a   8      4    3     14:30  BASE
01a1a80c   6      3    2     14:32  BASE
01a1a80e   4      2    1     14:34  BASE
01a1a80f   8      4    3     14:36  BASE
01a1a811   6      3    2     14:38  BASE
01a1a813   6      3    2     14:40  BASE
01a1a815   8      4    3     14:42  BASE
01a1a817   4      2    1     14:44  BASE
01a1a819   2      1    0     14:46  BASE
01a1a81a   12     6    5     14:49  CAND
01a1a81d   30     15   14    14:52  CAND
01a1a820   12     6    5     14:55  CAND
01a1a823   24     12   11    14:58  CAND
01a1a826   22     11   10    15:01  CAND
01a1a828   30     15   14    15:05  CAND
01a1a82b   22     11   10    15:08  CAND
01a1a82e   32     16   15    15:11  CAND
01a1a831   36     18   17    15:14  CAND
01a1a834   2      1    0     15:16  CAND

Adapter output: verdict noise, median shift 17.0, IQR ratio 4.5.

Partner Technologies

Didn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology.

Hackathon Experience

Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say noise instead of regression when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__

What 586 Real Traces Taught Us About LLM Agent Determinism

Vishwajitsinh Mahida — Wed, 04 Nov 2026 03:41:00 +0000

What 586 Real Traces Taught Us About LLM Agent Determinism

How temperature selectively degrades agent reasoning — and what that means for observability.

The Setup

On November 3, 2026, we ran 586 traces of a LangGraph folder-research agent against AgentPulse, our metadata-only observability platform. The agent used Ollama with qwen2.5:7b on an RTX 4060 (Legion), plus 6 traces from a Pixel 8 Pro running qwen3b via llama.cpp in Termux.

The experiment: same 10 questions, same corpus, four temperatures (0, 0.3, 0.7, 1.0). We measured trace shapes — span counts, durations, tool calls — to understand how sampling temperature affects agent behavior.

Why this matters: If you're building observability for AI agents, you need to know what "normal" looks like. Without a determinism baseline, every alert is a guess.

Finding 1: Temperature 0 Is Deterministic

At temp=0, qwen2.5:7b produced identical trace shapes across 20 runs per question. Nine out of ten questions had standard deviation 0.0 on span counts. The tenth (Q1) had sd 0.4 — still remarkably tight.

Question	Spans (temp=0)	SD
Q2-Q9 (except Q1)	6-6, 8-8, 2-2, 11-11	0.0
Q1	8-10	0.4

This is the baseline. When temp=0, trace-shape changes are genuine regression signal, not noise. Your alert thresholds can be tight.

Finding 2: Temperature Degrades Complex Reasoning First

Here's what surprised us. Temperature doesn't add uniform noise. It selectively degrades complex reasoning while simple lookups survive.

At temp=0.3:

Q1 (complex): sd 1.7 — already breaking
Q4 (complex): sd 2.0 — breaking
Q3, Q7, Q9 (simple): sd 0.0 — perfect

At temp=1.0:

Q7 (simple): sd 1.6 — finally breaks
Q8 (simple): sd 0.0 — still perfect

The pattern: Multi-file synthesis questions degrade at 0.3. Simple lookups survive to 1.0. The breakdown is question-dependent, not just temperature-dependent.

Implication: Your eval thresholds can't just be "wider at higher temp." They need to be per-question-type. A simple lookup at temp=0.7 should still be deterministic. A complex reasoning task at temp=0.3 is already noisy.

Finding 3: Compare Distributions, Not Means

Our first eval prototype used mean comparison. It got 7/10 "correct" — but the 3 false positives were instructive.

At temp=0.7, traces become bimodal: sometimes the agent does full work (8 spans), sometimes it produces a degenerate 2-span trace. The mean shifts, triggering a "regression" alert. But the median and IQR tell the real story: high variance with overlapping IQR = noise, not regression.

The rule:

Shifted median + similar IQR = regression (alert)
Same median + wider IQR = noise (don't alert)
Bimodal = noise (the agent is inconsistent, not systematically worse)

We validated this on 500 traces. Mean-based detection fails. Median/IQR works.

Finding 4: Three Devices, One Platform

We validated AgentPulse across three very different machines:

Legion (RTX 4060): 580 traces, qwen2.5:7b, ~6s/trace
Dell (i5-1135G7): 72h soak running, synthetic load
Pixel 8 Pro (Tensor G3): 6 traces, qwen3b via llama.cpp, Termux

The phone was the hard one. We fought through:

Rust ABI incompatibilities (uuid_utils on Python 3.13)
Missing SSH client
Model server configuration
Registry label approval

But it works. The phone agent runs, generates traces, and uploads them to the same workspace. Metadata-only observability works on mobile.

What We Built

From these findings, we built the quality-warning evidence adapter — the statistical engine that connects real trace data to AgentPulse's alerting system.

Core logic:

Compare baseline vs candidate using median and IQR (not mean)
Infer question type from baseline shape (simple_lookup vs complex_reasoning)
Adjust thresholds by question type (complex gets 1.5x tolerance)
Classify: no_change, noise, regression, or insufficient_data
Bimodal distributions → noise, never regression

The adapter is live on GitHub (codex/quality-warning-evidence, commit 3df33f0). The API integration follows.

The Takeaway

If you're building agent observability:

Establish your temp=0 baseline first. Without it, you're guessing.
Thresholds must be per-question-type. Complex reasoning degrades faster.
Compare distributions. Means lie when data is bimodal.
Test on real hardware. The phone taught us more than the GPU did.

Temperature doesn't just make agents "more random." It selectively kills complex reasoning first. Your monitoring needs to know the difference.

586 traces. 3 devices. 1 night. The data is real, the findings are publishable, and the code is on GitHub.