DEV Community

Cover image for We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.
Vishwajitsinh Mahida
Vishwajitsinh Mahida

Posted on

We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.

#mlhacks #devchallenge #hackathon

This is a submission for the MLH x DEV Writing Challenge

What I Built

I built AgentPulse, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels).

This week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. 3.8x increase from folder contents alone.

The evidence adapter verdict: noise (confidence 0.6) — not regression, because it requires 20 traces/set to claim regression. Correct conservative call on n=10.

What I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust.

Demo

Prime Agent span comparison

Session    Spans  LLM  Tool  Time   Set
--------   -----  ---  ----  -----  ----
01a1a806   8      4    3     14:26  BASE
01a1a80a   8      4    3     14:30  BASE
01a1a80c   6      3    2     14:32  BASE
01a1a80e   4      2    1     14:34  BASE
01a1a80f   8      4    3     14:36  BASE
01a1a811   6      3    2     14:38  BASE
01a1a813   6      3    2     14:40  BASE
01a1a815   8      4    3     14:42  BASE
01a1a817   4      2    1     14:44  BASE
01a1a819   2      1    0     14:46  BASE
01a1a81a   12     6    5     14:49  CAND
01a1a81d   30     15   14    14:52  CAND
01a1a820   12     6    5     14:55  CAND
01a1a823   24     12   11    14:58  CAND
01a1a826   22     11   10    15:01  CAND
01a1a828   30     15   14    15:05  CAND
01a1a82b   22     11   10    15:08  CAND
01a1a82e   32     16   15    15:11  CAND
01a1a831   36     18   17    15:14  CAND
01a1a834   2      1    0     15:16  CAND
Enter fullscreen mode Exit fullscreen mode

Adapter output: verdict noise, median shift 17.0, IQR ratio 4.5.

Partner Technologies

Didn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology.

Hackathon Experience

Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say noise instead of regression when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__

Top comments (0)