DEV Community

liao-zhizhi
liao-zhizhi

Posted on

Public trial: looking for real agent traces that break doneattr — an open-source diagnostic CLI for false “done” claims (not validated yet)

I maintain doneattr, an open-source diagnostic CLI, and I’m publicly looking for strangers to try it on real traces. To be clear: this is not a product launch, and it is not a claim that the attribution is validated. It is a diagnostic tool, and what I want most is failure feedback from real traces.

What it does: after the question “does this count as done?”, it asks “which reusable decision mode does this false ‘done’ claim belong to?” It outputs one of six mutually exclusive primary attributions plus evidence snippets. One-liner: Attribute false “done” claims to reusable decision modes.

Honest status: two dogfood rounds did not hold; the unclassified rate is still too high. The self-blind agreement only shows the reading is reproducible, not that the tool is validated. So I especially welcome cases where it misattributes, falls into unclassified, produces unreadable evidence, or lacks an adapter.

Best fit: you have agent traces (JSONL/JSON/OTel/Langfuse/harness verdicts) and want to diagnose false_success. It reads traces locally by default; MIT; not on PyPI yet, so clone from GitHub.

Worst fit: production acceptance gates, Web UI, settlement modules, or “accuracy endorsement”. Please don’t use it as an acceptance gate or settlement tool yet.

Try it:

bash
git clone https://github.com/liao-zhizhi/doneattr
cd doneattr
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
doneattr tests/fixtures/verification_skipped.jsonl
doneattr --format json path/to/trace.jsonl
Feedback: reply here or open a GitHub issue. Ideally include: trace type, the output primary_attr, and why you think it is a misattribution or should be unclassified. Please do not upload raw traces containing secrets, personal data, or customer data; redacted output JSON is fine. I can’t promise timely support, but I will read failure cases carefully.

Repo: https://github.com/liao-zhizhi/doneattr

(If this kind of open-source trial recruitment is not allowed by the community rules, please let me know or remove it.)

Top comments (0)