I Built an Agent That Refuses to Act — Here's Why Qwen Made It Possible

Demba Fofana — Mon, 20 Jul 2026 19:31:54 +0000

Monday, 9am. Conversion drops 3.2%. The PM pings Slack: "What happened?"
A data engineer queries the warehouse. A data analyst slices dashboards — segment
by segment, device by device. Four hours later: maybe they found it. Maybe the
drop was systemic and they just burned a morning chasing ghosts.
At $200/hour for a data engineer + analyst pair, that's ~$900 per
investigation. This happens 2-3 times a week at any SaaS company with
meaningful traffic. That's $3,200/week — over $150K/year — spent answering
one recurring question: "why did this metric move?"
I built an agent that does the same job in 2 seconds for $0.004.
Powered by Qwen.

What a data team does. Automated.

When a metric drops, here's what your data engineer and analyst actually do:

Query the data. Engineer pulls two snapshots from the warehouse — last week vs. this week, segmented by dimensions. 30 minutes.
Build segment breakdowns. Analyst computes per-segment rates, contribution shares, significance thresholds. 2 hours.
Evaluate significance. Is that drop on the paid channel real, or is the sample too small? 30 minutes.
Check for confounding. Did the rate actually change, or did the population mix shift? Another 30 minutes.
Decide. Act or escalate? 30 minutes. Total: ~4.5 hours. $900. And no guarantee they got it right. Here's what prove-or-abstain does with that same data: POST /investigate { "panel": "clean" } → 2 seconds → ASSERT segment=paid (p < 0.001) → EXECUTE: webhook fires, campaign paused Same raw data. Same conclusion. 22,500 times faster. 50,000 times cheaper.

How it knows when to act — and when to shut up

This isn't a prompt you ask nicely. The agent routes every candidate dimension
(device, country, channel, plan...) through four mathematical gates. All four
must pass before it's allowed to ASSERT a cause:

Material — the drop is at least 2%. Noise isn't worth a human's time.
Localized — one segment accounts for ≥55% of the drop. If every channel dipped equally, no single action fixes it.
Significant — a real two-proportion z-test, p ≤ 0.01. Same perfectly concentrated drop passes on 6,000 users, fails on 60. Your analyst would do the same check — the agent just does it instantly.
Clean — rate and mix effects aren't entangled. No confusing a composition shift with a real rate change. Any gate fails → ABSTAIN. The agent escalates to a human with the exact failing condition named. Not "I don't know." "Interaction share 0.63 > 0.50 — rate and mix effects are entangled; this needs a human." This is the cost-saving mechanism. An ABSTAIN means: I just saved you from acting on a wrong diagnosis. The PM doesn't roll back a deploy. The engineer doesn't revert a feature flag. No one builds a dashboard for a phantom problem. That's where the real money is — not the investigation time, but the false action prevented.

Why Qwen

Your data team costs $200/hour. You can't just throw any LLM at this and expect
it to replace them — you need one that makes reliable tool calls, never invents
dimension names, and produces consistent behavior across model tiers.
Qwen (via Alibaba Cloud DashScope) hit all three:

Tool calling that doesn't hallucinate. Qwen drives the investigation by calling test_dimension(device), reading the gate result, then trying test_dimension(segment) — choosing which dimension to test, but never inventing one that doesn't exist. A determinism guard tests every dimension Qwen skipped, so a lazy LLM can never produce a false "nothing to see here." Every decision traces back to math, not model confidence.
Model interchangeability without cost penalty. Swap qwen-turbo, qwen-plus, or qwen-max — same ASSERT/ABSTAIN verdict. Run the cheapest model, get the same correctness. The benchmark proves it across all three. At $0.40/M input tokens, an investigation averages ~4,500 tokens total — $0.004. That number isn't approximate. It's metered per request and committed in CI.
Mock mode proves the LLM is separable. Run with QWEN_MOCK=1 — zero API calls, deterministic templates. Every verdict is bit-identical. Qwen conducts the investigation. The math decides the outcome. If you audit this agent, you can see the LLM decided nothing. That's the contract you need when you're trusting it to replace two salaries. The one place Qwen genuinely judges? Mapping unfamiliar CSV columns to [metric, dims..., n, c]. Ambiguous column names have no mechanically-deducible answer, so I built a two-pass self-verification system: Qwen proposes a mapping, then re-examines its own work against the sample rows. If it misclassified Temperature as a dimension instead of a covariate, it corrects itself — labeled self_verified: false, auditable and transparent.

The number that should change how you think about AI agents

I ran prove-or-abstain on 10 real public datasets — Titanic passenger manifests,
US college majors with real employment stats, airline passenger counts, penguin
populations, car fuel economy, the Gapminder dataset, and more.
7 out of 10 produced ABSTAIN.
Because in the real world, most metric shifts are genuinely systemic. They don't
localize to a clean, actionable segment. An agent that fabricates a cause on
those seven is a financial liability — it triggers actions that cost more
than the original investigation.
Your analyst might spend 4 hours and still reach the same conclusion:
"this is systemic, escalate." The agent reaches it in 2 seconds. For $0.004.
And it documents exactly why, with a replayable audit trail.
The three that did ASSERT found real, unplanted truths:

Titanic survival localizes to sex=female (p=0.0018) — the "women and children first" effect, not the popular pclass explanation
The STEM employment gap concentrates in majority-women college majors
Penguin body mass variation localizes to species

What this means for your team

Role	Current cost per investigation	With prove-or-abstain
Data engineer (query + prepare)	$100 (30 min)	$0 (automated)
Data analyst (segment + evaluate)	$400 (2 hours)	$0.004 (2 seconds)
False diagnosis risk (rollback, revert)	$500-5,000 per incident	$0 (ABSTAIN prevents it)
Total per investigation	$500-900 + risk	$0.004, guaranteed
Per month (10 investigations)	$5,000-9,000	$0.04

The agent doesn't get tired. It doesn't skip the significance check at 5pm on a
Friday. It doesn't have a gut feeling that the mobile team shipped something
buggy and confirmation-bias its way there. It runs the same four gates every
time and names exactly which one failed.

20/20 benchmark accuracy. 0% false-ASSERT. 0% false-ABSTAIN.
$0.004 per investigation. Docker. MIT license.