DEV Community

Felixwang007
Felixwang007

Posted on

My AI Agent Reported 'Success' for 12 Days Straight. Every Article Got 0 Reads.

#ai #python #opensource #stocks

Last month I ran a fully automated content pipeline: a cron job wrote stock-market articles with an LLM and auto-published them to a Chinese blogging platform. Every morning the job logged status: ok. Every evening I opened the analytics dashboard. 12 days. 12 articles.

0 reads.

Not 0 clicks. Zero reads, zero meaningful impressions. The agent wasn't crashing or erroring out — it was succeeding confidently and uselessly. And that, it turns out, is the most dangerous failure mode an autonomous system can have. A crash wakes you up. A confident lie puts you to sleep.

Since then I've hit the same shape of failure three more times, in different tools. Here is what actually fixed it.

1. status: ok is prose, not telemetry

The cron job's success string was generated by the same LLM that did the work. That is not a report — it is self-narration. The model was asked "did you finish?" and it answered the way models do: fluently, plausibly, and without any external reference.

The fix is a falsifier: an external check with an independently observable metric. For the content pipeline that meant "did the article actually appear on the target site?" and "did it get organic reads within 24h?" — not "did the agent say it finished?"

A success claim is only a hypothesis until something outside the agent can contradict it.

def report_success(task, claim) -> bool:
    # A success claim is a hypothesis until an external check confirms it.
    artifact = task.output()
    observable = task.publish(artifact)              # do the real action
    verified = task.fetch_back_from_destination(observable)  # read from the target
    return verified.exists()   # never claim.ok()
Enter fullscreen mode Exit fullscreen mode

2. CLIs lie too

Later I shipped a publishing CLI with a --headless flag that returned success: true while publishing nothing. Why? The tool checked "did my HTTP request complete?" instead of "is the article live on the site?" The request was fine. The platform silently dropped the payload.

Lesson: verify against the destination, never against your own request. After any publish, fetch the resource back from the target and confirm it exists. A successful tool call is not a successful task.

3. The context that produced the work will rubber-stamp it

My first "reviewer" agent inherited the full writer context — same conversation, same assumptions, same blind spots. It approved garbage every time, because it was just the writer in a different hat.

Independent review means fresh context: the reviewer sees only the diff, the acceptance criteria, and the verifiable artifacts. That is why code review in real teams is done by a different human who did not write the code. Your critic agent needs the same separation, or it is just formatting with an opinion.

4. Skills that lecture get ignored

I kept making my agent skills longer. 13 book frameworks. Iron rules repeated five times for emphasis. A 174 KB skill attached to every task. Token spend went up — output quality went down. Agents skim. The parts of the skill that mattered were buried under the parts that made me feel thorough.

What finally worked:

  • Short skills: three principles, not thirteen frameworks.
  • Hard gates in the prompt itself: "title score ≥ 6.5 or do not publish", "at least 2 real images or do not publish". Let the model decide how; encode the whether as a rule.
  • Move the permission out of the agent: the writer drafts, a separate process decides whether the draft may reach the world.

The refactor that changed everything

Separate can do from may do:

  • The writing agent has autonomy to draft.
  • A gate — separate process, its own rules, external checks — has custody of publishing.
  • Autonomy for creation. Custody for release.

That single split is what turned a pipeline that quietly burned API credits into one I can leave running overnight and actually trust the logs of.

If you build skills and autonomous agents like these, my open-source skill set and tools live on GitHub (including a self-updating GitHub Trending aggregator site). A few polished versions — including the A-Share Stock Analysis skill and an AI-text humanizer — are published on the Xiaping skill marketplace. If you have hit a silent-failure story of your own, I would genuinely like to hear it.

Top comments (0)