I Built an Agent Skill to Stop AI From Answering the Wrong Question Beautifully

sirsws — Tue, 25 Aug 2026 01:57:03 +0000

AI systems are increasingly good at producing coherent answers. That creates a less obvious failure mode: the answer can be clear, detailed, and persuasive before the real problem has been identified.

The model may optimize a proxy while the underlying reality stays unchanged. It may turn an attractive plan into a commitment before testing its weakest assumption. It may explain an uncertain claim more elegantly than the evidence allows. And it may complete so much of the cognitive work that the user is left with neither an independent judgment nor a concrete action.

I built Judgment Loop as an open Agent Skill for this gap between thinking and acting.

It is not a prompt for making an AI sound deeper. It is a routing and reasoning guardrail for consequential choices, uncertain claims, recurring failures, serious learning, and evidence-based review.

The core idea

When consequences and uncertainty coexist, a useful answer should do more than produce a recommendation. It should make the current judgment falsifiable and keep the final action under the user's ownership.

Judgment Loop asks the agent to find the smallest sufficient path through these questions:

What reality is supposed to change?
Could the chosen metric improve while that reality remains the same?
Which claims are verified facts, reasonable inferences, or untested hypotheses?
What is the strongest reason the current judgment could fail?
What is the cheapest test that separates competing explanations?
What action should close now, and what belief should remain open to revision?

The workflow is deliberately proportional. A small reversible task should stay small. The full loop is reserved for decisions and claims where getting the target wrong has a meaningful cost.

Five modes instead of one giant framework

The skill routes between five modes:

Quick handles an undefined problem or attachment to a particular solution.
Decision examines downside, reversibility, tradeoffs, and exploration budgets.
Research separates mechanism, incremental contribution, evidence, boundaries, and falsification.
Learning checks whether understanding can be reconstructed, challenged, and transferred without AI.
Review compares predictions with results and updates what should be kept, removed, or tested next.

The agent selects the smallest sufficient mode. It should not invoke a ceremony merely because a framework exists.

A simple example

Suppose a user asks:

This paper reports a strong result. Should we integrate the method into production?

A fluent model can easily summarize the paper and produce an implementation plan. But that skips several load-bearing questions:

Was the reported improvement measured on the same target we care about?
Does the evidence establish a causal or production-relevant increment?
What assumptions differ between the paper and our environment?
What is the cheapest isolated experiment that could disconfirm the expected value?
What result would stop the integration rather than merely trigger more tuning?

Judgment Loop should lead with a provisional conclusion, expose the strongest failure reason, and propose an isolated falsification test before recommending production integration.

What the repository contains

The project includes:

a compact runtime SKILL.md;
separate references for decision, research, learning, and review modes;
realistic before-and-after examples;
positive, negative, and boundary trigger evals;
an OpenAI agent interface;
a maintained Simplified Chinese translation;
an MIT-0 license.

The negative trigger cases matter as much as the positive ones. The skill should stay out of simple facts, clear execution, low-risk reversible tasks, freeform creation, and emotional companionship alone.

Install

Using the universal Skills CLI:

npx skills add sirsws/judgment-loop

For a Codex-only global installation:

npx skills add sirsws/judgment-loop --skill judgment-loop -g -a codex -y

The repository is available at:

https://github.com/sirsws/judgment-loop

What I want to learn from publishing it

The main unresolved question is not whether the framework sounds reasonable. It is whether the routing and output contract improve real decisions without adding unnecessary process.

I am looking for concrete counterexamples:

cases where the skill activates but direct action would be better;
consequential cases where it fails to activate;
outputs that label evidence but still hide the load-bearing uncertainty;
situations where the AI completes the reasoning while the user cannot reconstruct or use it independently.

One successful example is only local evidence. If you try the skill, the most useful feedback is a case where its prediction, routing, or stopping condition breaks.