DEV Community: shizhe Lim

Stop Asking One Model to Do Everything: A Four-Layer Pattern for AI Workflows

shizhe Lim — Fri, 23 Oct 2026 09:48:31 +0000

OpenAI’s Decisions API became generally available this week. The interesting part is not simply that there is another API endpoint. It is the architectural distinction the release makes explicit:

Some AI tasks need generation. Many only need a bounded decision.

Asking the same general-purpose model to classify a request, select an action, draft a response, and approve the result may be convenient during prototyping. In production, however, those jobs have different requirements and failure modes.

A more testable AI workflow can be divided into four layers.

1. Deterministic rules for hard constraints

Some decisions should not depend on probabilistic inference.

Permissions, contractual restrictions, data-residency requirements, spending limits, and known security policies belong in normal application logic. A model can help identify relevant context, but it should not silently override a hard rule.

If a condition must always produce the same result, start with code.

2. Bounded models for repetitive judgments

Classification, prioritization, routing, and scoring are different from open-ended generation.

A bounded decision layer receives evidence and returns a constrained result: a probability, one choice from a known set, or a score against a rubric. This makes the output easier to validate and easier for the rest of the application to consume.

Typical examples include:

Routing a support request to the right queue
Scoring whether an agent action needs review
Choosing which model should handle the next step
Prioritizing documents for human inspection

This layer still needs evaluation. A probability is not automatically well calibrated, and a confidence threshold should be derived from representative application data rather than selected arbitrarily.

3. Generative models for open-ended work

Use generative models where the output space is genuinely open: drafting an explanation, synthesizing several documents, writing code, or planning a multi-step task.

The goal is not to avoid larger models. It is to reserve them for work where their broader capabilities provide value.

A simplified workflow might look like this:

if request violates a hard policy:
    block or escalate

decision = classify_and_route(request)

if decision is low-confidence or high-impact:
    send to human review
else:
    use the selected model and workflow

4. Human review for consequential uncertainty

A human-review path should be part of the architecture, not an emergency patch added after deployment.

Review is especially important when an action is irreversible, affects another person, accesses sensitive data, or falls outside the examples used during evaluation.

Evaluate the whole route

When testing this architecture, do not measure only raw model accuracy. Track:

Decision calibration
End-to-end latency
Cost per successfully completed workflow
Escalation and fallback rates
Performance drift as real inputs change
Whether each decision can be audited

The emerging lesson is that “Which model should we use?” is becoming the wrong first question.

A better question is: What kind of inference does each step actually require?

References

AI assistance disclosure: This article was drafted with AI assistance. The author reviewed and edited the final version and verified its factual claims against the linked primary sources.

The Missing Component in Agentic Software Engineering: A Machine-Checkable Oracle

shizhe Lim — Fri, 16 Oct 2026 06:23:05 +0000

A recent software-engineering experiment reportedly consumed more than 500 billion tokens over three months.

That number attracted attention, but it was not the most important result.

The project used a fleet of AI coding agents to reconstruct a large application in C++. After the first month, the output looked promising: the code was readable, the application launched, and visible features worked.

However, much of the implementation was semantically wrong.

Functions used incorrect signatures. Data structures were misinterpreted. Logic was invented or removed. Some simple operations were replaced with unnecessarily expensive abstractions.

The agents were producing activity, but the team had no reliable definition of correctness.

Why a Reviewer Agent Was Not Enough

The initial workflow included worker agents and a reviewer agent.

That sounds reasonable, but both sides were operating from the same incomplete description of the goal. When a worker explained why it had made a change, the reviewer could accept that explanation instead of independently verifying the behavior.

This exposes a common weakness in multi-agent systems:

Adding another model does not automatically create independent evidence.

A reviewer can find obvious defects, but it cannot consistently enforce requirements that have never been translated into objective checks.

The System Changed When It Gained an Oracle

The team eventually introduced an automated verification harness.

Each reconstructed function was compared with the corresponding reference output. The result was deliberately simple: PASS or FAIL.

That feedback loop made the workflow much more reliable:

An agent produced an implementation.
The verifier checked it independently.
Failed work returned to the agent with concrete feedback.
Passing work was recorded and checked again in CI.
The verification mechanism itself was protected from modification.

The important component was not another prompt. It was an oracle: an external system capable of deciding whether an output satisfied the requirement.

What Other Agent Workflows Can Learn

Most projects cannot compare compiled functions byte for byte, but the same pattern applies elsewhere.

For coding agents

Use tests, static analysis, type checks, security scans, performance budgets, and reproducible builds.

For data agents

Validate schemas, row counts, invariants, allowed ranges, lineage, and reconciliation totals.

For customer-support agents

Check policy compliance, required fields, escalation conditions, tool permissions, and whether cited account data actually exists.

For research agents

Require source availability, citation support, date checks, contradiction detection, and explicit reporting of missing evidence.

For workflow agents

Validate state transitions and permissions before allowing irreversible actions.

A Minimal Agent Verification Loop

A production workflow can start with a simple structure:

define acceptance criteria
set a cost and retry budget

while budget remains:
    generate or revise the result
    run an independent verifier

    if verification passes:
        accept the result
        record the evidence
        stop

    return structured failure feedback

escalate to a human or fail safely

The verifier should ideally be:

Independent from the generating agent
Deterministic where possible
Protected from modification
Easy to audit
Able to fail closed
Connected to cost and retry limits

Measure Verified Outcomes, Not Agent Activity

Token usage, tool calls, generated files, and completed subtasks can all look like progress.

None of them proves that the result is correct.

A more useful operational metric is:

total workflow cost / number of verified successful outcomes

This captures an important tradeoff. A cheaper model is not cheaper if it generates more failures, consumes more retries, or creates additional human review work.

At the same time, a strong verification harness may make less expensive models viable because they receive precise feedback.

The Bigger Lesson

Model capability still matters, but production Agent systems need more than intelligence.

They need boundaries, evidence, protected evaluation, retry controls, and a clear path for human escalation.

Before scaling the number of agents, define the mechanism that tells them—and your team—when the work is actually correct.

What machine-checkable oracle could you add to your current AI workflow?

Sources:

Maurice Heumann, “500+ Billion Tokens Later: Letting AI Agents Decompile a First-Person Shooter”
Tom’s Hardware coverage published October 12, 2026

AI disclosure: This article was prepared with AI assistance. Before publication, I reviewed the cited material and take responsibility for the final text.

Rules Engine vs. Decision Model vs. General LLM: A Practical Routing Framework

shizhe Lim — Wed, 07 Oct 2026 09:14:45 +0000

Choosing an AI model is becoming less important than deciding when each model—or non-model system—should be used.

OpenAI’s release of the Decisions API is one recent signal of this shift. Instead of generating an open-ended response, a decision endpoint can return structured predicates, choices, or scores.

But a dedicated decision model should not automatically replace rules engines or general-purpose LLMs. Each belongs in a different part of the stack.

1. Use rules when the decision is deterministic

A rules engine is usually the right choice when:

The policy is explicit and stable.
The inputs are already structured.
The result must be fully explainable.
A wrong decision creates compliance or financial risk.

Examples include permission checks, spending limits, required fields, and geographic restrictions.

Adding a model to these decisions may increase latency and introduce uncertainty without creating additional value.

2. Use a decision model when the input is ambiguous but the output is constrained

A decision model fits tasks such as:

Intent classification
Priority scoring
Tool selection
Model routing
Moderation triage
Mapping unstructured requests to a fixed set of actions

The input may be natural language or an image, but the output remains controlled.

The important production question is not simply whether the model returned a valid label. Teams also need to define:

A minimum confidence threshold
A fallback for low-confidence results
An evaluation dataset
Version and prompt tracking
Logging for later review

3. Use a general LLM when the task requires open-ended reasoning

A general-purpose model is more appropriate when the system must:

Compare incomplete evidence
Produce a detailed explanation
Plan multiple steps
Resolve an unfamiliar situation
Generate original content

These tasks benefit from broader reasoning, but they also tend to cost more and take longer than narrow classification or scoring.

A simple hybrid router

def route(request):
    if matches_deterministic_policy(request):
        return run_rules(request)

    decision = classify_request(request)

    if decision.confidence < 0.80:
        return run_reasoning_model(request)

    if decision.action_is_irreversible:
        return request_human_approval(decision)

    result = execute(decision)
    return verify_outcome(result)

The exact threshold should come from evaluation data, not from copying 0.80 into production. Different actions can also use different thresholds.

A support-ticket label might tolerate a lower threshold than a payment, account deletion, or infrastructure change.

Routing is only half of the system

A router can choose the correct model or tool and the workflow can still fail.

Production systems should verify the resulting state:

Did the selected tool actually run?
Did the database reach the required state?
Were unintended side effects created?
Should the workflow retry, escalate, or stop?

A successful API response is not necessarily a successful business outcome.

A practical selection rule

Use:

Rules for known policies
Decision models for constrained choices over ambiguous inputs
General LLMs for open-ended reasoning
Humans for high-impact or irreversible decisions
Outcome checks after every state-changing action

The goal is not to find one model that handles everything. It is to build a system that assigns each request to the most appropriate decision mechanism—and can explain, evaluate, and reverse that decision when necessary.

How are you implementing routing today? Are you using deterministic rules, an LLM classifier, a specialized decision model, or a hybrid system?

AI disclosure: This article was prepared with AI assistance and reviewed by the author for technical accuracy and final judgment.

Always-On AI Agents Need a Permission Architecture, Not Just Better Prompts

shizhe Lim — Tue, 06 Oct 2026 08:56:29 +0000

AI agents are moving from request-response tools to systems that can keep working after the user leaves.

That shift changes the security model.

A chatbot produces text for a person to review. An agent may read email, edit files, call an API, deploy code or initiate a transaction. Once a model can take actions, prompt quality is no longer enough.

The system needs a permission architecture.

Prompts are not access control

Instructions such as “never send an email without permission” are useful, but they are not a reliable security boundary.

The model is still interpreting language. It may encounter ambiguous instructions, untrusted content or conflicting context. A privileged action should therefore be enforced outside the model by a deterministic execution layer.

The model may propose an action. The surrounding system should decide whether that action is allowed.

1. Give each agent its own identity

Avoid letting multiple agents share one broad service account.

A separate identity makes it possible to answer basic operational questions:

Which agent requested the action?
Which user or workflow initiated it?
What resources was it allowed to access?
Can its access be revoked without affecting other agents?

Agent identity should be treated like workload identity, not like a convenient shared API key.

2. Separate reading from writing

Reading a calendar and deleting an event are not equivalent permissions.

Tool definitions should separate capabilities according to their consequences. For example:

email.read
email.draft
email.send
email.delete

An agent may receive read access by default while send or delete operations remain behind an approval step.

This also makes logs and security reviews much easier to understand.

3. Gate actions by consequence

Not every tool call needs human approval. Requiring confirmation for everything removes much of the value of automation.

Approval should be based on impact.

A useful starting point is:

Reversible, internal action: allow automatically
External but recoverable action: allow with notification
Irreversible or high-impact action: require approval
Action outside the agent’s defined scope: deny

The policy should live in code or configuration, not only in the system prompt.

An illustrative policy could look like this:

action: send_external_email
default: deny
requires:
  - scoped_agent_identity
  - human_approval
log:
  - initiating_user
  - destination
  - proposed_content
  - policy_decision

4. Keep an audit trail that explains actions

Logging only the final API request is not enough.

A useful audit record should connect:

The initiating user or scheduled workflow
The agent identity
The requested objective
The tool and resource accessed
The policy decision
The resulting state change

The goal is not to store every hidden reasoning token. It is to preserve enough evidence to reconstruct what the system attempted, what was authorized and what changed.

5. Design revocation and rollback before launch

Every long-running agent needs a clear stop mechanism.

Teams should be able to:

Revoke the agent’s credentials
Disable an individual tool
Cancel active work
Restore affected data where possible
Prevent retries after a failed or denied action

A kill switch without tested recovery procedures only solves half of the problem.

6. Avoid coupling permission policy to one model

Model providers and model versions change. Permissions should not have to be redesigned every time the underlying model changes.

Keep these layers separate:

Model selection
Business workflow
Tool execution
Permission policy
Audit and recovery

This separation also makes it easier to evaluate another model without accidentally changing the agent’s authority.

A pre-launch checklist

Before enabling an always-on agent, ask:

Does it have a unique identity?
Are read and write permissions separated?
Are irreversible actions approval-gated?
Can every external action be traced to a user and policy decision?
Can access be revoked immediately?
Has rollback actually been tested?
Can the model be replaced without rewriting the permission layer?

The most capable agent is not automatically the most useful one.

For production systems, trust comes from knowing what the agent can do, what it cannot do and exactly where it must stop and ask.

Which action in your current workflow would require a mandatory human approval gate?

AI disclosure: This article was developed with AI assistance and reviewed by the author before publication.

Managed Agent Runtimes Still Need a Control Plane

shizhe Lim — Fri, 02 Oct 2026 03:41:22 +0000

Hosted agent runtimes are making a lot of infrastructure feel like a solved problem.

Long-running sessions, context compaction, tool execution, sandboxes, and subagent coordination can now be consumed as platform features. That is useful — especially for teams that would rather spend their time on a product workflow than on maintaining an agent loop.

But I think we should separate the agent runtime from the agent control plane.

A runtime is where work executes. A control plane is where a team can answer operational questions after something goes wrong.

For a production agent, I would want the control plane to own at least:

Execution journal: inputs, tool calls, outputs, and decisions that can be inspected later.
Replay and recovery: a failed run should be resumable or reproducible without guessing what happened.
Approval gates: side-effecting actions need policy and a clear human-in-the-loop path.
Evaluation corpus: production failures should become regression cases.
Model-routing policy: different steps may need different models, latency budgets, or cost ceilings.

A hosted runtime can still be the right choice. It may reduce operational burden dramatically. But “managed” should not mean “opaque.”

My rule of thumb: let a platform operate the commodity infrastructure; keep evidence, governance, and decision policy portable enough that your team can investigate and improve the system.

How are you drawing this boundary? Is there a component you would never hand to a hosted agent platform?

Designing Permission Boundaries for Production AI Agents

shizhe Lim — Wed, 30 Sep 2026 09:47:27 +0000

AI disclosure: This article was prepared with AI assistance. The author reviewed and edited the technical content and remains responsible for its accuracy.

AI agent permissions are often treated as a binary decision:

The agent can use a tool
The agent cannot use a tool

That may be sufficient for a prototype. It is usually too coarse for a production workflow.

Reading a customer record, drafting an update, saving a draft, and sending the final message may all involve the same system. However, they do not have the same operational risk.

A safer design separates capability from authority.

A four-layer action model

One practical approach is to classify actions into four layers.

1. Observe

The agent reads information without changing external state.

Examples:

Retrieve a document
Read an API response
Inspect a repository
Query a customer record
Review system status

Read access should still be scoped. An agent rarely needs access to every record, directory, API field, or historical event.

2. Prepare

The agent produces an output that has not yet affected another system or person.

Examples:

Draft an email
Generate a SQL query
Propose a code change
Prepare a support response
Create an execution plan

This layer is useful because it separates reasoning from execution. A person or another policy component can inspect the proposed action before anything changes.

3. Act reversibly

The agent changes state, but the result can be reviewed and undone with limited cost.

Examples:

Create a pull request
Save a message as a draft
Add an item to a review queue
Create a temporary resource
Update a record with version history

Reversible does not mean risk-free. The system still needs validation, logging, retry controls, and a defined rollback path.

4. Commit irreversibly

The agent performs an action that is difficult, expensive, or impossible to reverse.

Examples:

Send an external message
Merge a pull request
Delete production data
Publish public content
Submit a payment
Change access permissions

These actions often need the strongest policy checks and, depending on the workflow, explicit human approval.

Put a policy check before every tool call

Permission decisions should be based on more than the tool name.

The same tool may support both low-risk and high-risk actions. For example, an email integration might read a message, save a draft, or send a final response.

A simplified policy layer might look like this:

ACTION_LEVELS = {
    "read_record": "observe",
    "draft_message": "prepare",
    "save_draft": "reversible",
    "send_message": "irreversible",
}

def authorize(action, context):
    level = ACTION_LEVELS[action]

    if not context.user_has_access:
        return False, "User does not have access"

    if not context.input_valid:
        return False, "Input validation failed"

    if level == "irreversible" and not context.human_approved:
        return False, "Human approval required"

    return True, "Authorized"

A production policy will be more detailed, but the important idea is simple:

The agent should not decide its own authority merely because it knows how to call a tool.

Make retries safe

Agent workflows frequently retry after timeouts, interrupted execution, or uncertain responses.

Without retry protection, an agent may repeat an external action even though the first attempt already succeeded.

Useful controls include:

An idempotency key for each intended side effect
An expected resource version
A record of the previous attempt
A clear completion state
A rollback or compensation reference

For example:

action_request = {
    "action": "send_message",
    "idempotency_key": "case-1842-final-response",
    "expected_version": 7,
    "requires_approval": True,
}

The idempotency key represents the intended action, not an individual retry. Repeating the request should not create a second side effect.

Persist intent, not only the final result

A useful execution record should explain what the agent intended to do and why it was allowed to continue.

Consider storing:

User or system objective
Plan version
Selected tool and arguments
Action classification
Policy-check result
Approval status and approver
Idempotency key
Tool response
Final workflow state
Resume position after interruption

This information makes failures easier to diagnose and interrupted workflows easier to resume safely.

Make approval requests informative

A human approval step should not be a generic “Allow” button.

Before asking for approval, show:

The exact proposed action
The destination or affected resource
The relevant input values
The expected external effect
Whether the action is reversible
A preview or diff when possible

A reviewer should be able to understand the consequence without reconstructing the entire agent session.

Test permission boundaries directly

Agent evaluation should include more than successful task completion.

Test situations such as:

The agent requests data outside its allowed scope
Input data changes after the plan is generated
A tool call times out after completing the action
The same action is retried
Approval is denied or expires
A workflow fails halfway through
Execution resumes after a restart
A reversible action cannot be rolled back

These scenarios reveal whether the surrounding system remains safe when the agent, tool, network, or user behaves unexpectedly.

A practical release checklist

Before allowing an agent to perform external actions, verify that:

Every tool has the minimum required scope
Read and write permissions are separated
Irreversible actions have explicit policy rules
Duplicate retries cannot repeat a side effect
Every external call can be traced
Approval requests show the exact consequence
Interrupted workflows can resume safely
Recovery procedures have been tested

Final thought

Agent autonomy is not defined by how many tools the system can access.

It is defined by which actions the agent may perform, under what conditions, and with what evidence.

A useful production agent does not need unlimited authority.

It needs clearly designed permission boundaries.

Which action in your agent workflow would you never allow without human approval?

Stop Reporting Only Average Agent Success

shizhe Lim — Tue, 29 Sep 2026 09:31:46 +0000

AI disclosure: This article was prepared with AI assistance. The author reviewed and edited the technical content and remains responsible for its accuracy.

An AI agent completes a task successfully during testing.

The demo works. The result looks correct. The team records a pass.

Then someone runs the same task again—and the agent chooses a different tool, changes an argument, follows another path, and fails.

This is not necessarily a capability problem. It is a consistency problem.

Average success can hide unreliable behavior

A recent IBM Research evaluation illustrates the difference.

On the AppWorld benchmark, a ReAct agent using GPT-4.1 achieved a Mean@5 score of 77.4%. In other words, it completed 77.4% of the attempted task runs successfully.

However, its Pass⁵ score was 53.0%. Only 53% of the tasks succeeded in all five repeated runs.

Both numbers are correct, but they answer different questions.

Mean@k: How often does the agent succeed on average?
Pass^k: For how many tasks does the agent succeed in every repeated run?
Pass@k: For how many tasks does at least one of the repeated runs succeed?

Pass@k is useful when a system can safely retry and verify its own answer.

Pass^k is more relevant when repeated failures are expensive, difficult to detect or unacceptable.

Calculating the consistency gap

Suppose evaluation results are stored as one Boolean list per task:

results = {
    "task_1": [True, True, True, True, True],
    "task_2": [True, False, True, True, False],
    "task_3": [False, False, True, False, False],
}

We can calculate the average success rate and the all-runs success rate separately:

task_runs = list(results.values())

total_runs = sum(len(runs) for runs in task_runs)
successful_runs = sum(sum(runs) for runs in task_runs)

mean_at_k = successful_runs / total_runs
pass_power_k = sum(all(runs) for runs in task_runs) / len(task_runs)
consistency_gap = mean_at_k - pass_power_k

print(f"Mean@k: {mean_at_k:.1%}")
print(f"Pass^k: {pass_power_k:.1%}")
print(f"Consistency gap: {consistency_gap:.1%}")

A large consistency gap means the agent can complete many tasks, but its success is not repeatable.

That distinction is easy to lose when a leaderboard reports only one aggregate score.

What an Agent evaluation should record

Repeating a task is useful only if the surrounding conditions are documented.

At minimum, record:

Model and model version
Prompt or instruction version
Available tools and their schemas
Tool permissions
Agent framework version
Temperature and other inference settings
Number of attempts
Evaluation or grader version
Token and tool-call cost
Failure stage and execution trace

Without this context, two teams may report scores for the “same” model while evaluating meaningfully different systems.

This is also why reproducible evaluation reporting matters. A model score without its inference budget, tool configuration and evaluation protocol is difficult to interpret—and even harder to reproduce.

Evaluate cost per consistent success

Agent economics should not stop at cost per request.

Consider measuring:

cost per consistent success =
total evaluation cost /
number of tasks that succeeded in every required run

A cheaper model may become expensive if it requires repeated attempts, human review or recovery from incorrect tool calls.

A more capable model may also be a poor choice if its behavior varies too much across identical workflows.

For multi-model systems, routing decisions should therefore consider at least:

Capability
Repeatability
Latency
Total recovery cost
Failure severity

The model with the best average benchmark score is not automatically the best production model.

Start with three repeated runs

Running every task dozens of times may be impractical.

A reasonable starting point is:

Run each critical task three times
Report Mean@3 and Pass³
Investigate tasks with inconsistent outcomes
Repeat after changing the model, prompt, tools or permissions
Increase the repetition count for high-risk workflows

Even three runs can expose instability that a single successful test would miss.

Final thought

A successful demo proves that an agent can complete a task.

A repeated-run evaluation helps show whether users can depend on it.

Production reliability starts when we stop asking only:

“Did the agent succeed?”

and begin asking:

“Will it succeed again under the same conditions?”

Have you started measuring repeated-run consistency in your Agent evaluations?

References

IBM Research, “Your Agent Aced the Task. Will It Do It Again?”
https://huggingface.co/blog/ibm-research/altk-evolve-consistency
UK AISI and EvalEval, reproducible evaluation reporting
https://huggingface.co/blog/evaleval-aisi