The Sandwich Architecture: Wrapping a Probabilistic LLM in Deterministic Code

The Pain — You gave your Agent a knowledge base, wrote SOPs, and built a skill pack. Yet when it executes, it skips the knowledge base and guesses on its own, or ignores the SOP flow entirely. The more you tune prompts and add tools, the more it "forgets."

What You'll Learn:

• Why "letting the Agent decide whether to retrieve" is a design flaw, not a behavior quirk
• The Sandwich Architecture: two deterministic layers (Pre-hook and Post-hook) wrapping one probabilistic LLM layer
• Runnable Python for forced context injection, constrained reasoning, and output validation with retry
• Three bonus hardening moves: a fresh session per task, scene isolation, and a 20-case eval set

This problem is so common that it deserves its own post. If you have ever shipped an Agent with a knowledge base, a standard operating procedure, and a handful of tools — only to watch it improvise — this one is for you.

1. The Problem: You Built Your "Determinism" on a Beach of Probability

Here is what you typically give an Agent:

• SOP → written into the System Prompt
• Knowledge base → exposed as a Tool, letting the Agent decide whether to query it
• Skill pack → attached to the Agent's side

But when it runs, the Agent "forgets" to query the knowledge base and "fails to follow" the SOP. Sound familiar?

The root cause is not a dumb Agent. It is a wrong architecture.

An LLM is, by nature, a probabilistic model. By handing the "should I retrieve the knowledge base?" decision to a probabilistic model, you are betting that — across enough invocations — it will never suffer attention drift, context overflow, or the "I already know this" illusion that makes it skip retrieval. It will eventually skip. Not because it is broken, but because that is what probabilistic sampling does at the tail of the distribution.

That is not the Agent's fault. You anchored a deterministic process on probabilistic self-discipline.

The Sandwich Architecture: deterministic code wraps the probabilistic LLM on both sides — that is the whole trick.

2. The Cure: The Sandwich Architecture — Deterministic Code Wrapping a Probabilistic LLM

Between 2024 and 2026, the industry converged on a core consensus. Practitioners at Anthropic, LangChain, OpenAI, and Google keep hammering on the same point:

For business scenarios with fixed processes, do not build a fully autonomous Agent that decides everything on its own. Build a system driven by deterministic code that calls the LLM only at cognitive decision points.

That is the Sandwich Architecture: two layers of deterministic code sandwiching one layer of probabilistic LLM reasoning.

+-----------------------------------------------------------------+
| PRE-HOOK  |  DETERMINISTIC CODE                                 |
| load_sop . vector_db.search . inject into prompt                |
+-----------------------------------------------------------------+
| LLM REASONING  |  PROBABILISTIC                                 |
| Agent reasons only . temperature=0.1 . whitelist                |
+-----------------------------------------------------------------+
| POST-HOOK  |  DETERMINISTIC CODE                                |
| validate tool calls . check output . retry                      |
+-----------------------------------------------------------------+

Everything the Agent must know is pushed in by code before it thinks. Everything the Agent produces is checked by code after it speaks. The only thing left for the probabilistic model is the reasoning in between — where probability actually helps.

Top Layer: Pre-hook — Don't Make the Agent Take a Multiple-Choice Test

Wrong approach: expose the knowledge base as a Tool and let the Agent decide inside its loop whether to call it. The decision to retrieve becomes one option among twenty — and it loses.

Right approach: before the task ever reaches the Agent, code forcibly retrieves the knowledge base and injects the results directly into the System Prompt. The Agent never gets the choice:

def prepare_context(scene_id, task_data):
    # 1. always fetch the SOP for this scene
    sop = load_sop(scene_id)

    # 2. always search the knowledge base
    docs = vector_db.search(
        query=task_data["summary"],
        top_k=3,
        filter={"scene": scene_id}
    )
    kb_text = "\n".join(d.page_content for d in docs)

    # 3. assemble the System Prompt
    system_prompt = f"""
[MANDATORY SOP - must be followed strictly] {sop}
[INJECTED LATEST KNOWLEDGE - do not guess on your own] {kb_text}

Working rules:
1. You do not need to retrieve anything. The latest and most accurate knowledge has been injected.
2. Execute strictly according to the SOP steps.
3. If the injected knowledge is insufficient to answer, output NEED_HUMAN_INTERVENTION.
"""
    return system_prompt

✅ Verified in production: I run this exact code every day. The moment the Agent "opens its eyes," the SOP and the knowledge are already on the table in front of it. There is no "should I query?" choice to make — the question was removed before the Agent existed.

🩸 Pitfall: My first version exposed the knowledge base as a Tool and let the Agent call it. Week one was fine. From week two, the Agent skipped retrieval more and more often. After I switched to forced injection, the problem disappeared completely.

💼 Value: "Whether to retrieve" goes from a multiple-choice question to a mandatory step. 100% execution rate — not 99%.

🧠 Cognitive upgrade: The first principle of commercializing an Agent — never hand choices to probability.

Middle Layer: The Reasoning Layer (Agent) — Reason Only, Don't Decide

This layer is the Agent you already have, but with three constraints:

def run_agent(scene_id, task_data):
    context = prepare_context(scene_id, task_data)

    agent = HermesRunner(
        system_prompt=context,
        tools=get_scene_tools(scene_id),   # only the tools this scene needs
        temperature=0.1,                   # keep it low to reduce drift
    )

    return agent.run(task_data["content"])

Temperature 0.1. In production, temperature must be 0.1 or lower. 0.7 is for writing poetry, not for processing business tasks. Every extra unit of entropy is a chance for the output to drift off the SOP.

Tool whitelist. Each scene only mounts the tools that scene needs. If scene A does not need search_web, physically remove it from the toolset. Fewer options, fewer wrong turns.

Max turns. Set a hard upper bound on the loop so the Agent cannot spin forever inside a retry cycle.

Bottom Layer: Post-hook — No Exit Until It's Done

The Agent finishing its output does not mean the task is done. The code layer checks whether the output is compliant:

def validate_and_retry(scene_id, task_data, max_retries=3):
    for attempt in range(max_retries):
        # run the agent
        response = run_agent(scene_id, task_data)
        logs = response.tool_logs

        # check 1: did it call the mandatory tool?
        if "search_database" not in [log.tool for log in logs]:
            task_data["content"] += "\n\n[VALIDATION ERROR] You skipped the database query step. Please re-run."
            continue

        # check 2: does the output contain the required compliance clause?
        if "compliance clause" not in response.text:
            task_data["content"] += "\n\n[VALIDATION ERROR] The output is missing the compliance clause required by the SOP."
            continue

        # all checks passed
        return response.text

    raise Exception(
        f"Agent violated the SOP {max_retries} times in a row in scene {scene_id}. Escalating to a human."
    )

The Post-hook gate: validate tool calls → validate output → retry (max 3) → escalate to a human.

✅ Verified in production: This is the logic behind the handoff_to_yuanbao.py module in my own system. Every article I publish passes through exactly this gate before it goes out.

🩸 Pitfall: Early versions "looked" finished but had skipped critical steps. Only after adding the validation layer did I actually know whether the Agent followed the SOP — the output alone never told me.

💼 Value: fail → retry → human escalation: three levels of backstop. Nothing imperfect leaves the door.

🧠 Cognitive upgrade: Output validation matters more than the output itself.

Before vs After: from "the Agent decides whether to query the KB" to "code injects the KB into the prompt before the Agent runs."

3. Three Extra Moves to Make the Architecture Rock-Solid

Move 1: A Fresh Session Per Task

Every scheduled task must start a new Agent Session with memory cleared. History from the previous task must not leak into the next one.

Wrong — accumulating history context:

agent.run(task_1)
agent.run(task_2)  # task_1's context pollutes task_2

Right — a brand-new session per task:

# right way: a brand-new session per task
def run_task(task_data):
    agent = fresh_agent()  # brand-new session
    return agent.run(task_data)

State that must survive across tasks is written to a JSON file on disk and read back by code. Do not rely on the Agent's memory for anything you actually need.

Move 2: Scene Isolation

Do not build one "super Agent" for all scenes. The bigger the blast radius of a single Agent, the more ways it can drift.

Wrong approach: one Agent + SOPs for 10 scenes + 20 tools.

Right approach:

Router -> detects the scene -> dispatches to the scene's dedicated Agent
Dedicated Agent carries only that scene's SOP + the minimum toolset

A scene-specific Agent with three tools fails in three places, not twenty. Containment is a feature.

Move 3: Build an Eval Set

Twenty typical tasks is enough. After every change to the prompt, the tools, or the SOP, run the eval set and check whether the Agent still follows the SOP. This is not hard — it is the same idea as evals_writing.py in my toolkit: a fixed set of inputs, a checklist of required behaviors, and a pass/fail verdict per case.

4. Where You Are Now

You are no longer the debugger who keeps tweaking prompts, bolting tools onto the Agent, and hoping it will self-discipline its way through the SOP. You are becoming a system designer who wraps a probabilistic LLM in deterministic code.

• The knowledge base is not for the Agent to query — code queries it for the Agent and feeds it straight to the mouth.
• The SOP is not for the Agent to read — code checks whether it actually executed it.
• The Loop is not for the Agent to roam freely — code keeps it re-checking on the right track.

Remember: 90% of "forgetting" problems are killed by architecture, not by model self-discipline.

Key takeaways — Entities: Hermes Agent, Harness Engineering, Loop Engineering. Value: Agent commercialization, SOP process hardening, deterministic architecture. Cognition: from prompt-tuning to the sandwich — deterministic code wrapping a probabilistic LLM.

---

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.