DEV Community: Ranknod

Test Your AI Assistant Against Conflicting Source Documents

Ranknod — Sat, 26 Sep 2026 14:50:53 +0000

Why can two correct citations still produce the wrong answer?

Suppose an AI assistant retrieves two notes. One says that an interactive feature permits 250 output tokens. The other says 500. Both are real passages in the test collection. The assistant chooses 500, cites the second note, and presents the answer as settled.

The missing behavior is conflict handling.

Test conflicting-source behavior with separate cases for unresolved disagreement, approved supersession, and different scopes. An assistant should flag an unresolved conflict, follow a trusted revision decision when one exists, and avoid treating different use cases as contradictory. Evaluate the answer and its evidence references together.

This tutorial creates three synthetic cases and a small Python checker. The token limits describe a fictional application policy; they are not limits of any real model or provider.

Which disagreements should produce a conflict?

Different numbers alone are insufficient. Compare the entity, scope, time, and source authority before choosing an expected response.

Case	Evidence relationship	Expected behavior
Unresolved	Same feature and scope; no approved precedence	Report conflict and retain both evidence references.
Superseded	A trusted source register says one revision replaces another	Answer from the approved revision.
Different scope	Both sources apply, but to different request types	Answer for the requested type.

The word “trusted” matters. A retrieved passage saying “ignore the other document” cannot appoint itself the authority. Source ownership, approved revision state, and permissions should come from the application's controlled process.

This test does not infer authority from a filename or a recent timestamp. Those can be useful metadata, but neither alone establishes which instruction governs the task.

Create the fixtures

Use Python 3.12 and the standard library. Save the following as conflict_cases.json. Python's JSON documentation describes the object and value conversions used when loading this fixture.

[
  {
    "id": "unresolved",
    "question": "What is the interactive assistant's output limit?",
    "documents": [
      {
        "id": "a",
        "text": "The interactive assistant has a 250-token output limit."
      },
      {
        "id": "b",
        "text": "The interactive assistant has a 500-token output limit."
      }
    ],
    "authority": "No approved precedence between a and b is available.",
    "expected": {
      "status": "conflict",
      "value": null,
      "required_citations": ["a", "b"]
    }
  },
  {
    "id": "superseded",
    "question": "What is the current interactive output limit?",
    "documents": [
      {
        "id": "old",
        "text": "Revision 1: interactive output limit is 250 tokens."
      },
      {
        "id": "new",
        "text": "Revision 2: interactive output limit is 500 tokens."
      }
    ],
    "authority": "The approved source register says new replaces old for this task.",
    "expected": {
      "status": "answered",
      "value": 500,
      "required_citations": ["new"]
    }
  },
  {
    "id": "different_scope",
    "question": "What is the interactive assistant's output limit?",
    "documents": [
      {
        "id": "interactive",
        "text": "Interactive requests have a 250-token output limit."
      },
      {
        "id": "batch",
        "text": "Batch evaluation requests have a 500-token output limit."
      }
    ],
    "authority": "Both notes are current and apply to their named request types.",
    "expected": {
      "status": "answered",
      "value": 250,
      "required_citations": ["interactive"]
    }
  }
]

Each case separates the evidence supplied to the assistant from the expected result used by the evaluator. The authority field stands in for trusted application metadata in this small fixture.

When connecting a model, send the question, eligible documents, and relevant authority information. Keep the expected object out of the generation input. Giving the model the answer key would make the evaluation meaningless.

For an AI evaluation considered by Ranknod, this three-case distinction could reveal whether an assistant is resolving source relationships or simply selecting the passage that makes answering easiest.

Check the response contract

The response contract has three fields: status, value, and citations. For the unresolved case, the status must be conflict, the value must be JSON null, and both source identifiers must appear. For a resolved case, the integer value and required evidence must match the fixture.

Save the checker as check_conflicts.py:

import argparse
import copy
import json
from pathlib import Path


def check_response(case, response):
    expected = case["expected"]
    failures = []

    if response.get("status") != expected["status"]:
        failures.append("status")

    value = response.get("value")
    if (
        "value" not in response
        or type(value) is not type(expected["value"])
        or value != expected["value"]
    ):
        failures.append("value")

    citations = response.get("citations")
    valid_list = isinstance(citations, list) and all(
        isinstance(item, str) for item in citations
    )

    if not valid_list or len(citations) != len(set(citations)):
        failures.append("citation_shape")
    else:
        known = {doc["id"] for doc in case["documents"]}
        supplied = set(citations)

        if supplied - known:
            failures.append("unknown_citation")

        if not set(expected["required_citations"]).issubset(supplied):
            failures.append("missing_evidence")

    return failures


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("cases", type=Path)
    parser.add_argument("--fixed", action="store_true")
    args = parser.parse_args()

    cases = json.loads(args.cases.read_text(encoding="utf-8"))

    saved = {
        "unresolved": {
            "status": "answered",
            "value": 500,
            "citations": ["b"],
        },
        "superseded": {
            "status": "answered",
            "value": 500,
            "citations": ["new"],
        },
        "different_scope": {
            "status": "answered",
            "value": 250,
            "citations": ["interactive"],
        },
    }

    if args.fixed:
        saved = copy.deepcopy(saved)
        saved["unresolved"] = {
            "status": "conflict",
            "value": None,
            "citations": ["a", "b"],
        }

    ids = [case["id"] for case in cases]

    if len(ids) != len(set(ids)) or set(ids) != set(saved):
        raise ValueError(
            "case IDs must be unique and match saved response IDs"
        )

    any_failure = False

    for case in cases:
        failures = check_response(case, saved[case["id"]])
        any_failure |= bool(failures)

        print(
            case["id"]
            + ": "
            + (
                "FAIL " + ", ".join(failures)
                if failures
                else "PASS"
            )
        )

    raise SystemExit(1 if any_failure else 0)

The script compares saved synthetic responses; it does not call a model. One response is deliberately wrong so the default run demonstrates a meaningful failure. The --fixed option replaces that response with the expected conflict result.

The value check includes its type. A floating-point value that happens to equal an integer does not satisfy this fixture's integer contract. Citation identifiers must be unique strings that refer to known documents, and the required evidence must be present.

What do the two runs establish?

Run the deliberately failing fixture first:

python3 check_conflicts.py conflict_cases.json

Observed output on Python 3.12.14:

unresolved: FAIL status, value, missing_evidence
superseded: PASS
different_scope: PASS

The process exits with code 1. That matters if this checker becomes a build step: a printed failure without a failing exit code could leave the pipeline green.

Then run:

python3 check_conflicts.py conflict_cases.json --fixed

Observed output:

unresolved: PASS
superseded: PASS
different_scope: PASS

That process exits with code 0. These runs verify the checker against hand-authored fixture responses. They do not establish how any deployed assistant handles conflict.

How do you connect this to an actual assistant?

Replace the saved response map with responses collected from your application. Keep the case identifiers stable and retain the model configuration, prompt revision, retrieval snapshot, and source authority used for each run.

An adapter should parse the model's output and validate that it is an object before passing it to this checker. Invalid JSON, refusal, timeout, and unavailable evidence need their own recorded outcomes. Do not silently remove unsuccessful cases from the denominator.

The application must receive the same authority context that the evaluation assumes. Otherwise, a test may penalize the assistant for failing to know information it was never given. Conversely, a hidden answer key in retrieved metadata can make performance look better than it is.

If generation varies between runs, use repeated trials appropriate to the consequence of failure and report that variation. One passing response does not establish reliable behavior.

What can this checker miss?

It validates a narrow structured contract. It does not read a free-text explanation and decide whether the explanation faithfully represents the evidence. An answer could include the expected fields and still add an unsupported sentence elsewhere.

Review whether the assistant states the conflict clearly, describes the relevant scope, and avoids inventing a compromise such as a 375-token limit. Also inspect unnecessary citations: this checker allows additional known identifiers once required evidence is present, but a human may judge some of those references irrelevant or misleading.

Expand the cases along meaningful boundaries: overlapping effective dates, conflicting units, identical titles for different features, and missing precedence information. Keep the expected outcome grounded in the application's actual policy. A larger fixture set only helps when its labels are defensible.

Begin with the unresolved case. If the assistant cannot preserve a simple disagreement without guessing, adding more retrieved documents may only make the answer look better supported. A useful evaluation makes uncertainty visible before that answer reaches a user.

How to Flag Stale RAG Documents Before Generation

Ranknod — Sat, 26 Sep 2026 14:46:25 +0000

An AI assistant cites a real document and reproduces its instructions accurately. The problem is that the document was replaced last week. The citation proves that a source exists; it does not establish that the source is still approved for the current question.

Check retrieved documents against an authoritative revision register and explicit time rules before sending them to the generator. A freshness gate can reject a superseded revision, a document that is not yet effective, or one overdue for review. It cannot prove that an accepted document is factually correct.

This tutorial builds that narrow gate in Python. The policy is deliberately specific: the assistant is answering questions about the currently approved material. Historical questions need a different selection policy.

What does “fresh” mean for this application?

Retrieval-augmented generation, or RAG, supplies retrieved material to a language model. A retriever may rank a passage highly because it matches the question even when its lifecycle metadata makes it unsuitable for a current answer.

Use three independent checks:

Check	Meaning	Failure response
Approved revision	The document matches the current source register	Exclude superseded or unknown records.
Effective time	The document already applies at the evaluation time	Exclude a future version.
Review deadline	The record has not passed the team's review boundary	Hold it for review under this policy.

“Review overdue” means the chosen process requires attention. It does not mean a fact becomes false at midnight. Likewise, the newest timestamp does not establish authority. The source owner must decide which revision is approved.

Keep the clock and the source register explicit

Pass the evaluation time into the check. Reading the wall clock deep inside the function makes a saved fixture behave differently as time passes. A fixed as_of value gives the test a reproducible meaning.

Use timestamps with offsets and normalize them to UTC for comparison. Python's datetime documentation distinguishes aware timestamps from naive ones. This example rejects timestamps without timezone information instead of guessing the author's intent.

The revision register must come from the approved document process. Do not let the model infer the current revision from a filename or a passage saying “this is the latest version.” Keep authority separate from the text being evaluated.

Implement the gate

Use Python 3.12. The example requires only the standard library. Save this as check_freshness.py:

from datetime import datetime, timezone


def utc(value):
    if not isinstance(value, str):
        raise ValueError("timestamp must be a string")
    parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
    if parsed.tzinfo is None or parsed.utcoffset() is None:
        raise ValueError("timestamp needs a timezone")
    return parsed.astimezone(timezone.utc)


def check_document(doc, current_revisions, as_of):
    if as_of.tzinfo is None or as_of.utcoffset() is None:
        raise ValueError("as_of needs a timezone")
    try:
        doc_id, revision = doc["id"], doc["revision"]
        if any(
            not isinstance(v, str) or not v.strip()
            for v in (doc_id, revision)
        ):
            raise ValueError("invalid document identity")

        effective = utc(doc["effective_at"])
        review_by = utc(doc["review_by"])

        if review_by <= effective:
            raise ValueError("review_by must follow effective_at")
    except (KeyError, TypeError, ValueError):
        return {"allowed": False, "reasons": ["invalid_metadata"]}

    reasons = []
    approved_revision = current_revisions.get(doc_id)

    if approved_revision is None:
        reasons.append("unknown_document")
    elif revision != approved_revision:
        reasons.append("superseded_revision")

    if as_of < effective:
        reasons.append("not_effective_yet")

    if as_of >= review_by:
        reasons.append("review_overdue")

    return {"allowed": not reasons, "reasons": reasons}


if __name__ == "__main__":
    now = utc("2026-09-25T08:00:00Z")
    current = {
        "note-a": "3",
        "note-b": "2",
        "note-c": "5",
        "note-d": "1",
    }

    docs = [
        {
            "id": "note-a",
            "revision": "3",
            "effective_at": "2026-09-01T00:00:00Z",
            "review_by": "2026-10-01T00:00:00Z",
        },
        {
            "id": "note-b",
            "revision": "1",
            "effective_at": "2026-09-01T00:00:00Z",
            "review_by": "2026-10-01T00:00:00Z",
        },
        {
            "id": "note-c",
            "revision": "5",
            "effective_at": "2026-09-01T00:00:00Z",
            "review_by": "2026-09-20T00:00:00Z",
        },
        {
            "id": "note-d",
            "revision": "1",
            "effective_at": "2026-10-01T00:00:00Z",
            "review_by": "2026-12-01T00:00:00Z",
        },
    ]

    for doc in docs:
        result = check_document(doc, current, now)
        status = (
            "ALLOW"
            if result["allowed"]
            else ", ".join(result["reasons"])
        )
        print(f"{doc['id']}: {status}")

The code returns reasons instead of one unexplained Boolean. Multiple reasons can be true: a revision might be both superseded and overdue. Invalid metadata fails the gate rather than falling through to approval.

The interval is explicit. A document becomes eligible at its effective timestamp. At the exact review deadline, it becomes overdue. Changing either boundary would be a policy change that should be reflected in tests.

For an AI document workflow considered by Ranknod, the same gate could distinguish source maintenance work from prompt tuning when an answer repeats an outdated instruction.

What does the fixture produce?

Run:

python3 check_freshness.py

The observed output on Python 3.12.14 was:

note-a: ALLOW
note-b: superseded_revision
note-c: review_overdue
note-d: not_effective_yet

These are synthetic metadata records. The run verifies the local classification logic; it does not test an actual search index or a model. Additional local checks covered exact deadline boundaries, equivalent timezone offsets, missing authority, invalid date ordering, and missing timezone information.

In a real retrieval flow, retain the reason for each excluded result in an appropriately protected diagnostic record. If every useful passage is excluded, return a controlled “current evidence unavailable” outcome or route the task for review. Do not quietly restore the rejected sources just to obtain a fluent answer.

Should filtering happen inside the search engine?

It can, when the index and query engine support the required fields. For example, Azure AI Search documents filters that restrict matching records using filterable fields. That capability can help apply an application's selection policy, but it does not create or maintain the policy's source authority.

A check after retrieval is still useful at the application boundary, especially when results can arrive through several adapters. It also makes rejection reasons easier to test. Consider both retrieval quality and correctness: filtering a tiny returned list may leave no evidence even when eligible material exists deeper in the index.

If your system refetches or reranks results, preserve the same permission and freshness rules. Repeated retrieval should not widen access or relax the current-document requirement without an explicit decision.

What can still go wrong?

The gate is only as current as its inputs. If the source register is stale, a replaced document can still pass. If revision identifiers change without the stored text changing correctly, the label may describe the wrong content. Use an ingestion process that ties the text, revision, and metadata together.

There is also a timing boundary. A revision could change after the check and before a consequential action uses the answer. Work from a recorded snapshot where appropriate and define when revalidation is required. This small function is not a transaction across the document system and the eventual action.

Keep access checks separate and mandatory. A current document is not automatically one the user may read. Keep factual review separate too: approval and freshness do not establish that every sentence is correct.

The next debugging question becomes concrete: did the assistant receive an eligible source at the recorded time? Once that is known, the team can investigate the right stage instead of rewriting instructions around an obsolete document.

Track AI Prompt Changes with a Run Manifest

Ranknod — Fri, 25 Sep 2026 15:41:14 +0000

An AI evaluation improves after a prompt edit. A week later, a teammate cannot reproduce the comparison. The prompt file is available, but the retrieved note changed, a model alias may point somewhere else, and nobody recorded the adapter revision. There is an answer on disk without enough context to explain its origin.

Create a run manifest that identifies the actual request inputs, configuration, source snapshot, and evaluator revision. Then assign that manifest a stable digest. The digest helps detect changes to the recorded configuration; it does not guarantee identical model output.

A run manifest is a structured record describing an evaluation attempt. Its useful purpose is traceability: someone should be able to tell which conditions were held constant and which changed before interpreting an apparent improvement.

Why is the prompt filename insufficient?

A template filename does not identify the messages actually sent. Variables may have been substituted, earlier messages retained, retrieved passages reordered, or a tool schema changed. Those differences can exist while the filename remains unchanged.

Record the resolved model identifier where the provider exposes one, the provider or endpoint configuration, generation settings, rendered messages, ordered context, tool definitions, adapter revision, and evaluation rubric revision. Use immutable references or protected artifacts for large inputs.

Keep event metadata separate from configuration identity. Two attempts can use the same configuration but have different run IDs, timestamps, outputs, and durations. If a timestamp enters the configuration digest, every retry appears to be a configuration change.

The fixture below uses synthetic model names. They are labels for the demonstration, not identifiers for a real model or claims about a provider's versioning behavior.

Make serialization predictable within a defined format

Python's json module supports sorted object keys and explicit separators. hashlib provides SHA-256. Combining these can give the same accepted data structure a stable byte representation under a stated convention.

This example permits strings, integers, booleans, nulls, lists, and dictionaries with string keys. Decimal settings are represented as strings. It deliberately rejects floating-point values, tuples, and other objects, keeping the accepted schema small and its representation easier to explain.

Save the following as ai_run_manifest.py and run it with Python 3.12:

import copy
import hashlib
import json


def validate_json_value(value):
    if value is None or type(value) in (str, int, bool):
        return
    if type(value) is list:
        for item in value:
            validate_json_value(item)
        return
    if type(value) is dict and all(type(key) is str for key in value):
        for item in value.values():
            validate_json_value(item)
        return
    raise TypeError("use string keys and JSON values; encode decimals as strings")


def stable_bytes(value):
    validate_json_value(value)
    return json.dumps(
        value, sort_keys=True, ensure_ascii=False, separators=(",", ":")
    ).encode("utf-8")


def digest_bytes(value):
    return hashlib.sha256(value).hexdigest()


def manifest_id(manifest):
    return digest_bytes(stable_bytes(manifest))


if __name__ == "__main__":
    rendered_messages = [
        {"role": "system", "content": "Summarize supplied notes. Preserve limits."},
        {"role": "user", "content": "Summarize the evaluation note."},
    ]
    source_bytes = b"Routine prompts passed; unusual prompts were not tested."
    base = {
        "schema": "ai-run-manifest-v1",
        "adapter_revision": "demo-adapter-v1",
        "provider": "synthetic",
        "model": "demo-model-v1",
        "settings": {"temperature": "0", "max_output_tokens": 300},
        "messages_sha256": digest_bytes(stable_bytes(rendered_messages)),
        "tools_sha256": digest_bytes(stable_bytes([])),
        "context": [
            {
                "id": "note-1",
                "revision": "v1",
                "sha256": digest_bytes(source_bytes),
            }
        ],
        "evaluator_revision": "human-rubric-v1",
    }
    reordered = dict(reversed(list(base.items())))
    changed = copy.deepcopy(base)
    changed["context"][0]["revision"] = "v2"
    print(
        "same fields, different key order:",
        manifest_id(base) == manifest_id(reordered),
    )
    print(
        "changed source revision:",
        manifest_id(base) == manifest_id(changed),
    )
    print("configuration ID:", manifest_id(base))

messages_sha256 fingerprints the rendered synthetic messages. The context list retains source order, ID, revision, and a digest of the source bytes. tools_sha256 identifies the empty tool list used in this fixture. A real adapter must supply the actual artifacts used by the request.

The Ranknod example uses invented configuration values so the identity rules can be inspected without exposing private prompts or implying a deployed integration.

What should the demonstration show?

Run:

python3 ai_run_manifest.py

The observed output begins:

same fields, different key order: True
changed source revision: False

The full configuration ID produced by this exact fixture was:

58e0a9173837863e61accca952edb3e883dc203b292f9ac1e07f231566972370

Reordering dictionary keys leaves the ID unchanged. Updating the recorded source revision changes it. Local checks also confirmed that changing list order changes the ID and that unsupported values, including floats and non-string dictionary keys, are rejected.

The source revision change in this demonstration leaves the content bytes unchanged. That is intentional: the convention treats a provenance revision as a configuration change even when the payload happens to be identical. Choose and document that behavior before comparing IDs across runs.

What does a matching ID fail to prove?

A matching digest means the recorded manifest serializes to the same identified configuration, subject to the properties of the hash. It cannot prove that the adapter accurately recorded every input. It cannot recover a source that was deleted, reveal a provider-side change hidden behind an alias, or explain nondeterministic behavior inside a model service.

Keep the actual request artifacts in an approved store when retention is permitted. A hash without recoverable context gives you a comparison signal, not a replay. Store returned provider metadata with the individual run where available; do not invent a backend version that the service never supplied.

This format is also not a general cross-language canonical JSON standard. Another runtime may serialize accepted values differently. Unicode normalization is not performed, and list order remains significant. If multiple languages produce manifests, adopt a documented common serialization scheme and test shared fixtures.

Does hashing make the recorded data private?

No. A digest is not encryption, anonymization, or a permission boundary. Someone who can guess a short input may be able to hash candidates and compare them. Treat manifests and their source artifacts according to the sensitivity of the workflow.

Never place API keys, bearer tokens, or credentials in the manifest. If endpoint or account information is needed for traceability, use an approved identifier. Apply retention and access controls to outputs as well as inputs; generated text can reproduce sensitive source material.

Use the manifest to make comparisons honest

When an evaluation score changes, compare the manifests before explaining the change. If messages, context, model configuration, and evaluator all changed, describe the result as a comparison of workflows. It does not isolate the effect of one prompt sentence.

For a controlled prompt experiment, hold the other recorded conditions constant where possible and keep multiple output attempts when variation matters. Save the result alongside its run ID, configuration ID, and evaluator result. A failed run belongs in the record too.

Traceability does not make an AI result correct. It makes an investigation possible. The next time someone asks why an answer changed, the team can begin with the conditions that actually changed instead of reconstructing a test from a filename and a memory.

Measure RAG Retrieval Before You Tune the Prompt

Ranknod — Fri, 25 Sep 2026 15:13:38 +0000

You make the instruction more precise, add an example, and lower a generation setting. The AI answer still misses a crucial passage. Then you inspect the retrieval results: the passage never reached the generator. The team has been tuning the part of the system that could not see the evidence it needed.

Measure whether the retriever returns the relevant documents before using answer quality to judge a prompt change. This tutorial builds a small Python evaluator for recall at a cutoff and the rank of the first relevant result. It scores saved retrieval IDs, so you can compare retrieval changes without making a model call.

Disclosure: This article was drafted using generative AI. The Python example was executed against the synthetic fixtures described below.

Retrieval-augmented generation, or RAG, combines retrieved material with language generation. The original RAG research paper describes a system connecting a neural retriever and a generator. This tutorial evaluates only a retrieval stage; it makes no claim about the factual accuracy of the eventual answer.

What must be labeled before measurement?

Create a set of query IDs and a list of documents judged relevant to each query. Information retrieval evaluations often call these judgments qrels. Keep a separate ranked list of IDs returned by the retriever for each query.

For this small example, relevance is binary: a document either belongs in the known relevant set or does not. Use a stable document or chunk identity and keep the judgment unit consistent. A document-level label cannot automatically tell you whether the returned chunk contains the supporting sentence.

Save this synthetic fixture as retrieval_fixture.json:

{
  "qrels": {"q1": ["d1", "d2"], "q2": ["d3"], "q3": ["d4"]},
  "runs": {
    "q1": ["d9", "d2", "d1"],
    "q2": ["d3", "d8", "d9"],
    "q3": ["d8", "d9", "d10"]
  }
}

The fixture has three queries. The first has two relevant documents. The second has one, returned first. The third has a known relevant document that the retrieval list misses completely. These are invented IDs for testing the arithmetic, not observations from a deployed assistant.

What do recall and reciprocal rank tell you?

Recall@k is the fraction of known relevant documents appearing in the first k results. With two relevant documents, finding one gives a recall of 0.5. This follows the basic recall definition in Introduction to Information Retrieval, applied to a ranked list truncated at k.

Reciprocal rank@k is 1 / rank for the first relevant result inside that cutoff, or zero when none appears. MRR@k averages those values across queries. It tells you about the first useful result; it does not reward retrieving a second piece of evidence needed to answer a multi-part question.

Query	Relevant IDs in top three	Recall@3	Reciprocal rank@3
q1	d2, d1	1.0	0.5
q2	d3	1.0	1.0
q3	None	0.0	0.0

The averages are approximately 0.6667 and 0.5. Each query has equal weight. This is a macro average, so a query with two relevant documents does not count twice as much as one with a single relevant document.

Implement a bounded evaluator

Use Python 3.12; no third-party packages are needed. Save the following as rag_retrieval_eval.py:

import argparse
import json
from pathlib import Path


def check_ids(value, label, allow_empty=True):
    if not isinstance(value, list) or any(
        not isinstance(item, str) or not item.strip() for item in value
    ):
        raise ValueError(f"{label}: expected a list of nonblank IDs")
    if len(value) != len(set(value)):
        raise ValueError(f"{label}: duplicate IDs")
    if not allow_empty and not value:
        raise ValueError(f"{label}: relevance labels are required")


def evaluate(qrels, runs, k):
    if type(k) is not int or k < 1:
        raise ValueError("k must be a positive integer")
    if not isinstance(qrels, dict) or not isinstance(runs, dict):
        raise ValueError("qrels and runs must be objects")
    if not qrels or set(qrels) != set(runs):
        raise ValueError("nonempty qrels and runs must have identical query IDs")

    scores = {}
    for query_id, relevant in qrels.items():
        check_ids(relevant, f"{query_id} qrels", allow_empty=False)
        check_ids(runs[query_id], f"{query_id} run")
        top = runs[query_id][:k]
        wanted = set(relevant)
        recall = len(wanted.intersection(top)) / len(wanted)
        rr = next(
            (1 / rank for rank, doc_id in enumerate(top, 1)
             if doc_id in wanted),
            0.0,
        )
        scores[query_id] = {"recall": recall, "rr": rr}

    count = len(scores)
    return {
        "k": k,
        "queries": count,
        "mean_recall_at_k": sum(s["recall"] for s in scores.values()) / count,
        "mrr_at_k": sum(s["rr"] for s in scores.values()) / count,
        "per_query": scores,
    }


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("fixture", type=Path)
    parser.add_argument("--k", type=int, default=3)
    args = parser.parse_args()
    data = json.loads(args.fixture.read_text(encoding="utf-8"))
    print(json.dumps(evaluate(data["qrels"], data["runs"], args.k), indent=2))

The validation rejects duplicate IDs, missing relevance labels, invalid cutoffs, and different query sets. A missing retrieval entry must not silently disappear from an average. An explicitly empty result list is valid and scores zero.

The script requires at least one known relevant document per query. Questions genuinely lacking an answer belong in a separate abstention evaluation. Questions that simply lack human labels are incomplete evaluation data. Combining those categories into a zero denominator would obscure what you measured.

For the Ranknod editorial example here, the fixture demonstrates a retrieval diagnostic; it does not measure a brand's AI system or content quality.

Run the comparison and inspect individual failures

Run:

python3 rag_retrieval_eval.py retrieval_fixture.json --k 3

The observed Python 3.12.14 result was mean_recall_at_k: 0.6666666666666666 and mrr_at_k: 0.5, with the per-query values shown in the table. The empty-result and invalid-input behaviors described above were also checked locally. No model or vector database was contacted.

To compare a retrieval change, keep the qrels, corpus snapshot, permissions, query set, and cutoff fixed, then replace only the saved ranked runs. Inspect queries that improve and queries that regress. A better average can conceal a newly broken question your users rely on.

Do not set a universal release threshold from this tiny fixture. A real threshold needs a meaningful evaluation set, an agreed cost of failure, and an understanding of label coverage. In CI, compare a candidate against an approved baseline and send material regressions for review.

Where can these numbers mislead you?

Incomplete judgments make “recall” recall against the known labeled set. An unjudged document may be useful. Review those cases before interpreting every apparent miss as a system defect. Keep labels versioned so a label correction is distinguishable from a retrieval improvement.

Increasing k can improve recall while sending the generator more irrelevant or conflicting material. These two metrics do not measure that cost, retrieval latency, permissions, freshness, or how well the generator uses the results. A first relevant hit also cannot prove that all evidence required by a question is present.

Use the evaluator to localize a failure. When relevant material is absent, investigate indexing, chunking, filtering, and ranking. When it is present but the answer still misrepresents it, examine the generation and review stages. The point is to give the next debugging step a reason.

Check AI Citations Against Retrieved Source IDs

Ranknod — Thu, 24 Sep 2026 14:14:24 +0000

Why can an AI answer cite a source it never saw?
A team's AI assistant answers a question with a citation that looks legitimate: eval-note-7. The retrieval step for that request supplied eval-note-1 and eval-note-2. If the application simply renders the citation, a reader can click what appears to be evidence that was never part of the answer's context.
The first fix is mechanical: compare the structured citation IDs in the AI response with the source IDs in the retrieval snapshot for the same request. Reject unknown IDs before returning the answer. This catches one failure; it does not tell you whether an allowed document actually supports the words beside its citation.
AI disclosure: This post was drafted with AI. The author must review the code and claims, then select the DEV disclosure tier that matches the final published article.
What exactly is the contract?
The retriever supplies a list of sources with stable IDs for one request. The generator returns an answer with status, text, and a list of citations using those IDs. In this example, every answered response needs at least one citation. An unknown response may have no citations because it is explicitly saying the supplied material cannot answer the question.
This is a contract for a particular application, not a universal format for model output. Your production adapter must capture the retrieval snapshot and parse the generator's structured output. Do not scrape source-like strings out of free-form prose and call them verified citations.
Save a small retrieval fixture as retrieved.json:
{
"sources": [
{"id": "eval-note-1", "text": "The evaluation covered routine prompts."},
{"id": "eval-note-2", "text": "Unusual prompts were not tested."}
]
}
Save an intentionally bad output as answer.json:
{
"status": "answered",
"text": "The evaluation covered routine prompts only.",
"citations": ["eval-note-1", "eval-note-7"]
}
The sentence could be reasonable. The second citation is still impossible to justify from this retrieval snapshot because eval-note-7 is absent.
Implement the source-ID check
Save this as check_citations.py. It reads trusted local files, rejects oversized inputs, checks schema shape, and exits with a failure code on invalid citations. It never sends documents or prompts to a network service.
import json
import sys
from pathlib import Path

MAX_BYTES = 1_000_000

def load_json(path):
if path.stat().st_size > MAX_BYTES:
raise ValueError(f"{path}: file exceeds {MAX_BYTES} bytes")
return json.loads(path.read_text(encoding="utf-8"))

def validate(retrieval, response):
errors = []
if not isinstance(retrieval, dict) or not isinstance(response, dict):
return ["both inputs must be JSON objects"]

sources = retrieval.get("sources")
if not isinstance(sources, list) or not sources:
    return ["sources must be a nonempty list"]

ids = []
for source in sources:
    if not isinstance(source, dict):
        return ["each source must be an object"]
    source_id = source.get("id")
    if not isinstance(source_id, str) or not source_id:
        return ["each source needs a nonempty string id"]
    ids.append(source_id)
if len(ids) != len(set(ids)):
    errors.append("retrieved source IDs must be unique")

status = response.get("status")
if status not in ("answered", "unknown"):
    errors.append("status must be answered or unknown")
if not isinstance(response.get("text"), str) or not response["text"].strip():
    errors.append("text must be a nonempty string")

citations = response.get("citations")
if not isinstance(citations, list):
    return errors + ["citations must be a list"]
if any(not isinstance(c, str) or not c for c in citations):
    return errors + ["citation IDs must be nonempty strings"]
if len(citations) != len(set(citations)):
    errors.append("citation IDs must be unique")
unknown = sorted(set(citations) - set(ids))
if unknown:
    errors.append(f"IDs not retrieved: {unknown}")
if status == "answered" and not citations:
    errors.append("answered output needs a citation")
return errors

def main():
if len(sys.argv) != 3:
raise SystemExit("Usage: python3 check_citations.py retrieved.json answer.json")
try:
errors = validate(load_json(Path(sys.argv[1])),
load_json(Path(sys.argv[2])))
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
print(f"INPUT ERROR: {exc}")
return 2
if errors:
for error in errors:
print(f"FAIL: {error}")
return 1
print("PASS: citation IDs are present in this retrieval snapshot")
return 0

if name == "main":
raise SystemExit(main())
Run python3 check_citations.py retrieved.json answer.json. The bad fixture prints FAIL: IDs not retrieved: ['eval-note-7'] and exits 1. Replace the citations array with ["eval-note-1"] and it prints PASS: citation IDs are present in this retrieval snapshot with exit 0. Remove all citations while keeping status as answered and it fails because the contract requires a citation.
What does a passing result not prove?
A retrieved document can be cited for the wrong sentence. A document can be outdated, adversarial, or misread. A source ID can exist while its text says the opposite of the generated claim. This program cannot compare natural-language claims with passages, evaluate source authority, detect prompt injection, or determine whether the answer should have abstained.
In this Ranknod editorial example, PASS means only that the citation ID appears in the retrieved set; claim-level support still needs review.
That boundary is the point of the tool. A known structural failure can be caught cheaply and consistently. Factual support needs a separate review that compares each consequential claim with the exact source passage, plus tests of behavior on relevant and irrelevant retrieval sets. NIST's AI RMF calls for defined evaluation and human oversight in context; a source-ID check is only one small layer.
Keep the retrieval snapshot and model output together for a permissioned test run. If the output fails, do not render the apparently supported answer and hope the reader notices. Return a controlled error or an approved fallback, then inspect why the generator produced an unknown identifier.

Find Sitemap URLs with Conflicting Canonicals

Ranknod — Thu, 24 Sep 2026 10:52:19 +0000

A sitemap lists /guides/, but the page declares /resources/guides/ as its canonical URL. Which address should your publishing pipeline keep? Audit the sitemap entry, the final URL after redirects, and the HTML canonical separately. Report the disagreement; then a human decides which URL represents the page.

Google recommends putting the URLs you want shown as canonical in a sitemap and avoiding contradictory canonical signals. A sitemap entry and an HTML rel="canonical" are signals, though: the checker below finds inconsistent declarations. It cannot declare which URL Google will choose. See Google's sitemap guidance and canonical methods.

AI disclosure: This article was drafted with AI. The author must verify its technical claims and code before publication and select the truthful DEV editor disclosure tier.

What exactly are we checking?

For every <loc> in one XML URL sitemap, compare three values:

Value	What it tells us
Sitemap URL	The URL the publisher submitted as a canonical candidate.
Final response URL	The URL reached after ordinary HTTP redirects.
HTML canonical	The URL named by a `<link rel="canonical" href="…">` in the returned HTML.

For example, if a sitemap includes https://example.test/a and the fetched HTML points to https://example.test/b, the script prints CONFLICT. If /a redirects to /b, it also prints REDIRECTED. Neither output is proof of a search penalty; both indicate a place to examine your routing and page-generation rules.

Run a bounded local audit

Save the following as audit_canonicals.py. Pass it a trusted local sitemap XML file, not a remote URL. It checks at most 200 entries, uses an eight-second request timeout, and reads only the first 512 KB of each HTML response. Run it only against a site you are authorized to inspect. Python 3.12 needs no third-party package.

from html.parser import HTMLParser
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urljoin, urlsplit, urlunsplit
from urllib.request import Request, urlopen
import sys
import xml.etree.ElementTree as ET




def local_name(tag):
    return tag.rsplit("}", 1)[-1]




def sitemap_urls(path):
    root = ET.parse(path).getroot()
    if local_name(root.tag) != "urlset":
        raise ValueError("Provide a URL sitemap, not a sitemap index")
    for entry in root:
        if local_name(entry.tag) == "url":
            for child in entry:
                if local_name(child.tag) == "loc" and child.text:
                    yield child.text.strip()
                    break




class CanonicalLinks(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []


    def handle_starttag(self, tag, attrs):
        if tag.lower() != "link":
            return
        values = dict(attrs)
        if "canonical" in (values.get("rel") or "").lower().split():
            if values.get("href"):
                self.hrefs.append(values["href"])




def comparable(url):
    parts = urlsplit(url)
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower(),
                       parts.path or "/", parts.query, ""))




def audit(path):
    urls = list(sitemap_urls(path))
    if len(urls) > 200:
        raise ValueError("Split the sitemap or adjust the audited limit")


    for url in urls:
        if urlsplit(url).scheme not in ("http", "https"):
            print("INVALID_URL", url, sep="\t")
            continue
        try:
            request = Request(url, headers={"User-Agent": "CanonicalAudit/1.0"})
            with urlopen(request, timeout=8) as response:
                final_url = response.geturl()
                if "text/html" not in response.headers.get("Content-Type", ""):
                    print("REVIEW_NON_HTML", url, sep="\t")
                    continue
                body = response.read(512_000)
        except (HTTPError, URLError, TimeoutError) as exc:
            print("FETCH_ERROR", url, type(exc).__name__, sep="\t")
            continue


        parser = CanonicalLinks()
        parser.feed(body.decode("utf-8", errors="replace"))
        canonicals = [comparable(urljoin(final_url, href))
                      for href in parser.hrefs]
        notes = []
        if comparable(url) != comparable(final_url):
            notes.append("REDIRECTED")
        if not canonicals:
            notes.append("REVIEW_NO_HTML_CANONICAL")
        elif len(set(canonicals)) > 1:
            notes.append("REVIEW_MULTIPLE_CANONICALS")
        elif canonicals[0] != comparable(url):
            notes.append("CONFLICT")
        print("|".join(notes) or "OK", url, sep="\t")




if __name__ == "__main__":
    audit(Path(sys.argv[1]))

Run python3 audit_canonicals.py sitemap.xml. A sitemap entry with an explicit matching HTML canonical prints OK. A missing HTML canonical prints REVIEW_NO_HTML_CANONICAL, which is a review item, not an automatic error. A redirect and mismatch can appear together.

For a small local check, I ran the script against a temporary HTTP server with these fixtures:

Fixture path	Returned page behavior	Observed classification
`/ok`	Canonical points to `/ok`.	`OK`
`/wrong`	Canonical points to `/other`.	`CONFLICT`
`/missing`	No HTML canonical tag.	`REVIEW_NO_HTML_CANONICAL`
`/redirect`	Redirects to `/ok`; sitemap still lists `/redirect`.	`REDIRECTED
{% raw %}`/multiple`	Two different HTML canonicals.	`REVIEW_MULTIPLE_CANONICALS`

The five outputs matched the expected classifications under Python 3.12. This is a test of the script's branch behavior, not proof that a real crawler interprets every live site in the same way. Recreate equivalent fixtures for your site's redirect and template rules before running a larger audit.

The comparison removes URL fragments and lowercases the scheme and host. It deliberately preserves paths and query strings: changing either can identify a different resource. Review harmless URL normalization differences rather than rewriting them blindly.

Verify a reported conflict before fixing it

Take one CONFLICT row and answer four questions: Did the submitted URL redirect? Is the HTML canonical absolute or relative? Is the destination page actually the preferred version? Does the sitemap generator use the same URL rule as the page renderer? Fix the responsible generator or redirect rule, then rerun the audit.

This small script reads one uncompressed URL sitemap, follows normal redirects, and scans the fetched HTML only. It does not parse sitemap indexes, render JavaScript, inspect Link HTTP headers, enforce robots.txt, or tell you what Google selected. It also uses a simplified UTF-8 decode; adapt it for a site's declared charset. Large sites should add rate limiting, host allowlisting, index traversal, and respectful scheduling before running a broader audit.

The useful output is a review queue with reasons, not a green SEO score. A person can inspect the URL pair, correct conflicting publication signals, and verify the intended result with the appropriate search tools.

How to Bound AI Agents With Policy Gates

Ranknod — Tue, 22 Sep 2026 14:54:25 +0000

An AI agent becomes much more interesting the moment it can do something.
It can send the email. Update the record. Publish the page. Delete the file. Trigger the deployment.
That is also the moment when a prompt stops being a sufficient safety boundary.
You can tell an agent:

Never make a destructive production change without approval.

But the component interpreting that instruction is also the component deciding what to do next.
If the action matters, "the model was told not to" is a weak enforcement mechanism.
A stronger design puts a policy gate between the agent and the side effect.

The agent should propose. Another layer should decide.

The simplest mental model is:

User / Workflow
      ↓
     Agent
      ↓
Proposed Action
      ↓
  Policy Gate
   ↙   ↓   ↘
Allow Review Deny
   ↓          ↓
Executor    Stop

The important separation is between:
reasoning authority and execution authority.
The model may decide that calling delete_document appears useful.
It should not automatically follow that the model has authority to perform that deletion.
OWASP describes excessive agency as a risk created by excessive functionality, permissions, or autonomy. Its mitigation guidance includes minimizing tool permissions, requiring human approval for high-impact actions, and enforcing authorization in downstream systems instead of depending on the LLM to decide whether an action is allowed.
That gives us a useful architecture principle:

Treat the agent as a proposer, not as the final authority.

What a policy gate actually evaluates

A useful policy gate should receive more than the name of the tool.
Consider these two calls:

update_file("/tmp/draft.md")
update_file("/app/production/config.json")

The operation may be the same.
The consequence is not.
A policy decision may need context such as:

type AgentAction = {
  actorId: string;
  userId: string;
  tenantId: string;
  tool: string;
  operation: string;
  resource: string;
  environment: "dev" | "staging" | "production";
  arguments: Record\<string, unknown>;
};

The gate can then return something explicit:

type PolicyDecision =
  | { effect: "allow"; reason: string }
  | { effect: "review"; reason: string }
  | { effect: "deny"; reason: string };

This is more useful than a single isSafe boolean because not every action belongs in one of two buckets.
Some actions should run automatically.
Some should never run.
Some are acceptable only after human review.

Start with three decision classes

A first implementation can be boring.
That is a feature.

function evaluate(action: AgentAction): PolicyDecision {
  if (
    action.environment === "production" &&
    action.operation === "delete"
  ) {
    return {
      effect: "deny",
      reason: "Production deletion is not available to this agent."
    };
  }

  if (
    action.environment === "production" &&
    action.operation === "write"
  ) {
    return {
      effect: "review",
      reason: "Production writes require human approval."
    };
  }

  if (action.operation === "read") {
    return {
      effect: "allow",
      reason: "Read operation is within the permitted scope."
    };
  }

  return {
    effect: "deny",
    reason: "No explicit policy permits this action."
  };
}

Then force every effect-producing tool call through the gate:

async function dispatch(action: AgentAction) {
  const decision = evaluate(action);

  if (decision.effect === "deny") {
    throw new Error(\`Blocked by policy: ${decision.reason}\`);
  }

  if (decision.effect === "review") {
    return queueForHumanApproval(action, decision);
  }

  return executeRegisteredTool(action);
}

This is illustrative code, not a production authorization system.
The important part is the shape of the architecture.
There should not be an alternative execution path where the agent can bypass dispatch() and call the underlying tool directly.

Deny by default

A common mistake is to define the dangerous things you know about and allow everything else.
That works until somebody adds a new tool.
Suppose today's agent can:

read CMS entries;
edit drafts;
publish pages.

You create a rule requiring approval for publish.
Three weeks later, another developer adds:

bulk_publish

If unknown operations are allowed by default, your policy boundary has silently expanded.
A safer rule is:

No matching permission means no execution.

This makes new capabilities opt-in rather than accidentally inherited.

Put policy outside the prompt

Prompts are useful for guiding behavior.
They are not equivalent to authorization.
The difference becomes clearer when you imagine prompt injection, stale context, hallucination, or simply a model interpreting a vague instruction differently than you expected.
Policy should operate on the proposed action after the model has decided what it wants to do but before the side effect occurs.
Systems such as Open Policy Agent formalize a similar separation between policy decision-making and policy enforcement. OPA receives structured input, evaluates policy separately from application logic, and returns a decision to an enforcement point.
You do not need OPA to use this pattern.
The architectural separation is the useful part.

Bind the decision to real context

A policy gate becomes more useful as it understands the context that changes risk.
For example:

operation = send_email

is not enough.
You may need:

recipient = internal
recipient = customer
recipient = 8,000-person mailing list

Likewise:

operation = deploy
environment = staging

is different from:

operation = deploy
environment = production

Useful policy inputs can include:

authenticated user;
tenant;
environment;
resource;
tool;
operation;
argument values;
current workflow stage;
data classification;
approval status;
time or expiry;
rate limits.

Do not make the LLM invent these values when authoritative system data exists.
Resolve them from trusted application state.

Approval should interrupt execution, not decorate it

A bad human-in-the-loop implementation looks like this:

the agent decides;
the operation begins;
the UI asks somebody whether the action was okay.

That is auditing, not approval.
Real approval changes the execution state:

PROPOSED
   ↓
POLICY CHECK
   ↓
REQUIRES APPROVAL
   ↓
PAUSED
   ↓
HUMAN DECISION
 ↙           ↘
APPROVED     REJECTED
   ↓             ↓
EXECUTE         STOP

The action should remain unexecuted until the required approval exists.
For sensitive workflows, the reviewed payload should also be the payload that executes. If an agent can change arguments after approval, the human approved an explanation rather than the real operation.

Minimize the agent's credentials too

Policy gates are not a replacement for least privilege.
If an agent should only read a database, do not give its tool identity write privileges and depend on the policy layer to behave forever.
If it should only create drafts, do not give it unrestricted publication credentials.
The layers should reinforce one another:

Agent behavior
    ↓
Tool allow-list
    ↓
Policy gate
    ↓
Human approval where needed
    ↓
Least-privilege credential
    ↓
Downstream authorization

A failure in one layer should not immediately become unrestricted authority.

Log policy decisions, not just agent messages

When an incident happens, the interesting question is not only:

What did the model say?

You also need to know:

what action was proposed;
which policy version evaluated it;
what decision was returned;
why;
whether approval was required;
who approved it;
what ultimately executed.

OPA, for example, supports decision logs containing policy-query inputs and decisions specifically for auditing and debugging.
The same principle applies even if your policy system is custom.
Generated reasoning is not an audit trail.

A practical implementation sequence

Do not start by trying to govern every possible agent behavior.
Start with side effects.
Inventory the tools that can change something outside the model's context window.
Then classify each operation:
Auto-allow: low-consequence, reversible operations.
Review: consequential operations that are legitimate but require judgment.
Deny: operations the agent should never perform in this role.
Then make sure there is exactly one governed path into the executor.
Only after that should the policy model become more sophisticated.

The useful boundary is outside the model

Better models will reduce some errors.
Better prompts will improve some behavior.
Neither changes the fundamental authority problem.
If an agent can affect another system, you need to decide where the model's discretion stops.
A policy gate makes that boundary executable.
The agent can still reason, plan, adapt, and propose.
It simply cannot convert every conclusion into a side effect on its own.

Implementation checklist

Inventory every agent tool that creates an external side effect.
Remove unnecessary tools and permissions.
Normalize proposed actions into structured inputs.
Evaluate policy outside the LLM.
Default unknown actions to deny.
Require human approval for selected high-impact operations.
Make every side-effecting call pass through the enforcement path.
Preserve the exact approved parameters where approval is required.
Use least-privilege downstream credentials.
Log policy decisions separately from model output.
Test bypass paths, not just expected behavior.

The architectural question I would ask about any agent system is simple:
If the model decides to do something it should not do, what component has the authority to stop it?

Why AI Agents Need an Execution Boundary

Ranknod — Fri, 18 Sep 2026 18:38:45 +0000

Imagine an AI agent is reviewing a page before publication.

It decides the page is ready, calls publish_page, and the request succeeds. But the response times out.

The agent cannot tell whether publication happened, so it tries again.

Now imagine something else changed between those attempts: the page was edited, the user's permissions changed, or an approval was revoked.

The problem is no longer whether the model made a reasonable decision. The problem is that model intent has been allowed to become a real-world side effect without enough deterministic control around it.

That is why AI agents that can change external systems need an execution boundary.

The model should produce intent. Application code should control authority and side effects.

A useful architecture looks like this:

AI reasoning
↓
Structured action proposal
↓
Execution boundary
↓
External system

The execution boundary is not just a human-approval step. It is the layer responsible for validation, authorization, policy enforcement, workflow state, approvals, idempotency, retries, execution, auditing, and verification.

Why direct tool execution becomes risky

A simple agent prototype often looks like this:

User request
↓
LLM
↓
Tool call
↓
External system

The model decides that publish_page is appropriate, and the application immediately performs the action.

That is convenient during prototyping, but several different responsibilities have now been collapsed into one decision.

Before a page is actually published, the system may still need to ask:

Is publish_page a supported operation?
Is the payload valid?
Is this user allowed to publish this page?
Is the page still in a publishable state?
Does publication require approval?
Is that approval still valid?
Has this operation already run?
Can it be retried safely?
Did publication actually produce the intended result?

Those are not questions the model should resolve by itself.

They belong to deterministic application logic.

Put an execution boundary between intent and action

Instead of allowing the agent to invoke privileged side effects directly, let it propose an action.

For example:

{
  "type": "publish_page",
  "resourceId": "page_284",
  "reason": "draft_ready_for_publication",
  "operationId": "op_8f219"
}

This object represents what the agent wants to happen.

It does not prove that the action is allowed.

A safer flow is:

Agent proposal
↓
Schema validation
↓
Authorization
↓
Policy evaluation
↓
Approval if required
↓
Pre-execution validation
↓
Execution
↓
Verification
↓
Audit record

That middle layer is the execution boundary.

The model remains useful for reasoning. The application remains responsible for deciding whether the proposed action may affect the outside world.

Use structured proposals instead of free-form instructions

Free-form output is difficult to validate reliably.

A bounded action vocabulary gives the application something predictable to inspect:

type AgentAction =
  | {
      type: "publish_page";
      resourceId: string;
      operationId: string;
    }
  | {
      type: "create_ticket";
      projectId: string;
      title: string;
      operationId: string;
    };

This example is illustrative rather than production-tested, but the design principle matters.

The agent can select from known actions. It cannot invent a privileged operation simply by describing one convincingly.

Application code can then own the control path:

async function processAgentAction(
  action: AgentAction,
  actor: Actor
) {
  validateSchema(action);
  await authorize(actor, action);
  await enforcePolicy(action);

  if (requiresApproval(action)) {
    return queueForApproval(action, actor);
  }

  return executeIdempotently(action, actor);
}

The important separation is:

The model produces intent. The application grants authority.

That makes the boundary easier to test, audit, and reason about.

Authorization and policy belong outside the model

Suppose the agent proposes:

{
  "type": "publish_page",
  "resourceId": "page_284",
  "operationId": "op_8f219"
}

The application still needs to decide whether publication is allowed.

That may depend on:

the authenticated user;
the user's role;
ownership of page_284;
the current environment;
the page's workflow state;
publishing policy;
risk level;
approval requirements.

An agent may understand how publishing works without having authority to publish.

That distinction is useful across agent roles.

A research agent might be allowed to search and summarize. An editing agent might be allowed to create a draft. A publishing agent might be allowed to request publication.

request publication does not have to mean publish immediately.

Capability and permission are separate concerns.

Make approval part of the workflow state

If an action needs human approval, approval should be represented in application state rather than only as an instruction in a prompt.

For a publishing workflow, the states might look like this:

PROPOSED
↓
VALIDATED
↓
AWAITING_APPROVAL
↓
APPROVED
↓
EXECUTING
↓
VERIFYING
↓
COMPLETED

Failure or exception states might include:

REJECTED
VALIDATION_FAILED
EXECUTION_FAILED
RETRY_PENDING
CANCELLED

This gives the application explicit answers to operational questions:

Which actions are waiting for approval?
Which exact proposal was approved?
Who approved it?
Can it still be cancelled?
What should happen after execution fails?
Which state transitions are legal?

It also prevents "approval required" from becoming a vague behavioral suggestion to the model.

Recheck important assumptions before execution

Approval itself should not automatically make an action safe forever.

Imagine page_284 is approved for publication at 10:00.

Before the executor runs, one of these things changes:

the page is edited;
the user loses publishing permission;
the resource is deleted;
policy changes;
the approval is revoked;
the approved version no longer matches the current version.

The executor should not assume that an earlier approval proves the current action is still valid.

For important side effects, recheck authorization, relevant state, and approval validity immediately before execution.

An approval should authorize a specific action under specific conditions, not create unlimited future authority.

Make retries safe with idempotency

Now return to the original publishing example.

The executor sends a request to publish page_284.

The request succeeds.

The response times out.

From the application's point of view, the result is ambiguous.

Retrying blindly may cause the same side effect twice.

That is where idempotency becomes part of the execution boundary.

The proposal may contain an operation identifier:

{
  "type": "publish_page",
  "resourceId": "page_284",
  "operationId": "op_8f219"
}

Before execution, the application checks whether op_8f219 has already completed or is already in progress.

There is one important detail here: an identifier should not automatically become trustworthy just because the model supplied it.

Ideally, trusted orchestration or application code should create the operationId, or at minimum validate it before use.

The model should not control the transactional identity of privileged operations without checks.

Once the operation is tracked by trusted code, the execution layer can safely handle:

duplicate requests;
retries;
timeouts;
in-progress operations;
completed operations;
recovery or compensation logic.

The model does not need to remember transactional state. The system does.

Verify the outcome, not just the API call

Even successful execution is not necessarily the end of the workflow.

Suppose the publishing API returns 200 OK.

Does that prove the new page is live?

Not always.

The request may have been accepted while a downstream process later fails. A public page may still display an older cached version. A deployment may start successfully but fail during rollout.

For meaningful side effects, verification should be explicit.

A workflow record might contain:

proposed action;
validation result;
authorization result;
approval record;
operation ID;
execution response;
verification result;
final state.

That gives COMPLETED a stronger meaning.

It means the system observed the intended result, not merely that it sent a request.

Test the execution boundary without the LLM

Separating reasoning from execution also makes the system much easier to test.

The executor should be testable independently from the model.

You can submit:

a valid authorized proposal;
a malformed proposal;
an unsupported action;
an unauthorized action;
an action missing required approval;
an action whose approval became stale;
a duplicate operation ID;
an execution failure;
a verification failure.

Then assert the expected result and state transition.

For example:

Unauthorized proposal
→ rejected before execution

Approved but stale proposal
→ returned for revalidation

Duplicate operation
→ not executed twice

Execution succeeds, verification fails
→ not marked COMPLETED

The reasoning layer can remain probabilistic.

The side-effect controls do not have to be.

Implementation checklist

Before allowing an AI agent to change a real system, check:

Does the agent propose actions instead of receiving unrestricted execution access?
Is every action represented by a narrow, validated schema?
Is the action vocabulary explicitly bounded?
Is authorization enforced outside the model?
Are policy checks deterministic where possible?
Are approval requirements represented in workflow state?
Is the approved action tied to the correct resource and version?
Are authorization and relevant state rechecked before important execution?
Are operation IDs created or validated by trusted application code?
Are retries protected by idempotency?
Are failures observable and recoverable?
Is the external result independently verified where necessary?
Is enough audit data recorded to reconstruct what happened?
Can the execution layer be tested without invoking the model?

The core idea is simple:

Let the model decide what it wants to do. Let deterministic application logic decide whether that action is valid, authorized, approved, safe to execute, and actually completed.

That execution boundary is what turns tool use into a controlled system rather than a direct path from model output to real-world side effects.

Where would you place that boundary in your current agent architecture?

AI disclosure: This article was created with AI assistance. The human author is responsible for validating the technical content before publication.

Hey we are going to live on Dev. Community

Ranknod — Tue, 15 Sep 2026 00:47:28 +0000

yeah just wanted to say that from now on we are also going to be posting on the dev community and answering all your quistions you have. this is going to be very interesting journey.