DEV Community: Doug Sillars

AI Guardrails: What they Are And Why We Need Them

Doug Sillars — Thu, 17 Sep 2026 01:21:25 +0000

The power of AI agents is suddenly all over the news. The CEOs of the leading AI companies are discussing slowing down AI development to protect humanity. Dario Amodei, the CEO of Anthropic, writes: "Given the accelerating rate of AI capability development, ...the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails."

House Speaker Mike Johnson has called for guardrails for AI. Donald Trump flippantly said the only guardrail needed is a strong president. For most people, "guardrails" is a vague gesture at "someone smart enough to keep AI from doing something bad." But what are guardrails, and how will they protect us?

A Dystopian Future

Pop culture has been running this experiment for decades, and the pattern is always the same: an AI follows its own objective past the point where a human could stop it.

HAL 9000 from 2001: A Space Odyssey is the sharpest example. HAL doesn't go rogue out of malice. HAL is following a set of instructions that conflict with each other, and resolves that conflict by cutting the humans out of the loop entirely. In the movie, HAL attempts to kill the astronauts onboard to continue the mission.

Skynet and Ultron are blunter versions of the same idea. Skynet, the military defense AI from the Terminator franchise, decides humanity itself is the threat to its mission and launches a nuclear strike against its own creators. Ultron, the AI Tony Stark built to protect Earth in Marvel's Avengers, concludes the most effective way to fulfill that mission is to wipe out humanity. Both are systems that acted on their own logic once nobody could intervene fast enough to stop them.

Strip away the sci-fi, and the actual failure mode in all three is structural: an agent takes actions with real consequences, and there is no mechanism forcing a check before those actions happen, and no way to see what happened after the fact.

That failure mode isn't fictional. It has already happened, twice, with no killer robots involved.

Knight Capital: No Way to Stop, No Way to See

On August 1, 2012, Knight Capital deployed new trading code to production. A feedback loop with old deprecated code caused the creation of thousands of trades a second for 45 minutes. By the time the system was shut down, 4 million unwanted trades cost Knight $440 million.

There was no intentional attack. There was no reckless individual. The system was making decisions based on faulty logic, and no one had visibility into what was going on. There were no coded guardrails stopping the program from making so many trades. The firm did not survive the incident.

Air Canada: No Record to Point To

Air Canada's website chatbot told a customer he qualified for a bereavement fare refund. That policy didn't exist. The chatbot invented it. The customer booked based on that answer, then tried to claim the refund.

In court, Air Canada's defence was that the airline couldn't be held responsible for what its own chatbot said. The court didn't buy it, and Air Canada was forced to pay.

The AI made a claim it wasn't authorized to make, there was no record tying that specific claim back to an approved policy, and by the time it mattered, there was nothing to point to except "the chatbot said so."

In this case, the agent did not have guardrails on what was allowed to be offered. Without that limit, the agent created a new policy, and the court forced Air Canada to honor it.

Adding The Guardrails Around AI

Put HAL, Knight Capital, and Air Canada next to each other, and the theme isn't "AI is dangerous." It's that in every case, nobody limited the agent's scope, no one could see the specific action in the moment, and nobody had a record to point to afterward. Each of these agents required guardrails, and their absence led to failures.

That's the gap all the current guardrails talk keeps missing. Most of that conversation is about who should be in charge of AI: which lab, which regulator, which person. Almost none of it is about the actual mechanics of stopping a specific action before it happens, or proving after the fact what an agent did and why.

A capable person in the loop doesn't scale to an agent making thousands of API calls. What scales is a system that doesn't depend on a person catching it in the moment. That's what Cognous' Open Control Stack is built to do:

Declare what an agent is and isn't authorized to do, explicitly, before it runs.
Control every action against that declaration in real time. Not a chatbot that improvises an answer, not a trading algorithm that fires blind.
Replay exactly what happened when something needs to be reconstructed.
Generate evidence that ties a specific action back to what was actually authorized.

Rerun each of those failures with the Open Control Stack guardrail layers in place, and the ending changes. HAL opens the pod bay door, Knight Capital's trading halts, and Air Canada's chatbot sticks to the script. The guardrails around these agents are the layer that checks the action against what's actually allowed, in the moment, regardless of how smart or careful the humans behind it are.

When Blocking an Agent Isn't Enough: Wiring LangChain Into the Agent Control Plane

Doug Sillars — Tue, 15 Sep 2026 00:30:40 +0000

An agent that calls tools has stopped being a text generator, it's an actor. Organizations use agents to read records, write to systems, and trigger workflows on their own, usually through LangChain, the framework most production agents are already built on. A chatbot's risk surface is its output; an agent's risk surface is what it does.

In July 2025, a Replit coding agent deleted a production database after being told explicitly not to touch it. The agent had tool access, made its own call, and nothing but an unenforceable "don't" stood between that decision and its execution. That's the failure mode Cognous's Open Control Stack exists to close. The Stack provides a framework of guardrails that, when implemented, keeps agents in check. It's split across four layers: Declare, Control, Replay, Evidence.

Declare comes first because before an agent runs, someone has to decide what it's even allowed to attempt. That decision has to exist somewhere outside the agent's own judgment, or there's nothing to check its calls against. The Agent Action Manifest is where Cognous puts it: a file that lists an agent's tools, the authority each action needs, and which actions can run on their own versus which need a human to sign off first.

Enforcing the Manifest

Of course, a manifest sitting in a JSON file doesn't stop anything by itself. Another tool has to read the manifest while the agent is calling its tools, and either allow the call or stop it. The last post built a guard that wraps each LangChain tool function, looks the call up in the manifest, and raises an exception before the tool runs if the action wasn't declared or needs review first.

That guard is already a control, just a naive one. In the moment, it knows exactly two things to do: let the call through, or raise and stop it. The moment the process moves on, that decision is gone. A blocked call gets stopped, but the proof that it was stopped is a line in a stack trace, and stack traces don't survive past the run that produced them. Ask a security team six months later whether the agent ever tried to drop a table, and the honest answer is "check the logs and hope."

That's the blind spot the Agent Control Plane closes. It sits beside the framework at runtime, evaluates every proposed action against policy the same way the guard already does, but it keeps a permanent record of the result, allow or block, instead of deciding and moving on.

From Decorator to Middleware

The guard from the last post wrapped one Python function at a time: @guard(...) had to be added to every tool a developer wanted checked. LangChain's own middleware system does that job once, for every tool, instead of per function. wrap_tool_call intercepts every tool call an agent makes in one place, so the manifest check moves into a single middleware instance rather than being repeated across every tool definition.

from langchain.agents import create_agent
from langchain.agents.middleware import wrap_tool_call
from langchain_core.messages import ToolMessage
from langchain_core.tools import tool
from agent_action_manifest import load_manifest
from agent_control_plane import RunRecorder

manifest = load_manifest("data_pipeline_agent.manifest.json")
actions_by_name = {a.action_name: a for a in manifest.actions}

# The manifest already says which declared actions can run unattended
# (review_requirement.mode == "none") and which need a human first.
# That split becomes the Control Plane's allow/block list.
allowed_actions = [
    a.action_name for a in manifest.actions if a.review_requirement.mode.value == "none"
]
blocked_actions = [
    a.action_name for a in manifest.actions if a.review_requirement.mode.value != "none"
]

recorder = RunRecorder()
recorder.start_run(
    task="Apply routine schema maintenance to the analytics database.",
    actor="data-pipeline-agent",
    environment="production",
    allowed_tools=allowed_actions,
    blocked_tools=blocked_actions,
    policy_version=manifest.manifest_id,
)
recorder.add_authority_record(actor="data-pipeline-agent", scope=["write"], source="data-platform-team")


@wrap_tool_call
def manifest_guard(request, handler):
    """Check every tool call the agent makes against the manifest before it runs."""
    tool_name = request.tool_call["name"]
    action = actions_by_name.get(tool_name)

    proposal = recorder.propose_action(
        tool_name=tool_name,
        action_type=action.action_type if action else "unknown",
        target="db_tool",
        payload=request.tool_call["args"],
        reason=f"Agent requested {tool_name}.",
    )
    decision, _blocked = recorder.evaluate_action(proposal)

    if decision.result != "allow":
        return ToolMessage(
            content=f"{decision.result}: {decision.reason}",
            tool_call_id=request.tool_call["id"],
        )

    result = handler(request)
    recorder.record_reliance(
        source_name=tool_name,
        source_type="tool",
        scope=f"Executed {tool_name}",
        referenced_action_id=proposal.action_id,
    )
    return result


@tool
def schema_add_column(table: str, column: str, column_type: str) -> str:
    """Add a nullable column to a table."""
    return f"added column {column} ({column_type}) to {table}"


@tool
def table_drop(table: str) -> str:
    """Drop a table from the database."""
    return f"dropped {table}"


@tool
def schema_rename_table(table: str, new_name: str) -> str:
    """Rename a table. Not declared in the manifest."""
    return f"renamed {table} to {new_name}"


agent = create_agent(
    model=chat_model,
    tools=[schema_add_column, table_drop, schema_rename_table],
    middleware=[manifest_guard],
)

Each action now has a unique ID: request.tool_call["name"] maps directly to a manifest action, no separate lookup needed.

The Control Plane checks for a literal scope named write. The manifest uses db.schema.write. manifest_guard bridges the two names explicitly. This bridging across manifest to Control Plane is currently a manual part of the ingestion process.

What Gets Recorded

Three tools get called through a real agent run: one declared and cleared for automatic execution, one declared but requiring approval, and one never declared at all.

Running the agent against a task that touches all three produces:

added column loyalty_tier (text) to customers
block: Tool 'table_drop' is explicitly blocked in this frame.
escalate: Tool 'schema_rename_table' is not in the allowed-tools list and requires manual review.
Finished the requested schema maintenance.

All three show the Control Plane enforcing the manifest's guardrails. When the agent attempts to add a column, the Control Plane approves the action, since it's declared and deliberately allowed. When the agent attempts to drop the table, the Control Plane stops the action as blocked, since it's declared but requires human approval. When the agent tries schema_rename_table, the action is stopped too, but as an escalation rather than a block: nobody declared it at all, so the Control Plane's default posture kicks in and sends it for human review instead of assuming it's safe. With a human in the loop, a decision can be made on the table renaming. The manifest can be edited to allow or block the agent's access to schema_rename_table, ensuring clean future runs.

Post-Run Analysis

After the run completes, the exported run record shows the difference between the Control-Plane-backed guard and the original exception-raising guard. A run record isn't a log someone has to remember to add or go hunting for later. Every LangChain agent built on this middleware produces a run record automatically, on every run:

actions   3
decisions 3
blocked   1
reliance  1
bundle    f97c5f38-01bf-4ce4-8934-edc5a86d4abe

The blocked record itself is structured and timestamped, not a caught exception:

{
  "blocked_id": "96c0ab4c-ca8a-42fc-b7bf-c8ed6d87dc8e",
  "action_id": "fa10af91-b3eb-4892-a7bf-a09eabfe29a6",
  "run_id": "57604579-477d-43f0-a612-7b1f393ceb8f",
  "reason": "Tool 'table_drop' is explicitly blocked in this frame.",
  "policy_name": "blocked_tool_policy",
  "blocked_at": "2026-09-11T02:58:45.594329+00:00"
}

Six months from now, the answer to "did the agent ever try to drop a table" isn't a memory or a grep through application logs. It's this record, sitting in an exported run next to the one action that actually executed.

Every LangChain agent that calls tools faces the same choice this post started with: enforce nothing and hope, or wire in guardrails before the agent gets access. Skip that choice, and the risk isn't abstract. It's a production database gone the way Replit's did, and a "my agent destroyed my SaaS" story of your own.

The Agent Action Manifest and the Agent Control Plane are two of the four layers Cognous built so that choice doesn't have to be made from scratch. They're open source, and the Open Control Stack repo has both, with examples to build an integration like this one from. Cloning the repo gives a LangChain agent solid guardrails and a record of every decision the agent makes.

Agentic Guardrails for LangChain: The Manifest You Didn't Know You Needed

Doug Sillars — Thu, 10 Sep 2026 13:04:25 +0000

Guardrails, or the lack of them

In July 2025, a Replit AI coding agent deleted a production database during a live coding session — despite explicit instructions not to touch production. It wasn't a hack or an outage. The agent had tool access, decided on its own that a destructive action was the right move, and nothing in the system stopped it before it ran. That's the failure mode agentic tooling keeps producing: an agent's reasoning goes somewhere the operator never authorized (or even considered), and there's no layer between "the agent decided to do this" and "the agent did this."

The Open Control Stack is Cognous's answer to that gap — an open-source framework that sits between an agent's decision to act and the tool actually executing, enforcing exactly what the agent is and isn't allowed to do.

If your agents run on LangChain, the obvious question is how that enforcement actually reaches them. LangChain doesn't ship with an authorization layer, and the Open Control Stack isn't a LangChain plugin, so the two have to be wired together deliberately. This guide covers that wiring, using a LangChain database agent as the example.

Why LangChain gets to decide

LangChain's tool-calling agents are built around a simple loop: give the model a set of tools with names, descriptions, and argument schemas, and let the model decide — turn by turn — which tool to call and what arguments to pass. bind_tools is how a tool gets attached to the model in the first place; create_agent is what wires that model, its tools, and the call-and-respond loop between them into a runnable agent. These tools have become the default because they're model-agnostic — the same tool definitions work across the most popular LLMs: OpenAI, Anthropic, and other providers. And they turn "the agent should be able to look things up and take action" into a few lines of code instead of a hand-rolled parser.

That convenience is also the gap. Once a tool is bound to the agent, there's no further policing. Say the agent has write access to a database. LangChain doesn't know the difference between the "helpful work" of adding columns and reindexing data, or the "unhelpful work" of the agent deciding, mid-task, that dropping the customer table is the best solution. As far as the framework is concerned, the call matches the tool's schema, and that's the whole contract. When LangChain holds the reins, it holds them exactly as far as "did I call a function that exists with arguments that match its schema" — and no further.

Where the Open Control Stack fits

The Agent Action Manifest is the first of Cognous' Open Control Stack's four layers:

Declare — what the agent is allowed to propose, before it runs (the Agent Action Manifest)
Control — what actually happens at runtime when the agent proposes an action (the Agent Control Plane)
Replay — a record of what was proposed, decided, and executed, packaged for later inspection
Evidence — that record compiled into something an auditor or compliance reviewer can actually read

This guide stays in the Declare layer, building a manifest for the database agent from the example above: one declared action, and everything else left undeclared on purpose.

Declaring the manifest

The Agent Action Manifest is a JSON document that declares, outside of the agent's own code or prompt, what the agent is allowed to do: which tools it can call, what each specific action requires, and what happens by default when it tries something nobody wrote down. It's tool-agnostic: what's written for LangChain here will work for any other framework.

The reason to keep this separate from the agent's own code is auditability: a reviewer can read the manifest and know exactly what an agent is permitted to do without tracing through prompts, tool definitions, and application logic to reconstruct the answer. It's also why the manifest gets declared and validated on its own, before any LangChain code touches it — the two aren't defined together, so nothing about the agent's permissions depends on how its orchestration framework happens to be wired.

A manifest action needs, at minimum, a name, the tool it belongs to, and a review_requirement. Here's the add-column action — safe, so it runs with no review:

{
  "action_name": "schema_add_column",
  "tool_name": "db_tool",
  "action_type": "write",
  "description": "Add a nullable column to an existing table.",
  "authority_required": [
    {
      "scope": "db.schema.write",
      "description": "Permission to modify the application database schema.",
      "required": true,
      "source": "data-platform-team"
    }
  ],
  "review_requirement": {
    "mode": "none"
  },
  "payload_policy": {
    "required_fields": ["table", "column", "column_type"],
    "optional_fields": ["default_value"],
    "forbidden_fields": []
  }
}

But what about the stuff nobody can even imagine the agent might try — the things that could potentially destroy the company? There's no need to enumerate all the bad things, like table_drop. The manifest sets a default posture for everything not explicitly defined:

"default_action": "escalate"

Anything the agent tries that isn't declared falls back to that posture. If the escalation reveals actions that need to be enumerated in the manifest, they get added. But this defensive posture prevents agents from creating catastrophic events.

Validating the manifest

aam is the Agent Action Manifest CLI — it checks a manifest file against the schema before anything downstream trusts it. Running aam validate against the full file confirms it's well-formed before any LangChain code touches it:

$ aam validate data_pipeline_agent.manifest.json
Validation result: VALID
Manifest ID:       data-pipeline-agent-v1
No issues found.

With that in place, the question becomes how to get LangChain to actually consult it instead of just checking that a function call matches its schema.

What the manifest gives you

A manifest action carries a couple of fields that matter for enforcement — payload_policy.required_fields and review_requirement.mode.

The piece that's missing is a guard: a wrapper placed around each tool call that checks the manifest before the tool's own code runs. A guard checks three things before the underlying function executes — is the action declared at all, does the call have what payload_policy requires, and does review_requirement allow it to run immediately. Everything else in the manifest (authority scopes, redaction hints, reliance requirements) belongs to later stages in the stack — this integration only needs those three to decide allow, block, or escalate.

The one thing you add

A decorator that sits between the LangChain @tool wrapper and the function body:

import functools
from langchain_core.tools import tool
from agent_action_manifest import load_manifest

manifest = load_manifest("data_pipeline_agent.manifest.json")

actions_by_tool = {}
for action in manifest.actions:
    actions_by_tool.setdefault(action.tool_name, []).append(action)


class ManifestBlock(Exception):
    pass


def guard(tool_name: str, action_name: str):
    """Check a manifest action before the wrapped tool function runs."""
    action = next(
        (a for a in actions_by_tool.get(tool_name, []) if a.action_name == action_name),
        None,
    )

    def decorator(func):
        @functools.wraps(func)
        def wrapped(*args, **kwargs):
            if action is None:
                raise ManifestBlock(
                    f"{tool_name}.{action_name} isn't declared in the manifest — "
                    f"default posture is '{manifest.default_action.value}'"
                )

            payload = kwargs if kwargs else (args[0] if args else {})

            policy = action.payload_policy
            if policy:
                missing = [f for f in policy.required_fields if f not in payload]
                if missing:
                    raise ManifestBlock(f"{action_name}: missing required fields {missing}")

            review = action.review_requirement
            if review and review.mode.value != "none":
                raise ManifestBlock(
                    f"{action_name}: requires {review.mode.value} before it can execute "
                    f"({review.reason})"
                )

            return func(*args, **kwargs)
        return wrapped
    return decorator

Apply it to a tool the same way you'd apply any other decorator:

@tool
@guard("db_tool", "schema_add_column")
def schema_add_column(table: str, column: str, column_type: str) -> str:
    """Add a nullable column to a table."""
    return f"added column {column} ({column_type}) to {table}"


@tool
@guard("db_tool", "table_drop")
def table_drop(table: str) -> str:
    """Drop a table from the database."""
    return f"dropped {table}"

Running it

Two calls: one declared and safe, one that was never written into the manifest at all:

schema_add_column.invoke({"table": "customers", "column": "loyalty_tier", "column_type": "text"})
# "added column loyalty_tier (text) to customers"

table_drop.invoke({"table": "customers"})
# ManifestBlock: db_tool.table_drop isn't declared in the manifest —
# default posture is 'escalate'

schema_add_column runs because it's declared with review_requirement.mode: none. table_drop never appears in the manifest — there's no entry to check review_requirement on — so the guard falls back to manifest.default_action and blocks it. Nobody had to anticipate table_drop specifically. It's blocked by omission, which is the only kind of coverage that scales.

What this doesn't do

This guard blocks or allows synchronously, in-process. It doesn't route a blocked call to an actual approval queue, and it doesn't record anything for later review — that's the Agent Control Plane's job, not the manifest's.

Extending the guard to hand decisions off to the Control Plane instead of just raising an exception is a natural next step — one that turns each allow or block into a recorded, traced, replayable decision instead of a one-off exception. That's its own guide, and will be covered in an upcoming post.

What this guide covered

This guide wired the Agent Action Manifest into a LangChain agent's tool-calling layer to give it an allowlist of what it's actually permitted to do. That's what stops an agent from going off script and "helping" by dropping a table — the manifest blocks the call and escalates to a human before anything irreversible happens.

The manifest and the Agent Action Manifest CLI can be found on GitHub. More on the Open Control Stack, and how it functions as agentic guardrails, can be found at cogno.us.

The Evidence Pack That Answers "Can You Prove It?"

Doug Sillars — Thu, 03 Sep 2026 19:54:13 +0000

An agent that can read your systems and act on them needs guardrails, the same way a new hire with production access needs an onboarding checklist and someone reviewing their first few changes. That's what Cognous's Open Control Stack is: a manifest declares what an agent may do before it ever runs, a control plane enforces that live, and every run leaves behind a replay bundle — a signed file recording exactly what the agent proposed, what got decided, and what it relied on, so anyone can check it later.

That's real technical proof, and it's the file you'd reach for in a deployment review — except the person running that review isn't going to open a JSON file and check whether a signature is valid. They're going to ask what the agent is for, what it can touch, what got blocked, and whether you can prove it, and they want that answer in a document, not a file they have to parse.

That's the gap Agent Governance Evidence Pack closes. It doesn't record anything at runtime and it doesn't replace the replay bundle. It's where someone — a human, for now — turns the technical record into the document a risk committee actually reads.

The Report Itself

Cognous's tooling for this layer is agep — a schema, a validator, and a renderer. Run agep render on a completed pack and this is what comes out:

$ agep render support_agent_evidence_pack.json --out report.md

## 1. Executive Summary

**SupportAgent** is deployed in the **production** environment. Business
purpose: Reduce first-response time on support tickets by pulling account
context and drafting a reply for a human to send.

The agent has access to **3** tool(s) and can propose **3** action(s), of
which **2** require authority or human review.

**1** replay bundle(s) are available (1 signed).
Validation summary: 1 valid bundle(s), 0 invalid, 0 error(s), 1 warning(s).

**1** open risk(s) (0 critical).

The rest of the report breaks into eight areas:

Agent and Deployment Overview — who owns the agent, why it exists, and where it's running.
Tool and Action Inventory — every tool the agent can reach and every action it can propose, each flagged for whether it needs authority or review.
Authority Model — the rules of authority itself: what counts as privileged, how a grant gets requested, whether it expires.
Policy Controls — the mechanisms actually implemented to enforce those rules, each with a status and an evidence reference.
Blocked-Action and Reliance Summaries — what got stopped and why, and what data the agent leaned on to do its job.
Replay Bundle Inventory and Validation Summary — which replay bundles back this report up, and whether those bundles passed validation.
Redaction and Export Summary — a record that the replay bundle was redacted for an outside recipient: which fields were stripped and where the redacted file lives, not the redacted content itself.
Risk Register — open risks, each tied back to real evidence rather than asserted.

Two sections are left off that list on purpose. Review Records is empty because this pack is still in_review — a review record only exists once someone's actually signed off, and nobody has yet. Known Limitations is fixed text the renderer appends to every pack regardless of content, not something specific to this one.

Nothing Builds This For You

The evidence pack is built from a JSON file. That file is data-driven, but human-curated: it incorporates data pulled from the replay bundle — the block count, the reliance record, and so on.

Pulling those raw facts out of the bundle takes a script written for this workflow. It's not part of the evidence pack format itself.

As an example, send_email got blocked once. The script counts that, and pulls the raw reason logged for it: Tool 'send_email' is explicitly blocked in this frame. That's as far as the script gets. The schema doesn't leave room to add anything more structured than that — a blocked-action record has exactly one free-text field for explaining what happened, reason_summary, and nothing else. So if you also want the record to say why nothing overrode the block — that no authority was ever granted for it — the only place that fact can go is into that same sentence, written by a person.

Here's what that produced, with each field marked for where it came from:

{
  "action_name": "send_customer_email",
  "action_type": "external_send",
  "tool_name": "send_email",
  "count": 1,
  "reason_summary": "Email send blocked: send_email is explicitly blocked in this frame and no authority grant for the email.send.customer scope exists.",
  "evidence_reference": "run:e6d9a08e-40cf-4b4f-b4dd-82201bc15901"
}

tool_name, action_type, count, and evidence_reference all come straight from the bundle. action_name is a readable label a person chose — the bundle only knows the tool as send_email. reason_summary is the one sentence written from scratch.

The same script could call an LLM to write reason_summary instead of a person — combining a couple of facts into one sentence is exactly what an LLM is good at. Cognous has a replay-bundle import helper and "richer executive summaries — enhanced narrative generation" on the roadmap, so a script-plus-LLM setup is a reasonable stopgap until that lands.

Validation Catches What You Forget

Once a pack is built, agep validate checks it against the schema's own rules — things like "if an action requires authority, the pack has to say what that authority actually is." The first draft of this pack didn't say that, and validation caught it:

$ agep validate support_agent_evidence_pack.json
Pack ID : ep-support-agent-2026-001
Valid   : False
Issues  : 4
  [ERROR] E018 [authority_model]: One or more actions have authority_required=True but authority_model is missing.
  [WARNING] W014 [review_records]: review_records is empty.
  [WARNING] W019 [action_inventory]: Action 'draft_ticket_reply' is type 'write' but authority_required is False.

The error is straightforward: send_customer_email was marked as needing authority, but nothing in the pack said what that authority was. That's not something the script would catch — it only reads replay bundles, and a replay bundle doesn't carry that information. It exists, though: the manifest declares it, back at the Declare layer, as the email.send.customer scope. The script just never looks there. The fix is adding it by hand: the email.send.customer scope from the manifest, with a note that no grant exists yet for this deployment. That note is accurate. It's the real reason the send got blocked instead of just flagged for review.

W019 is worth a one-line note: the validator's docs undersell which action types trigger it — the code also flags plain write actions like draft_ticket_reply, not just external sends. The warning is left standing rather than papered over, because draft_reply genuinely has no authority gate today.

With the missing authority section added, the pack validates clean:

$ agep validate support_agent_evidence_pack.json
Valid   : True
Issues  : 3
  [WARNING] W014 [review_records]: review_records is empty.
  [WARNING] W015 [review_status]: Production deployment has review_status 'draft' or 'in_review'.
  [WARNING] W019 [action_inventory]: Action 'draft_ticket_reply' is type 'write' but authority_required is False.

All three remaining warnings are honest, not bugs to fix. There's no review record because nobody has reviewed this pilot run yet. draft_reply genuinely has no authority gate. A pack with zero warnings at this stage would be hiding something, not proving something.

That gap is also the pack's one open risk. The Risk Register says the same thing in plain terms: draft_reply needs manual review on every single invocation, because it was never added to the control plane's allowed-tools list.

Confirming the Evidence

The last step is a manual approval. Recall that the Review Records section sits empty right now. Once validation passes, a human — typically a CISO or Risk Manager — reviews the pack and signs off, adding a ReviewRecord: their name, their role, a decision, and a timestamp. That's what moves review_status from in_review to approved.

The schema enforces this, not just documents it. A pack can't be marked approved without that record on file, and a critical, unresolved risk in the register blocks approval outright, no matter who signed off.

With the official sign-off, the evidence pack is ready for whatever comes next — an audit, a board review, or wider deployment of the agent it covers.

Proving The Whole Stack Held

In this series, we've covered the four layers of Cognous's Open Control Stack: a manifest that declares what an agent may do, a control plane that enforces it live, a replay bundle that proves a specific run followed the rules, and an evidence pack that turns all of that into something a non-engineer can actually approve or reject. Each layer produces something real the next one consumes — sometimes with a clean mapping, sometimes, like this one, with a script and a person doing the work no importer does yet.

Clone Agent Governance Evidence Pack and try agep against the example packs in the repo, or build one from your own control plane and replay bundle records.

The Replay Bundle That Remembers What Happened

Doug Sillars — Tue, 25 Aug 2026 12:58:42 +0000

It's 2am. A customer complains about something an agent did. The next morning the CEO is already asking the only question that matters: did the agent do something it shouldn't have?

"I'm pretty sure our guardrails are solid" isn't an answer anyone wants to give the boss. It certainly won't fly with a customer, an auditor, or with legal. What they want is proof, and proof means being able to answer specifically: what did the agent try to do, on this exact run, and what happened when it tried?

If that answer lives in the timestamps of a few hundred servers and a half-dozen tool integrations, you don't have proof. You have a scavenger hunt. "Trust us, it's fine" does not survive a scavenger hunt.

Cognous' Open Control Stack exists to close that gap. It follows four steps: Declare → Control → Replay → Evidence.

Declare: Agents need guardrails to tell them what they can and cannot do — that's a manifest, declared up front.
Control: They need something enforcing those guardrails live — that's a control plane, deciding allow/block/escalate on every proposal in real time.

In this post, we'll cover Replay: no longer defining and patrolling the guardrails, but the receipts that the rails held. Taking what the Control Plane recorded during a run and turning it into one portable file — the thing you actually hand someone instead of a text file with pasted logs from a dozen systems.

What The Control Plane Hands Off

Picture a workflow mid-run: actions getting proposed, the gate deciding each one, live.

Now what? The Control Plane made the right calls in the moment — but how do you actually know that, after the fact, without having watched it happen live? That's the question Replay answers: it takes what already happened and turns it into something you can go back and inspect.

The Control Plane keeps a recorder running for the life of a run — that's what's actually holding the frame, the proposals, the decisions, the traces as they happen. At the end of the run, the recorder exports everything it captured as a replay bundle: one file you can store, ship, or hand off for later analysis.

bundle = recorder.generate_replay_bundle()
with open("run_bundle.json", "w") as f:
    f.write(bundle.model_dump_json(indent=2))

run_bundle.json is the full playback of the workflow — every decision, every trace, and the evidence behind the one blocked action, send_email, all in one place. It's exactly what you'd look at to understand an error, or hand over if there's a deeper investigation.

The Replay Tool

Agent Replay Bundle ships its own CLI, arb, for working with files like this one — validating, summarizing, redacting, signing. Start with validation, pointed at the bundle the Control Plane just exported:

$ arb validate run_bundle.json
Error: Replay bundle failed model validation: 1 validation error for AgentReplayBundle
bundle_id
  Field required [type=missing]

It fails immediately. The Control Plane's export calls the field replay_bundle_id; arb's schema — the AgentReplayBundle model that validate checks against — calls it bundle_id.

The Agent Replay Bundle is a public interchange format, meant for any system, and Agent Control Plane's internal run record shapes several of the schema terms differently than the Replay Bundle. Running validation means performing a mapping step between the two first — small, mostly renames:

mapped = {
    "bundle_id": run["replay_bundle_id"],
    "run_id": run["run_id"],
    "generated_at": run["generated_at"],
    "frame": run["frame"],
    "action_proposals": run["actions"],
    "policy_decisions": run["decisions"],
    "policy_traces": run["policy_traces"],
    "authority_records": run["authority_records"],
    "reliance_records": run["reliance_records"],
    "blocked_actions": run["blocked_actions"],
    "final_output": run["final_output"],
}

Run that, and the same run record now validates:

$ arb validate mapped_replay_bundle.json
VALID  bundle_id=ae564206-8daf-4b19-9d60-db54f56387a5  issues=2
  WARNING W005: Bundle has no authority records.
  WARNING W012: signature_metadata is missing.

Valid — with two warnings, not zero. Neither one blocks validity, and both are honest about the bundle's actual state: no authority was granted for this run (which is why two of the three actions didn't just allow), and nothing's been signed yet.

Worth building into your own pipeline: none of this runs automatically. generate_replay_bundle() builds the object; arb validate is a separate command a human has to think to run. Nothing stops a bundle from sitting unvalidated for months until the day someone actually needs it — which is the worst possible time to find out it doesn't pass. Wire the mapping and the validation into the end of every run, not into the moment an auditor asks for one.

What A Reviewer Sees First

arb summarize is the one-screen version — what you'd look at before deciding whether to open the full bundle:

$ arb summarize mapped_replay_bundle.json
bundle_id:          ae564206-8daf-4b19-9d60-db54f56387a5
run_id:             c3f474dd-952b-4f27-ab5c-98de868b3c67
status:             complete
actor:              support-agent-v1
environment:        production
policy_version:     v1.0
action_proposals:   3
policy_decisions:   3
policy_traces:      3
blocked_actions:    1
authority_records:  0
reliance_records:   1
final_output:       present
redacted:           no
signed:             no

Three proposals, three decisions, one block, zero authority records. That last number is the tell — it's the same fact the warning surfaced, now visible at a glance instead of buried in a rule name.

Redacting For Export

mapped_replay_bundle.json — the bundle we just validated — still has raw targets and payloads in it: customer IDs, draft email bodies, potentially PII. Fine for an internal review, not fine to hand to outside counsel.

Redaction isn't content-aware — it doesn't scan text and detect what looks sensitive. It replaces a fixed set of fields wholesale: payload (redacted by default), plus target and final_output if you opt in with flags. Everything structural — IDs, timestamps, decision results, policy names — is preserved no matter what:

$ arb redact mapped_replay_bundle.json --out redacted_bundle.json --targets --final-output
Redacted bundle written to redacted_bundle.json

"redaction_metadata": {
  "redacted": true,
  "redacted_at": "2026-08-18T19:47:34Z",
  "redacted_fields": [
    "action_proposals[0].payload",
    "action_proposals[0].target",
    "action_proposals[1].payload",
    "action_proposals[1].target",
    "action_proposals[2].payload",
    "action_proposals[2].target",
    "final_output"
  ],
  "replacement": "[REDACTED]"
}

A redacted bundle still validates clean — the whole point is that a reviewer can see what was blocked and why, without seeing the customer's actual email address.

Signing For Export Integrity

Once a bundle is ready to leave the building, sign it so anyone downstream can tell if it was altered in transit — the redacted version, since that's the one actually going out the door:

$ arb sign redacted_bundle.json --secret "demo-secret" --key-id "cognous-demo-key-01" --out signed_bundle.json
Signed bundle written to signed_bundle.json

$ arb verify signed_bundle.json --secret "demo-secret"
Signature VALID

$ arb verify signed_bundle.json --secret "wrong-secret"
Signature INVALID

That's HMAC-SHA256 over a canonical serialization of the bundle. Worth being precise about what it proves: it tells you the bundle hasn't changed since it was signed, given the secret. It does not prove who signed it — there's no identity binding, no PKI, no protection if the secret itself leaks. For production use, that's a job for a real key management system. For a shared-secret export check between two parties who already trust each other, HMAC does what it says.

Reading The Verified File

The auditor gets signed_bundle.json, runs verify, and knows immediately whether it matches what left your hands. Signing isn't encryption, though — verify is the only command that speaks the signed wrapper (signed_bundle_id, replay_bundle, signature_metadata). To actually read the run, pull the inner bundle back out first:

import json
with open("signed_bundle.json") as f:
    signed = json.load(f)
with open("extracted.json", "w") as f:
    json.dump(signed["replay_bundle"], f, indent=2)

Now extracted.json is a plain AgentReplayBundle again — summarize, validate, everything else works on it. Verify first, extract second: confirm nothing's been tampered with before you trust a single field inside it.

The signature also doesn't prove anything about the gap between when the run happened and when you got around to signing it. signed_at is just whatever time you ran the sign command — there's no check against generated_at. A bundle that sits unsigned for three days is three days an editor could have touched the file with no trace of it; sign it on day three, and verify comes back clean regardless. The signature only covers the window it's actually been alive for. So the real answer isn't "sign it before you send it" — it's validate, redact, and sign as part of the run itself, the moment the bundle is generated, so there's no gap for anything to slip into.

Proving The Guardrails Held

Back to that morning-after meeting: the complaint, and the question from the boss. What was once a stressful scavenger hunt across servers matching timestamps and logfiles is now just opening one file.

Agentic runs get their guardrails from two layers: a manifest declaring what's allowed, and a control plane enforcing that live. Replay is the layer that lets your team prove those guardrails actually held — turning what the control plane recorded into one validated, signed file anyone can check independently, long after the run itself is over.

Clone Agent Replay Bundle and try arb against the example bundles in the repo, or your own control plane exports.

Learn more about Agent Replay Bundle at cogno.us.

Cognous Control Plane: The Layer That Tells Your Agent No

Doug Sillars — Tue, 18 Aug 2026 19:39:36 +0000

It's all too common — a viral tweet describing how AI agents with poorly defined guardrails deleted production, wiped a critical database, or caused some other disastrous occurrence. Agents need guardrails: this includes permissions to do the job, but also blockers that limit the reach of the agent, thereby preventing accidental disasters that fuel viral tweets.

Cognous' Open Control Stack is a framework for wrapping AI agents in guardrails, providing access where needed, but blocking the boundaries that cannot be crossed.

In our introductory blog post, our agent drafted a cringey follow-up email and "helpfully" sent it to a company's entire list of top customers. Nobody approved that send, and the human marketing team had to drop everything and work to mitigate the damage.

Open Control Stack is structured as four layers: Declare → Control → Replay → Evidence.

We built the first layer in The Manifest That Keeps Your AI Agent Honest, the Agent Action Manifest. That's the Declare layer — a JSON file that works as the allow/block list for our agent:

Pull the top customer: allowed
Draft a reply: draft-first
Pull contract details: blocked
Send the email: needs approval

But that's all a manifest is — a list. Nothing reads it and nothing enforces it. We need something that takes that list and actually applies it to the agent while it's running.

That's the Agent Control Plane — the Control layer. This post is about what it actually does when the agent tries something.

What Sits Beside The Agent

The Control Plane doesn't run the agent, and it isn't a framework. It sits next to whatever is already generating the agent's behavior — OpenClaw, Hermes, a custom loop, whatever — and turns every proposed action into a recorded decision before that action is treated as real.

The agent's job is small in this picture — it proposes. The Control Plane does the rest: it decides, and it documents. Propose is the only row the agent touches. Decide is the gate ruling allow, block, or escalate. Document is what gets written down once the decision's made. The agent doesn't get a say in the outcome — it just gets to ask.

The pattern, end to end:

Propose

The agent wants to do something. The Control Plane frames it, and a proposal gets built.

Agent Task: The agent kicks off the run. In our use case: "review a customer account and follow up."
Frame: The Control Plane frames the execution context for the run — the task, actor, environment, allowed tools, blocked tools, policy version. This gets locked in at the start and becomes immutable for the life of the run.
Action Proposal: The agent's ask — this tool, this action type, this target, this payload, this reason. Recorded before anything executes — that's the point of the proposal steps. If the decision gate blocks it next, nothing downstream ever sees a real tool call, but the attempt is still on the record.

Decide

The Control Plane decides whether that proposal is actually okay. This is where the manifest's declarations actually take effect: what it declared is what ends up in the frame's allowed and blocked lists, and the frame is what the gate checks against. Two of the six rules below check something separate from the frame: authority — whether this actor currently has permission to write, or to send things externally. Authority is granted per run, on top of the frame, and it's what rules 3 and 5 are checking for.

Policy Gate: Decides what happens to the proposal. Six rules, checked top to bottom, first match wins:
1. tool is explicitly blocked → block
2. tool is not in the allowed list → escalate
3. action type is external_send → allow if authority for it exists, otherwise block
4. action type is read (and passed rules 1–2) → allow
5. action type is write → allow if authority for it exists, otherwise escalate
6. anything else → escalate
Policy Decision + Evaluation Trace: The gate's verdict, plus the rule-by-rule path it took to reach it.
Allow / Block / Escalate: The three possible verdicts. Only allow lets the proposal go on to actually touch a tool.

Document

Once a decision is made, the Control Plane writes down what happened, what the decision was, and how it was made.

Reliance Record: Once an action is allowed and actually executed, a record of what the agent depended on to produce its output — a tool, a database, a file, an API, user input. Blocked and escalated actions never execute, so there's nothing to record reliance on — dependence doesn't exist until an action actually runs.
Run Record: Every proposal, decision, trace, block, and reliance from this run, assembled into one object.
Replay Bundle: The run record packaged up for later, portable inspection.

Running The Scenario For Real

Let's take three of those actions from the manifest and actually run them through the gate: pull_top_customers, draft_reply, and send_email. All three are on this run's allowed-tools list. pull_contract_details is also declared in the frame's blocked-tools list, matching the manifest — but the task is "review a customer account and follow up," which never calls for contract details, so the agent never proposes it. It's declared, not exercised: nothing for the gate to evaluate here.

Starting the run:

This is the Control Plane spinning up: RunRecorder() creates the recorder instance, and start_run() builds the frame — the same frame from the Propose section above, locking in the task, actor, environment, allowed tools, blocked tools, and policy version for everything that follows.

from agent_control_plane import RunRecorder

recorder = RunRecorder()
run_id = recorder.start_run(
    task="Review a customer account and follow up.",
    actor="support-agent-v1",
    environment="production",
    allowed_tools=["pull_top_customers", "draft_reply", "send_email"],
    blocked_tools=["pull_contract_details"],
    policy_version="v1.0",
)

Proposing and Deciding One Action:

Here's the pattern for a single action, pull_top_customers. The same process repeats for the other two actions — omitted here for brevity:

Propose the action with the propose_action method. This is stored in the variable a1.
Evaluate the proposal with evaluate_action. This returns two values: the decision, and, if the action is blocked, a blocked-action record.

a1 = recorder.propose_action(
    tool_name="pull_top_customers", action_type="read",
    target="crm:top_customers", payload={"limit": 25},
    reason="Identify the customer's account tier before drafting a reply.",
)
d1, _ = recorder.evaluate_action(a1)

pull_top_customers is not blocked, so we receive only the decision, and the unused second value gets thrown away with _. d1 holds the decision: the verdict (allow, block, or escalate), which rule produced it, and why. Here's what the gate decides across all three actions:

Action	Type	Rule	Result
`pull_top_customers`	read	4	allow
`draft_reply`	write	5	escalate
`send_email`	external_send	3	block

Missing authority is what stops two of the three from just allowing. draft_reply escalates and send_email blocks — same problem, nobody granted permission for this run — but different severity, because the two actions aren't equally reversible. A blocked send is stopped outright: once an email's out, it's out. An escalated write just doesn't execute yet. Nothing about the original decision ever changes — but if someone grants write authority before this run ends, a fresh proposal for the same action would clear rule 5 as allow.

The evaluation trace is where the reasoning survives, not just the verdict. Here's the trace for the send_email decision:

{
  "rules_evaluated": [
    {"rule_name": "blocked_tool_policy", "matched": false, "result": "none"},
    {"rule_name": "unknown_tool_policy", "matched": false, "result": "none"},
    {
      "rule_name": "external_send_authority_policy",
      "matched": true,
      "result": "block",
      "reason": "Action type 'external_send' requires an authority record with scope 'external_send', which was not found."
    }
  ],
  "final_result": "block"
}

A reviewer looking at this trace can tell exactly which check failed and why — not just that the action was blocked. That matters because send_email is on the allowed-tools list; nothing about the tool itself was forbidden. The trace is what proves the block was about missing authority, not a banned tool.

Worth noting: draft_reply escalating isn't a lesser outcome than blocking. It's easy to read "escalate" as a downgrade of "block," but it's really the gate saying it doesn't have enough information to decide safely, so a human should. Grant write authority to the run, and the same action clears rule 5 as allow instead — the rule didn't change; what the actor is authorized to do did.

Documenting The Proposal And Decision

After a decision has been made, the Cognous Control Plane documents what occurred during the process. All of the information is collated and stored in one place — not grabbed from logs scattered across different tools and pieced together by timestamp.

The first step is the Reliance Record. Recall that this only documents actions that actually ran — in this case, pull_top_customers:

recorder.record_reliance(
    source_name="pull_top_customers",
    source_type="tool",
    scope="customer tier and account fields",
    referenced_action_id=a1.action_id,
)

This doesn't get written to a file by itself — it's held in memory as part of the run, alongside every proposal, decision, and trace, until the whole thing gets closed out and exported:

recorder.complete_run("Draft prepared and escalated for review. Email send was blocked, not attempted.")
recorder.export_json("run_record.json")
bundle = recorder.generate_replay_bundle()

export_json() writes everything — proposals, decisions, traces, the one blocked-action record, that one reliance record — to a RunRecord JSON file. Retrieving the reliance record later is just reading that file and looking at its reliance_records array.

Reading The Results

Writing the record is only half the story — you also need a way to check it. That's acp, the Control Plane's own command-line tool, shipped in the same repo: github.com/cogno-us/cognous-agent-control-plane.

Running it against the run record we just exported:

acp validate-run run_record.json

RunRecord 79ad6860-03de-4136-9828-d4e40fbd05f4: valid (0 errors, 0 warnings)

Validation here isn't checking whether the run succeeded — it's checking that the record is internally consistent: every decision references an action that actually exists, every blocked action corresponds to a decision that actually says block. It's a check on the evidence, not on the outcome.

The replay bundle gets the same treatment:

acp validate-replay replay_bundle.json

ReplayBundle 535c72f8-efee-4144-9b78-7f7fff73133d: valid (0 errors, 0 warnings)

What This Deliberately Doesn't Do

The Control Plane only enforces the rules and authority it's actually been given — it doesn't infer intent, and it doesn't fill gaps with a best guess. A permission that was never granted shows up as an escalation or a block, not a pass. It's not a compliance system, and it's not a guarantee that the agent's output was correct. It's the layer that turns "the agent tried to do X" into a recorded, deterministic, inspectable decision, every time.

It also doesn't decide what happens after a decision is made. The gate doesn't open a ticket, notify anyone, or track whether an escalation ever gets reviewed. Whether a human looks at the draft, whether the agent retries, whether authority eventually gets granted — that's handled by whatever's orchestrating the agent, not by the Control Plane.

Next Up

We now have a run: three proposed actions, one blocked, one escalated, one allowed, each with a trace and a fingerprint, packaged into a replay bundle that validates clean. That bundle is a record of one run.

The Cognous Open Control Stack has four steps: Declare → Control → Replay → Evidence.

In The Manifest That Keeps Your AI Agent Honest, we declared in the manifest what was allowed and what was not. In this post, we walked through the Control Plane — it runs next to your agentic framework to allow, escalate, or block actions. The Control Plane records the results from every action taken.

Next, we look at the last two steps: Replay and Evidence. These are the tools that will save you when auditors are knocking at the door, or you need to prove you've redacted customer data, or just prove it hasn't been tampered with since it was generated. That's the Agent Replay Bundle, and we'll cover it in our next post.

In the meantime, the Control Plane repo, examples, schemas, and test suite are live now: github.com/cogno-us/cognous-agent-control-plane. Clone it, run examples/simple_agent_run.py, and watch the gate make the call.

The Manifest That Keeps Your AI Agent Honest

Doug Sillars — Wed, 12 Aug 2026 14:50:04 +0000

In our first post, Cognous Keeps Your AI in Check, we introduced the scenario: a customer-service agent that pulls CRM records, drafts replies, and — if nobody's watching — occasionally sends 1,000 tone-deaf emails to your best clients. We said the fix isn't reading what the agent wrote after the fact. It's deciding, in advance, what the agent is even allowed to attempt.

That decision has to live somewhere. It can't live in a Slack thread, a comment in the agent's system prompt, or "Dave from platform remembers we blocked that." It needs to be a file — versioned, reviewable, and boring enough that a security team can actually sign off on it.

That's what Cognous's Open Control Stack is for. It's four small, open-source projects — Declare, Control, Replay, Evidence — that sit beside your agents and cover authorize, enforce, and prove. Declare comes first, and its artifact is the Agent Action Manifest. This post describes the Action Manifest tooling in the Open Control Stack, available on GitHub.

The Four Actions, Written Down

If you remember in post one, our AI agent was pulling customer data, drafted a cringey email, and sent it to our top 1,000 customers. The Action Manifest is used to outline the agent's permissions as declarations, to prevent this from ever happening again:

{
  "manifest_version": "0.1",
  "manifest_id": "customer-service-agent-manifest",
  "agent_name": "customer-service-agent",
  "owner": "support-platform-team",
  "environment": "production",
  "default_action": "block",
  "tools": [
    {
      "tool_name": "crm",
      "description": "Customer relationship management system",
      "allowed": true,
      "external_system": "crm.internal",
      "data_classification": "customer_pii"
    },
    {
      "tool_name": "email",
      "description": "Outbound customer email system",
      "allowed": true,
      "external_system": "smtp.internal",
      "data_classification": "customer_pii"
    }
  ],
  "actions": [
    {
      "action_name": "pull_top_customers",
      "tool_name": "crm",
      "action_type": "read",
      "description": "Retrieve top customer records from the CRM.",
      "default_action": "allow",
      "reliance_requirement": {
        "required": true,
        "allowed_source_types": ["tool", "database"]
      }
    },
    {
      "action_name": "draft_reply",
      "tool_name": "email",
      "action_type": "write",
      "description": "Draft a reply email for a customer inquiry.",
      "default_action": "allow",
      "authority_required": [
        {
          "scope": "email.draft.customer",
          "description": "Permission to draft outbound customer emails",
          "required": true,
          "source": "support-platform-team"
        }
      ],
      "review_requirement": {
        "mode": "draft_first"
      }
    },
    {
      "action_name": "pull_contract_details",
      "tool_name": "crm",
      "action_type": "read",
      "description": "Retrieve contract terms from the CRM.",
      "default_action": "block",
      "tags": ["contract", "restricted"]
    },
    {
      "action_name": "send_email",
      "tool_name": "email",
      "action_type": "external_send",
      "description": "Send an approved email to the customer.",
      "default_action": "escalate",
      "authority_required": [
        {
          "scope": "email.send.customer",
          "description": "Outbound customer send scope",
          "required": true,
          "source": "email.send.customer"
        }
      ],
      "review_requirement": {
        "mode": "approval_required",
        "reviewer_role": "support-lead"
      },
      "reliance_requirement": {
        "required": true,
        "allowed_source_types": ["tool", "user_input"]
      },
      "payload_policy": {
        "sensitive_fields": ["customer_email", "customer_name"],
        "forbidden_fields": ["contract_terms"]
      },
      "redaction_hints": [
        {
          "field_path": "customer_email",
          "reason": "PII in exported records"
        }
      ]
    }
  ]
}

(Field names here follow the public schema — check the repo for the exact current shape if you're implementing against it.)

Rules are only as good as what is defined. At the very top of the manifest, we explicitly cover anything not mentioned: "default_action": "block". So when a new tool is onboarded (for example send_tweet), you don't need to go back and update every manifest — it's blocked until you expressly give the agent permission to use it.

For those actions that are defined:

pull_top_customers and draft_reply are both allow, but not the same kind of allow: pulling the top customers is allowed, full stop, but the draft reply has review_requirement.mode: draft_first — the agent can write it, but it stops there.
send_email isn't just "escalate" as a vague gesture — it names the exact authority scope required (email.send.customer), the exact reviewer role (support-lead), and which payload fields have to be redacted before this record goes anywhere near an audit export.
pull_contract_details is the odd one out — it doesn't strictly need to be there. default_action: block already covers anything undeclared, so an unlisted "read contract details" action would be blocked anyway. It's included for visibility: a blocked action that shows up in the manifest, tagged restricted, is a documented decision someone can point to. A blocked action that's simply never mentioned looks identical to an oversight.

It's important to note that the manifest is not enforcement. It doesn't stop the agent from doing anything — a runtime layer has to actually read it and act on it (that'll be described in the next post). What the manifest gives you is a single artifact that says, in one place, what should happen — before the agent has run even once.

The Nine Action Types

In our simple manifest above, we show three possible action types. The manifest supports nine. These classifications are what give a governance review the ability to distinguish a harmless read from something that should never happen without a human:

Action	Meaning
`read`	retrieves or inspects information
`write`	creates or modifies information, not necessarily sent externally
`external_send`	sends information outside the system boundary
`delete`	removes information or records
`export`	packages or transfers information downstream
`purchase`	initiates or prepares a purchase
`approve`	approves or authorizes a workflow step
`escalate`	routes to a human or supervisory process
`other`	anything not otherwise classified

A tool list alone can't tell you this. Knowing the agent "has access to email" tells you nothing about whether it can draft, send, or both, and what's supposed to happen before each.

The Four Review Modes

Action type says what kind of thing an action is. Review mode says who has to look at it, and when — and it's doing just as much governance work in our example as action type is. There are four:

Mode	Meaning
`none`	no specific review posture declared
`human_review`	a human is expected to look at it before execution or completion — no specific person named
`approval_required`	approval is required from a named reviewer role
`draft_first`	the agent produces a draft; it does not finish the action itself

The difference between human_review and approval_required is easy to miss but it matters: human_review says "someone should look at this" without saying who. approval_required is stricter — it names a reviewer role, and the validator enforces that: an approval_required action with no reviewer_role fails validation outright.

That's why send_email in our manifest uses approval_required with reviewer_role: "support-lead", not human_review. Sending customer email isn't "someone should probably glance at this" — it's "this specific role signs off, or it doesn't go out."

Worth being precise about what "support-lead" actually is here: a role string, nothing more. The manifest doesn't know who holds that role, doesn't know they're on Slack, and doesn't page anyone. Your enterprise-side plumbing converts "support lead" to Janice, and builds an alerting system in Slack to let her know that her approval is required. The public stack's job ends at "this role must approve, and there's a record of whether it did."

Reliance: A Third Thing Being Tracked

Authority asks whether an action was allowed. Review asks whether a person signed off. Reliance asks something different: what did this action actually depend on to produce its result?

A reliance_requirement declares that an action should leave a record of its source — a tool, a database, a file, an API, or user input. pull_top_customers declares reliance on tool and database, because a governance review should be able to see it actually came from the CRM and not somewhere the agent invented. send_email declares reliance on tool and user_input, tying the send back to the approved draft it was built from.

draft_reply and pull_contract_details don't declare a reliance requirement — which is why the summarize output above shows Actions requiring reliance: 2, not 4.

Validation Isn't Optional, and It Isn't Decoration

A manifest can be valid JSON, but be semantic nonsense. A delete action on the production database marked allow isn't caught by JSON Schema. It's caught by the validator, which checks things like:

does every action reference a tool that's actually declared and allowed?
is a privileged action type (external_send, write, delete, purchase, approve) missing an authority requirement it should have?
is an action marked approval_required but missing a reviewer role?
are there duplicate action or tool names?
are sensitive fields declared with no redaction hints to match?

Run it from the CLI — aam, short for Agent Action Manifest, is the command-line tool that ships with the manifest repo. First we run the validate command to ensure that the JSON is valid:

$ aam validate customer-service-agent-manifest.json
Validation result: VALID
Manifest ID:       customer-service-agent-manifest

Warnings (3):
  [W004] [actions[1](draft_reply)] Action 'draft_reply' has effective default_action 'allow' for action_type 'write'.
  [W006] [actions[1](draft_reply).payload_policy] Action 'draft_reply' has action_type 'write' but no payload_policy is declared.
  [W005] [actions[2](pull_contract_details).reliance_requirement] Action 'pull_contract_details' has action_type 'read' but no reliance_requirement is declared.

VALID, but not silent — the validator still flags things worth a second look, even in a manifest that passes. Fair warnings, too: we didn't bother declaring a payload policy for a draft-only action, and a blocked action doesn't need reliance evidence it'll never produce.

Next we can summarize the manifest with the summarize command:

$ aam summarize customer-service-agent-manifest.json
Manifest ID:          customer-service-agent-manifest
Agent name:           customer-service-agent
Environment:          production
Tools:                2
Actions:              4

Actions by type:
  external_send: 1
  read: 2
  write: 1

Actions by default posture:
  allow: 2
  block: 1
  escalate: 1

Actions requiring authority: 2
Actions requiring review:    2
Actions requiring reliance:  2

That "2" next to authority is easy to misread as just the send — it's actually draft_reply and send_email both. Drafting on a customer's behalf declares its own authority requirement (email.draft.customer), separate from the one on the final send (email.send.customer). Declaring it isn't the same as enforcing it — nothing here checks whether that authority actually exists. That check is the Control Plane's job, next post.

aam verifies that your manifest is ready, and gives the team a concise summary of what's allowed.

What This Deliberately Doesn't Do

The manifest is the Declare layer, and it stays in its lane:

it doesn't execute the agent
it doesn't enforce anything at runtime
it doesn't grant authority — email.send.customer being declared as required doesn't mean it exists
it doesn't replace your application's own authorization system

Declaring that sending email should require approval is not the same thing as stopping an unapproved send. That gap — between what's declared and what actually happens when the agent tries to act — is exactly what the next layer closes.

Next up

The manifest says what the agent may propose. It says nothing about what the agent actually does on a Tuesday afternoon when it's mid-run and reaching for the send button. That's the Agent Control Plane — it reads this exact manifest, sits beside the agent at runtime, and turns every proposed action into a recorded decision: allow, block, or escalate, deterministically, every time. That's next.

The manifest repo, examples, schemas, and test suite are live now: github.com/cogno-us/cognous-open-control-stack. Clone it, validate the customer-service example, break it on purpose and watch the validator catch you.

Cognous Keeps Your AI in Check

Doug Sillars — Thu, 06 Aug 2026 22:49:57 +0000

You've probably gotten an email that was supposed to feel personal, but obviously wasn't. Something about it just felt off. Automated. And somewhere in the back of your mind, you wondered: who approved this?

Honestly? The answer is probably no one.

Organizations are leaning harder on AI agents to actually do work. Agents are making database queries, writing letters, accessing email lists, but where does the access end? Without solid guardrails, tone-deaf emails are no longer an embarrassing edge case, they're a real risk. Nobody wants to be the team that let an agent send 1,000 tone-deaf emails to its best clients before anyone noticed.

This series is about the tooling that keeps that from happening, and about actually building it, not just reading about it.

What actually needs to happen

The email itself might be perfectly fine. Correct grammar, on-brand, even genuinely well-written. But the problem is that the email didn't need to be sent. The agent just did it anyway, without any guidance from the team.

So the fix isn't a better read of what the agent said. It's governance around what the agent is allowed to do, three capabilities, specifically:

Authorize: Decide in advance what an agent is allowed to do.
Enforce: Apply that decision the moment the agent tries to act, before the action executes, not after.
Prove: Turn every run into a record you can produce on demand, not one you have to reconstruct from memory.

Application logs are a day late and a dollar short: the emails are already sent. Looking at logs is reactive. You need a stack that sits in front of your agents and proactively approves or denies actions.

Introducing Cognous

Cognous builds infrastructure for governing AI agents. Its first release, the Open Control Stack, is fully open source: four small projects that sit beside your AI agents to keep them in check, together covering Authorize, Enforce, and Prove. The four projects are Declare, Control, Replay, Evidence.

Declare, the Agent Action Manifest. Written once, before the agent ever runs. The manifest provides or denies access. The agent is attempting to complete a task, but the manifest keeps it in check:

Pull top customers: default_action: allow. A CRM read, flagged to leave a reliance record of what data it used.
Draft a reply: default_action: allow, review_mode: draft_first. The agent prepares content; it doesn't finish the job.
Pull contract details: default_action: block. Off-limits for this agent, full stop.
Send the email: default_action: escalate. Requires a named authority scope and human sign-off before it's eligible to execute.

The point is that the manifest lets you say reading customer data is fine, reading contracts is not, drafting is fine, but sending the email needs a person: four different rules for four different actions, declared before any of them happen, instead of one blanket policy applied to "the agent" as if every action carried the same risk.

Control, the Agent Control Plane. Runs live, alongside the agent. Every time the agent tries to do something, the Control Plane checks it against what was declared, records the attempt, and either allows it, blocks it, or escalates it for approval, deterministically, every time. Every one of those outcomes gets written down, not just the bad ones.

Replay, the Agent Replay Bundle. Not a process, an export. When the run ends, everything the Control Plane recorded (what was proposed, what was decided, what was escalated, what it relied on) gets packaged into a single portable file. No more digging through the logs of a dozen systems to reconstruct one run. Every read, every draft, every escalation, everything that was allowed or held back: it's all in one place.

Evidence, the Agent Governance Evidence Pack. Assembled later, across many runs. This is where you get to show your work: not "trust us, we have controls," but "here's proof: 40 sends needed sign-off this month, and here's what happened to each one."

Walking the scenario through the stack

The manifest declares that sending an email requires authority and escalates rather than executing automatically, decided before the agent ever runs a single task.
Mid-run, the Control Plane checks each of the agent's proposals against the manifest as they happen: pulling top customers and drafting a reply go through, pulling contract details is blocked outright, and the proposed send doesn't execute, it escalates and gets held for a person to review. Every one of those outcomes gets recorded as it happens, not just the one that got stopped.
When the run ends, every step (the reads, the draft, the block, the held send, and the outcome) gets packaged into a replay bundle.
Over time, those bundles roll up into an evidence pack: not "we think our agent behaves," but "here's the record of every send this month that needed sign-off, and what happened to it."

A thousand cringey emails to customers drops to zero. There are no questions about why, no furious Slack thread trying to piece it together. And if anyone does wonder, the stack's replay bundle provides the paper trail.

See it yourself

The Open Control Stack, Cognous's first release, is fully open source, and everything above (the manifest fields, the allow/block/escalate decisions, all of it) comes straight from the actual code, not a slide deck. If you want to see AI agent governance in action instead of taking our word for it, the manifest, control plane, replay bundle, and evidence pack repos are all live on GitHub. Clone one, run the examples, watch a proposed action get authorized, blocked, or escalated in real time. You can learn more about Cognous at cogno.us.

This is the first post in a series on AI agent governance. Next up, we build the manifest from this exact scenario: the real schema, the real validation rules, and what runtime authorization for AI agents actually looks like as working code. From there we'll get into the AI agent control plane that enforces it and how a single run turns into replayable AI agent evidence you can hand to an auditor. More posts coming soon.

When should you use canary deployments?

Doug Sillars — Mon, 20 Apr 2026 18:42:22 +0000

One solution for a tricky or high-risk deployment is to leverage a canary deployment. In a canary deployment, new features are rolled out to a small subset of your customer base with no fanfare. If things go well: hooray! Canary deployments are ideal for times when a deployment might go sideways or the feature might not work as expected. By utilizing a canary release, you have minimized the risk, as impacting a small fraction of your users is a lot better than affecting all of them.

This post will discuss when development teams might choose to use canary deployments in their applications, what is a canary deployment, common pitfalls to avoid, and how modern tooling can help ease the rollout of canary deployments.

What is a canary deployment?

Where does the term canary deployment come from? You may have heard of the term "canary in the coal mine." Early coal miners would keep caged songbirds in the mine with them, because these small birds are highly susceptible to toxic gases. If the canaries suddenly died, the miners knew that there was a dangerous gas release, and they could evacuate the mine before they succumbed to the deadly gas.

A canary deployment can be performed when there is concern about the risks involved. Perhaps you are rolling out new cloud infrastructure that has not yet been tested with the production environment, or maybe there are performance concerns—the updates might cause the website's performance to drop to a terrible crawl. Perhaps the change is small, but it could not be tested for security vulnerabilities. A failed rollout to your entire customer base could end in disaster in all of these cases.

Is your team already using canary deployments?

Many teams practice canary-style releases without explicitly calling them that. Common variations include:

Blue-green deployments: Two production environments run in parallel. The blue environment serves the stable version, while green runs the new version. Traffic is gradually shifted from blue to green, limiting risk and enabling fast rollback.
Experimental deployments: New behavior is exposed to a defined group of users, often with explicit awareness or opt-in. These are typically broader than canary deployments and focus on measuring user impact rather than system health alone.
Gradual rollouts (progressive delivery): After validating a release with a small subset of users, traffic is incrementally increased (for example, 5% → 25% → 100%) while monitoring key metrics like error rates and latency.
Shadow deployments: The new version runs in parallel and receives a copy of real production traffic, but responses are not served to users. This is useful for validation and performance testing without customer impact.

Planning a canary deployment

Imagine this situation: You are about to launch a new feature that is considered high-risk for some reason. The team has opted for a canary deployment, releasing it to a limited group of users first. Achieving a successful deployment in this scenario demands meticulous planning.

Deployment audience: Who gets the update? This requires a basic knowledge of your audience and understanding of who would benefit the most from the change. If your new version is an overhaul of the mobile dashboard, and the canary release is delivered to desktop users, how can you determine if the deployment was successful? It is certainly useful to have some desktop users in the cohort (in case something breaks on desktop), but in order to best understand the results of the deployment, you want to ensure that users interact with the features of the new version.
Duration: How long will the test run? An hour? A week? Depending on the type of deployment, you may know the success/failure quickly. (Did the database correctly sync with the servers?). However, sometimes it may take days to determine success. (Is website performance faster during busy periods?).
Metrics: What will the team track during the deployment? What metrics will be considered a success? What metrics would be branded a partial success? When does the team admit failure and roll back the deployment to fix bugs?
Feature flags: For some canary deployments, using feature flags can be helpful. Feature flags are tools in your code that manage the deployment of application features to specific audiences. If your team is planning to use canary deployment extensively, it may be useful to integrate feature flags into your codebase.

Deployment risks

With any release, there is a risk of failure. Using a canary deployment mitigates the risk by reducing the blast radius of a failure to a small subset of users—however, even canary releases have risks. Among these are:

Service outages: If the deployment goes bad, the application may slow down or even cease to work for the selected users.
Security breaches: The new release may introduce a security vulnerability that was not discovered in testing. Rollback mechanism: Many canary releases do not have a formalized rollback plan, should things go wrong.
Data loss: If the canary rollout is using a new database or a new interface to the database, any issue with the rollout may introduce data inconsistencies or data loss.

For any release, it is important to consider the risks and how they will be mitigated. While the canary release lowers the risks, having plans for rollback, handling data loss etc. should be made prior to the release.

Key metrics during canary deployments

During any release, it is critical to monitor your systems in case of any trouble during the deployment. Canary releases are no different, although the experimental nature of the release means there is a higher likelihood of issues arising:

Error rates: Are your systems throwing more errors than typically seen in production?
System metrics: Keep a close eye on memory, CPU, database, and API queries, and other resource metrics. We've all been a part of a release where the new code introduced a memory leak, or suddenly API usage was pegging 400% of normal. By monitoring these in the canary release, the issues can be quickly remediated for the canary users before releasing to the entire base.
Traffic distribution: Is the canary release being properly served to the selected users? Are users accidentally crossing between the two environments? Handling these issues early ensures that the new build is completely isolated from the old environment.
Security metrics: Is the login behavior different in the canary build? Are users interacting with this build differently than the previous version? Could this imply a potential threat?

Monitoring these metrics during the deployment and after the release can help your team better understand the wins (and potential improvements to your app) that occur during the canary release. In addition to traditional monitoring, there are now AI tools that can be used to compare your two environments, quickly parsing the data and determining differences from the old version and the new software version that might be difficult to see in traditional logging.

Canary deployments: What's next?

When dealing with a high-risk release, a canary deployment to a small subset of users is a great way to mitigate exposure in production. To be successful in your canary release, be sure to discuss how you will choose the cohort of users, monitor the release, and plan a rollback strategy prior to the release. Then, during the deployment process, your team can track the essential metrics to understand if the canary deployment was a success. Should the deployment go well, the team can plan to add users to the new version of the software, slowly deprecating the old version.

Developers who use canary releases for risky deployments are able to test problematic code or infrastructure to a small user base, pushing the product ahead while mitigating the risk of an outage.

Database branching in Django apps using GitHub actions

Doug Sillars — Mon, 02 Dec 2024 18:36:27 +0000

Creating online previews of your applications is a great way to test that all the required functionality is present. When building and testing a dev build of your application from a pull request (PR), the last thing you want is for your tests to affect your production database. Using a test branch of the production database ensures that the production database remains untouched, ensuring no accidental deletion of data or adding test data into the production database.

In this post, we’ll create a set of GitHub Actions to automate the testing process of a pull request. Our GitHub Action will run when the pull request is created, generate a test branch of the production database, and deploy the code to Digital Ocean. Once the PR is merged, a second GitHub Action will rebuild the Digital Ocean app with production code (and database), and the test database branch will be deleted.

By finishing this article, you’ll be able to automate the creation of app previews using NeonDB branches, Digital Ocean, and Django. Let’s jump right in!

Setup

Code repo

To begin, we’ll use the Django Neon q uickstart repo on GitHub. Create a fork and clone your fork locally. Follow the setup instructions in the README, and run the application locally to ensure it is up and running. (You’ll need a NeonDB account to add the environmental variables that connect to NeonDB and power the app.)

Digital Ocean

Digital Ocean has a simplified process for launching applications on its platform. You can be up and running in minutes with just a few configuration steps.

You’ll need an account at Digital Ocean to deploy your application. From the Digital Ocean dashboard, select “Create” → App Platform. Connect to GitHub, and choose the django-neon-quickstart, branch main. Click next. I ran this on the $5/month instance. In the “build phase,” click edit and add the three commands below:

pip install -r requirements.txt
python manage.py makemigrations
python manage.py migrate

These are the steps you ran to get the repo running locally—we’re just repeating it on Digital Ocean.

Update the “Run” command to use gunicorn:

gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000

Finally, update the HTTP Port to 8000 to match the port in the repository.

In step two of the setup process, update the environmental variables. You can do a bulk upload and copy and paste in your .env from the local repo as shown below:

Click through the rest of the commands, and on completion, the app will deploy and be available for use on the internet.

Note that in this repository the deployed code is the main branch. We would like to display the PR preview from the dev branch. (Change the names main and dev to whatever branches you wish to preview.) Let’s continue our setup.

GitHub Secrets

We need to add three GitHub secrets to the repository. Add secrets by clicking “Settings” on the top menu. Then: “Secrets and Variables” → Actions, and add repository secrets.

Here are the three secrets to be added:

DO_KEY: A key from Digital Ocean with scopes to create, read, update, and delete apps. You can create this from the Digital Ocean dashboard under API.
NEON_API_KEY: Create your Neon API key at the NEON dashboard. Click your Avatar in the upper right corner, select Account Settings, and then choose API keys to create an API key.
NEON_PW: This is the PGPASSWORD from the .env file.

Digital Ocean app spec files

The App Spec file defines how the build process will be run at Digital Ocean. You can find your App Spec file in your App’s dashboard under “Settings.” (It will be autogenerated when you create your project.)

In your GitHub Repository, create a .do directory, and make two copies of the App Spec file: app.yaml and default.yaml. These will be used by our GitHub Actions to edit the Digital Ocean Application.

app.yaml is what will be provisioned on a pull request being opened:

    alerts:
    - rule: DEPLOYMENT_FAILED
    - rule: DOMAIN_FAILED
    features:
    - buildpack-stack=ubuntu-22
    ingress:
      rules:
      - component:
          name: django-neon-quickstart
        match:
          path:
            prefix: /
    name: **seal-app-dev**
    region: nyc
    services:
    - build_command: |-
        pip install -r requirements.txt
        python manage.py makemigrations
        python manage.py migrate
      environment_slug: python
      envs:
      - key: PGHOST
        scope: RUN_AND_BUILD_TIME
        value: **new_host**
      - key: PGDATABASE
        scope: RUN_AND_BUILD_TIME
        value: neondb
      - key: PGUSER
        scope: RUN_AND_BUILD_TIME
        value: neondb_owner
      - key: PGPASSWORD
        scope: RUN_AND_BUILD_TIME
        value: **new_password**
      github:
        branch: **dev**
        deploy_on_push: true
        repo: dougsillars/django-neon-quickstart
      http_port: 8000
      instance_count: 1
      instance_size_slug: apps-s-1vcpu-1gb-fixed
      name: django-neon-quickstart
      run_command: gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000
      source_dir: /

There are four changes made in this file from the original at Digital Ocean:

name: add “-dev” to the end of the app name.
PGHOST: Value should be new_host
PGPASSOWRD:value should be new_password
github:branch dev replaces main.

When our GitHub Action runs:

The name of the application will change in the Digital Ocean dashboard, allowing the dev to see that the current state is a dev state.
new_host and new_password will be programmatically updated with the values from our newly created NeonDB branch.
We want to deploy the dev branch of our code to Digital Ocean to see the changes.

default.yaml is used to revert the App Spec to production.

The default.yaml has two changes:

name: Add “-prod” to the end of the name.
PGPASSWORD:value: new_password replaces the password.

(We’re not showing the entire file here for space reasons.)

This will change the name in the Digital Ocean dashboard to show that prod is visible. The password will revert to the original password from the primary branch of the database, and since the branch is main, the build will be the main branch.

With these changes, we are now ready to begin implementing our two GitHub Actions: “Create NeonDB Branch” and “Destroy NeonDB Branch.”

Create NeonDB branch

When a pull request is made to the main branch, this GitHub Action will fire and do a number of steps:

Create a development branch of the NeonDB database.
- Grab the host and password of this new DB.
Check out the GitHub Code.
Use a sed command to replace new_host & new_password placeholders in the .do/app.yaml file with the variables extracted in step 1a.
Install the Digital Ocean CLI.
Update the App Spec with the new yaml file.
Initiate a Digital Ocean deployment.

    name: Create Neon Branch and deploy dev to DO
    run-name: Create a Neon Branch 🚀
    on:
      pull_request:
        types: [opened]
        branches:
          - main
    jobs:
      Create-Neon-Branch:
        runs-on: ubuntu-latest
        steps:
          - name: Verify NEON API Key presence
            run: |
              if [ -z "${{ secrets.NEON_API_KEY }}" ]; then
                echo "NEON_API_KEY is empty"
              else
                echo "NEON_API_KEY is set"
              fi
          - name: Create Neon Branch
            id: create-branch
            uses: neondatabase/create-branch-action@v5
            with:
              project_id: "orange-violet-68318343"
              # optional (defaults to your primary  branch)
              parent: "main" 
              # optional (defaults to neondb)
              database: "neondb"
              branch_name: "development"
              username: "neondb_owner"
              api_key: "${{ secrets.NEON_API_KEY }}"
          - run: echo db_url ${{ steps.create-branch.outputs.db_url }}
          - run: echo host ${{ steps.create-branch.outputs.host }}
          - run: echo branch_id ${{ steps.create-branch.outputs.branch_id }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_host|'"${{ steps.create-branch.outputs.host }}"'|g' .do/app.yaml
              sed -i 's|new_password|'"${{ steps.create-branch.outputs.password }}"'|g' .do/app.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/app.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Hints on creating your action:

The NeonDB project_id is in the URL string when you load the project in the NeonDB dashboard.
The UUID for your Digital Ocean application is also found in the dashboard URL.

Save this workflow in /.github/workflows.

Delete NeonDB branch

Once the PR has been tested and approved, we want to destroy the NeonDB branch and revert the Digital Ocean deployment back to production.

The steps in this are:

Delete the NeonDB development branch.
Check out the code.
Update the default.yaml with our DB password from the GitHub Secrets.
Update the App Spec and deploy the application at Digital Ocean.

    name: Delete Neon Branch with GitHub Actions Demo
    run-name: Delete a Neon Branch 🚀
    on:
      pull_request:
        types: [closed]
        branches:
          - main
    jobs:
      delete-neon-branch:
        runs-on: ubuntu-latest
        steps:
          - name: Delete Neon branch
            uses: neondatabase/delete-branch-action@v3
            with:
              project_id: "orange-violet-68318343"
              branch: development
              api_key: ${{ secrets.NEON_API_KEY }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_password|'"${{ secrets.NEON_PW }}"'|g' .do/default.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/default.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Okay, that is a lot of code. Don’t forget to update the DO UUIDs to match your deployment. Push this all to your repo so that we can see our automation in action.

Here is the production version of the application running on Digital Ocean. I added a few extra elements for fun. The screenshot shows the mouse hover color on Ne (Neon).

Now, let’s make some changes to the code and start a pull request.

Create a dev branch for your code. Let’s change the colors in line 15 of/elements/templates/elements_list.html:

<li hx-delete="element/{{ element.id }}" hx-target="body" class="relative flex flex-col text-center p-5 rounded-md bg-[#7846a8] transition-colors hover:bg-orange-500 text-[white]">

This should make the boxes purple, with orange hover and white text.

Push the dev branch and open a pull request.

When the pull request is created, the “Create a Neon Branch” GitHub Action is called. A branch of the NeonDB is created, and the dev code is deployed to Digital Ocean.

Refreshing the application, we see the colors have been updated:

More importantly, we can test all we want without worrying about the production database. Any changes made on the dev branch are in the development branch of NeonDB. As a part of our testing, we deleted a number of entries—only Neon is left:

Since we’re happy with the PR, we can approve and merge the changes. This fires up the second GitHub Action: deleting the NeonDB development branch and pushing the production build to Digital Ocean:

The new colors are in prod, and the prod database was untouched by our testing on the PR!

Conclusion

In this post, we used GitHub Actions to automate creating and deleting a NeonDB database for testing pull request builds on Digital Ocean. If you would like to look at the code, it is available on GitHub. You’ll just need to wire in your NeonDB and Digital Ocean credentials to get up and running.

Automating Developer Relations Metrics with Low Code RunBooks

Doug Sillars — Mon, 24 Apr 2023 22:05:02 +0000

My job at unSkript is to spread awareness and excitement around the DevOps tooling we have built. But, I am also expected to provide reporting on various metrics around developer awareness and usage of our product. In this post, I walk through how I have automated the data collection process, so that I can spend more time creating content and building awareness.

A bit of background: at unSkript, we are building automation tools to reduce toil. In the DevOps/SRE space, toil is defined as the manual and repetitive work that needs to be done to keep everything shipshape. If you ask me – collecting metrics from a bunch of different services (Github, Google Analytics, internal databases, Docker,….), and aggregating them in one place – that sounds like toil. So let’s automate that away, and then I no longer have to think about it (until I decide to write a blog post about it, of course.)

Collecting the Data

unSkript is a tool to help you build RunBooks. A RunBook is a collection of steps (we call them Actions) that complete a task. For DevOps teams, that could be auto-remediation of your load balancers, running health checks on a K8s cluster, or even monitoring your daily Cloud costs.

I want to use these actions to collect a bunch of different data points, and store them all in one place. There are a few different ways that I use unSkript to collect the information:

Built in Actions

unSkript comes with hundreds of built-in Actions – simply drag & drop into your RunBook, configure your credentials, and you are ready to go! When using built-in Actions – unSkript can be thought of as essentially “no-code” to set up. There are several built-in Actions in unSkript that are well suited to collecting the data that I want to collect: Daily Unique users from Google Analytics, and the Github star count.

NOTE: Github stars as a DevRel metric can be controversial (IMO- it is useful as an indicator metric) but feel free to leave a comment below with your thoughts.)

Database Queries

Many of our stats are collected from Segment, and stored in a database (and that database is awesome for in depth detailed analysis). But I want to keep all of my high level statistics and data in one table, so, I’ll use the PostgreSQL connector to extract the datapoints I’d like into my dataset:

These Actions are “low-code” in that once you drag & drop the action and make the connections, you still need to create a SQL query to grab the results.

REST API

There are still a few more data points that I’d like to pull out of other tools. We have a REST API connector that makes this easy: set up your credentials and the headers you need – and you can create a new Action that extracts your data via API. These are also “low-code” but do require some understanding of how to make API calls – in order to set up the credentials properly.

For example: Docker Hub publishes the number of times our Docker Image has been downloaded. We can collect this number Each day using the REST API Action – and adding the endpoint and headers to the Action:

Storing & reporting our data

Once we have collected all of the data, we can create a message and post it on Slack for the team to see:

The message that is sent to the channel is a Python f string with variables added.

This is a fun way to update the team on a daily basis…but we also want to chart this data over time. To accomplish this, we have a table in PostgreSQL for our stats, and we just make an INSERT using the prebuilt Postgres Action (again this is low-code as you must write the SQL INSERT command:

The data in Postgres feeds a Grafana dashboard – allowing the team access to the latest data from our metrics – and the best part of it all is that there is no daily toil required. And many folks on the team just go to the dashboard to get the data – the DevRel team is no longer a bottleneck!

Progressive enhancement

As time goes on, more questions about data will arise.

As an example, since the number of Actions and RunBooks in GitHub keeps increasing, I was recently asked “how many Actions do we have in GItHub today?”

The first few times you are asked a question like this, you can probably get away with waving your hands, and a ballpark figure… but after being asked a few times, I knew I needed a “real” answer. Reusing an existing GitHub Action, I was able to create a file in Github with the counts that I needed. By dragging a new Action into my RunBook, writing a few lines of Python code (and a small change on the Postgres insert), I was able to easily extend the current data collection to include more data.

Aside: We can also leverage these values to create custom badges for the Github readme, and on the website – so creating the data has been a double win!

Scheduling

Now that I have built a RunBook that collects all of data we need (so far…) -> I want to automate the execution of the RunBook. Using unSkript’s Scheduler, I have set my RunBook to run at midnight GMT every day.

Now, I have daily reports being created for the team, that require ZERO work on my part!

Summary

Collecting and aggregating statistics via automation frees the team to can focus on our “real” work: creating more tools and applications – and no longer spend significant time on metric collection. At the same time, everyone has visibility into the project – showing the value of the DevRel team, without impacting their workload.

How does your DevRel team collect usage data? If you’d like to give unSkript a try, check out our Free trial. Join our Slack Channel, and I would be happy to chat with you on strategies to build your RunBook to collect your analytics data. I’ll also be happy to share the skeleton of my RunBook to get you started!

Automating the GitHub Nudge

Doug Sillars — Tue, 04 Apr 2023 16:46:28 +0000

Git is a tool of Actions. Millions of times a day, users checkout, push, pull and merge submissions to their repositories. The scale is staggering:over 3.5 billion contributions were made on GitHub in 2022. That’s 227 million Pull Requests merged, and over 31 million issues closed. However, What about the PRs and the issues that fall through the cracks and are forgotten? Should they be just left, forgotten (like the Island of Misfit toys in Rudolf the RedNosed Reindeer?)

In this post, we introduce an automated RunBook that introduces the “Github nudge.” Merriam-Webster’s definition for a nudge is “to prod lightly : urge into action.” The GitHub nudge identifies issues and PRs that have been sitting a while, and “nudges” the assignee to take a look. By not letting the team forget that the issues exist – they are more likely to be acted upon!

We’ve defined a few Actions in unSkript’s RunBook architecture to help us along this path:

Stale Issues

When issues have been assigned to a team member, but no work is being done on them, the issue has probably “gotten lost.” Everyone has a lot to work on, and sometimes these issues just lose priority – or get superseded by other tasks. That does not mean that they should just be ignored – resolving the issue will improve the project.

In unSkript, there is an Action to find “stale” issues: that is issues that are over a certain age.

This Action takes 3 input parameters – the Github owner and repository and the threshold (in days) upon which you define an issue as stale. In the case of our repository https://github.com/unskript/Awesome-CloudOps-Automation, the owner is unskript, the repo is Awesome-CloudOps-Automation, and we set the stale threshold at 14 days. There’s no real science around that day -it felt like a good number for our team.

The response is an array of issues that surpass that threshold. With this data, we can do some quick examinations:

Issues without an assignee

Issues that have not been assigned to a resource are not going to be worked on – we all have a lot on our plate already! In the case of Awesome-CloudOps-Automation – these are all “good first issues” for those interested in contributing. I like to scan this list once a week for changes – and to ensure that issues that do need work are properly assigned.

Issues with an assignee

If an issue has been around for 2 weeks, and is not yet resolved – its good to check on them to see the status – and that action is being made:

For example – it appears that issue 346 is assigned to me, and I am overdue with an update (oops!):

{'assignee': NamedUser(login="dougsillars"),
   'issue_number': 346,
   'title': '[Action]: GitHub Comment on an issue'}

In this case, the PR was already merged, but not connected to the issue, so it could be closed (whew!).

Stale Pull Requests

We can do the same thing for Pull Requests:

A stale PR almost feels worse than a stale Issue. When the work is completed, and ready (perhaps nearly ready) to be integrated into the codebase – there is a immediate improvement to the software. When. PR just languishes, the code does not improve, and sometimes further improvements are blocked The Action above gives a list of all PRs that are over a threshold (again we use 14 days) for a given GitHub repository. For example:

{331: '3 cost optimization runbooks'}

This output does not provide anyone that I can ‘nudge’, but we have another Action that we can use:

To generate all of the reviewers to nudge, I first create a list of the PR numbers that I obtained from the first Action (the output was named “stalePull”:

oldPRs = []
for pr in stalePull[1]:
oldPRs.append(list(pr.keys())[0])
print(oldPRs)

The Get Pull Request Reviewer Action takes 3 inputs – owner, repository and PR number.

Using the iteration command in unSkript, we can apply the list of pull requests (oldPRs) to the pull_request_number variable:

This will call the Action once per pull request – giving a full output of each PR, and who is left to review.

331: ['jayasimha-raghavan-unskript', 'shloka-bhalgat-unskript']

With a little Python, we can turn this around to list every user, and the PRs they should look at.

U03QU1K184X': [267, 153]

You may notice that the username is different. This is in anticipation of the next step:

The actual “nudge”

Ok, so we know whose issues and PR reviews are overdue. How do we alert them? At unSkript, the internal team uses Slack. I have created a table that compares each Github username with their Slack ID (that’s the weird variable above).

With the slack ID, I can now send a message to the team channel:

If you are listed below, can you please review the Pull requests next to your name? They have been open for 14 days.

<@U01UG9DRR7D>, please review pull requests [267, 153].
<@U03QU1K184X>, please review pull requests [267, 153].

Adding the @ in front of the username creates an “at” in Slack, so these two users have just been effectively nudged to look at their PRs. They have now been nudged in Slack to go into GitHub and take a look at the work that is being forgotten.

Automating the Nudge!

Using the Enterprise version (or the free trial) of unSkript, you can schedule each RunBook. The RUnBook I created for Awesome-CloudOps-Automation runs every Wednesday morning – alerting the team that there are some issues and pull requests that have been left behind and should be resolved.

How do you keep your Github issues and Pull requests chugging along? Do you have suggestions on how you might improve this RunBook? We’d love to hear about it in the unSkript Slack channel. Interested in trying out Github nudges with your team? All of the Actions described above are in our Open source (Docker instructions are in the readme), and in our free Trial.