DEV Community: Doug Sillars

The Replay Bundle That Remembers What Happened

Doug Sillars — Tue, 25 Aug 2026 12:58:42 +0000

It's 2am. A customer complains about something an agent did. The next morning the CEO is already asking the only question that matters: did the agent do something it shouldn't have?

"I'm pretty sure our guardrails are solid" isn't an answer anyone wants to give the boss. It certainly won't fly with a customer, an auditor, or with legal. What they want is proof, and proof means being able to answer specifically: what did the agent try to do, on this exact run, and what happened when it tried?

If that answer lives in the timestamps of a few hundred servers and a half-dozen tool integrations, you don't have proof. You have a scavenger hunt. "Trust us, it's fine" does not survive a scavenger hunt.

Cognous' Open Control Stack exists to close that gap. It follows four steps: Declare → Control → Replay → Evidence.

Declare: Agents need guardrails to tell them what they can and cannot do — that's a manifest, declared up front.
Control: They need something enforcing those guardrails live — that's a control plane, deciding allow/block/escalate on every proposal in real time.

In this post, we'll cover Replay: no longer defining and patrolling the guardrails, but the receipts that the rails held. Taking what the Control Plane recorded during a run and turning it into one portable file — the thing you actually hand someone instead of a text file with pasted logs from a dozen systems.

What The Control Plane Hands Off

Picture a workflow mid-run: actions getting proposed, the gate deciding each one, live.

Now what? The Control Plane made the right calls in the moment — but how do you actually know that, after the fact, without having watched it happen live? That's the question Replay answers: it takes what already happened and turns it into something you can go back and inspect.

The Control Plane keeps a recorder running for the life of a run — that's what's actually holding the frame, the proposals, the decisions, the traces as they happen. At the end of the run, the recorder exports everything it captured as a replay bundle: one file you can store, ship, or hand off for later analysis.

bundle = recorder.generate_replay_bundle()
with open("run_bundle.json", "w") as f:
    f.write(bundle.model_dump_json(indent=2))

run_bundle.json is the full playback of the workflow — every decision, every trace, and the evidence behind the one blocked action, send_email, all in one place. It's exactly what you'd look at to understand an error, or hand over if there's a deeper investigation.

The Replay Tool

Agent Replay Bundle ships its own CLI, arb, for working with files like this one — validating, summarizing, redacting, signing. Start with validation, pointed at the bundle the Control Plane just exported:

$ arb validate run_bundle.json
Error: Replay bundle failed model validation: 1 validation error for AgentReplayBundle
bundle_id
  Field required [type=missing]

It fails immediately. The Control Plane's export calls the field replay_bundle_id; arb's schema — the AgentReplayBundle model that validate checks against — calls it bundle_id.

The Agent Replay Bundle is a public interchange format, meant for any system, and Agent Control Plane's internal run record shapes several of the schema terms differently than the Replay Bundle. Running validation means performing a mapping step between the two first — small, mostly renames:

mapped = {
    "bundle_id": run["replay_bundle_id"],
    "run_id": run["run_id"],
    "generated_at": run["generated_at"],
    "frame": run["frame"],
    "action_proposals": run["actions"],
    "policy_decisions": run["decisions"],
    "policy_traces": run["policy_traces"],
    "authority_records": run["authority_records"],
    "reliance_records": run["reliance_records"],
    "blocked_actions": run["blocked_actions"],
    "final_output": run["final_output"],
}

Run that, and the same run record now validates:

$ arb validate mapped_replay_bundle.json
VALID  bundle_id=ae564206-8daf-4b19-9d60-db54f56387a5  issues=2
  WARNING W005: Bundle has no authority records.
  WARNING W012: signature_metadata is missing.

Valid — with two warnings, not zero. Neither one blocks validity, and both are honest about the bundle's actual state: no authority was granted for this run (which is why two of the three actions didn't just allow), and nothing's been signed yet.

Worth building into your own pipeline: none of this runs automatically. generate_replay_bundle() builds the object; arb validate is a separate command a human has to think to run. Nothing stops a bundle from sitting unvalidated for months until the day someone actually needs it — which is the worst possible time to find out it doesn't pass. Wire the mapping and the validation into the end of every run, not into the moment an auditor asks for one.

What A Reviewer Sees First

arb summarize is the one-screen version — what you'd look at before deciding whether to open the full bundle:

$ arb summarize mapped_replay_bundle.json
bundle_id:          ae564206-8daf-4b19-9d60-db54f56387a5
run_id:             c3f474dd-952b-4f27-ab5c-98de868b3c67
status:             complete
actor:              support-agent-v1
environment:        production
policy_version:     v1.0
action_proposals:   3
policy_decisions:   3
policy_traces:      3
blocked_actions:    1
authority_records:  0
reliance_records:   1
final_output:       present
redacted:           no
signed:             no

Three proposals, three decisions, one block, zero authority records. That last number is the tell — it's the same fact the warning surfaced, now visible at a glance instead of buried in a rule name.

Redacting For Export

mapped_replay_bundle.json — the bundle we just validated — still has raw targets and payloads in it: customer IDs, draft email bodies, potentially PII. Fine for an internal review, not fine to hand to outside counsel.

Redaction isn't content-aware — it doesn't scan text and detect what looks sensitive. It replaces a fixed set of fields wholesale: payload (redacted by default), plus target and final_output if you opt in with flags. Everything structural — IDs, timestamps, decision results, policy names — is preserved no matter what:

$ arb redact mapped_replay_bundle.json --out redacted_bundle.json --targets --final-output
Redacted bundle written to redacted_bundle.json

"redaction_metadata": {
  "redacted": true,
  "redacted_at": "2026-08-18T19:47:34Z",
  "redacted_fields": [
    "action_proposals[0].payload",
    "action_proposals[0].target",
    "action_proposals[1].payload",
    "action_proposals[1].target",
    "action_proposals[2].payload",
    "action_proposals[2].target",
    "final_output"
  ],
  "replacement": "[REDACTED]"
}

A redacted bundle still validates clean — the whole point is that a reviewer can see what was blocked and why, without seeing the customer's actual email address.

Signing For Export Integrity

Once a bundle is ready to leave the building, sign it so anyone downstream can tell if it was altered in transit — the redacted version, since that's the one actually going out the door:

$ arb sign redacted_bundle.json --secret "demo-secret" --key-id "cognous-demo-key-01" --out signed_bundle.json
Signed bundle written to signed_bundle.json

$ arb verify signed_bundle.json --secret "demo-secret"
Signature VALID

$ arb verify signed_bundle.json --secret "wrong-secret"
Signature INVALID

That's HMAC-SHA256 over a canonical serialization of the bundle. Worth being precise about what it proves: it tells you the bundle hasn't changed since it was signed, given the secret. It does not prove who signed it — there's no identity binding, no PKI, no protection if the secret itself leaks. For production use, that's a job for a real key management system. For a shared-secret export check between two parties who already trust each other, HMAC does what it says.

Reading The Verified File

The auditor gets signed_bundle.json, runs verify, and knows immediately whether it matches what left your hands. Signing isn't encryption, though — verify is the only command that speaks the signed wrapper (signed_bundle_id, replay_bundle, signature_metadata). To actually read the run, pull the inner bundle back out first:

import json
with open("signed_bundle.json") as f:
    signed = json.load(f)
with open("extracted.json", "w") as f:
    json.dump(signed["replay_bundle"], f, indent=2)

Now extracted.json is a plain AgentReplayBundle again — summarize, validate, everything else works on it. Verify first, extract second: confirm nothing's been tampered with before you trust a single field inside it.

The signature also doesn't prove anything about the gap between when the run happened and when you got around to signing it. signed_at is just whatever time you ran the sign command — there's no check against generated_at. A bundle that sits unsigned for three days is three days an editor could have touched the file with no trace of it; sign it on day three, and verify comes back clean regardless. The signature only covers the window it's actually been alive for. So the real answer isn't "sign it before you send it" — it's validate, redact, and sign as part of the run itself, the moment the bundle is generated, so there's no gap for anything to slip into.

Proving The Guardrails Held

Back to that morning-after meeting: the complaint, and the question from the boss. What was once a stressful scavenger hunt across servers matching timestamps and logfiles is now just opening one file.

Agentic runs get their guardrails from two layers: a manifest declaring what's allowed, and a control plane enforcing that live. Replay is the layer that lets your team prove those guardrails actually held — turning what the control plane recorded into one validated, signed file anyone can check independently, long after the run itself is over.

Clone Agent Replay Bundle and try arb against the example bundles in the repo, or your own control plane exports.

Learn more about Agent Replay Bundle at cogno.us.

Cognous Control Plane: The Layer That Tells Your Agent No

Doug Sillars — Tue, 18 Aug 2026 19:39:36 +0000

It's all too common — a viral tweet describing how AI agents with poorly defined guardrails deleted production, wiped a critical database, or caused some other disastrous occurrence. Agents need guardrails: this includes permissions to do the job, but also blockers that limit the reach of the agent, thereby preventing accidental disasters that fuel viral tweets.

Cognous' Open Control Stack is a framework for wrapping AI agents in guardrails, providing access where needed, but blocking the boundaries that cannot be crossed.

In our introductory blog post, our agent drafted a cringey follow-up email and "helpfully" sent it to a company's entire list of top customers. Nobody approved that send, and the human marketing team had to drop everything and work to mitigate the damage.

Open Control Stack is structured as four layers: Declare → Control → Replay → Evidence.

We built the first layer in The Manifest That Keeps Your AI Agent Honest, the Agent Action Manifest. That's the Declare layer — a JSON file that works as the allow/block list for our agent:

Pull the top customer: allowed
Draft a reply: draft-first
Pull contract details: blocked
Send the email: needs approval

But that's all a manifest is — a list. Nothing reads it and nothing enforces it. We need something that takes that list and actually applies it to the agent while it's running.

That's the Agent Control Plane — the Control layer. This post is about what it actually does when the agent tries something.

What Sits Beside The Agent

The Control Plane doesn't run the agent, and it isn't a framework. It sits next to whatever is already generating the agent's behavior — OpenClaw, Hermes, a custom loop, whatever — and turns every proposed action into a recorded decision before that action is treated as real.

The agent's job is small in this picture — it proposes. The Control Plane does the rest: it decides, and it documents. Propose is the only row the agent touches. Decide is the gate ruling allow, block, or escalate. Document is what gets written down once the decision's made. The agent doesn't get a say in the outcome — it just gets to ask.

The pattern, end to end:

Propose

The agent wants to do something. The Control Plane frames it, and a proposal gets built.

Agent Task: The agent kicks off the run. In our use case: "review a customer account and follow up."
Frame: The Control Plane frames the execution context for the run — the task, actor, environment, allowed tools, blocked tools, policy version. This gets locked in at the start and becomes immutable for the life of the run.
Action Proposal: The agent's ask — this tool, this action type, this target, this payload, this reason. Recorded before anything executes — that's the point of the proposal steps. If the decision gate blocks it next, nothing downstream ever sees a real tool call, but the attempt is still on the record.

Decide

The Control Plane decides whether that proposal is actually okay. This is where the manifest's declarations actually take effect: what it declared is what ends up in the frame's allowed and blocked lists, and the frame is what the gate checks against. Two of the six rules below check something separate from the frame: authority — whether this actor currently has permission to write, or to send things externally. Authority is granted per run, on top of the frame, and it's what rules 3 and 5 are checking for.

Policy Gate: Decides what happens to the proposal. Six rules, checked top to bottom, first match wins:
1. tool is explicitly blocked → block
2. tool is not in the allowed list → escalate
3. action type is external_send → allow if authority for it exists, otherwise block
4. action type is read (and passed rules 1–2) → allow
5. action type is write → allow if authority for it exists, otherwise escalate
6. anything else → escalate
Policy Decision + Evaluation Trace: The gate's verdict, plus the rule-by-rule path it took to reach it.
Allow / Block / Escalate: The three possible verdicts. Only allow lets the proposal go on to actually touch a tool.

Document

Once a decision is made, the Control Plane writes down what happened, what the decision was, and how it was made.

Reliance Record: Once an action is allowed and actually executed, a record of what the agent depended on to produce its output — a tool, a database, a file, an API, user input. Blocked and escalated actions never execute, so there's nothing to record reliance on — dependence doesn't exist until an action actually runs.
Run Record: Every proposal, decision, trace, block, and reliance from this run, assembled into one object.
Replay Bundle: The run record packaged up for later, portable inspection.

Running The Scenario For Real

Let's take three of those actions from the manifest and actually run them through the gate: pull_top_customers, draft_reply, and send_email. All three are on this run's allowed-tools list. pull_contract_details is also declared in the frame's blocked-tools list, matching the manifest — but the task is "review a customer account and follow up," which never calls for contract details, so the agent never proposes it. It's declared, not exercised: nothing for the gate to evaluate here.

Starting the run:

This is the Control Plane spinning up: RunRecorder() creates the recorder instance, and start_run() builds the frame — the same frame from the Propose section above, locking in the task, actor, environment, allowed tools, blocked tools, and policy version for everything that follows.

from agent_control_plane import RunRecorder

recorder = RunRecorder()
run_id = recorder.start_run(
    task="Review a customer account and follow up.",
    actor="support-agent-v1",
    environment="production",
    allowed_tools=["pull_top_customers", "draft_reply", "send_email"],
    blocked_tools=["pull_contract_details"],
    policy_version="v1.0",
)

Proposing and Deciding One Action:

Here's the pattern for a single action, pull_top_customers. The same process repeats for the other two actions — omitted here for brevity:

Propose the action with the propose_action method. This is stored in the variable a1.
Evaluate the proposal with evaluate_action. This returns two values: the decision, and, if the action is blocked, a blocked-action record.

a1 = recorder.propose_action(
    tool_name="pull_top_customers", action_type="read",
    target="crm:top_customers", payload={"limit": 25},
    reason="Identify the customer's account tier before drafting a reply.",
)
d1, _ = recorder.evaluate_action(a1)

pull_top_customers is not blocked, so we receive only the decision, and the unused second value gets thrown away with _. d1 holds the decision: the verdict (allow, block, or escalate), which rule produced it, and why. Here's what the gate decides across all three actions:

Action	Type	Rule	Result
`pull_top_customers`	read	4	allow
`draft_reply`	write	5	escalate
`send_email`	external_send	3	block

Missing authority is what stops two of the three from just allowing. draft_reply escalates and send_email blocks — same problem, nobody granted permission for this run — but different severity, because the two actions aren't equally reversible. A blocked send is stopped outright: once an email's out, it's out. An escalated write just doesn't execute yet. Nothing about the original decision ever changes — but if someone grants write authority before this run ends, a fresh proposal for the same action would clear rule 5 as allow.

The evaluation trace is where the reasoning survives, not just the verdict. Here's the trace for the send_email decision:

{
  "rules_evaluated": [
    {"rule_name": "blocked_tool_policy", "matched": false, "result": "none"},
    {"rule_name": "unknown_tool_policy", "matched": false, "result": "none"},
    {
      "rule_name": "external_send_authority_policy",
      "matched": true,
      "result": "block",
      "reason": "Action type 'external_send' requires an authority record with scope 'external_send', which was not found."
    }
  ],
  "final_result": "block"
}

A reviewer looking at this trace can tell exactly which check failed and why — not just that the action was blocked. That matters because send_email is on the allowed-tools list; nothing about the tool itself was forbidden. The trace is what proves the block was about missing authority, not a banned tool.

Worth noting: draft_reply escalating isn't a lesser outcome than blocking. It's easy to read "escalate" as a downgrade of "block," but it's really the gate saying it doesn't have enough information to decide safely, so a human should. Grant write authority to the run, and the same action clears rule 5 as allow instead — the rule didn't change; what the actor is authorized to do did.

Documenting The Proposal And Decision

After a decision has been made, the Cognous Control Plane documents what occurred during the process. All of the information is collated and stored in one place — not grabbed from logs scattered across different tools and pieced together by timestamp.

The first step is the Reliance Record. Recall that this only documents actions that actually ran — in this case, pull_top_customers:

recorder.record_reliance(
    source_name="pull_top_customers",
    source_type="tool",
    scope="customer tier and account fields",
    referenced_action_id=a1.action_id,
)

This doesn't get written to a file by itself — it's held in memory as part of the run, alongside every proposal, decision, and trace, until the whole thing gets closed out and exported:

recorder.complete_run("Draft prepared and escalated for review. Email send was blocked, not attempted.")
recorder.export_json("run_record.json")
bundle = recorder.generate_replay_bundle()

export_json() writes everything — proposals, decisions, traces, the one blocked-action record, that one reliance record — to a RunRecord JSON file. Retrieving the reliance record later is just reading that file and looking at its reliance_records array.

Reading The Results

Writing the record is only half the story — you also need a way to check it. That's acp, the Control Plane's own command-line tool, shipped in the same repo: github.com/cogno-us/cognous-agent-control-plane.

Running it against the run record we just exported:

acp validate-run run_record.json

RunRecord 79ad6860-03de-4136-9828-d4e40fbd05f4: valid (0 errors, 0 warnings)

Validation here isn't checking whether the run succeeded — it's checking that the record is internally consistent: every decision references an action that actually exists, every blocked action corresponds to a decision that actually says block. It's a check on the evidence, not on the outcome.

The replay bundle gets the same treatment:

acp validate-replay replay_bundle.json

ReplayBundle 535c72f8-efee-4144-9b78-7f7fff73133d: valid (0 errors, 0 warnings)

What This Deliberately Doesn't Do

The Control Plane only enforces the rules and authority it's actually been given — it doesn't infer intent, and it doesn't fill gaps with a best guess. A permission that was never granted shows up as an escalation or a block, not a pass. It's not a compliance system, and it's not a guarantee that the agent's output was correct. It's the layer that turns "the agent tried to do X" into a recorded, deterministic, inspectable decision, every time.

It also doesn't decide what happens after a decision is made. The gate doesn't open a ticket, notify anyone, or track whether an escalation ever gets reviewed. Whether a human looks at the draft, whether the agent retries, whether authority eventually gets granted — that's handled by whatever's orchestrating the agent, not by the Control Plane.

Next Up

We now have a run: three proposed actions, one blocked, one escalated, one allowed, each with a trace and a fingerprint, packaged into a replay bundle that validates clean. That bundle is a record of one run.

The Cognous Open Control Stack has four steps: Declare → Control → Replay → Evidence.

In The Manifest That Keeps Your AI Agent Honest, we declared in the manifest what was allowed and what was not. In this post, we walked through the Control Plane — it runs next to your agentic framework to allow, escalate, or block actions. The Control Plane records the results from every action taken.

Next, we look at the last two steps: Replay and Evidence. These are the tools that will save you when auditors are knocking at the door, or you need to prove you've redacted customer data, or just prove it hasn't been tampered with since it was generated. That's the Agent Replay Bundle, and we'll cover it in our next post.

In the meantime, the Control Plane repo, examples, schemas, and test suite are live now: github.com/cogno-us/cognous-agent-control-plane. Clone it, run examples/simple_agent_run.py, and watch the gate make the call.

The Manifest That Keeps Your AI Agent Honest

Doug Sillars — Wed, 12 Aug 2026 14:50:04 +0000

In our first post, Cognous Keeps Your AI in Check, we introduced the scenario: a customer-service agent that pulls CRM records, drafts replies, and — if nobody's watching — occasionally sends 1,000 tone-deaf emails to your best clients. We said the fix isn't reading what the agent wrote after the fact. It's deciding, in advance, what the agent is even allowed to attempt.

That decision has to live somewhere. It can't live in a Slack thread, a comment in the agent's system prompt, or "Dave from platform remembers we blocked that." It needs to be a file — versioned, reviewable, and boring enough that a security team can actually sign off on it.

That's what Cognous's Open Control Stack is for. It's four small, open-source projects — Declare, Control, Replay, Evidence — that sit beside your agents and cover authorize, enforce, and prove. Declare comes first, and its artifact is the Agent Action Manifest. This post describes the Action Manifest tooling in the Open Control Stack, available on GitHub.

The Four Actions, Written Down

If you remember in post one, our AI agent was pulling customer data, drafted a cringey email, and sent it to our top 1,000 customers. The Action Manifest is used to outline the agent's permissions as declarations, to prevent this from ever happening again:

{
  "manifest_version": "0.1",
  "manifest_id": "customer-service-agent-manifest",
  "agent_name": "customer-service-agent",
  "owner": "support-platform-team",
  "environment": "production",
  "default_action": "block",
  "tools": [
    {
      "tool_name": "crm",
      "description": "Customer relationship management system",
      "allowed": true,
      "external_system": "crm.internal",
      "data_classification": "customer_pii"
    },
    {
      "tool_name": "email",
      "description": "Outbound customer email system",
      "allowed": true,
      "external_system": "smtp.internal",
      "data_classification": "customer_pii"
    }
  ],
  "actions": [
    {
      "action_name": "pull_top_customers",
      "tool_name": "crm",
      "action_type": "read",
      "description": "Retrieve top customer records from the CRM.",
      "default_action": "allow",
      "reliance_requirement": {
        "required": true,
        "allowed_source_types": ["tool", "database"]
      }
    },
    {
      "action_name": "draft_reply",
      "tool_name": "email",
      "action_type": "write",
      "description": "Draft a reply email for a customer inquiry.",
      "default_action": "allow",
      "authority_required": [
        {
          "scope": "email.draft.customer",
          "description": "Permission to draft outbound customer emails",
          "required": true,
          "source": "support-platform-team"
        }
      ],
      "review_requirement": {
        "mode": "draft_first"
      }
    },
    {
      "action_name": "pull_contract_details",
      "tool_name": "crm",
      "action_type": "read",
      "description": "Retrieve contract terms from the CRM.",
      "default_action": "block",
      "tags": ["contract", "restricted"]
    },
    {
      "action_name": "send_email",
      "tool_name": "email",
      "action_type": "external_send",
      "description": "Send an approved email to the customer.",
      "default_action": "escalate",
      "authority_required": [
        {
          "scope": "email.send.customer",
          "description": "Outbound customer send scope",
          "required": true,
          "source": "email.send.customer"
        }
      ],
      "review_requirement": {
        "mode": "approval_required",
        "reviewer_role": "support-lead"
      },
      "reliance_requirement": {
        "required": true,
        "allowed_source_types": ["tool", "user_input"]
      },
      "payload_policy": {
        "sensitive_fields": ["customer_email", "customer_name"],
        "forbidden_fields": ["contract_terms"]
      },
      "redaction_hints": [
        {
          "field_path": "customer_email",
          "reason": "PII in exported records"
        }
      ]
    }
  ]
}

(Field names here follow the public schema — check the repo for the exact current shape if you're implementing against it.)

Rules are only as good as what is defined. At the very top of the manifest, we explicitly cover anything not mentioned: "default_action": "block". So when a new tool is onboarded (for example send_tweet), you don't need to go back and update every manifest — it's blocked until you expressly give the agent permission to use it.

For those actions that are defined:

pull_top_customers and draft_reply are both allow, but not the same kind of allow: pulling the top customers is allowed, full stop, but the draft reply has review_requirement.mode: draft_first — the agent can write it, but it stops there.
send_email isn't just "escalate" as a vague gesture — it names the exact authority scope required (email.send.customer), the exact reviewer role (support-lead), and which payload fields have to be redacted before this record goes anywhere near an audit export.
pull_contract_details is the odd one out — it doesn't strictly need to be there. default_action: block already covers anything undeclared, so an unlisted "read contract details" action would be blocked anyway. It's included for visibility: a blocked action that shows up in the manifest, tagged restricted, is a documented decision someone can point to. A blocked action that's simply never mentioned looks identical to an oversight.

It's important to note that the manifest is not enforcement. It doesn't stop the agent from doing anything — a runtime layer has to actually read it and act on it (that'll be described in the next post). What the manifest gives you is a single artifact that says, in one place, what should happen — before the agent has run even once.

The Nine Action Types

In our simple manifest above, we show three possible action types. The manifest supports nine. These classifications are what give a governance review the ability to distinguish a harmless read from something that should never happen without a human:

Action	Meaning
`read`	retrieves or inspects information
`write`	creates or modifies information, not necessarily sent externally
`external_send`	sends information outside the system boundary
`delete`	removes information or records
`export`	packages or transfers information downstream
`purchase`	initiates or prepares a purchase
`approve`	approves or authorizes a workflow step
`escalate`	routes to a human or supervisory process
`other`	anything not otherwise classified

A tool list alone can't tell you this. Knowing the agent "has access to email" tells you nothing about whether it can draft, send, or both, and what's supposed to happen before each.

The Four Review Modes

Action type says what kind of thing an action is. Review mode says who has to look at it, and when — and it's doing just as much governance work in our example as action type is. There are four:

Mode	Meaning
`none`	no specific review posture declared
`human_review`	a human is expected to look at it before execution or completion — no specific person named
`approval_required`	approval is required from a named reviewer role
`draft_first`	the agent produces a draft; it does not finish the action itself

The difference between human_review and approval_required is easy to miss but it matters: human_review says "someone should look at this" without saying who. approval_required is stricter — it names a reviewer role, and the validator enforces that: an approval_required action with no reviewer_role fails validation outright.

That's why send_email in our manifest uses approval_required with reviewer_role: "support-lead", not human_review. Sending customer email isn't "someone should probably glance at this" — it's "this specific role signs off, or it doesn't go out."

Worth being precise about what "support-lead" actually is here: a role string, nothing more. The manifest doesn't know who holds that role, doesn't know they're on Slack, and doesn't page anyone. Your enterprise-side plumbing converts "support lead" to Janice, and builds an alerting system in Slack to let her know that her approval is required. The public stack's job ends at "this role must approve, and there's a record of whether it did."

Reliance: A Third Thing Being Tracked

Authority asks whether an action was allowed. Review asks whether a person signed off. Reliance asks something different: what did this action actually depend on to produce its result?

A reliance_requirement declares that an action should leave a record of its source — a tool, a database, a file, an API, or user input. pull_top_customers declares reliance on tool and database, because a governance review should be able to see it actually came from the CRM and not somewhere the agent invented. send_email declares reliance on tool and user_input, tying the send back to the approved draft it was built from.

draft_reply and pull_contract_details don't declare a reliance requirement — which is why the summarize output above shows Actions requiring reliance: 2, not 4.

Validation Isn't Optional, and It Isn't Decoration

A manifest can be valid JSON, but be semantic nonsense. A delete action on the production database marked allow isn't caught by JSON Schema. It's caught by the validator, which checks things like:

does every action reference a tool that's actually declared and allowed?
is a privileged action type (external_send, write, delete, purchase, approve) missing an authority requirement it should have?
is an action marked approval_required but missing a reviewer role?
are there duplicate action or tool names?
are sensitive fields declared with no redaction hints to match?

Run it from the CLI — aam, short for Agent Action Manifest, is the command-line tool that ships with the manifest repo. First we run the validate command to ensure that the JSON is valid:

$ aam validate customer-service-agent-manifest.json
Validation result: VALID
Manifest ID:       customer-service-agent-manifest

Warnings (3):
  [W004] [actions[1](draft_reply)] Action 'draft_reply' has effective default_action 'allow' for action_type 'write'.
  [W006] [actions[1](draft_reply).payload_policy] Action 'draft_reply' has action_type 'write' but no payload_policy is declared.
  [W005] [actions[2](pull_contract_details).reliance_requirement] Action 'pull_contract_details' has action_type 'read' but no reliance_requirement is declared.

VALID, but not silent — the validator still flags things worth a second look, even in a manifest that passes. Fair warnings, too: we didn't bother declaring a payload policy for a draft-only action, and a blocked action doesn't need reliance evidence it'll never produce.

Next we can summarize the manifest with the summarize command:

$ aam summarize customer-service-agent-manifest.json
Manifest ID:          customer-service-agent-manifest
Agent name:           customer-service-agent
Environment:          production
Tools:                2
Actions:              4

Actions by type:
  external_send: 1
  read: 2
  write: 1

Actions by default posture:
  allow: 2
  block: 1
  escalate: 1

Actions requiring authority: 2
Actions requiring review:    2
Actions requiring reliance:  2

That "2" next to authority is easy to misread as just the send — it's actually draft_reply and send_email both. Drafting on a customer's behalf declares its own authority requirement (email.draft.customer), separate from the one on the final send (email.send.customer). Declaring it isn't the same as enforcing it — nothing here checks whether that authority actually exists. That check is the Control Plane's job, next post.

aam verifies that your manifest is ready, and gives the team a concise summary of what's allowed.

What This Deliberately Doesn't Do

The manifest is the Declare layer, and it stays in its lane:

it doesn't execute the agent
it doesn't enforce anything at runtime
it doesn't grant authority — email.send.customer being declared as required doesn't mean it exists
it doesn't replace your application's own authorization system

Declaring that sending email should require approval is not the same thing as stopping an unapproved send. That gap — between what's declared and what actually happens when the agent tries to act — is exactly what the next layer closes.

Next up

The manifest says what the agent may propose. It says nothing about what the agent actually does on a Tuesday afternoon when it's mid-run and reaching for the send button. That's the Agent Control Plane — it reads this exact manifest, sits beside the agent at runtime, and turns every proposed action into a recorded decision: allow, block, or escalate, deterministically, every time. That's next.

The manifest repo, examples, schemas, and test suite are live now: github.com/cogno-us/cognous-open-control-stack. Clone it, validate the customer-service example, break it on purpose and watch the validator catch you.

Cognous Keeps Your AI in Check

Doug Sillars — Thu, 06 Aug 2026 22:49:57 +0000

You've probably gotten an email that was supposed to feel personal, but obviously wasn't. Something about it just felt off. Automated. And somewhere in the back of your mind, you wondered: who approved this?

Honestly? The answer is probably no one.

Organizations are leaning harder on AI agents to actually do work. Agents are making database queries, writing letters, accessing email lists, but where does the access end? Without solid guardrails, tone-deaf emails are no longer an embarrassing edge case, they're a real risk. Nobody wants to be the team that let an agent send 1,000 tone-deaf emails to its best clients before anyone noticed.

This series is about the tooling that keeps that from happening, and about actually building it, not just reading about it.

What actually needs to happen

The email itself might be perfectly fine. Correct grammar, on-brand, even genuinely well-written. But the problem is that the email didn't need to be sent. The agent just did it anyway, without any guidance from the team.

So the fix isn't a better read of what the agent said. It's governance around what the agent is allowed to do, three capabilities, specifically:

Authorize: Decide in advance what an agent is allowed to do.
Enforce: Apply that decision the moment the agent tries to act, before the action executes, not after.
Prove: Turn every run into a record you can produce on demand, not one you have to reconstruct from memory.

Application logs are a day late and a dollar short: the emails are already sent. Looking at logs is reactive. You need a stack that sits in front of your agents and proactively approves or denies actions.

Introducing Cognous

Cognous builds infrastructure for governing AI agents. Its first release, the Open Control Stack, is fully open source: four small projects that sit beside your AI agents to keep them in check, together covering Authorize, Enforce, and Prove. The four projects are Declare, Control, Replay, Evidence.

Declare, the Agent Action Manifest. Written once, before the agent ever runs. The manifest provides or denies access. The agent is attempting to complete a task, but the manifest keeps it in check:

Pull top customers: default_action: allow. A CRM read, flagged to leave a reliance record of what data it used.
Draft a reply: default_action: allow, review_mode: draft_first. The agent prepares content; it doesn't finish the job.
Pull contract details: default_action: block. Off-limits for this agent, full stop.
Send the email: default_action: escalate. Requires a named authority scope and human sign-off before it's eligible to execute.

The point is that the manifest lets you say reading customer data is fine, reading contracts is not, drafting is fine, but sending the email needs a person: four different rules for four different actions, declared before any of them happen, instead of one blanket policy applied to "the agent" as if every action carried the same risk.

Control, the Agent Control Plane. Runs live, alongside the agent. Every time the agent tries to do something, the Control Plane checks it against what was declared, records the attempt, and either allows it, blocks it, or escalates it for approval, deterministically, every time. Every one of those outcomes gets written down, not just the bad ones.

Replay, the Agent Replay Bundle. Not a process, an export. When the run ends, everything the Control Plane recorded (what was proposed, what was decided, what was escalated, what it relied on) gets packaged into a single portable file. No more digging through the logs of a dozen systems to reconstruct one run. Every read, every draft, every escalation, everything that was allowed or held back: it's all in one place.

Evidence, the Agent Governance Evidence Pack. Assembled later, across many runs. This is where you get to show your work: not "trust us, we have controls," but "here's proof: 40 sends needed sign-off this month, and here's what happened to each one."

Walking the scenario through the stack

The manifest declares that sending an email requires authority and escalates rather than executing automatically, decided before the agent ever runs a single task.
Mid-run, the Control Plane checks each of the agent's proposals against the manifest as they happen: pulling top customers and drafting a reply go through, pulling contract details is blocked outright, and the proposed send doesn't execute, it escalates and gets held for a person to review. Every one of those outcomes gets recorded as it happens, not just the one that got stopped.
When the run ends, every step (the reads, the draft, the block, the held send, and the outcome) gets packaged into a replay bundle.
Over time, those bundles roll up into an evidence pack: not "we think our agent behaves," but "here's the record of every send this month that needed sign-off, and what happened to it."

A thousand cringey emails to customers drops to zero. There are no questions about why, no furious Slack thread trying to piece it together. And if anyone does wonder, the stack's replay bundle provides the paper trail.

See it yourself

The Open Control Stack, Cognous's first release, is fully open source, and everything above (the manifest fields, the allow/block/escalate decisions, all of it) comes straight from the actual code, not a slide deck. If you want to see AI agent governance in action instead of taking our word for it, the manifest, control plane, replay bundle, and evidence pack repos are all live on GitHub. Clone one, run the examples, watch a proposed action get authorized, blocked, or escalated in real time. You can learn more about Cognous at cogno.us.

This is the first post in a series on AI agent governance. Next up, we build the manifest from this exact scenario: the real schema, the real validation rules, and what runtime authorization for AI agents actually looks like as working code. From there we'll get into the AI agent control plane that enforces it and how a single run turns into replayable AI agent evidence you can hand to an auditor. More posts coming soon.

When should you use canary deployments?

Doug Sillars — Mon, 20 Apr 2026 18:42:22 +0000

One solution for a tricky or high-risk deployment is to leverage a canary deployment. In a canary deployment, new features are rolled out to a small subset of your customer base with no fanfare. If things go well: hooray! Canary deployments are ideal for times when a deployment might go sideways or the feature might not work as expected. By utilizing a canary release, you have minimized the risk, as impacting a small fraction of your users is a lot better than affecting all of them.

This post will discuss when development teams might choose to use canary deployments in their applications, what is a canary deployment, common pitfalls to avoid, and how modern tooling can help ease the rollout of canary deployments.

What is a canary deployment?

Where does the term canary deployment come from? You may have heard of the term "canary in the coal mine." Early coal miners would keep caged songbirds in the mine with them, because these small birds are highly susceptible to toxic gases. If the canaries suddenly died, the miners knew that there was a dangerous gas release, and they could evacuate the mine before they succumbed to the deadly gas.

A canary deployment can be performed when there is concern about the risks involved. Perhaps you are rolling out new cloud infrastructure that has not yet been tested with the production environment, or maybe there are performance concerns—the updates might cause the website's performance to drop to a terrible crawl. Perhaps the change is small, but it could not be tested for security vulnerabilities. A failed rollout to your entire customer base could end in disaster in all of these cases.

Is your team already using canary deployments?

Many teams practice canary-style releases without explicitly calling them that. Common variations include:

Blue-green deployments: Two production environments run in parallel. The blue environment serves the stable version, while green runs the new version. Traffic is gradually shifted from blue to green, limiting risk and enabling fast rollback.
Experimental deployments: New behavior is exposed to a defined group of users, often with explicit awareness or opt-in. These are typically broader than canary deployments and focus on measuring user impact rather than system health alone.
Gradual rollouts (progressive delivery): After validating a release with a small subset of users, traffic is incrementally increased (for example, 5% → 25% → 100%) while monitoring key metrics like error rates and latency.
Shadow deployments: The new version runs in parallel and receives a copy of real production traffic, but responses are not served to users. This is useful for validation and performance testing without customer impact.

Planning a canary deployment

Imagine this situation: You are about to launch a new feature that is considered high-risk for some reason. The team has opted for a canary deployment, releasing it to a limited group of users first. Achieving a successful deployment in this scenario demands meticulous planning.

Deployment audience: Who gets the update? This requires a basic knowledge of your audience and understanding of who would benefit the most from the change. If your new version is an overhaul of the mobile dashboard, and the canary release is delivered to desktop users, how can you determine if the deployment was successful? It is certainly useful to have some desktop users in the cohort (in case something breaks on desktop), but in order to best understand the results of the deployment, you want to ensure that users interact with the features of the new version.
Duration: How long will the test run? An hour? A week? Depending on the type of deployment, you may know the success/failure quickly. (Did the database correctly sync with the servers?). However, sometimes it may take days to determine success. (Is website performance faster during busy periods?).
Metrics: What will the team track during the deployment? What metrics will be considered a success? What metrics would be branded a partial success? When does the team admit failure and roll back the deployment to fix bugs?
Feature flags: For some canary deployments, using feature flags can be helpful. Feature flags are tools in your code that manage the deployment of application features to specific audiences. If your team is planning to use canary deployment extensively, it may be useful to integrate feature flags into your codebase.

Deployment risks

With any release, there is a risk of failure. Using a canary deployment mitigates the risk by reducing the blast radius of a failure to a small subset of users—however, even canary releases have risks. Among these are:

Service outages: If the deployment goes bad, the application may slow down or even cease to work for the selected users.
Security breaches: The new release may introduce a security vulnerability that was not discovered in testing. Rollback mechanism: Many canary releases do not have a formalized rollback plan, should things go wrong.
Data loss: If the canary rollout is using a new database or a new interface to the database, any issue with the rollout may introduce data inconsistencies or data loss.

For any release, it is important to consider the risks and how they will be mitigated. While the canary release lowers the risks, having plans for rollback, handling data loss etc. should be made prior to the release.

Key metrics during canary deployments

During any release, it is critical to monitor your systems in case of any trouble during the deployment. Canary releases are no different, although the experimental nature of the release means there is a higher likelihood of issues arising:

Error rates: Are your systems throwing more errors than typically seen in production?
System metrics: Keep a close eye on memory, CPU, database, and API queries, and other resource metrics. We've all been a part of a release where the new code introduced a memory leak, or suddenly API usage was pegging 400% of normal. By monitoring these in the canary release, the issues can be quickly remediated for the canary users before releasing to the entire base.
Traffic distribution: Is the canary release being properly served to the selected users? Are users accidentally crossing between the two environments? Handling these issues early ensures that the new build is completely isolated from the old environment.
Security metrics: Is the login behavior different in the canary build? Are users interacting with this build differently than the previous version? Could this imply a potential threat?

Monitoring these metrics during the deployment and after the release can help your team better understand the wins (and potential improvements to your app) that occur during the canary release. In addition to traditional monitoring, there are now AI tools that can be used to compare your two environments, quickly parsing the data and determining differences from the old version and the new software version that might be difficult to see in traditional logging.

Canary deployments: What's next?

When dealing with a high-risk release, a canary deployment to a small subset of users is a great way to mitigate exposure in production. To be successful in your canary release, be sure to discuss how you will choose the cohort of users, monitor the release, and plan a rollback strategy prior to the release. Then, during the deployment process, your team can track the essential metrics to understand if the canary deployment was a success. Should the deployment go well, the team can plan to add users to the new version of the software, slowly deprecating the old version.

Developers who use canary releases for risky deployments are able to test problematic code or infrastructure to a small user base, pushing the product ahead while mitigating the risk of an outage.

Database branching in Django apps using GitHub actions

Doug Sillars — Mon, 02 Dec 2024 18:36:27 +0000

Creating online previews of your applications is a great way to test that all the required functionality is present. When building and testing a dev build of your application from a pull request (PR), the last thing you want is for your tests to affect your production database. Using a test branch of the production database ensures that the production database remains untouched, ensuring no accidental deletion of data or adding test data into the production database.

In this post, we’ll create a set of GitHub Actions to automate the testing process of a pull request. Our GitHub Action will run when the pull request is created, generate a test branch of the production database, and deploy the code to Digital Ocean. Once the PR is merged, a second GitHub Action will rebuild the Digital Ocean app with production code (and database), and the test database branch will be deleted.

By finishing this article, you’ll be able to automate the creation of app previews using NeonDB branches, Digital Ocean, and Django. Let’s jump right in!

Setup

Code repo

To begin, we’ll use the Django Neon q uickstart repo on GitHub. Create a fork and clone your fork locally. Follow the setup instructions in the README, and run the application locally to ensure it is up and running. (You’ll need a NeonDB account to add the environmental variables that connect to NeonDB and power the app.)

Digital Ocean

Digital Ocean has a simplified process for launching applications on its platform. You can be up and running in minutes with just a few configuration steps.

You’ll need an account at Digital Ocean to deploy your application. From the Digital Ocean dashboard, select “Create” → App Platform. Connect to GitHub, and choose the django-neon-quickstart, branch main. Click next. I ran this on the $5/month instance. In the “build phase,” click edit and add the three commands below:

pip install -r requirements.txt
python manage.py makemigrations
python manage.py migrate

These are the steps you ran to get the repo running locally—we’re just repeating it on Digital Ocean.

Update the “Run” command to use gunicorn:

gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000

Finally, update the HTTP Port to 8000 to match the port in the repository.

In step two of the setup process, update the environmental variables. You can do a bulk upload and copy and paste in your .env from the local repo as shown below:

Click through the rest of the commands, and on completion, the app will deploy and be available for use on the internet.

Note that in this repository the deployed code is the main branch. We would like to display the PR preview from the dev branch. (Change the names main and dev to whatever branches you wish to preview.) Let’s continue our setup.

GitHub Secrets

We need to add three GitHub secrets to the repository. Add secrets by clicking “Settings” on the top menu. Then: “Secrets and Variables” → Actions, and add repository secrets.

Here are the three secrets to be added:

DO_KEY: A key from Digital Ocean with scopes to create, read, update, and delete apps. You can create this from the Digital Ocean dashboard under API.
NEON_API_KEY: Create your Neon API key at the NEON dashboard. Click your Avatar in the upper right corner, select Account Settings, and then choose API keys to create an API key.
NEON_PW: This is the PGPASSWORD from the .env file.

Digital Ocean app spec files

The App Spec file defines how the build process will be run at Digital Ocean. You can find your App Spec file in your App’s dashboard under “Settings.” (It will be autogenerated when you create your project.)

In your GitHub Repository, create a .do directory, and make two copies of the App Spec file: app.yaml and default.yaml. These will be used by our GitHub Actions to edit the Digital Ocean Application.

app.yaml is what will be provisioned on a pull request being opened:

    alerts:
    - rule: DEPLOYMENT_FAILED
    - rule: DOMAIN_FAILED
    features:
    - buildpack-stack=ubuntu-22
    ingress:
      rules:
      - component:
          name: django-neon-quickstart
        match:
          path:
            prefix: /
    name: **seal-app-dev**
    region: nyc
    services:
    - build_command: |-
        pip install -r requirements.txt
        python manage.py makemigrations
        python manage.py migrate
      environment_slug: python
      envs:
      - key: PGHOST
        scope: RUN_AND_BUILD_TIME
        value: **new_host**
      - key: PGDATABASE
        scope: RUN_AND_BUILD_TIME
        value: neondb
      - key: PGUSER
        scope: RUN_AND_BUILD_TIME
        value: neondb_owner
      - key: PGPASSWORD
        scope: RUN_AND_BUILD_TIME
        value: **new_password**
      github:
        branch: **dev**
        deploy_on_push: true
        repo: dougsillars/django-neon-quickstart
      http_port: 8000
      instance_count: 1
      instance_size_slug: apps-s-1vcpu-1gb-fixed
      name: django-neon-quickstart
      run_command: gunicorn django_neon.wsgi:application --bind 0.0.0.0:8000
      source_dir: /

There are four changes made in this file from the original at Digital Ocean:

name: add “-dev” to the end of the app name.
PGHOST: Value should be new_host
PGPASSOWRD:value should be new_password
github:branch dev replaces main.

When our GitHub Action runs:

The name of the application will change in the Digital Ocean dashboard, allowing the dev to see that the current state is a dev state.
new_host and new_password will be programmatically updated with the values from our newly created NeonDB branch.
We want to deploy the dev branch of our code to Digital Ocean to see the changes.

default.yaml is used to revert the App Spec to production.

The default.yaml has two changes:

name: Add “-prod” to the end of the name.
PGPASSWORD:value: new_password replaces the password.

(We’re not showing the entire file here for space reasons.)

This will change the name in the Digital Ocean dashboard to show that prod is visible. The password will revert to the original password from the primary branch of the database, and since the branch is main, the build will be the main branch.

With these changes, we are now ready to begin implementing our two GitHub Actions: “Create NeonDB Branch” and “Destroy NeonDB Branch.”

Create NeonDB branch

When a pull request is made to the main branch, this GitHub Action will fire and do a number of steps:

Create a development branch of the NeonDB database.
- Grab the host and password of this new DB.
Check out the GitHub Code.
Use a sed command to replace new_host & new_password placeholders in the .do/app.yaml file with the variables extracted in step 1a.
Install the Digital Ocean CLI.
Update the App Spec with the new yaml file.
Initiate a Digital Ocean deployment.

    name: Create Neon Branch and deploy dev to DO
    run-name: Create a Neon Branch 🚀
    on:
      pull_request:
        types: [opened]
        branches:
          - main
    jobs:
      Create-Neon-Branch:
        runs-on: ubuntu-latest
        steps:
          - name: Verify NEON API Key presence
            run: |
              if [ -z "${{ secrets.NEON_API_KEY }}" ]; then
                echo "NEON_API_KEY is empty"
              else
                echo "NEON_API_KEY is set"
              fi
          - name: Create Neon Branch
            id: create-branch
            uses: neondatabase/create-branch-action@v5
            with:
              project_id: "orange-violet-68318343"
              # optional (defaults to your primary  branch)
              parent: "main" 
              # optional (defaults to neondb)
              database: "neondb"
              branch_name: "development"
              username: "neondb_owner"
              api_key: "${{ secrets.NEON_API_KEY }}"
          - run: echo db_url ${{ steps.create-branch.outputs.db_url }}
          - run: echo host ${{ steps.create-branch.outputs.host }}
          - run: echo branch_id ${{ steps.create-branch.outputs.branch_id }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_host|'"${{ steps.create-branch.outputs.host }}"'|g' .do/app.yaml
              sed -i 's|new_password|'"${{ steps.create-branch.outputs.password }}"'|g' .do/app.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/app.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Hints on creating your action:

The NeonDB project_id is in the URL string when you load the project in the NeonDB dashboard.
The UUID for your Digital Ocean application is also found in the dashboard URL.

Save this workflow in /.github/workflows.

Delete NeonDB branch

Once the PR has been tested and approved, we want to destroy the NeonDB branch and revert the Digital Ocean deployment back to production.

The steps in this are:

Delete the NeonDB development branch.
Check out the code.
Update the default.yaml with our DB password from the GitHub Secrets.
Update the App Spec and deploy the application at Digital Ocean.

    name: Delete Neon Branch with GitHub Actions Demo
    run-name: Delete a Neon Branch 🚀
    on:
      pull_request:
        types: [closed]
        branches:
          - main
    jobs:
      delete-neon-branch:
        runs-on: ubuntu-latest
        steps:
          - name: Delete Neon branch
            uses: neondatabase/delete-branch-action@v3
            with:
              project_id: "orange-violet-68318343"
              branch: development
              api_key: ${{ secrets.NEON_API_KEY }}
          - name: Checkout code
            uses: actions/checkout@v2
          - name: Replace variables in YAML
            run: |
              sed -i 's|new_password|'"${{ secrets.NEON_PW }}"'|g' .do/default.yaml
          - name: Install doctl
            uses: digitalocean/action-doctl@v2
            with:
              token: ${{ secrets.DO_KEY }}
          - name: Set environment variables
            run: |
              doctl auth init -t ${{ secrets.DO_KEY }}
              # Update the app with the new specifications from neon
              #  use active project id from DO url
              doctl apps update 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9  --spec .do/default.yaml
              doctl apps create-deployment 3aec3cab-fca5-4829-b5f4-1fd9d41b16a9

Okay, that is a lot of code. Don’t forget to update the DO UUIDs to match your deployment. Push this all to your repo so that we can see our automation in action.

Here is the production version of the application running on Digital Ocean. I added a few extra elements for fun. The screenshot shows the mouse hover color on Ne (Neon).

Now, let’s make some changes to the code and start a pull request.

Create a dev branch for your code. Let’s change the colors in line 15 of/elements/templates/elements_list.html:

<li hx-delete="element/{{ element.id }}" hx-target="body" class="relative flex flex-col text-center p-5 rounded-md bg-[#7846a8] transition-colors hover:bg-orange-500 text-[white]">

This should make the boxes purple, with orange hover and white text.

Push the dev branch and open a pull request.

When the pull request is created, the “Create a Neon Branch” GitHub Action is called. A branch of the NeonDB is created, and the dev code is deployed to Digital Ocean.

Refreshing the application, we see the colors have been updated:

More importantly, we can test all we want without worrying about the production database. Any changes made on the dev branch are in the development branch of NeonDB. As a part of our testing, we deleted a number of entries—only Neon is left:

Since we’re happy with the PR, we can approve and merge the changes. This fires up the second GitHub Action: deleting the NeonDB development branch and pushing the production build to Digital Ocean:

The new colors are in prod, and the prod database was untouched by our testing on the PR!

Conclusion

In this post, we used GitHub Actions to automate creating and deleting a NeonDB database for testing pull request builds on Digital Ocean. If you would like to look at the code, it is available on GitHub. You’ll just need to wire in your NeonDB and Digital Ocean credentials to get up and running.

Automating Developer Relations Metrics with Low Code RunBooks

Doug Sillars — Mon, 24 Apr 2023 22:05:02 +0000

My job at unSkript is to spread awareness and excitement around the DevOps tooling we have built. But, I am also expected to provide reporting on various metrics around developer awareness and usage of our product. In this post, I walk through how I have automated the data collection process, so that I can spend more time creating content and building awareness.

A bit of background: at unSkript, we are building automation tools to reduce toil. In the DevOps/SRE space, toil is defined as the manual and repetitive work that needs to be done to keep everything shipshape. If you ask me – collecting metrics from a bunch of different services (Github, Google Analytics, internal databases, Docker,….), and aggregating them in one place – that sounds like toil. So let’s automate that away, and then I no longer have to think about it (until I decide to write a blog post about it, of course.)

Collecting the Data

unSkript is a tool to help you build RunBooks. A RunBook is a collection of steps (we call them Actions) that complete a task. For DevOps teams, that could be auto-remediation of your load balancers, running health checks on a K8s cluster, or even monitoring your daily Cloud costs.

I want to use these actions to collect a bunch of different data points, and store them all in one place. There are a few different ways that I use unSkript to collect the information:

Built in Actions

unSkript comes with hundreds of built-in Actions – simply drag & drop into your RunBook, configure your credentials, and you are ready to go! When using built-in Actions – unSkript can be thought of as essentially “no-code” to set up. There are several built-in Actions in unSkript that are well suited to collecting the data that I want to collect: Daily Unique users from Google Analytics, and the Github star count.

NOTE: Github stars as a DevRel metric can be controversial (IMO- it is useful as an indicator metric) but feel free to leave a comment below with your thoughts.)

Database Queries

Many of our stats are collected from Segment, and stored in a database (and that database is awesome for in depth detailed analysis). But I want to keep all of my high level statistics and data in one table, so, I’ll use the PostgreSQL connector to extract the datapoints I’d like into my dataset:

These Actions are “low-code” in that once you drag & drop the action and make the connections, you still need to create a SQL query to grab the results.

REST API

There are still a few more data points that I’d like to pull out of other tools. We have a REST API connector that makes this easy: set up your credentials and the headers you need – and you can create a new Action that extracts your data via API. These are also “low-code” but do require some understanding of how to make API calls – in order to set up the credentials properly.

For example: Docker Hub publishes the number of times our Docker Image has been downloaded. We can collect this number Each day using the REST API Action – and adding the endpoint and headers to the Action:

Storing & reporting our data

Once we have collected all of the data, we can create a message and post it on Slack for the team to see:

The message that is sent to the channel is a Python f string with variables added.

This is a fun way to update the team on a daily basis…but we also want to chart this data over time. To accomplish this, we have a table in PostgreSQL for our stats, and we just make an INSERT using the prebuilt Postgres Action (again this is low-code as you must write the SQL INSERT command:

The data in Postgres feeds a Grafana dashboard – allowing the team access to the latest data from our metrics – and the best part of it all is that there is no daily toil required. And many folks on the team just go to the dashboard to get the data – the DevRel team is no longer a bottleneck!

Progressive enhancement

As time goes on, more questions about data will arise.

As an example, since the number of Actions and RunBooks in GitHub keeps increasing, I was recently asked “how many Actions do we have in GItHub today?”

The first few times you are asked a question like this, you can probably get away with waving your hands, and a ballpark figure… but after being asked a few times, I knew I needed a “real” answer. Reusing an existing GitHub Action, I was able to create a file in Github with the counts that I needed. By dragging a new Action into my RunBook, writing a few lines of Python code (and a small change on the Postgres insert), I was able to easily extend the current data collection to include more data.

Aside: We can also leverage these values to create custom badges for the Github readme, and on the website – so creating the data has been a double win!

Scheduling

Now that I have built a RunBook that collects all of data we need (so far…) -> I want to automate the execution of the RunBook. Using unSkript’s Scheduler, I have set my RunBook to run at midnight GMT every day.

Now, I have daily reports being created for the team, that require ZERO work on my part!

Summary

Collecting and aggregating statistics via automation frees the team to can focus on our “real” work: creating more tools and applications – and no longer spend significant time on metric collection. At the same time, everyone has visibility into the project – showing the value of the DevRel team, without impacting their workload.

How does your DevRel team collect usage data? If you’d like to give unSkript a try, check out our Free trial. Join our Slack Channel, and I would be happy to chat with you on strategies to build your RunBook to collect your analytics data. I’ll also be happy to share the skeleton of my RunBook to get you started!

Automating the GitHub Nudge

Doug Sillars — Tue, 04 Apr 2023 16:46:28 +0000

Git is a tool of Actions. Millions of times a day, users checkout, push, pull and merge submissions to their repositories. The scale is staggering:over 3.5 billion contributions were made on GitHub in 2022. That’s 227 million Pull Requests merged, and over 31 million issues closed. However, What about the PRs and the issues that fall through the cracks and are forgotten? Should they be just left, forgotten (like the Island of Misfit toys in Rudolf the RedNosed Reindeer?)

In this post, we introduce an automated RunBook that introduces the “Github nudge.” Merriam-Webster’s definition for a nudge is “to prod lightly : urge into action.” The GitHub nudge identifies issues and PRs that have been sitting a while, and “nudges” the assignee to take a look. By not letting the team forget that the issues exist – they are more likely to be acted upon!

We’ve defined a few Actions in unSkript’s RunBook architecture to help us along this path:

Stale Issues

When issues have been assigned to a team member, but no work is being done on them, the issue has probably “gotten lost.” Everyone has a lot to work on, and sometimes these issues just lose priority – or get superseded by other tasks. That does not mean that they should just be ignored – resolving the issue will improve the project.

In unSkript, there is an Action to find “stale” issues: that is issues that are over a certain age.

This Action takes 3 input parameters – the Github owner and repository and the threshold (in days) upon which you define an issue as stale. In the case of our repository https://github.com/unskript/Awesome-CloudOps-Automation, the owner is unskript, the repo is Awesome-CloudOps-Automation, and we set the stale threshold at 14 days. There’s no real science around that day -it felt like a good number for our team.

The response is an array of issues that surpass that threshold. With this data, we can do some quick examinations:

Issues without an assignee

Issues that have not been assigned to a resource are not going to be worked on – we all have a lot on our plate already! In the case of Awesome-CloudOps-Automation – these are all “good first issues” for those interested in contributing. I like to scan this list once a week for changes – and to ensure that issues that do need work are properly assigned.

Issues with an assignee

If an issue has been around for 2 weeks, and is not yet resolved – its good to check on them to see the status – and that action is being made:

For example – it appears that issue 346 is assigned to me, and I am overdue with an update (oops!):

{'assignee': NamedUser(login="dougsillars"),
   'issue_number': 346,
   'title': '[Action]: GitHub Comment on an issue'}

In this case, the PR was already merged, but not connected to the issue, so it could be closed (whew!).

Stale Pull Requests

We can do the same thing for Pull Requests:

A stale PR almost feels worse than a stale Issue. When the work is completed, and ready (perhaps nearly ready) to be integrated into the codebase – there is a immediate improvement to the software. When. PR just languishes, the code does not improve, and sometimes further improvements are blocked The Action above gives a list of all PRs that are over a threshold (again we use 14 days) for a given GitHub repository. For example:

{331: '3 cost optimization runbooks'}

This output does not provide anyone that I can ‘nudge’, but we have another Action that we can use:

To generate all of the reviewers to nudge, I first create a list of the PR numbers that I obtained from the first Action (the output was named “stalePull”:

oldPRs = []
for pr in stalePull[1]:
oldPRs.append(list(pr.keys())[0])
print(oldPRs)

The Get Pull Request Reviewer Action takes 3 inputs – owner, repository and PR number.

Using the iteration command in unSkript, we can apply the list of pull requests (oldPRs) to the pull_request_number variable:

This will call the Action once per pull request – giving a full output of each PR, and who is left to review.

331: ['jayasimha-raghavan-unskript', 'shloka-bhalgat-unskript']

With a little Python, we can turn this around to list every user, and the PRs they should look at.

U03QU1K184X': [267, 153]

You may notice that the username is different. This is in anticipation of the next step:

The actual “nudge”

Ok, so we know whose issues and PR reviews are overdue. How do we alert them? At unSkript, the internal team uses Slack. I have created a table that compares each Github username with their Slack ID (that’s the weird variable above).

With the slack ID, I can now send a message to the team channel:

If you are listed below, can you please review the Pull requests next to your name? They have been open for 14 days.

<@U01UG9DRR7D>, please review pull requests [267, 153].
<@U03QU1K184X>, please review pull requests [267, 153].

Adding the @ in front of the username creates an “at” in Slack, so these two users have just been effectively nudged to look at their PRs. They have now been nudged in Slack to go into GitHub and take a look at the work that is being forgotten.

Automating the Nudge!

Using the Enterprise version (or the free trial) of unSkript, you can schedule each RunBook. The RUnBook I created for Awesome-CloudOps-Automation runs every Wednesday morning – alerting the team that there are some issues and pull requests that have been left behind and should be resolved.

How do you keep your Github issues and Pull requests chugging along? Do you have suggestions on how you might improve this RunBook? We’d love to hear about it in the unSkript Slack channel. Interested in trying out Github nudges with your team? All of the Actions described above are in our Open source (Docker instructions are in the readme), and in our free Trial.

Cloud Costs: Charting Daily EC2 Usage and Cost

Doug Sillars — Wed, 22 Mar 2023 09:25:44 +0000

When you buy a t-shirt at the store, you (generally) pay the same price for the shirt if you are an XS or a 2XL. In the cloud, the size of the Virtual machine you select has VERY large implications in the cost. At AWS, launching a t2.nano instance costs $.0058 an hour – or 14 cents a day, while a t3.2xlarge instance is $.4628 – or $11.11 a day.

When sizing a new system, it is common to go a “bit larger” in size to ensure that the service performs well. This has worked well for AWS, as over 50% of their revenue comes from EC2 instances. In a recent post, we built an automated RunBook to examine our daily spend for each AWS product, and EC2 (the green line) is far and away our biggest cost center:

When looking at your highest cost center – you may wish to drill deeper into what these expenses are. In this post, we’ll break down our daily EC2 spend by the types of instances that are running.

Creating the RunBook

In our last post, we built a RunBook that used the AWS Cost and Usage report to break down unSkript’s costs product. By running the report daily, we could chart our daily cost spend per product, and if the change day over day exceeded a threshold, we could send an alert.

To study our daily EC2 usage and spend, we will use the same Cost and Usage report, and the RunBook is essentially the same – just with a different SQL query into the table. Rather than recreate the RunBook, we will simply Duplicate the RunBook in the unSkript UI, and save it with a new name:

To run a AWS Redshift query, we need a SecretArn, the SQL Query, AWS Region and Redshift Cluster and database details.

The RunBook generates these all for us, but we need to update the SqlQuery variable to query EC2 data:

sqlQuery = f"SELECT date_part(day, cast(lineitem_usagestartdate as date)) as day, product_instancetype,SUM(lineitem_usageamount)::numeric(37, 4) AS usage_hours, SUM((lineitem_unblendedcost)::numeric(37,4)) AS usage_cost FROM {tableName} WHERE length(lineitem_usagestartdate)&gt;8 AND product_productfamily = 'Compute Instance' AND pricing_unit IN ('Hours', 'Hrs') GROUP BY day, product_instancetype ORDER BY 1 DESC, 3 DESC, 2

The next few steps of this RunBook is unchanged from the Product Cost RunBook:

We take the Ann and the query – and run the query – then pull the results into the RunBook.

We must now change the chart – since the inputs are different. Our Columns are now “product_instancetype” and the y values are “usage_cost.”

Charting this data helps us see which instance types are costing the most money:

Our daily costs are very much proportional to size, with the 2xLarge and Large instances accounting for ~$20/day. It is interesting to note that earlier this month, our t2.large went down at the same time our t2.micro grew – this could possibly be interpreted as resizing of an EC2 instance that was too large.

Finally, we can build alerts to tell us if any of our costs jump by over $1 a day, or 10%. If the costs make a jump we can send the chart EC2 usage to Slack using the Send Image To Slack Action (coming this week to our Open Source). We also send this image every Monday for a historical record:

Conclusion

When it comes to FinOps – the team in charge of understanding and accounting for your cloud bill, one of the most critical tools to have is observability into your daily spend. If there is a large change – that might be ok – but it is good to know in advance, and double check that the increase is valid – thereby potentially avoiding a large bill at the end of the month.

In our previous post, we looked at all AWS products for daily cost spend, and in this post, we dug deeper to better understand our EC2 spend – the largest percentage of our AWS bill. Are you interested in trying out our alerting in unSkript? Try out our Open Source Docker build (and give us a star!), or sign up for a free cloud trial. Is there a segment of YOUR AWS bill that you’d like to investigate with us? Reach out in our Slack channel, and we’d be happy to help you create a RunBook for your use case!

Keeping your Cloud Costs in Check: Automated AWS Cost Charts and Alerting

Doug Sillars — Mon, 20 Mar 2023 16:03:51 +0000

Building and deploying infrastructure in the cloud is (by design) a very simple process. If a team is not being careful in their deployments, it can also become an expensive process. The interplay between finance teams and cloud teams has led to a new job function – FinOps. What is FinOps? This team (or team member) works with developer teams to understand cloud needs, negotiates better prices with cloud operators, and can help translate cloud expenses and needs to the finance team.

However, many companies working in the could don’t have the luxury of allocating a team member to understanding cloud costs, and it is left to the dev teams to do their best to mitigate surprise bill. Without a full-time FInOps professional, teams will need tooling to help them better understand and control their cloud bills.

The FinOps Institute has defined 6 domains for a FinOps tram:

Understanding Cloud Usage and Cost
Performance Tracking and Benchmarking
Real Time Decision Making
Cloud Rate Optimization
Cloud Usage Optimization
Organizational Alignment

In this post, we’ll describe a series of automated RunBooks that check the first three boxes (and help inform several others). What we’ll do is build automated reporting around the AWS Cost and Usage Report (CUR).

The AWS CUR

AWS’ Cost and Usage Report is a SQL report that breaks down your AWS spend in many different ways, and at a selected interval (hourly, daily, monthly). When generating a CUR report, a SQL file is placed into a S3 bucket at regular intervals. To set up a CUR report for your AWS Account, the AWS Documentation has a very nice tutorial. In this post, our CUR is updated daily, but for larger projects, you may want hourly granularity.

We set our CUR report to be sent into AWS Redshift. However, the daily updates are only added to the S3 file. To update our Redshift table, we must regularly take the file in SQL and update the database in Redshift.

We accomplish this by building a RunBook using a few new Actions:

When Making a RedShift Query, we need to know the Secret Manager ARN, the AWS Region, and the Redshift Cluster/database.

To get the Secret Arn, we use the AWS GET Secrets Manager ARNAction. This takes a secret name, and provides the ARN. (This does require Secrets Manager permission in your IAM credential.)

We then Create two SQL queries programatically. The table in Redshift is named awsbilling202303 (since it is currently March 2023). To ensure that the table name is always correct, we generate the query programmatically, so that the table name always has the format awsbilling<year><month>.

Next, we perform two SQL commands:

First, we TRUNCATE the table. This removes all of the rows, but keeps the columns:

truncate table awsbilling202303

Next, we COPY the rows from the SQL table in AWS:

copy awsbilling202303 from 's3://unskript-billing-doug/all/unskript-billing-doug/20230301-20230401/unskript-billing-doug-RedshiftManifest.json' credentials 'aws_iam_role=arn:aws:iam::<arn name>' region 'us-west-2' GZIP CSV IGNOREHEADER 1 TIMEFORMAT 'auto' manifest;

This query is provided in your S3 bucket.

With the RunBook created, we can schedule this RunBook to run daily, ensuring that the AWS table is always up to date.

NOTE: I am not a database expert. This is the “I have a hammer, so everything must be a nail” approach to updating the database. There is probably a more nuanced query that could be run.

Building Charts and Alerts

Now that the data is being populated into Redshift daily, we can begin exploring our data. In this second RunBook, we are going to extract the daily spent for each AWS service, plot the data, and create an alert for large changes in costs.

Our new RunBook begins the same way as our first RunBook – generating a SQL query and executing it at RedShift. This time we are querying Redshift for usage costs for every AWS product:

select lineitem_productcode, 
        date_part(day, cast(lineitem_usagestartdate as date)) as day, 
        SUM((lineitem_unblendedcost)::numeric(37,4)) as cost 
from awsbilling202303 
group by lineitem_productcode, day 
order by cost desc;

This query is then placed into a dataframe. Using this data, we can create a chart of our daily spend by AWS product:

Here is the same chart – just looking at the last seven days:

If we look back at the FinOps Institute bullet list, we are beginning the FinOps Institute bullets: Understanding Cloud Usage and Cost, as well as Tracking our Performance.

If we automate this RunBook to run daily (after the table is updated), we can generate this regular plot of our AWS Cloud Spend. In general, a daily chart with no change is not of great interest. But – if we can build an alert around our Cloud usage that finds significant increases in day to day cost – we can attach this chart to make the cost jump easy to identify.

Building an Alert

Every organization will have different thresholds for alerting. In the following code, we examine the 2 days previous (March 18 and 19) and look for increases of over 5%. Since many of the services in this chart are at very low spend rates, we add the additional filter that the change must be over $1. Looping over each service with the following:

if abs(todayCost-yesterdayCost) >1: 
   if delta >.05:
       #print( instance, delta,dfpivot.at[today, instance], dfpivot.at[yesterday, instance])
       bigchange[instance] = {"delta":delta, "todayCost":todayCost,"yesterdayCost":yesterdayCost}
       alertText = '@here There has been a large change in AWS Costs'
       alert = True
if date.today().weekday() == 0:
   alertText = 'Today is Monday, Here is the last week of AWS Costs'
   alert = True

If any of the changes in cost trigger this alert, we send an image on Slack with a ‘@here, there’s been a large change in AWS Spending” message. We also send the chart every day on Monday., so that there is a visual history of AWS spending.

When beginning to study and alert on your Cloud Spending it is important to start with simple reports and altering. This use case of daily spend by product, is a great start into our journey into the FinOps Institute’s next bullet points: Real Time Monitoring and Cloud Usage Optimization.

Conclusion

Many companies feel the monthly dread of “how big will my Cloud bill be this month?” By charting and alerting on your daily spend across all Cloud products, your team is less likely to be surprised by the beill at the end of the month. This data can also be used mitigate large changes that often result in bill “surprises.”

Using unSkript and the AWS Cost and Usage report, you can begin (or continue) your FinOps journey by better understanding where your costs are coming from and how they are changing day to day. Watch our blog for more posts on Cloud CostOps and how you can monitor your AWS bill.

AWS Service Quotas: Discovering Where you Stand

Doug Sillars — Fri, 17 Feb 2023 19:31:01 +0000

We’ve written a few posts in the last week about AWS Service Quotas. These are restrictions on services that are set by AWS (but can often be increased).

If our first post, we looked at New Actions in unSkript that can be used to determine quota values and request a quota increase. In this post, we’ll take the Actions a step further, and build an Action that compares the AWS quota to actual usage – generating an alert when a threshold is met.

Getting Started

To begin, we will need to consider how the Action will work. For any given service, we’ll need to query AWS at least twice:

Get the Quota Limit.
Determine the usage of a service.

Every query requires one call to complete Step 1. However, Step two can require many queries to complete the usage query. In the simplest case, we can do just one query:

Example: _Client VPN Endpoints per Region. _ If we query AWS for the list of endpoints in a region, we can simply get the length of the response to know how many endpoints exist.

However, there are times where there will be multiple queries:

Example: Routes per Client VPN Endpoint. In the first query, we get the list of VPN endpoints. In step 2, we must query every VPN endpoint to get the count of Routes. If there are 4 VPN endpoints, there will be a total of 5 calls made (On call to get the list of 4 VPN endpoints, and then one call to each of the four endpoints).

To account for these two options, we create an input Dictionary.

The Simple, one pass Dictionary

For the Describe AMIs call (only one Usage query is required), the Dict looks like this:

{'QuotaName':'AMIs','ServiceCode':'ec2','QuotaCode': 'L-B665C33B',
'ApiName': 'describe_images', 'ApiFilter' : '[]','ApiParam': 'Images',
'initialQuery': ''},

To get the Quota, we need the ServiceCode and the QuotaCode (If you need to obtain these variables, you can use the unSkript Action, or you can refer to the table in the unSkript Docs). The one usage API call will be made to the describe_images endpoint, and retrieve a list of Images. Counting this length gives us our usage.

The Two Pass Dictionary

To determine the Attachments per transit gateway, we must again get the quota from the Service Code and Quota Code. To get the count of attachments per transit gateway, we us the initalQuery array to make a first query.

The first query probes the describe_transit_gateways endpoint, to give a list of TransitGateways. _ In the second set of calls, we call the _describe_transit_gateway_attachments endpoint for each transit gateway. The filter has a string VARIABLE that is replaced with the TransitGatewayId for each gateway -ensuring that each call is made to a different transit gateway. We can then count the length of the response to find out how many attachments are in each transit gateway. If we have 12 transit gateways. we will have 12 usage reports.

{'QuotaName':'Attachments per transit gateway','ServiceCode':'ec2','QuotaCode': 'L-E0233F82',
'ApiName': 'describe_transit_gateway_attachments', 'ApiFilter' : '[{"Name": "transit-gateway-id","Values": ["VARIABLE"]}]',
'ApiParam': 'TransitGatewayAttachments',
'initialQuery': '["describe_transit_gateways","TransitGateways", "TransitGatewayId"]'},

Outliers

For most of our quota measurements, these two approaches work well. However, with over 2600 different quotas inside AWS, not all of them fit neatly into these two buckets. For example Multicast Network Interfaces per transit gateway requires 3 calls: Transit gateways -> Multicast Domains – > Domain attachments.

For others, there is custom code to iterate over. These require an extra if statement in the code to properly account for their usage.

Action Format

We can differentiate between the two types of query by looking at the ‘initialQuery’ parameter. If it is empty, we can do the Simple query, otherwise, do the double query (with a for loop that queries each initial result). For outliers, we can add specific code inside the if/else:

(this is simplified a bit from what actually runs):

for i in table: 
    #get quota 
    sq = sqClient.get_service_quota(ServiceCode=i.get('ServiceCode'),QuotaCode=i.get('QuotaCode')) 
    quotaValue =sq['Quota']['Value'] 

    #get usage     
    if i.get('initialQuery') = '': 
       res = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) 
       count = len(res) 
       percentage = count/quotaValue 
       combinedData = {'Quota Name': i.get('QuotaName'), 'Limit':quotaValue, 'used': count, 'percentage':percentage} 
       result.append( combinedData) 
       print(combinedData) else: res = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) 
       for j in res: #build the filter query with some simple substitutions res2 = aws_get_paginator(ec2Client, i.get('ApiName'), i.get('ApiParam'), Filters=filterList) count = len(res2) percentage = count/quotaValue objectResult = {j[initialQueryFilter] : count} quotaName = f"{i.get('QuotaName')} for {j[initialQueryFilter]}" combinedData = {'Quota Name': quotaName, 'Limit':quotaValue, 'used': count, 'percentage':percentage} result.append(combinedData)

Action Output

Once all of the values have been collected, the percentage utilized is compared to the warning percentage input. If the utilization is over the requested percentage, the Service data will be added to the output of the Action. With this information, the SRE responsible can decide the correct Action to take – either prune away some usage, or request an increase from AWS.

For example, testing all VPC Service quotas with a earning of 50% utilization gives the following data:

{'Instances': [{'Limit': 20.0,
                'Quota Name': 'VPCs Per Region',
                'percentage': 0.65,
                'used': 13},
               {'Limit': 20.0,
                'Quota Name': 'Internet gateways per Region',
                'percentage': 0.6,
                'used': 12},
               {'Limit': 5.0,
                'Quota Name': 'NAT gateways per Availability Zone',
                'percentage': 0.8,
                'used': 4},
               {'Limit': 50.0,
                'Quota Name': 'Routes per route table',
                'percentage': 0.5,
                'used': 25},
               {'Limit': 20.0,
                'Quota Name': 'Rules per network ACL',
                'percentage': 0.65,
                'used': 13}]}

Availability Today

As we publish this article, we have 2 Actions heading into unSkript:

A general AWS_ServiceQuota Compare Action that has the basic framework described above. This will likely require customization for each Quota you wish to test against.
AWS VPC Service Quota Warning. This Action takes all of the VPC service quotas (as of February 2023) and tests them against your infrastructure.

Coming Soon:

AWS EC2 Service Quota Warning. This Action will test your infrastructure against all EC2 Service Quotas, and warn you if you are approaching the quota threshold.

WE’re really excited to see how people use these Service Quota alerts in their infrastructure. If you have questions – feel free to reach out in our Slack Community. If you haven’t tried unSkript – try our OSS Docker Container, or use our free trial online!

AWS Service Quotas, or AWS has a LOT of Services!

Doug Sillars — Wed, 15 Feb 2023 18:03:21 +0000

In our recent post, we unveiled unSkript Actions that can query AWS Service Quotas. Service quotas are limits imposed by AWS on how many times a certain AWS feature can be used. Most of them are adjustable with a simple request, and our post showed how to determine your Service Quota values, AND request an increase using unSkript.

In this post, I thought it might be fun to dig into AWS Service Quotas a bit deeper, and get s general idea of how Service Quotas fit into the AWS landscape.

To get the Service Quota value, you need to know the Service Name, and the Quota Code. But how do you get these values?

AWS Service Names

We can get all of the AWS Service Names using the Service Node endpoint. Running this call, we find that there are 221 named services in AWS (as of Feb 15, 2023). AWS gets a lot of flak for their naming conventions, but with so many services, if course some are going to have sup-optimal names. Lucky for us, we’ll be using the ServiceCode, and not the Service Name, so “_ AWS Systems Manager Incident Manager Contacts ” is simply “ **_ssm-contacts** ” and “AWS IAM Identity Center (successor to AWS Single Sign-On)” is just “_ sso. _”

AWS Service Quotas

Next, we can run these 221 named services against the “List Service Quotas” endpoint to get all of the Service Quotas for all of the Services. Only 113 of AWS Services (51%) have features that have a service quota. Even with just half of services having quotas – there are a _ LOT _ of preset quotas in AWS: 2,629 of them in fact! (Feb 15,2023)

Service Code	Count of Quota Name
sagemaker	702
ec2	131
iotwireless	102
kinesisvideo	82
rekognition	80
personalize	66
cases	65
braket	62
elasticmapreduce	60
geo	60
comprehend	56
kms	53
lookoutmetrics	44
sns	44
logs	41
apigateway	38
iotcore	37
chime	36
forecast	33
ebs	32
glue	28
acm-pca	26
fsx	26
robomaker	24
rds	24
elasticloadbalancing	22
dataexchange	22
mediapackage	22
monitoring	22
cognito-idp	21
elasticfilesystem	21
vpc	21
fis	19
servicecatalog	17
omics	17
ssm-contacts	16
application-autoscaling	15
ssm	15
cassandra	14
frauddetector	14
imagebuilder	14
servicequotas	13
workspaces	13
textract	13
lex	13
events	12
amplify	11
appmesh	11
route53resolver	11
iottwinmaker	11
databrew	11
medialive	11
rolesanywhere	10
mgn	10
access-analyzer	9
eks	9
groundstation	9
nimble	9
dms	9
dynamodb	9
workspaces-web	8
ecr	8
ivschat	8
profile	8
batch	7
proton	7
mediastore	7
drs	6
appconfig	6
pinpoint	6
schemas	5
cloudformation	5
athena	5
m2	5
ivs	5
ec2-ipam	5
es	5
refactor-spaces	4
kafka	4
resiliencehub	4
app-integrations	4
network-firewall	4
resource-explorer-2	3
AWSCloudMap	3
qldb	3
ssm-sap	3
mediaconnect	3
auditmanager	3
sms	2
lambda	2
guardduty	2
ram	2
ses	2
autoscaling	2
fargate	2
cloudhsm	2
outposts	2
airflow	2
dlm	2
license-manager	2
ssm-guiconnect	1
connect-campaigns	1
connect	1
macie2	1
iotanalytics	1
emr-serverless	1
codeguru-profiler	1
autoscaling-plans	1
firehose	1
codeguru-reviewer	1
kinesis	1
grafana	1
ec2fastlaunch	1

As Machine Learning can be a very expensive process, it is no surprise that Amazon SageMaker leads the pack with over 700 service quotas. 2nd in line is an oldie but a goodie, Amazon’s EC2 (debuted in 2006!) with 131 quotas.

What do we know about quotas?

The longest quota name belongs to Rekognition, and it is quote a mouthful:

_ Transactions per second per account for the Amazon Rekognition Image personal protective equipment operation DetectProtectiveEquipment. _with a quota of 5. That’s a lot of words to say that the service can scan 5 frames per second to identify a helmet, face mask or gloves on anyone in the picture. This quota can be adjusted, if desired.

The largest Quota is ElasticFileSystem’s file size , weighing in at 52673613135872 bytes. (Which, if I did my math correctly, is 47.9 TB). This is a hard limit and cannot be adjusted.

The second largest quota is the Maximum number of rows in a dataset for Amazon Forecast, with a soft limit of 3 billion rows. You can request that this number be increased.

Quota adjustment

Of the 2,629 quotas, 2,003 can be adjusted (76%), and 626 (24%) cannot be changed.

Quota Units

Only 87 of our quotas have units (3.3%). 20 are time based, and the remaining 67 are data sizes (of varying magnitude):

Unit	Count
Millisecond	2
Second	18

These vary from 200 ms ( API Gateway Maximum Integration Timeout ) to 30 days: SageMaker’s Longest run time for an AutoML job from creation to termination. In case you were wondering, 30 days is also 2,592,000 seconds.)

When it comes to units, there’s nothing like arbitrarily multiplying by 1024 to change the units (and I see you MegaBits and your extra x8… but these are all throughput, so I’ll give that a pass).

Unit	Count
Bytes	11
Kilobytes	15
Megabits	4
Megabytes	13
Gigabytes	22
Terabytes	2

The smallest value is Lookout Metrics Value Length at 40 Bytes, and the largest is RDS Total storage for all DB instances at 100,000 GB (or 97 TB).

The winner for the oddest size measurement goes to Elasticfilesystem’s Throughput per NFC Client at 524.288 MegaBytes.

Summary

While looking at the giant list of AWS Service Quotas, I thought it might be fun to look at the data more closely. It remains to be seen whether the unSkript team will continue to let me use PivotTables to look at data.

More importantly, the list of Service quotas – with the Service Code and Quota Code are all in one table, and we have published the Feb 15, 2023 list in the unSkript Docs.

If you’re interested in learning more about unSkript, join our Slack Community, or check out our Open Source repo, you can run unSkript Open Source locally with Docker!

DEV Community: Doug Sillars

The Replay Bundle That Remembers What Happened

What The Control Plane Hands Off

The Replay Tool

What A Reviewer Sees First

Redacting For Export

Signing For Export Integrity

Reading The Verified File

Proving The Guardrails Held

Cognous Control Plane: The Layer That Tells Your Agent No

What Sits Beside The Agent

Propose

Decide

Document

Running The Scenario For Real

Documenting The Proposal And Decision

Reading The Results

What This Deliberately Doesn't Do

Next Up

The Manifest That Keeps Your AI Agent Honest

The Four Actions, Written Down

The Nine Action Types

The Four Review Modes

Reliance: A Third Thing Being Tracked

Validation Isn't Optional, and It Isn't Decoration

What This Deliberately Doesn't Do

Next up

Cognous Keeps Your AI in Check

What actually needs to happen

Introducing Cognous

Walking the scenario through the stack

See it yourself

When should you use canary deployments?

What is a canary deployment?

Is your team already using canary deployments?

Planning a canary deployment

Deployment risks

Key metrics during canary deployments

Canary deployments: What's next?

Database branching in Django apps using GitHub actions

Setup

Code repo

Digital Ocean

GitHub Secrets

Digital Ocean app spec files

Create NeonDB branch

Delete NeonDB branch

Conclusion

Automating Developer Relations Metrics with Low Code RunBooks

Collecting the Data

Built in Actions

Database Queries

REST API

Storing & reporting our data

Progressive enhancement

Scheduling

Summary

Automating the GitHub *Nudge*

Stale Issues

Issues without an assignee

Issues with an assignee

Stale Pull Requests

The actual “nudge”

Automating the Nudge!

Cloud Costs: Charting Daily EC2 Usage and Cost

Creating the RunBook

Conclusion

Keeping your Cloud Costs in Check: Automated AWS Cost Charts and Alerting

The AWS CUR

Building Charts and Alerts

Building an Alert

Conclusion

AWS Service Quotas: Discovering Where you Stand

Getting Started

The Simple, one pass Dictionary

The Two Pass Dictionary

Outliers

Action Format

Action Output

Availability Today

Automating the GitHub Nudge