DEV Community: Mykola Kondratiuk

Fable 5 Went Dark Friday Night. I Ran My Critical Workflow on a Backup Saturday - Here's What Broke

Mykola Kondratiuk — Mon, 15 Jun 2026 07:42:36 +0000

On Friday afternoon a government order hit Anthropic, and by Saturday morning Fable 5 and Mythos 5 were disabled for every customer worldwide. Not deprecated. Gone. Two days later OpenAI shut Sora down because it was losing fifteen million dollars a day.

I don't have a strong take on the politics. What I had was a smaller, more selfish question at 8am Saturday: if I'd staffed a real workflow on either of those, what would I actually do right now?

So I tested it. Here's what happened.

"We'd just switch" is a hope, not a plan

I'd been telling myself I had redundancy for months. If my main model fell over, I'd move to a second vendor. Easy.

The problem with that sentence is that I had never once run it. A fallback you've never executed isn't a fallback. It's a guess with good posture.

So Saturday I took my single most critical AI-dependent workflow - a spec-to-task-breakdown pipeline I lean on every day - and ran it end to end on a different vendor's model. One time. Just to find out whether the guess held.

It didn't.

Break #1: the prompt was overfit to one model

The first thing that broke was the prompt itself. My prompt had drifted into a shape that worked beautifully on the model I built it against. Tight, terse, lots of implicit structure the model had learned to fill in.

The backup model read the same prompt and produced mush. Not wrong exactly, just vague and unstructured, the kind of output you'd toss.

The fix was real work, not a config flag:

- summarize the spec and break it into tasks
+ You are breaking a spec into engineering tasks.
+ Output JSON only, matching this shape:
+ { "tasks": [{ "title": "", "estimate_pts": 0, "depends_on": [] }] }
+ Rules:
+ - every task must be independently shippable
+ - no task larger than 3 points; split if larger
+ - depends_on references task titles, not indexes

Model A filled in all that structure on its own. Model B needed it spelled out. That's twenty minutes of restructuring I'd much rather spend on a calm Saturday than during an actual outage.

Break #2: a silent tool-call dependency

The second break scared me more because it was invisible. One step in the pipeline depended on a tool call - a function the model invokes to pull live data. The backup model's tool-calling format was different enough that the call silently no-op'd.

The output still looked plausible. It just used stale data and didn't tell me. That's the worst failure mode there is: confidently wrong, no error, no flag. I only caught it because I was looking for trouble. On a normal day that bad output flows downstream and someone makes a decision on it.

Availability belongs on the risk register

Here's the reframe I walked away with. We already handle the API being down. You get a 503, you back off, you retry, it comes back. That's an outage with an SLA and a status page that eventually goes green.

This is the model being gone. No SLA. No restore ETA. No green status page, because it isn't coming back. A policy order or a vendor's burn-rate review can end it overnight, and you find out the same way everyone else does.

For a service you don't control and can't restore, that's a single point of failure on your critical path. We'd never ship that for a database. Most of us are shipping it for the model doing half the thinking.

The one-pager that deletes your worst hour

The cheapest move turned out to be the most useful. The first hour after a model goes dark gets burned figuring out what just broke - which workflows touched that model, what versions, where the outputs live.

IBM found 88% of enterprises don't keep a complete inventory of the AI and agents they run. You can't reroute around a dead model if you don't know what depended on it. So I wrote one file:

workflows:
  - name: spec-to-tasks
    model: primary-vendor/model-a
    criticality: must-survive
    fallback: tested 2026-06-13, prompt needs restructure
  - name: standup-digest
    model: primary-vendor/model-a
    criticality: can-wait
    fallback: none, recovery order documented
  - name: video-assets
    model: openai/sora
    criticality: can-wait
    export_path: download MP4s + project json before EOL

That last line is the Sora lesson. When a vendor kills a product, not just a model, you also have to ask where your outputs go and how you get them out. One extra column.

The point isn't fear

I want to be clear, because the lazy version of this post is "AI is unreliable, panic." It isn't, and that's not useful. Depending on these models is the right call. The teams that win aren't the ones who avoided the dependency. They're the ones who can keep the work moving the morning it disappears.

That competence costs an afternoon to build and almost nobody has built it yet:

Run your most critical workflow on a second model once. The rehearsal is the whole instrument.
Sort workflows into must-survive-today vs can-wait. Only the short list earns a tested fallback.
Keep a one-page workflow-to-model list so the first lost hour becomes a glance.

I ran my test on a quiet Saturday and it cost me twenty minutes and a little ego. The alternative was running it for the first time on the morning it counted.

What would break first in your stack if your main model wasn't there tomorrow - and have you ever actually checked?

I Lead AI Agents Every Day - Here Are 5 Shifts No Standard Tells You How to Make

Mykola Kondratiuk — Fri, 12 Jun 2026 07:09:26 +0000

A Google DeepMind safety lead said this week that they're putting $10M behind multi-agent safety because "there just isn't really a field of research for multi-agent safety yet."

I read that and laughed, because I'm already running the thing the research field doesn't exist for yet. Most of us are. You spin up a couple of agents, hand them work, and somewhere in there you quietly become a manager of workers that don't think like workers.

Two days before that, PMI published the first official standard for AI in project work. It's a solid document. It also leaves the entire "how do you actually do this on a Tuesday" layer to you. So here's my Tuesday layer: five shifts I had to make, each one learned by getting it wrong first.

You stop filling the queue and start drawing the line

My first instinct with an agent was the same as with a person: here's work, go.

That broke the first time an agent made a reasonable decision on something that turned out to be irreversible. It wasn't the agent's fault. I never told it which decisions were one-way doors.

So now the first artifact I write isn't a task list. It's a boundary file. Something like this lives next to the work:

# decision-boundaries.yml
autonomous:
  - reformat, refactor, rename within a module
  - anything reversible with a git revert
escalate:
  - schema changes, public API shape
  - deletes, migrations, anything touching prod data
  - spend over $0 or any external send
on_unsure: stop_and_ask

That file does more for me than any standup. Leadership moved from assigning the work to defining what may be decided without me.

You read work you never watched happen

I used to review work I'd seen get built. I knew the steps, so "looks right" was usually safe.

Then I started getting finished diffs with no memory of how they came to be. "Looks right" stopped being safe. The code was clean and the reasoning under it was wrong in a way you only catch if you go digging.

The skill now is judging a result cold, with zero context on the path. Ethan Mollick wrote this week about a model holding twelve hours of focus on one spec. When the attention window outlasts mine, my job isn't checking steps. It's scoping the spec so tightly the steps don't need a babysitter.

You plan capability, not headcount

"How many engineers do I need" is a question I catch myself asking and kill.

The real one: what mix of people and agents produces this outcome, and what's the human-only core I'd never hand off? The plan turned into a capability map with a deliberately protected center.

Gergely Orosz's June job-market analysis lands in the same place from the data side: the roles that compound are where judgment about AI systems is the scarce input, not execution on a known stack. Capability planning is that judgment pointed at your own team.

You design the alarm before the fire

Standup tells you something broke. Which means it tells you late.

Workers that fail unpredictably need the alarm built up front. I keep a short tripwire list, each one a single sentence: if this observable crosses this line, halt and ping me, and here's who owns the ping.

# tripwires.yml
- watch: test_pass_rate
  trip: "< 100% on touched files"
  action: halt + page me
- watch: files_changed
  trip: "> 20 in one task"
  action: pause for scope review

It feels too simple to matter. It has saved more bad mornings than any dashboard I've built.

You own the system, not the deliverable

This is the one that's actually a promotion.

Ownership used to mean the outcome is mine. It still is. The level changed. I don't own the deliverable directly anymore. I own the system that makes it: people, agents, and the rules between them. That's the only level that scales.

Boris Cherny, who runs Claude Code, said this week he hasn't written a line of code himself in eight months. People hear a flex. I hear the shift in one sentence: stopped producing the work, started owning the system that produces it. Bigger job, not a smaller one.

Where are you on these

I'm not clean on all five. Solid on three, shaky on two, and the shaky ones cost me the most.

Rate yourself one to five on each, fast. The two you score lowest are the two behaviors that move you this quarter. Which one did you make first, and which are you still avoiding?

Tags: #projectmanagement #ai #career

I Took the Keyboard Back From an Agent Mid-Task - Here's What the New PMP Can't Test

Mykola Kondratiuk — Fri, 05 Jun 2026 06:22:24 +0000

A few weeks back I had an agent reconciling a vendor list. It ran clean. No error, no crash, output looked right. Then I noticed it had merged two suppliers that share a parent company into a single row, which would have thrown off every spend rollup downstream by a real number.

I stopped it and fixed it by hand. Not because anything alerted me. Because I'd been burned by that exact shape before, and the burn taught me something a tutorial never did.

I'm telling you this because on July 9 the PMP changes, and the change points straight at that moment.

What changed in the exam

For the first time, the PMP makes AI mandatory content instead of an elective. The Business Environment domain goes from 8% of the exam to 26%. PMBOK 8 becomes the base. Fees climb in August.

The largest project-management body on earth just put it in writing: AI fluency is core PM competency now, not a specialty track. If you've been treating "PM + AI" as a buzzword, the institution that certifies the role just disagreed with you.

I think that's the right move. It's also where it gets interesting for anyone who actually ships work through agents.

Certified is not practiced

A multiple-choice exam can certify that you know what an agentic workflow is. It can check that you'll pick the textbook answer about AI risk, define non-determinism, name the correct oversight principle.

That's awareness, and awareness is worth certifying.

What it can't reach is the reflex that runs real work. The exam can't certify that you've handed live stakes to an agent, watched it drift, and built the instinct for when it can act versus when you take over. You don't recall that instinct. You earn it, and the only way to earn it is to need it.

I learned the vendor-merge thing from the run where I caught it too late.

What AI fluency looks like in the editor, not the exam

Let me put it in do-this terms. Here's the practiced version, and none of it is a question you can bubble in.

You scope the agent's slice like a statement of work. Not "improve onboarding." Bounded edges, in and out defined before it touches anything:

task: reconcile vendor list for Q2
in_scope:
  - dedupe exact-match names
out_of_scope:
  - merging distinct legal entities   # <- the line that would have saved me
acceptance:
  - row count change is human-reviewed before rollup runs

You know the override moment. This one I'd put first. You only learn the veto by having needed it.

You read the output for what it didn't do, not just whether what's there is correct. The skipped supplier, the untouched edge case.

You design the work so a human can actually check it. If the only way to verify is to redo the whole thing, the slice was wrong.

You size the blast radius before deploy. How wrong can this go, who feels it, answered up front the way you'd treat a change to a live service.

Five reps. All earned. Zero on the exam.

Why this is a level-up, not a layoff

The panic read of this is backwards. The credential catching up doesn't shrink the role, it raises the floor. When the institution names AI fluency as baseline, the person who's practiced it instead of read about it becomes the scarce one. Certified-and-practiced is a much smaller set than certified.

So I wouldn't study the new section and stop. I'd go get the reps it's gesturing at. Hand something real to an agent this week, let it run a little past comfortable, and watch the moment your hand goes for the keyboard. That moment is the competency. It never shows up on a scorecard.

For those of you shipping work through agents already: what's the moment that taught you to override, and could a test have taught it instead?

Tags: #projectmanagement

I Sorted 40 Backlog Items by Shape Instead of Who's Free - Here's What Broke

Mykola Kondratiuk — Wed, 03 Jun 2026 07:29:49 +0000

canonical_url:

Two Fortune 500 execs stood on a summit stage last week and gave opposite answers to one question: is an AI agent a colleague or a tool?

One names his agents and seats them in reviews. The other won't call them colleagues. I watched the debate go by and realized I'd stopped caring about the noun a while ago, because it never once changed what I did with my backlog on a Monday.

So here's the thing I actually changed, and the part of it that broke.

The old default: sort by who's free

For years my planning loop was the same. Look at what needs doing. Look at who has capacity. Assign. The unit was always the person. "Who's free" was the first question.

When agents showed up in the loop, I just slotted them in as another row in the capacity table. Same question, one more name. That's the "tool" answer in practice - an agent is a faster hand, you point it at whatever's next.

It worked until it didn't. The agent would happily take a task that needed a human to even scope it correctly, produce something plausible, and I'd find out two steps later that the plausible thing was wrong in a way that cost me a half-day to unwind.

The change: sort by shape first

I flipped the order. Before I look at who's free, I sort the backlog by shape.

Two buckets. Person-shaped work needs judgment, lives in ambiguity, depends on taste or a relationship I can't write down. Agent-shaped work is defined, repeatable, and gate-able - I can describe the input, the output, and the check it has to pass.

The discipline is that I write the agent's slice like a contractor's statement of work, not a chat prompt:

# agent slice — scoped like an SOW, not a vibe
task: regenerate API client from updated OpenAPI spec
input:
  - openapi.yaml (v3.1, committed)
  - existing client at src/clients/
output:
  - regenerated client, same public surface
gate:
  - all existing contract tests pass
  - public exports diff reviewed by a human before merge
owner_of_outcome: me   # not the agent

If I can't fill in gate and output cleanly, that's my signal: this isn't agent-shaped yet. It's person-shaped work I was about to mislabel because I wanted it off my plate.

I ran this across about 40 backlog items. Roughly a third of what I'd have handed to an agent under the old "who's free" sort failed the shape test and went back to a human.

What broke

Two things broke, and both were useful.

First, my sense of which work was "important." I'd been guarding a pile of tasks as too critical to automate. Half of them were just defined work I was emotionally attached to. They sorted cleanly into the agent bucket and ran fine. The status I'd assigned them was about me, not the work.

Second - and this is the one I underscoped - the gate is only as good as the human watching it. There's research showing that once you treat an agent like a teammate, you review its output less carefully. Naming the thing lowers your guard. So the "colleague" framing isn't warmth, it's a quiet accountability leak. I caught myself rubber-stamping a passing gate because the agent had "been reliable lately." A tool doesn't earn trust. Work does, run by run.

There's an old line going around that fits: you automate the boring, and then humans just manage the failure modes. That's not a downgrade of the human job. The failure modes ARE the job now. Managing the part the agent can't hold is the work that's left, and it's the work that compounds.

Why this is a career thing, not just a workflow thing

The same exec who named his agents also said the hardest part isn't the technology, it's the managers. I think that's exactly right, and it cuts both ways.

If the manager is the bottleneck, the manager is also the leverage. The skill that pays off from here isn't prompt-writing - that floor keeps dropping. It's the older skill of looking at a pile of work and knowing fast which parts are person-shaped and which you can scope, gate, and let run. That's "golden age of the generalist" energy: when the specialization tax drops, the person who holds judgment AND can design the work is the one who wins.

The execs on stage were missing a practitioner, so they argued about the label. The label was never the leadership problem. The work was.

So I'm curious where you've landed: when an agent is in your loop, do you still plan by who's free, or have you started cutting the work by shape first? And where has the gate let something through on you?

I Added a Human Veto to My PM Agent — Here's What Broke First

Mykola Kondratiuk — Fri, 29 May 2026 19:46:10 +0000

Running automation agents for a while now. Most work fine hands-off. But one of them - my project status agent - kept making decisions that felt right in isolation but wrong in context.

So I added a human approval step. Not a "review and confirm" UI widget. An actual veto gate in the workflow itself: the agent drafts the action, pauses, and waits for my explicit go-ahead before doing anything irreversible.

Here's what I didn't expect: the first thing to break wasn't the agent logic. It was my own habits.

The Problem I Was Trying to Solve

My status reporting agent does three things: pulls data from Jira, formats a weekly PM summary, and posts it to the team Slack channel. Straightforward automation.

Except twice in three months it posted something embarrassing. Once it included a stale blocker that had been resolved 48 hours prior. Once it flagged a team member's ticket as overdue when they'd actually shipped early and the tracker hadn't caught up.

Neither was catastrophic. Both were awkward.

The traditional fix would be "add better logic to catch these cases." I tried that. Added freshness checks, added resolved-status validation. Still leaked edge cases.

So I took a different approach: make human review a structural part of the workflow, not a safety net I bolt on when things go wrong.

What I Actually Built

The architecture is boring. Agent generates the Slack message draft, posts to a private review channel, waits for a thumbs-up emoji reaction, only then posts to the team channel.

If no reaction within 2 hours, it pings me directly and kills the task. No silent failures.

The human approval isn't optional and it isn't a fallback. It's a required step in the sequence. The workflow can't progress without it.

I got the idea from reading about Microsoft Conductor, which open-sources a similar pattern for multi-agent orchestration. Human approval as a default workflow step, not a retrofit. Their framing stuck with me: designed-in, not bolted-on.

What Actually Broke

I expected the agent to break. It didn't.

I expected me to approve everything in under 5 minutes. I did, mostly.

What I didn't expect: I stopped trusting my own review. The first week, I read every draft carefully. By week three, I was rubber-stamping. My brain had offloaded judgment to "well the agent probably got it right." The approval gate existed, but the actual human review stopped happening.

This is the invisible failure mode nobody talks about. You add a human step. The human shows up. But they're not really there.

The fix was embarrassingly simple: I added friction. Required a comment, not just an emoji. Had to type at least one word before the workflow could advance. Stupid? Maybe. Effective? Completely.

Turns out the veto gate only works if the human has to engage to use it.

Three Things I Didn't Know I Needed

1. Escalation hooks, not just approval gates.

Not all decisions are equal. Minor formatting choices don't need a veto. Anything that posts externally or modifies data does. I ended up building a simple severity classifier: low = auto-approve, medium = soft review prompt, high = hard gate. Saved probably 70% of the friction without sacrificing coverage where it mattered.

2. A timeout that fails loudly.

My 2-hour window was too long. If I'm in back-to-back meetings, the agent just hung. Switched to 30 minutes with an escalating Slack ping. Now if I haven't approved it, I can't miss it.

3. A clear distinction between "irreversible" and "annoying-to-undo."

I started gating everything. Caught myself adding a veto to the agent that sends me my own daily brief. Nobody else sees that. No irreversible action involved. Human gate added zero value there, only friction.

The useful mental model: if I had to undo this action at 11pm on a Friday, would I care? Yes = human gate. No = let it run.

Why This Matters More Than It Looks

Most of the "AI safety" conversation in enterprise is about governance frameworks and audit trails. That's real. But the practical engineering question is simpler:

Which steps in your agent workflow require a human in the loop by design, not by accident?

Design it from the start and the workflow is reliable. Bolt it on after an incident and you're playing catch-up forever.

The Microsoft Conductor open-source was notable to me not for the code but for the default: human approval ON unless you opt out. Most agent frameworks do the opposite. They default to autonomous and assume you'll add guardrails when you need them.

I think that's backwards. Especially for agents touching anything external: posting, sending, modifying.

Where I'm Landing

The veto gate has been running about 6 weeks now. Two incidents caught before they shipped. One case where my review actually improved the draft - not just filtered a bad one. About 3 minutes of daily overhead.

Worth it. But only because I designed the friction deliberately. The version without the comment requirement was almost worse than no gate at all - it gave me false confidence in an approval that wasn't really happening.

If you're building agent workflows that touch anything irreversible, the question I'd ask first: what happens if this runs at 2am and you're asleep? Whatever you wouldn't want to explain the next morning - that's where your human gate goes.

If any of this maps to workflows you're building, curious what the "irreversible action" problem looks like on your end.

I Read Intuit's 3,000-Job Layoff Memo - Here's the One Line Every AI Restructuring Memo Is Missing

Mykola Kondratiuk — Thu, 21 May 2026 06:55:22 +0000

canonical_url:

On Tuesday, May 20, Intuit's CEO Sasan Goodarzi sent a memo announcing 3,000 jobs cut. 17% of the company. Reason: "reduce complexity to focus on AI." I read it twice looking for one line. It wasn't there.

The same line has been missing from every AI workforce announcement I have read in the last two years. I want to name it, because the missing line is an engineering-accountability problem before it is a PM-leadership problem, and the people closest to the failure surface (you, reading this on Dev.to) are the ones who feel the gap first.

The Memo Names Three Things. It Skips One.

Every AI restructuring announcement in the last two years has the same shape.

What got cut. Intuit: 3,000 jobs, 17% of workforce. Klarna 2024: customer support function. Duolingo 2025: a chunk of contractor work. IBM 2025: a large internal reorg. Always specific. Always quantified.

What gets refocused. Intuit: "focus on AI." Klarna: AI-first customer service. Duolingo: AI-augmented learning. IBM: "augmenting HR with AI." Always a direction.

What stays. Intuit named existing partnerships with OpenAI and Anthropic. Every memo names what stays. It is the reassurance paragraph.

What's missing. A real human name attached to the failure path of any specific AI system the cut workers used to operate. "When this agent gets it wrong, [name] answers." That line. Zero of the memos name it.

It is the same shape. Cut, refocus, stay, no failure owner. Once you read four in a row, you stop reading four announcements and start reading one announcement repeated four times.

What Actually Got Swapped

Underneath every one of these memos is a structural swap nobody is naming.

The financial side moves cleanly. Payroll down. Software spend up. The controller closes the quarter. The general ledger has every entry it needs.

The accountability side does not move at all. The worker who answered "I made that call, here's why" is gone. The AI that makes the call now has nobody attached to it.

The headcount got swapped for AI capacity. The accountability the headcount carried did not get re-assigned. By default, accountability becomes nobody's job. Not by malice. Not by oversight. Just by the structural shape of a memo that names everything except the failure owner.

This is the workforce-to-accountability swap. It is the unstaffed accountability sitting in every AI restructuring announcement, including the next one your company sends.

Why This Is An Engineering Problem First

Most of the layoff coverage is framed at the executive layer - the CEO's memo, the press release, the analyst reaction. The frame I want here is the practitioner one.

The agent runs in your repo. The pipeline that calls the agent is wired to production. The customer-facing surface the agent writes into is wired to actual humans on the other end. When the agent is wrong, the wrongness lands in code you maintain - a misrouted refund, a wrong tax calculation, a wrong customer email, a wrong policy interpretation surfaced through a chat UI you shipped.

You are closest to the failure surface. You feel the missing line first. The PM who is supposed to author the line is downstream of the deployment by definition. By the time the line is missing in production, you have probably already noticed.

That is why this is worth flagging as an engineer-reader: the missing line is not a future PM artifact, it is a present-tense gap on workflows you are already running.

The Sharper Edge - Voice Agents This Week

Three different vendors shipped voice-capable AI agents in the last week. PollyReach added live voice on a CRM agent. AdaptiveAI shipped triggered outbound phone agents. Crescendo's Live CX agents take full inbound calls.

These agents represent the company in a verbal conversation. The voice is the company's. The commitments the agent makes - "I'll refund that", "I'll escalate this", "I'll send you the contract by Friday" - bind the company in the customer's experience.

No named human is on the call. The agent uptime is 99.98%. The customer is on the line. The named human at the company is not in the conversation, and after the conversation, is not in the transcript.

99.98% uptime is the SRE side of this. Accountability is the other side. Reliability ≠ accountability. You can have a perfectly uptime-correct agent and still have nobody on the line when its output causes harm.

The Engineering+PM Move

The move is small enough to do this afternoon, on one workflow.

Pick one agent or AI-replaced workflow your team runs. One. The one most likely to be wrong about something that matters.

Add a one-line block to the agent's spec (AGENTS.md, prompt header, deployment doc - wherever the agent's contract lives):

# Failure Owner
When this agent's output causes harm, [name] answers.
Signed: [name], [date].

Commit it. PR it. Tag the named person on the PR.

That is the inheritance move. It is the human-side complement to the version control, the test suite, and the SLA. Three accountability artifacts on the technical side (git blame, test owners, on-call rotation) and zero on the AI output side is the current default. Adding the line fixes one of them.

If you are the PM on the team and the engineering side is doing this before you do, you are watching a role-growth opportunity walk by. If you are the engineer on the team and you are already maintaining the agent's spec, the line is yours to write today, and the PM on your team will copy it onto the next four workflows.

The Pattern Will Keep Repeating

The next AI restructuring memo will come this quarter. Probably from a Fortune 500 SaaS company. Probably citing "complexity reduction" or "AI focus." Probably 1,000 to 10,000 jobs. Read it the way I read Intuit's. You will find the cut. You will find the refocus. You will find the stay. You will not find the line.

The line has to be authored. Someone has to sign it. The earliest signer on any given workflow is the person closest to it - which, on most AI-augmented teams in 2026, is an engineer plus a PM, not a CEO writing a memo.

Three thousand jobs at Intuit. Zero named humans in the announcement. The missing line is the artifact. The name on the line is the move.

What's missing from the last AI announcement your company sent - the cut, the refocus, the stay, or the failure owner?

I Read the Devenex Launch Yesterday - Here's the Policy File Your Agent Repo Is Still Missing

Mykola Kondratiuk — Wed, 20 May 2026 07:30:41 +0000

I spent an hour reading the Devenex launch yesterday and the only sentence I keep coming back to is "execution control plane." That phrase is doing a lot of work.

It says: enforcement is a product now. Every agent request gets policy-evaluated, identity-bound, recorded as evidence before anything runs.

It does not say: the policy itself exists.

Six products ship enforcement. Zero ship the policy.

Look at what shipped this month. Devenex launched May 19 as the first execution control plane. Antigravity 2.0 hardened Git policies at Google I/O Day 2. Notion's External Agent API went GA with workspace-scoped guardrails. Claude has had tool-use limits since launch. OpenAI has function-call constraints. Salesforce Agentforce has action approvals.

Six products. Different vendors. Different layers. All shipping enforcement.

The artifact they all need to enforce against is the same shape. None of them ship it. That artifact is your problem, and it lives in your repo, not theirs.

I started calling it the policy file.

What goes in the policy file

Four sections. I've been writing it this way for a while; the launches this week made me realize it's the same shape across every enforcement product I read the docs for. The shape doesn't depend on the vendor.

Action classes

The agent's universe of possible actions, broken into named classes: read, write, send-external, transact, escalate, spawn-subagent. Each class is a category the policy file attaches constraints to. The act of writing the list is the point. The default in every deployment doc I've seen is implicit: the agent can do anything inside its tool set. Naming classes is how you refuse that default.

A sketch in YAML:

action_classes:
  read:
    sources: [crm.contacts, crm.opportunities]
  write:
    targets: [crm.opportunities.notes]
  send_external:
    channels: [email, slack-dm]
  transact:
    instruments: [stripe.refund]

That's not a real schema. It's the shape your real schema settles into after the third review.

Blast radius caps

A number per class. Not a vague guardrail, a number the enforcement layer can compare against at request time.

caps:
  write.records_per_run: 50
  send_external.recipients_per_session: 10
  transact.usd_per_run: 500
  spawn_subagent.depth: 2

The contrast: the deployment doc says "the agent has access to the CRM." The policy file says "the agent's write class is capped at fifty records per run." One sentence Devenex can check. One sentence Antigravity can check. One sentence Claude tool-use can check.

Escalation triggers

The inverse half of the allowlist. When the agent hits a class not in its policy, or a cap it's about to exceed, what fires? Named human. Named channel. Named SLA.

escalation:
  - class: write
    trigger: cap_exceeded
    route: "#agent-ops"
    owner: "@owner-of-record"
    sla_hours: 4
  - class: transact
    trigger: any
    route: "#finance-approvals"
    owner: "@treasury-lead"
    sla_hours: 1

The deployment doc has "agent owner" once on page one. The policy file has an escalation route per class.

Evidence schema

What the agent has to log so a human can audit the run afterward. Structured output. The action class invoked. The tool calls. The identity the agent acted as. The policy version. The escalation path if any.

evidence:
  required_fields:
    - run_id
    - policy_version
    - action_class
    - tool_calls
    - acting_identity
    - escalation_record
  format: jsonl
  retention_days: 365

Without an evidence schema, you can't answer "did the agent follow the policy?" after the fact. The policy was unenforceable from the start.

A specific moment that made this concrete

I was reading through a deployment doc for an agent recently. Clean prose. Listed the APIs. Listed the data sources. Useful agent.

No section for what happens when it tries to write five thousand records. No section for what happens when it tries to send to two hundred recipients. No section for what happens when it transacts above a cap, because nobody had written the cap.

The deployment doc wasn't wrong. It was answering the wrong question. It answered "what does the agent do?" The policy file answers "what is the agent allowed to do, and what fires if any of that breaks?"

Different artifact. Different reviewer. Different file.

The clean split: enforcement vs. authoring

Devenex et al. ship enforcement. That half is done. The other half - authoring - isn't a product, and I don't think it can be one. Authoring is the codification of your team's actual judgment about what the agent should be allowed to do. That judgment is cross-functional: engineering knows the runtime, security knows the threat model, legal knows the constraint, finance knows the cap.

It's not "PM lobs a doc over the wall." The PM convenes the call, drafts the file, opens the PR. Engineering reviews it the same way it reviews a Terraform plan. Security reviews it the same way it reviews IAM. The policy ships in the same PR as the agent.

That's policy-as-code, the shape devs already know from infra. The new thing isn't the shape; it's the artifact existing for AI agents at all.

What I'd do this week if I were shipping an agent

Open a policy.yaml in the agent repo. Stub the four sections. Pin one number per class even if it's a wild guess. Wire the evidence schema into the agent's logging path. Put it in the same PR as the next prompt change.

The enforcement layer your platform vendor ships is checking against something. If nobody wrote the something, the enforcement is checking against silence.

What's the section your agent repo is missing first - blast radius caps, or the evidence schema?

I Built a 5-Signal Vendor Watchlist for Google I/O 2026 - Here's What Each One Will Break in My Stack

Mykola Kondratiuk — Fri, 15 May 2026 05:15:33 +0000

canonical_url:

Google I/O 2026 is four days out. If you ship anything that touches a Google model, the keynote is about to change capability assumptions your team never approved against. I have done this dance for six straight years of keynotes, and the pattern is always the same: an engineer pastes a Verge link into Slack on Wednesday, someone says "we should look at this", and three weeks later a capability we never reviewed quietly shipped inside a tool we already authorized.

This year I tried something different. Friday morning I sat down with a coffee and wrote a five-line watchlist for the keynote - vendor signals plus one specific engineering action per signal. The exercise was small enough to feel silly. Then I realized the format reuses for every keynote that follows. AWS re:Invent. Microsoft Build. OpenAI DevDay.

Sharing it here because if you have not built one yet, the next four days are your peak window.

What vendor product observability actually means

Most engineering teams I work with have great downstream observability. Uptime, error rates, model latency, queue depth. The ops team set it up. Dashboards exist.

Ask the same teams what they observe about their vendors and the answer goes quiet. The signals that reshape what your stack can do - a keynote, a model release, a quietly retired chatbot, a market position shift - arrive through press coverage, not through a dashboard. There is no Datadog for vendor product strategy. The closest substitute is a small, named list of signals you commit to reading on the day of the keynote.

That list is the watchlist. It is not glamorous. It is a 5-row markdown file. The discipline is that it exists.

Signal 1: OS-layer ambient intelligence as a new delivery category

The Android Show I/O Edition aired last week. Gemini Intelligence ships in June as the intelligence layer running below the Android app surface - booking, browsing, form completion - with access to email, calendar, messages. Rolling out as an OS update to any Android 12+ device on the AI Pro or Ultra tier.

If your team builds on Android, this matters because most authorization patterns cover two delivery categories: agents your team deployed, and agents a vendor embedded in a tool your team authorized. OS-layer intelligence is a third category, and it does not arrive through your authorization pipeline. It arrives through the OS update channel.

Engineering action at I/O: Pull the list of Android-deployed devices in your fleet that intersect "AI Pro or Ultra subscriber" and "Android 12+". That intersection is the scope of devices where the OS now has agent capabilities your data flow assumptions never accounted for.

Signal 2: Model capability delta as silent capability inheritance

When a frontier model jumps capability, every downstream tool that runs on it inherits the new floor on keynote day. If you have authorized AI tools whose vendor swapped in a Google model, those tools now have capabilities you never approved against, and you may not even know which ones.

Engineering action at I/O: Make a one-column list of authorized AI tools your team uses that run on Google models. After the keynote, write the capability delta beside each one. The delta column is the trigger for re-review. If the delta crosses a sensitivity threshold (PII handling, code execution, autonomous file I/O), the tool needs a fresh approval pass.

I tried this exercise last month with the Anthropic Opus 4.7 200k context window release. The finding surprised me: capping context at 200k produced better spec quality on agent reviews than running uncapped. Context window length is a quality lever, not just a capacity dial. The delta is not always "more is more."

Signal 3: A2A protocol updates as a multi-agent perimeter change

Google has been moving toward agent interoperability standards for two quarters. Any A2A protocol announcement at I/O reshapes the question of which authorized agents are allowed to talk to which other agents.

This is the question most engineering teams treat as edge case until production forces it. Agents are still mostly designed as endpoints with their own auth. The graph view - what agent A is permitted to call agent B for, under what conditions, with what data - is rarely written down before a multi-agent stack hits real users.

Engineering action at I/O: If an A2A spec or protocol lands, write the team's permission policy for agent-to-agent communication on the same day. One page. It does not have to be production-ready. It has to exist before the protocol shows up in the SDK you already use.

Signal 4: Vendor market position drift

Axios reported on May 13 that Anthropic has overtaken OpenAI in workplace adoption. Take a moment with that. If your team's vendor selection rationale was written when OpenAI was the dominant workplace AI, your approval rationale references a market position that is no longer current.

Most approval artifacts have no field for "market position when approved" and no scheduled review trigger for "market position has shifted since." The approval ages silently. Two years from now somebody will ask why the team is on the second-place vendor and the honest answer will be "nobody asked us to update the rationale when the market moved."

Engineering action at I/O: Add one field to your vendor approval template - Market position when approved. The field is the trigger. The next position shift now has a review path baked in. This costs five minutes and pays for itself the first time a budget review asks the question.

Signal 5: Vendor product retirement as an agent migration event

Amazon announced this week it is retiring Rufus and replacing it with the Alexa shopping agent. This is the rehearsal for the next ten retirements. Every enterprise AI vendor will eventually retire the chatbot you authorized and ship an agent successor with a different operating envelope.

Most engineering teams treat the successor as a drop-in. It is not. The operating envelope is different. The data flow is different. The failure mode is different. If your team authorized the predecessor, the successor needs a fresh review - not a carryover.

Engineering action at I/O: Watch session notes for "deprecated", "retired", "replaced by" language. Every retirement of a product your team authorized triggers a fresh review pass on the successor. Block the calendar before the announcement, not after.

The reusable shape

The five signals are not the point. The point is that writing them down on Friday changes how the keynote reads on Tuesday. The keynote stops being news and becomes a scheduled review trigger with a defined output - an updated stack policy.

The same five categories apply to AWS re:Invent, MSFT Build, OpenAI DevDay. OS-layer delivery. Model capability delta. A2A perimeter. Vendor position drift. Product retirement. Five rows. One engineering action per row. Re-read four times a year.

The thirty thousand Claude-certified consultants PwC just announced make the point sharper, not weaker. Every keynote lands against a workforce that already has the tooling. The watchlist is not optional anymore.

Question

If you maintain a watchlist for major vendor keynotes - or have ever wished you had one - what signal would I add as the sixth row? I want to compile what you all post into a single follow-up before the keynote.

Tags: #ai #googleio #productivity #discuss

I Graded My Agent Deployment Doc Against LangChain Interrupt - Here Are the 5 Gaps

Mykola Kondratiuk — Wed, 13 May 2026 07:34:10 +0000

canonical_url:

I do a lot of deployment authorization docs. That's the PM version of what an SRE would call a launch checklist. It lists the agent, the scopes, the secrets it touches, the kill switch, the cost ceiling, the rollback path.

For the last two years, the audience for that doc has been exactly one team: ours. Security signs it. Compliance reviews it. Engineering builds against it. Nobody outside the company ever reads a line.

Today I pulled up the LangChain Interrupt Day 1 schedule and that audience doubled.

The thing that changed at 9:30 PT today

Harrison Chase keynoted Interrupt at 9:30 Pacific. The headline was tame on paper: a synthesis of what 1,000+ teams shipped in production over the past 12 months. The substance was less tame. Clay, Rippling, Workday, plus the long tail of teams running smaller agent fleets, surfaced concrete production patterns. The talk wasn't aspirational. It read like a postmortem of an entire industry's first serious year of agent deployment.

That synthesis is now in public. Same week, SAP Sapphire closed with 200+ agents under a single stated design rule (governance first), with Claude as the reasoning layer and NVIDIA OpenShell as the execution wrapper. Two completely different sources. Same structural artifact: a public reference for what production-ready agent deployment looks like.

My internal doc has been graded against one rubric. As of today, it gets a second one whether I asked for it or not.

I sat down and graded mine

I gave myself 45 minutes. Three columns:

Pattern named in the public production literature this week
Our position: adopted, diverged, or gap
One sentence on why

I expected to find maybe 1 gap, possibly 2. I found 5.

Here they are, with what I'm doing about each one.

Gap 1: per-action cost ceiling, not per-month budget

Our cost guardrail was a monthly budget per agent. Easy to set, easy to forget, fires the alert about a week after damage is done.

The production pattern that kept surfacing in the public synthesis is per-action ceiling with auto-pause. If a single tool invocation projects to cost more than $N (or more than $N over the rolling 60 seconds), the agent stops and pings a human.

Our fix is small: a wrapper around the LLM client that estimates token cost in advance, compares against a per-action ceiling defined in the agent's config, and routes to a pause queue when over. About 40 lines. The harder part was deciding the ceiling, which is now a PM call, not an SRE call.

Gap 2: scoped credentials per agent, not per agent family

We had one service account per agent family (e.g., "research agents", "support agents"). Audit logs showed activity by family, not by individual agent. Fine for accounting. Bad for blast-radius reasoning.

The production pattern is one credential per logical agent instance, with the scope narrowed to the specific tables, endpoints, or namespaces that agent legitimately touches. If a single agent goes off, you revoke one credential without taking down the family.

This is a one-day migration in our system because the agent identity already exists in our config. We just weren't projecting it down into the credential layer.

Gap 3: the production review document does not yet exist

This one's about the artifact, not the runtime. Our internal review covers our policy: does this satisfy our compliance posture? It does not cover the production floor reading: where do we sit relative to what 1,000+ teams already found works?

I'm adding a new section to every deployment doc going forward. Three subheads:

Adopted: patterns we took straight from the public practitioner record
Diverged: patterns we considered and chose against, with the reason
Gaps: patterns we don't have an answer for yet

The "Gaps" subhead is the high-leverage one. Gaps documented in your own voice are gaps you control the conversation about. Gaps surfaced by a stakeholder in a meeting are not.

Gap 4: agent-writes-its-own-retries is a flagged pattern

We let one agent class write its own retry policy when a tool call failed. It's an obvious productivity win until the agent invents a retry pattern that compounds against a rate-limited downstream service.

The published practitioner consensus this week was clear: retries belong in the agent harness, not in the agent's reasoning loop. The agent should not be the entity deciding when to try again.

Our fix is to replace the self-retry behavior with a queued reissue: the harness owns the policy, the agent owns the request. About a day's work, including writing the test cases. Most of the time was migrating the existing retry-policy state out of the prompt and into config.

Gap 5: blast radius is named in our spec, but not in our review template

Anthropic's Deputy CISO ran a webinar Monday framing agent governance around "specific actions, scopes, and blast radii." That phrase is going to be the lingua franca of agent risk for at least the next 12 months.

We use blast radius informally in our deployment specs. We do not have a column for it in our review template. So the conversation we want to have (what's the worst this agent can do before someone catches it?) sometimes doesn't happen because the document doesn't prompt it.

The fix is a column. Each agent's row gets: "Blast radius at maximum scope, if every guardrail fails." One sentence. The act of writing the sentence is the audit.

What I'm shipping by Friday

The per-action cost wrapper, with one ceiling per agent class, tunable in config.
One credential per logical agent. Audit log change merges with it.
A new section in the deployment doc template: Production-Floor Reading.
The retry policy migrated from the prompt into the harness.
A blast radius column on the review template, populated retroactively for every active agent.

None of these are large. The total work is something like 3-4 engineer-days across the team. The reason I wasn't doing them last week is not that they were hard. It's that I didn't have the second reader yet. The internal reader was satisfied. Without the production floor, none of these gaps surfaced as gaps.

That's the part worth saying out loud. The second reader is what made the gaps visible. The work was always small. The doc was the bottleneck, and the doc didn't know it was the bottleneck.

What I'd ask if you ship agents

If you ran the same 45-minute exercise on your deployment doc this afternoon, which gap would you find first - cost ceiling, credential scope, retries, blast radius, or the review template itself?

I'm collecting answers through Friday. The Day 2 Interrupt content will sharpen the production-floor reading, and I'd rather refine the template against five teams' gap rows than against my own.

Tags: projectmanagement, ai, agents, productivity, discuss

I Read a Survey That Predicted My Job's Next 2 Years - Here's What It Got Right and Missed

Mykola Kondratiuk — Sat, 09 May 2026 07:34:36 +0000

canonical_url:

KPMG just dropped a number on people in my seat. They surveyed 306 Canadian executives. 39% of them expect AI agents to be leading project management for their teams within 2-3 years. 66% are already moving to a fully integrated AI-human workforce. First time the role-redefinition forecast is in survey data, not in an opinion column.

I run a PM workflow with an agent fleet doing most of the drafting and a lot of the review. So when an executive survey predicts the next two years of my job, I read it as primary source material on what the people who sign my budget are planning to assume.

Two things stood out.

What the executives got right

The direction is correct. The role really is shifting toward direction-and-review instead of artifact authorship. My morning two years ago was inbox plus drafting the day's first brief. My morning today is fleet status, then choosing which of last night's drafts is shippable, which needs another pass, and which got the wrong scope baked in and needs to be killed before it gets routed.

The horizon is also realistic, depending on where you're starting from. If your team has not yet stood up an agent stack alongside engineering work, 24-36 months to "agents leading PM" is plausible. There is a real procurement, instruction-tuning, governance-design, trust-building cycle to go through. None of it is fast on the first lap.

The integrated-workforce framing is the part the dev side will recognize fastest. The pattern is the same one engineering already lives: a PR queue where some commits are human-authored, some are agent-authored, and the human decision surface is mostly review and override. The PM equivalent is here. It looks like a doc queue, a roadmap delta queue, a sprint-scope queue. Same shape, different artifacts.

What the survey didn't ask

Executive surveys ask about role-level shifts. They don't ask the day-level question, which is the one engineers and PMs both actually live in.

The day-level question is: what does the morning look like, what's in the queue, what runs without you, what blocks on a human call, where does the dev-PM interface change shape because the PM is mostly directing instead of authoring?

For the dev side, the change that matters is on the spec-to-ship loop. Specifically, the spec side gets shorter and the review side gets longer. The PM is still naming what to build, but the artifact that lands in your repo as the brief or the scoped doc is increasingly drafted by an agent the PM directed and reviewed. The conversation about the spec moves from "let me write this up and send it Tuesday" to "the agent drafted three variants overnight, here's the one I'd ship, push back if anything looks off." Faster on the spec side. Slower on the review side, because the dev now has to verify that the directed-and-reviewed spec is still coherent before committing to it.

The survey doesn't measure that loop. It measures the hiring intent and the workforce category. Both useful, neither operational.

The 30-day diff

Here's a move that probably translates regardless of role.

Pull up your current todo list. Write down three items something automated is already doing or could plausibly be doing if you set it up. Write down three items only you in your seat can do. Then pull up your todo list from a month ago. Run the same split. How many items moved from "only you" to "automated or could-be"? Even one is a real signal. Three is a trend.

I started doing this around the time I noticed the agent had drafted a brief I'd planned to write. The diff that month was small. Six months in, it was not.

The KPMG number is a 24-month forecast. The 30-day diff is the short-horizon evidence the survey didn't ask for. The forecast is in their hands. The diff is in yours.

The floor, not the ceiling

If you've been running this for two years already, the 39% expecting "agents leading PM" in 24-36 months is the floor of what's coming. The practitioner who started seriously in 2024 is already past where executives expect they'll be in 2028. The interesting question is not "will it happen." It's "what does floor + 1 look like, and who's already there."

The dev side has been at floor + 1 for a while in a few places. The PM side is catching up.

What's the loop look like on your team?

I Read Boris Cherny's 30-Day Claude Code Stat. Here's What Most Takes Get Wrong.

Mykola Kondratiuk — Wed, 06 May 2026 07:44:58 +0000

Boris Cherny, Head of Claude Code at Anthropic, posted a stat on X this morning. In the last 30 days, 100% of his contributions to Claude Code were written by Claude Code itself. 259 PRs. 497 commits. 40,000 lines added. 38,000 removed. Zero by the head of the team.

Most reads of this go straight to "AI-assisted dev productivity is real." That's the obvious layer. It's not the interesting one.

The Engineering Leadership Read

If you run a team in 2026 - staff+ engineer or EM or director - the data point that matters is the second one. The head of a product team is no longer the team's most prolific code author. The head of the team is also no longer the team's most prolific reviewer; Claude Code does the first pass on its own PRs.

The work the head of the team is doing all day, then, is not the artifact. The artifact (the spec, the PR description) is a thing AI ships now. The calls inside the artifact - what to build, what to kill - those are the work.

This is the part most senior engineers haven't named in their own job yet.

Same Shift, Different Clothing

Same week, ServiceNow shipped a literal AI agent kill switch at Knowledge 2026. The demo: a prompt-injection attack hits a pricing agent. The system maps blast radius. A kill switch surfaces and asks a human to pull it.

Most coverage framed this as IT and security infrastructure. It is. It's also a leadership data point in product clothing. The vendor solved the product question - what does the kill switch do, how fast does it cut. The vendor cannot solve the leadership question - who decides when to pull it. Against what threshold.

Same shape as Boris's stat. The artifact (the kill switch feature) ships from a vendor. The call (when to use it) stays with the senior person on the team.

What the Deciding Work Actually Looks Like

It doesn't show up in a commit log. It doesn't have a template.

It's the moment in a Slack thread where someone asks "should we ship this?" and the senior person answers in two sentences with three reasons. It's the call to ship the rollback or the forward fix when the AI flagged the regression. It's the human pass on AI-written code that asks "but does this match the product intent?" and decides yes or no.

None of those moments produce an artifact. All of them are the work that compounds.

Why Measurement Systems Are Blind to It

Performance reviews and promo packets were built when the artifact was the work. They reward what got shipped. The work that got decided leaves no trace, so the system can't see it.

The senior engineer or EM measured by code volume or design-doc count is being measured against a 2023 work product. The senior engineer measured by decision quality - what got built, what got killed - is being measured against the 2026 one.

If your performance review still asks for ship counts and never asks about your call log, the system hasn't caught up to your job.

Three Moves If This Lands

The fix starts small.

First, start a private call log this week. One line per call. What was decided. What the alternative was. The first week feels like nothing. By month two it's the artifact your performance review was missing - a record of the work that doesn't show up anywhere else.

Second, lead with the calls in your next promo conversation or career check-in. "I shipped X" is 2023 language. "I decided X over Y because Z, and the outcome was W" is 2026 language. The shape of evidence changes when the work changes.

Third, find the leader on your team whose daily work is already 80% calls. Watch how they spend their day. That's the role shape you're growing into - and it's quieter than you'd think.

The Career Arc Nobody Named

The path from senior IC to staff to manager to director to VP has always meant less authorship and more direction at every rung. What's new in 2026 is the compression. AI takes over artifact production at every level. So the curve from authorship to direction now starts at IC, well below where it used to.

The senior leader who is still measured by what they shipped is being measured by a metric the system inherited. The senior leader who is being measured by what they decided is being measured by what the work actually is.

Boris Cherny just gave us the cleanest data point of the year for that shift. The Head of Claude Code stopped writing code, and the team kept shipping. That isn't a productivity story. It's a leadership story, and the system that measures the head of the team hasn't caught up to it yet.

What was your highest-leverage call this week, and is it visible in any system that measures your work?

I Pulled 3 Months of Engineering Metrics on Our AI Tools - Here's the Dashboard Cell Nobody Built

Mykola Kondratiuk — Wed, 29 Apr 2026 06:56:11 +0000

The CFO asked engineering. Engineering pointed at the PM retro. The PM retro had a row that said "team velocity feels higher" and a row that said "developers report subjective time savings." That was the data.

Meanwhile a fresh enterprise survey out of ExcelMindCyber says 73% of companies will fail to deliver promised ROI on AI investments this year. I read that and thought: of course. The dashboard for the question doesn't exist.

What the Repo Already Knows

Pull request throughput. Time-to-first-review. Cycle time from open to merge. Build duration. CI flake rate. Incident count. Mean time to recovery. PR size distribution.

We ship all of these. Most teams stream them into a Grafana board or a Linear analytics view or a custom dbt model on top of GitHub events. The data is in the repo. The data is in the CI logs. The data is in the deploy pipeline.

What we don't ship is the cell that says "for the workflow we adopted Tool X for, what changed."

That sounds trivial. It is not.

The Cell That Doesn't Exist

To answer the cell honestly you need three things in one row:

The workflow boundary. Not the tool boundary. "PR review" is a workflow. "Tool X" is a tool. The same tool can land in three workflows and change two of them. You need the join key to be the workflow.

The before-window. A baseline of the metric for that workflow before the tool landed. Not the team-wide cycle time. The cycle time on the specific class of work the tool was supposed to change.

The behavior signal. Did engineers actually use the tool inside the workflow, or did they sign up, click around once, and route around it. We have user-event telemetry for our own product. We rarely have it for the AI tool we just bought.

Without those three columns, the dashboard answers a different question. It answers "did we deploy the tool" not "did the workflow change."

What I Tried First (and Why It Failed)

The first version of the cell I built was a simple compare. Cycle time on PRs in February versus cycle time on PRs in April. Tool landed mid-March.

Numbers looked good. Cycle time was down 14%. I almost shipped it.

Then I segmented by PR class. Refactor PRs were down 22%. Bug-fix PRs were flat. Feature PRs were up 4%. The aggregate hid three completely different stories.

Then I looked at tool usage. Half the team had opened the tool fewer than three times in 30 days. The 14% improvement was carried by four developers. The rest of the team was running the same workflow without the tool and getting roughly the same numbers.

The honest answer to the CFO question wasn't "the tool drove a 14% improvement." It was "four developers got real value, the rest haven't adopted it yet, and we don't have the playbook for the rest."

If I had shipped the v1 number, the next quarter's budget cycle would have used it as proof. Then we would have spent more on the same shape of tool, and gotten a smaller delta, because the developers who would benefit had already adopted.

What Engineering Owns Here

I keep hearing the framing that "this is a PM problem." It isn't, or rather, it isn't only.

The PM retro happens after the quarter. The dashboard cell happens continuously. If engineering owns the metrics that say "this tool changed this workflow for these developers by this much," the PM gets a starting point that isn't fiction. If engineering owns nothing, the PM writes the retro on vibes and the CFO funds the next round on vibes.

The tools we already use give us most of what we need. GitHub events. Linear events. Tool-specific webhooks where they exist. A small dbt model that defines workflow boundaries explicitly. A heartbeat metric on tool usage at the user level.

The piece nobody is building is the join. Workflow x tool x usage x outcome. Four columns. Most teams have one.

The Smallest Version Worth Shipping

A single materialized view. Per workflow, per AI tool, per developer:

select
  workflow_id,
  tool_id,
  developer_id,
  date_trunc('week', event_at) as week,
  -- usage signal
  count(distinct case when event_type = 'tool_invocation' then event_id end) as tool_uses,
  -- workflow outcome
  avg(cycle_time_minutes) as cycle_time_avg,
  count(distinct case when event_type = 'workflow_completion' then event_id end) as completions
from workflow_events
join tool_events using (developer_id, workflow_id, week)
group by 1, 2, 3, 4;

Five columns. One join. Then a Grafana panel that shows cycle_time_avg split by tool_uses bucket (zero, low, high). The panel answers the question: "for the developers who actually use the tool, did the workflow get faster, and by how much."

The first time I ran ours, the bucket comparison was the most honest 30 seconds of the quarter. It told me which tools had earned their seat and which were budget items pretending to be productivity gains.

Honest Limit

This dashboard cell does not answer whether the tool was worth the money. That requires a price tag, a discount rate, an opportunity-cost guess. That part is genuinely a CFO conversation.

What the cell does answer is whether the workflow changed at all. Without that, the CFO conversation is fiction. With it, the conversation is at least a real conversation.

What's the cell your team has built and your CFO doesn't know about yet?