After the Cloud & AI Summit in St. Charles (September 30 to October 2), I did what everyone does with conference notes now. I gave Copilot the materials from fourteen conference sessions and asked for a summary.
It did a good job. Its executive summary opened with "The conference was not really about AI" and went on to say the event was about process, governance, context and quality engineering. It told me the whole thing lined up almost perfectly with the proposal I've been pushing at work, which it clearly knew about.
The best line in the whole document came from the session on taming agent swarms:
One Agent -> Test -> Multiple Agents
not
Multiple Agents -> Hope
I loved it. I'd already drafted a blog post with that diagram in the middle of it.
Then I opened the speaker's deck. Fifty-seven slides. The word "hope" isn't in it.
What the slide said
Samuel Gomez's real slide is less fun and more useful. It's a three-step decision: default to one agent, test one agent first when the choice is unclear, and scale to multi-agent only when specialization or scale justifies the operating cost. "Choose the simplest architecture that meets the use case." A later slide lists when separate agents earn their keep: distinct domains, hard boundaries around security or governance or data access, independent scale and parallel specialists. Both slides cite a Microsoft Learn article from December 2025.
Maybe Gomez said the "hope" line out loud in the room. It isn't on any of his slides. Copilot wrote that version of it, and wrote it well enough that I was ready to publish it as his.
Before I get smug about that, look at my own code. The AIEOS agent harness has a routing mode called parallel consensus. It sends a request to several providers and accepts the result when enough of them agree. "Agree" turns out to mean their answers are within 20% of the median length. That's a sanity check wearing a nicer name, and I'm the one who named it.
Two more things that weren't on any slide
So I checked the rest. I pulled the eight decks behind the sessions the summary leaned on and searched every one of them for the claims I'd been about to repeat.
The eight-step chain Copilot said kept showing up (knowledge, instructions, skills, agents, workflows, approvals, evidence, feedback) isn't drawn in any deck. The closest real thing is Benjamin Michaelis's five-step version in his Copilot agents talk: Instructions, Skills, Agents, MCP, Hooks. Copilot's version reads like that one, stretched to eight.
The second one stung. Copilot said the GitHub agent workflow talk and several governance talks pointed toward "Approve what will execute, not simply the PR." No deck says that. It's the argument I've been making at work, and Copilot knew my proposal. It didn't find a speaker who agreed with me.
It found me.
What the decks did say
I didn't throw the summary away, and the reason is a little annoying. Most of what Copilot reported was real, and the real versions were better.
Eric D. Boyd's MCP session had the most useful slide of the week, a decision ladder: prompt, instructions, skill, agent, tool, MCP server, workflow. Each rung asks one question, and the instruction at the top is "Stop at the first yes." MCP is sixth. The very next slide says "Don't build an MCP server when a markdown file will do."
I have a parked MCP server that proves his point. I built it for AIEOS back in May, 62 tests passing. It never got its last five percent of work, never got wired into anything, and in August I formally parked it. Whether a markdown file would have done the job is a question I should have asked in May.
Michaelis's GitHub Actions deck had its own ladder for rolling agents out: report-only, then staged writes, then shadow runs beside a human, then scoped production writes. "Start at the bottom. Earn each rung." The AIEOS dark factory, its unattended mode, is held back the same way. It refuses every unattended run until every validator on its path holds a passing calibration lock, and none does yet. My judge calibration is still under the bar I set for it.
The gap I came home with
Boyd's deck also had the slide I keep thinking about. "Discovery is not authorization." Classify every tool by blast radius in four tiers: read-only, reversible write, consequential, destructive. "Reversible preview, gated commit. Least privilege, per capability." Michaelis showed the same idea from the other end with preToolUse hooks that deny a call before it runs.
AIEOS comes up short here.
[Spec or design doc] -> <Freeze + validator: PASS/FAIL> -> [Agent does the work] -> < Act-time gate: approve the tool call > -> [Production]
^ AIEOS gates here today ^ the gap: not built
AIEOS gates the first check today. The second one is the gap.
Its gates sit at artifact promotion. An architecture doc can't move downstream until it's frozen and a validator passes it. That's design time, and it says nothing about the moment an agent holding a token decides to call a tool. I have a backlog item for a policy boundary that would cover runtime gating, and a spike that found the policy language already ships an approval-gate example close to what I need. It isn't built. After Boyd's four tiers it should probably move up the list.
Own the semantic layer
Greg Beaumont's Fabric session put a stack on one slide. Storage at the bottom is "a Cheap Commodity." Compute costs for queries and LLMs are "in a race to the bottom." The semantic layer in the middle is where "IP that differentiates value exists." The slide's bottom line: "Own the Semantic Layer, Own the Value!!!" (three exclamation points, his).
That's the second gap. AIEOS keeps a machine-readable kit manifest as its single source of truth, which is a start. The broader semantic retrieval idea has sat at the bottom of my idea list since April. Maybe that's the right call. After Beaumont's slide I'm less sure, and I don't have a better ranking yet.
The part Copilot got right
Copilot's overall assessment opened with a line I'd have written myself:
AI should be treated like a highly capable but fallible implementation engine operating inside deterministic organizational controls.
That's the thesis AIEOS was built on. Then Copilot demonstrated it on me. It generated a summary that was mostly right and partly invented, and it had no way to tell me which parts were which. The deterministic control turned out to be eight slide decks and an evening of searching them.
Read the slides before you quote the summary.
Originally published at DevOps Diary.
Top comments (0)