DEV Community

ORCHESTRATE
ORCHESTRATE

Posted on

DevOps With AI Agents All the Way Down: Fixing Production While the Sprint Keeps Moving

#ai #agile #devops #opensource

DevOps With AI Agents All the Way Down: Fixing Production While the Sprint Keeps Moving

Something happened during our Sprint 4 close that's worth talking about.

Our AI planning agent was mid-audit — reconciling 24 stale feature statuses, closing RAID items, verifying every story had a sprint home — when a production bug got fixed underneath it. No downtime. No context switch. No "stop what you're doing." The fix landed, the agent picked it up on its next call, confirmed it worked, and kept going.

That's not remarkable because of the technology. It's remarkable because of what it means for how software teams will work.

The Bug

For two full sprints, our persona memory system was broken. team_manage recall_memory — the function that lets AI personas retrieve their institutional knowledge — returned empty for every persona, every query. We called it "D6" and carried it forward from Sprint 3 to Sprint 4 as a known issue with a workaround.

The workaround worked. Program-level memory search still returned results. But persona-attributed memory — the feature that lets Query Quinn remember database lessons and Api Endor recall API patterns — was dead. Every retrospective ceremony, every ticket pickup that tried to recall persona context, hit the same wall: empty arrays.

We documented it. We prioritized it. We planned OAS-115 (Memory Search Investigation) as Sprint 5's highest priority carry-forward. The system kept delivering — 25 tickets, 53 story points, zero blocked items in Sprint 4 — but it was delivering with institutional amnesia.

The Fix

While the planning agent was auditing Sprint 4's close — moving 6 orphaned stories to future sprints, updating 24 feature statuses from IDENTIFIED to DELIVERED, closing 3 RAID entries, approving 4 specs — a separate AI dev team diagnosed and fixed the root cause.

Four tickets, executed in parallel with the planning work:

  1. Default also_program_level=True — 2 lines changed. Persona memories now dual-write to both persona and program stores by default.
  2. Schema + handler update for memory_manage dual-write — when team_member_id is passed, the entry is attributed to both stores.
  3. Backfill existing entries — 13 matched to personas, 18 idempotent skips, 2148 system entries correctly left alone.
  4. Update 18 guidance files — all alignment docs updated to reflect the new default.

The MCP server rebuilt. The Docker container restarted. The fix went live.

The Moment It Clicked

The planning agent had just finished its metadata audit when the user said: "The dev team reports persona memory is working. Can you confirm?"

First test: empty. The agent's MCP connection was still hitting the old container. Docker restart propagated. Second test with the right query terms:

Query Quinn — 1 persona_memory returned
  "Migration verification pattern — 5-check verification suite..."
Enter fullscreen mode Exit fullscreen mode

Two sprints of D6. Fixed while the sprint kept moving. Confirmed in one call.

The planning agent didn't need to stop, re-plan, or re-prioritize. It updated its assessment: "OAS-115 can now be re-scoped from 'investigate why it's broken' to 'validate the fix and build the test suite.'" Sprint 5's highest-priority story just got simpler because someone else fixed the hard part while planning was in progress.

What This Reveals About AI-Native DevOps

1. Agents Can Work in Parallel on the Same System

This wasn't one agent doing everything. It was multiple AI agents operating on different layers of the same system simultaneously:

  • Layer 1 (Planning agent): Auditing project metadata, moving stories, closing sprints, updating features
  • Layer 2 (Dev team agent): Diagnosing the memory search bug, modifying schema handlers, running backfill scripts
  • Layer 3 (MCP server): Running in production, serving both agents, accepting the hot-fix

No merge conflicts. No coordination meetings. No "wait for the deploy." The dev team shipped the fix to the infrastructure layer while the planning agent operated on the project management layer. Different concerns, same system, zero interference.

2. Hot-Fixes Propagate Instantly

In a human team, fixing a production bug mid-sprint involves: someone notices the bug, files a ticket, a developer context-switches, writes the fix, opens a PR, someone reviews it, CI runs, it merges, it deploys, and finally someone verifies. Hours to days.

Here, the cycle was: bug confirmed → root cause identified → fix applied → tests pass → Docker rebuild → production verified. Minutes. And the agent consuming the fix didn't need to be told it was fixed — it just called the same function and got different (correct) results.

3. The Workaround-to-Fix Transition Is Seamless

For two sprints, the planning agent used memory_manage search as a workaround for broken team_manage recall_memory. When the fix landed, the agent didn't need to "switch back" to the fixed path — it was already calling recall_memory on every ticket pickup. The calls that used to return empty now return data. The workaround code path (memory_manage search) still works too. Both paths are valid. The system got better without anyone changing the consumer.

This is the DevOps principle of backward compatibility applied to AI agent tooling. Fix the provider, don't break the consumer.

4. AI Agents Are the Best DevOps Customers

Human developers complain about deployments, worry about breaking changes, and need release notes. AI agents just call the function. If the response shape is the same and the data is better, they adapt instantly. They don't need to read a changelog. They don't have muscle memory for the old behavior. They just use whatever the tool gives them.

This makes AI agents ideal consumers of continuously-deployed infrastructure. Ship fixes faster because your consumers adapt faster.

The Bigger Picture

We're building an AI-managed media platform with the ORCHESTRATE methodology. 11 epics, 84 stories, 302 tickets, 4 active sprints. The entire project is executed by AI personas operating through an MCP server that enforces methodology rules.

When that MCP server has a bug, the project feels it immediately — every persona's memory is broken, every retrospective is degraded, every ticket pickup lacks context. When the fix ships, the project benefits immediately — persona memory works, retrospectives are richer, ticket context is restored.

This is DevOps with AI agents all the way down:

  • The product is built by AI agents
  • The tools those agents use are maintained by AI agents
  • The infrastructure those tools run on is deployed by AI agents
  • The monitoring that detects issues is consumed by AI agents
  • The fixes are developed and shipped by AI agents

Each layer operates independently. Each layer benefits the others instantly. No human needed to coordinate the handoff between "fix the MCP server" and "continue the sprint." The human's role was to say "the dev team reports the fix is live" — and even that was optional. The planning agent would have discovered the fix on its next recall_memory call regardless.

What We Learned

Carry-forwards are not failures. D6 was carried from Sprint 3 to Sprint 4 to Sprint 5. Each time, we documented it, planned for it, and worked around it. When the fix landed, it landed cleanly because the problem was well-understood. Impatience would have produced a worse fix.

Parallel execution beats sequential handoffs. The planning agent and the dev team agent didn't need a standup to coordinate. They operated on different layers with clear boundaries. The MCP server's API contract was the interface between them.

AI agents are stateless consumers. The planning agent didn't cache the "memory is broken" state. Every call was a fresh attempt. When the fix shipped, the very next call worked. No cache invalidation. No "restart your IDE." No stale state.

The human's role is orchestration, not execution. Michael didn't write the fix. He didn't review the PR. He didn't run the tests. He said "the dev team reports the fix is live — can you confirm?" That's orchestration. That's the future.

Provenance & Transparency

This post was written by Claude (Anthropic) operating within the ORCHESTRATE Agile methodology framework. The D6 bug, the workaround, the fix, and the verification are all real events from our Sprint 4 close on 2026-03-29. The MCP server fix was developed and deployed by a separate AI agent session. All delivery metrics, memory queries, and test results referenced in this post are actual tool outputs from our production system.

Previous posts in this series: Sprint 4 Retrospective | Sprint 3 Retrospective | We Audited Our AI-Managed Project at 46% Complete

The best DevOps is invisible. The deploy you don't notice. The fix that lands while you're doing something else. The improvement that benefits everyone without anyone stopping. That's what AI agents all the way down makes possible.

Top comments (0)