DEV Community

Ilya Gorban
Ilya Gorban

Posted on

An AI-Native Dev Lifecycle Across R&D: The Plan

#devops #ai #codereview #adlc

TL;DR. AI coding agents now open pull requests faster than people can review them. Instead of reviewing faster, I plan to decide which PRs need a human at all. A PR review agent puts each PR in one of three risk tiers by written rules: merge on green, AI review with a human after merge, or human review before merge. It starts in dry run: people decide as today, and every disagreement is a finding. After merge, an alert-triage agent links alerts to recent merges and proposes reverts, never merging them. Part 2 will report what survived real PRs.

This is an opinionated write-up of what I am planning as a solo DevOps engineer for a whole R&D organization, in one company's setup. It is a plan, not a result. Treat it as reasoning to weigh, not as the one right answer.

This is part 1 of a series on an AI-native dev lifecycle:

  1. This article
  2. What was adopted, and the first measured results (after implementation)

Why

Most of our pull requests are now opened by AI coding agents. That changed the shape of the review problem.

Waiting for a review is not the bottleneck. Attention is. Every PR that asks for a human competes with every other one, and an agent can open a dozen in the time it takes to read one carefully.

The obvious answers both fail:

  • Review everything. It does not scale, and a reviewer reading their thirtieth agent PR of the week is approving, not reviewing.
  • Trust the agent. Then nothing stands between a confident, wrong change and production.

Instead: spend human attention where a change can hurt production, and prove with data that the rest is safe without it.

What

Three risk tiers

Tier Gate Typical changes
A: merge on green Required checks only Human-facing docs, non-prod overlays, pin-only bumps
B: AI review, human after merge Required checks plus an AI review with no open findings; the owner is notified on merge and can ask for a revert Application code, workflow logic, tests, tooling, runtime markdown, agent skills and knowledge
C: human before merge Required checks plus one approval from the owning team Prod-grade config, shared CI libraries, infrastructure as code, secrets and permissions, every database change

Four rules make the tiers hold:

  1. The riskiest file decides. A docs edit bundled with a prod overlay is tier C.
  2. A tier can be raised, never lowered, and nothing an agent writes can lower it.
  3. Default-deny. A path that matches no tier list fails a deterministic check, so a new directory blocks until classified instead of silently becoming tier A.
  4. Database changes are always C. Queries, schema and permissions. In the history I mined, nine in ten database-touching PRs sat in ordinary application paths that a path rule would put in tier B.

The rule seed is the set of paths that already require approval from people with elevated permissions, such as production or infrastructure approvers. It is the closest thing to a written statement of "this path needs these people".

Two levels of knowledge

A reviewer, human or AI, needs two kinds of knowledge:

  1. The application. The shared conventions every app follows (how code, comments, docs, tests and infrastructure are written, and how a PR is made), plus each app's own contract: its interfaces, where its database contract lives, and which of its files are runtime.
  2. The change. What in this diff needs human eyes, and what can merge on green.

The first level exists because of one finding. Path-only tiering treats *.md as documentation, wrongly in two places. In one service, every "docs" PR in four weeks edited LLM prompt assets that ship with the service. In another repo, runbooks and agent knowledge files are rendered into a live agent's knowledge base. Both are behaviour changes that look like documentation. Only the application knows which files it reads at runtime, so every app has to declare them before any path rule can be trusted.

Agent knowledge is the same trap. An edit to a skill that does something important can break it. An in-repo knowledge base that is wordy or wrong bloats every later agent's context, or teaches it the wrong thing. Those files are reviewed for correctness and brevity, not waved through as docs.

Ground truth from history

A dry run needs the right answer for each PR. History gives a start:

  • reverts;
  • a follow-up fix PR that references this PR within a week;
  • changes requested by a reviewer, or review threads followed by new commits;
  • rubber-stamp approvals: no thread, no change, no follow-up.

They are candidates, not labels. On a spot check, a "follow-up fix" was sometimes a real regression and sometimes a sibling change or ordinary iteration. A shared changed file was mostly noise. So people confirm a seed set by hand, one row at a time: should the agent have said "merge" or "human" here, and why.

And the measure has to cover both directions:

  • false automerge: the agent would have merged something a human needed to see;
  • false human: the agent would have made a person wait for nothing;
  • noise: findings nobody acts on, because noisy AI review loses trust faster than no review.

How

Before merge: a read-only classifier

My proposed first version runs Claude Code headless in a GitHub Action:

  1. A workflow runs claude -p on each PR, with read-only tools, the repository's rules and the diff.
  2. --json-schema forces the shape of the verdict: tier, findings with cited evidence, the rules hit, a confidence.
  3. A small script, not the model, validates that JSON and posts it as a GitHub check. During the dry run the check is neutral.

The design is not tied to it: any agent that runs unattended with schema-shaped output fits, and agents can be compared on the same evaluation set.

Reviewing a PR that is still changing burns tokens. So the agent reviews only PRs marked ready, after the cheap required checks pass. The shared knowledge base tells every agent session to open PRs as drafts. The platform's own draft state beats a custom label: the ready-for-review event is the trigger, with no extra convention to forget.

The diff, PR text and comments come from whoever opened the PR: untrusted input. The agent that reads them holds no write token and no cluster access, and the merge decision is enforced by the repository ruleset, outside the model.

Alternatives I ranked:

  • Writing my own agent loop on an SDK. Full control, but then I own a harness, which is a commodity; what is ours is the knowledge and the evaluation set.
  • Hosted review bots. They produce comments rather than a verdict a script can enforce, and our per-application rules do not plug in easily.

After merge: link alerts to merges

Live tier A and B changes still need watching. The planned first version reuses an alert-triage agent I already run, built on the open-source HolmesGPT, which investigates alerts with runbooks and read-only access. It will gain one step: did this application deploy a merge shortly before the alert? If so, it names the PR, opens a revert PR and pages the owner. None of this is built yet, and the first thing to confirm is that the agent's toolsets can read which revision is actually deployed.

Three constraints:

  • It never merges its own revert, and never changes the cluster. A revert is a git change like any other.
  • Merge is not deploy. It attributes an alert to a merge only once the GitOps controller's synced revision includes the merge commit. A merge still awaiting sync caused nothing yet.
  • Different trust profile, different agent. The pre-merge reviewer reads untrusted diffs and must hold no authority. The post-merge one reads live system state and needs cluster read access. Putting both in one agent would join untrusted input to authority.

Feedback: teach the PR agent from outcomes

A feedback run, started by the agent's maintainer by hand or on a schedule, sets the agent's verdicts over a period against what happened next: human overrides, reverts, follow-up fixes, linked alerts. It proposes rule and knowledge changes as a tier C PR, because changing the PR agent's rules changes what merges unseen. Each confirmed miss joins the evaluation set.

The compliance gate

If you hold a security certification, your change-management control almost certainly requires review before a change ships. Automerge under that text and your audit evidence no longer matches what happens.

So the agent does not leave dry run until the change-management control has been read and either already covers AI review, or has been updated with the auditor's agreement. Check your control text before you automerge anything. My proposed shape is risk-tiered change management: human approval kept for tier C, and for A and B, required automated checks, the AI verdict recorded on the PR, owner notification and a revert window. Whether that satisfies an auditor is the auditor's call.

What I took from published ADLCs, and where I disagree

None of three outside sources targets my problem, more PRs than reviewers; most move human attention earlier, into specs and plans.

"Inside the ADLC" (Towards AI) describes a spec-driven pipeline with role-tiered agents. I took three ideas:

  • review as several read-only reviewers by dimension, each returning structured findings;
  • "a reviewer that can change code has a conflict of interest";
  • autonomy as an explicit list of stops, which I invert: list what forces human review, and everything else merges on green. That list is also the audit artefact.

Where I disagree: no outcome data, it reviews only its own code, and its wrap-up step merges unattended despite saying merging needs approval.

The PDD kit keeps one product model per application in Markdown, updated by the agents before code. It is the closest existing shape for my level-1 contract; my history adds runtime files and where the database contract lives. Its evidence is a launch post.

"The Agentic Awakening" (a venture investor's playbook) argues that AI review in CI is the cheapest autonomy step, that you should own your context and evaluations and buy the harness, and that a human gate stops prompt injection only when the decision is enforced outside the model. I took all three. I discount its evidence: about twenty interviews, mostly from one investor's portfolio, no control group, and lines of code called a vanity metric, then used as its headline.

What part 2 will report

Part 2 will report, against this plan:

  1. Dry-run accuracy against the confirmed seed set: false automerge, false human and noise, separately, per tier.
  2. Whether path rules plus per-app knowledge caught the runtime markdown and database changes that path rules alone missed.
  3. Tier mix and reviewer load: how many tier C requests each reviewer got per week, before and after.
  4. Stability, using the signals the DORA AI Capabilities Model ties to AI adoption: PR size, reverts of A and B merges, incidents traced to them, rework rate (follow-up fixes within a week) and change failure rate.
  5. The post-merge link: first, whether the agent can read the deployed revision at all; then how many alerts it tied to a recent merge, and how many of those it got right.
  6. What I dropped or changed, and why.

The success bar is fixed now so it cannot drift: fewer tier C requests per reviewer than today's total review requests, with no production incident traced to a tier A or B merge.

References

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Top comments (0)