An agent opened a pull request on our service last week. Six hundred lines. It rewrote how we handle webhook retries and deduplication, an area tha...
For further actions, you may consider blocking this person and/or reporting abuse
Hey, this article appears to have been generated with the assistance of ChatGPT or possibly some other AI tool.
We allow our community members to use AI assistance when writing articles as long as they abide by our guidelines. Please review the guidelines and edit your post to add a disclaimer.
Failure to follow these guidelines could result in DEV admin lowering the score of your post, making it less visible to the rest of the community. Or, if upon review we find this post to be particularly harmful, we may decide to unpublish it completely.
We hope you understand and take care to follow our guidelines going forward!
Honestly, I think Git needs to upgrade. Include the agent session with the PR, so you can read through the reasoning. Ideally, you would be able to pull it into your own model and continue conversation with it. That way you can ask questions that your model can explain. Imo, that's the future of code reviewing, because syntax is perfect, tests are all green, but to understand why, that needs adjusting in order to accommodate agents.
Yeah, this is where I think it goes. The session is the missing artifact. Right now the reasoning gets thrown away the moment the PR opens, and everyone downstream is reconstructing it by hand. Attaching it to the branch so a reviewer can actually interrogate it, ask their own model why a given choice was made, is what review looks like once the agent is the one writing.
The one thing I'd add from building this: the raw session is too noisy to just bolt on. Half of it is dead ends and tool output nobody needs. The signal is the decisions and the moments a human stepped in and steered, lifted out of the transcript. Pulling that out is most of the work, and it's why I'm shaping it as a story instead of a dump.
That's the part I'm building as Branch Story (which is live), it lives in a VS Code extension if you ever want to see the shape of it. But you've got the direction exactly right.
True, I mean if it's a week long dev cycle, it's millions and millions of tokens, far too much for the LLM to actively cache. Maybe worth using a codebase map then draw association to each file per-prompt, so when you query it, it can swap in just that chunk as context. Similar to what I built for V.E.L.O.C.I.T.Y. OS. I'll give it a look at some point, it wont really be all that usable to me, because the foundry runs swarms of instances at once, each with individual scoped tasks, etc and the ledger stores every decision already, so it's pretty good for understanding what, why, when, where, how, but for anyone using a single agent, or an ide in general, that seems extremely useful.
Yeah, you've got the architecture right. You don't cache the transcript, that's a losing game. The session gets keyed against a map of the codebase, decisions associated to the files and topics they touch, and you pull in just the relevant slice per query. That's basically how it works under the hood, so we're thinking the same way there.
And you're right that a decision ledger already gets you most of the what, why, when. That's the capture side, and it's the hard part, sounds like you've solved it well for velocity. The bit I'd pull apart is that a ledger is built for querying, and a human reviewer doesn't want to query, they want to read. Turning the ledger into something a person can go through top to bottom and come away holding the intent is the layer I'm actually building. Different job from storing it.
On the swarm point, that's actually the exact case I built the handoff piece for. When multiple agents work a branch, each one passes its knowledge forward so the next doesn't start from zero, and the story stitches across them instead of leaving the why scattered per instance. So it might sit closer to your foundry than you'd expect. Either way, appreciate you giving it a look whenever you get to it.
The missions and mission reports are pretty good at reviewing what the model did and why it was done in a way, but if you want a summary from A-Z, that's the final audit's job, as it produces 2 files, the full report, which is highly detailed and the summary, which covers all parts, but as an overview. Interesting that they're so similar, do you work dir-file or file-dir for your propagation?
Yeah, that two-file split maps almost exactly onto how I think about altitude. Your full report and your summary are the same instinct I'm working from, I just fold them into one surface instead of two files. The story reads at the summary altitude, and you expand any beat into the evidence sitting behind it. Progressive disclosure rather than two artifacts, but same idea: one level to read, one level to verify when a claim doesn't sit right.
On propagation I work file-dir, so bottom-up. The decision is the atomic unit and it anchors to the files it actually touched, then the directory, topic and branch views are rollups over those file-level decisions. The reason is that a reviewer is staring at a diff, which is inherently file-level, so anchoring the why at the file keeps the story lined up with exactly what they're looking at, and the higher-level narrative composes upward from there. Directory-first read cleaner on paper but it kept drifting off the actual change.
Curious which way you landed, since your missions sound scoped more like units of work than files, so I'd guess you lean dir-file. What pushed you there?
The part that would stop me cold is "it rewrote webhook retries and dedup, tests were green." Green on the existing suite is the weakest possible signal there, because the agent wrote the code AND stayed inside the assumptions your current tests already encode. For a change like that I gave up reviewing the diff as prose and started reviewing the test delta: does this introduce a failure mode the suite does not cover, and is there a case that would go red if the dedup key were subtly wrong. If the answer is no, the PR is not reviewable yet no matter how clean it reads, and the ask back to the agent is "add the test that fails without this change," not "explain the diff." That moves the hour from reconstructing intent to checking coverage, which is the part that actually scales when generation gets cheap.
The gap it leaves is that you still have to know what the failing case should be. In our dedup example the test that goes red is "two byte-identical payloads with different idempotency keys must not collapse," and nobody writes that test unless they already know the payloads can be identical. That knowledge is the same contextual thing that wasn't in the diff. So I'd run them together: the narrative tells you which invariant actually matters here, then you demand the test that proves the agent respected it. Coverage-checking scales, but only once you know what to cover.
The 600-line PR nobody wants to review is where the whole model breaks down. Review was the safety net, and it assumed a human volume of change. Agents blew past that, and LGTM on 600 lines nobody read is just trust with extra steps. The fix isn't heroic review, it's shrinking what has to be reviewed by hand: enforce the boilerplate and the standards mechanically so human attention goes to the handful of decisions that actually carry risk. A 600-line diff isn't the problem; 600 lines where every line is equally unverified is.
the bottleneck-moved framing is right, and the part i'd underline is buried in your own example. the dedup-by-payload mistake is the kind a narrative makes worse before it makes better, because the agent would have narrated that decision as sensible. "i deduplicated on payload for simplicity" reads fine. the story of the reasoning is still the agent's account of itself, and a confident wrong assumption produces a confident wrong narrative.
which is why the line at the end is the load-bearing one: grounded in the team's actual captured decisions, not reconstructed from the diff. that's the difference between a narrative you can check and one you just read. the first catches "you deduped on payload but our idempotency key exists for a reason." the second just makes the wrong call easier to approve because now it has a story attached.
so i think the real scarce thing isn't the narrative, it's the captured decision the narrative can be checked against. which tracks with you saying the capture side is where most of your work goes. that's the hard half and the right half.