<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hasan Madi</title>
    <description>The latest articles on DEV Community by Hasan Madi (@teledream).</description>
    <link>https://dev.to/teledream</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247989%2Fd579964b-360d-400a-a34e-0d2f94325eec.jpg</url>
      <title>DEV Community: Hasan Madi</title>
      <link>https://dev.to/teledream</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/teledream"/>
    <language>en</language>
    <item>
      <title>Git is not enough for agents</title>
      <dc:creator>Hasan Madi</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:48:29 +0000</pubDate>
      <link>https://dev.to/teledream/agents-need-an-undo-button-oh6</link>
      <guid>https://dev.to/teledream/agents-need-an-undo-button-oh6</guid>
      <description>&lt;p&gt;I feel that by now, most of us are increasingly running agents on YOLO/auto-mode (skipping approvals), the friction of approving the actions of every agent and its subsequent subagents is becoming impractical. This increases the need for snapshots. Yes, Git can hold previous copies of your code in case something went wrong, but it is ultimately version control software that assumes a level of human safety.&lt;/p&gt;

&lt;p&gt;In fact, Git increases the surface area by which an agent could plausibly mess up your repo:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;git clean -fdx&lt;/code&gt;: deletes untracked files, which Git can't undo.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;git reset --hard&lt;/code&gt;: clears uncommitted changes, and the reflog &lt;em&gt;doesn't&lt;/em&gt; bring them back.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;git push --force&lt;/code&gt;: rewrites the remote for everyone.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;git branch -D&lt;/code&gt; followed by &lt;code&gt;git gc --prune=now&lt;/code&gt; clears even the reflog's safety net.&lt;/li&gt;
&lt;li&gt;Anything you gitignore: &lt;code&gt;.env&lt;/code&gt;, a local db, &lt;code&gt;~/.config&lt;/code&gt;, untracked projects.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So, what is the alternative? Say that your agent runs in a sandbox with access to a hosted Git repo. Anything you do to protect that repo must either (a) ferry state out of the sandbox, or (b) restrict the agent's capacity within the sandbox itself. Both get messy quite quickly.&lt;/p&gt;

&lt;p&gt;I gave a third option a shot: snapshot the entire VM the agent runs on, frequently yet cheaply, and record the screen alongside it, creating a "livestream" effect where you can view and return to previous states. This article will discuss the architecture of this system, the work at the hypervisor level that was done to keep it fast, and why (I think) it serves as a practical undo button for agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;p&gt;First, I needed to pick a VM technology that would function as a good foundation. I originally started out with Firecracker microVMs, yet later ended up with the more obscure crosvm, which is behind ChromeOS's Linux support.&lt;/p&gt;

&lt;p&gt;Few have heard of crosvm despite Firecracker starting out as a fork of it. Something that makes crosvm unique is that it has &lt;code&gt;virtio-fs&lt;/code&gt;, which unlike Firecracker's &lt;code&gt;virtio-blk&lt;/code&gt;, is flexible enough to mount a host's ZFS dataset. As I'll later explain, this'll be the primitive the system builds upon.&lt;/p&gt;

&lt;p&gt;Firecracker was designed for short-lived serverless functions. crosvm was made for Chromebook users, making it one of the rare VMs designed from the ground up for user interaction. As such, it exposes a lot of graphical, audio, and passthrough devices that make it unique for sandboxing agents, where a user often needs to interact with the agent's environment. It also has Google's paranoid "process-per-device" security model, wherein each virtual device runs in its own sandbox, so compromising one doesn't compromise the whole VMM.&lt;/p&gt;

&lt;h3&gt;
  
  
  The filesystem: ZFS datasets under &lt;code&gt;virtio-fs&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The VM's home directory isn't actually a disk image. It's a ZFS dataset on the host, mounted into the guest with &lt;code&gt;virtio-fs&lt;/code&gt;. ZFS will do a lot of heavy lifting here.&lt;/p&gt;

&lt;p&gt;A ZFS snapshot is copy-on-write. As the name would suggest, it doesn't immediately copy any data, instead freezing the block pointers, and only blocks written afterward take new space. A snapshot of an 80GB &lt;code&gt;/home&lt;/code&gt; only takes up the space of the files that changed, and in the case of agents, this is often only kilobytes of source code. This system allows me to frequently snapshot without much consequence.&lt;/p&gt;

&lt;p&gt;With a disk image, the host sees a single opaque file, so you either (a) copy the whole thing or (b) get into the messy territory of guest VM tools.&lt;/p&gt;

&lt;h3&gt;
  
  
  The memory: checkpointing a running VM
&lt;/h3&gt;

&lt;p&gt;Snapshotting files isn't enough though. If you roll back an app to a previous filesystem, yet with the same memory, it'll probably just crash, or worse yet: corrupt something.&lt;/p&gt;

&lt;p&gt;So, a full snapshot includes a crosvm checkpoint of the RAM and device state bundled alongside the ZFS snapshot. The issue is taking a memory snapshot without causing a noticeable pause.&lt;/p&gt;

&lt;p&gt;You could pause the VM, write out its memory, then resume. With a few gigabytes in use that will take a few seconds, frequently.&lt;/p&gt;

&lt;p&gt;Instead the copy runs in rounds while the VM keeps going. The first round copies everything, and later rounds only copy new changes, a little like an incremental backup of memory where each round only copies what changed from the last.&lt;/p&gt;

&lt;p&gt;Some statistics from a demo VM:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Total time: 4.4s&lt;/li&gt;
&lt;li&gt;Pre-copy: 2.2s in 5 rounds&lt;/li&gt;
&lt;li&gt;vCPU pause: 58.9ms&lt;/li&gt;
&lt;li&gt;Disk hold: 67.1ms&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;(the disk hold is how long the filesystem is frozen while the ZFS snapshot is being taken)&lt;/p&gt;

&lt;p&gt;So from a user's perspective, snapshots are a blip that only takes 60ms.&lt;/p&gt;

&lt;p&gt;If the VM is writing memory faster than rounds can catch up, the snapshot waits for a calmer moment instead of forcing a long pause.&lt;/p&gt;

&lt;h3&gt;
  
  
  The recording: snapshots are a pain if you can't see them
&lt;/h3&gt;

&lt;p&gt;So, now you have a huge list of snapshots called &lt;code&gt;2026-11-16T11:12:57Z&lt;/code&gt;... To state the obvious: a user remembers what the screen showed before things broke, not the exact time.&lt;/p&gt;

&lt;p&gt;Alongside snapshots, the desktop's screen is continuously recorded as a video. Because we're talking about SSH here, most of the screen is text, not pixels, which gave me a unique opportunity to make the system even more efficient by creating a custom sort of video format that directly encodes unicode and compositor state, but that's for another article. Scrubbing back through the video doubles as the interface for undo: drag back until you see the state you want, and restore. The timeline maps each moment of the recording to the closest snapshot.&lt;/p&gt;

&lt;p&gt;Restoring doesn't clear the current state. It starts a new branch, as if you ran &lt;code&gt;git checkout -b&lt;/code&gt; but for the whole machine. If you went back too far, the branch you left is still there. Restoring a snapshot only takes about 7 to 9 seconds on our host.&lt;/p&gt;

&lt;p&gt;Basically, it's like a video player:&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv13n9zd8ne9jjcbm4bw5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv13n9zd8ne9jjcbm4bw5.png" title="Rewinding to a previous Claude Code state [all over SSH]" alt="the timeline bar with a branch hovered" width="799" height="223"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;
  
  
  The remainder: resource costs of the system
&lt;/h3&gt;

&lt;p&gt;It's pretty efficient, although there remains a subtle cost (which I think is worth it):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Disk&lt;/em&gt;: snapshots capture things that probably don't need to be captured, such as a build that takes up hundreds of megabytes. Copy-on-write limits the cost to only occurring once. RAM is also tough as it's gigabytes large and changes frequently. I've implemented a deduplication system which acts in chunks across snapshots, to relieve some pressure there.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Retention&lt;/em&gt;: despite efficiency, snapshots still take up a lot of space, and so older history is phased out.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  Agents need an undo button (closing)
&lt;/h3&gt;

&lt;p&gt;To summarize, an agent with approvals off will eventually do something you didn't want. Git's protection is undone by shell access. Anything outside Git isn't protected in the first place. Trying to restrict the agent's permissions within the VM, or trying to ferry state out of the VM, are both messy solutions.&lt;/p&gt;

&lt;p&gt;Snapshotting the full machine is the "nuclear option", and I've found it effective, and hope others will too.&lt;/p&gt;

&lt;p&gt;(snapshots are not a backup though, as they are all saved to the same place, so keep a copy of anything important elsewhere)&lt;/p&gt;

&lt;p&gt;I've made a public version, &lt;a href="https://zydra.sh" rel="noopener noreferrer"&gt;Zydra&lt;/a&gt;, for anyone to try (without signing up to anything), give it a try :)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh zydra.sh
ssh -p 443 zydra.sh &lt;span class="c"&gt;# From an office network&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



</description>
      <category>ai</category>
      <category>claude</category>
      <category>security</category>
      <category>linux</category>
    </item>
  </channel>
</rss>
