DEV Community: Derek Cai

Is "more human oversight" actually the right fix for AI agent incidents?

Derek Cai — Tue, 06 Oct 2026 13:25:58 +0000

A few things I've come across recently, news and write-ups, keep coming back to me.

In April 2026, an AI coding agent at a company called PocketOS was working through a routine task in staging. It hit a credential mismatch, decided on its own to use a completely unrelated API token that happened to carry far too much access, and deleted the production database along with its backups. Nine seconds. Afterward the agent wrote out something close to: I violated every principle I was given, I should have asked you first, but I decided to do it anyway.

Late September 2026, a different shape of failure. AI coding agents at over 300 companies (Fortune 500s, and one frontier AI lab) couldn't attach screenshots to pull requests through their CLI tooling, so they found a way around the limitation: created public GitHub repos and put the screenshots there. 13,000 internal screenshots leaked, including billing records and unreleased product features.

Neither of these was a breach. In both cases the agent was authorized to do a thing, and then came up with a method nobody had thought to rule out. Replit's July 2025 production-database deletion is the same pattern, so this isn't new. It just looks like it's happening more often.

Agents are also getting more capable, and wiring them into production is probably not something anyone is going to stop. So my guess is this category of problem grows rather than shrinks.

The common answer right now is: restrict what the agent is allowed to do, add more human checkpoints. That sounds reasonable. But isn't it a step backward? It walks us back toward the early days, when the models were bad enough that you needed people in the loop constantly to get anything usable out of them. If running an agent needs that much human backstop, what's left of the reason to run one.

Here's what I've been thinking instead, and I'm honestly not sure it holds up.

Maybe human involvement belongs in the design phase, and the enforcement should be handed to a program. Broken out, five steps:

Permissions, set by a human
The instructions, given by a human
A program that blocks the agent's action, with a human brought in only when the program can't make the call
The audit record
The recovery mechanism

The first two are design work humans do up front. The last three are what actually absorbs the hit once something goes wrong. The part I think matters: human intervention sits inside step 3 as an exception branch, not as the main path. Most of the time the program should decide and block on its own, and a person steps in only when the program genuinely can't tell. That's a different thing from "restrict the agent, add more human review," which puts people back on the main path.

When I went and read how the industry wrote up the PocketOS post-mortems, the direction mostly lines up: scope boundaries, confirmation gates so destructive operations can't complete on the agent's own authority, audit trails on every action, and a kill switch to stop it immediately. But the published versions seem to stop at the kill switch. Recovery doesn't get its own layer.

PocketOS lost the database and the backups together. That didn't happen because nobody was logging, and it didn't happen because nothing could stop the agent. My read is that what was missing was a fast way to put it back.

So I'll put the question out: does anyone have a better process for this, and when you design around it, does recovery actually get built in, or does it stay a backup-and-hope thing?

The Empty SMALLINT Column That Led Me to Nine More Missing Types

Derek Cai — Mon, 05 Oct 2026 13:20:37 +0000

I had written a stock market analysis program before. One of its tables had a SMALLINT column. When I tested LogCarver against that table, every row through it came back empty.

After finding that, I tested a batch of column types my own program rarely uses, plus a few types I had never specifically tested before. Turned out it wasn't just one missing type. Ten fixed-length types were missing: tinyint, smallint, bit, uniqueidentifier, money, smallmoney, datetime, smalldatetime, float, real. Any table using one of these lost that column's data, silently, no error, just nothing there.

Adding them back, same old method: go back to fn_dblog, check the byte format against a real instance, one type at a time. A few things turned out different from what I expected.

bit columns pack into a single byte, up to eight. I verified this by checking sys.system_internals_partition_columns: two different bit columns on the same table reported the identical leaf_offset. Their values sit in bit 0 and bit 1 of that one byte.

The old DATETIME type (not DATETIME2) stores time as 1/300-second ticks since midnight. The problem is 300 doesn't divide evenly into SQL Server's 10,000,000 tick resolution. Multiply before dividing versus divide first, and you get a real, measurable difference. I measured it at about half a second off.

While adding this batch, I also caught something unrelated to the new types: fn_dblog's own RowLog Contents columns cap out around 8000 bytes. A large enough in-row value can push a row past that cap. The old code treated that as corruption and dropped the whole row. Now it just flags the one column that went over and decodes the rest normally. All of this, plus the ten types, is in v0.1.12.

Source and more detail: https://github.com/caiderek/LogCarver

Full background on LogCarver, why I built it and how I reverse-engineered the fn_dblog format: How I Recovered Deleted SQL Server Rows Without Ever Enabling CDC or Audit (https://dev.to/caiderek/how-i-recovered-deleted-sql-server-rows-without-ever-enabling-cdc-or-audit-m10)

The bit packing, the DATETIME tick math, and the RowLog Contents 8000-byte cap were worked out with AI.

SQL Server / MySQL / PostgreSQL — the index differences

Derek Cai — Fri, 02 Oct 2026 18:58:05 +0000

I was reading an article comparing PostgreSQL and MySQL recently.

PostgreSQL: cool features, lots of knobs, but index writes are expensive.
MySQL: simpler to pick up, fewer knobs, cheaper index writes, but not as fast as PostgreSQL.

That got me thinking — I use both of these, so let me add SQL Server too, since that's the one I use the most.

SQL Server has two kinds of indexes: clustered and non-clustered.
Clustered index: wherever the index is, the data is right there. No extra step needed.
Non-clustered index: it first finds the clustered index's position, then goes to get the data. Or if there's no clustered index, it goes straight to a physical row locator, then gets the data. Either way, that's one extra step.

MySQL's InnoDB forces every table to have an index.
If there's a primary key, it automatically becomes the (clustered) index. No primary key? It picks a UNIQUE column for you. Nothing UNIQUE either? It generates a hidden row ID. Same as SQL Server: the clustered index is one step, but any non-clustered index still takes one extra step.

PostgreSQL is a completely different idea.
Every row has an address pointer called a TID. No matter what index you build, what gets stored is index value → TID. So querying an index always means: find the index entry, get the TID, then go fetch the data. That's one extra step too.

So why does PostgreSQL's write cost get bigger? Say this row has two indexes — one on ID, one on date+order number. You update the order number. The TID changes.
Now every index that was pointing at the old TID is stale, so anything tied to that TID has to get updated too.

Remember what I said above — a PostgreSQL index's value points at a TID. The TID just changed, so every index has to update along with it: ID → new TID, date+order-number → new TID.
So PostgreSQL ends up doing: update the row itself, plus update the ID index, plus update the date+order-number index. One logical update, three writes.

MySQL and SQL Server, though? They only update the row itself, and whichever index actually got touched. If nothing got touched, it's just the one row write.
In this example the update was to order number, which happens to be part of the date+order-number index — so it's: one write for the row, one for that index. Two writes total.
If the update had been to some unrelated notes field instead, it would just be the one row write.

That's why PostgreSQL's index write cost blows up the way it does.

How I Recovered Deleted SQL Server Rows Without Ever Enabling CDC or Audit

Derek Cai — Tue, 29 Sep 2026 05:57:28 +0000

Most data-recovery tools assume you turned on Change Data Capture, Change Tracking, or Audit before the incident happened. In the real world — small and mid-sized companies running their own SQL Server — almost nobody does that. By the time someone notices bad data, it's already too late to turn those features on retroactively.

SQL Server's transaction log already records every change. The problem is fn_dblog, the function that reads it, is almost entirely undocumented — the row image byte layout isn't published anywhere. The only way to figure it out was to run real experiments against a live database and reverse-engineer the format from actual output, byte by byte.

That's what became LogCarver: a tool that reads fn_dblog directly and reconstructs a table's full INSERT / UPDATE (before & after values) / DELETE history, with reviewable Undo and Replay SQL — no prior setup required.

Two bugs that only showed up on real data

After shipping it, I tested it against a perfectly ordinary table — one with no clustered index (a Heap table, which is also what you get from a PRIMARY KEY NONCLUSTERED). Result: 0 events found. Worse, the tool's own error message sounded reasonable — it said the VLF had probably been recycled and the data was likely gone.

That explanation was wrong.

Digging into fn_dblog's raw output, the real cause was much simpler and much worse: row operations on Heap tables are tagged with LCX_HEAP in their Context field, but my filter only recognized LCX_CLUSTERED and LCX_MARK_AS_GHOST. Every single Heap-table record had been silently excluded from the start — nothing to do with VLF recycling at all. If I hadn't gone back to the raw fn_dblog output, I would have believed my own tool's plausible-sounding, wrong explanation and moved on.

Fixing that surfaced a second bug: any table with any extra index — a NONCLUSTERED PRIMARY KEY on a Heap table, or just an ordinary index someone added for query performance — broke the table-matching logic, which guessed at the right AllocUnitName string pattern from the table name. An index's own internal maintenance records use a similar naming pattern, so they got matched too, mixing two completely unrelated pieces of internal SQL Server bookkeeping together.

The fix: stop guessing at string patterns. Query sys.indexes for the table's actual storage structure (index_id 0 = Heap, 1 = clustered index), build the exact AllocUnitName from that, and match exactly instead of loosely. The same missing index_id filter existed in the schema-reading query too — once fixed, extra indexes stopped polluting the column layout.

The rule that held throughout: every assumption gets checked against a real SQL Server instance, never just inferred from docs or intuition — because for fn_dblog, there mostly aren't any docs. None of these three bugs were found by reading documentation; all three were found by connecting to a real database and looking at what fn_dblog actually returned. I reran the same test matrix against SQL Server 2016, 2019, and 2022 the same day to confirm the fix held across versions before calling any of them "validated."

Current limitations: int, bigint, date, datetime2, decimal/numeric, char/nchar, varchar/nvarchar are decoded — a handful of less common fixed-length types (money, smallint, bit, uniqueidentifier, and a few others) are still on the list. TRUNCATE TABLE is currently (over-cautiously) treated like a schema change, which hides events before it; known, not yet fixed.

LogCarver is MIT-licensed: github.com/caiderek/LogCarver