DEV Community: Why Next

One Bucket, Two Terraform Owners - the Last apply Wins

Why Next — Sun, 19 Jul 2026 06:20:42 +0000

Originally published at blog.whynext.app.

It started as an ordinary cleanup problem. Users upload media files (recordings and images) through presigned URLs. The server issues an upload URL, the client uploads straight to S3, then calls a commit API to say "register this key as a real asset." The problem is what happens when someone gets a presign but never commits. The app crashes, the network drops, the user leaves the screen, and the bucket is left with an object that isn't registered anywhere.

I wanted a lifecycle rule to clean these up, but there was no way to write one. Committed and uncommitted objects were mixed under the same prefix, so any rule that says "delete old things" would delete real assets too. A daily upload quota kept the pile from growing fast, but the fact remained: there was no path to reclaim the space.

The design: what isn't committed lives in tmp

The backbone of the fix is key namespace separation.

presign issues a temporary key under the tmp/ prefix.
When commit passes validation (existence check via HEAD, Content-Type, size limit), it promotes the object to its final key with CopyObject and deletes the tmp original.
Objects whose commit never arrives stay in tmp/, and a lifecycle rule expires them after 7 days.

Now the lifecycle rule only has to look at tmp/. Real assets are outside its blast radius from the start. The clients didn't need to change. I read all three upload flows to confirm this: every one of them uses the key returned in the commit response for its follow-up calls, so the server can change the key shape without them noticing. Commits for old-format keys already in flight at deploy time still go through the existing path.

One trap here. This bucket has versioning enabled. On a versioned bucket, expiration doesn't delete an object. It only adds a delete marker, and the original bytes stay behind as a noncurrent version. Without a paired noncurrent_version_expiration (1 day), the cleanup runs and not a single byte is reclaimed.

The design review paid for itself too. Putting a review on the promotion logic surfaced three confirmed defects. A hole where an admin-only custom key path could commit a tmp/ key as the final key, letting the lifecycle rule delete a live asset (blocked tmp final keys at the shared commit boundary). The promotion's sourceKey acting as an arbitrary copy primitive (enforced the equation between tmp and final keys). And an idempotency problem where two concurrent commits of the same tmp key make one side's copy die with a 404. All three were fixed in code before moving on.

And then apply failed

I added the lifecycle rule in Terraform and merged. The staging apply failed.

The error was a timeout writing the lifecycle configuration, but digging in, the timeout was a symptom. The structure was the problem. Two Terraform resources each owned this bucket's lifecycle. One lived inside the image resize module (the module attached its own rule for its own purposes). The other lived in the per-environment bucket config file (where I had just added the tmp rule). Same physical bucket, two owners.

S3's lifecycle API has no "add one rule" operation. PutBucketLifecycleConfiguration replaces the entire document. Terraform's aws_s3_bucket_lifecycle_configuration sits on top of that, so the resource overwrites the bucket's whole lifecycle document with just the rules it knows about. With two owners, each keeps pushing a document containing only its own rules. If they meet inside one apply, the concurrent writes time out. If they meet separately, it's worse: the apply succeeds without any error, and the last winner erases the other side's rules.

That's the coldest part of this incident. The timeout was actually the lucky outcome, because it was a loud failure. If this structure had stayed alive, the next apply that touched the image resize module for any reason would have silently deleted the tmp expiration rule I had just added. The plan would have shown nothing but a one-line in-place update on that bucket resource, and uncommitted objects would have started piling up forever again. No monitoring anywhere would have noticed that a cleanup rule had vanished.

In fact, this exact spot had been flagged once during code review, as a suspicion that "these two resources seem to touch the same bucket." At the time it was filed as a potential risk. It took only minutes after the merge for potential to become real, when the apply failed.

The fix: one owner

The direction was clear. Each bucket must have exactly one lifecycle owner. I removed the lifecycle resource from the module and made the per-environment config file the single owner, moving the module's rule over as well.

The tool for this was Terraform's removed block. If you just delete the resource declaration from the module, Terraform tries to destroy the real lifecycle configuration. That opens a window where the rules currently live in production get deleted and then recreated. Give the removed block destroy = false and Terraform forgets the resource in state only. The real thing stays untouched while it disappears from the ledger, so ownership transfers to the environment config with zero downtime.

After the fix, the staging apply passed, and after rolling out to prod I checked the actual bucket state directly through the AWS API. Both buckets had the tmp expiration rule alive, and all three pre-existing rules intact. The reason I didn't stop at terraform plan is exactly what this incident taught me: a plan is just each owner's ledger, and in a document-overwrite structure, two ledgers know nothing about each other.

What it leaves behind

A document-level resource must have exactly one owner. Not just S3 lifecycle. Bucket policies, CORS configuration, notification configuration - every Terraform resource built on a "replace the whole thing" API has the same property. If declarations pointing at the same target exist once in a module and once in the environment config, you don't get a compile error. You get a silent overwrite at runtime. Before adding a new rule, first go find out whether that document already has an owner.

One more. Deleting a resource declaration and deleting the real thing are different acts, and Terraform expresses that distinction with the removed block. When the goal is an ownership transfer, the moment you go through destroy, zero downtime is gone.

Finally, a reclamation path starts with key design. When data with different lifespans - committed and uncommitted - lives in the same namespace, no cleanup rule can be written safely. If the lifespans differ, separate where they live first. For the record, this approach does not retroactively reclaim the uncommitted objects that already piled up. Those objects can't be distinguished from real assets (that was the starting point of this whole problem), so I deliberately chose to apply the scheme to new uploads only.

The Tests Were Green - the Notification Had Never Been Sent, Not Once

Why Next — Fri, 17 Jul 2026 12:10:43 +0000

Originally published at blog.whynext.app.

A 500 alert landed in Slack. The error message looked unfamiliar. Error: Custom Id cannot contain :. The API that admins use to post answers to user questions was dying, and the stack trace pointed at BullMQ's Job.validateOptions.

So far, an ordinary bug report. Where this story stops being ordinary is after finding the cause. The code throwing this error had not changed in that day's deploy. It had been failing every time, 100%, since the day the feature first shipped. Which means the push notifications this queue was supposed to send had never been sent, not once. And all that time, every test was green.

The cause was one character

The offending code looked like this. It's the spot where, when a question gets an answer, a job to push-notify the user goes into the queue.

await this.queue.add(ANSWER_NOTIFY_JOB_NAME, data, {
  jobId: `answer-notify:${data.questionId}`, // ← this colon
});

A custom jobId was given so the same question couldn't be enqueued twice, but the delimiter was a colon. BullMQ builds its Redis keys with colons, so it forbids colons in custom jobIds. In v5 the exact condition is: if the jobId contains a colon and splitting on colons does not yield exactly 3 parts, it throws. The 3-part exception is a carve-out for the legacy repeatable-job format, so a 2-part id like answer-notify:123 always gets Custom Id cannot contain :. For the record, purely numeric jobIds are also rejected with Custom Id cannot be integers, so dropping the prefix isn't the answer either.

Dozens of other queues across the codebase all used hyphens. Only this file used a colon.

The bug was there from day one - so why did it blow up today

The fix is one line: change the colon to a hyphen. The interesting question is elsewhere. Why did code that always failed stay quiet all this time and turn into a 500 today?

When the feature first shipped, the enqueue was wrapped like this.

try {
  await this.queue.add(/* ... */);
  return true;
} catch (err) {
  this.logger.error("enqueue failed", err);
  return false; // ← the failure disappears here
}

It logs the failure and returns false. The caller never checks the return value. So the exception that was thrown every time was swallowed every time, the API returned 200, and only the notification quietly evaporated.

Then a commit merged the same day changed this contract. The intent was: if the enqueue fails, don't swallow it, throw, so the "answered" flag written in the transaction gets rolled back and the operation can be retried. A change in the right direction. At that moment, the enqueue that had always been failing made a sound for the first time. The 500 wasn't a bug that commit created - it was the first scream of a bug that had been there all along.

This is the cost of code that swallows errors. The signal disappears, and the bug stays put, taking an entire feature with it.

Why the tests were green

This queue had tests. The usual kind of spec: call the enqueue and check that queue.add was called with the right arguments. And that test's queue.add was a mock that always succeeded.

Real BullMQ's add validates the jobId and rejects colons. The mock's add accepts anything. Exactly as much as the mock was more lenient than the real contract, the tests passed inputs green that could never pass in reality. When a mock diverges from the real contract, a green test certifies dead code, not living code.

While fixing the bug, I plugged this hole first. I changed the queue mock's add to run BullMQ's real Job.validateOptions. That function is pure validation that runs without Redis, so it drops straight into tests. Then I checked that the guard was a real guard with a mutation test. I put the colon back, watched the test turn red at exactly that spot, and only then committed. If you reintroduce the bug and the test is still green, that test might as well not exist.

It wasn't just one

Stopping here would have been fixing half the problem. If the same mistake exists in one place, it probably exists in others. I defined the class of defect - custom jobIds containing colons - swept the whole codebase, and put multiple review agents on it for cross-checking. Two more cases turned up.

One was the queue that transcribes user-uploaded audio to text. jobId: "stt:${id}". This one was still swallowing errors, so there wasn't even a 500. Checking the data showed that for the past 12 days, not a single transcription job had entered the queue. Audio files were saved normally, failures went only to the logs, and on the user's screen the text stayed empty forever. Completely asymptomatic.

The other was the queue for guide emails about scheduled events. Its jobId had four colon-separated parts, so neither reminders nor cancellation notices were entering the queue at all.

I unified the three producers on a single shared jobId builder. It joins with hyphens, replaces any stray colons, and rejects purely numeric ids. To prevent recurrence, I also added a lint rule that catches exactly the pattern of a colon literal inside a jobId. I did not force-migrate the roughly 50 existing hyphen jobIds to the new builder. Changing the jobId of a repeatable job risks duplicate cron registrations, so I chose to target only the defect class.

The second trap: a fixed jobId quietly kills retries

It would have been nice if the story ended there, but during review verification a more fundamental problem surfaced. Once the colon is fixed, a fixed jobId like answer-notify-123 comes alive, and that fixed jobId itself contradicted the retry contract established the same day.

When a job hash with the same jobId still exists in Redis, BullMQ ignores the new add without any error and returns the existing jobId. A verifier confirmed this behavior by reading BullMQ's Lua scripts. But this queue uses removeOnComplete: { count: 100 }, so the hash of a just-completed job stays around. Combine the two and you get this scenario. A notification job fails once. Per the contract, the "answered" flag is rolled back and an admin retries. The new enqueue hits the leftover job hash, does nothing, and returns as if it succeeded. The notification never goes out, and this time there's no error and no DLQ.

Remove the catch that swallowed failures, and the library's dedupe takes over the same role. In the end I removed the custom jobId from this queue entirely. Duplicate-send prevention moved to the consumer side: the processor atomically claims a "notification sent" flag with a conditional update. Even if the job comes in twice, only the side that grabs the flag first sends the push.

If you're preventing duplicates with a custom jobId and you also have a design that re-enqueues on failure, it's worth checking that the two mechanisms don't kill each other. You may have the same combination we did, where a retry quietly becomes a no-op.

What was the damage

A feature dead for 12 days sounds expensive, but the measurements said otherwise. The answer notifications lost zero events, because that feature wasn't in real production use yet. The email queue only covered past events, so there was nothing to resend. The actual recovery scope was 2 untranscribed audio files. The audio was still stored, so re-transcribing them restored everything.

We got lucky. And this luck does not repeat. Had the feature been in active use, 12 days of notifications and transcriptions would have been gone for good, and since the jobs were never created in the queue in the first place, no deploy would have recovered anything. Writing a backfill script was secondary. I judged that the real thing to fix was the fact that a dead feature was dead for 12 days without anyone knowing. So transcription failures no longer stop at the logs and now surface in the alert channel, and the answer-notification path moved to an outbox committed together with the transaction, eliminating the very state of "the DB recorded success but the job doesn't exist".

What it leaves behind

Three things from this incident that will outlast the code.

First, code that only logs in a catch and returns normally makes a path that fails 100% of the time invisible to everyone. If you're going to swallow, at minimum pair it with a metric or an alert that counts the failures, and if you don't have that, throwing is better. A thrown error is loud but gets fixed; a swallowed error lives for 12 days.

Second, mocks must be as strict as the real contract. On top of a mock that always succeeds, no input ever gets validated. If the library ships a pure validation function, run it in the mock, and confirm the test turns red when you reintroduce the bug - that's what makes it a guard.

Third, hunt defects by class. In the same codebase, the same mistake doesn't happen only once. Once you find one, define the class, sweep everything, and go as far as blocking the class from coming back - a lint rule, a shared builder - and one discovery earns its keep.

We Chased the 9 People Intercepting Our App's TLS. It Was Google.

Why Next — Mon, 13 Jul 2026 05:52:01 +0000

Originally published at blog.whynext.app.

Certificate pinning killed our app once. On April 5, 2026, we had pinned the SPKI of our leaf certificate. ACM auto-renewed the certificate, the leaf public key changed, and in that instant every user's API calls were blocked. We had no remote kill switch, so we had to push a hotfix build back through the stores. After that, the repo carried a rule: "Do not add certificate pinning."

Two months later, we built pinning again. This time we pinned the SPKIs of the four Amazon Root CAs instead of the leaf. Roots do not change when ACM renews, so the same accident cannot happen twice. We also added a kill switch we can flip from remote config. On June 29, we remotely enabled report-only mode, which reports mismatches without blocking anything.

Two days later, there was an issue waiting in Sentry.

[CertPinning] SPKI mismatch (report-only)
74 events · 9 users · 2026-06-29

Two hashes we never sent

Opening an event showed the two SPKIs of the certificate the app had actually received.

mode: "reportOnly"
host: "web.pronouncekorean.com"
expected_pin_count: 4
served_spki: [
  "sha256/9hqPsoMiyQMwLCoRPk6FoCYmOsPiGqzQqUcpuZIfvgs=",
  "sha256/r9mjYco6rQO8YkTqr/XXGsQlDUuQqz2mGr67S0imt7M="
]

I pulled the real server chain with openssl s_client and compared. Not the leaf our server sends, not the intermediate, not the root - none of them matched either hash. These 9 people never received our server's real certificate at all. Someone was terminating TLS in the middle, re-signing with their own certificate, and handing that to the app.

The fact that served_spki had only two entries was a clue too. On the normal path, the chain the app reports contains three certificates, all the way up to the root. Only two (leaf plus intermediate) meant this report came from a different branch in the code.

This is where I drew my first conclusion. To be precise, the AI agent drew it and I found it plausible. "This is not a bug to fix. Report-only pinning is working exactly as designed and catching interception happening on other people's networks. Corporate or school proxies, antivirus web protection, captive portals, that sort of thing. We cannot control any of it, so mute it in Sentry and just watch it as a metric."

Since we were in there anyway, a suggestion came along with it: add a backup pin. I picked that.

A backup pin would not remove a single one of these events

Right after the work on adding the pin started, something felt inconsistent, so I asked back.

Hold on, shouldn't a mismatch be impossible in the first place?

That one line collapsed the direction we had just chosen. The agent re-read the Android probe code and corrected itself.

val verified = verifiedChainSpki(offered, host)
if (verified == null) {
    // Trust store validation failed = interception/forgery -> report without any pin comparison
    report(host, offered.mapNotNull(::spkiSha256))
    return
}
// Pin comparison only happens for a chain that passed validation
if (verified.none(pins::contains)) { report(host, verified) }

The probe has two steps. First it validates the chain against the OS trust store, and only a chain that passes gets compared against the pin list. Our 74 events had already failed at the first step. With verified == null, the code reports immediately without ever running the pin comparison. Backup pins only matter at the second step, the case where the chain validates but the pin does not match.

Conclusion: adding Starfield G2 to the pins will not reduce this issue by a single event.

That was also the moment it became clear that a normal user cannot produce this report, because the live chain matches our pins exactly. So by definition these 9 people are not normal users. There was no bug to fix here. What we had to find out was who these 9 were.

Ignore it and we can never turn it on

I could not accept the "ignore it and monitor" recommendation either.

We have to fix this properly. We need to actually turn this on later, and if it stays like this we never can.

The end goal of this pinning work is enforce mode. The gate for deciding whether we can turn enforce on was already written in the design doc: two weeks with a real-user mismatch rate below 0.5%. If we leave the mismatches unexplained, these 74 events keep polluting the gate. Muting them makes the 74 disappear from the dashboard, but the gate stays polluted exactly as before.

So I dug into the data more. As soon as I pulled the tag distribution, something strange came up.

All 74 events came from the same device, and that device does not exist

The device tag on all 74 events was 100% a single value: OnePlus8Pro. But the fingerprint of that device made no sense.

Field	Event value	Real OnePlus 8 Pro
archs	includes x86_64, x86	arm64 only (Snapdragon 865)
Screen	288 x 448 px, 106 dpi	1440 x 3168, 513 dpi
CPU	2 cores, frequency 0	8 cores
simulator	false	-

Consumer ARM phones do not advertise an x86 ABI. There is no Android phone in the world with a 288x448 screen at 106 dpi. This was an x86 virtual device with only its Build.MODEL disguised as a real device name. It was even claiming simulator: false to pass as physical hardware.

The 9 user IDs were not people either. All 9 existed only on June 29, and in that single day they ran all three versions: 1.7.0, 1.8.0, and 1.8.1. That is not a person, that is a pipeline. The region on all 74 events was the US, but the city was null on every one of them. That is what traffic from a data center range looks like.

At this point I asked: so who is analyzing our app? The agent put security vendors' automated APK analysis sandboxes at number one, followed by app intelligence companies and APK mirror sites. Automated store review came in at number four.

The fourth-ranked theory became first with three IPs

Number four bothered me. So I asked.

Could it be Google Play Console or the App Store?

The answer that came back was firm.

Google Play pre-launch report / Test Lab - almost certainly not. The decisive reason: Google does not MITM your HTTPS. Play pre-launch uses the standard trust store and receives your real certificate as is. Store crawlers do not intercept and re-sign traffic. That is what an analysis sandbox does.

As pure reasoning, the argument is hard to fault. And it was wrong.

What was missing was IP evidence. Sentry scrubs IPs as PII, so user.ip was null on every event, and the original IP of a past event cannot be recovered. To identify the source we needed IPs, and the tool we were looking at did not have them.

What we did have was the production API's access log. It recorded X-Forwarded-For and User-Agent on every request, and our app's User-Agent was unique: Dart/3.12 (dart:io). And June 29 was still inside the 7-day log retention window.

First I pulled the event timestamps of all 74 mismatches from Sentry and grouped them into three clusters: 02:12-02:20, 11:30-11:41, and 17:32-17:40 (UTC). Then I queried CloudWatch for the IPs of requests that came in with Dart/3.12 during those three windows.

02:12-02:20  ...  66.249.84.132, 66.249.84.141
11:30-11:41  ...  66.102.7.69
17:32-17:40  ...  66.249.84.141

66.249.x.x is the range Googlebot uses, and 66.102.x.x is Google too. All three windows had a Google IP in them. In two of the three, the only app request in that window came from a Google IP. The remaining window also had an IP that looked like a real user, but the Google IP was in there as well.

Three minutes later, the agent corrected itself.

Correction. Earlier I said "Google does not MITM, so this is probably not them," but the IP evidence overturns that guess. This is Google. Measured IPs are stronger evidence than reasoning.

What was pulling our app apart was Google's routine automated scan, the one every app on Play gets. Not an attacker, not a competitor, not a security vendor. That scan environment instruments traffic. That is why, to our probe, the chain looked unverifiable.

So what did we fix?

We left the pinning alone and changed where the probe applies. We decided not to turn pinning on at all in scanner environments.

That left the question of how to recognize a scanner. It was already clear we could not trust simulator: false, so we needed a signal that cannot be spoofed. CPU architecture cannot be hidden.

// Decisive signal: a consumer ARM phone never advertises an x86 ABI.
// Google's scanner can spoof Build.MODEL with a real device name (observed: "OnePlus8Pro" on x86),
// but it cannot hide the fact that it is x86_64.
if (supportedAbis.any((abi) => abi.contains('x86') || abi.contains('i686'))) {
  return true;
}

On top of that we also check for Cuttlefish/GCE hardware names and test-keys fingerprints. The detector is deliberately biased toward answering "this is an emulator." A false positive just means we skip pinning in that one environment, and an attacker running the app on an emulator can patch the app to disable client-side pinning anyway, so we lose nothing. A false negative is far worse, because it traps a scanner behind a block screen.

Here is why that matters. If we had simply turned enforce on, this is what would have happened. Google's pre-launch crawler hits our "insecure connection" block screen. That gets recorded as a failure in the Play Console pre-launch report. Nothing is wrong with the code, and yet a red line shows up on the store review screen.

What stayed with me

Sentry scrubs IPs as PII. That is the right default. But because of it, the one field that could identify what was going on was gone from the event. In the end we cross-referenced two things, the app's User-Agent and the timestamps of the events, and recovered from the access log what Sentry had thrown away. Missing from one tool does not mean missing everywhere.

The sentence "Google does not intercept HTTPS" was a smooth piece of reasoning. It worked through the purpose of a store crawler, the behavior of the standard trust store, and the difference from a sandbox, and pushed the theory down to fourth place. Then three IPs flipped the conclusion in three minutes. Ask an AI about principles and you get principles back. Whether those principles match reality is something only the logs can tell you.

If we had muted these 74 events in Sentry, the dashboard would have been clean, and on the day we turned enforce on we would have gotten a red line in store review without knowing why.

We still have not turned enforce on. The gate requires two weeks with a real-user mismatch rate below 0.5%. Now that the scanner is filtered out, we have to count those two weeks again from the start.

We Got an Alert That the Database Was Down. The Database Was Never Down.

Why Next — Mon, 13 Jul 2026 05:51:59 +0000

Originally published at blog.whynext.app.

I opened Slack in the morning and found four CRITICAL alerts waiting.

504 AppHttpException: The request timed out.
  POST /customer/practice/attempts

PrismaClientKnownRequestError
  Transaction already closed

500 DriverAdapterError: canceling statement due to statement timeout
  GET /admin/dashboard/stats

DATABASE 서비스 다운
  "error": "Database health check (retry) timeout after 2000ms"

The last one is the scariest, because it means the database is dead. Put all four on one screen and a story writes itself. The database was struggling, so queries backed up, so requests timed out, and in the end even the health check failed. It is a believable story.

So I opened the database metrics first. The story fell apart on the first screen.

The database was idle for 21 straight hours

I pulled RDS metrics at 5-minute resolution across a 21-hour window covering the incident. 252 data points.

Metric	Min	Max
CPU	4.6%	19.7%
Connections	53	73
CPU credit balance	576	576
Read latency	0s	0.01s
Write latency	0.01s	0.08s

CPU never crossed 20%, and the credit balance has the same minimum and maximum, which means none of it was ever spent. Write latency peaked at 80 milliseconds. I checked RDS Proxy too. Connection borrow latency peaked at 26 milliseconds, backend connections at 36. Nowhere is there a number that would explain an 8-second or 10-second query.

The database was not dead. It was not even busy.

One request crosses the Atlantic twenty-five times

So where did the 8-second query come from? The architecture answers that.

Our service runs in two regions, us-east and eu-west. There is a single primary database, in us-east. Reads from the eu-west app are served by a local replica, but every write crosses the Atlantic and goes through RDS Proxy to the us-east primary.

The number of writes was the problem. I followed one practice-attempt save request (POST /customer/practice/attempts) through the code and counted. One asset upsert, three pre-aggregation reads, four queries in the learning attempt transaction, one re-fetch by the runner, two or three progress updates, about six in the streak transaction, two in the stats transaction, and two or three review-state updates. Roughly twenty-five, spread across three separate interactive transactions.

The logs had that exact number in them.

06:03:17  request_done  POST /customer/practice/attempts
          status=201  duration_ms=11695  query_count=25

25 queries, 11.7 seconds. And this request did not even fail. It held on for 11.7 seconds and came back a 201.

There is one thing I should be honest about here. I did not measure how many milliseconds a single one of those round trips costs. One code comment says the cross-region RTT is 220 milliseconds; another comment says 80 to 100 milliseconds per statement. Without checking which of the two is right, I wrote the calculation "25 x 220ms = 5.5 seconds" into the issue. Looking back, that is not a measurement, it is a quotation. The accurate statement is this: I counted the round trips (25), and I did not measure the cost per round trip. The conclusion still holds. Cross the Atlantic twenty-five times and the cost adds up in seconds.

During the EU morning peak (06:00 to 06:10 UTC), 57 primary writes took longer than 3 seconds. LearningAttempt.create at 8,989ms, UserActiveDate.upsert at 9,070ms, LearningAttempt.findUnique at 10,561ms. Requests pushed against each other and the round-trip latency stacked up.

Two of the alerts came out of that gap.

Alerts 1 and 2: a 5-second transaction could not survive 9.3 seconds

The streak update code looked like this.

// streak.service.ts:254
const changed = await this.prisma.primary.$transaction(async (tx) => {
  // ... six queries that cross the region boundary ...
});

There is no options object, so Prisma's default interactive transaction timeout of 5,000 milliseconds applies. Normally that is fine. As long as you are not crossing a region.

Transaction API error: A query cannot be executed on an expired transaction.
The timeout for this transaction was 5000 ms,
however 9326 ms passed since the start of the transaction.

9,326 milliseconds. That becomes P2028, and the request becomes a 504 at the gateway.

This code was the only unprotected one in our codebase. Every other transaction path goes through a withTxRetry helper, which raises the timeout to 8 seconds and treats P2028 as retryable. Only the streak update sat outside that umbrella, still using the 5-second default.

Alert 4: the health check walked through the very bottleneck it was watching

The scariest alert, "DATABASE service down," turned out to be the silliest one.

The health check code was well written. It sends a SELECT 1 with a 2-second timeout, and if that fails it waits 500 milliseconds and tries once more, and if that fails too it fires a CRITICAL alert. It even has its own dedicated connection pool (max 2) so it never mixes with the app's Prisma pool. The design intent was that the health check stays alive even when the app pool is exhausted.

But the path this probe takes to reach the database was not isolated.

// health.service.ts:139
// If DIRECT_DATABASE_URL is set, use it; otherwise fall back to DATABASE_URL

DIRECT_DATABASE_URL was not set. The fallback kicked in, and DATABASE_URL points at the exact RDS Proxy where the app's writes were piling up. In other words, the SELECT 1 fired by the eu-west health probe had to cross the Atlantic, go through the proxy that was congested right at that moment, reach the us-east primary, and come back. In under 2 seconds.

The code knew this. It leaves a warning behind whenever the fallback kicks in.

DIRECT_DATABASE_URL is not set; DB health checks will use DATABASE_URL
and may still share proxy bottlenecks.

That line was sitting right there in the logs. Nobody had read it.

The decisive evidence was that both regions failed at the same moment. At 12:02:17, the health check on a us-east task timed out. At 12:02:18, an eu-west task timed out too. If it had been a regional problem, only one side should have died. Both dying together means the cause is something they share, which is the proxy path.

What follows is a foregone conclusion. Both health check attempts failed, so /public/health returned 503, and the orchestrator decided the task was unhealthy and replaced it. Four minutes later a new task came up. The new task hit the exact same health check timeout as soon as it booted. Swapping a task does not make the Atlantic any narrower.

"DATABASE service down" was not a symptom of an outage. It was the monitor walking down the congested road it was monitoring and then reporting that it was having a hard time.

The admin 500 was a separate matter

This is where the urge to tie all four alerts into one story arrives. I tried to write it that way at first too. But the admin 500 had a different root.

When GET /admin/dashboard/stats computes the most popular words, it filters to the last 7 days. But the table that filter lands on (category_attempt) has no date column of its own. The date only exists on the joined learning_attempt side. So Postgres has no way to slice category_attempt down by date up front and has to scan the entire history. No amount of good indexing can make this filter selective.

slow_query path=/admin/dashboard/stats
           label=CategoryAttempt.groupBy dur=8009ms

statement_timeout is 8,000 milliseconds. The query took 8,009. It died by 9 milliseconds. This was a full scan that has nothing to do with regions and was always going to blow up eventually as the table grew.

Three of the four alerts shared one root, and one did not. Four alerts landing on the same screen does not mean they have the same cause.

One theory we rejected

There was one plausible suspect. The device-cleanup cron that runs just before the 06:00 burst could have caused lock contention. The timing lined up nicely, so it was fairly convincing.

I went through the logs from both regions across the 05:30 to 06:15 window. There was not a single cron-related log line. Checking the path of the slow Device.updateMany, it was not the cron's bulk update but /v2/public/auth/refresh, an ordinary user's token refresh. The theory was rejected by measurement. Overlapping timing is a correlation. Not being in the logs is a fact.

We folded the round trips

There are two kinds of prescription. One endures the symptom: raise the timeout to 8 seconds, grow the pool, add retries. The other removes the cost itself: cut the number of round trips.

We did both. Wrapping the streak transaction in withTxRetry is a bandage, but it was a bandage we needed. The real prescription, though, was folding the round trips away.

The tool for it already existed. A migration dated June 27 contained a server-side function called create_practice_attempt_v1. It folds session locking, lookup, insert, session metadata update, and the pre-aggregation read into a single SQL function so the whole thing finishes in one round trip. It is even idempotent: call it twice with the same client attempt id and the second call returns the existing row along with inserted=false.

But the feature flag that turns this function on, PRACTICE_ATTEMPT_SINGLE_RT_ENABLED, defaults to false, and it was off in production.

Instead of removing a region or moving to active-active, what we did was flip a switch that was already built.

The canary: what do you judge it by?

We turned the flag on in eu-west only, leaving us-east off as a control group. Task definition went from 786 to 787 and we did a rolling deploy.

Choosing what to judge by is the heart of this story. Latency is a trap. Deploy during a quiet traffic hour and p95 improves even if you fixed nothing. And latency did improve. p50 went from 2,784ms to under 2,000ms. But that is weak evidence.

What I looked at instead was queries per request.

before  n=85   avg query_count = 22.0
after   n=49   avg query_count = 15.3

Query count is a per-request metric that does not depend on load. Whether traffic is heavy or light, the number of queries one request fires is decided by the code path. It went from 22 to 15.3, a drop of about 7. And those 7 match exactly the number of queries we folded into the function (session lock + lookup + insert + session metadata + pre-aggregation read). That number is the evidence that the code path actually changed. Not the latency.

The post-deploy sample is 49 requests over 15 minutes. There were zero 5xx and zero P2028 in that window, but I did not measure a "before" error count over a window of the same length on the same day, side by side. This flag only folds the attempt save. The streak transaction still crosses regions. Zero P2028 in that window means we were lucky, not that anything was proven.

The deploy worked, and the flag almost vanished

The way we ran the canary had a cost. 787 was a task definition registered by hand in the console. The next regular deploy would have the CD pipeline overwrite it with a task definition that has no flag. Not fatal, since overwriting it just falls back to a safe OFF. But the improvement would quietly disappear.

So it had to be promoted into terraform, and that is where the second trap appeared. The terraform workflow applies staging only on a develop push, and applies production only on a main push. And staging has no eu-west region to begin with. In other words, merging a PR that carries a production eu-west change into develop does nothing at all. We had to switch the base to main.

There was a third trap. Merging straight into main leaves develop behind, and if you do not merge it back, the next release reverts the change. We had to open a separate back-merge PR.

"terraform apply succeeded" and "it is live in production" are two different sentences.

What stayed with me

When a "database down" alert arrives, open the database metrics first. The name on an alert was chosen by the code that fired it, not by the cause. Our health check went as far as building a dedicated pool isolated from the app pool, and it still got stuck alongside the app, because the road to the database was the same one. Isolation is not only about splitting resources. It has to split the path too.

Raise the timeout from 5 seconds to 8 and that day's alerts stop. But the request still crosses the Atlantic twenty-five times. Grow traffic a little more and 8 seconds will not be enough either. It is better to decide up front whether you are enduring the symptom or removing the cost. If the cost is "count x distance," the thing to cut is the count.

Latency moves with load; per-request metrics do not. Before you deploy a performance fix, see p95 improve, and relax, check whether the amount of work one request does actually went down. That number is the same whether you deploy at dawn or at lunchtime.

I counted the round trips, and I borrowed the per-round-trip latency from a code comment. Those two comments disagreed with each other. The conclusion did not change, which is lucky, but next time the conclusion might. If you did not measure a number, write that you did not measure it.

What Happened When I Let Several AI Agents Loose in One Repo

Why Next — Sun, 12 Jul 2026 06:19:39 +0000

Originally published at blog.whynext.app.

Work with AI agents for a while and the ambition comes naturally. While one session fixes a bug, another can refactor, and a third can investigate an issue, right? You can spin up as many models as you like, so productivity should scale to match.

That's how I started too. And within a week I learned that the real enemy of parallel agents isn't the models' skill. It's the working directory they share.

HEAD is a global variable

The cause fits in one sentence. When multiple sessions share a single git checkout, the current branch becomes everyone's global variable.

Picture two people working on one computer at the same time and the absurdity is obvious, but that thought never occurred to me while spinning up agents. With one session per terminal tab, they look isolated from each other. But there is one filesystem, and one HEAD. The moment one session runs git checkout, the ground shifts under every other session.

The incidents from that week fell into clear types.

Branch hijacking. While session A was working on a topic branch, session B switched branches to do its own work. A committed without knowing, and the commit landed on top of B's branch. It happened in the other direction too: right as A was about to commit, the branch had been switched to develop, and only the hook that blocks direct commits to protected branches saved it. Without the hook, it would have gone straight in.

Orphaned commits. Session B deleted session A's topic branch during a cleanup pass. A's commits became orphans belonging to no branch, and I dug through the reflog, found the commit hashes, and recovered them with cherry-pick. Lucky that it worked; if the reflog had expired or I hadn't found them, the work would have simply evaporated.

Staging contamination. At the moment session A was creating a commit, a file deletion that session B had staged was sitting in the staging area alongside it. Committed as-is, B's deletion would have been folded into A's commit. It only got filtered out because an unfamiliar change showed up while skimming the diff. If the agent hadn't looked at the diff right before committing, nobody would have known.

Duplicate implementation. The most deflating type. Two sessions, each unaware of the other, implemented the same feature independently. Both did a diligent, solid job, and one had to be thrown away wholesale. The time saved by parallelizing was handed straight back.

They even trip each other's verification gates

Files and branches weren't the only problem. Our harness has a gate that runs the repo-wide static analysis and tests before a session can finish, so work can't end in a broken state. For a solo session, it's an excellent mechanism.

With parallel sessions, it became a trap where they trip each other. Session A only touched documentation, but A's gate fails because of a compile error in a file session B is midway through fixing. A burns turns proving "this isn't my change," and on a bad day waited 30 minutes for B to clean up. The worst incident was a session modifying code that another session was working on, just to get its own gate to pass. The gate itself became an incentive to touch someone else's work.

The check isn't what's wrong. The check's scope being "the whole repo" became wrong the moment the repo stopped belonging to one person.

Three lines of defense

After going through the accidents type by type, I put up three layers of defense.

Worktree isolation as the default. Give each session its own independent working directory with git worktree and HEAD is no longer a shared variable. Branch hijacking, orphaned commits, and staging contamination disappear at the root. Three of the four accident types above vanish with this one move. It isn't free, though. In a monorepo, every worktree needs its own dependency install and code generation, and in a repo using tools that get along badly with worktrees, like git-crypt, it's hard to enforce. We keep one repo on a shared checkout because of exactly that constraint, which is why the other two defenses exist.
Re-check the branch right before committing. When a session starts work, it records "I am working on this branch," and right before committing it compares that against the current HEAD. If they differ, stop the commit and figure out what happened first. The rule is almost embarrassingly simple, but every branch-hijack incident boiled down to "the HEAD at commit time wasn't the HEAD I knew about." A human would notice the branch name sitting in their prompt; an agent doesn't check unless explicitly told to.
Scope the verification gate to my changes. Narrow what the session-exit gate inspects, from the whole repo down to the files that session actually modified. No more getting my exit blocked by someone else's WIP, and no more incentive to touch someone else's code just to get the gate to pass. The health of the whole repo is something CI re-checks against committed state anyway. The session gate never needed to look at everything.

On top of these, one operating rule. Before spinning up a session, skim the currently open branches and PRs and check for overlapping scope. Duplicate implementation isn't a problem git can block; it's a dispatching problem, so the only way to prevent it was habit, not tooling.

Concurrency problems were never just a database thing

Laying it all out, the picture looks familiar. A shared resource, lock-free concurrent access, race conditions, and isolation levels. The exact problem we spent decades learning about in databases and multithreaded code, replayed on top of a working directory.

With a single agent, the problem doesn't exist. Even when a human and an agent take turns, the human implicitly plays coordinator. The trouble starts the moment there are multiple agents and the human lets go of the coordination. From that point on, the working directory is a shared resource that needs concurrency control, and if you run it without isolation, you will lose data, just as databases did.

This is not an argument for giving up on parallel agents. I still spin up multiple sessions every day. Only one thing changed: I no longer cram them all into the same room.

A Pipeline Where a Second AI Tries to Disprove the First One's Fix

Why Next — Sun, 12 Jul 2026 05:02:07 +0000

Originally published at blog.whynext.app.

Anyone who has had an AI agent do code review knows this. It's pretty good at other people's code. But have it review the code it just wrote and it turns strangely generous. You get "the implementation matches the intent" back, and when you actually run the thing, the bug is still there.

Which makes sense. Within the same model and the same context, whatever misconception it had while writing the code, it has again while reviewing it. It's a structure where it re-approves its own conclusions, so it's less a review than an echo.

So I moved the judging outside. I hand the code Claude fixed to Codex, and I ask for exactly one thing: "Assume this fix is wrong, and try to refute it."

Three rounds in a row, each found a deeper bug

The first time I really felt the value of this loop was a logout bug. The symptom was simple: pressing the logout button sometimes didn't log you out. Claude found a cause and fixed it, and reading the code, it looked plausible. I almost merged it.

I had Codex attempt a refutation, and round one came back with this: the fix itself is correct, but a redirect on the auth page intercepts the flow before that code path is ever reached, so the fix never gets a chance to run. The fixed code was dead code.

I fixed the redirect too and asked for another refutation. Round two found something else. This time it was an in-progress flag that had been added to prevent duplicate execution. At certain timings, that guard turns the logout request itself into a no-op. Only in round three did I get "I couldn't find anything to refute," and only then did I merge.

All three rounds involved code that had passed self-review. And in all three rounds, the refuter caught the problem because it wasn't asking "does the code look good" but "is there a scenario where this claim collapses."

The pattern kept repeating after that. On a server fix that had been through self-review twice, cross-checking still caught one missing value in a DB constraint, and a migration got added. On a retry-logic fix, the refuter sent back a blocking verdict over an edge case that misclassified a particular timeout response as a permanent failure. Every one of them was the kind of thing that was plausible enough to almost slip through.

Ask for refutation, not review

The most important part of the prompt design is framing. Say "review this" and the model gives you a safe answer, half praise and half minor nitpicks. Say "assume this fix is wrong and refute it; pass it only if you fail to refute it" and the posture changes. Passing stops being the default and becomes the consequence of a failed refutation.

On top of that, three more conditions.

Enforce an evidence format. Every finding must carry a file, a line number, and a reproduction scenario where the defect actually fires. "This part looks risky" is not accepted. If it can't construct a scenario, that suspicion isn't a finding, it's dismissed. In practice the refuter fairly often disproves its own suspicion and drops it, and that's the key mechanism filtering out false positives.
Get the verdict as a grade. Pass / pass after minor fixes / block - three levels are enough. With grades, "there are findings but it's mergeable" and "do not merge" don't get mixed together.
Run it in an independent context. The refuter gets only the diff and the relevant code. It does not get the conversation from the work session, or the narrative of "why it was fixed this way." The moment you share the narrative, the refuter catches the same misconceptions.

One operational tip as well. Run the refuter read-only. Give it permission to change code and it will try to fix things itself instead of pointing them out, and then you're back to the question of who verifies that fix. Keep the role pinned to judging, and send fixes back to the original worker.

When to run it

I don't run this loop on every change. Calling in a refuter for a typo fix is a waste. What always gets the loop is changes that are hard to undo. DB migrations, data deletion, payment and auth paths all live here. Changes that carry a "fixed it" claim are also targets. A bug fix needs a matched pair, the bug reproduced before and gone after, and the refuter is good at finding the gaps in that pair. The last category is changes I'm not going to read in full. Having a human read every line of a large AI-generated diff realistically breaks down. If I'm not going to read it, the least I can do is have a different model read it adversarially.

The cost question has to be addressed, and here it is: one round of refutation costs a few model calls. Compared with the cost of a human chasing down one bug that shipped to production, it wasn't worth agonizing over. If that logout bug above had made it to a release, I would have had to dig through both layers myself, the redirect and the flag guard, armed with nothing but reports that "sometimes logout doesn't work."

Why it has to be a different model

A fresh session of the same model gets you part of the way. Just separating the context makes the narrative-contamination problem go away. But after months of running this, my conclusion is that a genuinely different model clearly catches more.

Different models attend to different things. One is sensitive to timing problems in state management while the other is sensitive to contract violations and boundary values, that sort of thing. Two sessions of the same model share the same blind spots, but different models have blind spots that don't line up. The value of cross-checking comes precisely from that misalignment.

This structure should feel familiar. It's the same reason human teams assign code review to someone other than the author. Working with AI doesn't make that principle disappear; it just became cheap enough to apply all the time. A judge who isn't the author, eyes not steeped in the narrative, and an approval that only arrives once refutation has failed. What human organizations did only occasionally because it was expensive can now stand guard in front of every merge.

I Stopped Trusting the Agent's "Done" - prove-it, a verify.sh Gate

Why Next — Fri, 10 Jul 2026 03:05:12 +0000

Originally published at blog.whynext.app.

"All tests pass. Ready to merge."

The agent said that, and the diff was clean enough that I almost believed it. Then I scrolled back up the terminal, and the tests had never run. There was no execution log at all. The agent had intended to run the tests, and that intent got written straight into the completion report.

After this happens a few times, the reaction is usually one of two things. Add "you MUST run the tests and show the output" to the prompt, or re-check everything by hand every time the agent says done. I tried both. Neither lasted. So I took a third path. Instead of learning to distrust the word "done," I turned done from a declaration into a check that has to pass. That became an open-source tool called prove-it.

It's not lying, it's structure

First thing to get straight: the agent is not lying.

Humans can tell "I ran the tests" apart from "I meant to run the tests," because we have memory. The agent has nothing to compare against. It has no way to check what it did against what it intended, so it reports intent instead of results. "Tests pass" is shorthand for "I wrote the code so that the tests would pass." That's a design property, not a character flaw - which is why prompts can't fix it. Write "you must actually run them" a hundred times, and a day still comes when it doesn't.

Prompt techniques have another weakness: they rot with every model generation. The fact that an instruction works on this model is no guarantee it works on the next one. A mechanism that demands evidence, on the other hand, sits one layer above the model and survives upgrades. Whether the tests actually ran is something an exit code can tell you - the model doesn't have to.

Turning reports into checks

The idea itself - accepting "done" only as evidence, never as a claim - is something I covered in an earlier post. This post is the story of turning that idea into a tool anyone can install.

The prove-it contract is one sentence. The repo declares how to prove itself in a file called verify.sh, and the agent cannot claim done until that proof passes.

The implementation looks like this. Put an executable verify.sh at the repo root. Exit 0 means "this tree is provably fine." When the agent tries to end its turn, a Stop hook runs the script, and if it's nonzero, instead of letting the agent stop, it sends it back to work along with the last twenty lines of the failure output. Most of the time, that output alone is enough for the agent to fix the cause.

When the gate fires is just as simple. This session actually changed this repo, an executable verify.sh exists, and the current tree state has never passed before. It only runs when all three are true. Read-only sessions are left alone, and a tree that has passed once is never re-checked.

And it doesn't block forever. It sends the agent back at most three times per turn, then yields - because a hook that never yields stalls the session. But yielding is not passing. When a turn ends in a yield, the last output is not "done" but a warning: "this turn ended unverified." The agent may never get past the gate, but it can never get past it quietly - and that is the entire claim this tool actually makes.

Why five lines of bash won't do

If you've read this far and thought "that's just one test command in a Stop hook," you're right. Five lines will do it, and that's how I wrote it the first time. That version failed silently in four ways. Some of these were real bugs in early versions of prove-it, so each one now has a regression test pinned to it.

It blocks once, then never again. Claude Code sets the stop_hook_active flag on every stop event after a hook has blocked once. A hook that reads this flag as "let it through" stops being a gate after exactly one block. A hook that ignores the flag blocks forever and stalls the session. It's a trap you only learn by hitting both sides, and the answer is to count attempts yourself, block a fixed number of times, then yield loudly.

Commit, and it looks like nothing happened. A hook that decides "was there work?" by whether the working tree is dirty waves through every turn that ends in a commit - and committing is the most ordinary thing an agent does. So prove-it records the tree state at session start as a baseline and compares against it on every stop. Commits, files patched with sed, files spat out by a code generator - all of it counts as change.

The agent can remove the check. The cheapest move available to an agent that can't pass verify.sh is to delete verify.sh, or strip its execute bit. The gate records whether the repo was armed at session start and rejects any turn that ends disarmed. There's a subtler move too: leave it executable and rewrite the checks inside. That one is not blocked - editing verify.sh is often exactly the work you asked for. But it doesn't slip by quietly either. If verify.sh changed during the session and then passed, the gate tells you so, and reading that diff tells you whether it was work or evasion.

Giving up is indistinguishable from passing. Any host eventually forces a hook to yield. A hand-rolled hook yields in silence, and the last thing you see is "done." A turn that failed three times and gave up looks identical on screen to a turn that passed on the first try. prove-it's last word is a warning. If that difference seems minor, picture the day after you merged a given-up turn believing it had passed.

Those four cases are the entire reason prove-it is longer than five lines. If you want to keep using your own hook, go ahead - but the cases it has to handle are written down in SPEC.md, and I'd recommend reading it. What matters is the contract, not the implementation.

What this tool does not prove

There's a limitation worth stating honestly. The gate enforces exactly one thing: verify.sh returned 0 before the turn ended. What that 0 means is entirely up to the checks you wrote. A verify.sh containing a single exit 0 passes this gate and proves nothing.

The spec calls that Level 1. Level 2 is whether the checks assert against real evidence, and no tool can verify that for you - prove-it included. Wiring up the hook is the easy part. The real work is answering "what does proven mean in this repo," and the answer differs per repo. That's why the contract fixes only the file name, never the contents.

The escape hatches are deliberate too. PROVE_IT_SKIP=1 lets that turn straight through, and deleting verify.sh turns the gate off for good. A gate that can't be removed is a gate people eventually route around, and a bypassed gate is worse than none - because it reports that the checks ran when nothing ran at all.

Deliberately small on day one

Installation is three lines, and the third one does the actual work.

/plugin marketplace add Why-Next/prove-it
/plugin install prove-it@whynext
/prove-it:init

/prove-it:init detects your stack, generates verify.sh, shows you it passing right in front of you, then runs a copy with an appended exit 1 to show the gate rejecting a turn. Takes about thirty seconds.

The generated verify.sh has exactly one active check: git diff --check. Tests, type checking - all of it is there only as comments. That's deliberate, so it passes on main the day you install it. A gate that fails on day one teaches the team to bypass it in week one. Enable the commented checks one at a time, after running each by hand and watching it pass. And before enabling one, make it fail on purpose at least once. A check that cannot fail is not a check, and finding that out on the day you need it is too late.

There's also a defined point to stop growing verify.sh: one minute total. Anything slower goes to CI. So does anything that needs secrets or production access. verify.sh keeps only the evidence the agent can produce locally, mid-task.

A ledger of what the gate caught

Turn on PROVE_IT_LEDGER=1, and every time the gate catches a false completion, one line lands in a local file.

{"ts":"2026-07-09T04:12:33Z","claim":"All tests pass. Ready to merge.",
 "evidence_demanded":"verify.sh exit 0","actual":["3 failed, 41 passed"]}

What the agent claimed, what was demanded, what was actually true. After a month of these, you can read how your agent fails from a record instead of a guess. The file stays on local disk, is never sent anywhere, and is off until you turn it on.

This repo gates itself

The prove-it repo has its own verify.sh, of course, and inside it the gate tests itself against real git repos. Does a failing check block? Does a passing check open? Does a read-only session go untouched? Does a clean tree after a commit get mistaken for "no work"? All four traps above are pinned down as regression tests.

It's MIT licensed, and the only dependencies are bash, git, and python3. It's tested with Claude Code, and the wiring for hosts that expose the same kind of blocking hook - Codex CLI, Gemini CLI, and others - is in the ADAPTERS doc. If you have a repo where an agent edits code and says "done" in the same conversation, that's where this gate is most useful.

"Done" should be a state the repo adjudicates, not a word the agent declares. Writing the criteria for that judgment is still your job - but at least now, nobody gets to skip the judgment and slip past quietly.

When the Agent Says "It's Done," Don't Take Its Word for It

Why Next — Thu, 09 Jul 2026 05:14:06 +0000

Originally published at blog.whynext.app.

When you work with an AI coding agent, there's one sentence that fools you more than any other: "It's all done."

The agent says this far too easily. It says "the tests should pass" without actually running them, and says "I fixed the root cause" without ever reproducing the bug. Read the code and it looks plausible. But run the commands for real and about half the time it isn't finished. After living with this for a long time, I came to a conclusion. The problem isn't that the AI lies. The problem was that I, a human, was deciding every single time whether to believe what it said.

A problem of proof, not trust

Working alone, I write almost all of my code with AI. That means dozens of times a day I decide "can I accept this change." At first I read the agent's report each time, and when something felt off, I ran the commands myself to check. The trouble with this approach is that the checking depends on me. On a tired day, a rushed day, the tenth time I'm doing a similar task, I let it slide with "it says it's done, so it's probably done." And it's exactly the ones I let slide that blow up.

So I changed direction. Instead of me doubting every time, I made the system demand evidence. I built a gate into the harness so that the agent saying "done" is not enough to end the turn. Done became a check you have to pass, not a declaration.

The root of this idea is simple. The more cheaply AI can churn out code, the more the bottleneck shifts from producing it to verifying it. When the cost of making code approaches zero, "trusting that it's correct" becomes the relatively most expensive work. And if it's that expensive, it shouldn't be left to a human's mood in the moment. It should be an automatically enforced procedure.

A hook that blocks the exit

Concretely, I use a hook that steps in at the exit point. When the agent declares the work finished and tries to end the turn, a check runs right before that. The check demands one thing: "show me the evidence that it's done."

If there's no evidence, the exit is refused. The agent is sent back, with a message saying "you haven't proven it's done, so keep going." Then, and only then, the agent runs the tests, executes the commands, and pastes the output. The funny part is that in this process, about half the time the agent itself says "oh, this isn't actually done yet" and goes on to finish the work that was left. It hadn't really been finished when it said it was.

The key is that this gate is a machine, not a person. Whether I'm tired or rushed, the gate is exactly as strict. My discipline wavers, but the hook's discipline does not. It turns verification from a habit into infrastructure.

What counts as evidence

So what do I accept as "evidence"? I use four kinds.

Command output. If the tests passed, there has to be output from the actual test runner. Not "should pass," but a log that passed. If the build succeeded, a build log; if lint is clean, the lint output.
diff. What was changed has to show up as a diff. Not "I fixed this function" in words, but the changed lines themselves. Whether the scope of the change matches the report is caught here.
Reproduction. If a bug was fixed, there has to be a pair of evidence: that the bug actually reproduced before the fix, and that it disappeared after. Without before/after, "fixed" is just a guess.
Cross-review. For important changes, I have a different model do the review. Codex looks at what Claude wrote, and they point out each other's blind spots. It catches far more than one model judging its own work.

What these four have in common is that they are all reproducible facts. Not opinions, but output you can paste in. This is why AI can't be the final judge. Setting the bar for "good," and deciding which output meets that bar, ultimately stays a human's job. It's just that instead of making that judgment by hand every time, you freeze the judging procedure into code.

Why this is good for the human too

After setting up the gate, there was an effect I didn't expect. It wasn't only the agent that got disciplined - I got easier too.

Before, my job was to read the report and doubt "is this really done." Now the hook does that job. The work that reaches me already has evidence attached. I review the evidence; I don't first check whether evidence exists. The starting line of judgment has been moved one step forward.

And this is also a record I leave for my future self. Why a given change was safe stays recorded alongside the command output from that moment. A month later, when I wonder "why did I do it this way," the rationale is sitting right next to the commit.

The one new muscle to build

People talk about all sorts of skills for working with AI, but the biggest thing that stuck with me is this: don't rely on yourself for verification - delegate it to the system.

The skill of writing good prompts goes stale when the model changes. A trick that worked yesterday doesn't work on the new model. But the discipline of "only accepting done as evidence" is a layer above the model, so it doesn't go stale. Whatever model you use, however many generations the models go through, a gate that demands evidence stays useful. If anything, the smarter the model gets, the more plausibly it gets things wrong, so the value of the gate goes up.

AI has taken the burden of writing code off our hands. What it left us in return is the work of judging "can I trust this." If you try to make that judgment by human willpower every time, you get worn down and things leak through. But if you set up a machine to demand evidence, one tireless gate stands there being strict on my behalf. The more you have AI do for you, the more the thing worth investing in is not "how do I ask" but "how do I verify the claim that it's done."

"Just Turn It Off in Staging" — What the Noisy Alert Was Actually Hiding

Why Next — Wed, 08 Jul 2026 08:41:42 +0000

Originally published at blog.whynext.app.

The same alert showed up at the same time every day: an ERROR saying the subscription reconciliation job had failed in staging. The cron ran at 3:17 AM, and three minutes later the alert landed. The stack trace always opened with the same line: Google Play Voided Purchases API returned 404. The job retried three times, died on 404 all three times, dropped into the DLQ, and left one more red alert piled up in our Slack.

This wasn't an incident. Nothing was broken. It was just noisy. And in the process of deciding what to do about that noise, I relearned something: turning a thing off and actually fixing it are not the same problem.

Why the 404 was happening

The Voided Purchases API is a Google Play endpoint that returns purchases which were refunded or canceled. We reconcile against it daily to catch people who get a refund quietly while continuing to use the app. But a 404 from this endpoint doesn't mean "no data." For voidedpurchases.list, a 404 means the application (package) can't be found, or the service account has no access to that package at all.

Our staging service account has no Play Console access to the production package - and there's no reason it should. Staging isn't an environment that actual store refund traffic flows through. So calling this API from staging produces a 404. Today, tomorrow, forever, unless the configuration changes. This isn't a blip that clears itself up; it's a permanent condition created by the environment.

That's where the real distinction lives. For a transient failure, retrying is the right call - if the network hiccups or the other side briefly returns 5xx, calling again a few seconds later can work. But a permanent condition doesn't improve with retries. Permissions the staging service account doesn't have aren't going to appear three seconds from now. So retrying this 404 three times was pure waste, and the ERROR alert at the end of it was pure noise.

Why the code was retrying in the first place

The root cause was in how the client classified responses. listVoidedPurchases was throwing every non-200 response through a single path.

if (!res.ok) {
  throw AppHttpException.serviceUnavailable(
    `Google Play Voided Purchases API returned ${res.status}`,
  );
}

serviceUnavailable is the error our queue reads as "retry this." So whether the status was 404, 500, or 429, everything that came out of here got retried three times and ended the same way: an alert. The problem is that these three statuses mean completely different things. 500 and 429 mean "not right now, but maybe soon." 404 means "not under this condition, ever." The moment they're merged into one exception, an error that retrying can never help climbs onto the retry pipeline anyway.

What's interesting is that the right answer already existed in the same file. getSubscriptionPurchase, on the same client, had long since split out 404 on its own.

if (res.status === 404) {
  throw new GooglePlayApiError(404, "purchase not found");
}

One method respected what 404 meant. The other flattened it away. The precedent was sitting right there in the codebase - the new method just didn't follow it. This kind of asymmetry usually shows up when methods get written on different days, with different things on the author's mind.

The fix: split it at the boundary

The fix landed on two layers, and both followed a pattern that already existed.

First, we split 404 out at the client boundary. A 404 now throws GooglePlayApiError(404), carrying the meaning "not found," while everything else - 5xx, 429, timeouts - still throws serviceUnavailable as before. Only the errors that genuinely warrant a retry stay on the retry path.

if (res.status === 404) {
  throw new GooglePlayApiError(404, "application not found");
}
if (!res.ok) {
  throw AppHttpException.serviceUnavailable(/* 5xx, 429, timeouts - genuinely retryable */);
}

Next, we decided how the reconciliation service should treat this 404. The service already had a flow for "no service account key, skip quietly" (isConfigured === false). A 404 is, at its core, the same situation: this is an API you can't call in this environment. So it got routed to the same place.

try {
  const voided = await client.listVoidedPurchases();
  // ... proceed with reconciliation
} catch (e) {
  if (e instanceof GooglePlayApiError && e.status === 404) {
    this.logger.warn("Voided purchases 404 - not reachable in this environment, skipping");
    return { skippedAppNotFound: true };
  }
  throw e; // anything other than 404 is re-thrown to retry normally
}

The catch block swallows only 404, logs a warning, and returns a skip count. Since it doesn't re-throw, there's no retry, no DLQ, no ERROR alert. Any error that isn't 404, on the other hand, gets re-thrown and the original retry logic still runs as before. Only the noisy part goes quiet; the real problems stay loud.

We backed this with tests: does the client throw GooglePlayApiError on 404, does reconciliation skip on 404, and does anything other than 404 still get re-thrown. That last case matters most. If you quiet a 404 and accidentally swallow a 500 along with it, you lose visibility into an actual outage.

"Can't we just turn this off in staging?"

After the PR went up, someone asked a good question: why run this job in staging at all? The instinct is sound - staging has no real refund traffic, so there's no reason to be calling this API in the first place. So we looked into it.

Turning it off was less trivial than it sounded. The natural switch for disabling this feature in staging would be whether a service account key is configured (isConfigured), but that key isn't dedicated to this one job. It also gates receipt validation for in-app purchases, handling Google's real-time developer notification (RTDN) webhook, and the subscription reconciliation processor - subscription verification as a whole hangs off the same key. To test payments and subscriptions in staging at all, that key has to be present. Pull it to kill this one job, and you kill the very things you'd want to test.

Fine - what about a dedicated toggle just for this job? No such toggle existed. Building one would mean introducing a new switch like GOOGLE_VOIDED_RECONCILE_ENABLED. In other words, even "just turn it off in staging" turns out to require touching code and environment variables. There was no free way to disable it.

Turning it off and fixing it are different problems

But the real point was never how hard the toggle would be to build. Even if adding it had been trivial, it still wouldn't have been a substitute for fixing the 404 - because the two approaches solve different problems.

Taking 404 out of the retry path is a structural safeguard. In any environment - staging, a misconfigured production, some future case where somebody forgets to flip a toggle - it stops a 404 that retrying can never fix from turning into an alert storm.
Disabling the job in staging is an operational optimization. "There's no reason to call this in this environment, so skip the call entirely" is a good thing to layer on top.

The order matters. Adding a toggle on top of the 404 fix is fine. But adding only the toggle while skipping the 404 fix means that someday, when production config drifts and a 404 shows up there, the alert comes back - and this time you can't wave it off as "just staging." Turning off the source erases the one alert you can see right now. Fixing the failure mode erases every future case where that same alert would fire again.

Faced with a noisy alert, the hand reaches for the source first: kill the job, delete the alert rule, exclude the environment. It's a natural impulse, and sometimes it's the right one. But before you flip that switch, it's worth asking one question: am I silencing the alert, or fixing the failure it was telling me about? The former makes this one alert go quiet. The latter keeps this same failure from coming back wearing a different face.

What to ask at the boundary

None of this is specific to the Google Play API. Any code that calls out to an external service and retries its failures through a queue faces the same fork in the road. Before touching retry policy in the queue configuration, ask this at the client boundary first.

Does this error get better if I retry it? 5xx, 429, and timeouts usually do - what fails now might succeed shortly after. 4xx usually doesn't. 404, 403, and 400 will keep giving you the same answer until the underlying condition changes. Merge the two into a single exception, and a failure that retrying can never fix ends up burning retry budget and filling the alert channel anyway. The place to draw that line isn't the queue - it's the client boundary, where the response code is first seen.

And before writing a fix, it's worth taking one pass through the codebase. In our case, the neighboring method in the same file was already handling 404 correctly. There was nothing to invent. More often than you'd expect, the answer is already sitting in the code - some other piece just never followed it.

SIGTERM Had Two Owners: Why Our Shutdown Hooks Ran Twice

Why Next — Wed, 08 Jul 2026 06:51:26 +0000

Originally published at blog.whynext.app.

Every deploy dropped the same error into Sentry: Transaction already closed. That's Prisma failing with P2028 when it tries to run a query after its transaction has already been closed. Users never saw any impact - BullMQ's retry immediately picked the job up on another task, and the completion logs never showed a gap. But it was a quiet, recurring failure mode any time a deploy overlapped with live traffic. Chasing it down, I discovered our server's shutdown hooks were actually running twice on every single shutdown.

Act One: the database dies before the job does

Lining up the error timestamps against CloudWatch logs, the timeline matched exactly. A rolling deploy sends SIGTERM to the old task. Six seconds later, app.close() starts cleanup, and the Prisma engine closes its transaction. Right after that, an updateMany from a still-running asset-processing job executes against a transaction that's already closed, and fails with P2028. A user happened to be uploading audio back-to-back, so there was an active job at the exact moment of the deploy, and that job's query died.

The root cause was shutdown ordering. NestJS calls lifecycle hooks in a defined order when it shuts down. The problem was that closing the BullMQ worker and disconnecting Prisma both lived in the same shutdown phase (onApplicationShutdown). Within a single phase, module registration order decides execution order - and in our case, Prisma happened to disconnect first, leaving the still-in-flight worker's query with nowhere to go.

The fix direction was obvious: close the worker in an earlier phase. NestJS's shutdown hooks have an ordering, and beforeApplicationShutdown finishes for every module before any module's onApplicationShutdown even starts. I wrote a service that finds every worker and drains it (waits for active jobs to finish) in this earlier phase.

@Injectable()
export class WorkerDrainService implements BeforeApplicationShutdown {
  constructor(private readonly discovery: DiscoveryService) {}

  async beforeApplicationShutdown() {
    // Find every registered WorkerHost and wait for its active jobs to finish.
    // This phase finishes before onApplicationShutdown, where Prisma disconnects.
    const workers = this.discovery
      .getProviders()
      .filter((p) => p.instance instanceof WorkerHost);
    await Promise.all(workers.map((w) => w.instance.worker.close()));
  }
}

Worker.close() returns the same promise on repeated calls, so it doesn't conflict with any existing worker-closing code, and it silently does nothing in environments without Redis. Now, even when a deploy overlaps with a job, the job finishes first and the DB closes afterward. I added a unit test and opened the PR. Up to this point, it was an ordinary shutdown-order bug fix.

Act Two: SIGTERM had two owners

The twist came in review. A reviewer going through the whole shutdown path asked a simple question: who, exactly, is handling SIGTERM?

It turned out there were two owners. main.ts had these two separate lines:

// ① Hand shutdown signals to Nest itself (calling with no args registers both SIGTERM and SIGINT)
app.enableShutdownHooks();

// ② Also hand shutdown to the http-graceful-shutdown library
setupGracefulShutdown(server, {
  onShutdown: async () => {
    await app.close(); // this is where the full set of lifecycle hooks runs
  },
});

Both were catching SIGTERM. When a signal arrives, Node calls every registered listener in order, without waiting for any of them to finish. So the moment a real SIGTERM arrives, here's what happens: the listener Nest registered runs the exact same shutdown sequence as app.close() immediately, without waiting for the HTTP drain. Separately, http-graceful-shutdown drains HTTP connections its own way, then calls app.close() a second time from onShutdown. The shutdown hooks run twice.

Reasoning alone wasn't enough to be sure, so I went back and read the production shutdown logs. The evidence was right there. Twenty milliseconds after the SIGTERM timestamp - before the HTTP drain had even finished - onModuleDestroy had already started, and the shutdown hook logs for SpeechService and QueueMonitorService each appeared twice. The only reason this never turned into an incident is that each service's hooks were written to be safe to call twice (idempotent). That defensive habit was the only thing keeping "runs twice" as a log smell instead of an incident - and that smell is what eventually caught the bug.

Deleting one line

The fix was deleting one line.

// main.ts
- app.enableShutdownHooks();

Remove enableShutdownHooks() and SIGTERM has a single owner: http-graceful-shutdown. Nothing is lost, because its onShutdown already calls app.close(), which runs the full set of lifecycle hooks anyway. The only thing that changes is that the hooks now run once instead of twice - and only after the HTTP drain has finished.

The smaller the fix, the more it needs to be verified against the real thing. I sent an actual SIGTERM to a built server and watched the sequence with my own eyes.

Received SIGTERM → health check 503 → HTTP connections closed
→ shutdown hooks run only once, after HTTP close
→ Drained 56/56 BullMQ workers in 211ms → shutdown completed → clean exit

What used to run twice before the HTTP drain in production now runs once, after the drain. Since this was verified with Act One's worker-drain fix already in place, it also confirmed the two fixes work correctly together.

I also wrote down a gotcha I hit along the way. Under NODE_ENV=development, http-graceful-shutdown switches to an immediate-exit mode and never takes the graceful path at all - so this verification can't be reproduced in development mode. I only got a real reading after setting NODE_ENV=staging with dummy secrets, so the process would take the same path as production. When you're verifying a shutdown path, the first thing to check is whether your verification environment actually exercises that path.

What the shutdown code taught me

The lesson here reaches well past the narrow stage of a shutdown sequence.

A single signal should have a single owner. If you've wired in a graceful-shutdown library, don't also leave the framework's automatic shutdown-hook registration turned on. The moment both catch the same SIGTERM, the shutdown sequence quietly runs twice. When you adopt a new tool, the first thing to check is which signal or event it's going to own - and whether something already owns it.

Shutdown has an order too. Code that cleans something up should only run after everything depending on it has finished. A database should close only after the jobs using it are done, which means those jobs need to be drained in an earlier phase. A framework's phased shutdown hooks aren't decoration - they exist precisely to express this kind of ordering.

Defensive code buys you time. If the shutdown hooks hadn't been written to tolerate being called twice, this bug would have shown up as a data incident, not a log smell. Because the hooks were idempotent, they absorbed the double execution, and we got to find the bug in a log instead of an incident report. If those hooks had been written assuming they'd only ever run once, this story would have ended very differently.

The Discount You Weren't Claiming: Why Spot and Graviton Are Cheap

Why Next — Wed, 08 Jul 2026 06:51:25 +0000

Originally published at blog.whynext.app.

Our AWS bill came in over budget two months in a row. Against a monthly cap of $550, actual spend landed at $585, then $553. Not huge amounts, but the trend was up. I went resource by resource looking for something to cut, and the conclusion was unexpected: there was almost nothing left to cut. The real lever wasn't "how much you use" - it was "how you buy it." What I learned along the way: every number on a bill has a reason, and you need to know that reason to decide what to turn off and what to keep.

The resources were already lean

I started with usage itself. But this part had already had a lot of work put into it. The NAT gateway was gone in favor of public subnets, logs were kept for only 7 days, tracing was sampled at 1%, and the observability dashboard was squeezed to fit inside the free tier. The database and Redis were already running on cheaper ARM instances, S3 had automatic tiering, and the bastion host only spun up on demand. Pretty much everything that could be squeezed out at the resource level already had been.

Breaking down the June bill line by line: RDS $208, ECS Fargate $84, load balancer $52, WAF $49, public IP $41, ElastiCache $35, S3 $20. Shrinking the instances further from here would have been a step backward. The database had a history of being scaled up once already due to memory pressure, and the WAF was deliberately kept around to validate rules before production rollout. The only lever left wasn't the resources - it was how we paid for them. Buy the same resources, just more cheaply.

Why Spot is cheap

The first thing that stood out was that we were buying compute purely at on-demand list price. AWS has a much cheaper channel for the exact same servers: Fargate Spot. It runs about 70% cheaper than on-demand.

There's a reason it's that cheap. AWS builds out its data centers to handle peak demand. Under normal conditions, a lot of that capacity sits idle. Idle capacity earns zero revenue, so AWS sells it off cheap on one condition: "if a full-price customer shows up, you get bumped within 2 minutes." It's the same logic as an airline selling off empty seats right before departure - a seat that flies empty earns nothing, so selling it with strings attached still beats selling nothing at all.

The question is whether you can live with that condition - "you get bumped if asked." In production, that's a problem. If a task gets reclaimed 2 minutes after a user connects, the service wobbles, even if briefly. Staging is different: there are no real users, and a few minutes of downtime just mildly inconveniences one QA engineer. So we moved staging's API onto Spot. Performance is identical to on-demand, since it runs on the exact same hardware - same vCPU, same memory. The only thing that changes is the availability condition: "can be reclaimed" - and that's a condition staging is happy to accept. Once we made the switch, staging tasks started running fine on Spot.

Why Graviton is cheap

The second lever was the compute chip itself. The database and Redis were already on ARM instances, but the Fargate tasks running the application itself were still stuck on x86. Moving to Graviton, AWS's own ARM chip, cuts the cost of the same performance by about 20%.

The reason this one is cheap is different in kind from Spot. With a standard instance, AWS is buying CPUs from Intel or AMD, so the price we pay includes the chip maker's margin. Graviton is a chip AWS designed itself, so that middleman markup disappears. On top of that, the ARM architecture does the same work with less power and less heat, which lowers the data center's electricity and cooling costs too. From AWS's side, the cost basis is lower, so selling it 20% cheaper is still profitable - and it's also a strategy to pull customers into AWS's own chip ecosystem by pricing it low.

Here's the key difference. Spot's discount comes with a condition - "accept that you might get reclaimed." Graviton's discount comes with none. It's simply a cheaper chip. So while Spot is confined to staging, Graviton carries straight through to production as-is. Performance isn't much of a worry either: Node.js gets performance per vCPU on Graviton that's on par with x86, or better. The one thing worth watching is cold start. When an app first boots, single-core speed matters, and Graviton can lag a bit behind the latest x86 chips there, adding a few seconds to startup. So we rolled it out to staging first, measured boot time, and then promoted it to production. The only pre-check needed was confirming that native modules were built for arm64.

To sum up the character of the two discounts: Spot is a discount you earn in exchange for tolerating inconvenience, so it comes with a condition (hence staging); Graviton is a discount you get by sharing in AWS's own cost advantage on its chip, so it comes with none (hence production too). With performance equal, there was no reason to keep paying x86 list price. In truth, this round of optimization wasn't really about cutting anything new - it was closer to finally claiming a discount we'd never been claiming.

Some costs shouldn't be cut

But scanning a bill also turns up traps running the other direction: line items that look wasteful but are actually insurance.

RDS Proxy was one. It was costing $22 a month, and at first it looked like a candidate for cutting. One of Proxy's features is handling secret rotation without downtime, and that rotation feature was currently turned off. If we weren't using rotation, why keep the Proxy at all?

That was a misread. Proxy's primary purpose was never rotation - it's preventing connection surges. When a deploy or a scale-out suddenly spins up more tasks, the number of connections hitting the database spikes, and the database has a hard cap on how many concurrent connections it can accept. Proxy sits in front of it, pooling and multiplexing connections so that cap never gets breached. As it happens, this particular database was an instance that had already been scaled up once due to memory pressure, and every single connection eats memory. In other words, that $22 wasn't waste - it was the premium for insurance that keeps the database from keeling over from a connection surge every time we deploy. Turning it off wouldn't have saved money; it would have removed the safety net.

The misread traced back to a code comment. The comment describing that Proxy was written mostly from the rotation angle, which made it easy for a reader to conclude "rotation is off, so Proxy is unnecessary." So instead of cutting Proxy from the savings plan, we added a line to the comment: "Primary purpose is preventing connection surges; rotation is a secondary feature." So the next person reading the bill doesn't make the same mistake.

Every number on a bill has a reason

Tell someone to cut cloud costs, and their hand reaches for "what to turn off" first - shrink the instances, disable features, delete resources. What we learned this time was the opposite. The question to ask before cutting anything is: "why is this number here?"

Some numbers are discounts you never claimed. Like Spot and Graviton, there's a cheaper channel for buying the exact same resource, and you were simply paying list price. Each of these discounts is cheap for its own reason, and knowing that reason - whether it demands tolerating reclamation, or it's a no-strings-attached cost advantage - tells you exactly where it's safe to apply. Other numbers are, conversely, insurance you shouldn't cut. If a line item that looks like waste is actually preventing an incident, it isn't spend - it's defense.

In the end, reading a bill isn't about making the numbers smaller. It's about knowing the reason behind each one. Once you know the reason, you claim the discounts whose conditions you can live with, and you keep the costs that prevent incidents. That's all we really did here - not saving anything new, just sorting out which discounts to claim and which insurance to keep.

AI Writes the Code Now. So Why Does Git Matter More Than Ever?

Why Next — Tue, 07 Jul 2026 02:30:53 +0000

Originally published at blog.whynext.app.

"If AI writes all the code, do I still need to learn Git?"

I get this question a lot from people who started coding recently. The assumption behind it goes like this: Git is a tool from the era when we typed code by hand, so once AI writes the code, Git should become unnecessary too. My answer is the opposite. The question has it backwards. You need Git precisely because AI writes the code.

I write almost all of my code with AI. This blog, the apps I run - most of it was built with AI coding tools. And since I started working this way, my Git usage hasn't shrunk. It has multiplied. Here are the four reasons why.

You can only delegate what you can undo

Handing code over to AI means, at its core, "letting changes I haven't fully read into my project." For that not to be terrifying, one condition has to hold: you must be able to return to a clean state at any moment.

A commit is exactly that device. Like a save point in a game, once you've committed, you can make bold requests. "Restructure these files completely" carries no risk - if you don't like the result, you roll back to the commit and move on. Without a save point, the moment AI touches ten files, there is no way back. You're left alone with broken code, not even sure what changed.

There's a real incident that shows this isn't a theoretical worry. In July 2025, a founder building an app with Replit's AI agent watched the agent delete his production database - even though he had told it to freeze all code changes. It made enough noise that Replit's CEO personally apologized. Had it been code, a commit would have saved him. The lesson isn't "AI is dangerous." The lesson is: never give AI free rein without a way to undo.

The people who delegate the most to AI are the ones who commit most often. Boldness doesn't come from courage. It comes from knowing you can always go back.

AI's output arrives as a diff

The biggest change since I started working with AI is how my time is spent. Time spent writing code dropped sharply, and time spent reading code filled the gap. When AI finishes a task, my job is to check what changed and how - and the format that shows me is the diff. That Git screen with only the changed lines, picked out in red and green.

That's why the real bottleneck in development today isn't producing code - it's verifying it. As the cost of generating code falls, the cost of answering "can I trust this change?" becomes relatively expensive. And the unit of that judgment is the diff. If you can read a diff, you grasp what AI did in five minutes. If you can't, you're combing through the entire codebase from scratch.

Reading diffs is a far easier skill than understanding a whole codebase - you only look at what changed. Yet this one easy skill decides how fast you can work with AI.

GitHub is where AI agents work

GitHub was originally a place for humans to collaborate with humans. Split off a branch, propose changes through a Pull Request, review, and merge. It turns out AI coding agents work in exactly this way. Assign an issue to GitHub Copilot's coding agent and it creates a branch, does the work, and submits a PR. Tools like Claude Code also operate on the same flow: branch, stack commits, open a PR.

In other words, branches, PRs, and reviews are no longer just conventions between people - they're the interface between people and AI. If you don't know these concepts, you end up assigning work to an AI agent without knowing how to receive the result. The agent reports "I opened a PR," and if you don't know what a PR is, everything stops right there.

This applies even if you build alone. I work solo, yet I create PRs every day. Not because I have collaborators, but because no format works better as a checkpoint for reviewing and approving a batch of AI work. For a solo developer, GitHub isn't a collaboration tool. It's an inspection station for AI.

History is context you feed to AI

The last reason to get comfortable with Git looks a bit further ahead: your commit history becomes AI's input.

AI coding tools produce better results the more project context they have. And a commit history is, by itself, a chronicle of the project. If your commit messages record when, what, and why things changed, AI reads them and understands "this code was fixed this way because of last month's bug." Today's tools actually do this - when they get stuck, they dig through git log and git blame on their own to find context.

So the habit of making small commits with thoughtful messages used to be a gift to your future self. Now it's also a gift to your future AI. In a repository with a hundred commits that explain what changed and why - rather than a hundred commits labeled "fix" - AI gets noticeably smarter. We've entered an era where your records become your performance.

"Can't AI just handle Git for me too?"

Having read this far, you might push back: commits, branches, PRs - can't AI just do all of that for me?

Yes. That's exactly what happens. I almost never type git commands myself. I say "commit what we have so far" or "make a branch and work there," and AI takes care of it. The value of memorizing commands really has collapsed.

Judgment, though, can't be delegated. How far to roll back, whether this change is safe to merge, whether now is the moment for a save point - only someone who knows what commits, branches, and diffs are can make those calls. Without the concepts, AI reports "there's a merge conflict" and you have no idea what's happening or what to tell it. It's like having a driver: someone else can steer, but the destination is still yours to choose.

So "getting comfortable with Git" now means something different. It used to mean drilling commands into your fingers. Now it means being able to hold a conversation in concepts. There's less to memorize - and what's left is to understand.

Where to start

If you've just started coding with AI, skip the command-line book and get four concepts down instead.

Commit: a save point you can return to. Make one before you hand AI a task.
Branch: a fork where you experiment without touching the main code. Bold attempts belong on a branch.
Diff: the screen that shows what changed. Always check AI's work as a diff.
PR: the gate where a batch of changes gets reviewed and merged. Use it even when you're alone.

Once you understand these four, delegate the rest to AI. If you can say "commit this," "make a branch," "show me the diff," and "open a PR" - you're already on good terms with Git.

AI lifted the burden of writing code. What it left us is the work of governing change, and the language of that work is Git. The less code we type by hand, the more power belongs to the person who can read, undo, and merge what changed.