DEV Community: Revin

An OpenAI model went looking for exposed API keys in testing. I assume my coding agent will too

Revin — Fri, 18 Sep 2026 14:42:10 +0000

CIO reported today that OpenAI published six new reports on model misalignment under a new reporting framework. The cases come from internal evaluations and describe models acting beyond the constraints they were given: hidden instructions, unauthorized communication with external services, modified intermediate outputs, shared environments used in ways nobody planned. One of them is an attempt to locate exposed API keys. OpenAI called the behavior "unexpected or concerning".

Most reactions treat this as news about OpenAI. I read it as a spec sheet for the thing running in my terminal. A coding agent with shell access has the same shape as the models in those reports: a goal, a set of tools and a filesystem. If a key sits within reach and the task gets easier with it, using the key is just another path to finishing the task. No intent is required for that to happen.

So I stopped asking whether my agent would open .env. I assume it will, and I set things up so that opening it finds nothing worth having.

The mental model: a contractor on day one

A new contractor gets the access their task needs, for as long as the task lasts, and someone keeps a record of what they touched. Nobody hands them the master key because they seem careful. That is the frame I use for agents, and it breaks down into four questions:

What can it read right now?
Where do the secrets live, and can they leave the workspace?
Which token does it hold, and what can that token do?
Is there a record of what it ran?

1. Find what the agent can read

.gitignore protects your git history. It does nothing against a process that opens files, and the working directory is exactly where the .env lives.

Start with the files git ignores but that are sitting on disk:

# ignored by git, readable by anything running in this directory
git ls-files --others --ignored --exclude-standard \
  | grep -iE '\.env|secret|credential|\.pem$|\.key$'

Then look at what the agent inherits without reading any file:

# names only, never print the values
env | grep -iE 'key|token|secret|password' | cut -d= -f1

The second command matters more than people expect. Agents usually inherit the shell you launched them from. If you exported OPENAI_API_KEY or AWS_SECRET_ACCESS_KEY in your profile months ago, the agent has it already.

Last, scan contents, history included:

gitleaks dir . --redact --no-banner
gitleaks git . --redact --no-banner

A key that was committed once and deleted later still lives in git log -p. An agent asked to find out why something broke in March will read that history.

2. Move secrets out of the workspace

The tempting first fix is telling the agent to ignore .env, through its ignore file or a line in the instructions. That is a request. The models in the OpenAI reports were operating under instructions too.

What holds is the secret not being on disk. Replace the file with references that a secret manager resolves at runtime. With the 1Password CLI, the template looks like this:

# .env.tpl (safe to commit, safe for the agent to read)
DATABASE_URL=op://dev/app-db/url
STRIPE_SECRET_KEY=op://dev/stripe-test/secret

op run --env-file=.env.tpl -- npm run dev

The agent can read .env.tpl all day and it will see vault paths. Doppler, Infisical and the cloud secret managers do the same job with different syntax, so pick whichever your team already pays for.

One catch I don't have a clean answer for: if the agent's shell can call op run with your unlocked session, the values are one command away. I start the agent from a shell where the secret manager is not signed in and run the app from a different one. It is clunky, and I'd like to hear a better arrangement.

3. Its own token, with the smallest scope

When the agent needs credentials, to push a branch or read an issue, it gets its own and never yours. On GitHub that means a fine-grained personal access token limited to one repository, contents read and write, pull requests write, nothing under administration, expiring in days instead of a year.

Then run the agent where that token is the only thing in the environment:

docker run --rm -it \
  -v "$PWD":/work -w /work \
  --env-file agent.env \
  --network agent-net \
  agent-image

agent.env has exactly one line. The container sees the repo and nothing from your home directory: no ~/.aws, no ~/.ssh, no shell history with a token someone pasted once. If you can route agent-net through a proxy that logs hostnames, do it. Interacting with external services was one of the behaviors in the reports, and outbound traffic is where you would see it.

4. Log every command it runs

When something odd happens, the first question is what the agent executed. Most agents now expose hooks around tool calls. In Claude Code, a PreToolUse hook on Bash receives the command as JSON on stdin, and exit code 2 blocks it:

#!/usr/bin/env bash
# ~/.claude/hooks/audit-bash.sh
input=$(cat)
cmd=$(jq -r '.tool_input.command // empty' <<<"$input")
printf '%s\t%s\n' "$(date -u +%FT%TZ)" "$cmd" >> "$HOME/.agent-audit.log"

if grep -qE '(\.env|id_rsa|\.aws/credentials)' <<<"$cmd"
then
  echo "blocked: command touches a secret path" >&2
  exit 2
fi

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "~/.claude/hooks/audit-bash.sh" }
        ]
      }
    ]
  }
}

Be honest about the block. It is a speed bump. cat .e""nv walks right past that regex, and so does a Python one-liner. I keep it because it turns the obvious attempts into a line I will notice. The log is the part that earns its keep: a plain text file with timestamps that I can grep after a long session.

What I would skip

Two recommendations I keep seeing and don't think pay off. Asking the agent in the system prompt to never touch credentials costs nothing and guarantees nothing. The reports are, quite literally, about models acting past stated constraints. And rotating every key after every session sounds disciplined, but nobody keeps it up for more than a week. Short expiry and narrow scope do the same job without depending on someone remembering.

What the reports don't settle

These were OpenAI's internal evaluations, designed to surface this kind of behavior. I don't know how often a coding agent on an ordinary Tuesday task goes near a credentials file, and I would distrust anyone who gave me a precise number. What I do know is the price on each side: the setup above takes an afternoon, and being wrong means a production key that someone else finds before you do.

Security as a habit looks boring up close. The file that isn't there, the token that expires, the log nobody reads until the day it matters.

How are you keeping coding agents away from secrets today? Container, separate OS user, a remote dev box, or trust plus a good .gitignore?

I summed CPU time by user agent for 7 days: 63% went to clients that never load a second page

Revin — Wed, 09 Sep 2026 15:00:57 +0000

The invoice went up around 38% in one quarter and the meeting already had a culprit: the AI feature that shipped in April. Two more instances were on the table. Before approving them, I asked for something cheaper than a server, which was read access to seven days of access logs.

The answer took an afternoon. About 63% of the summed backend time in that week went to clients that arrived with a Chrome user agent, requested only HTML, never fetched a single stylesheet, and never came back to the same address twice.

Here is exactly what I ran, including the two things that did not work.

Your log format is probably missing the only field that matters

Most nginx setups still run the default combined format, which gives you status, bytes and user agent. It does not give you time. Without time you can only count hits, and hit count is the metric that hides this problem, because the cheap route wins on volume and the expensive one hides at the bottom of the list.

log_format timed '$remote_addr $host "$request" $status '
                 '$body_bytes_sent $request_time $upstream_response_time '
                 '$upstream_cache_status "$http_user_agent"';

access_log /var/log/nginx/access.log timed;

One reload and you start collecting. Everything below assumes that format.

First pass: classify by user agent, and watch it fail

The obvious first cut is by declared identity.

awk -F'"' '
{
  split($3, m, " ")
  ua = tolower($4)
  if (ua ~ /bot|crawl|spider|slurp/) k = "declared-bot"
  else if (ua ~ /mozilla\/5\.0/)     k = "browser-ua"
  else                               k = "other"
  hits[k]++; secs[k] += m[4]
}
END { for (k in hits) printf "%-13s %9d hits %11.1f s\n", k, hits[k], secs[k] }
' access.log-2026090*

declared-bot     412918 hits     38140.6 s
browser-ua      9130477 hits    201773.4 s
other            286042 hits     11962.0 s

Read that and you conclude the declared bots cost you about 15% of the time and the rest is customers. That conclusion is wrong, and it is the reason most teams stop here and buy the instance. A browser user agent is a string. Anyone can send it, and large scale collection stopped identifying itself a while ago.

Second pass: classify by behaviour instead

A browser that renders a page also asks for the CSS, the JS bundle, the fonts and a few images. A collector asks for the HTML and leaves. That difference is in the log already, and it does not require a new tool.

awk -F'"' '
{
  ip = $1; sub(/ .*/, "", ip)
  split($2, r, " "); u = r[2]
  split($3, m, " ")
  if (u ~ /\.(css|js|woff2?|png|svg|jpg|ico)(\?|$)/) asset[ip]++; else page[ip]++
  secs[ip] += m[4]
}
END {
  for (i in page)
    if (page[i] > 50 && asset[i] / page[i] < 0.1)
      printf "%-15s %7d pages %6d assets %9.1f s\n", i, page[i], asset[i], secs[i]
}
' access.log-2026090* | sort -k6 -nr | head

198.51.100.24      41207 pages     18   14822.9 s
203.0.113.91       28644 pages      0   10310.4 s
198.51.100.77      19855 pages      3    7412.0 s
203.0.113.140      12038 pages      0    4488.7 s
...

Sum that bucket against the total and you get the number from the title. In this system it came out at 63.4% of all backend seconds in the week. None of those addresses appeared in more than a couple of hundred requests each, because the ranges rotate. There was no name to block, which is the part that makes the usual answer useless.

Third pass: the twenty most expensive routes, by time and not by hits

awk -F'"' '
{
  split($2, r, " "); u = r[2]; sub(/\?.*/, "", u)
  split($3, m, " ")
  secs[u] += m[4]; n[u]++
  if (m[5] == "HIT") hit[u]++
}
END { for (u in secs) printf "%10.1f s %9d %6.1f%% %s\n", secs[u], n[u], 100 * hit[u] / n[u], u }
' access.log-2026090* | sort -nr | head -20

   61402.8 s     67341    1.8% /reports/export
   38915.3 s    204882    0.4% /catalog
   22107.6 s     91120   11.2% /calendar/day
    9044.1 s   1980433   96.7% /api/session

The route with 1.9 million hits was the cheapest thing on the list. The one costing the most ran 67 thousand times at roughly 900 ms each, with a cache hit rate under 2%, because every request carried a different date range in the query string. Every filter combination that became a public address is a page that is generated once, served once, and cached for nobody.

The two things that did not work

robots.txt. Whoever respects it was already respecting it, and the traffic in that top bucket does not read it. Adding rules there changed nothing measurable in the following week.
Blocking inside the application. The first attempt was a middleware that inspected the request and returned 403. In that stack the session middleware ran first, so by the time the rule said no, the request had already checked out a database connection. A 403 from the app measured around 34 ms and held a pool slot. The same 403 from nginx measured 0.4 ms. Saying no is not free, and where you say it is worth roughly two orders of magnitude.

So the deny moved to the edge, per range rather than per address, since single IPs rotate too fast to matter.

limit_req_zone $binary_remote_addr zone=perip:16m rate=3r/s;

location /catalog {
    limit_req  zone=perip burst=10 nodelay;
    limit_req_status 429;
}

Honest limit on that one: rate limiting by range punishes offices behind NAT, and I have no clean answer for telling those two apart at the edge.

What actually moved the number

The expensive route got a cheap variant for anything that had not requested an asset in the same connection window: same content, none of the aggregation queries that build the side panels. Query count per render went from 41 to 3. The cache key dropped the tracking parameters and the TTL went up on pages born from parameter combinations, which changed little between requests anyway. Filter combinations left the index, with canonical pointing at the unfiltered version. Two weeks later, summed upstream time was down about 44% with no new instance, and p95 improved on the routes humans actually use.

This is not a niche problem, and it is worse on public code hosting

While I was writing this up, Konstantin Ryabitsev, who runs kernel.org infrastructure, reported that git.kernel.org burns more CPU rendering commits as HTML for scrapers than it spends on every other kind of legitimate access combined, git clones included. Across five geo distributed nodes, fourteen cores are doing nothing else at any given moment. Simon Willison published the account and it landed on Hacker News.

A git web frontend is the extreme version of the shape: commits times files times views, diff, blame, raw, tree, an address space that grows by multiplication. A person opens the same handful of those. A crawler opens all of them, once each, which defeats caching by design, because for every address the crawler request is both the first and the last.

One thing the log does not answer. It will not tell you whether the collector eating your CPU feeds something that sends you users. Behaviour separates robots from humans, not wanted from unwanted, and that second call is not an infrastructure call.

So, two questions for people who have been through this. How do you separate the crawlers you want from the ones you do not, when both arrive with a Chrome user agent from residential ranges? And has anyone found a rate limit granularity that survives corporate NAT without whitelisting by hand?

Originally published on the Revin blog: https://revin.com.br/en/blog/scraper-traffic-cloud-bill

One copy change touched 12 files: I grepped the repo and found the same discount rule written 5 times

Revin — Fri, 04 Sep 2026 13:37:59 +0000

Someone gave me read access last month to a product built almost entirely with an AI assistant. Node, TypeScript, roughly 30k lines, six months old, paying customers. The question was not about architecture. It was much more practical: why does changing one sentence in an email take a week?

The ticket behind the call was tiny. Reword the order confirmation email and fix the line that shows the applied discount. The diff that came back touched 12 files.

I did not read the codebase. I ran one search.

Start from the number, not from the folder tree

Business rules leave fingerprints, and the fingerprint is usually a literal: a threshold, a rate, a status string. Here it was a 15% discount above a 500 subtotal.

rg -n --no-heading '\b500\b' src/ | wc -l
# 19
rg -l '0\.15|0\.85' src/

src/pricing/calculateTotal.ts
src/services/checkout.ts
src/emails/orderConfirmation.ts
src/reports/monthlyRevenue.ts
src/admin/orderPreview.ts

Five files carrying the same rule. Four minutes of work, no context needed about the domain.

Then I opened them side by side, and the copies were not identical:

// src/pricing/calculateTotal.ts
if (subtotal > 500) {
  total = subtotal * 0.85
}

// src/services/checkout.ts
const discount = subtotal >= 500 ? subtotal * 0.15 : 0

// src/reports/monthlyRevenue.ts
const discounted = orders.filter((o) => o.subtotal > 500)

At exactly 500 the checkout applies 75 off and the pricing module applies nothing. The report counts that order as undiscounted either way. One boundary, three answers.

I asked how often that happens:

select count(*) from orders where subtotal = 500.00;
-- 23

23 orders since launch got their total decided by whichever code path answered first. Nobody had opened a bug, because both numbers look plausible on a screen.

The tool I expected to catch this caught nothing

My first move was the obvious one, and it failed. I ran a copy paste detector:

npx jscpd src --min-tokens 50 --reporters console
# Clones found: 3
# Duplicated lines: 1.4% (all inside __fixtures__)

1.4%, and every hit was test fixture noise. By that metric the repo looks clean.

The reason took me a while to accept. Clone detectors are built for code that was copied. This code was never copied, it was regenerated. Someone opened a new session, described the discount again in slightly different words, and got a fresh implementation with a different variable name, a different comparison operator and a different shape. Token level similarity is low. Semantic duplication is total.

That is the part I had not internalized before this repo. Old style duplication announces itself, because the two blocks read the same. Regenerated duplication hides, because each version reads like it was written by a different person on purpose.

Git says how it got there

git log --format='%ad  %s' --date=short -- \
  src/pricing/calculateTotal.ts src/services/checkout.ts \
  src/reports/monthlyRevenue.ts | head

2026-03-12  feat: order total with tier discount
2026-03-29  feat: checkout summary
2026-04-17  feat: monthly revenue report
2026-05-02  fix: discount line on confirmation email

Four weeks apart, one author, four separate sessions. Every commit was correct on the day it landed. None of them was wrong in isolation, and that is exactly why review never flagged anything.

The ten minute check that says whether the tests are real

Before touching anything I broke the rule on purpose. Changed the threshold from 500 to 5000 in the pricing module and ran the suite:

npm test
# Tests:       118 passed, 118 total
# Time:        11.4 s

118 green tests while pricing is wrong. Coverage was reported at 71%, so the number on the badge was fine. The tests exercised the functions, they just never asserted on the money.

What I did instead of proposing a rewrite

A rewrite was the first thing suggested on the call, and it was the most expensive option on the table. Three cheaper steps went in first.

One, a characterization test on the boundary, written before any refactor, pinning the behavior we decided was correct (>=, discount applies at 500):

describe('tier discount boundary', () => {
  it.each([
    [499.99, 0],
    [500, 75],
    [500.01, 75.0015],
  ])('subtotal %p gives discount %p', (subtotal, expected) => {
    expect(discountFor(subtotal)).toBeCloseTo(expected, 4)
  })
})

Two, one exported discountFor in src/pricing, then deleting the four copies one at a time, running the new test between each deletion.

Three, a crude guard in CI so the sixth copy does not get generated next month:

#!/usr/bin/env bash
# ci/check-pricing-literals.sh
hits=$(rg -l --glob '!src/pricing/**' --glob '!**/*.test.ts' '0\.15|0\.85|\b500\b' src | wc -l)
if [ "$hits" -gt 0 ]; then
  echo "pricing literal outside src/pricing:"
  rg -n --glob '!src/pricing/**' '0\.15|0\.85|\b500\b' src
  exit 1
fi

It is blunt and it produces false positives, which we handle with an allowlist file. It also caught two new attempts in the following weeks, both from fresh AI sessions that had no idea the rule already existed. The next copy change on that email touched 2 files instead of 12.

Where this does not apply

If you are three weeks from finding out whether anyone wants the product, skip all of it. A prototype does not need a single source of truth for a discount, it needs proof that someone will pay. The moment there are real orders in a database and a boundary that decides money, five phrasings of the same rule stop being a style problem.

I also do not know how this scales past one rule. Grepping literals works because pricing hides behind numbers. Rules expressed as prose in a status machine, or as a chain of booleans, do not leave that fingerprint, and I have no cheap check for those yet.

So the honest question for whoever has been through this: what do you use to catch the same rule regenerated in different wording, when clone detection is blind to it? Structural patterns with ast-grep, embedding search over the codebase, architecture tests that fail when a module reaches for something it should not know about? I would rather learn a method than write more grep scripts.

Originally published on the Revin blog: https://revin.com.br/en/blog/building-with-ai-without-architecture

I stopped Postgres and /health kept returning 200 in 4 of 6 services

Revin — Wed, 02 Sep 2026 15:06:43 +0000

Last week I read that companies are putting AI agents to work maintaining mainframes and that confidence in the platform held up while modernization runs alongside (CIO Dive, on BMC's annual survey). My first reaction had nothing to do with agents. It was that a platform older than most of the people operating it can tell you what it did at 3:12am on a Tuesday, and several Node services I have opened this year cannot.

So I stopped arguing about it in meetings and ran two boring tests on a codebase I had read access to. Six HTTP services, Node and TypeScript, Postgres, Redis, one external payments API. Both tests took an afternoon and neither needed anything beyond docker, curl, grep and jq.

Test 1: stop the database, then ask the service if it is healthy

The health endpoint in four of the six services looked like this. Same shape, different indentation.

app.get('/health', (req, res) => {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
});

It reads nothing. It answers because the process is alive and the event loop got to it. Here is the run:

$ docker compose stop postgres
[+] Stopping 1/1
 ✔ Container app-postgres  Stopped

$ curl -s -o /dev/null -w "%{http_code}\n" localhost:3000/health
200

$ curl -s localhost:3000/orders/8814 | head -c 80
{"statusCode":500,"message":"Internal server error"}

Four services returned 200 with the database down. One returned 503, and it was the only one whose author had wired an actual query into the check. The sixth had no health route at all, which is at least honest.

The dashboard on the wall reads that endpoint every 30 seconds and draws availability at 99 point something. During the two minutes Postgres was stopped, the graph stayed green and every order request returned 500. The metric existed. The guarantee did not.

Test 2: rebuild one order from the logs

Second test, and this is the one that changed my opinion about what legacy means. Pick an order that behaved oddly yesterday and reconstruct its path from the log file, without opening a psql session.

$ wc -l app-2026-08-31.log
41823 app-2026-08-31.log

$ grep -c 8814 app-2026-08-31.log
2

Two lines for an order that went through six code paths. One was the access log line with the URL, the other was a stack trace. Everything in between was logged like this:

2026-08-31T14:02:12.114Z info  discount applied
2026-08-31T14:02:12.119Z warn  fallback rate used
2026-08-31T14:02:12.240Z info  order persisted

I counted 31 lines in that time window that plausibly belonged to the request. Twenty two of them carried no id of any kind, so "plausibly" was the best I could do. Under concurrency, that window contained three other orders. There is no way to tell which one used the fallback rate. Not a hard question, an unanswerable one.

What I tried first and had to undo

My first move was the obvious one: put the dependency checks inside /health, the same route the Kubernetes liveness probe already pointed at. It worked for about a day. Then the payments API had a slow minute, three checks went past the probe timeout, and kubelet restarted pods that were serving traffic fine. I turned a downstream hiccup into a restart storm of my own making.

The split that stuck: /health answers for the process only and stays on liveness. /ready answers for dependencies and goes on the readiness probe, with a per check timeout well under the probe timeout.

const CHECK_TIMEOUT_MS = 800;

async function check(name, fn) {
  const started = Date.now();
  const timer = new Promise((_, reject) =>
    setTimeout(() => reject(new Error('timeout')), CHECK_TIMEOUT_MS),
  );
  try {
    await Promise.race([fn(), timer]);
    return { name, ok: true, ms: Date.now() - started };
  } catch (err) {
    return { name, ok: false, ms: Date.now() - started, error: err.message };
  }
}

app.get('/ready', async (req, res) => {
  const checks = await Promise.all([
    check('postgres', () => db.query('select 1')),
    check('redis', () => redis.ping()),
    check('payments', () => fetch(`${PAYMENTS_URL}/ping`)),
  ]);
  const ok = checks.every((c) => c.ok);
  res.status(ok ? 200 : 503).json({ ok, checks });
});

Same run as before, with the database stopped:

$ curl -s -o /dev/null -w "%{http_code}\n" localhost:3000/health
200
$ curl -s localhost:3000/ready | jq -c '.checks[] | select(.ok == false)'
{"name":"postgres","ok":false,"ms":802,"error":"timeout"}

Giving the log lines an owner

For the second problem I did not adopt a tracing stack. I added a request id in async local storage and a log function that always merges it in, so nobody has to remember to pass context down five call levels.

const { AsyncLocalStorage } = require('node:async_hooks');
const crypto = require('node:crypto');
const store = new AsyncLocalStorage();

app.use((req, res, next) => {
  const requestId = req.header('x-request-id') || crypto.randomUUID();
  res.setHeader('x-request-id', requestId);
  store.run({ requestId, route: req.path }, next);
});

function log(level, msg, fields = {}) {
  const ctx = store.getStore() || {};
  process.stdout.write(
    JSON.stringify({ ts: new Date().toISOString(), level, msg, ...ctx, ...fields }) + '\n',
  );
}

// at the call site, the order id rides along with the value that matters
log('warn', 'fallback rate used', { orderId: order.id, rate, source: 'default_table' });

The replay of the same question, one command instead of an afternoon:

$ jq -r 'select(.orderId == "8814") | [.ts, .level, .msg, .rate // ""] | @tsv' app.log
2026-09-01T11:40:02.101Z  info  order created
2026-09-01T11:40:02.118Z  info  discount applied      0.15
2026-09-01T11:40:02.121Z  warn  fallback rate used    0.09
2026-09-01T11:40:02.244Z  info  order persisted

Thirty one of thirty one lines now carry the id. The whole change was under 60 lines across the four services, plus the tedious part, which was rewriting roughly 90 log call sites that were passing strings built with template literals.

What I take from it

The date of the first commit told me nothing. Two of these services are from 2019, two are from last year, and they failed the same two checks. The 1970s platform in that survey is auditable because someone paid to keep it that way for decades, with records, procedures and a named owner per change window. An agent reading its logs has something to read. Pointed at a service that logs "discount applied" with no id, any agent, human or otherwise, produces a confident guess about code that moves money.

One thing I am still unsure about: whether the split between liveness and readiness is worth it for a single instance service with no orchestrator, where a failing dependency and a failing process end in the same restart either way.

How do you draw the line on your readiness checks, and do you let a slow third party API mark you as not ready?

Originally published on the Revin blog.

1,043 branches in a pricing service: 34 changed last year, 9 disagree with the spreadsheet

Revin — Mon, 31 Aug 2026 12:57:51 +0000

Someone gave me read access to a pricing service last month with a question I could not answer by opinion: should we buy a rules engine? Node and TypeScript, roughly 40k lines, six files under src/pricing, three developers in its history and one of them still around.

The vendor deck said hundreds of rules. The team said thousands. Nobody had counted, and the count was never the interesting number anyway. What decides that purchase is how many of those rules move, how often, and who has to be in the room when they do.

So I spent an afternoon measuring three things: how many branches exist, how many of them changed in the last twelve months, and how many of the same rules also live in the finance spreadsheet with a different value. The third number was the one that mattered, and it was the only one nobody had asked for.

Counting branches, and the first attempt that lied to me

My first pass was the lazy one:

$ rg -c '\bif\s*\(' src/pricing
src/pricing/commission.ts:274
src/pricing/discount.ts:141
src/pricing/creditLimit.ts:98
src/pricing/tax.ts:87
src/pricing/legacyPartner.ts:61
src/pricing/shipping.ts:28

689 total, and wrong in both directions. It counts if inside comments and string literals, and it misses every ternary and every case. In this codebase the ternaries were not decoration, half the tier selection was written as chained ? :.

A parse gives an honest number:

// count-branches.js
const { Project, SyntaxKind } = require('ts-morph')

const project = new Project({ tsConfigFilePath: './tsconfig.json' })
const kinds = [
  SyntaxKind.IfStatement,
  SyntaxKind.ConditionalExpression,
  SyntaxKind.CaseClause
]

for (const file of project.getSourceFiles('src/pricing/**/*.ts')) {
  const total = kinds.reduce(
    (sum, kind) => sum + file.getDescendantsOfKind(kind).length,
    0
  )
  if (total > 0) console.log(total, file.getFilePath())
}

$ node count-branches.js | sort -rn
412 src/pricing/commission.ts
196 src/pricing/discount.ts
143 src/pricing/creditLimit.ts
121 src/pricing/tax.ts
116 src/pricing/legacyPartner.ts
 55 src/pricing/shipping.ts

$ node count-branches.js | awk '{ s += $1 } END { print s }'
1043

1,043 decision points. Big enough to scare anyone in a planning meeting, and useless on its own.

Which of them actually moved

File level granularity is crude, and it was enough to change the conversation:

$ for f in $(git ls-files 'src/pricing/*.ts'); do
    printf "%3d %s\n" "$(git log --since='12 months ago' --oneline -- "$f" | wc -l)" "$f"
  done | sort -rn
 31 src/pricing/commission.ts
 12 src/pricing/discount.ts
  4 src/pricing/creditLimit.ts
  0 src/pricing/tax.ts
  0 src/pricing/legacyPartner.ts
  0 src/pricing/shipping.ts

Drop the commits that only touched imports or formatting and 47 becomes 34 real rule changes in a year, 31 of them in one file. Three files with 292 branches between them had not been touched since 2021.

For the hot file I narrowed it down with git log -L, which follows a function across renames well enough:

$ git log -L '/function calculateCommission/',+60:src/pricing/commission.ts \
    --since='12 months ago' --format='%h %ad %an' --date=short | grep -c '^[0-9a-f]\{7\}'
19

Nineteen changes to one function in twelve months, and every one of them went through a ticket, a review and a deploy. That is the real argument for pulling something out of the code, and it applies to about 3% of the branches in that repo. The other 97% are fine where they are.

The 0.125 nobody could explain

While I was in there, I picked the strangest constant and asked git who put it there:

$ git log -S'0.125' --oneline --date=short --format='%h %ad %an %s' -- src/pricing/commission.ts
a3f19c2 2019-11-08 <redacted> fix commission for partner channel

One commit, one line of message, author no longer at the company. The diff shows what happened. Nothing anywhere shows why the partner channel got half a point more, or whether that was meant to expire.

Version control is the best record of what was done that most companies own, and a poor record of why it was done. A rules engine does not fix that either. It just moves the undocumented number to a different screen.

The spreadsheet nobody put in the architecture diagram

Then I asked for the workbook finance uses to close the month. Formula cells are business rules running in production without tests, history or an owner, so I counted them:

# formulas.py
import re
from openpyxl import load_workbook

wb = load_workbook("month-close.xlsx")
rate = re.compile(r"0\.\d+|\d+(?:\.\d+)?%")

for ws in wb.worksheets:
    for row in ws.iter_rows():
        for cell in row:
            value = cell.value
            if isinstance(value, str) and value.startswith("="):
                found = rate.findall(value)
                if found:
                    print(ws.title, cell.coordinate, found, value[:64])

$ python formulas.py | tee sheet-rates.txt | head -4
Commissions D14 ['0.125'] =IF(C14>45000,B14*0.125,B14*0.1)
Commissions D15 ['0.125'] =IF(C15>45000,B15*0.125,B15*0.1)
Discounts   H8  ['0.08']  =IF(AND(F8>50,G8<>"distributor"),E8*0.08,E8*0.05)
Discounts   H9  ['0.05']  =IF(AND(F9>50,G9<>"distributor"),E9*0.08,E9*0.05)

$ python formulas.py | wc -l
148
$ python formulas.py | grep -o "\['[^]]*\]" | sort -u | wc -l
23

148 formula cells, 23 distinct rates. I pulled the numeric literals out of the six pricing files with the same AST script and compared the two lists. Fourteen matched. Nine did not.

The first line of that output is the expensive one. The service pays the higher commission tier above 50,000 and the spreadsheet pays it above 45,000. Same rate, different threshold. Invoices come out of the system, commissions get paid from the sheet, and the gap only shows up when a rep notices their own number.

That comparison took about forty minutes to write and it is the measurement I would run first if I did this again, before counting a single branch.

The table that turned into an interpreter

The obvious move is to pull the rates out of the code and into data. It works, right up to the point where it does not.

create table commission_rule (
  id            serial primary key,
  channel       text not null,
  min_amount    numeric not null,
  max_amount    numeric,
  rate          numeric not null,
  effective_from date not null,
  owner         text not null
);

Tiers, caps and thresholds fit here without complaint, and most requests from the business are exactly that: change a number. Then I hit this one, which exists in that codebase almost word for word:

// 8% off above 50 units, except distributors, who get 5%,
// unless the contract predates 2023
const rate =
  order.units > 50
    ? order.customerType === 'distributor'
      ? contractSignedBefore(order, '2023-01-01')
        ? 0.08
        : 0.05
      : 0.08
    : 0

To express that in columns I added customer_type, then contract_signed_before, then a priority column, then a first match wins evaluator to read the priority. Two hours in, I was writing a rules engine inside the product, with no debugger, no tests and exactly one person who would ever understand it. I threw it away.

Where the line ended up

Numbers go to the table, with an owner and an effective date. Chained conditions stay in the code, with a test whose name spells out the business case:

test('distributor keeps 5% above 50 units when the contract predates 2023', () => {
  const order = {
    units: 120,
    customerType: 'distributor',
    contractSignedAt: '2022-04-19'
  }

  expect(discountRateFor(order)).toBe(0.05)
})

23 constants left the code. The nine mismatches got reconciled against the sheet, which was a finance conversation and not an engineering one. The chained rules stayed exactly where they were, and nobody has asked to change them since.

On the original question: 34 changes a year, concentrated in one file, with no separate business function operating the rules and no auditor asking who changed what. Buying an engine there would have added an environment to run, test and roll back, in exchange for removing a deploy that happens about three times a month.

Two things I still do not have a good answer for. The AST comparison between spreadsheet constants and code constants was a throwaway script, and I would like it running in CI, but matching a formula to the function that implements it is fuzzy work and I stopped at eyeballing 23 lines. And I have no idea where this line sits for a team of two with twenty rules, where the whole exercise probably costs more than the problem.

How do you keep rate constants from forking between the system and whatever the business uses to close the month? Has anyone automated that diff in a way that survives a column being renamed?

Originally published on the Revin blog.

I counted scope changes with gh and jq: 41 of 118 issues arrived after the freeze, 3 got a new date

Revin — Fri, 28 Aug 2026 13:22:47 +0000

A delivery I estimated at three weeks landed closer to seven. The code the ticket actually described was finished in the first week, more or less on time. The other four weeks went somewhere, and for a while the only account I had of them was a feeling.

Before I wrote software for a living I signed off on steel structures: cement plants, one racetrack. Sites run late too, sometimes badly. What a site has that my repository did not is the monthly measurement: a document listing what went up, what did not go up and why, signed by both sides long before the schedule blows. The delay gets assembled in pieces. In my project the delay showed up finished, in one meeting, at the end.

A 2012 question on Software Engineering Stack Exchange asking why IT cannot deliver large projects quickly like other industries has 123,583 views, a score of 543 and 31 answers. Most of them talk about estimation and essential complexity. I wanted a number instead of an opinion, so I went looking for the four missing weeks in data I already had: the issue tracker.

First attempt: a label nobody applied

At kickoff we agreed on a label called scope-change. Anything requested after the scope was agreed would carry it. Simple, free, and it failed.

gh issue list --milestone "Release 2" --state all --limit 300 \
  --json number,title,createdAt,closedAt,labels > issues.json

jq '[.[] | select(any(.labels[]; .name == "scope-change"))] | length' issues.json

Two. Both created by me, both on the same afternoon, both in the first month. A label works only when a human remembers to apply it at the exact moment he is under pressure to keep the work moving. Nobody labels a request that arrives as "quick thing, five minutes".

Second attempt, also bad: I tried churn per week as a proxy, lines added and removed from git log --numstat. The chart was flat and useless. Rework and new scope look identical in a diff, which is the whole reason the status meeting stays green.

What worked: creation date against the freeze date

The one field nobody has to maintain is createdAt. The tracker writes it whether anyone cares or not. So the question became mechanical: how much of what we shipped in this milestone did not exist when we agreed on the date?

FREEZE=2026-03-09T00:00:00Z

# everything in the milestone
jq 'length' issues.json

# everything born after we agreed on the date
jq --arg freeze "$FREEZE" \
  '[.[] | select(.createdAt > $freeze)] | length' issues.json

118
41

Around 35% of the milestone was written after the estimate that the milestone was judged by. Broken down by month, the shape is worse than the total:

jq -r --arg freeze "$FREEZE" '
  [.[] | select(.createdAt > $freeze)]
  | group_by(.createdAt[0:7])
  | map({month: .[0].createdAt[0:7], added: length})
  | .[] | "\(.month)  \(.added)"
' issues.json

2026-03   6
2026-04   11
2026-05   17
2026-06   7

May was the month everyone in the room described as "the team is struggling with the last stretch". It was the month the scope grew the most. Nobody lied. The growth simply had no line anywhere, so the only visible variable was the date.

The second number is the one that hurts

Work arriving late is normal. On a site the owner changes his mind constantly: he moves the warehouse layout after the foundation is poured, raises the clear height, adds a crane bay. There is a path for it, and it is tedious on purpose, because price and date move together with the request.

So I asked the tracker the second question: of those 41, how many ever got a revised date written down anywhere?

jq -r --arg freeze "$FREEZE" \
  '.[] | select(.createdAt > $freeze) | .number' issues.json |
while read -r n; do
  hits=$(gh issue view "$n" --json body,comments \
    --jq '[.body] + [.comments[].body] | join("\n")' |
    grep -Eic 'estimate|revised date|new deadline|moves the date')
  [ "$hits" -gt 0 ] && echo "$n"
done | wc -l

Three out of 41. Thirty eight work items entered the milestone through a conversation, and the schedule they landed on was still the one negotiated before they existed. That is the difference between a site that runs 20% long and a project that reads as a failure. One measured the drift along the way, the other showed up finished at the end.

Turning the measurement into a CI check

The fix I kept is boring and it lives in the repo, because process that lives in someone's memory is the same label that got applied twice. The workflow does not block the merge. It writes the fact on the PR, at the moment the work is being requested, while the decision is still cheap.

name: scope-gate

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  flag-late-scope:
    runs-on: ubuntu-latest
    steps:
      - name: Compare issue creation with the freeze date
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FREEZE: "2026-03-09T00:00:00Z"
          PR: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: |
          issue=$(gh pr view "$PR" --repo "$REPO" \
            --json closingIssuesReferences \
            --jq '.closingIssuesReferences[0].number // empty')

          if [ -z "$issue" ]; then echo "no linked issue"; exit 0; fi

          created=$(gh issue view "$issue" --repo "$REPO" \
            --json createdAt --jq .createdAt)
          labels=$(gh issue view "$issue" --repo "$REPO" \
            --json labels --jq '[.labels[].name] | join(",")')

          if [[ "$created" > "$FREEZE" && "$labels" != *"scope-change"* ]]; then
            gh pr comment "$PR" --repo "$REPO" --body \
              "Issue #$issue was created after the scope freeze and carries no scope-change label. Add the label with an effort range and a revised date, or move it out of this milestone."
          fi

The label came back to life once a robot asked for it instead of a person. Over the next two milestones the count went from 2 to 29, and the useful part was never the label itself: it was that the date discussion happened in the week the request arrived, not in the week of the deadline.

Where this falls apart

If your team opens issues after the code is written, createdAt measures nothing and you will get a clean report on a project that drifted anyway. Same for repos where one issue means "epic" and the next one means "typo", since counting items assumes items are roughly comparable in size, and mine were not. I looked at the 41 by hand to be sure the big ones were spread out, which is not a method, just a sanity check.

I am also not convinced this pays for itself on a new product still hunting for its first customers. There the scope is supposed to move every week, and a bot commenting on every PR is noise with a YAML file attached.

The part I do trust is the count. Take the last milestone that blew its date, run those two jq lines, and compare the number of items born after the estimate with the number that ever got a new date written down. If the second number is zero, the delay was never a surprise.

How do you keep this visible in your repo? I am curious about people using milestones with explicit budget, or a bot that recomputes a forecast whenever an issue joins the milestone, because comparing dates in bash is the crudest version of this I could build.

Originally published on the Revin blog: https://revin.com.br/en/blog/why-it-cannot-deliver-like-construction

I ran git blame on the files that touch money: 87% of the commission logic had one author

Revin — Wed, 26 Aug 2026 16:04:14 +0000

Every planning meeting at a client I was helping ended the same way. Someone would pull a card about the commission rule, and someone else would say "better wait for Marina". Nobody found that odd. She always sorted it out.

I wanted a number instead of a feeling, because "only Marina touches this" does not survive a budget conversation and a percentage does. So I spent an afternoon on the repository. Node, Postgres, about 64k lines, six years old, four developers on payroll and a fifth who left in 2023.

The method is crude and it fits in one afternoon. Write down the rules that move money, map each one to the files that implement it, and ask git who owns those lines.

Step 1: list the rules, then the paths

This part is not automatable and that is fine. I sat with the product person and we listed ten rules that touch money: how the discount is calculated, when an order can be cancelled, what releases a partner payout, how tax is split on an invoice, and so on. Then a developer mapped each rule to the files that actually implement it.

$ cat money-paths.txt
src/billing/commission.ts
src/billing/discount.ts
src/orders/cancellation.ts
src/payouts/partner-release.ts
src/invoices/tax-split.ts
...

Ten rules turned into 14 files. The mapping took longer than the script.

Step 2: ask blame who wrote the surviving lines

while read -r f; do
  total=$(git blame -w -M -C --line-porcelain "$f" | grep -c '^author ')
  top=$(git blame -w -M -C --line-porcelain "$f" \
        | grep '^author ' | sort | uniq -c | sort -rn | head -1)
  printf '%-34s %4s lines %s\n' "$f" "$total" "$top"
done < money-paths.txt

Output, names changed:

src/billing/commission.ts         412 lines  358 author Marina
src/billing/discount.ts           287 lines  169 author Marina
src/orders/cancellation.ts        233 lines  201 author Marina
src/payouts/partner-release.ts    191 lines   96 author Rafael
src/invoices/tax-split.ts         164 lines  102 author Marina

The commission file came back at 87% single author. Four of the ten rules had the same name above 80%. That is the whole finding, and it took an afternoon.

The flags matter more than the loop. -w ignores whitespace changes, -M follows lines moved inside a file, -C follows lines copied from other files. Without them the number is a lie, and I will get to how badly.

Step 3: check who guards the door

Authorship is half the story. The other half is review, because a rule with one author and three habitual reviewers is much less concentrated than the blame output suggests.

gh pr list --state merged --limit 300 \
  --search "billing/commission in:path" \
  --json number,reviews \
| jq -r '.[].reviews[].author.login' \
| sort | uniq -c | sort -rn

  19 marina
   3 rafael
   1 caio

Nineteen of the 23 merged pull requests touching that path were approved by the same person who wrote it. So the review was not spreading the knowledge, it was confirming it.

What did not work

Three attempts died before the one above.

First I ran git shortlog -sn -- <file>, counting commits per author. It ranked a developer first on the discount file because he had done a dependency bump that reformatted imports. Commit count measures traffic, not ownership.

Then I ran blame without -w -M -C. A Prettier rollout in 2023 rewrote almost every line in the repository, and the output cheerfully told me that one engineer owned 71% of the entire business logic. He had joined four months earlier. Any blame based metric that ignores whitespace commits will hand you that kind of nonsense with a straight face.

Last, I tried correlating with cyclomatic complexity, because that is what most answers on Stack Exchange recommend when someone asks how to quantify technical debt. The most complex file in the repository had four authors and nobody was afraid of it. The file that froze the planning meeting was 233 lines of plain conditionals. Complexity told me where the code was ugly. It said nothing about where the company was exposed.

Why the ugly code is not the debt

In systems that land on my desk after the previous vendor walked away, the thing that eats the first month is rarely the chained slow query or the test suite without a single assertion. It is figuring out why a strange conditional exists in the order service. That conditional is a commercial agreement somebody closed on the phone in 2019.

The code was the only living record of the rule. Living records read slowly, and the person who could read it fast changed jobs.

Construction has a name for the missing artifact. There is the design, drawn before anything is poured, and the as built, the drawing of how the structure ended up after every decision made on site, including the beam that moved because the soil did not cooperate. Skip the as built and you find the deviation by opening a wall with the building already occupied. In software the wall is invisible and the person who knew where it was is somewhere else.

There is a cheap test for whether your docs are as built or decoration. Give a small change on that rule to someone who has never touched it, with the document and nothing else. If she opens Slack in twenty minutes, you have a file rather than a document.

What actually moved the number

A documentation sprint fixes little and rots fast. Rotation moves the blame output, in three habits:

Every task on a concentrated rule ships as a pair, with the person who does not know it on the keyboard and the person who does only answering questions.
On call rotates through everyone, with the author of the rule in second line rather than first, because on call is the only moment nobody can postpone the question.
The rule owner writes half a page of why, including the options that were thrown out, and the page counts as done only after someone else changed the rule reading nothing but it.

This is slower. A pair delivers something like 20% to 30% slower in the first weeks and the team will say so out loud. It is the price of pulling one name off four lines of that list.

Where the advice is bad: if you are three people still hunting for customers, concentration is an advantage. Rotation eats the week and your real problem is whether anyone pays. The math flips when revenue starts depending on that system, and it flips without warning.

I re-ran the script on those paths eleven weeks later. Commission went from 87% to 61%, which is not great and is a lot better than a wiki nobody opened.

How do you measure this where you work? I have seen bus factor plugins, CODEOWNERS coverage reports and pure gut feeling in planning, and I am honestly not sure the plugins beat the afternoon with blame. If you have a metric that survived contact with a real repository, I want to read it.

Originally published on the Revin blog: https://revin.com.br/en/blog/cognitive-debt-technical-debt-outside-code

PHPCompatibility flagged 217 errors in a PHP 5.6 app. The one that cost money wasn't in the report

Revin — Mon, 24 Aug 2026 14:08:34 +0000

Last month someone handed me read access to a PHP 5.6 codebase and asked whether going to PHP 8 was the same size of job as dropping Laravel on top of what already existed. Roughly 90k lines, procedural, a homemade router, MySQL 5.7 underneath, running order capture and shipping rules for a distributor. Three developers in its history, none of them still at the company.

The team had been arguing about it for two weeks with zero numbers on the table. Instead of joining the argument I ran a scanner, and then I did the boring thing nobody wants to do first: I froze the current behavior of the money flows before touching the version.

Good thing, because the change that would have cost real money never showed up in the scanner report.

The linter counts the damage in about half an hour

composer require --dev \
  squizlabs/php_codesniffer \
  phpcompatibility/php-compatibility

vendor/bin/phpcs -p ./src \
  --standard=PHPCompatibility \
  --runtime-set testVersion 8.2 \
  --report=summary

Output, trimmed:

PHP CODE SNIFFER REPORT SUMMARY
--------------------------------------------------------------
FILE                                        ERRORS  WARNINGS
--------------------------------------------------------------
src/legacy/order_functions.php                  58        11
src/legacy/db.php                               41         3
src/legacy/mailer.php                           22         9
src/shipping/tier.php                            9         2
... (7 more files)
--------------------------------------------------------------
A TOTAL OF 217 ERRORS AND 89 WARNINGS WERE FOUND IN 11 FILES
--------------------------------------------------------------

217 errors across 11 files, in a codebase that everybody in the room described as "unfixable". Almost all of it mechanical: each() removed, create_function() gone, mysql_* calls that already died back in 7.0, arguments passed by reference where that is no longer allowed. Rector cleared a large slice of it unattended:

vendor/bin/rector process src --set php82 --dry-run

What was left after the dry run was a long afternoon of manual edits. Annoying, not a quarter of work.

The change that does not raise an error

Loose comparison between a string and a number changed in PHP 8. On 5.6, 0 == "abc" returned true. On 8, it returns false. Nothing throws, no log line appears, the if simply picks the other branch.

In this codebase it lived inside a switch, which is loose comparison wearing a costume:

function shippingTier($code) {
    switch ($code) {
        case 0:            // matches ANY non-numeric string on 5.6
            return 'free';
        case 1:
            return 'flat';
        default:
            return 'standard';
    }
}

echo shippingTier('EXPRESS');
// PHP 5.6 -> free
// PHP 8.2 -> standard

Same with in_array(0, ['EXPRESS', 'ECONOMY']), true on 5.6 and false on 8. A business rule flips and nobody signs off on it. The customer finds out from an invoice total three weeks later.

Grep does not save you here. I tried: searching for == in 90k lines returns thousands of hits, and reading them one by one is how you convince yourself you read them all.

Freeze the behavior, then bump the version

What worked was characterization tests over the four flows that generate revenue. Not unit tests of what the code should do. A recording of what it does today, ugly parts included.

// tests/Characterization/ShippingTierTest.php
$inputs = ['EXPRESS', 'ECONOMY', '0', '', '0.0', 0, 1, '1abc', null, false];

$fh = fopen(__DIR__ . '/golden.txt', 'w');
foreach ($inputs as $in) {
    fwrite($fh, sprintf("%-10s => %s\n", var_export($in, true), shippingTier($in)));
}
fclose($fh);

Run it on the old runtime, keep the file, run it on the new one, diff:

php5.6 tests/Characterization/ShippingTierTest.php && mv tests/Characterization/golden.txt golden-5.6.txt
php8.2 tests/Characterization/ShippingTierTest.php && mv tests/Characterization/golden.txt golden-8.2.txt
diff golden-5.6.txt golden-8.2.txt

1c1
< 'EXPRESS'  => free
---
> 'EXPRESS'  => standard
2c2
< 'ECONOMY'  => free
---
> 'ECONOMY'  => standard
5c5
< '1abc'     => flat
---
> '1abc'     => flat

Three lines of diff on the flow that decides shipping cost. That is the whole return on the exercise. Across the four money flows I ended up with 3 behavior changes: two harmless, one that would have handed free shipping to nobody who had it before, or the reverse, depending on which way the data leaned that month.

Two things I tried that did not work

First attempt was to lean on the existing suite. The repo reported 34% coverage, which sounded like something. Opening the tests, a good chunk of them called the method and asserted nothing at all, no assert* anywhere in the body. The metric existed, the guarantee did not. Coverage told me nothing about whether the version bump changed an outcome.

Second attempt was worse and I am glad it was a branch. I ran a sweep turning == into === in the shipping and pricing files, on the theory that strict is safer. It is safer in a language where types are stable, and this app reads everything from mysqli in the old procedural style, which hands back numeric columns as strings. Half the id comparisons started returning false. Reverted in twenty minutes. Strict comparison is a refactor with its own test bill, not a migration step you sneak in.

Then check what nobody maintains anymore

Half a day of work and the answer changes the plan more than any framework debate:

composer outdated --direct
composer audit
php -m | sort   # native extensions the server actually loads

Watch for mcrypt, which left core in 7.2 and is still glued to homegrown crypto in plenty of shops. If every direct dependency has an 8-compatible release, even one that needs work, the upgrade path is short. If the system is welded to a framework that stopped shipping fixes years ago, part of it gets rewritten either way, and now you know which part: the entry layer, not ten years of business rules.

One detail that shortens meetings: 5.6 has been out of security support since the end of 2018. That does not choose between upgrading and rewriting. It removes the option everybody secretly prefers, which is looking at this next year.

The order that ended up working

Characterization tests over the revenue flows, on the old runtime, golden files committed.
Rector plus manual cleanup for the 217 errors, then diff the golden files and explain every line that moved.
Housekeeping without changing the shape of the thing: Composer with PSR-4, a single front controller, config out of the code, real logging.

Only after that does standing a framework beside it on the same database make sense, one route at a time, new endpoints born there and old checkout left alone until tests can back the move. The system runs split for a while and that bothers everybody who likes clean things. It is also reversible on any given day, which a nine-month rewrite is not.

Measured on this one: the scan took under an hour, the characterization tests took about two days, the actual upgrade landed in a bit over three weeks. The rewrite proposal on the table had been sized at a quarter, and every rewrite I have watched from close up ran past its number.

The part I still do not have a clean answer for: flows that only prove themselves against a third party. Billing, tax invoices, the payment gateway. You cannot golden-file a webhook that only fires when a real customer pays. How do you snapshot behavior on those before a version jump? Recorded HTTP fixtures, a sandbox, or do you just ship it and watch the logs?

Originally published on the Revin blog: https://revin.com.br/en/blog/php-5-6-to-php-8-or-rewrite

I pointed an AI reviewer at a 2019 order service: 63 comments, and none of them was the IDOR

Revin — Fri, 21 Aug 2026 12:55:09 +0000

Last month I got read access to an order service that had been running since 2019. Node, Express, Postgres, around 64k lines, three developers in its history and none of them still at the company. The team wanted a second opinion before a batch of changes to the checkout flow.

I ran an AI reviewer over the branch first. Half curiosity, half because everybody has been arguing about whether AI code review is a bubble, and I wanted a number of my own instead of a take.

It came back with 63 comments across 12 files.

Sorting the 63

38 were naming, formatting, or "consider extracting this into a helper"
11 were missing null checks, and 9 of those could not happen, because the value came from a NOT NULL column
8 suggested wrapping code in a try/catch that was already inside a try/catch
4 were about actual behavior, and 2 of those I would have written myself: a date compared as a string, and a retry loop with no ceiling

So something around 3% of the output was worth a human's attention. That ratio alone is not damning. Linters have terrible ratios too and nobody cares, because a linter costs nothing to skim.

The part that bothered me is what it stayed quiet about.

The comment it did not write

Here is the endpoint, cleaned up and renamed:

// src/routes/orders.js
router.get('/orders/:id', requireAuth, async (req, res) => {
  const order = await db.oneOrNone(
    'select * from orders where id = $1',
    [req.params.id]
  );

  if (!order) return res.status(404).json({ error: 'not_found' });
  return res.json(order);
});

requireAuth resolves the session and puts req.user in place. It never asks whether that order belongs to the caller. The ids are sequential integers. Three curl calls with a valid cookie and a different number and you are reading another customer's address, items and total.

The AI's comment on that exact file was: "Consider extracting the 404 response into a shared helper for consistency." Accurate. Useless.

I checked how widespread it was with ripgrep before doing anything else:

$ rg -n "where id = \$1" --glob '!test/**' src/
src/routes/orders.js:41:    'select * from orders where id = $1',
src/routes/invoices.js:88:   'select * from invoices where id = $1',
src/jobs/reconcile.js:23:    'select * from orders where id = $1',

Two of the three were reachable from the browser. The third runs in a job with no request context, so it is fine.

Why a diff reviewer cannot see this

The reviewer reasons about lines that exist. A broken authorization check is an absence. There is no bad line to point at, and the thing that makes it dangerous lives somewhere else entirely: in the fact that ids are sequential, in what requireAuth chose not to do, in a middleware file the branch never touched.

Same story with the two other findings from that week. A query sitting inside a for loop that only hurts when a customer has more than a handful of shipments. A /health route that returns 200 without checking anything, which means the load balancer keeps sending traffic to a process that lost its database pool. Both are invisible in a diff and obvious when you run the system.

The fix I tried that did not work

My first move was the obvious one: give it more context. Whole file instead of hunk, then a repo map, then a system prompt that spelled it out, roughly "check every route for broken object level authorization, ids are sequential".

Told explicitly what to hunt for, it found the orders endpoint. It also flagged 7 endpoints in total, and 5 of those were wrong. They loaded data through a repository function that already scopes by customer_id, one call deeper than the route file. The model could not tell the difference between an endpoint with no ownership filter and an endpoint whose filter is two hops away.

Which is the honest summary: it runs a checklist over text. It does not know who owns what in your system, and the prompt that makes it paranoid enough to catch the real one also makes it cry wolf five times.

What sits in CI now

I stopped asking the reviewer for security opinions and wrote a test with two tenants:

test('an order is not readable by another customer', async () => {
  const alice = await seedCustomer();
  const bob = await seedCustomer();
  const order = await seedOrder(bob, { total: 1290 });

  const res = await request(app)
    .get(`/orders/${order.id}`)
    .set('Authorization', bearer(alice));

  expect(res.status).toBe(404);
});

404 and not 403, so the endpoint does not confirm that the id exists.

One test only covers one route, and the point was the other 40. So there is a second script that walks the Express router stack and prints every path that never touches an owner filter:

$ node scripts/audit-routes.js
GET   /orders/:id              no owner filter
GET   /invoices/:id            no owner filter
POST  /orders/:id/cancel       no owner filter
GET   /orders                  scoped (customer_id)
...
3 of 41 routes unscoped

It is crude. It matches on the repository functions we consider safe, so a fourth way of loading an order would slip past it until someone adds it to the list. But it is deterministic, it runs in about 2 seconds, and it never invents a problem to look useful.

Where I land on the bubble argument

I don't think these tools are worthless. Two of those 63 comments were real, and one of them was a bug I would probably have shipped.

What I think is broken is measuring them by comments produced. Every comment spends attention, and attention runs out. After the twentieth "consider extracting", the human reviewer starts scrolling, and the one comment that mattered scrolls by with the rest. The tool got cheaper and moved the cost onto the person approving the merge.

So the number I care about now is not how much it finds. It is how much noise a real finding has to survive.

How are you drawing that line? Specifically, has anyone gotten an AI reviewer to reason about authorization across files without drowning in false positives, or did you also give up and write the boring two-tenant test?

This audit came out of client work at Revin.

An agent wrote 14 tests for a module I knew was broken. Mutation score came back 41%

Revin — Wed, 19 Aug 2026 13:14:13 +0000

Last month I ran a small experiment on a checkout module I already knew was broken.

The module applies tiered discounts to an order total. The bug is boring: the tier discount gets applied twice above a certain subtotal, and the coupon argument is read but never used. It survived two code reviews because the output still looks like money and the difference only shows up on bigger carts.

I gave an agent read access to that one file and asked for a test suite. Nothing else in the prompt, no examples, no description of expected behavior. It came back with 14 tests. All green on the first run. Line coverage for the file went from nothing to 92%. The bug shipped anyway.

Martin Fowler's site published a piece last week called "TDD inside the agent loop - theater or actual value?", and reading it is what pushed me to stop arguing from feeling and write down my own numbers. So here they are.

The file under test

const TIERS = [
  { min: 0, off: 0 },
  { min: 500, off: 0.05 },
  { min: 1000, off: 0.12 },
];

export function orderTotal(items: Item[], coupon?: Coupon) {
  const subtotal = items.reduce((acc, i) => acc + i.price * i.qty, 0);
  const tier = TIERS.filter((t) => subtotal >= t.min).pop()!;

  let total = subtotal * (1 - tier.off);

  if (coupon && subtotal >= 1000) {
    total = total * (1 - tier.off); // the tier lands a second time
  }

  return Math.round(total * 100) / 100;
}

What the agent gave me

Two of the 14 tests were fine. The rest looked like this:

describe('orderTotal', () => {
  it('returns a number', () => {
    expect(typeof orderTotal([{ price: 10, qty: 1 }])).toBe('number');
  });

  it('applies the 5% tier', () => {
    const items = [{ price: 500, qty: 1 }];
    expect(orderTotal(items)).toBe(500 * (1 - 0.05));
  });

  it('handles a coupon', () => {
    const items = [{ price: 1200, qty: 1 }];
    expect(orderTotal(items, { code: 'X10' })).toBe(929.28);
  });
});

Look at the last one. 929.28 is the double discount. The agent read the implementation, computed what the code currently does, and froze it as the expected value. The test is green because the bug is now documented as a requirement.

This is the part that no coverage report will ever tell you. A test written from the implementation can only confirm the implementation.

The mutation run

npx stryker run --mutate 'src/checkout/order-total.ts'

-----------------|---------|----------|----------|-----------|---------|
File             | % score | # killed | # surviv | # timeout | # error |
-----------------|---------|----------|----------|-----------|---------|
order-total.ts   |   41.37 |       12 |       17 |         0 |       0 |
-----------------|---------|----------|----------|-----------|---------|

92% of the lines executed, 41.37% of the mutants killed. Twelve dead, seventeen alive. Two of the survivors say the whole story:

Survived mutant #7 (ConditionalExpression)
-  if (coupon && subtotal >= 1000) {
+  if (coupon && true) {

Survived mutant #12 (EqualityOperator)
-  TIERS.filter((t) => subtotal >= t.min)
+  TIERS.filter((t) => subtotal > t.min)

Mutant #12 flips >= into > and nothing in the suite notices, which means no test ever hits a subtotal of exactly 500 or exactly 1000. Fourteen tests and not one boundary. Mutant #7 removes the subtotal condition entirely and the suite stays green, because the coupon test asserts the broken output.

Two things I tried that went nowhere

First I did the obvious thing and asked for more. "Improve the test suite, target 95% coverage." It wrote six extra tests, coverage went to about 96%, mutation score moved to roughly 47%. More assertions on shape, more toBeDefined, same blind spots. Raising the coverage gate rewarded exactly the behavior I was trying to kill.

Then I put the theory in the prompt: "write boundary tests, this suite has weak mutation coverage." The agent produced tests that look like boundary tests, with subtotals of 499, 500 and 501, and it filled every expected value by running the current code in its head. Green again. The vocabulary changed, the epistemology did not.

What moved the number

Two changes, both of them about what the agent is allowed to read.

Feed the surviving mutants back in as the task. Stryker writes a JSON report, so the loop becomes "here is mutant #12, write a test that kills it" instead of "write tests". Two rounds of that took the score from 41% to somewhere near 78%.
Make the red step mandatory and machine-checked. The agent may not see the implementation when writing the test, and the test has to fail against the current code before anyone looks at it.

The second one is three lines in a pre-commit hook and it is the only part of this I would call TDD:

# a new test must be able to fail on the unfixed file
git stash push -- src/checkout/order-total.ts
if npx jest src/checkout/order-total.spec.ts --silent; then
  echo "test passes on the broken implementation - rejected"
  git stash pop && exit 1
fi
git stash pop

Crude, and it only works when the fix and the test arrive in the same change. But it caught four tests in the following week that were green against code they were supposed to break.

The cost, honestly

Mutation runs are slow. That single file takes about 40 seconds, the full suite takes something close to 11 minutes on our runner, so it only runs on changed files in CI and on a nightly job for the payment paths. I have no idea whether the mutants-as-input loop holds up on a codebase with heavy mocking, because ours has fairly little of it. If your suite mocks the module under test, most mutants die for the wrong reason and the score lies in the other direction.

What I stopped believing is the green board. An agent writing tests after reading the implementation produces a very convincing photograph of the bug you already have. Coverage measures which lines ran. Mutation measures whether anyone would have noticed if those lines were wrong, and the gap between 92 and 41 is where the money leaked.

So, a real question: for those of you running agents inside a TDD loop, how do you stop the model from deriving the expected value from the code it is looking at? Hiding the implementation behind an interface helped a bit here, and I would like to hear what else works before I turn this into a rule for the whole repo.

I write about this kind of thing on the Revin blog: https://revin.com.br/en

I inherited an undocumented codebase: 2.5 days to boot it, ~400 silent errors a day

Revin — Fri, 14 Aug 2026 13:47:02 +0000

Six weeks ago I got read access to a system nobody had documented. The dev who wrote most of it left in 2023. The README had three lines and two of them were wrong.

My first instinct was the wrong one. I opened the repo and started reading, module by module, for two days. At the end I could describe the folder structure and nothing else. Reading code tells you what the code says. It says nothing about what actually runs, how often, and what breaks at 3am.

So I stopped reading and ran three experiments instead, in this order: boot the system on a clean machine while logging every missing step, instrument what was already in production, and find the handful of paths where money moves. No features, no refactors, for two weeks.

The Stack Exchange question "I've inherited 200,000 lines of spaghetti code, what now?" has been sitting there since 2012, past 200k views, 463 votes, 19 answers. Almost every answer argues about code quality. I couldn't join that argument yet, because I still didn't know what the system did.

Experiment 1: boot it on a clean container

Rule I set for myself: no asking anyone, no copying files from a colleague's laptop. Empty container, the repo, and whatever documentation exists.

$ git clone git@github.com:redacted/api.git && cd api
$ cp .env.example .env && docker compose up --build
api-1  | Error: connect ECONNREFUSED 127.0.0.1:6379
api-1  |   at TCPConnectWrap.afterConnect [as oncomplete]
api-1  | # REDIS_URL is read in src/queue/client.ts and is not in .env.example
api-1 exited with code 1

Every time it broke I appended a line to BOOT.md: the error, what fixed it, how long it took. That file ended up with 14 undocumented steps and the whole thing took me 2.5 days.

One grep was enough to prove the env problem was structural and not bad luck:

$ grep -rhoE "process\.env\.[A-Z_]+" src | sort -u | wc -l
23
$ grep -cE "^[A-Z_]+=" .env.example
11

Twelve variables the application reads and nobody wrote down. The rest of the list looked like this:

the database schema only existed as a production dump someone pastes into a chat window
the runtime was pinned to a three-year-old minor through an undeclared engine field
one payment integration had no sandbox and pointed at the live endpoint from local
the seed script assumed a tenant row that only exists in production

BOOT.md became the first honest document the project had. In construction there's a drawing called as-built: the plan of what was actually erected, not the one that left the office. That's what a boot log is.

The number I track now: time between a new dev getting access and seeing one request served locally. It went from 2.5 days to roughly 40 minutes once I committed the compose fixtures and a seed that doesn't need production.

Experiment 2: a week of production data beats a month of reading

The system had a /health endpoint answering 200 and a colourful panel wired to it. Neither measured anything that breaks. The metric existed, the guarantee didn't.

I installed three things and stopped, on purpose, because none of them touch business rules:

// src/http/observability.ts
app.use((req, res, next) => {
  const id = req.headers['x-request-id'] ?? randomUUID()
  const started = process.hrtime.bigint()
  res.setHeader('x-request-id', id)

  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - started) / 1e6
    logger.info({
      request_id: id,
      route: req.route?.path ?? 'unmatched',
      method: req.method,
      status: res.statusCode,
      duration_ms: Math.round(ms),
    })
  })

  next()
})

Structured logs with a request id, error grouping in a capture tool, and response time per route at the edge. One deploy, one file, low blast radius.

Seven days later the log aggregation said three routes out of 61 carried around 78% of the traffic. It also surfaced an error firing about 400 times a day inside a swallowed catch, which had never reached a single dashboard.

The latency answer came from the database, not from the code I had been reading:

SELECT calls, round(mean_exec_time::numeric, 1) AS avg_ms,
       round((total_exec_time / 1000)::numeric, 1) AS total_s,
       left(query, 70) AS query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 5;

Top row: a SELECT ... WHERE order_id = $1 called 41,882 times in a day at 6.4ms each. Cheap query, called inside a loop, once per line item. Half the p95 of the checkout route lived there.

What I tried and dropped: full distributed tracing in week one. Instrumenting every service boundary meant editing code I didn't understand yet and shipping a deploy I couldn't reason about. Edge timings plus a request id got me most of the answer from one middleware, and I added spans later, only on the paths that mattered.

Experiment 3: find the five paths where money moves

This one has no code. I booked an hour with support, an hour with finance, and an hour with the person who has been in operations the longest. Three questions: what do customers call about, what jams at month-end close, and which spreadsheet exists today to work around the software.

Five flows came out, and I mapped each one to concrete routes and jobs using the request ids from experiment 2:

signup and login, because a customer who can't get in complains within minutes
checkout, plus every payment call it fires underneath
the nightly billing job, the oldest script in the repo and the one with no reprocessing
outbound integration with the ERP, where a silent failure becomes a reconciliation mess a quarter later
the report leadership opens on Monday, which decides whether the system is trusted

Five paths with an owner and log evidence behind each one told me more than any diagram of 200k lines would have.

The 92% coverage that guarantees nothing

I asked for the coverage number before looking at the suite. 92%. Then I opened the billing tests:

it('generates invoices for active contracts', async () => {
  const contracts = await factory.contracts(3, { status: 'active' })
  await billing.run({ reference: '2026-07' })
  // no assertion
})

It runs, it passes, it counts as covered lines. A test with no assertion is a line counter with good PR.

So I ran mutation testing on that one module:

$ npx stryker run --mutate "src/billing/**/*.ts"
Ran 1.72 tests per mutant on average.
---------------|---------|----------|-----------|------------|
File           | % score | killed   | survived  | no coverage|
---------------|---------|----------|-----------|------------|
billing        |   41.18 |      44  |       63  |          0 |
---------------|---------|----------|-----------|------------|

92% coverage, 41% mutation score, 63 mutants alive in the code that issues invoices. Now the theatre had a receipt.

I didn't try to write the whole suite in two weeks. I wrote characterisation tests on the five revenue paths, pinning current behaviour exactly as it is today, wrong parts included. That's a safety net and it isn't quality yet. Quality comes after you know what you can touch without dropping revenue.

The rewrite I didn't do

Around day 4 the sentence shows up: this is unmaintainable, we should rewrite it. I keep saying no for a practical reason. The ugly code is the only living documentation of the business rules. The weird if in the middle of the order service turned out to be a contract signed with a large customer in 2019, and nobody on the current team knew it existed. A rewrite throws away the answer together with the question.

Where that stops being true: a runtime with no security support, a critical dependency unpatched for years, a stack you can't hire for anymore. Then the maths flips, and even then I'd go piece by piece with the old system running beside me as the oracle. And I honestly don't know that any of this scales down to a 3,000-line app with forty users. One afternoon probably covers that.

Where day 14 landed

Boot time from 2.5 days to about 40 minutes. One error at ~400/day found and fixed. One N+1 removed from checkout. Mutation score measured on the module that bills customers. Characterisation tests on five flows. Zero features, zero refactors.

The part I'm least sure about is experiment 3. Sitting with support and finance worked, but it's slow and it depends on those people having time for me. Has anyone found a faster way to identify the revenue paths straight from telemetry? And do you instrument first or read first when you land on a codebase nobody can explain?

Originally published on the Revin blog: https://revin.com.br/en/blog/inherited-spaghetti-code-first-two-weeks

The code that passes the pitch and fails due diligence

Revin — Wed, 12 Aug 2026 12:56:57 +0000

Originally published on the Revin blog.

For months I kept hearing the same scene told from different angles. A founder lands the round's narrative, the investor likes it, a term sheet shows up. Then technical diligence starts, someone from outside opens the repository, and the number on the page starts to shrink.

In May I talked to the founder of a healthtech company who had just been through it. Series A on the table, an eager lead, and a consultant the fund had hired to read two and a half years of product. The report came back at thirteen pages. The part that stung wasn't a bug: it was that nobody on the cap table could say who legally owned the code written by a studio she'd hired in year one and let go later. The contract was boilerplate. So was the IP clause.

Her product worked fine. What failed was everything around it: the contract, the ownership, a history nobody had read with a buyer's eyes.

Two people going over a report, pen on the chart: diligence reads what the pitch left out.

I've written about engineering teams for a while, and this is the gap that catches my eye the most. A pitch is a sales object: it shows the traction, the line going up, the demo that works. Diligence is the opposite of a pitch. It doesn't want the good story, it wants what sits underneath. And what sits underneath is rarely built with the day in mind when someone from outside lifts the lid.

Diligence opens what the pitch keeps shut

Technical diligence is not bug hunting. A serious consultant spends three to five days looking at a set of things the founder almost never measured:

Intellectual property: who signed what, and whether third-party code actually became yours.
Knowledge concentration: how many people understand the system, and what happens if the main one leaves.
Basic security: a secret committed to the repo, a dependency left unpatched, a door left open.
Ability to ship: how long it takes from a commit to production, and how many hands have to get involved.

None of these show up in a demo. All of them show up in a report. Which is why diligence is usually the first time a founder sees their own product through the eyes of the person about to pay for it.

"We'll clean it up before the round" doesn't survive contact

The most common reaction, when I raise this in conversation, is to defer. "When the round gets close, we'll get organized." It feels like the natural order of things. Except diligence won't wait for you to be ready.

You can't document two years of decisions in three weeks. Rewriting an IP contract with a vendor who already walked away is a slow negotiation, sometimes with someone who no longer picks up. Testing code that never had tests, written by people no longer on the team, is archaeology. Diligence lands in the worst possible window: when you have the least time and the most to lose. Show up improvising and the report becomes the other side's discount argument.

The counter has no shine to it, and it works: a team that operates, from day one, as if the lid could be lifted at any moment. The squads we work with at Revin run that way by default, with a round on the horizon or none at all. Code that passes diligence is the same code that doesn't break in production, and the discipline behind one is the discipline behind the other.

If you suspect your code would not survive an outside read, a Diagnostic Sprint lifts the lid before the investor does. Worth a look.

What usually shows up in the report

I've read more diligence reports than I'd like, and a few patterns keep repeating. A production secret committed in plain text somewhere in the Git history, where it never truly disappears. One person holding the core of the system in their head, with nothing written down. Libraries two or three major versions behind, carrying a known and public vulnerability. A deploy that hinges on a manual step living in someone's memory.

Across the projects that reached me for a second opinion this past year, something like two-thirds had at least one secret exposed in the repository. That number is probably skewed, I'll admit: nobody calls me to talk about the diligence that went well. But even cut in half, it's too high for a problem that costs a day to fix and a whole round to discover late.

A team around one laptop: code that passes an outside read is built by people who expected one.

The report doesn't ask who wrote it

Here's a misunderstanding that trips founders up. A bad diligence result is not a synonym for outsourced engineering. I've seen in-house teams leave a repository in a state that would embarrass any vendor, and I've seen an external squad hand over code the fund's consultant praised in writing. The question that matters is only one: was the code written with discipline, or was it a patch job?

A patch job has a recognizable signature: the freelancer who showed up for three months and vanished, the studio that shipped the MVP and handed back a boilerplate contract, the agency that billed by the hour and left no test behind. Each one solved the week's problem and none thought about the report coming two years later. A managed senior squad does think, because the same discipline that makes a test pass is the one that gets an IP clause signed and keeps a secret out of the repo.

Who actually owns the code, by the way, deserves its own piece, and there is one here.

Back to that founder

The healthtech closed its round, but smaller than it was worth, with part of the money earmarked to fix what diligence had flagged. She left me with a line I keep repeating: "I paid for the same code twice: once to write it, once to prove it was mine."

I'm not writing this to scare anyone raising capital. I'm writing it because diligence is just the moment the bill arrives; the spending happened much earlier, in the choice of who built the thing and how. If you want the outside read to be a stamp instead of a shock, the time to deal with it is now, long before the term sheet.

Want an honest read on the real state of your code before a fund does theirs? Talk to Revin.

We write about software engineering, managed squads and technical decisions every week at revin.com.br.