DEV Community

Revin
Revin

Posted on Originally published at revin.com.br

I inherited an undocumented codebase: 2.5 days to boot it, ~400 silent errors a day

Six weeks ago I got read access to a system nobody had documented. The dev who wrote most of it left in 2023. The README had three lines and two of them were wrong.

My first instinct was the wrong one. I opened the repo and started reading, module by module, for two days. At the end I could describe the folder structure and nothing else. Reading code tells you what the code says. It says nothing about what actually runs, how often, and what breaks at 3am.

So I stopped reading and ran three experiments instead, in this order: boot the system on a clean machine while logging every missing step, instrument what was already in production, and find the handful of paths where money moves. No features, no refactors, for two weeks.

The Stack Exchange question "I've inherited 200,000 lines of spaghetti code, what now?" has been sitting there since 2012, past 200k views, 463 votes, 19 answers. Almost every answer argues about code quality. I couldn't join that argument yet, because I still didn't know what the system did.

Experiment 1: boot it on a clean container

Rule I set for myself: no asking anyone, no copying files from a colleague's laptop. Empty container, the repo, and whatever documentation exists.

$ git clone git@github.com:redacted/api.git && cd api
$ cp .env.example .env && docker compose up --build
api-1  | Error: connect ECONNREFUSED 127.0.0.1:6379
api-1  |   at TCPConnectWrap.afterConnect [as oncomplete]
api-1  | # REDIS_URL is read in src/queue/client.ts and is not in .env.example
api-1 exited with code 1
Enter fullscreen mode Exit fullscreen mode

Every time it broke I appended a line to BOOT.md: the error, what fixed it, how long it took. That file ended up with 14 undocumented steps and the whole thing took me 2.5 days.

One grep was enough to prove the env problem was structural and not bad luck:

$ grep -rhoE "process\.env\.[A-Z_]+" src | sort -u | wc -l
23
$ grep -cE "^[A-Z_]+=" .env.example
11
Enter fullscreen mode Exit fullscreen mode

Twelve variables the application reads and nobody wrote down. The rest of the list looked like this:

  • the database schema only existed as a production dump someone pastes into a chat window
  • the runtime was pinned to a three-year-old minor through an undeclared engine field
  • one payment integration had no sandbox and pointed at the live endpoint from local
  • the seed script assumed a tenant row that only exists in production

BOOT.md became the first honest document the project had. In construction there's a drawing called as-built: the plan of what was actually erected, not the one that left the office. That's what a boot log is.

The number I track now: time between a new dev getting access and seeing one request served locally. It went from 2.5 days to roughly 40 minutes once I committed the compose fixtures and a seed that doesn't need production.

Experiment 2: a week of production data beats a month of reading

The system had a /health endpoint answering 200 and a colourful panel wired to it. Neither measured anything that breaks. The metric existed, the guarantee didn't.

I installed three things and stopped, on purpose, because none of them touch business rules:

// src/http/observability.ts
app.use((req, res, next) => {
  const id = req.headers['x-request-id'] ?? randomUUID()
  const started = process.hrtime.bigint()
  res.setHeader('x-request-id', id)

  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - started) / 1e6
    logger.info({
      request_id: id,
      route: req.route?.path ?? 'unmatched',
      method: req.method,
      status: res.statusCode,
      duration_ms: Math.round(ms),
    })
  })

  next()
})
Enter fullscreen mode Exit fullscreen mode

Structured logs with a request id, error grouping in a capture tool, and response time per route at the edge. One deploy, one file, low blast radius.

Seven days later the log aggregation said three routes out of 61 carried around 78% of the traffic. It also surfaced an error firing about 400 times a day inside a swallowed catch, which had never reached a single dashboard.

The latency answer came from the database, not from the code I had been reading:

SELECT calls, round(mean_exec_time::numeric, 1) AS avg_ms,
       round((total_exec_time / 1000)::numeric, 1) AS total_s,
       left(query, 70) AS query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 5;
Enter fullscreen mode Exit fullscreen mode

Top row: a SELECT ... WHERE order_id = $1 called 41,882 times in a day at 6.4ms each. Cheap query, called inside a loop, once per line item. Half the p95 of the checkout route lived there.

What I tried and dropped: full distributed tracing in week one. Instrumenting every service boundary meant editing code I didn't understand yet and shipping a deploy I couldn't reason about. Edge timings plus a request id got me most of the answer from one middleware, and I added spans later, only on the paths that mattered.

Experiment 3: find the five paths where money moves

This one has no code. I booked an hour with support, an hour with finance, and an hour with the person who has been in operations the longest. Three questions: what do customers call about, what jams at month-end close, and which spreadsheet exists today to work around the software.

Five flows came out, and I mapped each one to concrete routes and jobs using the request ids from experiment 2:

  • signup and login, because a customer who can't get in complains within minutes
  • checkout, plus every payment call it fires underneath
  • the nightly billing job, the oldest script in the repo and the one with no reprocessing
  • outbound integration with the ERP, where a silent failure becomes a reconciliation mess a quarter later
  • the report leadership opens on Monday, which decides whether the system is trusted

Five paths with an owner and log evidence behind each one told me more than any diagram of 200k lines would have.

The 92% coverage that guarantees nothing

I asked for the coverage number before looking at the suite. 92%. Then I opened the billing tests:

it('generates invoices for active contracts', async () => {
  const contracts = await factory.contracts(3, { status: 'active' })
  await billing.run({ reference: '2026-07' })
  // no assertion
})
Enter fullscreen mode Exit fullscreen mode

It runs, it passes, it counts as covered lines. A test with no assertion is a line counter with good PR.

So I ran mutation testing on that one module:

$ npx stryker run --mutate "src/billing/**/*.ts"
Ran 1.72 tests per mutant on average.
---------------|---------|----------|-----------|------------|
File           | % score | killed   | survived  | no coverage|
---------------|---------|----------|-----------|------------|
billing        |   41.18 |      44  |       63  |          0 |
---------------|---------|----------|-----------|------------|
Enter fullscreen mode Exit fullscreen mode

92% coverage, 41% mutation score, 63 mutants alive in the code that issues invoices. Now the theatre had a receipt.

I didn't try to write the whole suite in two weeks. I wrote characterisation tests on the five revenue paths, pinning current behaviour exactly as it is today, wrong parts included. That's a safety net and it isn't quality yet. Quality comes after you know what you can touch without dropping revenue.

The rewrite I didn't do

Around day 4 the sentence shows up: this is unmaintainable, we should rewrite it. I keep saying no for a practical reason. The ugly code is the only living documentation of the business rules. The weird if in the middle of the order service turned out to be a contract signed with a large customer in 2019, and nobody on the current team knew it existed. A rewrite throws away the answer together with the question.

Where that stops being true: a runtime with no security support, a critical dependency unpatched for years, a stack you can't hire for anymore. Then the maths flips, and even then I'd go piece by piece with the old system running beside me as the oracle. And I honestly don't know that any of this scales down to a 3,000-line app with forty users. One afternoon probably covers that.

Where day 14 landed

Boot time from 2.5 days to about 40 minutes. One error at ~400/day found and fixed. One N+1 removed from checkout. Mutation score measured on the module that bills customers. Characterisation tests on five flows. Zero features, zero refactors.

The part I'm least sure about is experiment 3. Sitting with support and finance worked, but it's slow and it depends on those people having time for me. Has anyone found a faster way to identify the revenue paths straight from telemetry? And do you instrument first or read first when you land on a codebase nobody can explain?


Originally published on the Revin blog: https://revin.com.br/en/blog/inherited-spaghetti-code-first-two-weeks

Top comments (0)