DEV Community: Kazu

"nginx doesn't run your config top to bottom: two traps with rewrite, try_files, and if"

Kazu — Tue, 11 Aug 2026 13:01:00 +0000

You write a rewrite, then a try_files right under it, and you read it as "the rewrite on top runs first, the try_files below runs after." You put an if inside a location and read it as "when the condition is true, this stuff gets added to the outer config." Both readings trace the config file straight down, line by line, in the order it's written. And both have burned me.

Let me show you the two symptoms first. In each case the config reads cleanly, yet the behavior doesn't line up. Why that happens — the order in which nginx processes a request — is what this post is about.

Symptom 1: rewrite and try_files side by side, and you get a 404

In some location you want to rewrite the request once, then fall back to a static file. The naive version looks like this:

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri $uri/ =404;
}

The intent is "rewrite /app/foo to /public/foo, serve the file if it exists, 404 otherwise." Since rewrite is on top and try_files is below, the rewrite takes effect first and then try_files handles the fallback — that's how it reads.

But throw /app/logo.png at it and you get a 404, even when /public/logo.png really exists.

$ ls /public/logo.png
/public/logo.png            # the file is definitely there

$ curl -sI http://localhost/app/logo.png
HTTP/1.1 404 Not Found      # and yet, 404

The try_files in this location is never evaluated once.

Symptom 2: config placed inside an `if` block doesn't take effect

In another location you want to add one header conditionally. The naive version looks like this:

location /download/ {
    add_header X-Served-By "nginx" always;

    if ($arg_debug = "1") {
        add_header X-Debug "on" always;   # only want this when ?debug=1
    }

    root /var/www;
    try_files $uri =404;
}

The intent is "always add X-Served-By, and add X-Debug on top of it only when ?debug=1." But hit /download/foo?debug=1 and X-Debug shows up while X-Served-By, which you set on the outside, has vanished. The moment the condition goes true, part of the outer config drops.

Both configs read cleanly when you scan the lines top to bottom. And both miss.

How it works: nginx's HTTP processing phases

nginx runs each request through a fixed sequence of processing phases. Which phase a given directive runs in is determined per directive type, and has nothing to do with where the line sits in the config file. Both symptoms come out of exactly this one fact.

Start with the big picture. Lay out the phases an HTTP request passes through, in order, and you get this (source: the official Phases section of the Development Guide).

post-read       right after the request is read
    ↓
server-rewrite  rewrite directly under server (before location selection)
    ↓
find-config     pick the location
    ↓
rewrite         rewrite / if inside the location
    ↓
post-rewrite    if the URI changed, go back to find-config
    ↓
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         serving files via proxy_pass / root
    ↓
log             access log

The same rewrite runs before location selection (server-rewrite) when written directly under server, and after it (rewrite) when written inside a location. Where you put it changes the phase, which changes the order it runs in. What decides the order is not the line sequence but "which block, and which type of directive" — that's how you read this list. Now let's follow how each of the two symptoms travels along this timeline.

Symptom 1, followed by phase

The part that matters for rewrite ... last and try_files is this stretch.

  ┌─▶ find-config     pick the location (first pass: /app/. with last, the second pass picks a different one)
  │       ↓
  │   rewrite         rewrite ... last rewrites to /public/foo
  │       ↓
  └── post-rewrite    URI changed → back to find-config
          ↓
      precontent      try_files runs here. we're now in the different location picked on the second pass.
                      the try_files written in the original /app/ is never reached → 404

The last in rewrite ... last means "once you've rewritten the URI, redo location selection with the new URI." So the instant /app/foo becomes /public/foo, the post-rewrite phase calls find-config back and picks a location all over again. The rewritten /public/foo no longer matches /app/, so it moves to a different location. Meanwhile try_files runs way later, in the precontent phase. It may get evaluated wherever the request landed, but the try_files written in the original /app/ is never reached again. If the destination has no file and no fallback, that's where the 404 comes from. It's not that your =404 fired — the request dead-ended somewhere else, never reaching the original try_files. "Written below, so it runs after" is not what happens; the phases differ, so regardless of line position the rewrite fires first and jumps out of the location.

Symptom 2, followed by phase

if and add_header take effect in two phases that are far apart.

find-config     pick the location (/download/)
    ↓
rewrite         if is true → switch the config context to the inner block
    ↓           (the switch carries over into all later phases)
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         apply add_header when the response is being sent
                it looks at the block we switched into (X-Debug only)
                → the outer X-Served-By is gone

An if inside a location is an "implicit nested location" — it creates a config block of its own. And that if is a rewrite-phase directive. When the condition goes true, this phase switches the request's config context into the inner block, and that state carries over into the phases that follow. Much later, when add_header is applied at the point of sending response headers, what it reads is the block we switched into. All that's inside is X-Debug, and on top of that, add_header does not inherit from the outer scope once there's even one in the same block (it replaces rather than accumulates), so X-Served-By disappears entirely. If the condition is false, no switch happens and X-Served-By shows up as usual. The outer header vanishing is not about line order — it's because the switch that happened in the rewrite phase is still in effect in the later phases. What people call if is evil is, at bottom, this "context switch in the rewrite phase."

Fixes

Symptom 1

If you want to keep processing in the same location after the rewrite, change last to break. With break, the post-rewrite return to find-config doesn't happen, so the request can go down to the precontent try_files while staying in the same location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 break;   # last → break
    try_files $uri $uri/ =404;
}

If instead you want to route the request to a different location depending on the URI, leave last as is. In that case, put the fallback in the destination location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
}

location /public/ {
    root /var/www;
    try_files $uri $uri/ =404;   # put the fallback in the destination
}

Which one is right comes down to a design call: do you want to keep processing in the same location after the rewrite, or hand off to a different one?

Symptom 2

Don't put content-handling directives (add_header, proxy_pass, try_files, and so on) inside an if. Treat return and rewrite as about the only things that are safe inside if.

The condition here is a query argument (?debug=1), and location can't match on the query string. For branching on a value like this, push it out of if and into a map that builds a variable.

map $arg_debug $x_debug {
    default "";
    "1"     "on";
}

server {
    location /download/ {
        add_header X-Served-By "nginx" always;
        add_header X-Debug     $x_debug always;
        root /var/www;
        try_files $uri =404;
    }
}

The two add_header lines sit at the same level, so no nested block replaces anything and both take effect. When $arg_debug isn't 1, $x_debug is empty, and nginx won't emit an add_header with an empty value, so X-Debug shows up only during debugging.

Checking which location you're in right now

The feel for phases sticks once you get your hands dirty. The quick way is to plant one marker in a suspect location to tell whether the request settled there.

location /app/ {
    add_header X-Loc "app" always;   # marker that shows up if you stay in this location
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri =404;
}

Look at the headers with curl -sI http://localhost/app/foo, and if X-Loc: app isn't there, the request has already left this location via rewrite ... last (the add_header you wrote here has no effect once it's gone). Change last to break, hit it again, and if X-Loc: app appears, the request stayed in the same location and made it down to precontent. One marker header lets you see, across phases, which location the request is in right now.

Wrapping up

When you read an nginx config, your eyes follow the lines top to bottom. But what nginx actually does is run the request along a timeline of phases. When you're questioning a config, don't trace the line order — trace it in this order instead: "which phase does this directive run in? and at that point, which location is the request in?" That usually makes things line up.

nginx Tools for When Your Config Gets Out of Hand

Kazu — Tue, 04 Aug 2026 13:02:00 +0000

nginx config files grow. What starts simple gets layers added on top: another reverse proxy, TLS settings, routing logic that branches and branches again.

Before long you're looking at 20+ location blocks and can't say off the top of your head which one handles which request. You add a single add_header line and some other header silently disappears. You fix an alias path and prod throws 404s while staging is fine. nginx -t says "syntax is ok." But you don't feel ok about it.

Every config change comes with a quiet dread: will this break something? And that dread usually has a basis. nginx config mistakes are silent. Nothing shows up in the error log. The browser gets a normal-looking response. You don't find out something is broken until much later.

nginx itself is well-documented, but the ecosystem around it is harder to survey. Validation tools, config generators, test frameworks — searching turns up scattered information. This article organizes the common pain points into six categories and maps the tools that address each one.

Problem 1: Prevent config mistakes upfront
Problem 2: Generate config files
Problem 3: Automate testing
Problem 4: Collect metrics
Problem 5: Run nginx on Kubernetes
Problem 6: Visualize logs
Gaps that still aren't filled

Problem 1: Prevent config mistakes upfront

nginx -t checks syntax. But syntactically valid config can still have security problems. SSRF, path traversal via alias (e.g., a request to /files/../../etc/passwd reaches files it shouldn't), HTTP splitting — none of these trigger a syntax error.

Gixy: static analysis focused on security

https://github.com/dvershinin/gixy

Originally built at Yandex, now actively maintained as a community fork. It checks for well-known security risks: SSRF, alias traversal, HTTP splitting. Install the gixy-ng fork from PyPI.

pip install gixy-ng
gixy /etc/nginx/nginx.conf

Output looks like this — it tells you which directive is the problem and why:

[WARN] [http_splitting] Possible HTTP-Splitting via HTTP header.
Directive: proxy_set_header X-Forwarded-For $http_x_forwarded_for;

A good fit when you've just inherited an nginx config and want a quick security audit before touching it. Detection is limited to known patterns and there's no custom rule support, but as a first security gate in CI it offers the best return on effort.

crossplane: treat config as data

https://github.com/nginxinc/crossplane

A Python library that parses nginx config files into JSON. Useful not just for validation but for dynamically generating config or loading it in tests.

import crossplane

payload = crossplane.parse('/etc/nginx/nginx.conf')

# Check for parse errors
for error in payload['errors']:
    print(f"Error: {error}")

# Walk the config tree to find specific directives
def find_directives(block, name):
    for item in block:
        if item.get('directive') == name:
            yield item
        if 'block' in item:
            yield from find_directives(item['block'], name)

config = payload['config'][0]['parsed']

# Example: flag any proxy_pass that uses plain HTTP
for d in find_directives(config, 'proxy_pass'):
    url = d['args'][0]
    if url.startswith('http://'):
        print(f"Warning: proxy_pass uses plain HTTP: {url}")

The parsed output is a tree of directives and their arguments. With a traversal like find_directives, you can express project-specific rules in Python: "are all proxy_pass values HTTPS?", "is server_tokens off set?" Where Gixy is limited to known patterns, crossplane lets you write your own.

Scripting against config files and automating structural checks in CI are where this earns its place. If you don't have a concrete need to generate or programmatically inspect config, skip it for now. Adding it "because it looks useful" tends to mean it sits unused.

Which to choose

Goal	Pick
Find security issues quickly	Gixy
Manipulate or validate config in code	crossplane

Gixy alone covers the security CI gate. If you're writing config generation or structural validation logic in Python, you need crossplane. They don't overlap, so using both is fine.

Problem 2: Generate config files

Writing TLS config from scratch means making a lot of small decisions: cipher suites, HTTP/2 support, HSTS headers. Starting from a known-good template beats hand-rolling it and getting something subtly wrong.

nginxconfig.io: generate best-practice config in the browser

https://www.digitalocean.com/community/tools/nginx

A web app by DigitalOcean that generates a complete nginx config from a series of UI choices. Select your TLS cipher suites, gzip settings, HTTP/2, security headers, and it outputs ready-to-use config files. Free, no account required.

Use it when bootstrapping a new server or bringing TLS settings up to current best practices. It's an excellent starting point, but don't deploy the output unchanged. Treat it as a template and adjust for your project's requirements.

nginx-proxy: auto-generate config in Docker environments

https://github.com/nginx-proxy/nginx-proxy

A container that watches Docker start/stop events and automatically updates nginx config. Set a VIRTUAL_HOST environment variable on your container and nginx-proxy generates the reverse proxy config for it.

services:
  app:
    image: my-app
    environment:
      - VIRTUAL_HOST=example.com
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

When app starts, nginx-proxy automatically creates config to route example.com traffic to it.

In Docker Compose setups where manually updating nginx on every deploy is friction, this removes that burden. Traefik and Caddy do the same thing. If you're not already committed to nginx (existing config, team familiarity), compare them first. If you're already running nginx, nginx-proxy is the natural fit.

Which to choose

Environment	Pick
Regular Linux server	nginxconfig.io
Docker / Docker Compose	nginx-proxy

No Docker? nginxconfig.io is enough. In Docker environments, nginx-proxy's automation pays off.

Problem 3: Automate testing

Manually sending test requests after every config change doesn't scale. Automating it means the same checks run every time.

Test::Nginx: declarative test framework

https://github.com/openresty/test-nginx

A Perl-based test framework from the OpenResty community. It starts a real nginx instance, sends requests, and checks responses. The format is declarative — you describe what config to use, what request to send, and what response to expect. Tests read clearly as a result.

=== TEST 1: basic proxy
--- config
location /api {
    proxy_pass http://127.0.0.1:8080;
}
--- request
GET /api/users
--- response_body
[{"id":1}]

This fits nginx module development and verifying that config changes don't alter behavior. Perl is a real barrier for most teams today. If you're not writing custom nginx modules, hurl covers most of what you actually need.

hurl: a DSL for HTTP testing

https://hurl.dev/

hurl lets you write HTTP requests and assertions in a plain-text format, then run them from the shell. Easy to slot into CI. Not nginx-specific — it works for HTTP testing in general.

GET http://localhost/api/users
HTTP 200
[Asserts]
header "Content-Type" contains "application/json"
jsonpath "$.length" > 0

hurl --test api.hurl

Reach for it when you want to confirm API behavior hasn't broken after a config change, or when the team has no Perl experience. Because it's not nginx-specific, it doesn't go deep on nginx internals. It works well for endpoint reachability checks, but for testing precise nginx behavior (match precedence, path rewriting), Test::Nginx is the right tool.

Which to choose

Situation	Pick
Testing nginx config or modules directly	Test::Nginx
Checking that the API behind nginx still works	hurl
Team has no Perl experience	hurl

Test::Nginx for precise nginx behavior verification. hurl for HTTP interface testing with a lower adoption cost.

Problem 4: Collect metrics

Without visibility into how many requests nginx is handling and where the bottlenecks are, there's no basis for making informed decisions when problems surface.

nginx-module-vts: embedded stats module

https://github.com/vozlt/nginx-module-vts

A module that adds traffic statistics to nginx — request counts, error rates, latency per virtual host and upstream. Built in at compile time. Exposes a /status endpoint that returns JSON.

vhost_traffic_status_zone;

server {
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
}

Detailed traffic stats are the payoff — when you have a build environment and can compile nginx from source. Requiring a source build is the biggest barrier. If you're using a packaged nginx install, this isn't an option. If you already have Prometheus, the overhead of a custom build may not be worth it.

nginx-prometheus-exporter: bridge stub_status to Prometheus

https://github.com/nginxinc/nginx-prometheus-exporter

Reads nginx's built-in stub_status data (connection counts, request totals) and exposes it in Prometheus format. Runs as a separate process alongside nginx — no changes to nginx itself required.

location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

./nginx-prometheus-exporter -nginx.scrape-uri=http://localhost/nginx_status

In Prometheus + Grafana stacks, this is the easiest way to add nginx metrics without touching nginx itself. stub_status only gives you connection counts and request totals — no per-upstream breakdown. Confirm that's enough before committing.

Which to choose

Situation	Pick
Already running Prometheus	nginx-prometheus-exporter
Need per-host, per-upstream, per-status-code stats	nginx-module-vts
Using packaged nginx and don't want to rebuild	nginx-prometheus-exporter

If you're on Prometheus, nginx-prometheus-exporter is by far the easier path. If you need granular metrics and control your build, look at vts.

Problem 5: Run nginx on Kubernetes

When exposing services externally in Kubernetes, you use an Ingress resource — a rule that maps incoming paths to backend services. nginx-based Ingress controllers are widely used for this. Two controllers with similar names exist, and they're frequently confused.

ingress-nginx: the Kubernetes community controller

https://github.com/kubernetes/ingress-nginx

Maintained by the Kubernetes community (kubernetes/ingress-nginx). Used as the default by many Kubernetes distributions. Large amount of documentation and community knowledge available.

In March 2025, a set of serious vulnerabilities called IngressNightmare was disclosed (CVE-2025-1974 and others, CVSS 9.8). Remote code execution was possible through the admission controller component. Fixed in versions 1.12.1 and 1.11.5. If you're running an older version, upgrade now.

On a general-purpose Kubernetes cluster running standard Ingress, this is the obvious starting point. Switching Ingress controllers later is painful, so the initial choice matters. When in doubt, ingress-nginx has fewer dead ends from a documentation standpoint.

kubernetes-ingress: the NGINX Inc. controller

https://github.com/nginxinc/kubernetes-ingress

Maintained directly by NGINX Inc. (nginxinc/kubernetes-ingress). Supports both open-source nginx and NGINX Plus (the commercial version).

The case for it is NGINX Plus features (advanced load balancing, active health checks) or a support contract with NGINX Inc. If you don't need NGINX Plus, there's little reason to choose this over ingress-nginx. For OSS nginx, the community controller has better coverage.

Which to choose

Situation	Pick
Community resources matter	ingress-nginx
Planning to use NGINX Plus	kubernetes-ingress
Starting on EKS / GKE or similar managed cluster	ingress-nginx

If you go with ingress-nginx, run version 1.12.1 or 1.11.5 or later. Check your current version before anything else.

Problem 6: Visualize logs

nginx access logs pile up as text. Figuring out which paths are getting hammered, or where errors are spiking, is hard to do from raw log files.

GoAccess: real-time stats in the terminal

https://goaccess.io/

A TUI tool that aggregates nginx access logs and displays them in real time. Graphs and stats update live in the terminal. No log server or database needed — just install and run. Can also generate HTML reports for browser viewing.

goaccess /var/log/nginx/access.log -c
# Starts in interactive mode to select the log format

# Stream live logs in COMBINED format
tail -f /var/log/nginx/access.log | goaccess --log-format=COMBINED -

Reach for it when you need answers quickly, especially on a server with no log collection infrastructure in place. It reads local log files directly, so multi-server environments are awkward. Think of it as a per-server quick-look tool.

Loki + Grafana Alloy: long-term log storage and search

https://grafana.com/oss/loki/ / https://grafana.com/docs/alloy/latest/

Loki stores and queries logs efficiently, with Grafana for dashboards. Grafana Alloy handles log collection. Alloy replaces Promtail, which reached end-of-life in March 2026 — for new setups, use Alloy.

Example Alloy config (sending nginx logs to Loki):

loki.source.file "nginx" {
  targets = [{
    __path__ = "/var/log/nginx/access.log",
    job      = "nginx",
  }]
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

It earns its place when you need to aggregate logs from multiple servers, search historical logs by time range, or view nginx logs alongside other metrics in Grafana. Running the full stack yourself carries real operational overhead. Start with Grafana Cloud's free tier, or bring this in when someone on the team already knows how to operate Grafana.

Which to choose

Situation	Pick
Need to check something right now	GoAccess
Already running Grafana	Loki + Grafana Alloy
Need long-term log retention and cross-server search	Loki + Grafana Alloy
Debugging with direct SSH access to the server	GoAccess

GoAccess gives you immediate answers. Loki + Alloy is for ongoing operations. They're not mutually exclusive — GoAccess for fast situational awareness, Loki for accumulating and analyzing over time.

Summary: match tool to problem

Problem	Tool
Find security risks in config	Gixy
Manipulate config programmatically	crossplane
Generate correct config (Linux server)	nginxconfig.io
Auto-manage Docker reverse proxies	nginx-proxy
Test nginx config behavior	Test::Nginx
Add HTTP connectivity checks to CI	hurl
Collect metrics via Prometheus	nginx-prometheus-exporter
Need detailed per-upstream stats (module)	nginx-module-vts
Set up Ingress on Kubernetes	ingress-nginx (1.12.1+)
Need NGINX Plus features	kubernetes-ingress
Visualize logs immediately	GoAccess
Long-term log storage and search	Loki + Grafana Alloy

nginx itself is stable and mature, but the ecosystem moves fast — Promtail's EOL is a good example. Periodically check the maintenance status of tools you rely on, or you'll find out about a support cutoff at the worst time.

Gaps that still aren't filled

The tools covered here leave some problems unsolved or underserved. If you know tools that address any of these, drop them in the comments.

Dynamic upstream management

Updating upstreams without an nginx reload is a common requirement. With open-source nginx, the only options are OpenResty (Lua scripting for dynamic upstream control) or NGINX Plus (API-driven management). Vanilla nginx requires rewriting config and reloading.

Config drift tracking across environments

There's no standard tool for tracking and visualizing how nginx config differs between dev, staging, and production. Git-based management is the practical answer, but cross-environment diff review and detecting prod-only config still means manual work.

IDE / LSP support

VSCode extensions for nginx.conf exist, but completion and inline validation are incomplete. No editor today reliably explains location block match priority or how proxy_pass transforms paths.

WAF

ModSecurity has a solid track record as an nginx WAF, but rule management and false-positive tuning are heavy ongoing work. There's nothing you can drop in and operate at a low maintenance level.

https://github.com/owasp-modsecurity/ModSecurity-nginx

Distributed tracing

The nginx OpenTelemetry module (ngx_otel_module) shipped in 2023 but is still maturing. Trace collection with Jaeger or Tempo is possible, but documentation and real-world operational experience are thin.

https://github.com/nginxinc/nginx-otel

$request_time Looks Fast, But Users Say It''s Slow: Breaking nginx Latency into 4 Parts

Kazu — Tue, 28 Jul 2026 13:03:00 +0000

Your nginx $request_time looks fine in the dashboard — p95 is 120ms, totally reasonable.

But users keep reporting that things "occasionally freeze." You dig through the logs looking for slow requests and find nothing obvious. Overall p95 is healthy. The upstream app team says "our side is fast." nginx looks fast too. Yet the complaints are real. You don't know where to look.

$request_time collapses everything that happened during the request into a single number. Fast phases and slow phases all get averaged together, making it impossible to see where the time actually went.

nginx does record timestamps at each intermediate checkpoint, though. Subtract them, and latency breaks into 4 distinct parts.

The log variables covered here and what they measure are the same regardless of nginx version.

What one number hides

$request_time measures everything from when nginx reads the first byte from the client to when it finishes sending the last byte of the response. In a reverse proxy setup, that single number includes at least all of:

time to establish a connection to the upstream backend
time until the backend starts returning a response (backend processing time)
time to receive the full response from the backend
nginx's own processing and the time to send the response back to the client

All of this gets summed into one $request_time. So even if p95 is 120ms, you cannot tell from that number alone whether the breakdown is "20ms backend, 100ms sending" or "100ms connecting, 20ms everything else." Those scenarios call for completely different fixes, but $request_time erases the distinction.

Also worth noting: the TLS handshake on the client side happens before $request_time starts, since the clock starts at "first byte received." If you suspect TLS overhead and stare at $request_time, it won't show up there.

Before you start: add variables to your log format

The breakdown analysis below requires $upstream_connect_time, $upstream_header_time, and $upstream_response_time to be present in your access log. Open your own logs and there's a good chance none of these are there.

The reason is simple: nginx's default combined log format doesn't include any of them. combined logs eight fields — remote address, user, timestamp, request line, status, response bytes, referer, and user agent. $request_time isn't even there. Upstream timing variables are excluded by design.

Until you define a custom log_format and reload nginx, this breakdown is physically impossible. You can't subtract what doesn't exist. Fix the log definition first.

log_format latency '$remote_addr "$request" $status '
                   'rt=$request_time '
                   'uct=$upstream_connect_time '
                   'uht=$upstream_header_time '
                   'urt=$upstream_response_time';

access_log /var/log/nginx/access.log latency;

After adding this and running nginx -s reload, the raw data you need starts appearing in logs. Logs written before the reload won't have these fields, so there's no way to retroactively see the breakdown for historical requests.

There's also an approach that captures timing directly from the kernel without touching the nginx config. ngxray explores that direction (currently in development).

Breaking it into 4 parts

When nginx operates as a reverse proxy, it records three timestamps for the upstream interaction. These are the basis for the breakdown.

$upstream_connect_time: time until the connection to the upstream is established (includes TLS handshake if the upstream is HTTPS)
$upstream_header_time: time until nginx starts receiving the response headers from the upstream
$upstream_response_time: time until nginx has received the full response from the upstream

All three are measured from the same starting point — the moment nginx begins processing the upstream. They're not three independent intervals; they're cumulative values. $upstream_response_time contains $upstream_header_time, and $upstream_header_time contains $upstream_connect_time. They're nested.

Once you understand they're cumulative, subtraction gives you the individual intervals.

Part	Formula	What it measures
① Connect	`$upstream_connect_time`	Time to connect to upstream (TLS included)
② Backend processing	`$upstream_header_time − $upstream_connect_time`	Time until upstream starts responding (TTFB)
③ Response transfer	`$upstream_response_time − $upstream_header_time`	Time to receive the full response from upstream
④ Client + nginx internal	`$request_time − $upstream_response_time`	Request body read, nginx internals, sending to client

Don't read ④ as "nginx is slow." It captures request body reading, rewrite/access phases, and the time to finish delivering the response to the client. When users on slow connections are the only ones complaining, ①②③ will look fine while ④ carries all the time. Slow clients only become visible through this breakdown. This assumes proxy_buffering is on (the default), where nginx buffers the full upstream response before sending to the client. If buffering is off, slow-client impact can bleed back into ③, so suspect both in that case.

And don't look at averages. Break each of the four parts into p50 / p95 / p99 separately. Whichever part has a spiking p99 is your answer.

For example, laying out p50 / p95 / p99 per part might reveal a distribution like this (these numbers are for illustration):

Part             p50     p95     p99
① connect         1ms     2ms     3ms
② backend        15ms    40ms    60ms
③ transfer        3ms     8ms    12ms
④ client+nginx    2ms     5ms   380ms   ← here

Even when $request_time p95 looks clean at 120ms, the p99 for ④ alone can spike to 380ms. That's what "occasionally freezes" actually is — buried in the overall average and p95, but visible as soon as you look at p99 per part.

Once you know which part is the problem, the next action follows directly: ① is thick → upstream keepalive probably isn't configured (covered in the next section); ② is thick → backend processing is the bottleneck; ③ is thick → look at transfer size or upstream bandwidth; ④ is thick → suspect client connection speed or response body size.

Even the nginx official blog's explanation writes that $upstream_header_time measures the time "between establishing a connection and …" — it's ambiguous whether that excludes $upstream_connect_time or includes it. Rather than memorizing each variable's exact definition, it's more reliable to internalize the structure: all three accumulate from the same origin, and you subtract to get intervals.

Computing p50/p95/p99 per part

"Look at p99 per part" is easy to say, but how do you actually get those numbers? Using the latency format defined above, here's an awk example to compute percentiles.

To extract percentiles for part ④, client-side (rt − urt):

awk '{
  for (i = 1; i <= NF; i++) {
    if ($i ~ /^rt=/)  { rt  = substr($i, 4) }
    if ($i ~ /^urt=/) { urt = substr($i, 5) }
  }
  if (rt != "" && urt != "" && urt + 0 >= 0) a[++n] = rt - urt
}
END {
  asort(a)
  printf "p50=%.3f  p95=%.3f  p99=%.3f\n",
    a[int(n * 0.50)], a[int(n * 0.95)], a[int(n * 0.99)]
}' /var/log/nginx/access.log

The structure is the same for every other part. For ② backend processing use uht - uct; for ③ response transfer use urt - uht.

In production you can run equivalent aggregations in Loki, CloudWatch Logs Insights, or Datadog log queries. The key is not "overall p95" but "p99 per part, listed separately." No matter how good a single number looks, the problem stays hidden until you split the breakdown and look at each part's p99.

When ① is always non-zero

When you break down the parts, you occasionally see a pattern where ②③④ are all small but $upstream_connect_time registers as non-zero on every single request. The responses themselves are lightweight, but there's a small constant overhead on every request.

The TCP connection to the upstream isn't being reused. When connections are kept alive and reused, $upstream_connect_time should be near 0 (or -) after the first request. If it's consistently non-zero on every request, nginx is creating a new connection for each one — keepalive to the upstream isn't configured. This tends to get overlooked in long-running systems.

Two directives work together here:

upstream backend {
    server 127.0.0.1:3000;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

keepalive 32 sets the maximum number of persistent connections the worker keeps in the pool. proxy_http_version 1.1 and proxy_set_header Connection "" are required because HTTP/1.0 sends Connection: close by default, which causes the upstream to close the connection immediately. Without those two lines, keepalive does nothing.

After adding the config and running nginx -s reload, check the uct= values in the access log. The first request will be non-zero since it's establishing the connection. If subsequent requests show 0.000 or -, connection reuse is working.

Repeatedly opening new connections also leads to TIME_WAIT buildup and ephemeral port exhaustion, but that's a different layer of concern. For this article, the key read is: "① consistently non-zero = connections aren't being reused."

When the breakdown breaks: commas and colons

There's another pattern that breaks the subtraction: looking at a $upstream_* value and finding more than one number.

uct=0.001, 0.003 uht=0.012, 0.045 urt=0.012, 1.230

This isn't a broken log. nginx logs multiple values when a single request involves multiple upstream interactions. The delimiter carries meaning.

Comma-separated (0.001, 0.003): the request hit multiple upstream servers in sequence. The typical case is the first server failed and the second was tried as a retry/failover.
Colon-separated (0.001 : 0.003): an internal redirect via X-Accel-Redirect or similar occurred, spanning multiple upstream groups.

If you treat this as a single value and compute $request_time − $upstream_response_time, the subtraction breaks completely. The string "0.012, 1.230" isn't a number.

More importantly, multiple values are a signal of something abnormal. Two comma-separated values mean there was a retry — the first upstream attempt failed. That failure was buried in the average and invisible. When a slow request shows comma-separated upstream times, investigate the retry before worrying about the interval breakdown.

Closing: where is your time going?

Rather than wrapping this up as a checklist of solutions, here are questions to bring to your own logs.

Does your access log include $upstream_connect_time, $upstream_header_time, and $upstream_response_time? Are you still running the combined format?
Are you looking at p50 / p95 / p99 separately for each of the 4 parts, not just $request_time as a whole?
Have you ever computed $request_time − $upstream_response_time (part ④, client + nginx internal)?
Is $upstream_connect_time non-zero on every request?
Can you interpret what it means when $upstream_* has multiple values separated by commas or colons?
Can you explain "numbers look fast but users say slow" using per-part p99 rather than averages?

If any answer is no, you haven't broken down latency yet. As long as that's the case, "fast numbers, slow experience" stays a permanent contradiction.

$request_time doesn't lie. It just compresses too much into one number. Split it four ways and you can always pinpoint where the time went. Start by opening your access log and checking whether $upstream_response_time is even being recorded.

References (official documentation)

Every variable and behavior discussed here is documented in the official nginx docs.

ngx_http_upstream_module: definitions of $upstream_connect_time, $upstream_header_time, $upstream_response_time; meaning of comma/colon separators; TLS handshake handling
ngx_http_log_module: log_format and the default combined format definition
Using NGINX Logging for Application Performance Monitoring: upstream timing variable usage
nginx source ngx_http_upstream.c: confirmation that all three variables share the same origin (u->start_time) and the nested structure

Silent nginx Config Bugs That Pass nginx -t: if, location, add_header, alias

Kazu — Tue, 21 Jul 2026 13:02:00 +0000

nginx -t returns syntax is ok and test is successful. nginx -s reload finishes without a complaint. Nothing shows up in the error log.

And yet the behavior is wrong.

I've done this to myself more times than I'd like to admit. The one that hurt most was a CSP header I shipped to production that never reached a single browser for a full day — I only noticed because I got suspicious that no violation reports were coming in. There have been others: a file I put under /static/ getting swallowed by a different location, old behavior lingering after a reload. In every case nginx -t passed, and nothing was grammatically wrong anywhere.

The nature of this gap is clear: nginx -t validates only syntactic correctness, and never looks at semantic correctness.

What `nginx -t` actually guarantees for you

What nginx -t does, roughly speaking, is check "is this config file in a shape that nginx can load?" Does the directive name exist? Is the argument count right? Are the {} blocks closed? Are the value types valid? In compiler terms, it goes about as far as syntax errors and maybe type checking.

Everything past that point — "will this config process requests the way you intended?" — is outside the scope of nginx -t. This is where the blind spot is. When the test passes, you quietly reread it as "the config is correct." But all that passed was the syntax test.

Outside nginx -t, there are three layers of breakage.

Semantic level — the config isn't read the way you intended. The grammar is perfect.
Security level — it works exactly as you intended. It just also works exactly as the attacker intended.
Runtime level — the config text is irrelevant. It's the behavior of the reload operation itself.

The higher layers are the ones you step on daily, and their cause is easy to look for inside the config file. The lower you go, the more it becomes the kind of incident where reading the config file gives you no answer at all. We'll go top to bottom. Reading this with your own nginx.conf open beside you should help.

Note that this article assumes the behavior of the nginx 1.31 (mainline) series. The stable series is 1.30, and the behavior covered here — if, location selection, add_header inheritance, alias, and reload — is the same across both. Where a version makes a difference, I'll call it out explicitly.

Semantic level: the grammar is correct, but it's read differently

The contents of `if` aren't evaluated in the order you think

When you want to "branch on a condition" in an nginx config, if is the first thing you reach for. But an if inside a location has long been called "if is evil" in the nginx community. It sounds like a religious argument if you only hear the name, but it's a statement about behavior.

First, why it breaks. if is a directive of the rewrite module, evaluated in an early phase of request processing (the rewrite phase). And the nastier part: when you write an if inside a location, that if block is treated as an implicit nested location. When the condition is true, nginx switches the request's entire configuration context into that inner location. Not every directive gets dropped (many settings like root or proxy_set_header carry over). But content-processing directives like try_files or proxy_pass may not carry over the way you expect. That's the trap.

Let's look at a concrete example. A common, simple setup: serve a static site, but route just the requests under /api/ to a backend. It's completely valid grammatically and passes nginx -t:

location / {
    root /var/www/html;
    try_files $uri $uri/ =404;      # serve static files

    if ($uri ~ ^/api/) {
        proxy_pass http://backend;  # route only /api/ to the backend...
    }
}

The author's intent is "normally serve static files, and only proxy to the backend when the path starts with /api/." But if creates an implicit nested location, so when the condition is true the config context switches wholesale into it. Inside there's only proxy_pass; the outer content processing like root and try_files isn't carried into the switched-to location. Worse, once there's an if in the location, try_files can stop being evaluated as expected even for requests where the condition is false. The grammar is correct and nginx -t says nothing, so you get "occasional 404s" and "occasional un-proxied requests" without ever noticing.

Detection and avoidance: The things that are genuinely safe inside an if in a location are, in practice, only return ..., rewrite ... last, set ... (and break). For anything else — try_files, proxy_pass, header manipulation — the moment you want to do it inside an if, that's a sign the design is wrong. When you want to "route by path" like here, the answer is not if but splitting the location. Routing in nginx is location's job; if breaks it by cutting in.

Rewriting the earlier example by splitting the location looks like this:

location / {
    root /var/www/html;
    try_files $uri $uri/ =404;      # static serving is this block's only job
}

location /api/ {
    proxy_pass http://backend;      # /api/ is a separate location from the start
}

Making /api/ its own location means no implicit nesting and no config-context switch. The static-serving location / and the proxying location /api/ each hold only their own processing. Replacing conditional branching with "location as routing" rather than "if as a control statement" is the basic form of avoiding if. When you want to branch on a value rather than a path, push the condition out into a map (a lookup table that maps an input value to an output value) with the same mindset, and use the resulting variable.

Location priority is not the order you wrote them in

When you write multiple location blocks, don't you assume "the one written higher up takes priority"? I did. And it bit me.

nginx's location selection runs on a rule that differs from the order you wrote. The procedure is this:

If an = (exact match) location matches the URI exactly, it's decided right there and nothing else is examined.
Otherwise, among the prefix locations (no modifier, or ^~), pick and remember the one with the longest match. Order within the config file is irrelevant here. The longest match wins.
If the selected prefix has ^~, it's decided there and regex is not checked.
Otherwise, check the regex locations (~, ~*) in written order, and the first that matches wins. Here, written order matters.
If no regex matches, use the prefix remembered in step 2.

"Prefixes are longest-match (order-independent), regexes are first-match by written order, and regexes take priority over prefixes." That last point is what gets people. A reproduction:

location /static/ {
    root /var/www;
}

location ~ \.(png|jpe?g|gif)$ {
    expires 30d;
    root /var/www/cache;   # <- a different root
}

Let's trace a request to /static/logo.png. As a prefix, /static/ is the longest match and gets "remembered." But /static/ has no ^~. So nginx proceeds to the regex check, \.(png|jpe?g|gif)$ matches, and that one wins. The result: /static/logo.png goes looking in /var/www/cache. The /static/ block is skipped past. The grammar is perfect, nginx -t passes. Only your assumption — "I'm handling all static files under /static/" — is off.

Detection: Don't guess which location you're hitting — confirm it. The easy way is to temporarily add a marker header to the suspect locations:

location /static/ { add_header X-Loc "static" always; ... }
location ~ \.(png|jpe?g|gif)$ { add_header X-Loc "regex" always; ... }

Then hit curl -sI http://localhost/static/logo.png | grep X-Loc and you'll know instantly which one answered. Nailing down "where am I hitting right now" as a fact comes first.

Once you have that, if you want to lock it down, put ^~ on the prefix to stop the regex check itself:

location ^~ /static/ {   # decided the moment this prefix is the longest match; regex is not examined
    root /var/www;
}

With `add_header`, adding one in the child makes all the parents disappear

This is the trap that most symbolizes the "semantic level" of the three layers, and the one I was stuck on the longest.

As a premise, many people write security headers like HSTS or CSP once in the server block, intending them to apply to every location.

server {
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header Content-Security-Policy "default-src 'self'" always;

    location /api/ {
        add_header Cache-Control "no-store" always;   # <- the moment you add this
        proxy_pass http://backend;
    }
}

You just added one cache-control header to /api/. But nginx's add_header is specified so that if there's even one add_header at the current level, it does not inherit any add_header from the parent level at all. It's replacement, not merging. The result: HSTS and CSP disappear from the /api/ response, and all that's left is Cache-Control. Only the security headers go silently missing.

This "array-style directives don't inherit (don't merge with) the parent once defined in a child" behavior is not unique to add_header. proxy_set_header and fastcgi_param have the same trap. Write this, for example, and Host and X-Forwarded-For stop being passed to the backend:

server {
    proxy_set_header Host            $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location /api/ {
        proxy_set_header X-Request-Id $request_id;   # <- just added one
        proxy_pass http://backend;
        # neither Host nor X-Forwarded-For is inherited here
    }
}

Detection: Missing headers — the only sure method is to actually look at the response. At each endpoint where you "think" you set a header, compare against the real thing.

curl -sI https://example.com/         | grep -i -E 'strict-transport|content-security'
curl -sI https://example.com/api/foo  | grep -i -E 'strict-transport|content-security'

Present at the top but gone under /api/ is the classic signature of this trap.

How to deal with it: The most portable fix is to re-list every header you need in the child block. Split the common part into a separate file and include it, so you gather the duplication into one place while re-declaring it at each level:

# security-headers.conf
add_header Strict-Transport-Security "max-age=63072000" always;
add_header Content-Security-Policy   "default-src 'self'" always;

server {
    include security-headers.conf;

    location /api/ {
        include security-headers.conf;      # re-read and re-declare in the child too
        add_header Cache-Control "no-store" always;
        proxy_pass http://backend;
    }
}

The add_header_inherit merge; added in nginx 1.29.3 lets you change the behavior to "inherit the parent, then add the child" (it's in both the current stable 1.30 series and mainline 1.31 series). But it's not in the 1.28-and-earlier stable series, so it may not be available on your version — getting the premise "it's replacement" into your bones is more effective first.

Forget one trailing slash on `alias`, and you can climb outside the directory

Everything so far has been about "behavior that differs from intent." This alias trap is the same kind of mistake — a single-slash difference — but the quality of the outcome is different. The config looks like it works as intended and does, yet you can read files outside the published directory. It's not just a behavior bug; it's a security hole as-is.

A reproduction. It's the commonplace setup of "I want to serve static files from a different directory":

location /assets {           # <- no trailing slash
    alias /var/www/static/;  # <- this one has a trailing slash
}

/assets/logo.png is served normally. nginx -t passes, and as far as the browser is concerned there's no problem. It looks like it works as intended.

Why it's dangerous. alias builds the real file path by replacing the portion of the URI that matched the location (here, /assets) with the value of alias. When the location is /assets (no slash), the string following /assets gets stuck on directly at the end. So a request for /assets../ resolves to /var/www/static/ + ../ = /var/www/static/../ = /var/www/. That means with a little trick like /assets../../etc/passwd, you can climb outside the intended published directory. It's a classic path traversal (an attack that walks back up directories to read non-public files on the server). Normal access never notices it at all.

Detection and fix: The way to spot it is simple — line up the trailing slashes on the location and the alias. Either both have one, or neither does. In the example above, add a trailing slash to the location side too, like this:

location /assets/ {          # <- add the trailing slash so they match
    alias /var/www/static/;  # <- alias side has a trailing slash too
}

Now /assets../ no longer matches this location in the first place, and the climb is gone. Normal requests like /assets/logo.png are still served exactly as before.

If the location and the directory name match in the first place, the safest move is to drop alias and lean on root instead. root doesn't replace the location portion — it appends the URI as-is under the directory — so this trailing-slash problem simply can't occur:

location /assets/ {
    root /var/www/static;    # /assets/logo.png -> /var/www/static/assets/logo.png
}

(Note that root appends the URI path verbatim, so the real directory has to be laid out as /var/www/static/assets/ too. When you don't want to change an existing physical layout, take the alias + matched-trailing-slash route instead.)

To audit an existing config, use nginx -T (below) to dump all locations, and eyeball just the blocks containing alias for the trailing-slash correspondence. To reproduce it at hand, poke one level up with curl: if curl -sI 'http://localhost/assets../' returns not a 200 or 403 but "something that shouldn't be visible," you've got a hit.

Security level: it works exactly as intended, which is why it's dangerous

The semantic-level traps were "behavior that differs from intent." From here, the quality changes. The config works exactly as you intended. It just also works exactly as the attacker intended. So it passes not only nginx -t but even a visual review, on a "well, it's working" basis.

Passing a client-touchable variable straight into `proxy_pass`

Using a variable for the proxy_pass destination lets you decide the destination dynamically per request. Handy. But it's a different story if the source of that variable is client-derived.

location /fetch/ {
    proxy_pass http://$arg_target;   # ?target=... decides the destination
}

With a request like /fetch/?target=internal-admin:8080/, nginx can relay the request to any host inside your network. This is a textbook entry point for SSRF (server-side request forgery, an attack that uses your server as a stepping stone to send requests to places it otherwise couldn't reach). Including a variable in proxy_pass also changes how the URI is handled and how name resolution behaves (you need a resolver if you use a hostname), and while you're distracted by those details, it's easy to overlook the danger that the destination is client-controlled.

I won't chase this further here — SSRF is a topic that deserves its own article. The one thing I want to nail down in this piece: "nginx -t passes" and "it's safe" are entirely separate things. Whether a value the client can influence flows into proxy_pass, a rewrite destination, or a root/alias path — that perspective is completely outside the syntax check.

Runtime level: the config text is no longer relevant

The last layer has a different flavor. Here the contents of the config file are correct. What "looks" broken is the behavior of the reload operation itself. Stare at the config and the answer isn't written there.

After a reload, the old behavior lingers for a while

You fix the config and nginx -s reload. It succeeds. And yet, for a while, the old behavior is observed. It looks like a bug, but this is nginx's normal behavior.

When it receives a reload (SIGHUP to the master process), nginx doesn't suddenly switch everything over. It starts new worker processes with the new config and tells the old workers to shut down gracefully. Gracefully — that's the point. The old workers stop accepting new connections, but they take care of requests already in flight to the end before terminating.

Normally this finishes in an instant, so you don't notice. The problem is when there are long-lived connections. WebSockets, large file downloads, long polling. An old worker holding one of these stays put until that connection ends. In other words, a window forms where the old and new configs are alive at the same time. That's the true identity of "the old behavior lingers after a reload." The config has been updated. It's just that a worker with the old config is still running.

Detection: This one you look at processes, not the config.

ps -eo pid,ppid,command | grep '[n]ginx'

Old workers linger with the label nginx: worker process is shutting down. If this "shutting down" keeps sitting alongside normal workers (nginx: worker process), that's the substance of old-and-new coexisting. If you want to cap how long they linger, set worker_shutdown_timeout:

worker_shutdown_timeout 30s;   # force old workers to terminate after at most 30 seconds

The important thing here is not to chase this as a bug. The cause isn't written in the config file — it's in the single fact that "reload is that kind of operation."

Also, confirming worker generations with ps requires being able to log into that server. I'm also working on a small tool that peeks — from the outside, without touching the running nginx at all — at which worker generation is holding connections after a reload, using eBPF (ngxray, work in progress).

Summary: how to fill in the space outside the syntax check

nginx -t doesn't lie. It just answers a narrow question. "Is it in a loadable shape?" — it answers that. "Is it read as intended?" "Can it be abused by an attacker?" "What happens during a reload?" — it was never answering those to begin with.

Each of the three layers is filled in differently.

Semantic level (if, location priority, add_header inheritance drops) can be caught by looking at the actual response. Marker headers and the difference in curl -I. Get into the habit of comparing behavior with curl -sI rather than reading the config and convincing yourself. Flattening the "config that's actually in effect" once with nginx -T (uppercase; it dumps the final config with all includes expanded) also helps.
Security level is about tracing where client-derived values flow. Whether external input reaches proxy_pass, a destination, or a path.
Runtime level is about looking at processes and connections, not the config. Check worker generations with ps.

The syntax check is only the first sheet of the layers. From the second sheet down, for now, I fill it in with human observation and discipline. Having a machine watch this space — reading the config not just by syntax but by "meaning," warning about location overtakes and header inheritance drops — static analysis, and a mechanism that continuously observes the actual responses, is the topic I want to think about next.

For now, one line is all I want you to take home. When nginx -t passes, the next thing to hit is curl -sI. That the syntax is correct and that it works as intended are two things you confirm separately.

References (official documentation)

The behavior in this article can all be backed up by official documentation. When you suspect your config, this is the first place to check.

ngx_http_rewrite_module (if) and If Is Evil — the story of if creating an implicit location
The location directive — the selection algorithm (longest prefix and regex priority)
ngx_http_headers_module (add_header) — inheritance rules and add_header_inherit, added in 1.29.3
The alias directive — how the real file path is assembled
proxy_pass — behavior when using a variable in the destination, and resolver
Controlling nginx and worker_shutdown_timeout — reload and graceful shutdown

"How I Designed a Go Linter's Rule Engine So Adding a Rule Is One Line"

Kazu — Tue, 14 Jul 2026 13:02:00 +0000

"Just add one more check" — and before you know it, a switch grows another branch, the config grows another case, and you're copy-pasting the same setup you already wrote somewhere else. If you've ever worked on code where rules or checks keep piling up, you probably recognize this kind of rot.

gomarklint is a Markdown linter, and it currently ships 24 checks (rules). Yet adding a new one touches just a function, one line in a table, and two spots in the config. The switch doesn't grow. No wiring to add. The cost of adding a rule stays flat no matter how many there are. This post takes that apart from the inside — how that state is kept.

First, let's run gomarklint

Before we go inside, let's see what the tool actually does, once. gomarklint is simple: it reads Markdown files, finds structural and link problems, and points at them with a file:line. Say you have a README with a link that goes nowhere.

# My Project

See the [docs]() for details.

Run gomarklint on it and you get:

$ gomarklint README.md

Errors in README.md:
  README.md:3: [error] no-empty-links: link has empty destination: [docs]()

✖ 1 issues found
✓ Checked 1 file(s), 3 line(s) in 1ms

Because an [error] was printed, the exit code is non-zero. In CI, this is where the build fails.

When this post says "rule," it means one of those checks that produces a single line of output like that. "Find empty links." "Check that heading levels don't skip." "Check for duplicate headings." There are 24 such checks today, and each one returns "which line, and what's wrong." gomarklint's own job is to collect those findings (the LintError you'll see in code shortly), sort them, and print them.

Here's the real question. How do you keep those 24 checks at "adding one is a one-line change"? We'll open it up in order — the diagnostic type, how rules are laid out, the shared preprocessing that skips code and HTML, and how severity wires into the exit code — and finish by adding a real rule through an actual PR (#106).

Even if you're not writing a linter — a validator, a CI check, a rule-based plugin system — if you write "a mechanism where items keep growing" in Go, this should be a template you can copy directly.

Rules pile up, and code usually rots

Take a check like that "find empty links" one, and now imagine 24 of them. Built naively, the dispatch and configuration rot a little more with every rule you add. The usual four traps:

Every new rule makes some switch longer. You can't tell where a new rule should be registered, so you get missed registrations and double registrations.
Small setup logic like "don't flag [x]() inside a code block" gets reimplemented in each rule. Fix it in one, and another stays stale.
The items you can write in the config file and the default values in code become two sources of truth, and one gets updated while the other drifts.
The "error or warning?" decision — the same one that just set our exit code — plus its counting and its effect on the exit code get scattered around, and CI fails or passes not quite as intended.

gomarklint would have stepped on every one of these if left alone. This post walks through how each is avoided, in order.

First, express diagnostics, config, and severity as minimal types

At the center of the engine sit three small types. Everything wobbles if these wobble, so they're kept deliberately plain.

The first is the violation itself.

type LintError struct {
    File     string
    Line     int
    Rule     string
    Message  string
    Severity string
}

The point is that Rule and Severity are in there but are filled in by the engine, not the rule — as we'll see later. A rule function only has to return "which line, and what happened."

The second is per-rule configuration.

type RuleConfig struct {
    Enabled  bool
    Severity RuleSeverity
    Options  map[string]interface{}
}

The third is severity, which has only three values.

const (
    SeverityError   RuleSeverity = "error"   // presence of this makes the exit non-zero
    SeverityWarning RuleSeverity = "warning" // reported, but doesn't fail the run
    SeverityOff     RuleSeverity = "off"     // disabled
)

If there's an error, CI fails. Hold onto just that one fact, and the rest of the branching comes back to it.

Rules are functions, not an interface

This is what changed the most in the engine.

The obvious design is to define a Rule interface and make each rule implement Check(). gomarklint doesn't do that. A rule is just a function, and the functions are laid out in a table.

They even split into two families by signature. One takes just the raw slice of lines — the simple rules.

var simpleRules = []struct {
    name string
    fn   func(string, []string, int) []rule.LintError
}{
    {"final-blank-line", rule.CheckFinalBlankLine},
}

The other takes a shared preprocessing result, *preprocess.Context, which we'll get to. Most rules live on this side now.

var contextRules = []struct {
    name string
    fn   func(string, *preprocess.Context, int) []rule.LintError
}{
    {"no-bare-urls", rule.CheckNoBareURLs},
    {"single-h1", rule.CheckSingleH1},
    {"duplicate-heading", rule.CheckDuplicateHeadings},
    {"unclosed-code-block", rule.CheckUnclosedCodeBlocks},
    {"empty-alt-text", rule.CheckEmptyAltText},
    {"no-empty-links", rule.CheckNoEmptyLinks},
    // …and more
}

Running the rules is then just iterating that table.

for _, r := range contextRules {
    if l.config.IsEnabled(r.name) {
        errs = append(errs, l.withSeverity(r.fn(path, ctx, offset), r.name)...)
    }
}

Only the rules that take options — like heading-level's minLevel or the style of consistent-*, and max-line-length — are kept out of the table and listed below with explicit ifs, because they take extra arguments. The trade is: "the option-free majority goes in a table, the option-taking minority is spelled out."

There's a cost to not using an interface. You can't do dynamic extension like dropping in a rule as a plugin at runtime. But the payoff was bigger.

Rules become stateless pure functions. Give input, get output — so tests fall straight out as table-driven.
Which rules run is fully visible by reading one slice. Nothing hides behind an abstraction.
Registering a new rule is, literally, one line in a slice.

Write "skip inside code blocks" once

Of those traps, the one that hurts most quietly is "the logic to skip inside code blocks scattered across every rule."

The heading rule, the link rule, the emphasis rule — all of them want to say "but not inside a fenced code block." Done naively, each rule starts counting "am I inside a fence right now?" on its own. And then one rule misses indented code blocks, another misses HTML comments — inconsistencies creep in. This actually happened in gomarklint, and we took inventory of it in issue #337.

The answer is preprocess.Scan. It walks the file's lines exactly once, classifies each line into "fenced code / indented code / HTML block / HTML comment," and hands it to every rule as a *preprocess.Context.

ctx := preprocess.Scan(lines)
allErrors := l.collectLineErrors(path, lines, ctx, offset)

The rule side no longer counts anything itself. It just asks the shared verdict.

if inBlockContext(ctx, i) {
    continue // inside code/HTML, so this rule ignores it
}

The clever bit here is that Context deliberately does not offer a single "is this line skippable?" convenience method. It exposes the four contexts individually. There's a reason: max-line-length and no-hard-tabs, for instance, do want to look inside fenced code (line length and tabs should be flagged even there). So "what to skip" is the rule's choice. The inBlockContext helper for the majority is just a thin OR over those four.

func inBlockContext(ctx *preprocess.Context, i int) bool {
    return ctx.InFencedCode(i) || ctx.InIndentedCode(i) ||
        ctx.InHTMLBlock(i) || ctx.InHTMLComment(i)
}

Context is trimmed hard for speed. It borrows the input line slice rather than copying it. The context flags are packed one byte per line, and the "sanitized line" with inline code blanked out is stored in a map only for the lines that differ. Since Scan runs over every file, that one byte matters. But that's a story about designing for speed, so I'll leave the deep dive to the benchmarking post. Here the design win — "write the skip logic once and share it with everyone" — is the real point.

Severity and exit code are wired on the engine side

I said the rule functions fill in neither Rule nor Severity. What fills them is this withSeverity.

func (l *Linter) withSeverity(errs []rule.LintError, ruleName string) []rule.LintError {
    sev := l.config.RuleSeverity(ruleName)
    for i := range errs {
        errs[i].Rule = ruleName
        errs[i].Severity = sev
    }
    return errs
}

The rule body returns only the "facts," and whether it's an error or a warning is decided by the config. The same rule can be an error for one team and a warning for another. Finally, the number of errors is counted, and if there's even one, the exit is non-zero. The basis for the CI gate is consolidated into this one place.

Prevent config/code drift with a contract

The flexibility of .gomarklint.json comes from RuleConfig.UnmarshalJSON accepting three shorthands.

{
  "rules": {
    "no-empty-links": true,                        // enabled / error
    "no-trailing-punctuation": "warning",          // enabled / warning
    "max-line-length": { "enabled": true, "lineLength": 120 }  // full spec
  }
}

true means "enabled / error," "warning" means "enabled / warning," and an object is a full spec. The writer's convenience (the human) is absorbed by one method on the parser side.

Here's the two-sources-of-truth trap. There's a default definition in the Go code too, config.Default(), and the JSON string that gomarklint init emits (DefaultConfigJSON) carries the same list. These two are kept in sync by a manual contract — and the code says so in a comment. It is not auto-generated. When you add a rule, you update both; that's the promise.

Actually adding one: no-empty-links (MD042)

With the tooling in place, let's trace it through a real PR. It's no-empty-links (MD042 in markdownlint terms, empty-link detection), added in issue #106. It catches links whose destination is empty, like [text]().

1. Write one rule function

In internal/rule/no_empty_links.go, write one function with the contextRules signature. The parts we've built so far just work.

func CheckNoEmptyLinks(filename string, ctx *preprocess.Context, offset int) []LintError {
    var errs []LintError

    for i := 0; i < ctx.Len(); i++ {
        if inBlockContext(ctx, i) {
            continue // ignore links inside code/HTML
        }

        // Line with inline code blanked out, so a link-looking
        // `[x]()` inside a code span isn't a false positive.
        line := ctx.Sanitized(i)
        if !strings.Contains(line, "](") {
            continue
        }

        for _, match := range findEmptyLinks(line) {
            errs = append(errs, LintError{
                File:    filename,
                Line:    offset + i + 1,
                Message: fmt.Sprintf("no-empty-links: link has empty destination: %s", match),
            })
        }
    }
    return errs
}

All that's written here is the judgment of "what counts as an empty link" (findEmptyLinks catches [](), [](#), [](<>), and the image variants). "Skip code blocks," "blank out inline code," "adjust the line number by offset" — those all just call the shared parts that already exist. Rule and Severity aren't filled in — the engine adds them later.

2. Add one line to the table

var contextRules = []struct { /* … */ }{
    // …
    {"no-empty-links", rule.CheckNoEmptyLinks}, // ← just this
}

Dispatch is done. No switch, no wiring to add.

3. Register in config (two spots)

Add no-empty-links to both config.Default() and DefaultConfigJSON. This is the "keep in sync by hand" contract in practice.

4. Add tests

Write internal/rule/no_empty_links_test.go table-driven, like the other rules. Because the rule is a pure function, you just line up input lines and expected violations.

Of the four steps, the only one where you really "think" is step 1. The rest is one line, two spots, and a boilerplate test. Even at 24 rules, this shape hasn't broken.

Extensibility isn't adding layers

Many linters get slower and harder to extend as they grow, because every rule leans on a shared, complex framework. gomarklint went the other way. Each rule is an independent, stateless function, and only the thing everyone needs — the code/HTML context check — was pulled out into a single preprocessing pass. Simplicity scales better than abstraction. Even now, with the engine carrying 24 rules, that still holds.

The result is:

Adding a rule is one function + one line in a table + two config spots,
Which rules run is visible by reading a slice,
It scales linearly with repository size.

Extensibility is often assumed to mean stacking layers — but sometimes it's about not stacking them. If you ever want to enforce your own documentation style guide, you can copy this shape directly. Write one function, add one line to the table. That's it.

For the background on parsing and the overall architecture, my earlier post Inside gomarklint: Architecture, Rule Engine, and How to Extend It is also worth a look (the engine has since evolved into the shape described here).

shinagawa-web / gomarklint

Catch broken links before your readers do.

gomarklint

English | 日本語

Catch broken links before your readers do — and keep your Markdown clean while you're at it. 100,000+ lines in ~170ms, single binary, no Node.js required.

Quick install (macOS / Linux):

curl -fsSL https://raw.githubusercontent.com/shinagawa-web/gomarklint/main/install.sh | sh

Download binary (no Go required):

Download the latest binary for your platform from GitHub Releases.

# macOS / Linux
tar -xzf gomarklint_Darwin_x86_64.tar.gz
sudo mv gomarklint /usr/local/bin/
# or install to user-local directory (no sudo required)
mkdir -p ~/.local/bin && mv gomarklint ~/.local/bin/

# Windows (PowerShell)
Expand-Archive -Path gomarklint_Windows_x86_64.zip -DestinationPath "$env:LOCALAPPDATA\Programs\gomarklint"
# Add to PATH (run once)
[Environment]::SetEnvironmentVariable("PATH", $env:PATH + ";$env:LOCALAPPDATA\Programs\gomarklint", "User")

Via Homebrew:

brew install shinagawa-web/tap/gomarklint

Via npm:

npm install -g @shinagawa-web/gomarklint

Via go install:

go install github.com/shinagawa-web/gomarklint/v3@latest

Catch broken…

View on GitHub

Tired of re-recording my README demo GIF by hand, I turned the whole demo environment into code

Kazu — Tue, 07 Jul 2026 13:01:00 +0000

The demo GIF at the top of an OSS README. That little clip that shows you what a tool does in five seconds, before a single line of prose does. That one.

I used to re-record mine by hand. Every single time.

Fire up the screen recorder, tidy up the terminal, type the command, stop at a good moment, export to GIF. Three minutes if it goes well. It usually doesn't. I fat-finger one character in the command. I hit that final q a beat too early and the recording ends before the screen I wanted to show even appears. Again. This time I notice, after exporting, that the terminal font size is different from last time. Again.

That's how "just one GIF" would casually swallow 30 minutes.

Hand-recording produces a slightly different picture every time

The biggest problem with recording by hand is that the picture changes every time you redo it.

Font, size, color theme, shell prompt, window width. All of it depends on the state of the machine you recorded on, at that moment. My prompt is set up to show the current git branch, and once, after a re-record, the README GIF alone had some unrelated repo name baked into it.

The quieter problem is staleness. You tweak the UI a little. You add a line to the header. You change a keybinding. Every time, the GIF gets a bit more out of date. But re-recording is a pain, so it stays stale. The very top of the README goes on proudly showing a screen that no longer exists in the current version. The first thing a visitor sees is the thing that gets updated least.

Anyone who's shipped a CLI tool as OSS has probably felt this one.

A TUI demo is a whole other level of annoying

pgincident, a tool I maintain, is a TUI for the first response to a Postgres incident. It's the full-screen-in-the-terminal kind.

A TUI demo is hard in a way a single CLI command isn't. If there's no "state" on the screen, the demo means nothing.

For an ls GIF, you just run it in a directory with a few files and it looks fine. But what pgincident wants to show you is a screen of "what's happening in this Postgres right now." A long query is running. A lock wait queue is forming. There's a session that opened a transaction and walked away. Only when all of that is on the screen at once does the viewer go "ah, so that's what this tool is."

Connect to a clean, quiet Postgres where nothing is happening, and all you record is a screen of empty rows that says nothing. That's not a demo.

But manufacturing a real long query, a real lock, and a real idle-in-transaction all at once, conveniently, just for the demo — doing that by hand every time isn't realistic. To create a lock artificially you open two sessions, deliberately make one wait, keep that state alive while you start recording in a third terminal, and... just the choreography is a headache. It's a pain to do once, and the thought of reproducing it every time I change the UI means the GIF stays stale even longer.

The `.tape` file — writing terminal operations down

This is where vhs comes in. It's a Charm tool, and in one line: you write a script of terminal operations, and it runs them exactly as written and exports a GIF or MP4.

The script goes in a text file with a .tape extension. Type types characters, Enter presses enter, Sleep waits. Set pins down the appearance. This isn't a full reference article, so it's faster to just look at the real thing. Here's pgincident's demo.tape.

# pgincident demo
# Run via: ./scripts/record-demo.sh

Output docs/demo.gif

Set FontSize 14
Set Width 1200
Set Height 700
Set Theme "Dracula"
Set Shell "bash"

# Start pgincident with the dev config
Type "./pgincident --config dev/pgincident-dev.toml"
Sleep 500ms
Enter

# Overview screen — let it render and refresh once
Sleep 4s

# Navigate to Dashboard screen
Type "o"
Sleep 4s

# Move cursor down to a Long-running query row and open SQL detail
Type "j"
Sleep 300ms
Enter
Sleep 3s

# Close detail and quit
Type "q"
Sleep 500ms

You can pretty much read what it does. Font size, window width, height, color theme — all pinned explicitly by the Set lines up top. None of my machine's settings leak in. Type "o" moves to the Dashboard screen, Type "j" moves the cursor down a row, Enter opens the SQL detail — the gaze of a README reader, turned straight into a script.

Even this alone almost entirely kills the "slightly different picture every time" problem. Same script, same picture. No re-shoots for typos. If I mistype something, I fix one character in the .tape and run it again.

But `.tape` alone is only half of it

So far the operations have become code. But remember the TUI difficulty from earlier. The "state" that shows up on screen still isn't set up anywhere.

A script that launches ./pgincident is useless if the Postgres it connects to is empty — all you record is an empty screen. The script only knows how to operate. It doesn't know what should be on the screen.

So I wrapped the .tape in one more script from the outside. That's scripts/record-demo.sh.

#!/usr/bin/env bash
set -euo pipefail

cd "$(dirname "$0")/.."

echo "==> Building pgincident..."
go build -o pgincident ./cmd/pgincident

echo "==> Starting Postgres..."
docker compose up -d postgres

echo "==> Waiting for Postgres to be ready..."
until docker compose exec -T postgres pg_isready -U postgres -q; do
  sleep 1
done

echo "==> Starting load generator..."
docker compose exec -T postgres psql -U postgres < dev/loadgen_setup.sql
go run ./cmd/pgincident-loadgen &
LOADGEN_PID=$!
trap 'kill $LOADGEN_PID 2>/dev/null || true' EXIT

echo "==> Waiting 5s for load to build up..."
sleep 5

echo "==> Recording demo..."
vhs demo.tape

echo "==> Done! -> docs/demo.gif"

Reading top to bottom, here's what it does. First it go builds pgincident. It starts Postgres with Docker. It waits until pg_isready says the connection is up. It pipes in the load-generation SQL and runs the load generator in the background. It waits 5 seconds for the load to build up. Only then does it call vhs demo.tape. When the recording finishes, the trap kills the load generator and cleans up.

The .tape records the operations, and this script sets up the state that shows on screen. Build, Postgres startup, load generation, recording, cleanup — all folded into a single script, so the whole demo environment is reproducible. This is the part I most wanted to say. Unless you widen "record the demo" into "assemble the demo environment and record it," you can't fully turn a stateful TUI demo into code.

You type ./scripts/record-demo.sh. After that, with a real long query, a real lock, and a real idle-in-transaction lined up on screen, one GIF gets generated. The manual choreography that used to give me a headache became a single command.

What actually creates that "state"?

Inside record-demo.sh, the line that looks the most unremarkable and does the most work is go run ./cmd/pgincident-loadgen. That's the part that artificially stokes the "real incident" you see on screen.

The prep is dev/loadgen_setup.sql. It's an unremarkable script that just creates two small tables for the load to hit.

CREATE TABLE IF NOT EXISTS loadgen_accounts (
    id         bigint      PRIMARY KEY,
    balance    numeric(12,2) NOT NULL DEFAULT 0,
    touched_at timestamptz   NOT NULL DEFAULT now()
);

-- 10k rows gives cache hit ratio room to move
INSERT INTO loadgen_accounts (id, balance)
SELECT i, (random() * 10000)::numeric(12,2)
FROM   generate_series(1, 10000) AS g(i)
ON CONFLICT DO NOTHING;

-- pgincident_dev only has pg_monitor privileges. Grant DML so the simulator can run
GRANT SELECT, UPDATE ON loadgen_accounts TO pgincident_dev;

Here's where it gets interesting. The load generator itself (Go) deliberately produces, one by one, the three categories that pgincident's Dashboard shows.

Long-running queries come from analytical queries with a pg_sleep baked in. It runs short ones that finish in a few seconds and long ones that squat for 5–8 minutes, with staggered start times.

WITH paused AS (SELECT pg_sleep($1))
SELECT a.id, a.balance, b.id, b.balance
FROM   paused, loadgen_accounts a
JOIN   loadgen_accounts b ON b.id != a.id
WHERE  a.id = $2 AND b.id = $3

Lock waits come from the classic wait queue: one session grabs a row with FOR UPDATE and holds it, while another goes for the same row and gets stuck waiting.
Idle in transaction comes from a BEGIN, one query, and then just sitting there without committing or rolling back. pgincident's threshold is 30 seconds, so it sleeps long enough to cross it.
On top of that, a light OLTP load (a mix of SELECT / UPDATE) runs in the background so that TPS and cache hit ratio look "alive" on screen.

And here's the biggest trick for the sake of the recording: each worker is written to start on a staggered schedule. The idle ones so that "at least one is always visibly past the 30-second threshold," the long queries so that "two cycles half-overlap." The point is that the load itself is designed so that whenever vhs demo.tape runs, the Dashboard's three categories are all filled in at once, just right.

"Turning the whole demo environment into code," taken to its conclusion, includes this. Writing operations in a .tape will never produce this "broken just right" state.

The README just embeds this GIF as the top image.

![pgincident demo](docs/demo.gif)

What changed after turning it into code

Honestly, I didn't invent anything dramatic. Neither vhs nor a shell script is a novel technology. Even so, the day-to-day feel clearly changed.

Anyone who runs it gets the same look and the same composition. Font, size, theme, the way load is applied — it's all in the script. It won't match down to which query lands in the top row on any given run, but the look and composition of the picture don't depend on my machine's mood. When a future contributor wants to update the GIF, they can read the .tape and know what's on screen.

It reviews in a diff and lives in git. A GIF is binary, so you can't read the diff of its contents, but .tape and record-demo.sh are just text. "Bumped a Sleep from 3s to 4s," "reordered the operations" — that shows up in the PR diff. Demo changes ride on the same rails as code changes.

Change the UI, regenerate. Added a line to the header. Changed a keybinding. Before, it was "re-recording is a pain, so leave it." Now it's just typing ./scripts/record-demo.sh again. I won't claim staleness is gone — forget to run the regen and it stays stale. But the weight of the "it's a pain" excuse dropped from 30 minutes of manual work to a single command.

Let me be honest about one thing too. Right now record-demo.sh is a local script I run by hand. I'm not auto-regenerating the GIF in CI. When I change the UI, I still have to notice "ah, gotta regenerate the GIF" myself and type it myself. Ideally there's room to get to where any UI-touching change makes CI re-bake the GIF and commit it. That's the next move. What I've got so far is that regeneration is one command.

Gotchas and tips

A few spots that actually took some doing.

Tuning Sleep takes the most work. A TUI refreshes the screen periodically. Right after launch, the data isn't drawn yet. If Sleep is too short, you record an empty or half-drawn screen. The reason demo.tape waits a generous Sleep 4s after launch is to let the Overview screen refresh once and fill in. Too long, though, and the GIF drags. I went back and forth a few times here — "shorten it, record, didn't make it, lengthen it a bit." It's an adjustment a single CLI command never needs; it's a TUI-specific chore.

How well the load is crafted decides how convincing the demo is. If the load the generator throws is too weak, not a single long query crosses the threshold and the screen stays sparse. Too strong and the screen fills up so much you can't tell what to look at. Creating the "broken just right" state is unglamorous but the single biggest thing for demo-ness.

Don't mix in secrets. The script is text and goes into git. And of course the GIF bakes in exactly the characters you typed and the screen you showed. Be careful not to write a production DSN or password into a Type, or to record a screen connected to production. That pgincident's demo uses a dev config (dev/pgincident-dev.toml) and connects to a local Docker Postgres is partly for this reason.

GIF size. Crank up Set Width / Set Height and the file naturally gets heavier. This image sits at the top of the README, so slow loading defeats the purpose. Width 1200 × height 700 was the sweet spot where the text isn't crushed and the file isn't too heavy.

Demos go from something you "record" to something you "write and generate"

Looking back, what changed is the kind of work. From the manual labor of "recording" a demo — moving your hands, failing, re-shooting — to "writing and generating" one: turning the script and the environment into code, and baking it with a command.

And for a stateful TUI demo, writing the operations in a .tape isn't enough. Only when you go all the way to setting up the "state" on screen — a real long query, a real lock — and turn the whole demo environment into code does it become reproducible. That, I think, is the key thing when you turn a TUI demo into code.

If you're re-recording your own OSS README demo GIF by hand every time, start with a single .tape. And if the screen you want to show needs "state," try folding the setup of that state into the script too.

By the way, what pgincident actually is as a tool, I wrote up in another article. It's the story of that first 30 seconds when Postgres slows down at 2 a.m.

"Couldn't you just use pg_activity? — staking out one corner of Postgres incident triage"

Kazu — Wed, 01 Jul 2026 13:01:00 +0000

Earlier, I wrote about a 2 a.m. incident. Production is slow, you open psql, the cursor blinks, and your hand freezes on the very first query. I introduced pgincident as a tool that breaks that "first-response paralysis" by giving you the whole picture on one screen.

When I show that article to people, there's almost always one thing that comes back first.

"Couldn't you just use pg_activity for that?"

Or "isn't pgcenter enough?" Fair question. Terminal-based Postgres monitors have been around forever. Anyone can picture "that kind of screen" the moment you say top, and I've personally been bailed out by both of those tools in the field more times than I can count. If I'm going to add one more TUI to the pile, I owe people a real explanation of what makes it different from the established standards.

This article is that explanation. It's not a case that pgincident beats everything. After taking honest stock of where the existing tools are strong and where they have a harder time reaching, where pgincident stands turns out to be a fairly narrow corner: managed Postgres × the Mac in front of you × first-response triage. That's where this lands. Read it as a stock-take, not a sales pitch.

First, let me give credit — the existing tools are good

I don't want to open by talking anything down, so let me say it up front: pg_activity and pgcenter are both well-made tools.

(One caveat before we start: everything I say in this article about how the tools behave is as of pg_activity v3.6.2 / pgcenter v0.10.1 / pgincident v0.6.0. OS support, required privileges, and managed-environment behavior can all change between versions, so read every "works / doesn't work / gets stripped" here as a statement about those three specific versions.)

pg_activity is a top-style activity viewer written in Python. It's been around a long time, it's mature, and if you're SSH'd into a production host it's still first-rate. Instead of re-typing pg_stat_activity by hand, the running queries line up sorted by duration and refresh on their own. Part of what I said I "wanted" in the last article has been sitting right here all along.

pgcenter is written in Go and is also of the top lineage. What's genuinely impressive about pgcenter is that it can correlate system stats — CPU, I/O — with Postgres stats on one screen. "Is the DB slow because of the queries, or because the host's I/O is saturated?" You can make that call by looking at the OS and the DB side by side on a single screen. If you want to do this on a self-managed Linux box, pgcenter is still a top pick.

This isn't me propping it up in the abstract. There's a concrete time it saved me. On a self-managed Linux production box (Postgres I'd stood up myself on EC2), API latency one day ballooned to about three times its usual. Staring at the slow query log, I couldn't single out one bad query — everything was uniformly slow. That kind of slowness is the nasty kind, because you can't pin the blame on a single query. I SSH'd into the production host and launched pgcenter as the postgres user. On that familiar top-lineage screen, the first thing that jumped out was the I/O wait row, with disk utilization basically pegged. The bottom half of the same screen had the Postgres-side stats, and I could see checkpoints firing far more often than usual. OS-side I/O saturation and DB-side checkpoint flurry, stacked one above the other, visible at the same time. "This isn't the queries — the host's I/O is jammed, and the trigger is checkpoints." I had that hunch within about ten seconds of opening the screen. If I'd been cross-referencing pg_stat_bgwriter and iostat in separate terminals, it would have taken a lot longer.

That's what correlating OS and DB on one screen buys you. It's pgcenter's strongest suit, and where it works, it really works.

So this isn't a piece about killing the old thing to sell the new one. The question is where "where it works" actually is.

What the descendants of `top` assume

pg_activity and pgcenter are similar at the root of their design. Both are descendants of top, and the basic shape is "show what's running right now as a single sorted list." And both deliver their best experience under one assumption.

That assumption, roughly put, is that you're sitting on the host where the DB is running.

Concretely, it comes down to this. pgcenter's signature feature — correlation with system stats — pulls CPU and I/O information from the OS. So the tool has to run on the same host as the DB. pg_activity, too, if you want it to show system information and temp-file information, assumes you run it on the same host as the OS user running postgres (or root).

A word on privileges — this one's easy to get wrong, so let me be precise. Both tools deliver their best experience when the connecting role is a SUPERUSER. But it's not that they refuse to run without it. pg_activity drops into degraded mode if you're not a superuser and stops showing system and temp-file information — but you can still see the activity list itself. pgcenter is also happiest with a superuser, but it'll run with privileges sufficient to read the stats. It's not "SUPERUSER required," it's "without it, some of it gets stripped." That's a distinction worth keeping.

Either way, the precondition for the best experience is "on the DB host, on Linux, with sufficient privileges." As long as that holds, both tools are strong.

Where the assumption falls apart — RDS, Cloud SQL, Aurora

The problem is that in a lot of recent incidents, that assumption falls apart wholesale.

When your production Postgres runs on a managed service (RDS, Cloud SQL, Aurora), that precondition breaks in three ways at once. (1) you can't become the OS user, (2) you can't get SUPERUSER, (3) you can't get system stats. Let's take them in order.

First, (1), you can't become the OS user of the DB host. You don't get to touch the internals of a managed service. The whole premise of SSH'ing in and launching the tool as the postgres user simply doesn't hold.

Next, (2), you can't get SUPERUSER. Managed services generally won't hand you a SUPERUSER-equivalent role. So you're structurally locked out of that "SUPERUSER gives you the best experience" case from earlier.

And (3), this is the one that stings most. The correlation with system stats — pgcenter's killer feature — simply doesn't work on managed services, because the mechanism for pulling CPU and I/O from the host can't reach inside the walls of a managed DB. pg_activity has an --rds flag, but with it, system stats don't show up.

Here, though, the damage lands differently on the two tools, so let me write it out fairly. pgcenter has its single biggest selling point — correlating OS and DB — vanish. The top half of that side-by-side screen, the one that let me catch the I/O saturation on EC2 in one shot, stops meaning anything at all. pg_activity, by contrast, loses the add-on information (system stats, temp-file info), but its core as an activity viewer survives. The main act — watching running queries sorted by duration, auto-refreshing — works just fine on a managed service too. pgcenter loses a pillar; pg_activity loses a branch. Both "get weaker on managed services," but that difference is not small.

In fact, I once got burned by exactly this. During a morning incident with a slow Aurora (PostgreSQL-compatible), I typed out pgcenter on my Mac out of habit. It wouldn't even start — pgcenter is Linux-only and the Mac is unsupported (this isn't degradation, it just plain doesn't run). So I pointed pg_activity at the Aurora instance with the --rds flag, and this time it started up fine. The activity list showed up too. But the system-information columns I always leaned on were completely blank. CPU, I/O, temp files, all empty. Which makes sense when you think about it — you can't reach the OS inside the managed wall — so this is what degraded looks like under --rds. Not "doesn't run," but "comes back as half of itself."

That swing-and-a-miss — it starts, but the half you most wanted to see is blank — wore on me more than I'd have expected. On EC2 I'd had a hunch within ten seconds of opening the screen, and now the same tool just wouldn't do the same work. In the end, that morning I couldn't use either of my usual standards in its proper form, and I had to form a hunch off the activity list plus guesswork in my head. The first thing you open during a late-night or early-morning incident is usually not the production host but the terminal on your own Mac. And in that "first thing you open," both tools fall short of their best at the same time.

For the record, this isn't a defect in either tool. It's just that the world their design assumed (self-managed Linux hosts) and the world incidents now happen in (managed × the machine in front of you) have drifted apart. The world moved over these past few years; the tools didn't degrade.

A difference in trade-offs — deliberately dropping system stats

Here's the design call pgincident makes. To be fair, this isn't "pgincident has more features." If anything, it's throwing features away.

pgincident deliberately does not carry system stats (CPU / I/O / memory). The README even marks this as an explicit non-goal: "that's pgcenter's job; this one focuses on Postgres internals."

This might look like a weakness. And it is — for someone who wants to correlate OS and DB on self-managed Linux, pgcenter is clearly the better choice. I'll grant that.

But as a design decision it's coherent. On managed services you can't get system stats anyway. A feature built on top of something you can't get is dead on arrival if managed is your main case. So instead of depending on it, you pour everything into the range you can get with the pg_monitor role: activity, locks, idle in transaction, and the big-picture overview of overall health. Unlike SUPERUSER, pg_monitor is a role that can be granted on managed services, so you could say pgincident shifts the starting point of its design so that the best case "still holds on a managed service."

Why can pg_monitor be granted on managed services while SUPERUSER is withheld? This is worth one paragraph of context. SUPERUSER bypasses permission checks wholesale. It's a sweeping privilege that reaches the filesystem, server configuration, and other people's data. Hand that to a user and AWS or GCP can no longer hold the management boundary between the OS and the infrastructure. So managed services structurally don't issue SUPERUSER. pg_monitor, on the other hand, is a narrowly scoped role that bundles up only the reads you need for monitoring — all columns of pg_stat_activity, the various stats views. Because what it can see is limited to monitoring stats, the managed side can grant it without worry. By planting its starting point here, pgincident stands from the outset on the side of "the privilege a managed service can comfortably issue," rather than "the privilege a managed service structurally can't."

One honest note: pgincident's managed-service support itself is, on the repo's SQL catalog, still largely unverified (most of the testing so far has been on local PG16). So I won't write "battle-tested on RDS." What I'm pointing at here isn't a verification track record but a choice of privilege model and architecture — the design's starting point sits on managed services and pg_monitor. Same kind of TUI, but the best case it assumes faces a different direction.

The other difference is the "shape" — one list vs. overview-then-dig

Apart from the privilege and platform story, the shape of the screen itself is different. And personally I think this is the more important connection back to the last article.

The basis of the top family is "show what's running right now as a single sorted list." Sort by duration, sort by resource. This is good for seeing "what's heaviest." But it doesn't directly help with that paralysis I wrote about last time, the one where your hand stops at "what kind of incident is this even?" A list gives you a "heaviest-things ranking," but it doesn't give you the one-frame picture of "what's the overall state right now." Are connections exhausted? Are locks chaining? Is an idle in transaction session holding on? You need that hunch before the list.

pgincident reverses the order. The first thing you get is the Overview — a one-frame picture of overall health. Connection count, TPS, cache hit ratio, checkpoints, autovacuum. Each has a threshold, and the shaky metrics get highlighted in color. Here you get a read on it first — "this is the connection-exhaustion line," "no, this is a lock" — and only then drill down into that direction's category (activity / locks / idle in transaction). Overview first, then dig.

Last time, I wrote this: "Because I was digging without an overview, I dug holes on instinct and filled them back in, and 30 minutes went by." A single top-family list is an excellent tool for the "digging" side. What pgincident is trying to add is the one "overview" frame that comes before it, and that's the direct answer to the first-response paralysis I described in the first article. The comparison is only a means; the theme has been "the quality of the first response" the whole time.

Just one comparison table

I'll pick the axes carefully and lay down a single table. Think of it not as a binary win/lose table but as one for reading "under which assumption does which tool work." (Restated: the behavior here is as of pg_activity v3.6.2 / pgcenter v0.10.1 / pgincident v0.6.0.)

	pg_activity	pgcenter	pgincident
Language	Python	Go	Go
OS	Runs on Mac (system stats need a Linux host)	Linux-only (won't start on Mac)	Linux + macOS
Privileges	Best with SUPERUSER / degrades without it	Ideal with SUPERUSER / works with stats-read privileges	Designed around `pg_monitor`
System-stats correlation	(needs same host + OS privileges)	best-in-class (needs same host, Linux)	deliberately absent (non-goal)
Managed DB as the main arena	secondary	secondary (system stats vanish)	the primary design assumption
Maturity / adoption	high, proven track record	high	new
Screen shape	one sorted list	OS+DB correlated on one screen	overview → drill into a category

The bold cells are where each tool clearly wins. For system-stats correlation, pgcenter; for maturity and track record, pg_activity; for putting managed services at the center from the start, pgincident. Read the table this way and you can see that what pgincident fills is the bottom-right corner — only the case of "managed is the main arena, pg_monitor is the starting point, and I want to come in from an overview."

A quick word on tools in other categories

Let me also address the "but we already run Datadog for monitoring" question. This is less a comparison than a sorting-out of layers.

Metrics platforms (Datadog, Grafana + postgres_exporter, pganalyze) are for continuously accumulating metrics and watching trends and anomalies over time. Catching "it's been gradually getting worse since last week" is this layer's job, and that's where it earns its keep. But that's the "look back later / keep a standing watch" layer, and it lives on a different time axis from "the first 30 seconds right after the alert fires, what's happening in the terminal in front of me right now." I think of it as a different layer, not a competitor.

GUIs (pgAdmin, DBeaver) are tools for browsing schemas and writing queries. They're not what you open in that moment when the cursor is blinking at the start of an incident. Different category too.

Conclusion — "the best tool" depends on the situation

The conclusion of an honest stock-take comes out like this.

If you're on self-managed Linux and want to look at OS and DB together to sort out "is it the host or the DB," then pgcenter. Its system-stats correlation is still first-rate.
If you're SSH'd into the production host and want a mature, proven standard, then pg_activity. It's ahead on maturity and adoption.
If you want long-term trends and standing monitoring, then Datadog / Grafana / pganalyze. That's a different layer.
And if you want to get oriented right after the alert fires, on managed Postgres, from the Mac in front of you, then the tool trying to fill that corner is pgincident.

This is a story about "the design's starting point sits on managed × pg_monitor," not about "a track record of field verification piling up on RDS." Verification has mostly progressed on local PG16 so far, and that honestly remains a weak point. Even so, as far as I know there's no other first-response TUI whose assumed best case faces managed services from the start. What I'm asserting is the design's direction, not a track record.

pgincident isn't a tool I built to beat everything. By dropping system stats, narrowing the feature set, and shifting its assumed best case to "managed × pg_monitor," it's trying to take on just the one corner that the existing tools have a structurally harder time reaching.

My answer to "couldn't you just use pg_activity?" is "in many cases, yes, exactly." If you can put it on the production host, on Linux, and want to see all the way down to system stats, the existing tools are the better call. But if it's 2 a.m. and you want to get into RDS from the Mac in front of you and form a hunch about which category of incident this is within 30 seconds — for that one narrow case, and that case alone, pgincident exists.

So the conclusion of this stock-take isn't "use this," it's "choose by the situation." If you want to SSH into EC2 and look at the whole host, pgcenter; for a mature standard living on the production host, pg_activity; for tracking trends, Datadog; and for managed × the machine in front of you × first response, pgincident. Tools are stuck to their situations, and no single one covers all of them. Pick based on the situation you most often find yourself in. For me, that just happened to be "RDS from the Mac, late at night."

pgincident is on GitHub.

The first 30 seconds of a Postgres incident: why they take 30 minutes

Kazu — Tue, 23 Jun 2026 13:02:00 +0000

2 a.m. PagerDuty goes off. "Production is slow."

You open your laptop, fire up psql with bleary eyes, and connect to production. The prompt comes up. The cursor blinks after production=>.

And your hands stop.

Now, where was I supposed to look first? Do I get an overview of the whole DB, or do I start drilling into individual queries? It's been a while since the last incident, and the first move doesn't come to me.

"What kind of incident is this even?" — and you freeze

"Production is slow." If you could act on that much information, this would be easy. Is the slowness in the code or the DB? Is one runaway slow query thrashing, or is the overall load up? Are connections exhausted? Is a chain of lock waits piling up somewhere? Is an idle in transaction session holding a transaction open?

These are all different incidents. Different places to look, different moves to make. Do you check pg_stat_activity? pg_locks? Cache hit ratio in pg_stat_database? Before you can decide which pg_stat_* to hit, you need a hunch about which category of incident this is.

But you don't have anything yet to form that hunch.

What you need, really, is the big picture. The goal is to narrow it down. "Ah, this is connection exhaustion." "No, this is a lock." Only once you've placed the incident into a category does the first query become obvious. And for that, you first want to see — on one screen — what the entire database looks like at this exact moment. How are connections doing? Is TPS spiking or dropping? Is anything waiting? You don't want the overview for its own sake. It's the footing you need before you can make the call.

But what psql gives you is one query, one snapshot. Fire off a SELECT and you get back a single slice. No big picture. You want the whole picture, and you have to start by guessing the right query to get a view of it.

So you pick the first query on instinct. Usually it's "pg_stat_activity for now." Not wrong. But you're running it without knowing whether it's the right call. With no hunch yet, you're running the very query that was supposed to give you one.

Guess wrong and you pay for it in extra time. Once, I saw the connection count was unusually high, decided "connection exhaustion," suspected the app's connection pool, and nearly started a conversation about adding servers. The actual culprit was a single long query thrown by a nightly batch. It had settled in and clogged everything behind it, and the connections were just stacking up as a result. I'd looked at the result (connection count) and mistaken it for the cause (the long query). If I'd been able to see the whole thing on one screen first, I'd probably have caught it in a couple of minutes.

These few minutes — from the alert firing to the first keystroke — are the most nerve-wracking part.

You don't have the queries memorized

Say you commit to "pg_stat_activity first." The next problem arrives.

You can't write that query, properly filtered, from memory.

SELECT * FROM pg_stat_activity you can write. You can — but run it and 143 rows scroll past, and you can't tell which one is the culprit. What you want is "only the active, long-running queries, ordered by duration descending, only the columns I need." Can you write that SELECT, with the state = 'active' filter and the now() - query_start math, at 2 a.m. with nothing in front of you? I can't.

So you open a browser. You search "postgres long running queries." You open Stack Overflow. You copy a query. You paste it into psql. ERROR: column "waiting" does not exist. An old answer referencing a column that was removed in PG 9.6. You fix it.

Locks are even worse. JOIN pg_locks and pg_stat_activity to produce the pairs of which session is blocking which — how many people have that query memorized? You search again. You paste again.

Then you remember, "we put a query collection in the team wiki," and go looking for it. It does exist. But it was last updated two years ago, and half of it doesn't run as-is. Figuring out which ones still work and which are stale is, again, more searching. In the middle of an incident, you're bouncing between search results and a wiki instead of your own repository.

Is that the cause, or the result?

So you go through all that trouble and finally produce a list of long-running queries. At the top sits a query that's been running for 2 minutes 14 seconds.

Is this the culprit? You don't know.

Is the query itself heavy and slow, or is it being blocked somewhere else and just sitting there long as a result of waiting? One snapshot can't separate the two. You think you've grabbed the cause, when you might be looking at the victim.

To separate them, you also need to look at the lock side. Which means another query. And then you reconcile both in your head. At 2 a.m.

And you blink, and 30 minutes are gone

Queries are snapshots, so if you want to follow how things change, you have to re-run them by hand. Hit pg_stat_activity once. The situation shifts. Hit it again. Recall the history with ↑, hit enter. Run it again.

Written out, the first response looks like this:

Search "postgres running queries"
→ Copy a query from Stack Overflow
→ Fix it for a PG version mismatch
→ Sort by duration
→ Wonder: is this the cause or the result?
→ Search for a separate locks query
→ Paste and run
→ The situation changed, so re-run everything
→ …

It feels like an instant. Your hands never stop. Searching, pasting, fixing, re-running. And yet you glance at the clock and 30 minutes have passed.

For those 30 minutes, the incident channel in Slack keeps growing. When someone asks "what's the status?", all you can say is "still investigating." Your hands haven't stopped. You've been hitting something the whole time. And yet you haven't even taken the first step toward recovery. You don't even have a hunch about the cause.

One tool has a tagline: "the first 30 seconds of a Postgres incident." But with nothing in hand, those "30 seconds" quietly balloon into 30 minutes of manual queries and searching. Where it should have taken 30 seconds, 30 minutes go by. That gap is what hits hardest in a late-night incident.

What I actually wanted was "now, on one screen"

Looking back, what I wanted in the first 30 seconds was always the same thing.

"What does the database look like right now" — on one screen. Are connections filling up? Has cache hit ratio dropped? Are there long-running queries? Is a lock queue forming? Is a session sitting with a transaction open? I want to see this as one picture first, without memorizing queries and without searching.

Then, having placed it — "this is connection exhaustion," "no, this is a lock" — I want to drill down only in that direction. Overview first, then dig. The order was always supposed to be this, and yet with psql I had no choice but to start from the one query for "digging."

Because I was digging without an overview, I dug holes on instinct and filled them back in, and 30 minutes went by.

pgincident — putting that into one TUI

pgincident is a tool that packs that "overview first, then dig" into a single TUI in your terminal. Instead of the many queries you'd hit in psql, you take your guess from an overall-health overview screen, then drop into a category dashboard to dig.

Setup is nothing dramatic. On macOS, you install it with Homebrew.

brew tap shinagawa-web/tap
brew install pgincident

On Linux / macOS, a one-liner works too.

curl -fsSL https://raw.githubusercontent.com/shinagawa-web/pgincident/main/install.sh | sh

Getting started is three steps. First, generate a config file.

pgincident --init
# Created /your/project/.pgincident.toml

Write your connection details into the generated .pgincident.toml.

[connections.default]
dsn = "postgres://user:password@localhost:5432/mydb"

[thresholds]
long_running        = "5s"
idle_in_transaction = "30s"

Then just launch it.

pgincident

The connecting role doesn't have to be a superuser. As long as it's a member of the pg_monitor role, it works. This is a quiet detail, but it matters — on managed Postgres like RDS or Cloud SQL, you often can't get a superuser at all.

What you see in the first 30 seconds

When you launch it, the first thing you get is the Overview screen. That "now, on one screen" is right here.

primary  10.0.1.42:5432  PG 16.1                              interval: 5.0s
──────────────────────────────────────────────────────────────────────────
  DB Health Overview
──────────────────────────────────────────────────────────────────────────

  Metric                Value                 Status
  ──────────────────────────────────────────────────
  Connections           142 / 200 (71%)       OK
  TPS                   2340                  OK
  Cache hit             99.2%                 OK
  Checkpoints           req: 0                OK
  Autovacuum            0 workers             OK

──────────────────────────────────────────────────────────────────────────
[o]dashboard  [q]uit  [+/-]interval  [?]help

Connections, TPS, Cache hit, Checkpoints, Autovacuum. Each has a threshold set, and the badge reads OK when healthy, WARN when shaky, CRIT when bad. If your setup has a replication standby, a Replication lag row joins them.

This is exactly what I wanted at 2 a.m. If Connections is red at 90%, this looks like connection exhaustion. If Cache hit has dropped, another lead opens up. Without memorizing queries, without searching, the hunch about "which category of incident" forms right here. And this screen refreshes itself on the interval you set, so there's no re-running it by hand.

Once you've placed it, press o to drop into the Dashboard screen. Here, the things you'd hit with several queries in psql line up in three categories.

Long-running queries — active queries still running past the threshold (default 5s)
Locks — pairs of the blocking and the blocked sessions
Idle in transaction — sessions left holding a transaction open past the threshold (default 30s)

That "cause or result?" separation you were stuck on moves forward here. The list of long-running queries and the blocking relationships of locks are on the same screen at the same time. If the top long-running query shows up on the blocked side, it's not the culprit — it's the victim. No need to re-run a separate query and reconcile it in your head.

Press Enter on a long-running query row and its full text opens in an overlay. Formatted with line breaks, keywords highlighted.

┌─ Query Detail ──────────────────────────────────────────────────────────┐
│ PID: 12345   user: app_user   duration: 00:02:14.32   state: active     │
│ ─────────────────────────────────────────────────────────────────────── │
│ SELECT                                                                   │
│   u.id, u.name, u.email,                                                │
│   o.id AS order_id, o.status, o.total_amount                            │
│ FROM users u                                                             │
│ JOIN orders o ON o.user_id = u.id                                       │
│ WHERE u.status = 'active'                                               │
│   AND o.created_at > NOW() - INTERVAL '7 days'                         │
│ ORDER BY o.created_at DESC                                              │
│ LIMIT 100                                                               │
└─────────────────────────────────────────────────────────────────────────┘
[any key] close

If you've configured multiple DBs, c switches the connection (the use case being to move between a primary and a replica). You adjust the polling interval with + / -, and Tab moves between sections.

Getting the "first 30 seconds" back

What you actually needed in the first response to an incident wasn't a new query collection, or pg_stat_* syntax to re-learn. It was being able to take in "what's happening right now" as an overview on one screen, place your guess, and then dig — and to walk that order without searching or re-pasting. Because I had no way to get the overview, I kept digging on instinct — holes I'd just fill back in. What pgincident gives back is that first overview.

This article covers only the first 30 seconds, from launch — the core experience. Deeper dives into lock chains, and the investigation further down the line, are for another post.

2 a.m., those few minutes where your hands stopped after production=>. So that they don't turn into 30 minutes, have the one-screen now on hand first — that alone changes the first response quite a bit.

How to safely remove a Rails column: finding every real reference before you delete

Kazu — Tue, 16 Jun 2026 13:03:00 +0000

Every Rails project has at least one of these. A model with an old column that's probably not used anymore. "Probably" is the scary part. If something in production is still referencing it, deleting the column breaks the app. NoMethodError, in production.

You know what that looks like. It's 11:30 PM the night before sprint planning. You're tidying up the Article model — the kind of low-stakes cleanup you save for when nothing urgent is on fire. You spot summary in db/schema.rb. It doesn't appear in any recent ticket. The last commit touching it was fourteen months ago. You look at the column definition: t.text :summary. Probably a description field from some old feature. You open the controller. You don't see it used. You check the views quickly. Nothing obvious.

You convince yourself it's probably safe to delete. You run the migration, deploy, go to sleep.

At 7:15 AM your on-call pager fires. Five hundred errors per minute. NoMethodError: undefined method 'summary' for an instance of Article. Users are getting blank pages. Logs are flooding. Slack has twelve messages: "site down?" "was it that deploy last night?" Your stomach drops. You push a rollback. The errors stop. Now you spend the morning in a postmortem figuring out that an admin reporting feature — a rarely-used export endpoint buried in app/reports/ — was still calling article.summary to build a CSV. Nobody thought to check it. You didn't even know that file existed. The column wasn't unused; it just looked unused.

That's why nobody deletes anything: you can't be sure, so you don't. It's a reasonable call — except there was never a way to get sure.

The obvious thing to try: search for the column name in VS Code. In one Rails project, searching for summary returned 3,847 results. I started going through them and quickly noticed: almost none were the real thing. <summary> tags in ERB templates — the HTML accordion element. Translation keys in locale files. Description strings in RSpec examples. Actual code accessing article.summary: 9 results.

I gave up somewhere around result 50.

Why full-text search isn't enough

VS Code search and grep answer "does this string appear anywhere in this file?" That's useful for a lot of things. But when you want to know "is this column actually referenced in code?", text search picks up way too much. The column name in a string literal, in a comment, as an HTML tag name: it all counts as a hit. Sorting through them is manual work.

Those 3,847 VS Code results were full-text matches. Narrowing with grep -rn "\bsummary\b" --include="*.rb" to Ruby files left 847. Here's what those broke down to:

Type	Count
`<summary>` tags in ERB templates (HTML element)	312
Translation keys in locale files (`summary:`, etc.)	218
Description strings in RSpec `describe` / `it` blocks	157
Comments, variable names, unrelated strings	151
Actual column accesses	9

summary is particularly tricky because HTML5 has a <details>/<summary> accordion element. If your project uses that tag in templates, every view file is a potential hit. To text search, <summary> and article.summary are the same thing: a string match.

Even filtering to Ruby files, any 'summary' string in a serializer field list or a comment still hits. "Ruby files only" and "actual column access" are completely different things.

The more you refine the regex, the more you start wondering whether the regex itself is missing something. You end up needing to verify the verification.

And summary is not even the worst case. Consider status, name, or type — column names that appear in dozens of unrelated contexts throughout a typical Rails project. Variable names, hash keys, RSpec subject descriptions, i18n keys, FactoryBot attributes. Hundreds of hits. Same problem, worse noise.

Try it right now if you're curious. Open a Rails project you've been on for a year. Run:

grep -rn "\bstatus\b" --include="*.rb" ./

Count the results. Most of them have nothing to do with the status column you're thinking about. They're local variables, hash keys in unrelated parts of the app, describe "updates the status" in test files. The signal you need — "is article.status actually accessed somewhere?" — is buried in hundreds of lines of noise.

There's also the psychological cost. You open VS Code, run the search, see 847 results for status, and your shoulders drop. You close the tab. You tell yourself you'll check it later. "Later" never comes. Nobody should have to hand-verify 847 results to answer a yes/no question.

"I searched, couldn't check everything, left it alone." Most Rails developers have been here. You want to delete the column but can't. Checking properly is possible, but it costs more time than the cleanup is worth. So the column stays, and they pile up.

So the columns accumulate. Here's what that does to a codebase.

The schema bloats. You open db/schema.rb and it's 600 lines. You scroll past columns you half-recognize — legacy_body, old_slug, deprecated_export_format, summary — added by developers who've since moved on, tied to tickets that closed eighteen months ago. Nobody knows what they did, so nobody touches them. Every SELECT * drags them along. Every Article.new builds an object with two dozen attributes, most of them nil because they've been unused for a year.

A new developer joins and runs Article.column_names to understand the schema, then stares at the output. "What's legacy_body? What does deprecated_export_format mean?" They ask in Slack. Nobody knows for certain, and the answer is "don't touch those." Reasonable in isolation. But repeat that across five models and it hardens into an unspoken rule: don't touch anything you didn't write.

Design options narrow with it. You want to add a content_format column, but you can't tell whether legacy_body and body are two competing implementations of the same concept or two genuinely different things. So the new feature gets bolted onto the side of the model instead of replacing the old thing cleanly. Every migration feels slightly riskier because the schema is full of things nobody understands, and the unease compounds until nobody touches anything at all. Six months later, another developer hits the same dead end.

This is the same kind of debt as missing tests: it accumulates quietly and you can never point to the moment it started. The real cost is cognitive load. Every unused column is a small tax on everyone who reads the model. Thirty columns, two years of new developers, one noisy on-call incident caused by a column that was supposed to be gone: that's how "we never clean up old columns" becomes a drag that's hard to measure and impossible to ignore. And none of it is a skills problem — the tool to check properly just didn't exist yet.

How colref reads code structure instead of text

What you actually wanted to know was: where is article.summary referenced in Ruby code? colref answers that, ignoring ERB <summary> tags and locale-file summary: keys.

How does it tell the difference? Instead of treating code as a sequence of characters, it reads the code structure.

When you write article.summary, Ruby sees "call the summary method on the article object" — a specific structure. <summary> is written as an HTML tag name — structurally, it's not a method call. :summary is written as a symbol. Reading code structure makes those differences detectable. Only places written as object.column_name get picked up. HTML tags, symbols, and strings that happen to contain summary are ignored.

Text search is Ctrl+F. Reading code structure is closer to a human reading through every line — except it handles thousands of lines in a second.

Here's what that looks like in practice:

Method	Hits (for `summary`)	What it sees
VS Code full-text search	3,847	All string matches
grep `\bsummary\b`	847	Word-boundary matches
colref	9	Actual column accesses only

3,847 or 847 becomes 9. Whether you can act on the results depends entirely on how many there are.

When you get 9 results: open each one. app/controllers/articles_controller.rb:42 means go to that line and check whether article.summary is actually being accessed there. Nine results takes maybe 15 minutes.

A few things you'll encounter while reviewing results: the column appearing in a migration file (colref skips migrations, but if it surfaced it, the migration is just recording the column's history, not actively using it). You might also see test factories or fixtures that assign the column's value. If you delete the column and forget to clean up the factory, your test suite will fail. That's not a reason not to delete — it's just something to handle as part of the deletion.

When you get zero: you have a fact. "Not found in Ruby code" is different from "I think it's probably fine." It's the signal to move on: dynamic access patterns, templates, Strong Parameters, serializers. Treat a zero from colref as the first check, with several more still to run.

The shift is from "check 847 things" to "check 9 things, then a handful of specific files." That's the difference between a task you'll defer indefinitely and one you'll do today.

Installation

Add to your Gemfile:

bundle add colref --group development

Or install globally:

gem install colref

Specify the model name, field name, and your project directory:

colref check --orm rails --model Article --field summary ./

Results come back as filename:line_number. Each one is something you can open directly.

What zero results doesn't cover

Zero results doesn't mean "safe to delete." It means "not found in Ruby code," and the gap between those two is where columns come back to bite you.

Dynamic Access Pattern: colref detects literal-symbol forms like article.send(:summary) and article.read_attribute(:summary) — these appear in results with a [symbol] confidence label, meaning they need manual verification. What colref cannot catch is when the method name is stored in a variable:

# colref cannot catch this — field name is in a variable, not on the same call
[:title, :summary, :body].each do |attr|
  puts article.send(attr)
end

The variable form can't be reliably caught with a single grep either — send.*summary only matches lines where send and summary appear together, which misses the loop above entirely. To find this pattern, read every send, public_send, and read_attribute call site directly. As a starting point:

grep -rn "send.*summary\|read_attribute.*summary" --include="*.rb" ./

This catches the literal form (article.send(:summary)) that colref already surfaces, and it's a useful double-check. But don't treat zero results as proof there's no variable-form usage — scan the call sites by eye for the loop pattern.

Symbol Permit Pattern: The column name appears as a symbol in permit inside controllers, which colref does not detect:

# controller
def article_params
  params.require(:article).permit(:title, :summary)
end

colref won't detect this :summary symbol. Opening the controller file directly is the reliable check.

Serializer Field Pattern: Blueprinter, JSONAPI::Serializer, ActiveModelSerializers — all list fields as strings or symbols:

# Blueprinter
class ArticleBlueprint < Blueprinter::Base
  fields :title, :summary
end

# JSONAPI::Serializer
class ArticleSerializer
  include JSONAPI::Serializer
  attributes :title, :summary
end

# ActiveModelSerializers
class ArticleSerializer < ActiveModel::Serializer
  attributes :id, :title, :summary
end

Determining which model the :summary symbol in a list refers to requires tracing class inheritance, which colref doesn't handle yet. Serializer files tend to be few in number — opening them directly is the reliable check.

ERB templates: <%= @article.summary %> lives in .html.erb files. colref only scans .rb files. Check templates separately:

grep -rn "summary" --include="*.erb" ./

Note that <summary> tags will also hit here, so results will be noisy. Narrowing to @article.summary or article.summary is more practical.

ActiveAdmin / RailsAdmin: If you're displaying or editing the column in an admin interface, the reference is likely a string or symbol there too. I've seen column :summary sitting in an ActiveAdmin show block for a column that had been "confirmed removed" twice already — nobody checked the admin file because it only gets opened once a month. If your project uses either, check those files as well.

Checking serializers and admin files by eye sounds tedious, but in practice it takes a few minutes. These files tend to be organized by model. Open app/serializers/article_serializer.rb, find the relevant serializer, check the attributes list. Open app/admin/article.rb if you use ActiveAdmin. This isn't a grep problem. You open two files and look, and you're done — no tooling required.

colref (Ruby attribute accesses) + grep (variable dynamic patterns and templates) + manual check (Strong Parameters, serializers, admin) covers the vast majority of real-world Rails codebases. There are edge cases colref doesn't handle yet; the Detection Patterns docs list them. For most projects, this three-part check is enough to move from "I think it's probably unused" to "I have confirmed it's unused."

Once you've gone through all of that and colref returns zero, that's a grounded deletion rather than a guess.

The procedure

Here's the full sequence I run.

# 1. Check for column accesses in Ruby code
colref check --orm rails --model Article --field summary ./

# 2. Check for dynamic access (catches literal form; scan call sites for variable form)
grep -rn "send.*summary\|read_attribute.*summary" --include="*.rb" ./

# 3. Check ERB templates (narrow to .summary access)
grep -rn "\.summary" --include="*.erb" ./

# 4. Check Strong Parameters (controllers)
grep -rn ":summary\|'summary'" --include="*.rb" app/controllers/

# 5. Check serializers and admin
grep -rn ":summary\|'summary'" --include="*.rb" app/serializers/ app/admin/

# 6. Generate the removal migration
rails generate migration RemoveSummaryFromArticles summary:string

# 7. Apply to the schema
rails db:migrate

Steps 2–5 are still grep — colref doesn't solve everything. But step 1 cuts 847 results down to 9. The "too many results to check, left it alone" situation: this is the one place that changes.

One more thing about steps 6 and 7: give the migration a descriptive name like RemoveSummaryFromArticles. Six months from now, someone scanning migration filenames can see what changed and when without opening every file. Run the migration locally and make sure your test suite passes before deploying. When you're confident about a deletion it's tempting to skip verification. Don't. If a factory is still setting the deleted column, tests will catch it before production does.

The whole process — run colref, run the checklist, generate the migration, run tests locally, deploy — takes maybe 30 minutes for a column that's actually unused. Compare that to leaving it in db/schema.rb for another year because you couldn't confirm it was safe.

The difference between "I think it's probably unused" and "zero results in Ruby code, no dynamic send call sites, nothing in templates" matters when something goes wrong. Knowing what you checked tells you exactly where the cause wasn't — which narrows down where it was. A grounded deletion makes the debugging faster.

First run: try a column you know is used

If you don't have a deletion candidate in mind, start with a column you know is in use — something like title on Article:

colref check --orm rails --model Article --field title ./

If title is actively used, you'll get multiple results with file and line number:

app/controllers/articles_controller.rb:42
app/helpers/articles_helper.rb:11
app/serializers/article_serializer.rb:5

Seeing what a real result looks like makes it easier to judge zero results later. Then try a column you've been wondering about. Close to zero? Move to steps 2–5.

From installation to first run: under five minutes. Running it is faster than reading the README.

What do you do when you get 3 results? Open all three. For each one: is this code still running in production? If a reference is inside a clearly dead code path — wrapped in a feature flag that was turned off, or a method that's never called — it doesn't count as a real reference. If it's live code, the column is still in use. But 3 results is a manageable number. You can make that judgment call.

What if you get 0 results? Don't stop there. Run steps 2–5. Zero from colref, nothing from the dynamic access grep, clean templates, nothing in the controllers or serializers: that's multiple independent checks pointing the same direction. At that point you have something solid to stand on.

Even without a deletion candidate right now, colref fits into routine schema review. Scan db/schema.rb, spot something that looks unused, run colref. Zero results — it goes on the list. "Probably unused" becomes "not referenced in Ruby code" in 30 seconds. Do this periodically on projects you maintain. Every few months scan the migration history for columns you don't recognize, run colref on them, and build a short list. Some end up staying because they're used in ways colref doesn't detect. But a few always turn out to be genuinely gone: references removed over time, nobody noticed, nobody cleaned it up. Those get deleted.

FAQ

Can colref be added to CI?

Yes. The use case is catching references to a deleted column that sneak back in through someone's PR. colref check exits with code 0 when there are zero results, so it fits as a step in GitHub Actions or CircleCI:

- name: Check removed column references
  run: colref check --orm rails --model Article --field summary ./

When something other than zero results comes back, the CI step fails. That prevents deleted column references from ever making it to main.

What should the team know before using this?

The important thing to communicate is what colref doesn't detect — Strong Parameters, serializers, ERB templates, and variable-symbol dynamic access. If "colref returned zero so it's safe" becomes the assumption without those additional checks, things get missed. Documenting a checklist alongside colref — "colref covers direct access in Ruby code; run these greps and open these files for the rest" — means new team members get the full picture from day one.

colref is still in development. If something doesn't work or you get unexpected results, open an issue at github.com/shinagawa-web/colref. Real usage feedback is what shapes the priorities.

colref currently supports Django and Rails. For the roadmap, see issue #74.

How many columns are you sitting on that you haven't been able to delete?

"Renaming `user` when grep can't tell which model you mean"

Kazu — Tue, 09 Jun 2026 13:05:00 +0000

I stopped mid-task trying to add an editor field to the Article model.

The problem was the existing user field. Adding editor would leave Article with two foreign keys to User sitting side by side:

class Article(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    editor = models.ForeignKey(User, on_delete=models.CASCADE, related_name='edited_articles')

I could already see the code review comment. "What is user here? The author? The last person who touched it?"

The right move was to rename user to author first, then add editor. Leave it as-is and that ambiguity is baked into the codebase permanently. Every new developer who opens the model will have the same question. The field name is wrong and everyone who encounters it will eventually notice.

The problem was how much that rename scared me. user doesn't live only inside the Article model definition. It gets accessed as article.user in views, output as user_id in serializers, listed in admin's list_display, returned as a response key the frontend depends on. Every one of those places needs to change to author or author_id. The serializer's fields list, the frontend's expected key names — all of it.

Miss one and here's what happens: you renamed article.user to article.author in views.py but missed fields = ['user_id'] in the serializer. The API can no longer return author_id. The frontend's expecting user_id and that screen breaks. Or you run the migration and there's a reference still sitting somewhere — AttributeError: 'Article' object has no attribute 'user' starts appearing in production logs. The kind of bug you find after the migration is already live. The logs fill up, Slack lights up, someone asks if we should roll back.

So the right order is: find every reference first, then rename. That means knowing every file and every line before touching the model definition or running any migration. The obvious tool is grep.

grep -rn "\.user" --include="*.py" ./

It returned 340 hits.

I ran colref on the same codebase. It returned 4 hits for Article --field user and 3 hits for Article --field user_id. Seven results total — every actual Article.user reference in the codebase.

340 became 7. The gap is what this article is about.

The same problem comes up with any field name that multiple models share. status, name, created_by, assigned_to — grep on a common field name returns results from every model that uses it, and there is no way to filter by which model you care about. The rename that looks straightforward from the model definition turns into a search problem the moment you try to verify what will break.

The Disambiguation Problem: Ten Models, One Field Name

grep returned 340 results because user is not specific to Article. Every model in the project that relates to a user has a user field. Here's where those 340 hits actually came from:

Model	Hits
Comment.user	89
Order.user	62
Profile.user	58
Payment.user	47
other models	78
Article.user / Article.user_id	7

Seven out of 340. The other 333 are legitimate field accesses — not noise, not comments, not file extensions. Real code that correctly uses .user on those other models. Text search has no way to distinguish them from what you're looking for.

This is a different problem from the grep-noise issue that comes up when checking field deletions, where string literals, comments, and file paths inflate the count. That problem is covered in an earlier article in this series. Here the results are all real code. The issue is that user appears on ten models, and text search cannot tell which model any given obj.user belongs to.

You could try narrowing the search. Filter to files that import Article, or grep for article.user with a lowercase a expecting the variable name to match. Every refinement introduces a new failure mode. What if a view stores the article in a variable named obj or instance? What if it comes from get_object_or_404 and the variable name is different from what you searched for? A tighter grep pattern gives you more confidence in the hits it returns, but zero information about what it missed. You end up needing to verify your verification.

Checking all 340 results by hand isn't impossible but it's exactly the kind of task that gets postponed indefinitely. You open the first file, see that it's Comment.user, close it, open the next one — Order.user — and somewhere around result 30 you realize this is going to take the rest of the afternoon. You close the tab and tell yourself you'll do it tomorrow. Tomorrow you have other things. The rename doesn't happen and the ambiguity stays.

The field that should have been renamed six months ago is still called user. And the next time someone needs to add another user relationship to the model, the situation is even more tangled.

What you need is a search that understands which model you're asking about.

The Scope Filter: Telling colref Which Model You Mean

colref reads code as structure rather than text. It parses source files into an AST and targets attribute-access nodes — the nodes that represent obj.field in running code. String literals, comments, and template paths are invisible to it. Crucially, it accepts a --model flag.

pipx install colref
# or
pip install colref

For a ForeignKey field, two access patterns appear in real code:

article.user      # fetches the User object (triggers a JOIN)
article.user_id   # fetches the FK integer directly (no JOIN)

Performance-conscious code often uses article.user_id to avoid the join when only the ID is needed. Both need to be found and updated, so run both:

# find obj.user accesses on Article instances
colref check --orm django --model Article --field user ./

# find obj.user_id accesses on Article instances
colref check --orm django --model Article --field user_id ./

# --field user
app/views.py:58
app/api.py:23
app/admin.py:11
app/tests/test_views.py:34

# --field user_id
app/serializers.py:18
app/api.py:29
app/tests/test_api.py:41

Seven results. Every one is an Article.user or Article.user_id access. Comment.user, Order.user, every other model's user field — excluded.

How does it tell the difference? colref reads the project's models.py files first and builds a map of which fields belong to which model. It confirms that Article has a field named user. Then it walks the AST looking for attribute-access nodes and applies the model context: comment.user is classified as a Comment instance access and filtered out. It's not searching for the string user — it's reading the syntax tree and reasoning about which model each access belongs to.

Seven results is a number you can act on. Open each file, verify the context, update the reference. At a reasonable pace that's fifteen to twenty minutes of work. For each result, the question is simple: is this code still active in production? If the access is inside a function you know is live, update it. If it's inside commented-out code or a block that clearly never runs, note it and move on. Seven results means you can make that judgment for each one without losing track of where you are.

Once you've gone through colref's output and updated every location, there are two categories colref doesn't cover: serializer fields lists and admin class attributes. Neither is detected because determining which model a string like 'user_id' refers to inside a fields = [...] list requires tracing class inheritance, which colref doesn't handle yet. Check those directly before running the migration:

grep -rn "user_id" --include="*.py" ./app/serializers.py ./app/admin.py ./app/forms.py

These files are limited in number. In most projects you have one serializers file, one admin file, maybe one forms file. Opening each and searching for user_id takes a few minutes.

The Trap: One Wrong Keystroke Drops Your Data

After updating every reference colref found and checking the serializers manually, the next step is generating the migration:

python manage.py makemigrations --name rename_article_user_to_author

Django detects that the old field is gone and a new one appeared, and asks:

Did you rename article.user to article.author (a ForeignKey)? [y/N]

The default is N. If you're not paying attention and hit Enter, or answer n because you're not sure what Django is asking, the resulting migration looks like this:

# What you get if you answer n — do not use this
operations = [
    migrations.RemoveField(
        model_name='article',
        name='user',
    ),
    migrations.AddField(
        model_name='article',
        name='author',
        field=models.ForeignKey(...),
    ),
]

RemoveField drops the user_id column. AddField creates a new empty author_id column. All the data that was in user_id — every article's author relationship — is gone. On a production database with existing rows, this means every Article.author is now null or missing, depending on whether the field allows null. If it doesn't allow null, the migration fails partway through. Either way, not a situation you want to be in.

Answer y. That generates a RenameField migration:

# What you get if you answer y — this is what you want
operations = [
    migrations.RenameField(
        model_name='article',
        old_name='user',
        new_name='author',
    ),
]

RenameField renames the column in place. The data stays exactly where it is. Every row that had a user_id value now has that same value under author_id.

If you'd rather skip the interactive prompt and write the migration directly, use RenameField explicitly:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ("app", "0042_previous_migration"),
    ]

    operations = [
        migrations.RenameField(
            model_name="article",
            old_name="user",
            new_name="author",
        ),
    ]

That's the safest approach: write the migration by hand, review it, apply it. No interactive prompt to accidentally misread.

One more check before applying: run the migration against your local database and confirm your test suite still passes. If a test factory is still setting user=... on Article instead of author=..., the tests will catch it here rather than in production. Factory and fixture files are another category colref doesn't cover — they often set fields using keyword arguments that look like function calls, not attribute accesses. A passing test suite after the migration is the confirmation that nothing was missed.

The API Surface: Renaming Propagates Past Python

Running python manage.py migrate isn't the end of the rename. The model field changed from user to author, which means the database column changed from user_id to author_id. Anything that reads data from the API and expected user_id in the response now receives author_id instead. That's a breaking change.

The places that need to change:

TypeScript type definitions that declare an Article interface with a userId field
API client code that reads response.data.user_id
OpenAPI schemas, and any client SDKs generated from them
Tests that check response payloads against user_id

A team that generates TypeScript types from the OpenAPI schema will have userId in dozens of components, not just one. A mobile app consuming the same API has its own model types. An analytics pipeline reading the JSON might extract user_id by name. None of these are visible from the Django codebase. The Python rename is only half the work — the other half is knowing what else consumed that response key and how quickly those consumers can be updated.

If the backend and frontend deploy together — same release, same CI run — update both in one change and deploy together. The rename on the Python side, the type definition update on the TypeScript side, deployed atomically. That's the cleanest path when it's available.

If backend and frontend deploy independently, there's a window between the backend deploy (which changes the response key from user_id to author_id) and the frontend deploy (which updates the code to read author_id). During that window the frontend is broken.

Django REST Framework's source parameter lets you keep the old response key temporarily:

class ArticleSerializer(serializers.ModelSerializer):
    # temporary bridge: API still returns user_id while the frontend catches up
    user_id = serializers.IntegerField(source='author_id', read_only=True)

    class Meta:
        model = Article
        fields = ['id', 'title', 'user_id', ...]

With this in place, the API continues returning user_id after the migration while the frontend deploys its update. Once the frontend is deployed and no longer reads user_id, remove the explicit field declaration and update fields to include author_id instead.

More steps, but no breaking window. Teams doing continuous deployment find this easier to manage than trying to coordinate a simultaneous Python-and-frontend release: Python rename + migration in one deploy, frontend update in the next, serializer bridge removed in a third cleanup.

The Naming Debt: Why `user` Became a Liability

Having made it through the rename, it's worth asking how the situation came up in the first place.

user describes an implementation fact — "this is a ForeignKey to User" — without saying anything about what that relationship means. Is it the author? The assignee? The last editor? The name doesn't tell you. That gap becomes a problem the moment you need a second user relationship on the same model, and it becomes a grep problem the moment you need to check where the field is used.

user is also the most common ForeignKey name in Django codebases by a wide margin. Comment, Order, Payment, Profile — every model that touches users tends to call the field user. That's why grep returned 334 irrelevant results. The name isn't specific to any model or any relationship. It's the field name equivalent of naming a variable data.

What would have been different with author from the start?

grep -rn "\.author" --include="*.py" ./

Far fewer models have an author ForeignKey. Results would stay in the dozens and remain checkable by hand. You might not have needed colref at all.

The principle for naming ForeignKey fields: use the name of the relationship, not the name of the target model. Instead of user, use author, assignee, reviewer, approver — whatever the relationship actually means in your domain. The field name should describe why the relationship exists, not just where it points.

The test is simple: can a developer reading the model definition understand the field's purpose without looking at any other file? author passes that test. user doesn't. A field named user on an Article model tells you the type of the related object but nothing about why the relationship exists. That gap grows more costly every time a new developer joins the team or a new relationship gets added to the model.

That's straightforward to apply on a new project, when no one has committed to a name yet. On an inherited codebase, a product that's been running for years, "it should have been author from the start" is not actionable. user is already in views, serializers, tests, and migrations. The rename has to happen, and the question is only how to do it without breaking anything in production.

That's where colref is useful. Seven results instead of 340. A fifteen-minute update instead of an indefinitely postponed task.

The rename that's been sitting on the backlog is usually not blocked by difficulty. It's blocked by the upfront cost of a check that feels too expensive to run. Reducing that cost from hours to minutes is what makes the work actually happen.

colref supports Django and Rails. For the full list of what it detects and the current roadmap, see github.com/shinagawa-web/colref.

Appendix: The rename procedure

Step 1: Find every reference.

colref check --orm django --model Article --field user ./
colref check --orm django --model Article --field user_id ./

Update each location colref returns. Then check serializers, admin, and forms for string references colref does not detect:

grep -rn "user_id" --include="*.py" ./app/serializers.py ./app/admin.py ./app/forms.py

For the full list of what colref detects and does not, see the Detection Patterns docs.

Step 2: Rename the field in the model.

class Article(models.Model):
    author = models.ForeignKey(User, on_delete=models.CASCADE, related_name='articles')

Step 3: Generate the migration. Answer y.

python manage.py makemigrations --name rename_article_user_to_author

When Django asks "Did you rename article.user to article.author?", answer y. n generates RemoveField + AddField and drops your data. y generates RenameField and keeps it.

Step 4: Run your tests locally, then apply.

python manage.py migrate

Step 5: Handle the API surface.

Update TypeScript types, API clients, and OpenAPI schemas. If the frontend cannot deploy at the same time, add user_id = serializers.IntegerField(source='author_id') to the serializer as a temporary bridge, then remove it once the frontend has deployed.

How to safely remove a Django model field: finding every real reference before you delete

Kazu — Tue, 02 Jun 2026 13:00:00 +0000

Every Django project has at least one of these. A model with an old field that's probably not used anymore. "Probably" is the scary part. If something in production is still referencing it, deleting the column breaks the app. AttributeError, in production.

So you don't delete it. You want to, but you can't figure out how to check safely, so it just sits there. The column takes up space in every query. The field clutters the model definition. And it keeps sitting there, month after month, because the cost of confirming it's unused feels higher than the cost of leaving it.

Think about what happens when AttributeError: 'Article' object has no attribute 'summary' hits production. Users get 500 errors every time they open the page. Logs flood. Slack lights up. "Was it that deploy we just pushed?" Already considering a rollback. And the cause was deleting a field you thought was unused.

That's why nobody deletes anything. You can't be sure, so you don't. That's the right call. The problem is there was no way to get sure.

I tried the obvious thing: searching for the field name in VS Code. Hundreds of hits. I started opening them one by one and immediately noticed most aren't real references. File paths, comments, unrelated variable names. I searched for the .html field in one Django project and got 1,202 results. The actual code accessing that field: 10 results.

1,192 were noise. But you couldn't know that upfront.

Why search returns 1,202 hits

When you give up because there are too many results, that's not a failure on your part. VS Code search and grep just weren't built for this.

These tools answer "does this string appear anywhere in this file?" That's useful for a lot of things. But when you want to know "is this field actually referenced in code?", text search picks up way too much. The field name in a string literal, in a comment, as part of a filename: it all counts as a hit. Sorting through them is manual work.

Here's what those 1,202 results for .html broke down to:

Type	Count
File extensions (`layout.html`, etc.)	1,087
Unrelated strings	27
Comments	21
Other noise	57
Actual field accesses	10

You can't check 1,200 results. Giving up was the right call. The issue wasn't how you used the tool. You were using the wrong tool.

And .html isn't a special case. Say you want to delete a title field on an Article model. "title" shows up everywhere in a codebase: variable names, dictionary keys, comments, strings. Hundreds of results. Same problem.

Common field names are the worst. status, name, type, created_at — these appear in dozens of unrelated contexts throughout a typical Django project. The search that was supposed to answer a simple question becomes 40 minutes of opening files, closing files, and losing track of what you've already checked.

"I searched, couldn't check everything, left it alone." Most Django developers have been here. You want to delete it but can't. The check isn't impossible, it's just too expensive. So the field stays. They accumulate.

This compounds over time. A project that's two years old might have thirty fields that nobody's confident about. The developers who added them have moved on. The tickets that motivated them are closed. The tests, if they exist, pass regardless of whether the field is used. There's no mechanism forcing a cleanup, just the gradual intuition that the schema is getting harder to understand.

If you know regex, you might try narrowing it: grep -rn "\bhtml\b" --include="*.py" to limit to Python files. Still the same problem. "html" as a dictionary key, in a comment — it all still hits. "Python files only" and "actual field access" are completely different things.

The more you refine the regex, the more you start wondering whether the regex itself is missing something. You end up needing to verify the verification. The tool that was supposed to save you time has become another source of doubt.

When undeletable fields pile up, here's what actually happens.

The schema bloats. Ten, twenty unused columns accumulate. A new developer opens the model, scans the fields. "What's this one for?" They try to find out, can't, and decide not to touch it. Reasonable. But when that pattern repeats, you end up with an unspoken rule: don't touch this model.

Every migration feels slightly more risky. The unease builds until nobody touches it at all. Six months later, another developer thinks the same thing. The cycle repeats.

This is technical debt in the same way missing tests are. An unresolvable cost that accumulates. Development slows. Onboarding takes longer. Nobody intended this, but the project gets heavier over time.

This isn't a skills problem. The tool didn't exist yet.

Here's the situation I kept ending up in: I'd look at a field, feel like it was probably unused, open VS Code to check, get overwhelmed by results, close it, and go do something else. The field would still be there six months later. The next developer would go through the same loop. The field would still be there a year later.

At some point the schema becomes archaeology. Fields with names like legacy_content, old_slug, deprecated_flag: nobody knows what they do, nobody wants to touch them, and the project carries them forever. Every SELECT * is slightly slower. Every new developer's mental model of the data is slightly more confused.

The real cost is cognitive load. Every unused field is a small tax on everyone who reads the model. Multiply that by thirty fields and two years of new developers and you start to see why "we never clean up old fields" becomes an invisible drag on velocity.

How colref reads code structure instead of text

What you actually wanted to know was: where is obj.html referenced in code? colref returns exactly that. File paths and string contents ignored.

How does it tell the difference? Instead of treating code as a sequence of characters, it reads the code structure.

embed.html in Python means "read the html attribute of the embed object": a specific structure. "pages/publish.html" is string data, not an attribute access. Reading code structure makes that difference detectable. Only places written as object.field_name get picked up. The .html that appears inside a string is ignored.

If text search is like pressing Ctrl+F on a page, reading code structure is closer to a human reading through every line. Except it handles thousands of lines in an instant.

It scans Python code and returns only .field_name accesses, with file and line number.

Method	Hits (for `.html`)	What it sees
VS Code full-text search	3,534	All string matches
grep `\.html\b`	1,202	Word-boundary matches
colref	10	Actual field accesses only

3,534 or 1,202 becomes 10. Whether you can act on the results depends entirely on how many there are.

When you get 10 results: open each one. views.py:42 means go to that line and check whether obj.html is actually being accessed. Real reference — can't delete. Not a real reference — skip. Ten results takes 10–15 minutes.

A few common things you'll see when reviewing results: the field name appearing in a migration file (colref skips migrations, but if it didn't, this would be a false positive; the migration is just recording the history of the field's existence, not actively using it). You might also see test factories or fixtures that set the field value. Worth noting: if you delete the field and forget to update the factory, your test suite will break. That's not a reason not to delete, it's just something to clean up as part of the deletion.

When you get zero: you have a fact. "No references found in Python code." That's different from "I think it's probably unused." Move to the next step: checking getattr, templates, Admin, Forms, and Serializers. Zero from colref is the starting point, not the finish line.

The shift is from "check 1,200 things" to "check 10 things, then a handful of specific files." That's the difference between a task you'll defer indefinitely and one you'll do today.

For the technical details of how code structure is read, see ARCHITECTURE.md.

Installation

Install via pipx:

pipx install colref

Or with pip:

pip install colref

Specify the model name, field name, and your project directory:

colref check --orm django --model Embed --field html ./

Results come back as filename:line_number. Each one is something you can open directly. Ten results takes maybe ten minutes to verify. Nothing compared to scrolling through 1,202 results, losing your place, and giving up halfway through.

A note on model names: use the class name exactly as it appears in your models file, including capitalization. Embed, not embed or EMBED. Field names are case-sensitive too: html, not HTML. If you get zero results for a field you know is used, double-check the casing first.

The ./ at the end is the path to scan. You can point it at a specific app directory if you want to narrow it down, but pointing at the project root works fine and makes sure nothing gets missed.

What zero results doesn't cover

Zero results doesn't mean "safe to delete." It means "not found in Python code."

Dynamic access: getattr(obj, field_name) with the field name in a variable won't be detected. Check separately:

grep -rn "getattr" --include="*.py" ./ | grep your_field

Django templates: {{ page.html }} lives in .html files. colref only scans .py. Check templates separately:

grep -rn "your_field" --include="*.html" ./

Django Admin, Forms, and DRF Serializers: This is the easiest one to miss. None of these are detected:

# Django Admin
class ArticleAdmin(admin.ModelAdmin):
    list_display = ['title']
    list_filter = ['title']

# Django Forms
class ArticleForm(forms.ModelForm):
    class Meta:
        fields = ['title']

# DRF Serializer
class ArticleSerializer(serializers.ModelSerializer):
    class Meta:
        fields = ['title']

Determining which model the string 'title' in a list refers to requires tracing class inheritance, which colref doesn't handle yet. Check these separately:

grep -rn "your_field" --include="*.py" ./

This grep has the same noise problem as full-text search. Opening the Admin, Forms, and Serializer files directly is more reliable. In most projects there aren't many of them.

These are exactly the places a Django beginner might not think to check. You verify the views, the serializers feel obvious after you remember them, but Django Admin is easy to forget, especially if the admin configuration lives in a file you rarely open. I've seen list_display hold a reference to a field that had been "confirmed deleted" twice already. The admin file just wasn't in anyone's mental checklist.

Once you've checked all of the above and colref returns zero, that's a grounded deletion: confirmed in Python code, checked getattr, templates, Admin/Forms/Serializers. Not "I think it's probably fine."

Checking Admin/Forms/Serializers by eye sounds tedious, but in practice it takes a few minutes. These files tend to be organized by model. Open admin.py, search for the model name, check list_display and related attributes. Open serializers.py, find the relevant serializer, check fields. Open forms.py if you have one. It's not a grep problem, it's an "open three files and look" problem. That's manageable even without a tool.

colref (Python attribute accesses) + grep (dynamic patterns and templates) + manual check (Admin/Forms/Serializers) covers the vast majority of real-world Django codebases. There are edge cases colref doesn't handle yet; the Detection Patterns docs list them. For most projects, this three-part check is enough to move from "I think it's probably unused" to "I have confirmed it's unused."

The five-step procedure

# 1. Check for field accesses in Python code
colref check --orm django --model YourModel --field your_field ./

# 2. Check for dynamic access
grep -rn "getattr" --include="*.py" ./ | grep your_field

# 3. Check templates
grep -rn "your_field" --include="*.html" ./

# 4. Delete the field and generate the migration
python manage.py makemigrations --name remove_your_field

# 5. Apply to the schema
python manage.py migrate

Steps 2 and 3 are still grep — colref doesn't solve everything. But step 1 cuts 1,202 results down to 10. The "too many results to check, left it alone" situation: this is the one place that changes.

The difference between "probably unused, I think" and "zero results in Python code, no getattr, nothing in templates" is real. If something breaks in production, knowing what you checked tells you exactly where to look. You know the cause came from outside your checked scope: a dynamic reference, a template, a pattern colref doesn't handle yet. The cause is narrowed. Grounded deletion makes debugging faster when things go wrong.

One more thing about step 4 and 5: don't skip makemigrations --name. Giving the migration a descriptive name like remove_summary_field makes the history readable. Six months from now, someone scanning migration filenames can see what changed and when without opening every file.

Before deploying, run the migration locally and make sure your test suite passes. When you're confident about a deletion it's tempting to skip the verification. Don't. If a test factory is still setting the deleted field, the tests will catch it before production does.

The whole process — run colref, check the checklist, generate the migration, run tests locally, deploy — takes maybe 30 minutes for a field that's actually unused. Compare that to leaving the field there indefinitely because you couldn't confirm it was safe to remove.

First run: try a field you know is used

If you don't have a candidate field in mind, try a field you know is used, something like title on Article:

colref check --orm django --model Article --field title ./

If title is in use, you'll get multiple results with file and line number:

app/views.py:42
app/serializers.py:18
app/templates/article_detail.py:11

Seeing what a real result looks like makes it easier to judge zero results later. Then try a field you've been wondering about. Close to zero? Move to steps 2 and 3.

From installation to first run: under five minutes. No need to read the README first. Running it is faster than reading about it.

What do you do when you get 3 results? Open all three. For each one: is this code still running in production? If a reference is inside a function that's clearly dead code, something wrapped in if False or commented out, it doesn't count. If it's live code, the field is still in use. But 3 results is a manageable number. You can make that judgment call.

What if you get 0 results? Don't stop there. Run steps 2 and 3. Zero from colref, zero from getattr grep, zero from template grep: that's three independent checks. At that point, also check your Admin, Forms, and Serializer files by eye. If all of that is clear, you have something solid to stand on.

Even without a deletion candidate right now, colref is useful for routine schema review. Scan migration history, spot something that looks unused, run colref. Zero results? It goes on the deletion candidate list. "Probably unused" becomes "not referenced in Python code" in 30 seconds.

I do this periodically on projects I maintain. Every few months I scan the migration history for fields I don't recognize, run colref on them, and build a short list. It takes maybe 20 minutes and usually turns up one or two candidates worth investigating further. Some end up staying because they're used in ways colref doesn't detect yet. But a few always turn out to be genuinely gone: references removed over time, nobody noticed, nobody cleaned it up. Those get deleted.

colref is still in development. If something doesn't work or you get unexpected results, open an issue at github.com/shinagawa-web/colref. Real usage feedback is what shapes the priorities. A bug report with a concrete example is more useful than ten feature requests.

colref currently supports Django and Rails. For the roadmap, see issue #74.

If you find a field that colref misses, something it should have flagged but didn't, that's especially useful to report. The detection gap around Admin, Forms, and Serializers is a known limitation, but there may be patterns in your codebase that nobody's encountered yet. The tool gets better with more real-world cases.

How many fields are you sitting on that you haven't been able to delete?

"How I Cut My Go Markdown Linter's Benchmark by 81%"

Kazu — Tue, 26 May 2026 13:00:00 +0000

When I started optimizing gomarklint, I had no benchmarks. I had unit tests. I had coverage. But I had no idea what the linter actually cost to run on a real document.

Here's what I found, what I changed, and what I'd do differently next time.

What is gomarklint?

gomarklint is a Go-based CLI Markdown linter I've been building as an open-source project. The pitch: catch broken links before your readers do, keep your Markdown clean, single binary, no Node.js required.

The main alternative most teams reach for is markdownlint, which works well but requires a Node.js runtime. For Go projects running in a lean CI environment, pulling in Node.js just to lint Markdown felt like the wrong tradeoff. gomarklint ships as a standalone binary installable via Homebrew, npm, or go install, and integrates with GitHub Actions and pre-commit out of the box.

The rule set covers around 25 checks: structural rules like heading-level (no H4 appearing under H2 without an H3 in between), content rules like no-bare-urls and fenced-code-language, and link validation including internal anchor checking. Each rule emits diagnostics with file path, line number, and severity — error causes a non-zero exit, making it safe as a CI gate.

Internally, each rule is a function that receives the full file content as a slice of lines and returns a slice of violations. No shared parse tree, no AST, just functions over strings. That made it easy to add rules quickly.

The current release processes 100,000+ lines in under 170ms. A typical 200-line README lints in under 0.2 ms across all rules, and a large documentation site with hundreds of files fits in a CI step nobody notices. Getting there required 20 PRs over three weeks, each measured before it merged.

The Starting Point: No Visibility

gomarklint's rules like heading-level, no-bare-urls, fenced-code-language, and around 25 others each receive the full file content split into lines and return a slice of violations. Rules run independently, no shared parse tree, no AST, just functions over strings.

That architecture is easy to extend. But every rule doing anything non-trivial had to solve the same problem on its own: "is this line inside a code block?" Headings inside fenced blocks aren't real headings. URLs inside fenced blocks aren't real URLs. Every rule that cared had to figure it out independently.

The solution that grew organically was a shared utility called GetCodeBlockLineRanges. It scanned the entire document, built a list of [start, end] line ranges for every fenced block, and returned them. Any rule could then call isInCodeBlock(lineNumber, ranges) to check membership via a linear search.

I didn't notice this was a problem because I never measured it.

Step 1: Build a Benchmark You Can Trust

Before optimizing anything, I needed a benchmark I could actually rely on. The existing per-rule _bench_test.go files were isolated and not included in CI comparisons, so they gave no signal about end-to-end cost.

I rewrote the benchmark around a single generateComplexMarkdown(n int) function that produces a realistic n-section document (headings, paragraphs, lists, fenced code blocks, images, links) exercising every rule's scan path without producing any violations:

func writeIntro(sb *strings.Builder) {
    sb.WriteString("# Main Title\n\n")
    sb.WriteString("This is the introduction to the document.\n\n")
}

func writeHeading(sb *strings.Builder, i int) {
    fmt.Fprintf(sb, "## Section %d\n\n", i)
}

func writeParagraph(sb *strings.Builder) {
    sb.WriteString("This section contains *important* information. ")
    sb.WriteString("Here are some **key** details that you should know about.\n\n")
}

func writeList(sb *strings.Builder) {
    sb.WriteString("Key points:\n\n")
    sb.WriteString("- First important point\n")
    sb.WriteString("- Second critical detail\n")
    sb.WriteString("- Third consideration\n\n")
}

func writeCodeBlock(sb *strings.Builder) {
    sb.WriteString("```

go\n")
    sb.WriteString("func example() error {\n")
    sb.WriteString("    return nil\n")
    sb.WriteString("}\n")
    sb.WriteString("

```\n\n")
}

func writeLinks(sb *strings.Builder, i int) {
    sb.WriteString("Useful resources:\n\n")
    fmt.Fprintf(sb, "- [Documentation](https://example.com/docs/%d)\n", i)
    fmt.Fprintf(sb, "- [GitHub](https://github.com/project/%d)\n", i)
    sb.WriteString("\n")
}

func writeImage(sb *strings.Builder, i int) {
    fmt.Fprintf(sb, "![Diagram %d](diagram%d.png)\n\n", i, i)
}

func writeSubsection(sb *strings.Builder, i int) {
    fmt.Fprintf(sb, "### Subsection %d.1\n\n", i)
    sb.WriteString("More detailed information goes here.\n\n")
}

func generateComplexMarkdown(sections int) string {
    var sb strings.Builder
    writeIntro(&sb)
    for i := 1; i <= sections; i++ {
        writeHeading(&sb, i)
        writeParagraph(&sb)
        writeList(&sb)
        if i%2 == 0 {
            writeCodeBlock(&sb)
        }
        if i%3 == 0 {
            writeLinks(&sb, i)
        }
        if i%4 == 0 {
            writeImage(&sb, i)
        }
        writeSubsection(&sb, i)
    }
    result := sb.String()
    return result[:len(result)-1]
}

That last constraint — zero violations — matters. A benchmark that triggers violations measures error-reporting cost, not scanning cost. I added a guard test to enforce it:

func TestBenchmarkContentIsViolationFree(t *testing.T) {
    content := generateComplexMarkdown(1000)
    results, err := lint.LintContent(content, benchmarkConfig())
    require.NoError(t, err)
    assert.Empty(t, results)
}

This test runs in CI on every PR in the series. If anyone accidentally adds a violation to the benchmark content, the build breaks before the numbers become meaningless.

The benchmark also pays forward. Now that it runs on every PR, any new rule added to gomarklint gets automatically checked for performance regression before it merges. If a new no-trailing-spaces rule adds 15% to the geomean, benchstat surfaces that number in the PR diff — before it ships, not after. Without a benchmark in CI, performance regressions from new features are invisible until someone notices the linter "feels slower" on a large repo.

Step 2: Profile Before Touching Anything

With the benchmark in place, I ran a CPU and allocation profile against the 1000-section document:

go test -bench=BenchmarkFullLinting -benchtime=5s \
    -cpuprofile=cpu.prof -memprofile=mem.prof ./cmd/
go tool pprof -top cpu.prof

The top result:

Showing top 5 nodes out of 42
      flat  flat%   sum%        cum   cum%
   6.32s  63.72% 63.72%      6.32s 63.72%  isInCodeBlock
   0.89s   8.97% 72.69%      0.89s  8.97%  strings.TrimSpace
   0.54s   5.44% 78.14%      0.54s  5.44%  regexp.(*Regexp).FindStringSubmatch
   ...

isInCodeBlock was a shared helper called from multiple rules. Each call invoked GetCodeBlockLineRanges, which allocated a [][2]int slice by scanning the entire document to find fence boundaries, then performed a linear search through that slice to answer one question: "is line N inside a code block?"

That's O(n × k) per rule per line, where n is the number of lines and k is the number of code blocks. In a document with 6 rules calling it and 50 code blocks, every line triggered 6 linear searches over 50 ranges. The cost multiplied silently across every rule that wanted to skip code block content.

The allocation profile added a second finding: CheckHeadingLevels was calling regexp.MustCompile(atxHeadingPattern) inside the function body on every invocation — not once at package init. That single oversight was allocating ~16 MB of heap per benchmark run and showed up clearly in -memprofile output.

Step 3: Fix the Biggest Problem First

Both issues were in the heading-level rule. I fixed them together in PR #182:

Before:

func CheckHeadingLevels(filename string, lines []string, offset int, minLevel int) []LintError {
    var errs []LintError
    prevLevel := 0
    headingRegex := regexp.MustCompile(`^(#{1,6})\s+`) // compiled on every call
    codeBlockRanges, _ := GetCodeBlockLineRanges(lines) // O(n) alloc

    for i, line := range lines {
        if isInCodeBlock(i+1, codeBlockRanges) { // O(k) linear search per line
            continue
        }
        matches := headingRegex.FindStringSubmatch(line)
        if matches != nil {
            currentLevel := len(matches[1])
            // ... violation checks
            prevLevel = currentLevel
        }
    }
    return errs
}

After:

// atxHeadingLevel returns the heading level (1–6) or 0.
// Pure byte scan — no regex, no allocation.
func atxHeadingLevel(line string) int {
    level := 0
    for level < len(line) && line[level] == '#' {
        level++
    }
    if level == 0 || level > 6 {
        return 0
    }
    if level == len(line) || line[level] == ' ' || line[level] == '\t' {
        return level
    }
    return 0
}

func CheckHeadingLevels(filename string, lines []string, offset int, minLevel int) []LintError {
    var errs []LintError
    prevLevel := 0
    inCodeBlock := false
    var fenceMarker string

    for i, line := range lines {
        if len(line) == 0 {
            continue
        }
        first := firstNonSpaceByte(line)

        // Inline fence tracking — no allocation, no O(n×k) lookup.
        if inCodeBlock {
            if first == fenceMarker[0] {
                trimmed := strings.TrimSpace(line)
                if IsClosingFence(trimmed, fenceMarker) {
                    inCodeBlock = false
                    fenceMarker = ""
                }
            }
            continue
        }

        if first != '#' && first != '`' && first != '~' {
            continue
        }

        trimmed := strings.TrimSpace(line)
        if marker := openingFenceMarker(trimmed); marker != "" {
            inCodeBlock = true
            fenceMarker = marker
            continue
        }
        if first != '#' {
            continue
        }

        currentLevel := atxHeadingLevel(trimmed)
        if currentLevel == 0 {
            continue
        }
        // ... violation checks
        prevLevel = currentLevel
    }
    return errs
}

The result on CI (AMD EPYC, 1000-section document):

metric	before	after	delta
time/op (geomean)	52.55 ms	14.78 ms	−72%
memory/op	2.108 Mi	1.912 Mi	−9%
allocs/op	9,225	4,677	−49%

One PR. 72% of the time gone.

Step 4: Turn the Pattern Into a System

The heading-level fix revealed a reusable pattern: the firstNonSpaceByte prefilter. Most rules only care about lines starting with a specific byte. A heading starts with #. A fenced code block starts with ` or ~. A hard tab starts with \t.

Reading the first non-whitespace byte costs almost nothing — one loop over leading spaces, then a byte read. Calling strings.TrimSpace on every line is not free, especially when 95% of lines would be skipped anyway.

I applied this prefilter across 14 more rules over 14 subsequent PRs:

func firstNonSpaceByte(s string) byte {
    for i := 0; i < len(s); i++ {
        c := s[i]
        if c != ' ' && c != '\t' && c != '\r' && c != '\n' {
            return c
        }
    }
    return 0
}

Each individual PR produced a modest gain — 3% to 15% per rule. But they compounded:

PR	Rule	time delta
#193	no-bare-urls	−15.5%
#195	no-emphasis-as-heading	−8.8%
#190	empty-alt-text	−6.4%
#194	duplicate-heading	−3.0% (+ −12.6% memory)
#192	no-empty-links	−2.8%

The Final Numbers

Starting from the benchmark baseline after the scaffolding work, to the last PR in the series:

metric	start	end	improvement
time/op (geomean)	74.37 ms	13.88 ms	−81%
memory/op	2.221 Mi	1.654 Mi	−25%
allocs/op	9,533	4,510	−53%

Twenty PRs over three weeks. Each one measured, each one merged only after the CI benchmark comparison confirmed no regression.

What I'd Do Differently

Profile before writing a single optimization. I got lucky that the biggest problem (isInCodeBlock at 63%) was immediately obvious. In a more complex codebase, guessing the hotspot first would have sent me optimizing rules that contributed 0.5% of total CPU while the real bottleneck sat untouched. The 20-minute setup cost of a proper benchmark pays back on the first targeted fix.

Treat the benchmark content as a first-class artifact. A benchmark that produces violations is measuring error-reporting cost, not scanning cost. A benchmark whose content drifts rule-by-rule becomes impossible to interpret. The TestBenchmarkContentIsViolationFree guard test caught three cases mid-series where a content addition accidentally triggered a violation. Without it, I would have been optimizing against corrupted numbers and wondering why the geomean kept moving.

Two PRs in this series landed within measurement noise. One was closed without merging. Having numbers in each PR body made that obvious: the CI benchstat output showed ~ (p=0.485) and there was nothing to argue about. A single "performance" PR with twenty changes mixed together would have buried those non-results.

Writing before/after benchmark numbers in every PR description did something I didn't expect: it made the series reviewable after the fact. Looking back at 20 PRs, I can tell exactly which optimization accounted for which fraction of the total gain. The firstNonSpaceByte prefilter was worth applying to 14 rules because I could see, rule by rule, that it kept working. That's how I found the pattern in the first place.

Try It

gomarklint is open source: github.com/shinagawa-web/gomarklint. The entire optimization series is tracked under issue #146, with each PR linking back to it and carrying its own before/after benchmark numbers.

If you're running a Go linter or any line-by-line text processor, these two patterns are worth trying before anything fancier: inline fence tracking instead of pre-computed ranges, and a first-byte prefilter before any string work. Zero dependencies, easy to test, and between them they accounted for almost all of the 81%.

DEV Community: Kazu

"nginx doesn't run your config top to bottom: two traps with rewrite, try_files, and if"

Symptom 1: rewrite and try_files side by side, and you get a 404

Symptom 2: config placed inside an if block doesn't take effect

How it works: nginx's HTTP processing phases

Symptom 1, followed by phase

Symptom 2, followed by phase

Fixes

Symptom 1

Symptom 2

Checking which location you're in right now

Wrapping up

nginx Tools for When Your Config Gets Out of Hand

Problem 1: Prevent config mistakes upfront

Gixy: static analysis focused on security

crossplane: treat config as data

Which to choose

Problem 2: Generate config files

nginxconfig.io: generate best-practice config in the browser

nginx-proxy: auto-generate config in Docker environments

Which to choose

Problem 3: Automate testing

Test::Nginx: declarative test framework

hurl: a DSL for HTTP testing

Which to choose

Problem 4: Collect metrics

nginx-module-vts: embedded stats module

nginx-prometheus-exporter: bridge stub_status to Prometheus

Which to choose

Problem 5: Run nginx on Kubernetes

ingress-nginx: the Kubernetes community controller

kubernetes-ingress: the NGINX Inc. controller

Which to choose

Problem 6: Visualize logs

GoAccess: real-time stats in the terminal

Loki + Grafana Alloy: long-term log storage and search

Which to choose

Summary: match tool to problem

Gaps that still aren't filled

Dynamic upstream management

Config drift tracking across environments

IDE / LSP support

WAF

Distributed tracing

$request_time Looks Fast, But Users Say It''s Slow: Breaking nginx Latency into 4 Parts

What one number hides

Before you start: add variables to your log format

Breaking it into 4 parts

Computing p50/p95/p99 per part

When ① is always non-zero

When the breakdown breaks: commas and colons

Closing: where is your time going?

References (official documentation)

Silent nginx Config Bugs That Pass nginx -t: if, location, add_header, alias

What nginx -t actually guarantees for you

Semantic level: the grammar is correct, but it's read differently

The contents of if aren't evaluated in the order you think

Location priority is not the order you wrote them in

With add_header, adding one in the child makes all the parents disappear

Forget one trailing slash on alias, and you can climb outside the directory

Security level: it works exactly as intended, which is why it's dangerous

Passing a client-touchable variable straight into proxy_pass

Runtime level: the config text is no longer relevant

After a reload, the old behavior lingers for a while

Summary: how to fill in the space outside the syntax check

References (official documentation)

"How I Designed a Go Linter's Rule Engine So Adding a Rule Is One Line"

First, let's run gomarklint

Rules pile up, and code usually rots

First, express diagnostics, config, and severity as minimal types

Rules are functions, not an interface

Write "skip inside code blocks" once

Severity and exit code are wired on the engine side

Prevent config/code drift with a contract

Actually adding one: no-empty-links (MD042)

1. Write one rule function

2. Add one line to the table

3. Register in config (two spots)

4. Add tests

Extensibility isn't adding layers

Symptom 2: config placed inside an `if` block doesn't take effect

What `nginx -t` actually guarantees for you

The contents of `if` aren't evaluated in the order you think

With `add_header`, adding one in the child makes all the parents disappear

Forget one trailing slash on `alias`, and you can climb outside the directory

Passing a client-touchable variable straight into `proxy_pass`

The `.tape` file — writing terminal operations down

But `.tape` alone is only half of it

What the descendants of `top` assume

The Naming Debt: Why `user` Became a Liability