<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Eric Gitangu</title>
    <description>The latest articles on DEV Community by Eric Gitangu (@ericgitangu).</description>
    <link>https://dev.to/ericgitangu</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2671445%2F799ba2bd-acbc-4721-bdbf-9549fd6e1d5d.jpeg</url>
      <title>DEV Community: Eric Gitangu</title>
      <link>https://dev.to/ericgitangu</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ericgitangu"/>
    <language>en</language>
    <item>
      <title>git skipped my pre-push hook and pushed anyway: the gate that lost its executable bit</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Mon, 16 Nov 2026 20:04:14 +0000</pubDate>
      <link>https://dev.to/ericgitangu/git-skipped-my-pre-push-hook-and-pushed-anyway-the-gate-that-lost-its-executable-bit-4k0g</link>
      <guid>https://dev.to/ericgitangu/git-skipped-my-pre-push-hook-and-pushed-anyway-the-gate-that-lost-its-executable-bit-4k0g</guid>
      <description>&lt;p&gt;&lt;em&gt;By Eric Gitangu. Originally published at &lt;a href="https://developer.ericgitangu.com/blog/git-skipped-my-pre-push-hook" rel="noopener noreferrer"&gt;developer.ericgitangu.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I work on a mobility/fintech platform. This one is about a safety check on &lt;code&gt;git push&lt;/code&gt; that stopped running, said so in a hint nobody reads, and what I changed so it can't go quiet again.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;One of the web apps, an admin console, has a pre-push hook. Before a push leaves the laptop, the hook runs a set of gates that check the app still agrees with the API contract it is built against. A failing gate refuses the push.&lt;/p&gt;

&lt;p&gt;The hooks are tracked in the repository and &lt;code&gt;core.hooksPath&lt;/code&gt; points git at that directory, so every clone gets the same gates.&lt;/p&gt;

&lt;h2&gt;
  
  
  Problem one: the gate judged the wrong thing
&lt;/h2&gt;

&lt;p&gt;The gates read the working tree. A push sends commits. Those are not always the same thing.&lt;/p&gt;

&lt;p&gt;On a stale checkout, the gates refused the push of an unrelated commit that was already green. From a linked worktree, they couldn't find a package they depend on.&lt;/p&gt;

&lt;p&gt;So I changed what the hook judges. git gives pre-push one line per pushed ref on stdin, and the hook now works from that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A deletion has nothing to gate.&lt;/li&gt;
&lt;li&gt;Each distinct pushed commit is gated once. A tag is gated on the commit it points at.&lt;/li&gt;
&lt;li&gt;If the commit is the checked-out HEAD and the checkout is clean (no tracked changes, no untracked files that aren't ignored), it is gated in place.&lt;/li&gt;
&lt;li&gt;Otherwise the commit is checked out into a temporary worktree and gated there, with its own dependencies installed from its own lockfile, frozen, install scripts off. Never a link to another checkout's dependencies.&lt;/li&gt;
&lt;li&gt;If that install fails, the push is refused. Nothing was gated, so nothing goes.&lt;/li&gt;
&lt;li&gt;The temporary worktree is removed on exit, failure, Ctrl-C and hang-up.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The shape of it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r local_ref local_sha remote_ref remote_sha&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$local_sha"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; *&lt;span class="o"&gt;[&lt;/span&gt;!0&lt;span class="o"&gt;]&lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; *&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;   &lt;span class="c"&gt;# all zeros: a deletion&lt;/span&gt;

  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$local_sha"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"$head"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$(git status --porcelain)"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    run_gates .                        &lt;span class="c"&gt;# clean HEAD: gate in place&lt;/span&gt;
  &lt;span class="k"&gt;else&lt;/span&gt;
    git worktree add -q --detach &lt;span class="s2"&gt;"$tmp"&lt;/span&gt; &lt;span class="s2"&gt;"$local_sha"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1
    install_deps &lt;span class="s2"&gt;"$tmp"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1      &lt;span class="c"&gt;# this commit's lockfile, frozen&lt;/span&gt;
    run_gates &lt;span class="s2"&gt;"$tmp"&lt;/span&gt;    &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1
  &lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That went in, tests and all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Problem two: the fix switched the gate off
&lt;/h2&gt;

&lt;p&gt;The next day I found the hook file had been committed with mode &lt;code&gt;100644&lt;/code&gt;. The rewrite had dropped its executable bit.&lt;/p&gt;

&lt;p&gt;Here is what git does with a hook that is not executable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;hint: The '.githooks/pre-push' hook was ignored because it's not set as executable.
hint: You can disable this warning with `git config advice.ignoredHook false`.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then it pushes. Exit code 0.&lt;/p&gt;

&lt;p&gt;So pushes from any checkout of that commit ran no pre-push gate at all. The commit that made the gate smarter also made it not run.&lt;/p&gt;

&lt;p&gt;I'm not claiming anything bad went out in that window. I have no evidence that it did. The finding is narrower and it was enough for me: the check was absent and nothing failed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The decision
&lt;/h2&gt;

&lt;p&gt;Restoring the mode is a one-line fix. It also leaves the same hole for the next rewrite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A gate that can be missing without anything failing isn't a gate. Make its absence a failure.&lt;/li&gt;
&lt;li&gt;Catch it in two places: on the laptop at the first commit, and in CI where no local setting can hide it.&lt;/li&gt;
&lt;li&gt;Failing test first, with a fixture that reproduces the silent skip.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The check
&lt;/h2&gt;

&lt;p&gt;It asks two questions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What mode did git store?&lt;/strong&gt; That is what every fresh clone gets. The file on disk can lie: it can be executable locally and still be &lt;code&gt;100644&lt;/code&gt; in git.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(git ls-files --stage -- .githooks/pre-push | cut -d' ' -f1)"&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$mode"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"100755"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; refuse &lt;span class="s2"&gt;"pre-push is stored as $mode"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Will git find an executable hook in this checkout?&lt;/strong&gt; This covers a hooks path that is unset, one pointed at nowhere, and linked worktrees.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(git rev-parse --git-path hooks)"&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; -x &lt;span class="s2"&gt;"$dir/pre-push"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; refuse &lt;span class="s2"&gt;"no executable pre-push in $dir"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each refusal prints the reason and the fix, which is &lt;code&gt;git update-index --chmod=+x&lt;/code&gt; on the file and a commit.&lt;/p&gt;

&lt;p&gt;Then the wiring:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The pre-commit hook calls the check first. A checkout whose pre-push can't run fails at its first commit, not after a week of ungated pushes.&lt;/li&gt;
&lt;li&gt;CI runs the stored-mode half on every push and pull request. CI has no local hooks config, so it only reads what git stored. It also runs the check's own test.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The test builds throwaway repositories and covers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a good hook in a linked worktree, where a real push runs it&lt;/li&gt;
&lt;li&gt;a hook stored as &lt;code&gt;100644&lt;/code&gt;: the check refuses, and the fixture proves a real push skips the hook&lt;/li&gt;
&lt;li&gt;executable on disk but &lt;code&gt;100644&lt;/code&gt; in git: still refused&lt;/li&gt;
&lt;li&gt;hooks path unset, and hooks path set to &lt;code&gt;/dev/null&lt;/code&gt;: both refused&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The second case matters most to me. The test doesn't trust my reading of git's behaviour. It makes git do it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is live, and what isn't
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The commit-aware pre-push, the check, the pre-commit call and the CI job are on the app's main branch, and CI runs the job on every push and pull request.&lt;/li&gt;
&lt;li&gt;This is developer tooling. "Shipped" here means every checkout of main, not a server.&lt;/li&gt;
&lt;li&gt;The same repository already had the rule for its credential scan: if the scanner isn't installed, the commit is blocked.&lt;/li&gt;
&lt;li&gt;Limits: pre-commit can be skipped too, so CI is the backstop for the mode. The check proves the hook can run. It does not prove the gates inside it are good.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;git skips a non-executable hook with a hint, not an error. Hints scroll past.&lt;/li&gt;
&lt;li&gt;The mode stored in git is the one that counts. Read it with &lt;code&gt;git ls-files --stage&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A push gate should judge the commit being pushed, with that commit's dependencies.&lt;/li&gt;
&lt;li&gt;If a check can be absent, test for its absence. In CI too.&lt;/li&gt;
&lt;li&gt;Reproduce the failure in a fixture. Don't assert what you believe the tool does.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The hook had one job and no permission to do it. Now it's &lt;code&gt;+x&lt;/code&gt;, and if that ever changes, the build gets hooked first.&lt;/p&gt;

</description>
      <category>git</category>
      <category>devops</category>
      <category>testing</category>
      <category>security</category>
    </item>
    <item>
      <title>HTTP 200, status 0, nothing wrong: teaching an API to tell empty from broken</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Sun, 15 Nov 2026 17:44:47 +0000</pubDate>
      <link>https://dev.to/ericgitangu/http-200-status-0-nothing-wrong-teaching-an-api-to-tell-empty-from-broken-3hh</link>
      <guid>https://dev.to/ericgitangu/http-200-status-0-nothing-wrong-teaching-an-api-to-tell-empty-from-broken-3hh</guid>
      <description>&lt;p&gt;I'm Eric Gitangu. I work on the backend of a mobility/fintech platform. This is one problem from the last two weeks: an API that answered "nothing nearby" and "the store is down" with the same shape, and how I taught it the difference without breaking a single installed app.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;The mobile API wraps every answer in an envelope. The HTTP status is 200 and a &lt;code&gt;status&lt;/code&gt; field inside says whether it worked: &lt;code&gt;1&lt;/code&gt; is success, &lt;code&gt;0&lt;/code&gt; is failure.&lt;/p&gt;

&lt;p&gt;That contract is older than most of the code around it, and the apps in people's pockets depend on it.&lt;/p&gt;

&lt;p&gt;A monitor watches for silent failures: 200 on the wire, status 0 in the body. When I read a few days of its log, a lot of what it had caught was not failure at all. A driver with no rides nearby. An empty queue. An empty document list. A setting nobody had configured yet. Each one answered status 0, because at some point "no results" was written down the error path.&lt;/p&gt;

&lt;p&gt;Three things follow from that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The app can't tell "nothing here" from "something broke".&lt;/strong&gt; Both arrive as failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The monitor counts ordinary days as failures.&lt;/strong&gt; An alert that pages on a normal state teaches people to ignore the alert.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A real fault hides in that noise.&lt;/strong&gt; Same shape, same count.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The decision
&lt;/h2&gt;

&lt;p&gt;The textbook fix is real HTTP status codes. Not available: installed builds read the envelope, and I can't recall an app from someone's phone.&lt;/p&gt;

&lt;p&gt;So the rules were:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No existing key, header or status changes for a client that did not ask for it.&lt;/li&gt;
&lt;li&gt;Status 0 splits into three classes: &lt;strong&gt;normal state&lt;/strong&gt; (nothing to show), &lt;strong&gt;refusal&lt;/strong&gt; (you may not, and here is why) and &lt;strong&gt;fault&lt;/strong&gt; (we broke).&lt;/li&gt;
&lt;li&gt;Every status-0 answer carries a stable, machine-readable code.&lt;/li&gt;
&lt;li&gt;Classification happens in one place, not at every call site.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  One choke point
&lt;/h2&gt;

&lt;p&gt;A hook sits at the last point before the response leaves. If the answer is JSON with status 0 and no code, it looks the message up in a catalog and adds a code and a reason. It only adds. It also publishes the code in a response header, so the access log and the monitor see it without parsing bodies.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;CATALOG&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"No rides available"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"no_results"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="s2"&gt;"normal"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Account not verified"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"unverified"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="s2"&gt;"refusal"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Could not read store"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"store_fault"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"fault"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;is_app_request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"error_code"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;route&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;ECHOES_ONE_TIME_CODE&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"code"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;CATALOG&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"unclassified"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unknown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;# add only: no key removed or changed&lt;/span&gt;
    &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;STABLE_CODES&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;   &lt;span class="c1"&gt;# header gets stable codes only&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A complete, runnable version with the baseline test is in &lt;a href="https://gist.github.com/ericgitangu/876b7ec7cf816d0f7b7e995a93650c75" rel="noopener noreferrer"&gt;this gist&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The monitor then counts only what is unclassified or uncoded. Everything else is a known answer with a name.&lt;/p&gt;

&lt;p&gt;To stop the pile growing back, a test walks the source for status-0 sites and compares them with a committed baseline. A new site with no code and no catalog entry fails the build. The debt can shrink. It cannot grow.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three things that bit
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The hook was too greedy.&lt;/strong&gt; Payment providers call routes on the same API, and a provider's callback body is the provider's contract, not mine. The classifier now applies only to requests that carry the app's session token header, and provider routes are excluded by name. That fix went in the same day as the hook.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;code&lt;/code&gt; was already taken.&lt;/strong&gt; Several endpoints return a one-time code in a field called &lt;code&gt;code&lt;/code&gt;, and installed builds read it as exactly that. Put a refusal code there and an installed build reads it as the one-time code. On those routes the refusal goes in &lt;code&gt;error_code&lt;/code&gt;, and the header copies only values from the stable list, so nothing that isn't a stable code reaches the access log through it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The version gate could never open.&lt;/strong&gt; The plan was to answer normal states with status 1 only for app builds at or above a minimum version, so older builds kept today's behaviour. I built it, tested it and merged it. Then it turned out the installed apps send no version. The gate compared against nothing and stayed shut.&lt;/p&gt;

&lt;p&gt;It was replaced by a per-route switch: a list of route keys, empty by default. A listed route answers status 1 with the same keys and empty values. An unlisted route answers exactly as before. Both carry the stable code.&lt;/p&gt;

&lt;p&gt;A feature flag keyed on a signal you have never seen in your own logs is a wish.&lt;/p&gt;

&lt;h2&gt;
  
  
  The telemetry that saw nothing
&lt;/h2&gt;

&lt;p&gt;The mobile telemetry was written from a hook that runs at request shutdown. That hook runs before the buffered response is flushed, so it never saw the body.&lt;/p&gt;

&lt;p&gt;It moved into the same output hook as the classifier and now records the envelope status and the code. The monitors count by code: on money routes an allowlist of normal-state codes is skipped, and anything unclassified, unknown or missing still counts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Empty is not unreadable
&lt;/h2&gt;

&lt;p&gt;One follow-up shows the pattern in small. The endpoint that lists a user's saved cards treated "this user has no cards" and "the card vault can't be read" alike.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No cards: status 1, an empty list, code &lt;code&gt;no_results&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Vault unreadable: status 0, code &lt;code&gt;unavailable&lt;/code&gt;, an honest message instead of the library's raw error text.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before changing it I read the client sources to see which keys they read and how they type them. No key was removed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is live, and what isn't
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The classifier, the catalog, the baseline test, the per-route switch, the telemetry change and the monitor changes are in production. So is the saved-cards split.&lt;/li&gt;
&lt;li&gt;The switch is per route and off by default. Turning a route on is a config decision made route by route, so I'm not claiming a number here.&lt;/li&gt;
&lt;li&gt;The next step is built and waiting in a release that is not in production yet: real faults on money paths answer a real 5xx (a store fault is a 503 with a retry hint) instead of 200 with status 0. The request still stops and nothing is written; only the HTTP answer changes. I'm not claiming that one until it ships.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Empty is a result. Treat it like one.&lt;/li&gt;
&lt;li&gt;Normal state, refusal and fault are three different answers. Give each a name a machine can read.&lt;/li&gt;
&lt;li&gt;Classify at one choke point, then fence it with a baseline so new unnamed failures fail the build.&lt;/li&gt;
&lt;li&gt;Before you gate on a client signal, check the client actually sends it.&lt;/li&gt;
&lt;li&gt;Read the client code before you touch a field. &lt;code&gt;code&lt;/code&gt; meant two things and the apps only knew one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;200 OK with status 0 is Schrödinger's response: you don't know if it's dead until you open the body. Better to write the answer on the box.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://developer.ericgitangu.com/blog/http-200-status-0-empty-is-not-an-error" rel="noopener noreferrer"&gt;developer.ericgitangu.com&lt;/a&gt;. More engineering write-ups on the &lt;a href="https://developer.ericgitangu.com/blog" rel="noopener noreferrer"&gt;blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>architecture</category>
      <category>monitoring</category>
      <category>backend</category>
    </item>
    <item>
      <title>The server had files git never met: fixing config drift with a pinned manifest</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Sat, 14 Nov 2026 14:10:20 +0000</pubDate>
      <link>https://dev.to/ericgitangu/the-server-had-files-git-never-met-fixing-config-drift-with-a-pinned-manifest-io9</link>
      <guid>https://dev.to/ericgitangu/the-server-had-files-git-never-met-fixing-config-drift-with-a-pinned-manifest-io9</guid>
      <description>&lt;p&gt;&lt;em&gt;By Eric Gitangu. Originally published at &lt;a href="https://developer.ericgitangu.com/blog/server-config-drift-git-manifest" rel="noopener noreferrer"&gt;developer.ericgitangu.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I work on the backend of a mobility/fintech platform. This is one problem from last week: configuration drift on a production server, and how I made git the record without touching the server.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Every deploy script we had installed something: application code, edge config, scheduled jobs. The server also held files that no script installed. Monitoring config and alert rules. A few scheduled jobs. Service units. Login hardening. Privilege rules. Each one was put there by hand, for a good reason, by someone who then went to lunch.&lt;/p&gt;

&lt;p&gt;Those files were a large part of the hardening, and they lived in exactly one place.&lt;/p&gt;

&lt;p&gt;That gives you two quiet ways to lose them:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Rebuild the server.&lt;/strong&gt; Everything the scripts install comes back. Everything else does not, and nothing tells you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Copy repo to server.&lt;/strong&gt; Where the repo holds an older copy of a file, the copy reverts a live fix.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Neither fails loudly. You find out when the alert that should have fired didn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  The decision
&lt;/h2&gt;

&lt;p&gt;The tempting fix is to write installers for everything and re-apply from git. That changes a production server to solve a record-keeping problem, and it assumes git knows better than the machine that is actually serving traffic. It didn't.&lt;/p&gt;

&lt;p&gt;So I went the other way. The server is right today; git catches up. Server to repo, read-only, nothing changed on the server.&lt;/p&gt;

&lt;h2&gt;
  
  
  The manifest
&lt;/h2&gt;

&lt;p&gt;Each server-only file is mirrored into the repo and gets one row in a manifest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# server_path          repo_path                 owner group mode sha256_of_live_file  status
/etc/example/app.conf  ops/mirror/app.conf       root  root  644  3f9a...c1            verbatim
/etc/example/db.conf   ops/mirror/db.conf        root  root  600  b07e...9d            redacted
/usr/local/bin/probe   ops/mirror/probe          root  root  755  91c4...e2            repo-ahead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The hash is always the hash of the &lt;strong&gt;live&lt;/strong&gt; file. The status says how the repo copy relates to it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;verbatim&lt;/strong&gt;: the repo copy must hash to the live hash. Byte for byte.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;redacted&lt;/strong&gt;: the file holds a secret, so the repo copy carries a placeholder. Here the repo copy must &lt;strong&gt;not&lt;/strong&gt; hash like the live file. If it matches, someone committed the secret. Same check, opposite sign.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;repo-ahead&lt;/strong&gt;: the repo carries a change that isn't installed yet. The row still pins what is live today, so any further edit on the server still shows as drift. Install it, flip the row to verbatim.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That third status matters more than it looks. Without it, the first file that is legitimately mid-change forces you to either lie in the manifest or switch the check off.&lt;/p&gt;

&lt;h2&gt;
  
  
  One check, two modes
&lt;/h2&gt;

&lt;p&gt;One read-only script reads the manifest. It never writes anywhere.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Against a checkout&lt;/strong&gt; (CI, pre-push, tests): every repo copy exists and relates to its pinned hash the way its status says.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On the server&lt;/strong&gt;: every live file exists, has the pinned owner, group and mode, and hashes to the pinned hash.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It names every finding instead of dying on the first one. A missing file reads as "a rebuild lost this, reinstall from the repo copy". A changed hash reads as "someone edited the server, re-mirror and bump the row".&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check_server&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;findings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;server_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"MISSING {p}: reinstall from {r.repo_path}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;st&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;oct&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_mode&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="mo"&gt;0o7777&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"PERMS   {p}: want {r.owner} {r.group} {r.mode}, got {' '.join(got)}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;live_hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"DRIFT   {p}: changed since it was mirrored"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;findings&lt;/span&gt;  &lt;span class="c1"&gt;# all of them, not the first&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A complete, runnable version of the check with a sample manifest is in &lt;a href="https://gist.github.com/ericgitangu/e3e247db8ee66209107593bced39c330" rel="noopener noreferrer"&gt;this gist&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The first run on the production server passed. A release later, a few of those files changed through the deploy, and the hashes were re-pinned in a commit. That is the whole workflow: a change to the server is either a commit or an incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making it bite
&lt;/h2&gt;

&lt;p&gt;A check nobody runs is a README. So it went into the path that already has everyone's attention.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The deploy refuses to start while the server differs from git.&lt;/strong&gt; Every other pre-deploy gate reasons from the repo. If the server isn't what the repo says, those gates are describing a different machine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;There is an override, because 3am exists.&lt;/strong&gt; It needs a written reason. It is logged and it raises an alert. And if the record can't be written, the override is refused. No unrecorded bypass.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Detection between deploys.&lt;/strong&gt; Kernel audit rules watch writes by human sessions to the application and config directories. The deploy tooling records its own windows, so its writes are subtracted. What's left is exported as metrics with alerts on top, and a full check runs nightly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two fixes the same night
&lt;/h2&gt;

&lt;p&gt;The audit scan was rewritten to filter in a single pass, so a long release build can't make it outlast its own timeout. A drift detector that times out during deploys is a detector with a blind spot exactly where the writes are.&lt;/p&gt;

&lt;p&gt;And the gate originally said "drift" without saying which. Now it names what it is refusing on. A gate that says no without saying why gets overridden; that's not a people problem, it's an error-message problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is live, and what isn't
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The mirror, manifest and check are in the repo. The server-side check ran read-only on production and passed.&lt;/li&gt;
&lt;li&gt;The deploy gate and the detection code are in the release that is now live in production.&lt;/li&gt;
&lt;li&gt;A prevention layer that tightens who can change the server, and how, is built, tested and rehearsed in dry-run. It gets applied only with a person present, so I'm not claiming it here. Locking yourself out of production is also a kind of drift.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;If no script installs a file, that file is a rumour. Write it down with a hash.&lt;/li&gt;
&lt;li&gt;Record first, enforce second. Git has to be right before it gets to be the boss.&lt;/li&gt;
&lt;li&gt;Give "mid-change" a legal state, or people will switch the check off.&lt;/li&gt;
&lt;li&gt;For redacted files, a matching hash is the failure.&lt;/li&gt;
&lt;li&gt;An override should cost one sentence of explanation and always leave a trace.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Drift isn't a bug you fix once. It's entropy with sudo. The best you can do is make it loud.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>security</category>
      <category>sre</category>
      <category>linux</category>
    </item>
    <item>
      <title>Money paths that fail closed: what I changed on a legacy payments platform</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Fri, 06 Nov 2026 07:49:24 +0000</pubDate>
      <link>https://dev.to/ericgitangu/money-paths-that-fail-closed-what-i-changed-on-a-legacy-payments-platform-4hm4</link>
      <guid>https://dev.to/ericgitangu/money-paths-that-fail-closed-what-i-changed-on-a-legacy-payments-platform-4hm4</guid>
      <description>&lt;p&gt;I own architecture and security for a payments and mobility platform in East Africa. Three markets, five payment rails (three mobile-money providers, two card processors), a PHP monolith that predates me, and MongoDB as the system of record. Not the stack I would pick on a blank page, but a bank-issued card and real wallets run on it, so the job was to make the money paths boring before anything else.&lt;/p&gt;

&lt;p&gt;This is what that looked like. No company names, and I have kept to what is actually shipped. The open items are at the end.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule one: claim before effect
&lt;/h2&gt;

&lt;p&gt;Every money operation claims a unique, one-time key before it does anything. A second attempt with the same key is refused: no second charge, no second ledger entry. The primitive is small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;key = idem.key("outstanding_deduct", trans_id)   # deterministic, namespaced
if key == "":  legacy_path()                     # unsafe/empty input -&amp;gt; old behaviour, logged
try:   idem.claim(key)                           # insert into a dedicated collection,
                                                 # unique partial index on key
except DuplicateKey:                             # Mongo 11000 = someone already did this
       return ok_noop                            # ack, do nothing
deduct(trans_id)                                 # the effect comes strictly after the claim
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is wired across roughly 45 money operations. Different sites use whichever dedupe fits: an atomic status compare-and-set (Paid -&amp;gt; Syncing before a rider credit), a unique insert on a repay key for mobile-money callbacks, the provider event id for webhooks, and the processor's own idempotency header on live card top-ups.&lt;/p&gt;

&lt;p&gt;The subtle decisions are not in the happy path:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Duplicate = refuse.&lt;/strong&gt; That is the fail-closed part and it is non-negotiable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Claim write itself fails (infra fault) = let the money through, log loudly.&lt;/strong&gt; We chose not to block legitimate payments because the idempotency store had a bad minute. That is fail-open for one specific fault and it is documented as such.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Crash after the claim, before the effect = the operation is un-retryable.&lt;/strong&gt; It lands in a logged reconcile queue instead of a retry loop. The window is narrow; it is not zero.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The P0 we found while wiring this: a shared driver-debt deduction path had no idempotency at all on the non-primary rails. One guard now covers every rail.&lt;/p&gt;

&lt;p&gt;Proof was cheap and worth doing: eight unit tests pin the key contract, a review confirmed claim-before-effect ordering at every site, and a live gate test showed the first insert winning and the duplicate returning 11000 with exactly one row left behind.&lt;/p&gt;

&lt;h2&gt;
  
  
  Callbacks: never trust the body
&lt;/h2&gt;

&lt;p&gt;Mobile-money providers call you back. Rules we enforce on that path:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Callbacks are accepted only from the provider's network (IP allow-list). The bank rail's signature check is built and waiting on production values.&lt;/li&gt;
&lt;li&gt;On confirmation we credit the &lt;strong&gt;stored&lt;/strong&gt; push amount, never the amount in the callback body. Body-vs-stored mismatch is rejected and logged.&lt;/li&gt;
&lt;li&gt;A re-delivered success is a no-op that still returns an ACK, so the gateway stops retrying.&lt;/li&gt;
&lt;li&gt;Card webhooks are signature-verified; an invalid signature is rejected. Hosted-checkout returns are verified server-side with the provider before any credit, and a background reconciler sweeps the credits the return call missed.&lt;/li&gt;
&lt;li&gt;Repeated fraud declines increment a counter and switch card top-up off past a threshold.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The callback recorder
&lt;/h2&gt;

&lt;p&gt;On 1 November an unmatched callback was lost. We could not say what the bank had sent. The fix went live the next morning: every payment callback from the bank partner leaves a redacted row, matched or not. Ids, result code, amount, receipt, last four digits of the phone, a &lt;code&gt;matched_pending&lt;/code&gt; flag, the action taken and why.&lt;/p&gt;

&lt;p&gt;Two design points. The recorder runs &lt;strong&gt;after&lt;/strong&gt; the 403 gate, so it never records unauthenticated noise. And its write is isolated from the decision, so a recorder fault cannot change the reply to the bank. The failure conditions are explicit: no row, a row with a full phone number or any card number, a credit without a row, or a 500 on the callback path. An admin health view shows the services up or down, the event table, and Force/Hide actions with a 14-day purge; a concurrent Force gets a 409, and only root can grant the Force right.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the money audit found
&lt;/h2&gt;

&lt;p&gt;Four live defects, all shipped fixes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A wallet double-count, fixed with an atomic compare-and-set.&lt;/li&gt;
&lt;li&gt;An admin adjustment that could push a wallet negative.&lt;/li&gt;
&lt;li&gt;A top-up race.&lt;/li&gt;
&lt;li&gt;A double-send.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Debit, credit and ledger entry now commit in one multi-document transaction on the critical paths, on a three-node replica set with majority-acknowledged writes. Every top-up, payout and card charge also appends a de-duplicated, best-effort audit event that never rolls back the money; the audit log is hash-chained and strips PAN, card, CVV and OTP keys on the way in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Releases that refuse to be hand-copied
&lt;/h2&gt;

&lt;p&gt;The incident that justified the release gate: a payment cron job was dead for nine days because a load-time compatibility fault was swallowed silently. A separate hand-copy to the live host left a stale dependency map and more than a thousand stray files.&lt;/p&gt;

&lt;p&gt;The live release path now:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Compile and load every module in a sandbox.&lt;/li&gt;
&lt;li&gt;A separate sign-off flag for any change touching money code (&lt;code&gt;MONEY_RELEASE=yes&lt;/code&gt;, set by a human).&lt;/li&gt;
&lt;li&gt;Build the new version alongside the running one and switch in a single instant.&lt;/li&gt;
&lt;li&gt;Automatic post-release health checks.&lt;/li&gt;
&lt;li&gt;Automatic rollback if a check definitively fails; the last several releases stay on disk for an instant manual revert.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Hand-copied code is refused; only releases built from the shared repository are accepted. Access is SSH key-only, elevation is scoped to release actions, and every use is logged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping card data out of the building
&lt;/h2&gt;

&lt;p&gt;Card entry happens in the processors' hosted sessions and hosted fields. The platform receives a session id or a token and keeps token id, last four, expiry and scheme. Payment pages have no card inputs, load one allow-listed third-party script, and relay results by &lt;code&gt;postMessage&lt;/code&gt; to an allow-listed origin. Retired card-handling endpoints return 410.&lt;/p&gt;

&lt;p&gt;We checked rather than asserted: a scan of all 218 collections found no CVV value and no card-number field; a scan of 79 log files (3.4 million lines, 15 days) found zero Luhn-valid numbers. The one exception, three 2017-era payout records with card-shaped values in a bank-account field, was masked to last four. Request bodies are not logged, phone numbers are redacted in callback paths, the bank OAuth token cache is sealed, and backups are encrypted with the key held offline.&lt;/p&gt;

&lt;p&gt;Honest caveat: one issuer card-reveal flow passes clear PAN and CVV through the platform for a single request. It is gated by KYC, a 300-second single-use OTP token, ten reveals per hour, &lt;code&gt;no-store&lt;/code&gt;, and an audit write that must succeed before the reveal is returned. Whether that stays or moves to an issuer-hosted view is an open decision with the bank and the assessor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seeing the quiet failures
&lt;/h2&gt;

&lt;p&gt;Prometheus, Grafana with Alertmanager, Loki for logs, Tempo traces through an OpenTelemetry collector, exporters for host, database and web server. The piece I would carry to any platform is the silent-failure detector: it classifies response envelopes so that a money or auth failure returning a 200 still raises an alert. The legacy system had none of this.&lt;/p&gt;

&lt;h2&gt;
  
  
  Still open
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Move the money-critical surfaces (wallet, ledger, settlements, card transactions) to PostgreSQL for constraints and versioned migrations; Mongo keeps rides, location and telemetry.&lt;/li&gt;
&lt;li&gt;A balance-reconciliation job, and re-queuing a handful of stalled ledger fan-out rows.&lt;/li&gt;
&lt;li&gt;Enforce MFA for every admin (available, not yet enforced everywhere), central write-once logs with 12-month retention, file-integrity monitoring, a Luhn-checked log scrubber, an external pen test, and the assessor-led review.&lt;/li&gt;
&lt;li&gt;The idempotency collection grows without bound. It is tiny; a TTL is a future option, not a problem today.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A question for you
&lt;/h2&gt;

&lt;p&gt;When the idempotency store itself is down, does your money path fail open or fail closed? We chose open for that one fault and closed for everything else. I would like to hear where others drew the line, and what it cost them.&lt;/p&gt;




&lt;p&gt;Eric Gitangu, Nairobi. Site: &lt;a href="https://developer.ericgitangu.com" rel="noopener noreferrer"&gt;https://developer.ericgitangu.com&lt;/a&gt;. LinkedIn: &lt;a href="https://www.linkedin.com/in/ericgitangu" rel="noopener noreferrer"&gt;https://www.linkedin.com/in/ericgitangu&lt;/a&gt;. GitHub: &lt;a href="https://github.com/ericgitangu" rel="noopener noreferrer"&gt;https://github.com/ericgitangu&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>fintech</category>
      <category>architecture</category>
      <category>security</category>
      <category>backend</category>
    </item>
    <item>
      <title>No double booking, enforced by Postgres: building a clinic booking system</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Wed, 28 Oct 2026 18:53:04 +0000</pubDate>
      <link>https://dev.to/ericgitangu/no-double-booking-enforced-by-postgres-building-a-clinic-booking-system-5ddl</link>
      <guid>https://dev.to/ericgitangu/no-double-booking-enforced-by-postgres-building-a-clinic-booking-system-5ddl</guid>
      <description>&lt;p&gt;Two patients click the same 10:30 slot at the same instant. Both requests check whether the slot is free. Both see "free". Both insert. You now have a double booking, and no amount of careful application code fixes it, because check-then-insert has a window in the middle by construction.&lt;/p&gt;

&lt;p&gt;I built a clinic appointment system, Idyllic Clinic, as a personal project and it grew well past what I first planned. The headline requirement stayed the same throughout: &lt;strong&gt;a booked slot is never offered or booked twice.&lt;/strong&gt; This post is about how I made the database, not the application, the thing that guarantees it.&lt;/p&gt;

&lt;p&gt;Live: &lt;a href="https://clinic.ericgitangu.com" rel="noopener noreferrer"&gt;clinic.ericgitangu.com&lt;/a&gt;. Every decision below has an entry, including its cost, at &lt;a href="https://clinic.ericgitangu.com/decisions" rel="noopener noreferrer"&gt;clinic.ericgitangu.com/decisions&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt; is not enough
&lt;/h2&gt;

&lt;p&gt;The usual answer is "lock the row". But slots here are not stored. Doctors have working hours, a slot is 30 minutes, and free slots are computed on request: cut each working window into 30-minute steps, subtract the booked starts, drop anything inside the one-hour lead time. There is no slot row to lock, and you cannot lock a row that does not exist yet.&lt;/p&gt;

&lt;p&gt;So uniqueness has to live on the appointment itself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;uq_appt_doctor_slot_booked&lt;/span&gt;
  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;appointments&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;doctor_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;starts_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'booked'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The service just inserts and maps the unique violation (SQLSTATE 23505, matched by constraint name) to &lt;code&gt;409 SLOT_TAKEN&lt;/code&gt;. There is deliberately &lt;strong&gt;no "is it taken?" pre-check&lt;/strong&gt;: any such check is a race window and adds nothing the index does not already enforce. Because the index is partial, a cancelled row drops out of it, so the slot frees up while the history stays. A second index on &lt;code&gt;(patient_id, starts_at)&lt;/code&gt; stops one patient holding two slots at the same instant.&lt;/p&gt;

&lt;p&gt;The trade-off, which I state on the decisions page: this is correct only for fixed, aligned 30-minute slots. Variable-length appointments would mean migrating to an &lt;code&gt;EXCLUDE USING gist&lt;/code&gt; constraint.&lt;/p&gt;

&lt;h2&gt;
  
  
  A race test that cannot pass vacuously
&lt;/h2&gt;

&lt;p&gt;A concurrency test that passes because the requests happened to serialize is worse than no test. So the harness is built to fail if it stops being concurrent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;25 POSTs for one slot are released at an &lt;code&gt;asyncio.Barrier&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A test-only &lt;code&gt;pg_sleep&lt;/code&gt; holds the transaction open for 200 ms while the test samples &lt;code&gt;pg_locks&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;It repeats over 10 slots with a pool large enough for all 25 requests to be in flight, and asserts the peak number of transactions blocked on the uniqueness check is at least 5.&lt;/li&gt;
&lt;li&gt;Per slot it asserts exactly one &lt;code&gt;201&lt;/code&gt;, twenty-four &lt;code&gt;409 SLOT_TAKEN&lt;/code&gt;, and one booked row.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then a control: the same kind of barrier against a database &lt;strong&gt;with the index dropped&lt;/strong&gt;, using a naive check-then-insert, asserting that more than one row gets booked. That proves the index is what holds, not luck. Production behaves the same way: 20 simultaneous requests for one slot returned one &lt;code&gt;201&lt;/code&gt; and nineteen &lt;code&gt;409&lt;/code&gt;s. The README has a copy-paste script to reproduce that against the live API.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;API:&lt;/strong&gt; FastAPI on Cloud Run, the single system of record. Every booking rule, idempotency key and RFC 9457 &lt;code&gt;problem+json&lt;/code&gt; error lives here. It is IAM-only; a direct request gets a 403.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database:&lt;/strong&gt; Neon Postgres 17. Migrations run as a &lt;code&gt;clinic_owner&lt;/code&gt; role in a Cloud Run Job before each deploy; the service runs as &lt;code&gt;clinic_app&lt;/code&gt; with no DDL rights. Migrations are expand-only so the previous revision keeps working.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Web:&lt;/strong&gt; two Next.js apps on Vercel (patient booking and an admin dashboard) in one pnpm workspace. They are thin BFF layers that reach the API by exchanging Vercel's OIDC token through workload identity federation for a Cloud Run ID token. No credential in the browser, no stored key.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Live updates:&lt;/strong&gt; slot events go through an outbox relay to Ably after each commit, with polling as the floor, so a missed push costs at most a minute of staleness and never a wrong booking. Staff events carry patient names, so they use a small read-only Go service on Cloud Run (&lt;code&gt;SELECT&lt;/code&gt;-only database role) instead of a third-party pub/sub.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Types:&lt;/strong&gt; OpenAPI 3.1 contract, with TypeScript types, zod schemas and an SDK generated from it. CI fails if generated files drift.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some other details I care about: time is stored as UTC &lt;code&gt;timestamptz&lt;/code&gt; with the clinic zone as data, naive datetimes are rejected with &lt;code&gt;422 TIMEZONE_REQUIRED&lt;/code&gt;, and the clock is injected so tests pin time and pass on any weekday. Booking retries replay through an idempotency key, so a lost response does not tell the real booker they lost the slot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stack
&lt;/h2&gt;

&lt;p&gt;Python 3.12, FastAPI, SQLAlchemy (async) with asyncpg, Alembic, Next.js 16, React 19, TypeScript, Better Auth, Go with pgx, Postgres 17 on Neon, Terraform on GCP, Vercel, CircleCI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Testing, CI and deploy
&lt;/h2&gt;

&lt;p&gt;Backend tests run against real Postgres 17 with no mocked repositories. The &lt;code&gt;e2e-real&lt;/code&gt; CI job brings up the real API and database with Docker Compose and runs Playwright scenarios against them, including double booking and cancelling twice. The backend job fails under 85% coverage, with a separate gate holding the pure domain code at 95%. Schemathesis, Lighthouse CI, axe-core, gitleaks, Semgrep, Trivy and OWASP ZAP are wired in as gates.&lt;/p&gt;

&lt;p&gt;Delivery is CircleCI with keyless OIDC to workload identity federation: a tagged no-traffic Cloud Run revision, a smoke test, promotion, and rollback automation. I should be upfront that no rollback has actually had to fire yet; the traffic move is covered by CI script tests, not a real incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I did not build
&lt;/h2&gt;

&lt;p&gt;I would rather you hear the gaps from me. The public API is unauthenticated so Swagger's Try it out works, which means anyone holding an appointment UUID can change it through that path; the signed-in web screens close this, the raw API does not. There is no rate limiting inside the API, only in the Vercel proxy. The &lt;code&gt;/book&lt;/code&gt; page misses the 2.5 s mobile LCP budget (2.80 s measured locally). The 10x and 100x scaling notes are deductions from configuration, not load tests, and are labelled as such.&lt;/p&gt;

&lt;p&gt;Visit my site for my other DIY pet-projects at developer.ericgitangu.com and the corresponding repo &lt;a href="https://github.com/ericgitangu" rel="noopener noreferrer"&gt;https://github.com/ericgitangu&lt;/a&gt; - happy coding 💻🥷&lt;/p&gt;

&lt;h2&gt;
  
  
  Take a look
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Live app: &lt;a href="https://clinic.ericgitangu.com" rel="noopener noreferrer"&gt;clinic.ericgitangu.com&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Every decision and its cost: &lt;a href="https://clinic.ericgitangu.com/decisions" rel="noopener noreferrer"&gt;clinic.ericgitangu.com/decisions&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code: &lt;a href="https://github.com/ericgitangu/clinic-booking" rel="noopener noreferrer"&gt;github.com/ericgitangu/clinic-booking&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;More of my work: &lt;a href="https://developer.ericgitangu.com" rel="noopener noreferrer"&gt;developer.ericgitangu.com&lt;/a&gt;, &lt;a href="https://github.com/ericgitangu" rel="noopener noreferrer"&gt;github.com/ericgitangu&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you have solved the variable-length slot case with &lt;code&gt;EXCLUDE USING gist&lt;/code&gt; in production, I would like to hear how it went.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>python</category>
      <category>softwareengineering</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Your Next.js App Router pages aren't indexed? Check the canonical you didn't set</title>
      <dc:creator>Eric Gitangu</dc:creator>
      <pubDate>Tue, 27 Oct 2026 19:59:45 +0000</pubDate>
      <link>https://dev.to/ericgitangu/your-nextjs-app-router-pages-arent-indexed-check-the-canonical-you-didnt-set-2kle</link>
      <guid>https://dev.to/ericgitangu/your-nextjs-app-router-pages-arent-indexed-check-the-canonical-you-didnt-set-2kle</guid>
      <description>&lt;p&gt;I relaunched my portfolio on the Next.js App Router, submitted the sitemap, and waited. A week later Google Search Console still said &lt;strong&gt;0 pages indexed&lt;/strong&gt; — everything sat in &lt;em&gt;"Crawled – currently not indexed."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;My first instinct was the usual: time and domain authority. Both are real for a new site. But before you blame Google, it's worth ruling out the self-inflicted version — because in my case there &lt;em&gt;was&lt;/em&gt; one, and it's a trap that's very easy to fall into with the App Router's metadata API.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gotcha: an inherited canonical
&lt;/h2&gt;

&lt;p&gt;In the App Router you set metadata in &lt;code&gt;layout.tsx&lt;/code&gt; or &lt;code&gt;page.tsx&lt;/code&gt;. It's tempting to put your canonical URL once, in the &lt;strong&gt;root layout&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/layout.tsx — DON'T do this&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Metadata&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;metadataBase&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://example.com&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;alternates&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;canonical&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Metadata &lt;strong&gt;inherits&lt;/strong&gt;. So every route under that layout — &lt;code&gt;/about&lt;/code&gt;, &lt;code&gt;/projects&lt;/code&gt;, every post — renders &lt;code&gt;&amp;lt;link rel="canonical" href="https://example.com/"&amp;gt;&lt;/code&gt;. You've just told Google that every page on your site is a duplicate of the homepage. Google obliges and indexes… the homepage, maybe.&lt;/p&gt;

&lt;p&gt;The fix is a per-route canonical. A tiny helper keeps it honest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/lib/seo.ts&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;Metadata&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;next&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;pageMetadata&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="p"&gt;}:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}):&lt;/span&gt; &lt;span class="nx"&gt;Metadata&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;alternates&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;canonical&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="c1"&gt;// self-referencing, per route&lt;/span&gt;
    &lt;span class="na"&gt;openGraph&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/about/layout.tsx&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;metadata&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;pageMetadata&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;About&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;…&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/about&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now each route points at itself. Verify it the way Google does — inspect a URL in Search Console and check that &lt;strong&gt;Google-selected canonical&lt;/strong&gt; matches your &lt;strong&gt;user-declared canonical&lt;/strong&gt;. If they disagree, nothing else you do will get that page indexed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three more App Router SEO fixes that actually moved the needle
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Noindex your preview/alias domains.&lt;/strong&gt; Vercel gives every deployment a &lt;code&gt;*.vercel.app&lt;/code&gt; URL, and they're all live and crawlable. Left alone, they compete with your real domain as duplicates. Send a noindex header to every host except the canonical one — no middleware needed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// next.config.mjs&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nf"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
    &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/:path*&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;missing&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;host&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;example.com&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;}],&lt;/span&gt;
    &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;X-Robots-Tag&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;noindex, nofollow&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;}],&lt;/span&gt;
  &lt;span class="p"&gt;}];&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Self-host your Open Graph image.&lt;/strong&gt; I'd been pointing &lt;code&gt;og:image&lt;/code&gt; at a LinkedIn CDN URL — which expires and isn't fetchable by some scrapers (WhatsApp never rendered it). Generate it at the edge instead with &lt;code&gt;next/og&lt;/code&gt;, so it's a real static asset at &lt;code&gt;/opengraph-image&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/opengraph-image.tsx&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;ImageResponse&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;next/server&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;630&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;OG&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ImageResponse&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="cm"&gt;/* your JSX */&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;size&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;3. Add real structured data.&lt;/strong&gt; A &lt;code&gt;Person&lt;/code&gt; + &lt;code&gt;BreadcrumbList&lt;/code&gt; JSON-LD block in your layout gives Google an entity to attach your name to. It's cheap and it's one of the few signals you fully control on a new site.&lt;/p&gt;

&lt;h2&gt;
  
  
  And the part that &lt;em&gt;is&lt;/em&gt; just time
&lt;/h2&gt;

&lt;p&gt;After all that, some pages still read &lt;em&gt;"Crawled – currently not indexed,"&lt;/em&gt; and that's the honest truth about a new site: &lt;strong&gt;crawl ≠ index.&lt;/strong&gt; Google crawled them fine (fetch successful, indexing allowed) and simply hasn't decided they're worth storing yet. The levers that change its mind are mundane — a sitemap it trusts, a handful of referring links from pages it already crawls often, internally-linked hub pages, and a few weeks. "Request indexing" nudges a specific URL; it doesn't buy authority.&lt;/p&gt;

&lt;p&gt;So: fix the canonical first (that's the bug), then be patient about the rest (that's the physics).&lt;/p&gt;




&lt;p&gt;The code above is live on my site — &lt;a href="https://developer.ericgitangu.com" rel="noopener noreferrer"&gt;developer.ericgitangu.com&lt;/a&gt; — and if you want to see the per-page canonicals and JSON-LD in the wild, the &lt;a href="https://developer.ericgitangu.com/projects" rel="noopener noreferrer"&gt;projects section&lt;/a&gt; renders a &lt;code&gt;SoftwareSourceCode&lt;/code&gt; block per entry. I'm Eric Gitangu, a software engineer working on distributed and fintech systems; happy to compare notes in the comments.&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>seo</category>
      <category>webdev</category>
      <category>react</category>
    </item>
  </channel>
</rss>
