<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Zanicar</title>
    <description>The latest articles on DEV Community by Zanicar (@zanicar).</description>
    <link>https://dev.to/zanicar</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3511747%2F0b3800a2-7e79-44ab-ac8e-2c4367ed12a8.png</url>
      <title>DEV Community: Zanicar</title>
      <link>https://dev.to/zanicar</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/zanicar"/>
    <language>en</language>
    <item>
      <title>How I Built a 30M req/s Zero-Alloc IAM Engine in Go (And Hit the RWMutex Wall)</title>
      <dc:creator>Zanicar</dc:creator>
      <pubDate>Tue, 17 Nov 2026 09:30:38 +0000</pubDate>
      <link>https://dev.to/zanicar/how-i-built-a-30m-reqs-zero-alloc-iam-engine-in-go-and-hit-the-rwmutex-wall-243o</link>
      <guid>https://dev.to/zanicar/how-i-built-a-30m-reqs-zero-alloc-iam-engine-in-go-and-hit-the-rwmutex-wall-243o</guid>
      <description>&lt;p&gt;When building Identity &amp;amp; Access Management (IAM) systems, authentication and authorization sit directly on the hot path of every incoming request. Over the past decade—across three or four complete from-scratch rewrites—I have been refining &lt;strong&gt;VK Auth&lt;/strong&gt;, an idiomatic Go IAM engine built around zero-allocation read paths, minimal memory overhead, and deterministic tail latencies.&lt;/p&gt;

&lt;p&gt;After benchmarking both the token verification engine (&lt;code&gt;controller/&lt;/code&gt;) and the authorization tree (&lt;code&gt;graph/&lt;/code&gt;) across millions of users and memberships on a Ryzen 7950X, we hit an interesting systems ceiling: &lt;strong&gt;graph traversal became so fast that Go’s &lt;code&gt;sync.RWMutex&lt;/code&gt; atomic reader counter now accounts for over half of saturated CPU.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here is a breakdown of the numbers, why we explicitly rejected lock-free copy-on-write in favor of state-delta encoding (&lt;code&gt;adr-017&lt;/code&gt;), and how we solved phased tree updates without breaking invariants.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Authentication (&lt;code&gt;controller/&lt;/code&gt;): Up to 30.2M Verifications/s
&lt;/h2&gt;

&lt;p&gt;In the first benchmark suite, &lt;code&gt;controller/&lt;/code&gt; handles token-session verification across both pure in-memory stores and a disk-backed Badger store, with and without observability enabled:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqc2opfj4nh9vkzrn0b0w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqc2opfj4nh9vkzrn0b0w.png" alt="controller: verifying a session" width="800" height="371"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Peak Throughput:&lt;/strong&gt; &lt;strong&gt;30.2M verifications/s&lt;/strong&gt; at 10k users (&lt;code&gt;1.66 µs&lt;/code&gt; p99) and &lt;strong&gt;28.1M verifications/s&lt;/strong&gt; at 1,000,000 users (&lt;code&gt;2.30 µs&lt;/code&gt; p99).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disk-Backed Parity:&lt;/strong&gt; Running 1,000,000 observed users on &lt;strong&gt;Badger on disk&lt;/strong&gt; sustains &lt;strong&gt;27.4M verifications/s&lt;/strong&gt; at the exact same &lt;strong&gt;2.30 µs&lt;/strong&gt; saturated p99, while shrinking heap usage after GC from 3.2 GB down to &lt;strong&gt;2.1 GB&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scale Headroom:&lt;/strong&gt; Even at 2,600,000 users, saturated throughput holds at &lt;strong&gt;26.1M verifications/s&lt;/strong&gt; (&lt;code&gt;2.56 µs&lt;/code&gt; p99).&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  2. Authorization (&lt;code&gt;graph/&lt;/code&gt;): 24.3M Answers/s &amp;amp; Zero Read Allocs
&lt;/h2&gt;

&lt;p&gt;Unlike &lt;code&gt;controller/&lt;/code&gt;, &lt;code&gt;graph/&lt;/code&gt; serves authorization queries directly from an in-memory tree backed by a &lt;strong&gt;publish-then-write&lt;/strong&gt; persistence mechanism. Rather than testing toy microbenchmarks, we generated realistic organizational topologies—ranging from multinational enterprises and universities to dense chat/guild graphs:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F10zirunf6ylgvtwis9po.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F10zirunf6ylgvtwis9po.png" alt="graph: answering an authorization at the target" width="800" height="152"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Zero Read-Path Allocations:&lt;/strong&gt; Both scoped and unscoped authorization checks execute with &lt;strong&gt;0 heap allocations&lt;/strong&gt;, keeping garbage collection off the hot path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Target Throughput:&lt;/strong&gt; &lt;strong&gt;24.1M – 25.2M answers/s&lt;/strong&gt; saturated across Multinational, Education, and Guild targets at a flat &lt;strong&gt;2.30 µs&lt;/strong&gt; saturated p99.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disk-Backed Durability:&lt;/strong&gt; Backing the Multinational target with a 2.2 GB Badger store still answers &lt;strong&gt;24.3M queries/s&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Even when pushing the tree into heavy stress ranges—scaling up to &lt;strong&gt;100,611 groups&lt;/strong&gt; (Guild) or &lt;strong&gt;1,000,000 groups and 11.45M memberships&lt;/strong&gt; (Chat)—latency degrades gracefully:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fehpz2hl3o57geupx02ak.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fehpz2hl3o57geupx02ak.png" alt="Below the target, and the stress ranges" width="799" height="220"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Why We Rejected Lock-Free Pointer Swaps (&lt;code&gt;ADR-017&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;When you vary tree depth (climbing 0 to 12 guardian levels beneath a membership) or branching factor (from 2 to 128), throughput barely moves—dropping only 4–7% across a 12-level climb.&lt;/p&gt;

&lt;p&gt;The reason is hardware-level lock contention:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Under saturated concurrent reads on a 16-core machine (32 threads), &lt;strong&gt;more than half of the CPU time on the Multinational target (and nearly 90% on the smallest trees) is spent inside &lt;code&gt;sync.RWMutex&lt;/code&gt;'s shared reader counter&lt;/strong&gt;—cache-line bouncing across cores—rather than traversing the tree itself.&lt;/li&gt;
&lt;li&gt;A common reflex in Go is to eliminate &lt;code&gt;RWMutex&lt;/code&gt; via &lt;strong&gt;lock-free atomic pointer swaps&lt;/strong&gt; (&lt;code&gt;ADR-010&lt;/code&gt;'s original model). In &lt;code&gt;ADR-017&lt;/code&gt;, we explicitly rejected both copy-based alternatives:

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Whole-tree rebuilds (&lt;code&gt;ADR-010&lt;/code&gt;):&lt;/strong&gt; Building a mutated copy and swapping a pointer makes transition time—and therefore &lt;strong&gt;permission revocation latency&lt;/strong&gt;—proportional to the size of the &lt;em&gt;entire tree&lt;/em&gt; rather than the size of the &lt;em&gt;change&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structural sharing (persistent path-copying):&lt;/strong&gt; Keeps revocation latency flat, but forces every structure (including the permission universe) into persistent data structures, adding massive comprehension and security complexity.&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Solution (&lt;code&gt;In-Place Fold Under Exclusion&lt;/code&gt;):&lt;/strong&gt; The tree separates &lt;strong&gt;canonical state (N)&lt;/strong&gt; from a &lt;strong&gt;pending delta (N+1 - N)&lt;/strong&gt;. Writers compose mutations into the delta without blocking readers, then take the write lock &lt;strong&gt;only to fold the delta into canonical state in place&lt;/strong&gt;. Revocation latency scales strictly with the size of the delta, and Go's race detector can verify correctness by inspection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why We Accepted the &lt;code&gt;28M/s&lt;/code&gt; Ceiling (&lt;code&gt;§6&lt;/code&gt;):&lt;/strong&gt; Because &lt;code&gt;RLock&lt;/code&gt; contends on a shared counter, throughput caps around &lt;strong&gt;28M answers/s&lt;/strong&gt; regardless of core count. In &lt;code&gt;ADR-017 §6&lt;/code&gt;, we explicitly state and accept this limit because it sits &lt;strong&gt;~480× above plausible enterprise peak demand&lt;/strong&gt; (&lt;code&gt;50,000&lt;/code&gt; checks/s)—and any deployment exceeding it can shard the hierarchy (&lt;code&gt;ADR-016&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Phased Writes Without Breaking Invariants
&lt;/h2&gt;

&lt;p&gt;The hardest engineering challenge wasn't read speed—it was updating the authorization tree safely while millions of reads are in flight.&lt;/p&gt;

&lt;p&gt;Moving from single atomic writes to a &lt;strong&gt;phased write mechanism&lt;/strong&gt; introduces &lt;em&gt;irreducible complexity&lt;/em&gt;: multiple tree invariants must remain intact across intermediate states so concurrent readers never observe torn permissions. Combined with state-delta encoding, this yields atomic visibility in the best case and strict eventual consistency in the worst—without ever violating security invariants.&lt;/p&gt;




&lt;h2&gt;
  
  
  Engineering With Intent (Preventing AI Refactor Loops)
&lt;/h2&gt;

&lt;p&gt;Retrofitting a complex mechanism like phased writes onto a codebase that has evolved across multiple rewrites requires strict architectural guardrails. I structured the workspace around &lt;strong&gt;Vigil Keep&lt;/strong&gt; (&lt;code&gt;ibe-workflow-template&lt;/code&gt;)—an Intent-Based Engineering (IBE) workflow combining structured context management and explicit Architecture Decision Records (ADRs).&lt;/p&gt;

&lt;p&gt;When using AI coding agents on irreducible systems problems, letting the agent iterate blindly at the code level causes context confusion and endless loops. Instead, we enforce a strict directional ladder (&lt;strong&gt;Influence drives down; feedback rises only as high as necessary&lt;/strong&gt;):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Forward Drive:&lt;/strong&gt; &lt;code&gt;Intent → Decisions (ADRs) → Design / Specs → Implementation&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backward Validation:&lt;/strong&gt; If implementation fails against intent, feedback steps up &lt;em&gt;only&lt;/em&gt; to the first rung capable of resolving the missing constraint (Implementation → Specs → Decisions → Intent).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forward Remediation:&lt;/strong&gt; Once the constraint or trade-off is locked in the ADR/Spec (such as &lt;code&gt;adr-017&lt;/code&gt;), execution drives down once and concludes—or escalates to a human if intents conflict.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;em&gt;(You can see a CSS diagram of how influence and feedback move across these rungs on &lt;a href="https://noctranet.com/approach" rel="noopener noreferrer"&gt;Noctranet's Approach page&lt;/a&gt;.)&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Explore the Codebase
&lt;/h2&gt;

&lt;p&gt;VK Auth is open-source under Apache 2.0 and split into focused modules:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;graph/&lt;/code&gt;&lt;/strong&gt; — In-memory authorization tree, zero-alloc traversal, and state-delta phased writes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;controller/&lt;/code&gt;&lt;/strong&gt; — Token and session verification engine&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;register/&lt;/code&gt;&lt;/strong&gt; — Identity and credential management&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;workspace/&lt;/code&gt;&lt;/strong&gt; — Reproducible benchmark generators, stress suites, and &lt;code&gt;PERFORMANCE.md&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check out the repositories and ADRs on GitLab: &lt;strong&gt;&lt;a href="https://gitlab.com/vigilkeep/auth" rel="noopener noreferrer"&gt;https://gitlab.com/vigilkeep/auth&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you’ve weighed &lt;code&gt;sync.RWMutex&lt;/code&gt; + state-delta mutations against copy-on-write structures in high-throughput Go services, I’d love to compare notes in the comments!&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Cover photo by &lt;a href="https://unsplash.com/@ling_hua" rel="noopener noreferrer"&gt;ling hua&lt;/a&gt; on &lt;a href="https://unsplash.com" rel="noopener noreferrer"&gt;Unsplash&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>go</category>
      <category>performance</category>
      <category>architecture</category>
      <category>showdev</category>
    </item>
  </channel>
</rss>
