<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Kennedy Njoroge </title>
    <description>The latest articles on DEV Community by Kennedy Njoroge  (@kyien).</description>
    <link>https://dev.to/kyien</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F977128%2F29ac404a-5951-4417-9573-fe8612076e23.jpeg</url>
      <title>DEV Community: Kennedy Njoroge </title>
      <link>https://dev.to/kyien</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/kyien"/>
    <language>en</language>
    <item>
      <title>Autoscaling doesn't save you at 30,000 requests per second — your dependencies do</title>
      <dc:creator>Kennedy Njoroge </dc:creator>
      <pubDate>Mon, 05 Oct 2026 09:57:01 +0000</pubDate>
      <link>https://dev.to/kyien/autoscaling-doesnt-save-you-at-30000-requests-per-second-your-dependencies-do-1o5e</link>
      <guid>https://dev.to/kyien/autoscaling-doesnt-save-you-at-30000-requests-per-second-your-dependencies-do-1o5e</guid>
      <description>&lt;p&gt;There's a comfortable story about peak traffic: load goes up, the autoscaler notices, pods multiply, everyone keeps their evening. I've watched that story hold, and I've watched it fail in a specific and instructive way — the autoscaler worked perfectly and the platform degraded anyway.&lt;/p&gt;

&lt;p&gt;Running transaction paths at sustained peaks in the tens of thousands of requests per second teaches you that horizontal scaling is the easy half of the problem. The hard half is that everything your new pods talk to did not scale, and now there are more of them asking.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure that taught me this
&lt;/h2&gt;

&lt;p&gt;Scale-up triggers. Pods go from 12 to 40 in about ninety seconds. Each one opens its configured pool of 20 database connections on startup.&lt;/p&gt;

&lt;p&gt;The database's connection limit is 500.&lt;/p&gt;

&lt;p&gt;Twelve pods needed 240 connections and everything was fine. Forty pods want 800. The database starts refusing connections, the pods that can't connect fail their readiness probes, the orchestrator restarts them, they try to connect again, and you have built a very efficient machine for denying yourself service. The autoscaler, meanwhile, sees rising latency and scales up further.&lt;/p&gt;

&lt;p&gt;The dashboards looked bizarre: CPU low, memory low, replica count healthy, error rate vertical.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Autoscaling converts a capacity problem into a concurrency problem on whatever you depend on.&lt;/strong&gt; If that dependency is a fixed-size resource — a connection limit, a license seat count, a rate-limited upstream, a legacy system with a thread pool — scaling out makes things worse, not better, and it does so fast.&lt;/p&gt;

&lt;h2&gt;
  
  
  Budget connections globally, not per pod
&lt;/h2&gt;

&lt;p&gt;The fix is to stop thinking about per-pod configuration and start thinking about a global budget:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pool_size_per_pod × max_replicas ≤ dependency_limit × safety_factor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With a 500-connection database, 0.8 safety factor (leave headroom for migrations, admin sessions, the other service nobody told you about), and a 40-replica ceiling:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pool_size ≤ (500 × 0.8) / 40 = 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then pin both halves so neither drifts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;autoscaling/v2&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HorizontalPodAutoscaler&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;minReplicas&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;12&lt;/span&gt;
  &lt;span class="na"&gt;maxReplicas&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;40&lt;/span&gt;          &lt;span class="c1"&gt;# ← this is a dependency constraint, not a cost one&lt;/span&gt;
  &lt;span class="na"&gt;metrics&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pods&lt;/span&gt;
      &lt;span class="na"&gt;pods&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;metric&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http_inflight_requests&lt;/span&gt;
        &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AverageValue&lt;/span&gt;
          &lt;span class="na"&gt;averageValue&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;35"&lt;/span&gt;
  &lt;span class="na"&gt;behavior&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;scaleUp&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;stabilizationWindowSeconds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt;
      &lt;span class="na"&gt;policies&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Percent&lt;/span&gt;
          &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;50&lt;/span&gt;
          &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt;
    &lt;span class="na"&gt;scaleDown&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;stabilizationWindowSeconds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;600&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things worth calling out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;maxReplicas&lt;/code&gt; is load-bearing.&lt;/strong&gt; Document &lt;em&gt;why&lt;/em&gt; it's 40, next to the number, or someone will raise it during an incident to "give it more capacity" and cause the exact outage they're trying to stop.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scale on in-flight requests, not CPU.&lt;/strong&gt; A service waiting on a slow downstream has low CPU and is in serious trouble. In-flight count — or queue depth — tracks saturation. CPU tracks work, and at peak the problem is usually waiting, not working.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Asymmetric windows.&lt;/strong&gt; Scale up fast, scale down slowly. Flapping replicas means constantly re-establishing connections, which is precisely the pressure you're trying to avoid.&lt;/p&gt;

&lt;h2&gt;
  
  
  Retry budgets, or how three services become twenty-seven
&lt;/h2&gt;

&lt;p&gt;Service A calls B calls C. Each retries three times on failure, which is the default in a lot of client libraries and sounds reasonable in isolation.&lt;/p&gt;

&lt;p&gt;C has a bad minute. B's single call becomes 3. A's single call becomes 9. Your users' single click becomes 27 requests to a service that is already struggling. Retries turn a brownout into an outage, reliably, and the blame lands on whoever's service fell over last rather than whoever configured the retries.&lt;/p&gt;

&lt;p&gt;Two rules that have held up for me:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Retry at one layer only.&lt;/strong&gt; Usually the outermost one that can still do something useful. Everything beneath it fails fast and propagates.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cap retries as a fraction of total traffic, not per request.&lt;/strong&gt; This is what service meshes mean by a retry budget — "retries may not exceed 20% of requests to this destination". Under normal conditions nobody notices. Under failure, the budget exhausts and retries simply stop, which is exactly the behaviour you want and exactly the behaviour per-request configuration cannot express.&lt;/p&gt;

&lt;p&gt;Pair it with a circuit breaker so that a hard-down dependency fails in microseconds instead of consuming a connection for a 30-second timeout:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;outlierDetection&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;consecutive5xxErrors&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
  &lt;span class="na"&gt;interval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10s&lt;/span&gt;
  &lt;span class="na"&gt;baseEjectionTime&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
  &lt;span class="na"&gt;maxEjectionPercent&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;50&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;maxEjectionPercent: 50&lt;/code&gt; is the detail people skip. Without it, a bad deploy or a network partition can get every endpoint ejected and you've turned a partial failure into a total one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failover you haven't run is not a failover
&lt;/h2&gt;

&lt;p&gt;Every platform I've worked on had a documented DR procedure. The useful question is never whether it exists, it's: &lt;strong&gt;when did a human last execute it, and how long did it take?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If the answer is "during the audit, and about forty minutes", you don't have failover. You have a document. Forty minutes of manual steps under pressure is where people typo a hostname into a production config.&lt;/p&gt;

&lt;p&gt;Making it real means making it a script, and making the script run on a normal Tuesday:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail

&lt;span class="nv"&gt;TARGET_SITE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;:?usage:&lt;span class="p"&gt; failover.sh &amp;lt;dr|primary&amp;gt;&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="c"&gt;# 1. verify the target is actually healthy before sending traffic to it&lt;/span&gt;
./healthcheck.sh &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$TARGET_SITE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"target unhealthy, aborting"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# 2. pre-warm — a cold site that receives 100% of peak traffic will fall over&lt;/span&gt;
ansible-playbook &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="s2"&gt;"inventory/&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;TARGET_SITE&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; scale-up.yml &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--extra-vars&lt;/span&gt; &lt;span class="s2"&gt;"target_replicas=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;PEAK_REPLICAS&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
./wait-for-ready.sh &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$TARGET_SITE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--timeout&lt;/span&gt; 180

&lt;span class="c"&gt;# 3. shift traffic in steps, checking error rate between each&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;weight &lt;span class="k"&gt;in &lt;/span&gt;10 25 50 100&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  ./set-traffic-weight.sh &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$TARGET_SITE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$weight&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="nb"&gt;sleep &lt;/span&gt;30
  ./assert-error-rate-below.sh 0.01 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; ./set-traffic-weight.sh &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$TARGET_SITE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 0&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The pre-warm step is the one most often missing. Shifting full peak traffic onto a site running at minimum replicas gives you a cold start, an empty cache, and a thundering herd against the database, all at once — and the resulting outage gets attributed to the failover rather than to the lack of pre-warm.&lt;/p&gt;

&lt;p&gt;The traffic ramp matters for the same reason: it gives you a cheap, reversible answer to "is the DR site actually serving correctly" before you're fully committed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Telemetry that triggers things, not just dashboards
&lt;/h2&gt;

&lt;p&gt;Most observability setups stop at displaying. The step after that is letting signals drive action — scale-up, pre-warm, failover — with clear, bounded authority.&lt;/p&gt;

&lt;p&gt;What I'd automate, in order of how confidently I'd do it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Scale-up on saturation.&lt;/strong&gt; Safe, reversible, happens constantly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pre-warm the DR site on sustained primary degradation.&lt;/strong&gt; Also safe — you're just spending some compute on standby capacity. Do this well before you'd consider failing over.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Traffic shift.&lt;/strong&gt; Automate the &lt;em&gt;steps&lt;/em&gt;, gate the &lt;em&gt;decision&lt;/em&gt; on a human, at least until you've watched it work a dozen times. Automated failover with a flaky health signal gives you a system that fails over and back repeatedly, which is worse than either site being down.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And whatever you automate, the signal needs to be multi-source. A single probe failing is a probe problem until something else agrees with it. Requiring agreement between an error-rate signal and a synthetic transaction before acting cuts false positives dramatically, at the cost of a little latency on genuine failures. That's a good trade for anything that moves traffic between sites.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check this week
&lt;/h2&gt;

&lt;p&gt;If peak is coming and you've got an afternoon:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Multiply your &lt;strong&gt;pool size by max replicas&lt;/strong&gt;. Compare that number against every fixed-limit dependency. Do it for database connections, and then do it again for anything with a rate limit.&lt;/li&gt;
&lt;li&gt;Trace a request through your stack and &lt;strong&gt;count the retry layers&lt;/strong&gt;. If it's more than one, fix that before peak.&lt;/li&gt;
&lt;li&gt;Ask when the &lt;strong&gt;failover was last executed by a human&lt;/strong&gt;. If it's more than six months, schedule a rehearsal rather than a review.&lt;/li&gt;
&lt;li&gt;Check what your HPA &lt;strong&gt;actually scales on&lt;/strong&gt;. If it's CPU, ask what happens when the service is slow but idle.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is exotic. It's just the half of scaling that doesn't show up in the autoscaler's dashboard, which is why it's the half that's usually missing.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Writing up what I learn about reliability, backend systems and data infrastructure. If you've had an autoscaler make an outage worse, I'd like to hear how.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>sre</category>
      <category>kubernetes</category>
      <category>observability</category>
      <category>devops</category>
    </item>
    <item>
      <title>A cache key ate 99.9% of my records and the pipeline looked green</title>
      <dc:creator>Kennedy Njoroge </dc:creator>
      <pubDate>Mon, 05 Oct 2026 09:53:33 +0000</pubDate>
      <link>https://dev.to/kyien/a-cache-key-ate-999-of-my-records-and-the-pipeline-looked-green-37i0</link>
      <guid>https://dev.to/kyien/a-cache-key-ate-999-of-my-records-and-the-pipeline-looked-green-37i0</guid>
      <description>&lt;p&gt;The worst pipeline bug I've shipped didn't throw. No stack trace, no failed job, no alert. Every run went green, finished inside its window, and wrote roughly one record out of every thousand it was handed.&lt;/p&gt;

&lt;p&gt;The job's own metrics said it was healthy, because the job's own metrics counted runs, not rows. It took a business user asking why a campaign list looked short to surface it.&lt;/p&gt;

&lt;p&gt;This post is about that class of bug — the pipeline that succeeds at doing nothing — and the three habits that make it structurally impossible rather than merely unlikely.&lt;/p&gt;

&lt;h2&gt;
  
  
  How you lose 99.9% of your data without noticing
&lt;/h2&gt;

&lt;p&gt;Simplified, the stage looked like this. Records arrive in batches, get enriched against a reference lookup, and are staged before reconciliation downstream.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="c1"&gt;// the bug, reduced&lt;/span&gt;
&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;cacheKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getAccountId&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="nc"&gt;Enrichment&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cacheKey&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lookupService&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;fetch&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;put&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cacheKey&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;stage&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spot it? &lt;code&gt;accountId&lt;/code&gt; isn't unique per record. One account generates many records per batch — different products, different billing cycles, different dates. The key collapsed thousands of distinct records onto a handful of cache entries, and the downstream staging write was keyed off the same value. Last write wins. Everything else evaporated.&lt;/p&gt;

&lt;p&gt;The job processed every record. It just persisted almost none of them.&lt;/p&gt;

&lt;p&gt;The fix is one line, and the one line is not the lesson:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;cacheKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;join&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"|"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getAccountId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getProductCode&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getBillingCycle&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getRecordDate&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The lesson is that nothing in the system was capable of noticing. That's the actual defect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Habit 1: count rows at every boundary, and assert on the ratio
&lt;/h2&gt;

&lt;p&gt;A pipeline stage should know how many records it received and how many it emitted, and should refuse to call itself successful when those numbers disagree in a way nobody authorised.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;StageResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;stage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;emitted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;rejected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;reason_counts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;assert_sane&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_ratio&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;
        &lt;span class="n"&gt;accounted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;emitted&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rejected&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;accounted&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;PipelineError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stage&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;accounted&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; records vanished&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;emitted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;min_ratio&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;PipelineError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stage&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: emitted &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;emitted&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;emitted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;) — below floor. &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rejections: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reason_counts&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two distinct checks doing different jobs. The first is conservation: every record either came out or was explicitly rejected with a reason. A record that is neither emitted nor rejected has &lt;em&gt;vanished&lt;/em&gt;, and vanishing is always a bug. The second is a ratio floor: even when everything is accounted for, a sudden collapse in throughput means something upstream changed.&lt;/p&gt;

&lt;p&gt;Deliberate filtering goes through &lt;code&gt;rejected&lt;/code&gt; with a reason. That way "we dropped 40% because they were test accounts" is visible and intentional, and "we dropped 40% because of a cache key" is a hard failure.&lt;/p&gt;

&lt;p&gt;I now treat a stage without row-conservation accounting as untested, regardless of how many unit tests it has. My own cache-key bug passed its unit tests, because the test fixture had one record per account.&lt;/p&gt;

&lt;h2&gt;
  
  
  Habit 2: make reruns boring
&lt;/h2&gt;

&lt;p&gt;The question that separates a pipeline you can operate from one you can't: &lt;em&gt;what happens if I run yesterday's batch again, right now?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If the answer involves checking anything first, the pipeline is fragile, because reruns are not an edge case. Reruns happen after an outage, after a bad upstream file, after a bug fix, and always under time pressure with someone asking when it'll be done.&lt;/p&gt;

&lt;p&gt;Idempotency isn't a property you bolt on. It comes from one decision: &lt;strong&gt;every record has a deterministic natural key, and writes are keyed on it.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;staging_campaign_records&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;record_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;product_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;billing_cycle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;record_date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;loaded_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch_id&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(:&lt;/span&gt;&lt;span class="n"&gt;record_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;product_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;billing_cycle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;record_date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SYSTIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;CONFLICT&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record_key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;DO&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;amount&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;EXCLUDED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;loaded_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;EXCLUDED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loaded_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;batch_id&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;EXCLUDED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The record key is derived from the business meaning of the row — the combination of fields that genuinely identifies it — never from a sequence, a load timestamp, or a UUID generated at read time. Those are all "different every run", which is the opposite of what you need.&lt;/p&gt;

&lt;p&gt;Then, crucially, a unique constraint on &lt;code&gt;record_key&lt;/code&gt; in the database. Not a check in application code. Application checks race; a unique index is a promise the storage engine keeps even when two workers process overlapping batches.&lt;/p&gt;

&lt;p&gt;Note what this gives you beyond safety: deleting and reloading a day's data becomes a thing you can do at 2 p.m. on a Tuesday without ceremony. That changes how fast you can fix things.&lt;/p&gt;

&lt;h2&gt;
  
  
  Habit 3: let history be history (SCD Type 2 without tears)
&lt;/h2&gt;

&lt;p&gt;The other silent corruption is overwriting a dimension in place. Customer changes segment; you &lt;code&gt;UPDATE&lt;/code&gt; the row; every report that ever referenced the old segment is now retroactively wrong, including the ones already circulated.&lt;/p&gt;

&lt;p&gt;Type 2 slowly changing dimensions solve this by never updating a fact, only closing it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- close the current row when the tracked attributes actually changed&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;dim_customer&lt;/span&gt;
   &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;valid_to&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;effective_from&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;INTERVAL&lt;/span&gt; &lt;span class="s1"&gt;'1 day'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;is_current&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;FALSE&lt;/span&gt;
 &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;
   &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;is_current&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;TRUE&lt;/span&gt;
   &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;segment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tariff_plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
       &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(:&lt;/span&gt;&lt;span class="n"&gt;segment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;tariff_plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;-- open a new one&lt;/span&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;dim_customer&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;segment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tariff_plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;valid_from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;valid_to&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;is_current&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;segment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;tariff_plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;effective_from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;DATE&lt;/span&gt; &lt;span class="s1"&gt;'9999-12-31'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;TRUE&lt;/span&gt;
 &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
       &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;dim_customer&lt;/span&gt;
        &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;
          &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;is_current&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;TRUE&lt;/span&gt;
 &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;IS DISTINCT FROM&lt;/code&gt; is doing real work there — it's null-safe, so a genuine &lt;code&gt;NULL → 'value'&lt;/code&gt; transition is detected rather than silently skipped the way &lt;code&gt;!=&lt;/code&gt; would skip it. And the &lt;code&gt;WHERE NOT EXISTS&lt;/code&gt; on the insert makes the pair idempotent: run it twice with the same input and you get one version, not two.&lt;/p&gt;

&lt;p&gt;Then querying as-of any point in time is trivial, which is the whole point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;segment&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;fact_usage&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;
  &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;dim_customer&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;
   &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage_date&lt;/span&gt; &lt;span class="k"&gt;BETWEEN&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valid_from&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valid_to&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reports stop changing their answers about the past. Disputes become checkable.&lt;/p&gt;

&lt;h2&gt;
  
  
  The check that would have caught me
&lt;/h2&gt;

&lt;p&gt;Row conservation, as a single assertion, at every stage boundary. That's it. The cache-key bug would have failed its very first run with &lt;code&gt;emitted 1,204/1,100,000 (0.1%) — below floor&lt;/code&gt;, instead of looking healthy for weeks.&lt;/p&gt;

&lt;p&gt;Everything else in this post is about making that assertion safe to act on: if reruns are idempotent, failing a batch loudly costs you nothing, so you can afford to fail loudly on anything suspicious. A pipeline that can't be safely rerun is a pipeline under pressure to keep going when it shouldn't — and that pressure is where silent data loss actually comes from.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Count what goes in and what comes out.&lt;/strong&gt; A record that is neither emitted nor rejected has vanished.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make reruns boring.&lt;/strong&gt; Deterministic natural keys, upserts, unique constraints in the database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never overwrite history.&lt;/strong&gt; Close the old row, open a new one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of it is clever. That's rather the point — clever is what ate my records.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;More on backend systems, pipelines and reliability as I go. If you've got a favourite silent-failure story, I'd genuinely like to read it.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>python</category>
      <category>architecture</category>
      <category>sql</category>
    </item>
    <item>
      <title>The boring part of an AI agent is the part that works</title>
      <dc:creator>Kennedy Njoroge </dc:creator>
      <pubDate>Mon, 05 Oct 2026 09:52:14 +0000</pubDate>
      <link>https://dev.to/kyien/the-boring-part-of-an-ai-agent-is-the-part-that-works-3bhn</link>
      <guid>https://dev.to/kyien/the-boring-part-of-an-ai-agent-is-the-part-that-works-3bhn</guid>
      <description>&lt;p&gt;Most of the time an incident spends "being worked on" isn't work. It's waiting to reach someone who can act on it. An alert fires, an email lands in a shared inbox, somebody skims it, guesses which team owns it, forwards it, and that team decides it isn't theirs. On a bad night that loop costs you an hour before anyone has opened a terminal.&lt;/p&gt;

&lt;p&gt;I built an agent to collapse that loop for a BSS platform, and it took mean time to resolution on routed incidents from around two hours to about ten minutes. What surprised me is how little of that came from the language model. The model does one small job in the middle. Everything that makes the system trustworthy is plumbing.&lt;/p&gt;

&lt;p&gt;This post is about the plumbing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shape of the problem
&lt;/h2&gt;

&lt;p&gt;Incoming incident email is semi-structured in the worst way: there's a template, three systems ignore it, two append their own stack traces, and the one that matters most sends plain prose from a human. Regex gets you maybe 60% of the way and then rots every time an upstream team edits their alert body.&lt;/p&gt;

&lt;p&gt;That 40% tail is exactly where an LLM earns its place. But "classify this email" is not the system. The system is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;poll inbox → normalise → classify → resolve owner → create ticket → notify → observe
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model is step three. Steps one, two, four, five, six and seven decide whether anyone trusts it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Normalise before you classify
&lt;/h2&gt;

&lt;p&gt;The single highest-leverage thing I did was aggressively strip the email before the model ever saw it. Signatures, legal footers, quoted reply chains, base64 inline images, and the forty lines of &lt;code&gt;Received:&lt;/code&gt; headers are pure token cost and pure distraction.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;email.parser&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BytesParser&lt;/span&gt;

&lt;span class="n"&gt;QUOTE_MARKERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-----Original Message-----&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;From:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;On ... wrote:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extract_body&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;BytesParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;parsebytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_body&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;preferencelist&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;plain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;html&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_content&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;

    &lt;span class="c1"&gt;# cut the reply chain at the first quote marker
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;marker&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;QUOTE_MARKERS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;idx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;marker&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\n{3,}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;[:4000]&lt;/code&gt; matters. Truncation is a feature. An incident email that needs more than 4000 characters of context to classify is one a human should be reading anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the model return a shape, not a sentence
&lt;/h2&gt;

&lt;p&gt;The classification call is deliberately small and deliberately constrained. I ask for structured output with a fixed enum of teams, a confidence score, and a one-line rationale.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;

&lt;span class="n"&gt;Team&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;billing&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;provisioning&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;network&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;identity&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;integrations&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-platform&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Triage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;team&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Team&lt;/span&gt;
    &lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sev1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sev2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sev3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sev4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;confidence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rationale&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details that are easy to skip and expensive to skip:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;unknown&lt;/code&gt; is a valid answer.&lt;/strong&gt; If the enum has no escape hatch, the model will pick the nearest plausible team and sound certain about it. Giving it somewhere honest to go is the cheapest accuracy win available.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The rationale is capped.&lt;/strong&gt; Not because anyone reads it in the happy path, but because when triage goes wrong at 3 a.m. the rationale is the only artefact that tells you whether the model misread the email or the owner map was stale. A 200-character cap keeps it a signal rather than an essay.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Confidence is the model's, and it's directional at best.&lt;/strong&gt; Treat it as a tiebreaker, not a probability. I threshold it, but the threshold is tuned against observed outcomes, not against what the number claims to mean.&lt;/p&gt;

&lt;h2&gt;
  
  
  The owner map is not the model's problem
&lt;/h2&gt;

&lt;p&gt;Here's the part that no amount of prompting fixes: the model can tell you a ticket is about billing. It cannot know that billing's on-call rotated yesterday, or that the billing team split into two squads last quarter.&lt;/p&gt;

&lt;p&gt;So the model resolves to a &lt;em&gt;team&lt;/em&gt;, never to a &lt;em&gt;person&lt;/em&gt;. A separate lookup against the directory — in my case Microsoft Graph — resolves the team to whoever is actually on call right now.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;resolve_owner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;team&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;graph&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Owner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;group&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OWNER_GROUPS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;team&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;            &lt;span class="c1"&gt;# static, version-controlled mapping
&lt;/span&gt;    &lt;span class="n"&gt;members&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group_members&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;on_call&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on_call_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# calendar-driven
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;on_call&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;members&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This split is what makes the thing survive org changes. When a team reorganises, somebody edits a YAML file. Nobody retrains anything, nobody rewrites a prompt. The LLM's view of the world is "what is this email about", which is stable. The volatile knowledge lives in systems that are already the source of truth for it.&lt;/p&gt;

&lt;p&gt;If you take one thing from this post: &lt;strong&gt;never let the model hold facts that another system owns.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Route low confidence to a human, loudly
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Mail&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;extract_body&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;triage&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;team&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.72&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;post_to_triage_channel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;low_confidence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;

    &lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;resolve_owner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;team&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;graph&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ticket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;create_ticket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ticket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;triage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fallback is not a failure path, it's the &lt;em&gt;default&lt;/em&gt; path for anything ambiguous, and it lands somewhere visible. Early on, roughly one in six emails went to the human channel. That felt like a bad number until I looked at what was in it: most were genuinely ambiguous, and the ones that weren't told me exactly which alert template to fix upstream. The fallback queue is your best source of improvement work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Idempotency, because email is not a queue
&lt;/h2&gt;

&lt;p&gt;Mail APIs will hand you the same message twice. Your poller will crash between "created ticket" and "marked as read". Somebody will replay a batch to test something.&lt;/p&gt;

&lt;p&gt;Every message gets a deterministic key, and ticket creation is conditional on it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;dedupe_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Mail&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# message-id is stable across redelivery; subject+sender is the fallback
&lt;/span&gt;    &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message_id&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sender&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;subject&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sent_at&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then a unique index on that key in whatever store backs you. Not a cache — a unique index. A cache will happily evict the thing under memory pressure and you'll find out when duplicate tickets land during your busiest hour. I have opinions about cache keys and silent data loss that I'll save for another post.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd measure from day one
&lt;/h2&gt;

&lt;p&gt;I started with "did it classify correctly", which turns out to be the least useful metric, because you can only compute it by hand on a sample.&lt;/p&gt;

&lt;p&gt;What actually drove decisions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reroute rate&lt;/strong&gt; — tickets reassigned within 30 minutes of creation. This is your real accuracy signal and it's free, because humans generate it just by doing their jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fallback rate&lt;/strong&gt; — share going to the human channel. Trending up means an upstream alert format changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time to first human action&lt;/strong&gt; — the number the whole project exists to move. Not time to resolution; you don't control that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per message&lt;/strong&gt; — unglamorous, but it's what lets you argue for the system's continued existence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reroute rate is the one I'd instrument first in any triage system. It's ground truth that costs nothing to collect.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest summary
&lt;/h2&gt;

&lt;p&gt;The model is a classifier with good manners about unstructured text. That's genuinely useful and regex could not have done it. But the two-hour-to-ten-minute number came from: removing a human hop, resolving on-call automatically, and making the ambiguous cases land somewhere visible instead of somewhere polite.&lt;/p&gt;

&lt;p&gt;If you're building something similar, budget your time accordingly. The prompt will take an afternoon. The owner map, the idempotency, the fallback path and the metrics will take the rest of the month, and they're the reason it still works six months later.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I write about backend engineering, data pipelines and reliability. If you've built something in this space and solved the owner-mapping problem differently, I'd like to hear it.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>devops</category>
      <category>automation</category>
    </item>
    <item>
      <title>Interoperability: The Key to Unlocking Blockchain's Full Potential</title>
      <dc:creator>Kennedy Njoroge </dc:creator>
      <pubDate>Tue, 14 May 2024 13:55:43 +0000</pubDate>
      <link>https://dev.to/kyien/interoperability-the-key-to-unlocking-blockchains-full-potential-28h5</link>
      <guid>https://dev.to/kyien/interoperability-the-key-to-unlocking-blockchains-full-potential-28h5</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Blockchain technology is rapidly evolving and has spawned many independent networks. Each network has its own unique advantages, but their isolation has prevented them from exchanging data and assets seamlessly across platforms. Interoperability, on the other hand, is a revolutionary trend that seeks to bridge these networks and foster collaboration, creating unprecedented opportunities.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is Interoperability?
&lt;/h2&gt;

&lt;p&gt;Interoperability in blockchain world is when different blockchains can communicate and share information in a seamless manner. This can include token exchange, smart contract data exchange, or even the creation of entire decentralized applications.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is interoperability important?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1.Enhanced Liquidity:&lt;/strong&gt; Interoperability enables assets to move freely between blockchains, boosting liquidity and expanding market opportunities.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Scalability:&lt;/strong&gt;Interoperability can alleviate congestion on individual blockchains by distributing the workload across multiple networks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.Increased Utilization:&lt;/strong&gt; DApps can take advantage of the functionalities of multiple blockchains, creating new use cases and enhancing the user experience.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4.Collaboration:&lt;/strong&gt; Projects can collaborate more effectively, combining their strengths to create innovative solutions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Interoperability in Action: Cosmos Network
&lt;/h2&gt;

&lt;p&gt;One of the most prominent examples of interoperability is the Cosmos Network, which uses the IBC protocol to securely and efficiently communicate between interconnected blockchains.Here is an example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JavaScript
// IBC Module (Simplified)
class IBCModule {
  constructor(sourceChain, destinationChain) {
    this.sourceChain = sourceChain;
    this.destinationChain = destinationChain;
  }

  createPacket(data) {
    return {
      sourceChain: this.sourceChain,
      destinationChain: this.destinationChain,
      sequence: generateUniqueSequence(), // In reality, this would be managed meticulously
      timeoutHeight: getCurrentBlockHeight() + 100, // Timeout if not processed by dest. chain
      data: data,
    };
  }

  sendPacket(packet) {
    // 1. Sign the packet (omitted for simplicity, but crucial for security)
    // 2. Relay the packet to the source chain's IBC module
    //    This involves complex network communication, often via 'relayer' nodes
    sourceChain.ibcModule.submitPacket(packet); 
  }

  receivePacket(packet) {
    // 1. Verify packet authenticity, signatures, etc.
    // 2. Process the data based on its type
    switch (packet.data.type) {
      case 'tokenTransfer':
        processTokenTransfer(packet.data);
        break;
      // ... other data types
    }
  }
}

// Usage Example
const cosmosHubIBC = new IBCModule('Cosmos Hub', 'Osmosis');
const transferData = {
  type: 'tokenTransfer',
  amount: '100 ATOM',
  recipientAddress: 'osmo1...' // Osmosis address
};

const ibcPacket = cosmosHubIBC.createPacket(transferData);
cosmosHubIBC.sendPacket(ibcPacket);

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this simplified example, an IBC packet is used to transfer 100 ATOM tokens from the Cosmos Hub to the Osmosis blockchain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Challenges and Future Viewpoint
&lt;/h2&gt;

&lt;p&gt;Whereas interoperability holds monstrous guarantee, challenges still persist. These incorporate:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Governance:&lt;/strong&gt; Establishing fair and transparent governance models for interconnected blockchains is a complex task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Standardization:&lt;/strong&gt; Developing standardized protocols and frameworks is crucial for widespread adoption.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Security&lt;/strong&gt;: Ensuring secure cross-chain communication is paramount to prevent vulnerabilities and exploits.&lt;/p&gt;

&lt;p&gt;Despite these challenges, the future of blockchain interoperability is bright. As the technology matures, we can expect to see:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-chain DeFi:&lt;/strong&gt; Decentralized finance applications that operate across multiple blockchains, offering users more diverse options.&lt;br&gt;
&lt;strong&gt;Interconnected Metaverse:&lt;/strong&gt; Virtual worlds that seamlessly connect with one another, enabling the transfer of digital assets and identities.&lt;br&gt;
&lt;strong&gt;Universal Blockchain Identity:&lt;/strong&gt; A single, self-sovereign identity that can be used across all blockchain platforms.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Interoperability is a game-changer for the blockchain industry. By enabling seamless communication and collaboration between networks, it unlocks a wealth of new possibilities for innovation and growth. As we move towards a more interconnected future, interoperability will play a pivotal role in shaping the next generation of blockchain applications and services.&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
