<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ayush Soni</title>
    <description>The latest articles on DEV Community by Ayush Soni (@ayushwing).</description>
    <link>https://dev.to/ayushwing</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4210786%2F105aad99-2265-499a-8c6f-fa58f937f3cb.png</url>
      <title>DEV Community: Ayush Soni</title>
      <link>https://dev.to/ayushwing</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ayushwing"/>
    <language>en</language>
    <item>
      <title>Debugging production: a method, not a hunch</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Sat, 14 Nov 2026 04:16:36 +0000</pubDate>
      <link>https://dev.to/ayushwing/debugging-production-a-method-not-a-hunch-34k4</link>
      <guid>https://dev.to/ayushwing/debugging-production-a-method-not-a-hunch-34k4</guid>
      <description>&lt;p&gt;Production incidents reward method over brilliance. The fastest way to a fix is rarely knowing every corner of the system; it's not skipping steps. When an alert fires, the instinct is to open the logs and start reading. That instinct costs hours.&lt;/p&gt;

&lt;p&gt;Here's the method I use.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Stop the bleeding first
&lt;/h2&gt;

&lt;p&gt;Your first job is to reduce user impact, not to understand the problem. Understanding can happen after users are fine.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Roll back&lt;/strong&gt; the last deploy if the timing fits, even if you're "pretty sure" it isn't the cause. A rollback that doesn't help is cheap information.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fail over&lt;/strong&gt; to another region or replica.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disable&lt;/strong&gt; the feature flag behind the new code path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shed load&lt;/strong&gt;: rate-limit the noisy client, pause the batch job, scale out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Declare the incident, name one person to coordinate and one to communicate, and post updates on a fixed rhythm. A quiet incident channel makes everyone else start their own investigation in parallel.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Build a timeline
&lt;/h2&gt;

&lt;p&gt;"It's slow" isn't a starting point; "p99 latency on checkout went from 300 ms to 12 s at 09:02 UTC" is. Pin down:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;When&lt;/strong&gt; exactly it started (and whether it was a step or a ramp).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What&lt;/strong&gt; is affected: which endpoints, error codes, regions, customers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What isn't&lt;/strong&gt; affected. Healthy neighbours are as informative as broken ones.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A step change usually means something changed. A slow ramp usually means something filled up: a disk, a queue, a connection pool, a cache that's slowly expiring.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Ask "what changed?"
&lt;/h2&gt;

&lt;p&gt;Most incidents are caused by a change. Check, in roughly this order:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;Where to look&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Code deploys&lt;/td&gt;
&lt;td&gt;Deploy log, around the start time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Configuration and feature flags&lt;/td&gt;
&lt;td&gt;Flag history, config repo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Infrastructure&lt;/td&gt;
&lt;td&gt;Scaling events, node replacements, certificate renewals, DNS changes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Traffic&lt;/td&gt;
&lt;td&gt;Volume, shape, a new client, a bot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependencies&lt;/td&gt;
&lt;td&gt;Status pages, error rates on outbound calls&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;A new tenant, a huge record, a backfill job running&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A shared, searchable change log across all of these is one of the highest-value investments a platform team can make. Without it, step 3 becomes a poll of the whole engineering department.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Form a hypothesis you can test quickly
&lt;/h2&gt;

&lt;p&gt;"Something's wrong with the database" isn't testable. "Connection pool exhaustion on the order service since the 09:00 deploy, which raised the default timeout" is: check pool metrics and the diff in five minutes.&lt;/p&gt;

&lt;p&gt;Write the hypothesis in the incident channel before you test it. It keeps the team aligned, stops three people testing the same idea, and makes the timeline easy to write afterwards. When a hypothesis is disproved, say so and move on — don't keep half-believing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Narrow by halves
&lt;/h2&gt;

&lt;p&gt;When there's no obvious change, bisect the problem space:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One &lt;strong&gt;region&lt;/strong&gt; or all of them?&lt;/li&gt;
&lt;li&gt;One &lt;strong&gt;endpoint&lt;/strong&gt; or all of them?&lt;/li&gt;
&lt;li&gt;One &lt;strong&gt;customer&lt;/strong&gt; or all of them? (A single tenant with unusual data is behind more incidents than people expect.)&lt;/li&gt;
&lt;li&gt;One &lt;strong&gt;host or pod&lt;/strong&gt; or all of them? (One bad node behind a load balancer looks like a small, random error rate.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Before or after&lt;/strong&gt; a dependency? Traces answer this directly: where does the time go?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each question halves the search space. Five good questions narrow almost anything.&lt;/p&gt;

&lt;p&gt;This is where observability earns its cost: structured logs with a trace ID, traces across service boundaries (including message queues), and metrics you can slice by endpoint, tenant and host. If you can't slice, you can't bisect. (I packaged my defaults for this into a &lt;a href="https://ayushwing.com/blog/spring-boot-starter-observability" rel="noopener noreferrer"&gt;Spring Boot starter&lt;/a&gt;.)&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Write it down
&lt;/h2&gt;

&lt;p&gt;After the incident, write a short, blameless review:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Timeline&lt;/strong&gt; — from first symptom to recovery, with timestamps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Impact&lt;/strong&gt; — who was affected, for how long, how badly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cause&lt;/strong&gt; — the technical chain of events, as far as you know it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Detection&lt;/strong&gt; — how did we find out, and could we have found out sooner?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Follow-ups&lt;/strong&gt; — a small number of concrete actions with owners: a missing alert, a guardrail, a runbook step.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The detection question is often the most valuable. "The customer told us" is a finding in itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Habits that make the method faster
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dashboards built for questions&lt;/strong&gt;, not decoration: the four golden signals (latency, traffic, errors, saturation) per service, sliceable by endpoint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runbooks&lt;/strong&gt; for the alerts you have, with the first three things to check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Practice&lt;/strong&gt;: game days where you break something on purpose in a safe environment and run the method.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Mitigate before you investigate.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Build a precise &lt;strong&gt;timeline&lt;/strong&gt;, then ask &lt;strong&gt;what changed&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Work from &lt;strong&gt;testable hypotheses&lt;/strong&gt;, said out loud.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bisect&lt;/strong&gt; by region, endpoint, tenant, host and dependency.&lt;/li&gt;
&lt;li&gt;Finish with a &lt;strong&gt;blameless review&lt;/strong&gt; that improves detection, not just the fix.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/debugging-production" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>debugging</category>
      <category>sre</category>
      <category>incidents</category>
      <category>observability</category>
    </item>
    <item>
      <title>One dependency for logs, traces and metrics: writing a Spring Boot starter</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Wed, 11 Nov 2026 09:07:51 +0000</pubDate>
      <link>https://dev.to/ayushwing/one-dependency-for-logs-traces-and-metrics-writing-a-spring-boot-starter-2fgg</link>
      <guid>https://dev.to/ayushwing/one-dependency-for-logs-traces-and-metrics-writing-a-spring-boot-starter-2fgg</guid>
      <description>&lt;p&gt;The worst time to discover that your services log differently is during an incident. It happens easily: each service is set up by a different person at a different time, so you end up with different field names, trace IDs in some services but not others, and a correlation ID that stops at the first hop. In a platform with several services I've worked on, the answer was a shared module for logging, auth and tracing; this starter is that idea rebuilt in the open.&lt;/p&gt;

&lt;p&gt;The fix isn't a wiki page of conventions. It's making the right setup the default by putting it in a dependency. That's what &lt;a href="https://github.com/ayushwing/spring-boot-starter-observability" rel="noopener noreferrer"&gt;spring-boot-starter-observability&lt;/a&gt; does: add one dependency to a Spring Boot 3 app and you get:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Structured JSON logs&lt;/strong&gt; with &lt;code&gt;traceId&lt;/code&gt;, &lt;code&gt;spanId&lt;/code&gt;, &lt;code&gt;requestId&lt;/code&gt; and &lt;code&gt;correlationId&lt;/code&gt; on every line.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OpenTelemetry tracing&lt;/strong&gt;: a server span per HTTP request, spans across Kafka producers and consumers using the W3C &lt;code&gt;traceparent&lt;/code&gt; header, and &lt;code&gt;@Traced&lt;/code&gt; for any method.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Micrometer metrics&lt;/strong&gt;: JVM metrics plus &lt;code&gt;@Timed&lt;/code&gt; and &lt;code&gt;@Counted&lt;/code&gt; on any method, and a health indicator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correlation IDs&lt;/strong&gt; that cross service boundaries via &lt;code&gt;X-Correlation-Id&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What a starter actually is
&lt;/h2&gt;

&lt;p&gt;A Spring Boot starter is two small ideas:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A list of auto-configuration classes in &lt;code&gt;META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports&lt;/code&gt;, which Spring Boot loads at startup.&lt;/li&gt;
&lt;li&gt;Those classes create beans &lt;strong&gt;only when it makes sense&lt;/strong&gt;, using &lt;code&gt;@Conditional...&lt;/code&gt; annotations.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The project is split the way the Spring team splits its own starters:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Module&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;starter-core&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Annotations (&lt;code&gt;@Traced&lt;/code&gt;) and constants; no auto-configuration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;starter-autoconfigure&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Logging, tracing and metrics auto-configuration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;starter-sample&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;A product API that shows everything working&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Rule 1: every bean backs off
&lt;/h2&gt;

&lt;p&gt;The most important annotation in a starter is &lt;code&gt;@ConditionalOnMissingBean&lt;/code&gt;. If the application defines its own &lt;code&gt;SpanExporter&lt;/code&gt; or &lt;code&gt;SdkTracerProvider&lt;/code&gt;, the starter's version quietly steps aside:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@AutoConfiguration&lt;/span&gt;
&lt;span class="nd"&gt;@ConditionalOnClass&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;OpenTelemetrySdk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;OtlpGrpcSpanExporter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nd"&gt;@ConditionalOnProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prefix&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"observability.tracing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"enabled"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="n"&gt;havingValue&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;matchIfMissing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ObservabilityTracingAutoConfiguration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

    &lt;span class="nd"&gt;@Bean&lt;/span&gt;
    &lt;span class="nd"&gt;@ConditionalOnMissingBean&lt;/span&gt;
    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;SpanExporter&lt;/span&gt; &lt;span class="nf"&gt;otlpGrpcSpanExporter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ObservabilityProperties&lt;/span&gt; &lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each piece is also gated on the classpath and on a property. Kafka tracing only appears when Kafka's interceptor classes are present &lt;strong&gt;and&lt;/strong&gt; a &lt;code&gt;Tracer&lt;/code&gt; bean exists; the servlet filter and MVC interceptor only appear in servlet web applications. A starter that forces beans on you gets ripped out within a quarter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule 2: observability never stops the app
&lt;/h2&gt;

&lt;p&gt;If the trace collector is down, your service should still serve traffic. If the exporter can't even be built — a malformed endpoint, say — the auto-configuration catches the failure, logs a warning and returns a no-op exporter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Exception&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Failed to initialize OTLP span exporter (endpoint: {}). "&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
             &lt;span class="s2"&gt;"Traces will not be exported: {}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tracingProps&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getEndpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;SpanExporter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;composite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// no-op: the app starts, spans are discarded&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the collector is simply unreachable at runtime, the OpenTelemetry SDK exports in the background and drops what it can't send. Either way, requests keep flowing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Logs that carry the trace
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;RequestContextFilter&lt;/code&gt; runs once per request and fills the SLF4J MDC, so every log line written while handling the request carries the same identifiers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kt"&gt;String&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getHeader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"X-Correlation-Id"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;correlationId&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isBlank&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;correlationId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;MDC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;put&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"correlationId"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setHeader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"X-Correlation-Id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything is cleared in a &lt;code&gt;finally&lt;/code&gt; block — without that, values leak into the next request that reuses the pooled thread, which produces some of the most confusing logs you'll ever read.&lt;/p&gt;

&lt;p&gt;The filter is configurable with properties (&lt;code&gt;include-headers&lt;/code&gt;, &lt;code&gt;header-filter&lt;/code&gt;, &lt;code&gt;custom-fields&lt;/code&gt;) so a team can, for example, add &lt;code&gt;tenantId&lt;/code&gt; to every line without writing code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tracing across Kafka
&lt;/h2&gt;

&lt;p&gt;HTTP has standard instrumentation in most stacks; message queues are where traces usually break. The producer interceptor writes a W3C header onto each record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="c1"&gt;// version-traceId-spanId-traceFlags&lt;/span&gt;
&lt;span class="kt"&gt;String&lt;/span&gt; &lt;span class="n"&gt;traceparent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"00-"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;traceId&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;spanId&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;traceFlags&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"traceparent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;traceparent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getBytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;StandardCharsets&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UTF_8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The consumer interceptor reads it back and starts a child span, so a trace that begins with an HTTP request continues through the topic into the consumer. In Jaeger you see one trace across both services instead of two unrelated ones.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2n31ybnacof4ok5igv4p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2n31ybnacof4ok5igv4p.png" alt="An HTTP request passes through the request-context filter and tracing interceptor; Kafka messages pass through producer and consumer interceptors; all emit JSON logs, OpenTelemetry spans and Micrometer metrics." width="800" height="784"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Entry points on the left, the three signals on the right. &lt;a href="https://ayushwing.com/diagrams/observability-starter.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Three gaps I found reviewing my own starter
&lt;/h2&gt;

&lt;p&gt;I re-read the code as if it were someone else's pull request. Three findings:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;HTTP spans don't continue the caller's trace.&lt;/strong&gt; The tracing interceptor starts the server span with &lt;code&gt;setParent(Context.current())&lt;/code&gt;, but nothing extracts an incoming &lt;code&gt;traceparent&lt;/code&gt; header first. Kafka hops are linked; HTTP hops start a new trace. The fix is to run the OpenTelemetry &lt;code&gt;TextMapPropagator&lt;/code&gt; over the request headers before starting the span.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two different trace IDs.&lt;/strong&gt; The log filter takes its &lt;code&gt;traceId&lt;/code&gt; from an &lt;code&gt;X-Trace-Id&lt;/code&gt; header (or falls back to the request ID), while spans have OpenTelemetry's own trace ID. Logs and traces should share one ID; the filter should read it from the current span.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;http.route&lt;/code&gt; uses the raw URI.&lt;/strong&gt; &lt;code&gt;/products/123&lt;/code&gt; and &lt;code&gt;/products/456&lt;/code&gt; become different routes, which explodes metric cardinality. It should use Spring's matched pattern (&lt;code&gt;/products/{id}&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;None of these show up in a demo. All three show up in production, during the incident you built the starter for. Fixes are the next commits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Put conventions in a dependency, not a document.&lt;/li&gt;
&lt;li&gt;In a starter, &lt;code&gt;@ConditionalOnMissingBean&lt;/code&gt; is a feature: let teams override anything.&lt;/li&gt;
&lt;li&gt;Observability failures must degrade to "no data", never to "no service".&lt;/li&gt;
&lt;li&gt;Trace propagation is only as good as its weakest hop — test every protocol your services speak.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Code, tests and the sample app are on &lt;a href="https://github.com/ayushwing/spring-boot-starter-observability" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/spring-boot-starter-observability" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>springboot</category>
      <category>observability</category>
      <category>opentelemetry</category>
      <category>java</category>
    </item>
    <item>
      <title>The strangler fig in practice: carving a service out of a monolith</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Tue, 10 Nov 2026 12:39:05 +0000</pubDate>
      <link>https://dev.to/ayushwing/the-strangler-fig-in-practice-carving-a-service-out-of-a-monolith-48k1</link>
      <guid>https://dev.to/ayushwing/the-strangler-fig-in-practice-carving-a-service-out-of-a-monolith-48k1</guid>
      <description>&lt;p&gt;I've been part of breaking up monoliths more than once — checkout, refund and wallet flows at an e-commerce marketplace, and a monolithic email system at a travel company. The lesson I'd pass on from both: don't rewrite, carve.&lt;/p&gt;

&lt;p&gt;Big-bang rewrites tend to stall for a simple reason: the old system keeps changing while you rebuild it, so the target moves, and nothing reaches production until everything does. The &lt;strong&gt;strangler fig&lt;/strong&gt; pattern, named by Martin Fowler after a vine that grows around a tree until it can stand alone, does the opposite. You put something in front of the monolith, move &lt;strong&gt;one capability at a time&lt;/strong&gt; into new code, and send traffic there piece by piece. The monolith shrinks; production never stops.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr7k0v447g83hg6umtnaf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr7k0v447g83hg6umtnaf.png" alt="Clients call a router. The router sends refund requests to a new refund service and everything else to the monolith. Over time more routes move to new services." width="800" height="305"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The router is the seam: each route can point at old or new code. &lt;a href="https://ayushwing.com/diagrams/strangler-fig.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: pick the first slice
&lt;/h2&gt;

&lt;p&gt;The first slice decides whether the project gets a second one. Good candidates have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Clear boundaries&lt;/strong&gt;: a capability with its own inputs and outputs, like "issue a refund" or "render and send an email", rather than something tangled through every request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Real value&lt;/strong&gt;: something that is slow to change today, scales differently, or fails differently from the rest. A slice nobody cares about won't earn the next quarter's funding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Manageable data&lt;/strong&gt;: ideally it owns its tables, or can.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Avoid both extremes. The easiest slice teaches you nothing; the scariest one (core checkout, usually) is a bad place to learn how your routing, deploys and monitoring behave.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: route at the edge
&lt;/h2&gt;

&lt;p&gt;Put a routing layer in front of the monolith — an API gateway, a reverse proxy, or a thin facade inside the monolith that forwards certain calls. The important property: &lt;strong&gt;switching a route between old and new is configuration, not a deploy&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Route by whatever is natural: a URL path, a message type, a tenant, a percentage of users. Start with internal users or one low-risk segment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: shadow, then shift
&lt;/h2&gt;

&lt;p&gt;Before the new service serves real users, prove it with real traffic:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Shadow&lt;/strong&gt;: the router sends each request to both. Users get the monolith's response; the new service's response is compared and mismatches are logged. Only do this for reads or for writes that are safe to duplicate (or that the new service performs in a dry-run mode).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shift&lt;/strong&gt;: move a small percentage of real traffic to the new service and watch error rates, latency and business metrics. Increase in steps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the way back open&lt;/strong&gt;: until the old code path is deleted, flipping the route back is your rollback.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Shadow comparison is where you find the undocumented behaviour — the rounding rule, the special case for one partner, the field the monolith silently fills in. Every monolith has these, and they're why "just rewrite it" estimates are always wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: move data ownership deliberately
&lt;/h2&gt;

&lt;p&gt;Routing requests is the easy part. Data is where these projects stall.&lt;/p&gt;

&lt;p&gt;The trap is a &lt;strong&gt;shared database&lt;/strong&gt;: the new service reads and writes the monolith's tables "for now". It ships quickly, and from then on both codebases are coupled through the schema — neither can change a column without the other. You've built a distributed monolith.&lt;/p&gt;

&lt;p&gt;Move ownership in stages, using the same expand–contract approach as for schema changes (see &lt;a href="https://ayushwing.com/blog/expand-contract-migrations" rel="noopener noreferrer"&gt;Zero-downtime migrations with expand–contract&lt;/a&gt;):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The new service gets its &lt;strong&gt;own store&lt;/strong&gt;. Data is synchronised from the monolith (change data capture, or events) while the monolith still owns writes.&lt;/li&gt;
&lt;li&gt;Writes move to the new service; the monolith receives updates through events or an API instead of writing the tables itself.&lt;/li&gt;
&lt;li&gt;The monolith stops touching those tables; they're removed from its schema.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At each step, one system is the &lt;strong&gt;source of truth&lt;/strong&gt; for each piece of data, and everyone knows which one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: decide what "done" means
&lt;/h2&gt;

&lt;p&gt;Strangler projects can go on forever. Decide early:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which capabilities &lt;strong&gt;should&lt;/strong&gt; move (because they change often, scale differently or need separate ownership)?&lt;/li&gt;
&lt;li&gt;Which can &lt;strong&gt;stay&lt;/strong&gt; in a smaller, stable monolith? A well-structured monolith for the parts that rarely change is a perfectly good outcome.&lt;/li&gt;
&lt;li&gt;When will the &lt;strong&gt;old code paths be deleted&lt;/strong&gt;? Every route that has fully moved should have its monolith code removed within a sprint or two. Dead paths behind flags are how a migration turns into permanent debt.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mistake&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;th&gt;Instead&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Starting with the hardest capability&lt;/td&gt;
&lt;td&gt;Months without a production win; project loses support&lt;/td&gt;
&lt;td&gt;Start with a valuable, bounded slice&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shared database "for now"&lt;/td&gt;
&lt;td&gt;Distributed monolith; schema changes need two teams&lt;/td&gt;
&lt;td&gt;Separate stores, synchronised, one source of truth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No shadow phase&lt;/td&gt;
&lt;td&gt;Undocumented behaviour discovered by customers&lt;/td&gt;
&lt;td&gt;Compare responses before shifting traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Never deleting old paths&lt;/td&gt;
&lt;td&gt;Two implementations to maintain forever&lt;/td&gt;
&lt;td&gt;Delete within a sprint of full cutover&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Replace the monolith &lt;strong&gt;one capability at a time&lt;/strong&gt;, never all at once.&lt;/li&gt;
&lt;li&gt;Make routing switchable by &lt;strong&gt;configuration&lt;/strong&gt;, so every step can be reversed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shadow&lt;/strong&gt; real traffic before serving it; that's where hidden behaviour shows up.&lt;/li&gt;
&lt;li&gt;Treat &lt;strong&gt;data ownership&lt;/strong&gt; as the real migration.&lt;/li&gt;
&lt;li&gt;Define "done" — and delete the old code when you get there.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/strangler-fig-in-practice" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>microservices</category>
      <category>architecture</category>
      <category>migrations</category>
      <category>monolith</category>
    </item>
    <item>
      <title>Retries without a retry storm: timeouts, backoff, jitter and circuit breakers</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Mon, 09 Nov 2026 05:56:21 +0000</pubDate>
      <link>https://dev.to/ayushwing/retries-without-a-retry-storm-timeouts-backoff-jitter-and-circuit-breakers-48i2</link>
      <guid>https://dev.to/ayushwing/retries-without-a-retry-storm-timeouts-backoff-jitter-and-circuit-breakers-48i2</guid>
      <description>&lt;p&gt;Retries are the cheapest reliability feature there is: a request fails because of a dropped packet or a pod restart, you try again, it works. Most transient failures are gone in milliseconds.&lt;/p&gt;

&lt;p&gt;They're also one of the most common ways a small incident becomes a large one. A database hiccups for a minute. Every caller retries. The callers of those callers retry too. The database — the one thing that needed less traffic to recover — now receives several times its normal load, and the one-minute blip becomes a long outage. This pattern is often called a &lt;strong&gt;retry storm&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Here are the four tools that keep retries on your side.&lt;/p&gt;

&lt;h2&gt;
  
  
  The arithmetic of retry amplification
&lt;/h2&gt;

&lt;p&gt;Imagine a request that goes through three services before it reaches a database, and each service retries a failed call 3 times in total (1 attempt + 2 retries).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g02ceg66xo2qds8u4yq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g02ceg66xo2qds8u4yq.png" alt="One client request becomes 3 calls to service B, 9 calls to service C, and 27 calls to the database when every layer retries 3 times." width="800" height="338"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Every layer that retries multiplies the load on the layers below. &lt;a href="https://ayushwing.com/diagrams/retry-amplification.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;When the database is failing, one user request becomes 3 × 3 × 3 = &lt;strong&gt;27 database calls&lt;/strong&gt;. Add users refreshing the page and you've multiplied your peak load by an order of magnitude at the exact moment you can least afford it.&lt;/p&gt;
&lt;h2&gt;
  
  
  Tool 1: timeouts with a budget
&lt;/h2&gt;

&lt;p&gt;A retry without a timeout is a request waiting forever, holding a thread and a connection. Every outbound call needs a timeout, and the timeouts need to fit together:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The user-facing request has an overall &lt;strong&gt;budget&lt;/strong&gt; — say 2 seconds.&lt;/li&gt;
&lt;li&gt;Each hop below gets &lt;strong&gt;less&lt;/strong&gt; than its caller, minus the time already spent. If service A gives B 1.5 s, B can't give C 2 s.&lt;/li&gt;
&lt;li&gt;A deadline can travel with the request (gRPC does this natively; with HTTP you can pass a header) so a service deep in the stack knows when nobody is waiting for its answer any more.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The common mistake is the default: many HTTP clients ship with very long or infinite timeouts. Set them explicitly, per dependency.&lt;/p&gt;
&lt;h2&gt;
  
  
  Tool 2: exponential backoff with jitter
&lt;/h2&gt;

&lt;p&gt;Retrying immediately mostly hits the same failure. Retrying after a delay that &lt;strong&gt;grows&lt;/strong&gt; — 100 ms, 200 ms, 400 ms — gives the dependency time to recover.&lt;/p&gt;

&lt;p&gt;But backoff alone isn't enough. If 10,000 clients all failed at 12:00:00.000 and all wait exactly 100 ms, they all retry at 12:00:00.100 — a synchronized wave. &lt;strong&gt;Jitter&lt;/strong&gt; spreads them out by randomising each delay. AWS's architecture blog has a well-known analysis showing that "full jitter" (a random delay between zero and the backoff ceiling) dramatically reduces both total calls and completion time compared with plain exponential backoff.&lt;/p&gt;

&lt;p&gt;With Resilience4j in a Spring service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;RetryConfig&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RetryConfig&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;custom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;maxAttempts&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;                                   &lt;span class="c1"&gt;// 1 call + 2 retries&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;intervalFunction&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;IntervalFunction&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofExponentialRandomBackoff&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
                &lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofMillis&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt;                   &lt;span class="c1"&gt;// initial interval&lt;/span&gt;
                &lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;                                      &lt;span class="c1"&gt;// multiplier&lt;/span&gt;
                &lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;                                     &lt;span class="c1"&gt;// randomization factor (jitter)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;retryOnException&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="nc"&gt;IOException&lt;/span&gt;
                            &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="nc"&gt;HttpServerErrorException&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ServiceUnavailable&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;

&lt;span class="nc"&gt;Retry&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"inventory"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="nc"&gt;Supplier&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Stock&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;decorateSupplier&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;retry&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;inventoryClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getStock&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;retryOnException&lt;/code&gt; predicate: retry &lt;strong&gt;transient&lt;/strong&gt; failures (I/O errors, 503s, 429s), never &lt;code&gt;400 Bad Request&lt;/code&gt; or &lt;code&gt;404 Not Found&lt;/code&gt;. Those will fail the same way every time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tool 3: retry in one place, and only what's safe
&lt;/h2&gt;

&lt;p&gt;Two rules remove most of the amplification:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Retry at one layer.&lt;/strong&gt; Usually that's the layer closest to the failing dependency, or the edge, but not both. If the service right above the database already retries with backoff, the services above it should fail fast and let their own callers decide.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retry only idempotent operations.&lt;/strong&gt; A &lt;code&gt;GET&lt;/code&gt; is safe. A &lt;code&gt;POST /payments&lt;/code&gt; is only safe to retry if it carries an idempotency key the server honours — otherwise a retry after a lost response charges twice. (I wrote about that pattern in &lt;a href="https://ayushwing.com/blog/idempotency-keys" rel="noopener noreferrer"&gt;Idempotency keys: making retries safe for money&lt;/a&gt;.)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A &lt;strong&gt;retry budget&lt;/strong&gt; is a useful extra guard: allow retries to add at most, say, 10% on top of normal traffic. When the budget is exhausted, fail fast. Envoy and several service meshes support this directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tool 4: circuit breakers
&lt;/h2&gt;

&lt;p&gt;When a dependency is clearly down, even well-behaved retries are wasted work. A &lt;strong&gt;circuit breaker&lt;/strong&gt; tracks recent failures and, past a threshold, "opens": calls fail immediately without touching the network. After a cool-down, it lets a few trial calls through ("half-open"); if they succeed, it closes again.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;CircuitBreakerConfig&lt;/span&gt; &lt;span class="n"&gt;cb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;CircuitBreakerConfig&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;custom&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;slidingWindowSize&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;                            &lt;span class="c1"&gt;// look at the last 50 calls&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;failureRateThreshold&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;                         &lt;span class="c1"&gt;// open at 50% failures&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;slowCallDurationThreshold&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofSeconds&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;slowCallRateThreshold&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;                        &lt;span class="c1"&gt;// slow calls count too&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;waitDurationInOpenState&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofSeconds&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;  &lt;span class="c1"&gt;// cool-down&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;permittedNumberOfCallsInHalfOpenState&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Count slow calls, not just errors.&lt;/strong&gt; A dependency that answers in 9 seconds is worse than one that errors in 9 milliseconds — it holds your threads. Slow-call thresholds catch that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Have a fallback.&lt;/strong&gt; When the breaker is open, return something useful: a cached value, a default, a degraded response ("prices updating"), or a clear error. "Open breaker → 500" protects the dependency but not your users.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Putting it together
&lt;/h2&gt;

&lt;p&gt;The order of decorators matters. A sensible stack, from outside in:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Timeout&lt;/strong&gt; for the whole operation (the budget).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retry&lt;/strong&gt; with exponential backoff and jitter, on transient errors only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Circuit breaker&lt;/strong&gt;, so retries stop as soon as the dependency is known to be down.&lt;/li&gt;
&lt;li&gt;The actual call, with its own connect and read timeouts.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And watch it: retry counts, breaker state changes and timeout rates belong on the same dashboard as error rates. A rising retry rate is often the first sign of trouble, minutes before errors reach users.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Retries multiply load across layers: count your layers before you add a retry.&lt;/li&gt;
&lt;li&gt;Always pair retries with &lt;strong&gt;timeouts&lt;/strong&gt;, &lt;strong&gt;exponential backoff&lt;/strong&gt; and &lt;strong&gt;jitter&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Retry only &lt;strong&gt;transient&lt;/strong&gt; errors, only &lt;strong&gt;idempotent&lt;/strong&gt; operations, in &lt;strong&gt;one&lt;/strong&gt; layer.&lt;/li&gt;
&lt;li&gt;Use &lt;strong&gt;circuit breakers&lt;/strong&gt; with slow-call detection and a real fallback.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/retries-without-a-retry-storm" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>resilience</category>
      <category>java</category>
      <category>microservices</category>
    </item>
    <item>
      <title>Kafka consumers in production: ordering, poison messages and lag</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Fri, 06 Nov 2026 08:29:46 +0000</pubDate>
      <link>https://dev.to/ayushwing/kafka-consumers-in-production-ordering-poison-messages-and-lag-2ng7</link>
      <guid>https://dev.to/ayushwing/kafka-consumers-in-production-ordering-poison-messages-and-lag-2ng7</guid>
      <description>&lt;p&gt;Kafka brokers are remarkably reliable. When a Kafka-based system misbehaves, the cause is usually on the consumer side: a message nobody can process, a loop that's too slow, a partition key someone changed. I've built Kafka pipelines for high-volume email personalization and for ML platform services, and these are the rules I'd hand any team before their consumers go to production.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Ordering is per partition, and the key decides the partition
&lt;/h2&gt;

&lt;p&gt;Kafka guarantees order &lt;strong&gt;within a partition&lt;/strong&gt;, not across a topic. The producer's message key decides the partition (by hash), so all messages with the same key land on the same partition, in order.&lt;/p&gt;

&lt;p&gt;That makes the key a design decision:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Events about &lt;strong&gt;one order&lt;/strong&gt; must be processed in order (&lt;code&gt;Created&lt;/code&gt; before &lt;code&gt;Cancelled&lt;/code&gt;) → key by &lt;code&gt;orderId&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Events about &lt;strong&gt;one customer&lt;/strong&gt; must be in order → key by &lt;code&gt;customerId&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;No ordering needed → no key, and you get even spreading.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two warnings. First, &lt;strong&gt;changing the key or the partition count&lt;/strong&gt; moves keys to different partitions; for a while, old and new events for the same order can be processed out of order by different consumers. Plan those changes like a migration. Second, a &lt;strong&gt;hot key&lt;/strong&gt; (one huge customer) means one hot partition, and that partition's consumer becomes your throughput ceiling.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. At-least-once: commit after processing, de-duplicate always
&lt;/h2&gt;

&lt;p&gt;The safe default is: poll, process, &lt;strong&gt;then&lt;/strong&gt; commit the offset. If the consumer crashes after processing but before committing, the messages are redelivered — so you get &lt;strong&gt;at-least-once&lt;/strong&gt; delivery.&lt;/p&gt;

&lt;p&gt;That means duplicates are normal, not exceptional, and processing must be idempotent: record the message's unique ID (or a business key) in the same transaction as the side effect and skip anything already recorded. Committing &lt;strong&gt;before&lt;/strong&gt; processing gives at-most-once instead: no duplicates, but a crash loses messages. For business events, that's rarely the trade you want.&lt;/p&gt;

&lt;p&gt;With Spring Kafka, the defaults already commit after the listener returns. Keep &lt;code&gt;enable.auto.commit=false&lt;/code&gt; and let the container manage it.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Poison messages: don't let one record block a partition
&lt;/h2&gt;

&lt;p&gt;A &lt;strong&gt;poison message&lt;/strong&gt; is one that will never succeed: malformed JSON, a schema the consumer doesn't know, a reference to something that doesn't exist. If the consumer retries it forever, every message behind it on that partition waits — consumer lag climbs, and downstream systems go quiet.&lt;/p&gt;

&lt;p&gt;The fix is to classify errors:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Error&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;th&gt;What to do&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Transient&lt;/td&gt;
&lt;td&gt;Database timeout, downstream 503&lt;/td&gt;
&lt;td&gt;Retry with backoff, a few times&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Permanent&lt;/td&gt;
&lt;td&gt;Deserialisation failure, validation error&lt;/td&gt;
&lt;td&gt;Don't retry; send to a dead-letter topic&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Spring Kafka's &lt;code&gt;DefaultErrorHandler&lt;/code&gt; does exactly this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Bean&lt;/span&gt;
&lt;span class="nc"&gt;DefaultErrorHandler&lt;/span&gt; &lt;span class="nf"&gt;errorHandler&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;KafkaTemplate&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Object&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Object&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;recoverer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;DeadLetterPublishingRecoverer&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;         &lt;span class="c1"&gt;// -&amp;gt; &amp;lt;topic&amp;gt;.DLT&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;backoff&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ExponentialBackOffWithMaxRetries&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;backoff&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setInitialInterval&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;backoff&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setMultiplier&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;handler&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;DefaultErrorHandler&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;recoverer&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;backoff&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;addNotRetryableExceptions&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;DeserializationException&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
                                      &lt;span class="nc"&gt;ValidationException&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;       &lt;span class="c1"&gt;// straight to DLT&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pair it with &lt;code&gt;ErrorHandlingDeserializer&lt;/code&gt; so a record that can't even be deserialised reaches the error handler instead of crashing the consumer in a loop.&lt;/p&gt;

&lt;p&gt;A dead-letter topic is only useful if someone looks at it: alert when it receives messages, and have a way to &lt;strong&gt;replay&lt;/strong&gt; them once the bug is fixed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpu6jebb8yq8nnl9us1i6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpu6jebb8yq8nnl9us1i6.png" alt="A consumer processes a record. Success commits the offset. A transient error retries with backoff. A permanent error, or exhausted retries, sends the record to a dead-letter topic and moves on." width="800" height="397"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Retry what can succeed later; park what never will. &lt;a href="https://ayushwing.com/diagrams/kafka-consumer-errors.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Keep the poll loop fast
&lt;/h2&gt;

&lt;p&gt;A consumer must call &lt;code&gt;poll()&lt;/code&gt; again within &lt;code&gt;max.poll.interval.ms&lt;/code&gt; (5 minutes by default). If processing a batch takes longer — say, an external API call per record that suddenly takes 10 seconds — the broker decides the consumer is dead, triggers a &lt;strong&gt;rebalance&lt;/strong&gt;, and hands its partitions to another consumer, which then hits the same slow API. You get a rebalance storm and almost no progress.&lt;/p&gt;

&lt;p&gt;Ways out:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Lower &lt;code&gt;max.poll.records&lt;/code&gt; so each batch finishes comfortably in time.&lt;/li&gt;
&lt;li&gt;Put &lt;strong&gt;timeouts&lt;/strong&gt; on every external call inside the listener.&lt;/li&gt;
&lt;li&gt;For genuinely long work, hand it off: write a task to a table or another topic and acknowledge the message quickly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Use the &lt;strong&gt;cooperative sticky&lt;/strong&gt; assignor (&lt;code&gt;partition.assignment.strategy=CooperativeStickyAssignor&lt;/code&gt;) so rebalances move only the partitions that need to move instead of stopping every consumer in the group.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. The one metric: consumer lag
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Consumer lag&lt;/strong&gt; is the number of messages produced but not yet processed, per partition. It's the most honest measure of whether your consumer is keeping up, and it catches every problem above:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Lag rising on &lt;strong&gt;one&lt;/strong&gt; partition → a hot key or a poison message.&lt;/li&gt;
&lt;li&gt;Lag rising on &lt;strong&gt;all&lt;/strong&gt; partitions → consumers too slow or too few, or a slow downstream.&lt;/li&gt;
&lt;li&gt;Lag flat but &lt;strong&gt;non-zero for hours&lt;/strong&gt; → something stuck.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Alert on lag (or better, on estimated time-to-catch-up), not on consumer CPU. And scale consumers up to — but not beyond — the partition count: extra consumers in a group sit idle.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist before go-live
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] The partition key matches the ordering the business needs.&lt;/li&gt;
&lt;li&gt;[ ] Processing is idempotent; duplicates are tested.&lt;/li&gt;
&lt;li&gt;[ ] Transient and permanent errors are handled differently; a DLT exists, is alerted on, and can be replayed.&lt;/li&gt;
&lt;li&gt;[ ] Every external call in the listener has a timeout; batches finish well inside &lt;code&gt;max.poll.interval.ms&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] Lag per partition is on a dashboard with an alert.&lt;/li&gt;
&lt;li&gt;[ ] You know what happens when you deploy: consumers restart, partitions rebalance, and nothing is lost or duplicated beyond what idempotency handles.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Ordering is per partition; the &lt;strong&gt;key&lt;/strong&gt; is a design decision.&lt;/li&gt;
&lt;li&gt;At-least-once is the default: &lt;strong&gt;de-duplicate&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dead-letter&lt;/strong&gt; poison messages instead of retrying them forever.&lt;/li&gt;
&lt;li&gt;Keep the &lt;strong&gt;poll loop&lt;/strong&gt; fast, and alert on &lt;strong&gt;lag&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/kafka-consumers-in-production" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kafka</category>
      <category>distributedsystems</category>
      <category>java</category>
      <category>eventdriven</category>
    </item>
    <item>
      <title>GraphQL Federation with Java subgraphs, without the N+1</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Thu, 05 Nov 2026 17:59:50 +0000</pubDate>
      <link>https://dev.to/ayushwing/graphql-federation-with-java-subgraphs-without-the-n1-15i9</link>
      <guid>https://dev.to/ayushwing/graphql-federation-with-java-subgraphs-without-the-n1-15i9</guid>
      <description>&lt;p&gt;When I worked on GraphQL APIs that sat in front of several backend services, the hard part was never the query language. It was the joins: one screen needs a product from one service, its reviews from another and each reviewer's name from a third. Do that in a single GraphQL server and you get a "god schema" that every team must change. Do it naively across services and you get a storm of small requests.&lt;/p&gt;

&lt;p&gt;Apollo Federation is one answer: each team owns a &lt;strong&gt;subgraph&lt;/strong&gt; for its own types, and a &lt;strong&gt;router&lt;/strong&gt; composes them into one &lt;strong&gt;supergraph&lt;/strong&gt;. I built &lt;a href="https://github.com/ayushwing/graphql-federation-playground" rel="noopener noreferrer"&gt;graphql-federation-playground&lt;/a&gt; to understand the patterns you hit in production, with Java subgraphs instead of the usual Node examples.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Service&lt;/th&gt;
&lt;th&gt;Port&lt;/th&gt;
&lt;th&gt;Owns&lt;/th&gt;
&lt;th&gt;Extends&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;user-subgraph&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;8081&lt;/td&gt;
&lt;td&gt;&lt;code&gt;User&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;product-subgraph&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;8082&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Product&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;User&lt;/code&gt; with &lt;code&gt;products&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;review-subgraph&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;8083&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Review&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Product&lt;/code&gt; with &lt;code&gt;reviews&lt;/code&gt;, &lt;code&gt;averageRating&lt;/code&gt;, &lt;code&gt;reviewCount&lt;/code&gt;; &lt;code&gt;User&lt;/code&gt; with &lt;code&gt;reviews&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apollo Router&lt;/td&gt;
&lt;td&gt;4000&lt;/td&gt;
&lt;td&gt;the supergraph&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each subgraph is a Spring Boot 3 app using the &lt;a href="https://netflix.github.io/dgs/" rel="noopener noreferrer"&gt;Netflix DGS framework&lt;/a&gt;. A single &lt;code&gt;docker compose up&lt;/code&gt; boots all three, the router and Jaeger.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v49fibd4iq2dj9kjk0f.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v49fibd4iq2dj9kjk0f.png" alt="A client sends one query to Apollo Router, which plans sub-queries to the user, product and review subgraphs and joins their results by entity key." width="799" height="289"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The router plans the query; each subgraph resolves only the fields it owns. &lt;a href="https://ayushwing.com/diagrams/graphql-federation.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Entities: a type with a key
&lt;/h2&gt;

&lt;p&gt;The idea that makes federation work is the &lt;strong&gt;entity&lt;/strong&gt;: a type that several subgraphs can contribute fields to, identified by a key. The review subgraph doesn't own &lt;code&gt;Product&lt;/code&gt;, but it can add fields to it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight graphql"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;"&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="err"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;extends&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;external&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Review&lt;/span&gt;&lt;span class="p"&gt;!]!&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;averageRating&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Float&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;reviewCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Int&lt;/span&gt;&lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When a client asks for a product's name and its reviews, the router first fetches the product from the product subgraph, then sends the review subgraph an &lt;code&gt;_entities&lt;/code&gt; query containing just &lt;code&gt;{ __typename: "Product", id: "p1" }&lt;/code&gt;. The review subgraph turns that key into an object and resolves the fields it owns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@DgsEntityFetcher&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"Product"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Product&lt;/span&gt; &lt;span class="nf"&gt;fetchProductEntity&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Object&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;Product&lt;/span&gt;&lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"id"&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reverse direction is equally small. A &lt;code&gt;Review&lt;/code&gt; stores only an &lt;code&gt;authorId&lt;/code&gt;, so the review subgraph returns a &lt;strong&gt;reference&lt;/strong&gt; and lets the router fetch the rest from the user subgraph:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@DgsData&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parentType&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"Review"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"author"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Object&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;author&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;DataFetchingEnvironment&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Review&lt;/span&gt; &lt;span class="n"&gt;review&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getSource&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"__typename"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"User"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"id"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;review&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getAuthorId&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No service calls another service directly. Ownership stays clean: the user team can change how users are stored without the review team noticing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The N+1 problem, multiplied
&lt;/h2&gt;

&lt;p&gt;Here's the catch. Ask for 50 products with their reviews and the router sends one &lt;code&gt;_entities&lt;/code&gt; request with 50 product keys. If &lt;code&gt;Product.reviews&lt;/code&gt; runs a repository query per product, that's 50 queries for one request — and in federation the pattern repeats at every hop.&lt;/p&gt;

&lt;p&gt;The fix is a &lt;strong&gt;DataLoader&lt;/strong&gt;: resolvers ask for a key, the loader collects every key requested in the same execution tick, and fetches them in one batch.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@DgsDataLoader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"reviewsByProduct"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ReviewsByProductDataLoader&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="nc"&gt;BatchLoader&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="nd"&gt;@Override&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;CompletionStage&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;productIds&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;CompletableFuture&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;supplyAsync&lt;/span&gt;&lt;span class="o"&gt;(()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;byProduct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
                    &lt;span class="n"&gt;reviewRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;findByProductIds&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;copyOf&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;productIds&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
            &lt;span class="c1"&gt;// results must come back in the same order as the keys&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;productIds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;map&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;byProduct&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getOrDefault&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;()))&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toList&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
        &lt;span class="o"&gt;},&lt;/span&gt; &lt;span class="n"&gt;executor&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The resolver then becomes a one-liner returning a future:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@DgsData&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parentType&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"Product"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"reviews"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;CompletableFuture&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;productReviews&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;DataFetchingEnvironment&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Product&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getSource&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;getDataLoader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"reviewsByProduct"&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;load&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details worth calling out: the batch function &lt;strong&gt;must&lt;/strong&gt; return results in key order (a missing product gets an empty list, not a gap), and keys are de-duplicated before they hit the repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure every resolver
&lt;/h2&gt;

&lt;p&gt;I added a &lt;code&gt;ResolverTimingInstrumentation&lt;/code&gt; to each subgraph. It extends graphql-java's &lt;code&gt;SimplePerformantInstrumentation&lt;/code&gt;, times every field fetch, records a Micrometer timer &lt;code&gt;graphql.resolver.duration&lt;/code&gt; tagged with parent type, field and status, and logs anything slower than a threshold (&lt;code&gt;observability.graphql.slow-threshold-ms&lt;/code&gt;, default 50 ms) at WARN.&lt;/p&gt;

&lt;p&gt;That is how I found the N+1 that's still in the code — and I've left it there as the example for this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  The N+1 I didn't fix (yet)
&lt;/h2&gt;

&lt;p&gt;Look at &lt;code&gt;averageRating&lt;/code&gt; and &lt;code&gt;reviewCount&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@DgsData&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parentType&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"Product"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"averageRating"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Double&lt;/span&gt; &lt;span class="nf"&gt;averageRating&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;DataFetchingEnvironment&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Product&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getSource&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Review&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;reviews&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reviewRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;findByProductId&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
    &lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both call the repository per product. Request &lt;code&gt;reviews&lt;/code&gt;, &lt;code&gt;averageRating&lt;/code&gt; and &lt;code&gt;reviewCount&lt;/code&gt; for 50 products and you get one batched query plus 100 individual ones. The timing logs show it immediately: the batched field takes one slow-ish call, the other two fields take many fast ones that add up.&lt;/p&gt;

&lt;p&gt;The fix is to derive both from the same DataLoader — load the reviews once per product and compute the average and count from the cached list. DataLoaders cache per request, so the second and third field reuse the first fetch for free.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do differently in production
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Bound the DataLoader thread pool.&lt;/strong&gt; The playground uses &lt;code&gt;Executors.newCachedThreadPool()&lt;/code&gt;, which grows without limit under load. Use a fixed pool sized to your database connection pool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add query cost limits at the router.&lt;/strong&gt; Federation makes deep queries cheap to write and expensive to run. Depth and cost limits belong at the edge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat the supergraph schema as a contract.&lt;/strong&gt; Run schema composition checks in CI for every subgraph change, so a breaking change fails a build instead of the router.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Federation is about &lt;strong&gt;ownership&lt;/strong&gt;: each team owns its types and contributes fields to shared entities by key.&lt;/li&gt;
&lt;li&gt;Every field that crosses an entity boundary needs a &lt;strong&gt;DataLoader&lt;/strong&gt;, or your API slows down as lists grow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-resolver timing&lt;/strong&gt; turns "the API feels slow" into a specific field name.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The full playground — three subgraphs, router config, subscriptions and tests — is on &lt;a href="https://github.com/ayushwing/graphql-federation-playground" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/graphql-federation-java" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>graphql</category>
      <category>java</category>
      <category>microservices</category>
      <category>performance</category>
    </item>
    <item>
      <title>Design docs that get approved</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Wed, 04 Nov 2026 07:09:08 +0000</pubDate>
      <link>https://dev.to/ayushwing/design-docs-that-get-approved-2a6a</link>
      <guid>https://dev.to/ayushwing/design-docs-that-get-approved-2a6a</guid>
      <description>&lt;p&gt;At staff level, much of your impact flows through documents: design docs, proposals, decision records. A good one gets a team aligned in a week; a bad one generates forty comments and a meeting that ends with "let's take this offline".&lt;/p&gt;

&lt;p&gt;The difference is rarely writing quality. It's whether the doc is built around a &lt;strong&gt;decision&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one-page structure
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Context
&lt;/h3&gt;

&lt;p&gt;What's true today, with numbers. Who's affected. Why now.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Order events are published after the database commit. During deploys we lose events: 37 lost last month, each fixed by hand. Three teams consume these events.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;(Numbers in examples are illustrative.)&lt;/em&gt; Two or three paragraphs. If the context is long, link to it.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Goals and non-goals
&lt;/h3&gt;

&lt;p&gt;Goals are what success looks like, measurably. &lt;strong&gt;Non-goals&lt;/strong&gt; are what this doc deliberately doesn't solve.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Goal:&lt;/strong&gt; no order event is lost on crash or deploy. &lt;strong&gt;Non-goal:&lt;/strong&gt; exactly-once delivery; consumers already de-duplicate. &lt;strong&gt;Non-goal:&lt;/strong&gt; migrating other event types this quarter.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Non-goals prevent half the review comments. "What about X?" — "Out of scope, see non-goals" is a much shorter conversation.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Options considered
&lt;/h3&gt;

&lt;p&gt;At least two &lt;strong&gt;real&lt;/strong&gt; options, plus "do nothing" when it's plausible. A table makes trade-offs comparable:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Transactional outbox + polling&lt;/th&gt;
&lt;th&gt;Outbox + CDC (Debezium)&lt;/th&gt;
&lt;th&gt;Distributed transaction&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Loses events on crash?&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;New infrastructure&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;Kafka Connect + Debezium&lt;/td&gt;
&lt;td&gt;XA coordinator&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Publish latency&lt;/td&gt;
&lt;td&gt;~0.5 s&lt;/td&gt;
&lt;td&gt;&amp;lt; 100 ms&lt;/td&gt;
&lt;td&gt;Synchronous&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Effort&lt;/td&gt;
&lt;td&gt;1 sprint&lt;/td&gt;
&lt;td&gt;2–3 sprints&lt;/td&gt;
&lt;td&gt;High, and fragile&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A doc with one option isn't a design doc; it's an announcement, and reviewers will invent the alternatives for you in the comments.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. The decision
&lt;/h3&gt;

&lt;p&gt;State it plainly, and say why the winner wins &lt;strong&gt;for this context&lt;/strong&gt;. "Outbox with polling: no new infrastructure, latency is acceptable for these consumers, and we can move to CDC later without changing producers."&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Risks and early signals
&lt;/h3&gt;

&lt;p&gt;What could go wrong, and how you'll notice early. "Outbox table growth — alert on oldest unpublished row older than 60 s; nightly cleanup job."&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Rollout and rollback
&lt;/h3&gt;

&lt;p&gt;How it ships in steps, and how each step is undone. If the rollback is "restore from backup", the rollout needs more steps.&lt;/p&gt;

&lt;p&gt;That's the whole doc. One to three pages. Diagrams are welcome where they replace paragraphs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running the review
&lt;/h2&gt;

&lt;p&gt;The doc is half the work. The other half is how you take it through review:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Pre-read with the toughest reviewer, privately.&lt;/strong&gt; The person most likely to object should see it first, one-to-one. Their objections improve the doc, and they arrive at the group review as a co-author rather than a critic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Say what you need.&lt;/strong&gt; "Looking for a decision between options A and B by Friday", not "feedback welcome".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Comments in the doc, decisions in the doc.&lt;/strong&gt; When a thread resolves, update the text. A doc whose body disagrees with its comments confuses everyone who reads it later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time-box.&lt;/strong&gt; If a review meeting is needed, 30 minutes, with the decision as the agenda.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Record the outcome.&lt;/strong&gt; Add a short "Decision" block at the top with the date and who agreed. Six months later, that's what people will look for.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Lightweight decisions: ADRs
&lt;/h2&gt;

&lt;p&gt;Not every decision needs a full design doc. For smaller ones, an &lt;strong&gt;Architecture Decision Record&lt;/strong&gt; — title, context, decision, consequences, in half a page, stored in the repo next to the code — is enough. The point is the same: future engineers can find &lt;em&gt;why&lt;/em&gt; things are the way they are, instead of reverse-engineering it from the code or from someone's memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common reasons docs stall
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Endless "what about X?" comments&lt;/td&gt;
&lt;td&gt;No non-goals&lt;/td&gt;
&lt;td&gt;Add them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reviewers propose alternatives&lt;/td&gt;
&lt;td&gt;Only one option presented&lt;/td&gt;
&lt;td&gt;Present 2–3 with a trade-off table&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long debate, no outcome&lt;/td&gt;
&lt;td&gt;No decision asked for&lt;/td&gt;
&lt;td&gt;State the decision needed and the deadline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Approved, then re-litigated&lt;/td&gt;
&lt;td&gt;Outcome not recorded&lt;/td&gt;
&lt;td&gt;Add a decision block with date and approvers&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Build the doc around a &lt;strong&gt;decision&lt;/strong&gt;, not a description.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Non-goals&lt;/strong&gt; and an &lt;strong&gt;options table&lt;/strong&gt; do most of the work.&lt;/li&gt;
&lt;li&gt;Pre-review with the &lt;strong&gt;toughest reviewer&lt;/strong&gt; first.&lt;/li&gt;
&lt;li&gt;Record the &lt;strong&gt;outcome&lt;/strong&gt; where people will find it later.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/design-docs-that-get-approved" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>leadership</category>
      <category>career</category>
      <category>documentation</category>
    </item>
    <item>
      <title>The transactional outbox: stop losing events between your database and Kafka</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Tue, 03 Nov 2026 02:16:34 +0000</pubDate>
      <link>https://dev.to/ayushwing/the-transactional-outbox-stop-losing-events-between-your-database-and-kafka-4adp</link>
      <guid>https://dev.to/ayushwing/the-transactional-outbox-stop-losing-events-between-your-database-and-kafka-4adp</guid>
      <description>&lt;p&gt;This is one of the most common bugs in event-driven systems, and it hides in two innocent lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="n"&gt;orderRepository&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;save&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;                 &lt;span class="c1"&gt;// 1. commit to the database&lt;/span&gt;
&lt;span class="n"&gt;kafkaTemplate&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;send&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"orders"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orderCreated&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;  &lt;span class="c1"&gt;// 2. publish the event&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the process dies between line 1 and line 2 — a deploy, an OOM kill, a lost connection to the broker — the order exists, but the event never goes out. Billing never bills, the warehouse never ships, and nothing errors, because from every service's point of view nothing happened.&lt;/p&gt;

&lt;p&gt;Swapping the lines doesn't help. Publish first, and if the database commit then fails you've announced an order that doesn't exist.&lt;/p&gt;

&lt;p&gt;The underlying problem is a &lt;strong&gt;dual write&lt;/strong&gt;: two systems, no shared transaction. You can't make a Postgres commit and a Kafka publish atomic without distributed transactions, which most teams rightly avoid. But you &lt;em&gt;can&lt;/em&gt; make two writes to the same database atomic. That's the whole idea of the outbox.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw6l1uglfwlfp5s0pje2o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw6l1uglfwlfp5s0pje2o.png" alt="The service writes the order and an outbox row in one database transaction. A relay reads unpublished outbox rows and publishes them to Kafka, then marks them published. Consumers process events idempotently." width="799" height="469"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The business write and the event are committed together; publishing happens afterwards, as often as needed. &lt;a href="https://ayushwing.com/diagrams/transactional-outbox.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;In the &lt;strong&gt;same transaction&lt;/strong&gt; as the business change, insert a row into an &lt;code&gt;outbox&lt;/code&gt; table describing the event.&lt;/li&gt;
&lt;li&gt;A separate &lt;strong&gt;relay&lt;/strong&gt; reads new outbox rows and publishes them to Kafka.&lt;/li&gt;
&lt;li&gt;Once Kafka acknowledges, the relay marks the row as published (or deletes it).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If the service crashes after the commit, the outbox row is still there and the relay publishes it later. If the transaction rolls back, the outbox row rolls back with it. The event exists if and only if the order does.&lt;/p&gt;
&lt;h2&gt;
  
  
  The table and the write
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;outbox&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt;            &lt;span class="n"&gt;BIGSERIAL&lt;/span&gt;    &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;aggregate_id&lt;/span&gt;  &lt;span class="nb"&gt;TEXT&lt;/span&gt;         &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;      &lt;span class="c1"&gt;-- e.g. the order id; used as the Kafka key&lt;/span&gt;
    &lt;span class="n"&gt;event_type&lt;/span&gt;    &lt;span class="nb"&gt;TEXT&lt;/span&gt;         &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;      &lt;span class="c1"&gt;-- OrderCreated, OrderCancelled, ...&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt;       &lt;span class="n"&gt;JSONB&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;created_at&lt;/span&gt;    &lt;span class="n"&gt;TIMESTAMPTZ&lt;/span&gt;  &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;published_at&lt;/span&gt;  &lt;span class="n"&gt;TIMESTAMPTZ&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;outbox_unpublished&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;outbox&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Transactional&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;Order&lt;/span&gt; &lt;span class="nf"&gt;placeOrder&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;PlaceOrder&lt;/span&gt; &lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Order&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;save&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Order&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="n"&gt;outbox&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;save&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OutboxEvent&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="s"&gt;"OrderCreated"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;toJson&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="o"&gt;)));&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// both rows commit together, or neither does&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The partial index keeps the relay's query fast no matter how many published rows you keep around.&lt;/p&gt;
&lt;h2&gt;
  
  
  Relay option 1: polling
&lt;/h2&gt;

&lt;p&gt;The simplest relay is a scheduled loop:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Scheduled&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fixedDelay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nd"&gt;@Transactional&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;relay&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;List&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;OutboxEvent&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;jdbc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;query&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""
        SELECT id, aggregate_id, event_type, payload
        FROM outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT 100
        FOR UPDATE SKIP LOCKED
        """&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;outboxRowMapper&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;OutboxEvent&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;kafka&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;send&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"orders"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;aggregateId&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;()).&lt;/span&gt;&lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;   &lt;span class="c1"&gt;// wait for the ack&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;jdbc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;update&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"UPDATE outbox SET published_at = now() WHERE id = ANY(?)"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt; lets you run several relay instances: each one locks a batch and the others skip it instead of waiting or double-publishing. Using &lt;code&gt;aggregate_id&lt;/code&gt; as the Kafka key keeps all events for one order on one partition, so consumers see them in order.&lt;/p&gt;

&lt;p&gt;Polling is easy to understand, needs no new infrastructure, and its latency (half a second here) is fine for most business events.&lt;/p&gt;

&lt;h2&gt;
  
  
  Relay option 2: change data capture
&lt;/h2&gt;

&lt;p&gt;If you need lower latency or you don't want polling load on the database, use &lt;strong&gt;change data capture&lt;/strong&gt;: a tool like &lt;a href="https://debezium.io" rel="noopener noreferrer"&gt;Debezium&lt;/a&gt; reads the database's replication log (the Postgres WAL, the MySQL binlog) and publishes each new outbox row. Debezium even ships an "outbox event router" that maps outbox columns onto Kafka topics and keys.&lt;/p&gt;

&lt;p&gt;The trade-off is operational: you now run Kafka Connect and Debezium, and you need to manage replication slots so they don't hold back WAL cleanup. For a team already running Kafka Connect it's the better relay. For a team that isn't, polling is usually enough to start.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part people forget: duplicates
&lt;/h2&gt;

&lt;p&gt;The outbox guarantees &lt;strong&gt;at-least-once&lt;/strong&gt;, not exactly-once. If the relay publishes a batch and crashes before marking it published, the next run publishes it again. So consumers must be idempotent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Give every event a unique ID (the outbox &lt;code&gt;id&lt;/code&gt;, or a UUID in the payload).&lt;/li&gt;
&lt;li&gt;In the consumer, record processed event IDs in the &lt;strong&gt;same transaction&lt;/strong&gt; as the consumer's own side effect, and skip IDs it has already seen.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's the same idea as idempotency keys for APIs, applied to messages — see &lt;a href="https://ayushwing.com/blog/idempotency-keys" rel="noopener noreferrer"&gt;Idempotency keys: making retries safe for money&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Operating it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Clean up.&lt;/strong&gt; Delete published rows older than a few days with a scheduled job, or partition the table by day and drop old partitions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Alert on the oldest unpublished row.&lt;/strong&gt; &lt;code&gt;now() - min(created_at) WHERE published_at IS NULL&lt;/code&gt; is the single best health metric: if it grows, events are stuck.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep payloads self-contained.&lt;/strong&gt; Put the data consumers need in the event, rather than making every consumer call back to your API — otherwise you've rebuilt the coupling the events were meant to remove.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;"Commit, then publish" is a dual write, and it loses events on crashes.&lt;/li&gt;
&lt;li&gt;Write the event into an &lt;strong&gt;outbox table in the same transaction&lt;/strong&gt;; publish it afterwards.&lt;/li&gt;
&lt;li&gt;Relay by &lt;strong&gt;polling with &lt;code&gt;SKIP LOCKED&lt;/code&gt;&lt;/strong&gt; to start; move to &lt;strong&gt;CDC&lt;/strong&gt; when latency or load demands it.&lt;/li&gt;
&lt;li&gt;The outbox is at-least-once: &lt;strong&gt;consumers must de-duplicate&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/transactional-outbox" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kafka</category>
      <category>distributedsystems</category>
      <category>microservices</category>
      <category>java</category>
    </item>
    <item>
      <title>Choose a database by access pattern, not by hype</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Mon, 02 Nov 2026 09:25:05 +0000</pubDate>
      <link>https://dev.to/ayushwing/choose-a-database-by-access-pattern-not-by-hype-1o9l</link>
      <guid>https://dev.to/ayushwing/choose-a-database-by-access-pattern-not-by-hype-1o9l</guid>
      <description>&lt;p&gt;Across ten years of backend work I've shipped production systems on MySQL, PostgreSQL, DynamoDB, MongoDB, Elasticsearch, Neo4j, Redis, Redshift and BigQuery. The rule I keep coming back to: write down the &lt;strong&gt;queries&lt;/strong&gt; before you pick the database.&lt;/p&gt;

&lt;p&gt;"We need something that scales" isn't a requirement. "Fetch an order by ID in under 10 ms at 5,000 requests per second", "find products similar to this one by shared attributes", "sum revenue by region for the last two years" — those are requirements, and each one points at a different kind of store.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with an access-pattern list
&lt;/h2&gt;

&lt;p&gt;Before choosing, list for each entity:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Example answer&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;How is it looked up?&lt;/td&gt;
&lt;td&gt;By &lt;code&gt;orderId&lt;/code&gt;; by &lt;code&gt;customerId&lt;/code&gt; + date range&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;What filters and sorts?&lt;/td&gt;
&lt;td&gt;Status, created date, newest first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read/write ratio and peak rate&lt;/td&gt;
&lt;td&gt;20:1, 3,000 reads/s at peak&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consistency needs&lt;/td&gt;
&lt;td&gt;Read-your-own-writes for the owner&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Relationships&lt;/td&gt;
&lt;td&gt;Order → items → products&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Analytics&lt;/td&gt;
&lt;td&gt;Daily revenue by category, two years of history&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Ten minutes on this table saves months of fighting the wrong store.&lt;/p&gt;

&lt;h2&gt;
  
  
  The main families, by the query they're good at
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Relational (PostgreSQL, MySQL) — the default.&lt;/strong&gt; Joins, transactions, constraints, flexible queries you didn't predict. Most systems should start here and leave only when a specific access pattern demands it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;email&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'PAID'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'7 days'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Key-value / wide-column (DynamoDB, Cassandra) — known patterns at huge scale.&lt;/strong&gt; Predictable single-digit-millisecond lookups by key, practically unlimited throughput, and you must design keys around your queries up front. Ad-hoc queries are expensive or impossible.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GetItem   PK = "TENANT#acme"   SK = "PROFILE"
Query     PK = "TENANT#acme"   SK begins_with "EVENT#2026-10"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Document (MongoDB) — aggregates read and written whole.&lt;/strong&gt; Good when an entity is naturally a nested document (a product with variable attributes, a form submission) and you mostly load it by ID.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Search (Elasticsearch, OpenSearch) — text, filters, ranking.&lt;/strong&gt; Full-text relevance, faceted filters, fuzzy matching, aggregations for filter counts. Treat it as a &lt;strong&gt;derived index&lt;/strong&gt;, fed from your source of truth, not as the source of truth itself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"bool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"must"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"match"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"oak dining table"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"filter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"term"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"in_stock"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"range"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"lte"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;40000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}}}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Graph (Neo4j) — relationships several hops deep.&lt;/strong&gt; "Which products share categories and attributes with this one?", "what depends on this feature?". In SQL these become recursive joins; in a graph database they're a short pattern. I've used it for product recommendations and for ML metadata lineage.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight cypher"&gt;&lt;code&gt;&lt;span class="k"&gt;MATCH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="py"&gt;p:&lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt; &lt;span class="ss"&gt;{&lt;/span&gt;&lt;span class="py"&gt;id:&lt;/span&gt; &lt;span class="n"&gt;$id&lt;/span&gt;&lt;span class="ss"&gt;})&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="ss"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;:HAS_ATTRIBUTE&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;IN_CATEGORY&lt;/span&gt;&lt;span class="ss"&gt;]&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="ss"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;:HAS_ATTRIBUTE&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;IN_CATEGORY&lt;/span&gt;&lt;span class="ss"&gt;]&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="py"&gt;other:&lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;other&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;
&lt;span class="k"&gt;RETURN&lt;/span&gt; &lt;span class="n"&gt;other.id&lt;/span&gt;&lt;span class="ss"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;shared&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;shared&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="ss"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Cache (Redis) — hot reads, counters, short-lived state.&lt;/strong&gt; Sessions, rate limits, leaderboards, cached responses. Plan for it being empty after a restart.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Warehouse (BigQuery, Redshift, Snowflake) — scans over history.&lt;/strong&gt; Columnar storage that makes "sum revenue by month and category over two years" cheap. Not for serving user requests.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;DATE_TRUNC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;MONTH&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;revenue&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;analytics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;orders&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;order_date&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;DATE_SUB&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;CURRENT_DATE&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;INTERVAL&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="nb"&gt;YEAR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  When you need two (or three)
&lt;/h2&gt;

&lt;p&gt;Real systems often combine stores: Postgres for orders, Elasticsearch for product search, a warehouse for reporting. That's fine — if you decide three things explicitly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Which store is the source of truth&lt;/strong&gt; for each piece of data. Everything else is derived and can be rebuilt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How derived stores are fed&lt;/strong&gt;: events, change data capture, or batch loads — and how stale they may be. Search results a few seconds behind are usually fine; a balance a few seconds behind usually isn't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How you rebuild&lt;/strong&gt; a derived store from scratch. If you can't re-index search from the source of truth, it has quietly become a second source of truth.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The anti-pattern is the one in the picture at the end of this post: each new feature adds a store because it's fashionable, with no answer to those three questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick decision guide
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;If your dominant query is…&lt;/th&gt;
&lt;th&gt;Start with&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Transactions, joins, flexible filters&lt;/td&gt;
&lt;td&gt;PostgreSQL / MySQL&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lookups by key at very high, predictable scale&lt;/td&gt;
&lt;td&gt;DynamoDB / Cassandra&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Whole nested documents by ID&lt;/td&gt;
&lt;td&gt;MongoDB (or JSONB in Postgres)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full-text search, facets, ranking&lt;/td&gt;
&lt;td&gt;Elasticsearch / OpenSearch, fed from the source of truth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multi-hop relationships&lt;/td&gt;
&lt;td&gt;Neo4j&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Aggregations over large history&lt;/td&gt;
&lt;td&gt;BigQuery / Redshift / Snowflake&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hot reads, counters, ephemeral state&lt;/td&gt;
&lt;td&gt;Redis&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Write the &lt;strong&gt;access-pattern list&lt;/strong&gt; before choosing a database.&lt;/li&gt;
&lt;li&gt;Start &lt;strong&gt;relational&lt;/strong&gt;; move a workload out when a specific query demands it.&lt;/li&gt;
&lt;li&gt;Search indexes and caches are &lt;strong&gt;derived&lt;/strong&gt; — feed them from the source of truth and be able to rebuild them.&lt;/li&gt;
&lt;li&gt;When you use several stores, name the &lt;strong&gt;source of truth&lt;/strong&gt; for every piece of data.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/choose-a-database-by-access-pattern" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>database</category>
      <category>architecture</category>
      <category>systemdesign</category>
      <category>backend</category>
    </item>
    <item>
      <title>Durable workflows vs. a queue and a cron job</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Sun, 01 Nov 2026 07:25:02 +0000</pubDate>
      <link>https://dev.to/ayushwing/durable-workflows-vs-a-queue-and-a-cron-job-1p4h</link>
      <guid>https://dev.to/ayushwing/durable-workflows-vs-a-queue-and-a-cron-job-1p4h</guid>
      <description>&lt;p&gt;Some work doesn't fit in a request. Moving a customer's data to new infrastructure. Onboarding a merchant across five systems. Generating, approving and sending a large batch of documents. These jobs have many steps, take minutes to days, call systems that fail, and must survive deploys and crashes.&lt;/p&gt;

&lt;p&gt;Most teams start the same way: a queue, a cron job and a status column.&lt;/p&gt;

&lt;h2&gt;
  
  
  What queue + cron becomes
&lt;/h2&gt;

&lt;p&gt;Version one is clean: a job row with &lt;code&gt;status = PENDING&lt;/code&gt;, a worker that picks it up, does the work, sets &lt;code&gt;DONE&lt;/code&gt;. Then reality adds requirements, one at a time:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A step fails transiently → add a &lt;code&gt;retries&lt;/code&gt; column and a &lt;code&gt;RETRYING&lt;/code&gt; status.&lt;/li&gt;
&lt;li&gt;A worker dies mid-job → add &lt;code&gt;locked_by&lt;/code&gt; and &lt;code&gt;locked_at&lt;/code&gt;, and a cron that releases stale locks.&lt;/li&gt;
&lt;li&gt;Step 4 must not run twice → add per-step status columns.&lt;/li&gt;
&lt;li&gt;A step needs to wait for a human approval or a 24-hour cool-down → add &lt;code&gt;resume_after&lt;/code&gt; and another cron.&lt;/li&gt;
&lt;li&gt;Step 6 fails permanently after steps 1–5 succeeded → write undo logic, scattered across the codebase.&lt;/li&gt;
&lt;li&gt;Nobody can tell where a job is → build an admin page that reads all these columns.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each addition is reasonable. Together they're a workflow engine — one with no tests for its failure paths, state spread across columns, and behaviour that only one engineer fully understands. (The picture at the end of this post is only slightly exaggerated.)&lt;/p&gt;

&lt;h2&gt;
  
  
  What a durable workflow engine gives you
&lt;/h2&gt;

&lt;p&gt;A durable workflow engine — &lt;a href="https://temporal.io" rel="noopener noreferrer"&gt;Temporal&lt;/a&gt; is the one I've used; Cadence, AWS Step Functions and others share the idea — lets you write the process as ordinary code while the engine persists every step's progress:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@WorkflowInterface&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;DataMoveWorkflow&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nd"&gt;@WorkflowMethod&lt;/span&gt;
    &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;move&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;DataMoveWorkflowImpl&lt;/span&gt; &lt;span class="kd"&gt;implements&lt;/span&gt; &lt;span class="nc"&gt;DataMoveWorkflow&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;MoveActivities&lt;/span&gt; &lt;span class="n"&gt;act&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Workflow&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;newActivityStub&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;MoveActivities&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
            &lt;span class="nc"&gt;ActivityOptions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;newBuilder&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setStartToCloseTimeout&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofMinutes&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setRetryOptions&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;RetryOptions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;newBuilder&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;
                            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setInitialInterval&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofSeconds&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
                            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setBackoffCoefficient&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
                            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setMaximumAttempts&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
                            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;

    &lt;span class="nd"&gt;@Override&lt;/span&gt;
    &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;move&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;Saga&lt;/span&gt; &lt;span class="n"&gt;saga&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Saga&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Saga&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;Builder&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;preflightChecks&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

            &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;provisionTarget&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;saga&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;addCompensation&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nl"&gt;act:&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;releaseTarget&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;copySnapshot&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;syncChanges&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;       &lt;span class="c1"&gt;// catch up writes made during the copy&lt;/span&gt;
            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;verifyChecksums&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;switchTraffic&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;saga&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;addCompensation&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nl"&gt;act:&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;switchTrafficBack&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

            &lt;span class="nc"&gt;Workflow&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;sleep&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofHours&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;     &lt;span class="c1"&gt;// soak period; costs nothing while waiting&lt;/span&gt;
            &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;decommissionSource&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;accountId&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ActivityFailure&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;saga&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;compensate&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;                        &lt;span class="c1"&gt;// undo completed steps in reverse order&lt;/span&gt;
            &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What the engine handles for you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Durability.&lt;/strong&gt; Each completed activity is recorded. If the worker crashes after &lt;code&gt;copySnapshot&lt;/code&gt;, a new worker replays the history and continues at &lt;code&gt;syncChanges&lt;/code&gt; — it doesn't copy again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retries and timeouts per step&lt;/strong&gt;, as configuration rather than columns.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long waits.&lt;/strong&gt; &lt;code&gt;Workflow.sleep(Duration.ofHours(24))&lt;/code&gt; holds no thread and no lock; the engine wakes the workflow when the timer fires. The same goes for waiting on a &lt;strong&gt;signal&lt;/strong&gt;, such as a human approval.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compensation.&lt;/strong&gt; The &lt;code&gt;Saga&lt;/code&gt; helper records how to undo each step and runs the compensations in reverse if a later step fails.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Visibility.&lt;/strong&gt; Every workflow's current step, history and inputs are queryable — the admin page you'd otherwise build.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffg5po187s8y2n2r78ffn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffg5po187s8y2n2r78ffn.png" alt="Workflow steps: preflight, provision, copy, sync, verify, switch traffic, soak, decommission. Failure after provisioning or switching triggers compensation in reverse order." width="800" height="411"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Steps run forward; compensations run backward. &lt;a href="https://ayushwing.com/diagrams/durable-workflow.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The price: determinism
&lt;/h2&gt;

&lt;p&gt;Workflow code is &lt;strong&gt;replayed&lt;/strong&gt; to rebuild state after a crash, so it must make the same decisions every time it runs with the same history. In practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No &lt;code&gt;Thread.sleep&lt;/code&gt;, &lt;code&gt;System.currentTimeMillis()&lt;/code&gt;, &lt;code&gt;UUID.randomUUID()&lt;/code&gt; or &lt;code&gt;Random&lt;/code&gt; in workflow code — use &lt;code&gt;Workflow.sleep&lt;/code&gt;, &lt;code&gt;Workflow.currentTimeMillis()&lt;/code&gt;, &lt;code&gt;Workflow.randomUUID()&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;No I/O in workflow code. Database calls, HTTP calls and file access go in &lt;strong&gt;activities&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Changing a running workflow's logic needs &lt;strong&gt;versioning&lt;/strong&gt; (&lt;code&gt;Workflow.getVersion&lt;/code&gt;), because in-flight workflows replay against the old history.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Activities, on the other hand, are normal code — and because they're retried, they must be &lt;strong&gt;idempotent&lt;/strong&gt; (see &lt;a href="https://ayushwing.com/blog/idempotency-keys" rel="noopener noreferrer"&gt;Idempotency keys&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;These rules take a week to internalise. They also force a clean separation between "what happens in what order" (workflow) and "how each step talks to the world" (activities), which tends to make the code easier to read.&lt;/p&gt;

&lt;h2&gt;
  
  
  When it's worth it — and when it isn't
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Situation&lt;/th&gt;
&lt;th&gt;Queue + worker&lt;/th&gt;
&lt;th&gt;Durable workflow&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;One step, idempotent, seconds long&lt;/td&gt;
&lt;td&gt;✅ Simple and enough&lt;/td&gt;
&lt;td&gt;Overkill&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2–3 steps, no waits, rare failures&lt;/td&gt;
&lt;td&gt;✅ Usually fine&lt;/td&gt;
&lt;td&gt;Optional&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Many steps, external systems, partial failure needs undo&lt;/td&gt;
&lt;td&gt;Grows into a hand-built engine&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Waits of hours or days, human approvals&lt;/td&gt;
&lt;td&gt;Crons and &lt;code&gt;resume_after&lt;/code&gt; columns&lt;/td&gt;
&lt;td&gt;✅ Timers and signals&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;You must answer "where exactly is job X?"&lt;/td&gt;
&lt;td&gt;Build an admin page&lt;/td&gt;
&lt;td&gt;✅ Built in&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The real cost is operational: you run (or pay for) the engine's cluster and learn its model. For one small job, that cost dominates. For a platform that runs long, multi-step processes every day — migrations, onboarding, provisioning — it pays for itself quickly, because every new process gets retries, timers, visibility and compensation for free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A queue + cron + status table grows into a workflow engine; decide early whether to build one or adopt one.&lt;/li&gt;
&lt;li&gt;Durable workflows give &lt;strong&gt;resumability, per-step retries, cheap long waits, compensation and visibility&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The price is &lt;strong&gt;determinism&lt;/strong&gt; in workflow code and &lt;strong&gt;idempotency&lt;/strong&gt; in activities.&lt;/li&gt;
&lt;li&gt;Use it for &lt;strong&gt;long, multi-step, failure-prone&lt;/strong&gt; processes; keep simple jobs simple.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/durable-workflows-vs-queue-and-cron" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>temporal</category>
      <category>distributedsystems</category>
      <category>java</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Idempotency keys: making retries safe for money</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Sat, 31 Oct 2026 16:31:17 +0000</pubDate>
      <link>https://dev.to/ayushwing/idempotency-keys-making-retries-safe-for-money-383c</link>
      <guid>https://dev.to/ayushwing/idempotency-keys-making-retries-safe-for-money-383c</guid>
      <description>&lt;p&gt;A request that times out may still have succeeded. That single fact is why wallet and refund systems — I've built both — have to be designed around retries from day one.&lt;/p&gt;

&lt;p&gt;Here is the sequence. A client calls &lt;code&gt;POST /refunds&lt;/code&gt;. The server debits the merchant, credits the customer, commits — and then the response is lost: a load balancer timeout, a pod restart, a mobile network drop. The client sees an error and does the reasonable thing: it retries. The server, seeing what looks like a new request, refunds again.&lt;/p&gt;

&lt;p&gt;Retries are not the bug. Retries are essential; networks fail all the time. The bug is an API that can't tell a retry from a new request.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern
&lt;/h2&gt;

&lt;p&gt;An &lt;strong&gt;idempotency key&lt;/strong&gt; is a unique value the client generates once per &lt;em&gt;operation&lt;/em&gt; — not per HTTP attempt — and sends with every attempt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /refunds
Idempotency-Key: 6f1c2a9e-7d3b-4f0a-9a51-2c8e0b7d4e11
Content-Type: application/json

{"orderId": "A-1042", "amount": 1499}
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server promises: for a given key, the operation happens &lt;strong&gt;at most once&lt;/strong&gt;, and every attempt gets &lt;strong&gt;the same response&lt;/strong&gt;. Stripe popularised this design for its public API, and it applies to any endpoint that moves money, sends a message or creates something a user would notice twice.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0sf08h3t6ufcf1fwayoy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0sf08h3t6ufcf1fwayoy.png" alt="First attempt: reserve the key, do the work, store the response. Retry with the same key: find the stored response and return it without doing the work again." width="800" height="503"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Reserve first, work second, replay on retry. &lt;a href="https://ayushwing.com/diagrams/idempotency-keys.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The table
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;idempotency_keys&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;key&lt;/span&gt;            &lt;span class="nb"&gt;TEXT&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;client_id&lt;/span&gt;      &lt;span class="nb"&gt;TEXT&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;request_hash&lt;/span&gt;   &lt;span class="nb"&gt;TEXT&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- SHA-256 of method + path + body&lt;/span&gt;
    &lt;span class="n"&gt;status&lt;/span&gt;         &lt;span class="nb"&gt;TEXT&lt;/span&gt;        &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- IN_PROGRESS | COMPLETED&lt;/span&gt;
    &lt;span class="n"&gt;response_code&lt;/span&gt;  &lt;span class="nb"&gt;INT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;response_body&lt;/span&gt;  &lt;span class="n"&gt;JSONB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;created_at&lt;/span&gt;     &lt;span class="n"&gt;TIMESTAMPTZ&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Scope keys per client: two different clients may well generate the same string, and that must not leak one client's response to the other.&lt;/p&gt;

&lt;h2&gt;
  
  
  The flow, with the edge cases
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nd"&gt;@PostMapping&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"/refunds"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;ResponseEntity&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;RefundResponse&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;refund&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nd"&gt;@RequestHeader&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Idempotency-Key"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
                                             &lt;span class="nd"&gt;@RequestBody&lt;/span&gt; &lt;span class="nc"&gt;RefundRequest&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
                                             &lt;span class="nc"&gt;Principal&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;hash&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sha256&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="c1"&gt;// 1. Reserve the key BEFORE doing any work. The primary key makes this atomic.&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;inserted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;jdbc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;update&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""
        INSERT INTO idempotency_keys (key, client_id, request_hash, status)
        VALUES (?, ?, ?, 'IN_PROGRESS')
        ON CONFLICT (client_id, key) DO NOTHING
        """&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getName&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;hash&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;inserted&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;StoredKey&lt;/span&gt; &lt;span class="n"&gt;existing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;keys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;find&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getName&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(!&lt;/span&gt;&lt;span class="n"&gt;existing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;requestHash&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hash&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;ResponseEntity&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;unprocessableEntity&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;   &lt;span class="c1"&gt;// same key, different body&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;existing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nc"&gt;Status&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;IN_PROGRESS&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;ResponseEntity&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;409&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;              &lt;span class="c1"&gt;// concurrent retry: try again later&lt;/span&gt;
        &lt;span class="o"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;existing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toResponse&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;                               &lt;span class="c1"&gt;// replay the stored result&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;// 2. Do the work and record the result in ONE transaction.&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;execute&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;RefundResponse&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;refunds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;issue&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="n"&gt;keys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;complete&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getName&lt;/span&gt;&lt;span class="o"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;201&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;ResponseEntity&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;201&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="o"&gt;});&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each branch exists because of a real failure mode:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reserve before you work.&lt;/strong&gt; If you check for the key, do the refund, and only then insert the key, two retries arriving 5 ms apart both pass the check. The &lt;code&gt;INSERT ... ON CONFLICT DO NOTHING&lt;/code&gt; against the primary key is the lock: exactly one attempt wins the reservation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Same key, different body.&lt;/strong&gt; A client bug that reuses a key for a different refund must not get the first refund's response — that would silently hide the second operation. Hash the request and reject mismatches with a &lt;code&gt;422&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In progress.&lt;/strong&gt; A retry that arrives while the first attempt is still running gets a &lt;code&gt;409&lt;/code&gt;. The client backs off and retries; by then the first attempt has finished and the retry gets the stored response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Work and result commit together.&lt;/strong&gt; If the refund commits but storing the response doesn't, the next retry finds an &lt;code&gt;IN_PROGRESS&lt;/code&gt; row forever. Keep the business write and the key update in the same database transaction.&lt;/p&gt;

&lt;h2&gt;
  
  
  When the work isn't in your database
&lt;/h2&gt;

&lt;p&gt;The single-transaction trick only works when the side effect lives in the same database as the key table. When step 2 calls an external payment provider, you can't roll that back. Two things help:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Pass the key downstream.&lt;/strong&gt; Most payment providers accept their own idempotency key. Derive it from yours (for example &lt;code&gt;refund:{clientId}:{key}&lt;/code&gt;), so a retry after a crash asks the provider for the same operation instead of a new one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recover stuck rows.&lt;/strong&gt; A process can die between the provider call and the commit. A sweeper that finds rows stuck &lt;code&gt;IN_PROGRESS&lt;/code&gt; for longer than your longest timeout can query the provider by the derived key and complete or release the row.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How long to keep keys
&lt;/h2&gt;

&lt;p&gt;Keep keys at least as long as any client will retry, plus a margin. Stripe keeps them for 24 hours; for internal services where queues can redeliver messages days later, I'd lean longer. Delete old rows with a scheduled job — not inside the request path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where else this applies
&lt;/h2&gt;

&lt;p&gt;The same idea shows up whenever "at least once" delivery meets "exactly once" side effects:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Queue consumers.&lt;/strong&gt; Kafka and SQS redeliver. Use the message ID (or a business ID) as the key and record it in the same transaction as the effect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Webhooks.&lt;/strong&gt; Providers retry webhooks; store the event ID before acting on it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scheduled jobs.&lt;/strong&gt; A job that runs twice after a deploy should find its own key and stop.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Retries are necessary; duplicate side effects are optional.&lt;/li&gt;
&lt;li&gt;The key belongs to the &lt;strong&gt;operation&lt;/strong&gt;, generated by the client once and reused on every attempt.&lt;/li&gt;
&lt;li&gt;Reserve the key atomically &lt;strong&gt;before&lt;/strong&gt; the work, store the response &lt;strong&gt;with&lt;/strong&gt; the work, and replay it on retry.&lt;/li&gt;
&lt;li&gt;Reject a reused key with a different body, and pass keys to every downstream system that moves money.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/idempotency-keys" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>payments</category>
      <category>java</category>
      <category>api</category>
    </item>
    <item>
      <title>Same AI, two developers: why the experienced one spends fewer tokens</title>
      <dc:creator>Ayush Soni</dc:creator>
      <pubDate>Fri, 30 Oct 2026 03:36:05 +0000</pubDate>
      <link>https://dev.to/ayushwing/same-ai-two-developers-why-the-experienced-one-spends-fewer-tokens-gd8</link>
      <guid>https://dev.to/ayushwing/same-ai-two-developers-why-the-experienced-one-spends-fewer-tokens-gd8</guid>
      <description>&lt;p&gt;Everyone is building apps with AI now. My cousin built one. Your manager built one. Somebody built one during a meeting about whether we should build one.&lt;/p&gt;

&lt;p&gt;That's genuinely great. But it raises a fair question: if a first-time builder and an engineer with ten years of experience use &lt;strong&gt;the same model&lt;/strong&gt; with &lt;strong&gt;the same prompt&lt;/strong&gt;, what's actually different?&lt;/p&gt;

&lt;p&gt;Let's run the experiment in our heads. Two developers, same AI, same request:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Build me a task-tracker app with user login."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Meet &lt;strong&gt;Nia&lt;/strong&gt;, the native AI builder: smart, fast, has never shipped software before. And &lt;strong&gt;Sam&lt;/strong&gt;, the seasoned engineer, who has broken production enough times to be careful.&lt;/p&gt;

&lt;h2&gt;
  
  
  Day 1: no difference at all
&lt;/h2&gt;

&lt;p&gt;Both get a working demo in an afternoon. Login page, task list, a nice dark theme, a little confetti when you finish a task. Both post a screenshot. Both get 40 likes.&lt;/p&gt;

&lt;p&gt;If you only look at day 1, experience doesn't matter. This is the part of the story everyone's excited about, and they're right to be. The AI did in an afternoon what used to take a week.&lt;/p&gt;

&lt;p&gt;The difference shows up later, the way it always does in software: &lt;strong&gt;in the second month, when you change things.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the tokens go
&lt;/h2&gt;

&lt;p&gt;Here's Nia's day 3. A user reports that tasks sometimes disappear.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Nia:  tasks disappear sometimes, fix it
AI:   (rewrites the storage layer)
Nia:  now login is broken
AI:   (rewrites the auth flow)
Nia:  login works but tasks still disappear
AI:   (adds a retry, a cache, and a second database "for reliability")
Nia:  it's slower now and still broken. FIX IT PROPERLY
AI:   (apologises, rewrites the storage layer again)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every one of those messages drags the whole conversation and a growing chunk of the codebase into context again. Every fix touches more files than the bug lives in. After a week the app works most of the time, nobody knows why, and the token bill looks like a phone number.&lt;/p&gt;

&lt;p&gt;Sam's day 3, same bug:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sam:  Read CLAUDE.md. Tasks occasionally disappear after editing.
      Don't change code yet: find where tasks are written and list
      the ways two writes could race.
AI:   Two tabs saving the same task: last write wins, no version check.
Sam:  Write a failing test that reproduces it with two concurrent saves.
AI:   (test fails, as expected)
Sam:  Fix it with a version column and a conditional update. Keep the diff small.
AI:   (12-line change, test passes)
Sam:  (reads the diff, commits)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four messages. One small diff. A test that will catch the bug if it ever comes back. Sam spent &lt;strong&gt;fewer&lt;/strong&gt; tokens than Nia, not more. Experience didn't make Sam write better prompts in some magic way. It made Sam ask for &lt;strong&gt;smaller, checkable steps&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;(If that bug looks familiar, it's the same race I wrote about in &lt;a href="https://ayushwing.com/blog/business-workflows-as-state-machines" rel="noopener noreferrer"&gt;Model business workflows as state machines&lt;/a&gt;. AI doesn't remove these problems. It just reaches them faster.)&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F69aqi11w9fqsu0fpb8px.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F69aqi11w9fqsu0fpb8px.png" alt="Two loops. Prompt and pray: ask for a whole feature, run it, see something broken, say fix it, repeat. Plan, test, ship: plan, write a failing test, make one small change, read the diff, commit, then take the next step." width="799" height="380"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Same model, two loops. Only one of them ends. &lt;a href="https://ayushwing.com/diagrams/two-ai-loops.html" rel="noopener noreferrer"&gt;Explore the interactive diagram&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The side-by-side
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Nia (first-time builder)&lt;/th&gt;
&lt;th&gt;Sam (experienced engineer)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;First prompt&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;"Build the whole app"&lt;/td&gt;
&lt;td&gt;"Plan the app; don't write code yet"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Size of each request&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;A feature, or "fix everything"&lt;/td&gt;
&lt;td&gt;One change you can review in two minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Re-explains the project every session&lt;/td&gt;
&lt;td&gt;A project instructions file the AI reads every time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Definition of done&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;"It looks like it works"&lt;/td&gt;
&lt;td&gt;A test passes, and the diff makes sense&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;When it breaks&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;"Fix it" (again)&lt;/td&gt;
&lt;td&gt;"Explain why before changing anything"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Tools&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Pastes logs and docs into the chat&lt;/td&gt;
&lt;td&gt;Gives the AI tools: skills, plugins, MCP servers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Dependencies&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Whatever the AI suggested&lt;/td&gt;
&lt;td&gt;Asks why each one is needed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Secrets&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;API key in the code, "temporarily"&lt;/td&gt;
&lt;td&gt;Environment variables from the start&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Git&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;One giant commit called "final"&lt;/td&gt;
&lt;td&gt;A commit after every green step&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Day 30&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Afraid to touch it&lt;/td&gt;
&lt;td&gt;Adds features on a Friday&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of Sam's column is about AI. It's just engineering: small steps, tests, version control, knowing what "done" means. &lt;strong&gt;AI is a multiplier, and it multiplies whatever habits you bring.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What makes AI-written code sustainable
&lt;/h2&gt;

&lt;p&gt;"Sustainable" is a fancy word for one question: &lt;strong&gt;can someone change this code next month without breaking it?&lt;/strong&gt; "Someone" includes you, and it includes the AI.&lt;/p&gt;

&lt;p&gt;Four things decide the answer:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Tests that describe behaviour.&lt;/strong&gt; A test suite is the only spec the AI can't misread. With tests, "refactor this" is safe. Without them, every change is a gamble the AI makes with confidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small, readable diffs.&lt;/strong&gt; If you can't explain a change in a sentence, you can't review it, and unreviewed code is how plausible-but-wrong bugs ship. My favourite: a refund endpoint that retries on timeout and pays twice. It looks perfectly correct in a diff (&lt;a href="https://ayushwing.com/blog/idempotency-keys" rel="noopener noreferrer"&gt;here's how to prevent it&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Written-down decisions.&lt;/strong&gt; A project instructions file (&lt;code&gt;CLAUDE.md&lt;/code&gt;, &lt;code&gt;AGENTS.md&lt;/code&gt;, &lt;code&gt;.cursorrules&lt;/code&gt;, whatever your tool reads) that says: how to run the tests, what the architecture is, what not to touch. The AI stops rediscovering your codebase every session, and that's most of the token savings right there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Boring choices.&lt;/strong&gt; One database, not five. A library everyone knows, not the one the AI half-remembers. AI is very good at writing glue for popular tools and very bad at debugging an obscure one.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Skills, plugins and tools: stop pasting, start equipping
&lt;/h2&gt;

&lt;p&gt;The biggest token sink I see is &lt;strong&gt;pasting&lt;/strong&gt;: the same error log, the same docs page, the same "here's how our project works" paragraph, over and over.&lt;/p&gt;

&lt;p&gt;Experienced developers give the AI &lt;strong&gt;tools&lt;/strong&gt; instead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Skills and plugins&lt;/strong&gt; package a repeatable way of doing something, such as a review checklist, a deploy process or an audit, so you don't explain it every time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP servers&lt;/strong&gt; let the AI read your tickets, query a staging database or check a deploy itself, instead of you copying the output in. (I built &lt;a href="https://ayushwing.com/blog/java-mcp-server" rel="noopener noreferrer"&gt;one for Java services&lt;/a&gt; for exactly this reason.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your own scripts:&lt;/strong&gt; a test command, a screenshot script, a linter. The AI can run them and check its own work.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A real example: this site. While building it with an AI assistant:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/tt-a1i/archify" rel="noopener noreferrer"&gt;Archify&lt;/a&gt; drew the diagrams, with its own validation checks, instead of the AI hand-placing boxes.&lt;/li&gt;
&lt;li&gt;An SEO audit skill (&lt;a href="https://github.com/AgricIDaniel/claude-seo" rel="noopener noreferrer"&gt;claude-seo&lt;/a&gt;) ran its checklist against the built site and found a real bug: &lt;code&gt;robots.txt&lt;/code&gt; pointed at a sitemap that no longer existed.&lt;/li&gt;
&lt;li&gt;Every visual change was screenshotted in a headless browser before it shipped, which caught a meme overflowing on phones and a diagram that rendered as a broken image.&lt;/li&gt;
&lt;li&gt;And the homepage stack section went through two designs I didn't like. Before the third attempt, I was asked to pick a concept from four sketches. That one stuck.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one isn't a tool at all. It's the oldest engineering habit there is: &lt;strong&gt;agree on what you're building before you build it.&lt;/strong&gt; It turns out it works on AI too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Signs you're vibe coding
&lt;/h2&gt;

&lt;p&gt;Score yourself honestly. One point each:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Your most-used prompt is "fix it".&lt;/li&gt;
&lt;li&gt;[ ] You've said "PLEASE" to a compiler in capital letters.&lt;/li&gt;
&lt;li&gt;[ ] Your app has 3 state-management libraries and you know the name of one.&lt;/li&gt;
&lt;li&gt;[ ] There is a file called &lt;code&gt;utils2_final_REAL.ts&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] Your git history is one commit: "init".&lt;/li&gt;
&lt;li&gt;[ ] You don't know what happens if two users click Save at the same time.&lt;/li&gt;
&lt;li&gt;[ ] The AI added a cache "for performance" and you were too scared to ask why.&lt;/li&gt;
&lt;li&gt;[ ] Your API key is in the frontend, but "only for now".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;0–2:&lt;/strong&gt; you're Sam. &lt;strong&gt;3–5:&lt;/strong&gt; you're on the way. &lt;strong&gt;6–8:&lt;/strong&gt; you're Nia, and that's fine. Every Sam started as a Nia. The good news is that every habit in Sam's column can be learned, and the AI will happily help you learn it if you ask it to explain instead of just fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  The playbook
&lt;/h2&gt;

&lt;p&gt;If you want to work like Sam starting tomorrow, here's the whole thing:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Plan before code.&lt;/strong&gt; Ask for a plan and push back on it before any code exists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write a project instructions file&lt;/strong&gt; on day 1: how to run, how to test, the rules.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One change per request&lt;/strong&gt;, small enough to review in two minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test first&lt;/strong&gt; for anything that matters: write the failing test, then the fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read every diff.&lt;/strong&gt; If you don't understand it, ask "why" before you accept it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commit after every green step&lt;/strong&gt;, so you can always go back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Equip, don't paste:&lt;/strong&gt; skills, plugins, MCP servers and scripts the AI can run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When stuck, ask for a diagnosis, not a fix.&lt;/strong&gt; "Explain why this fails" costs less than five rewrites.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On day 1, AI makes everyone equal. By day 30, habits make the difference.&lt;/li&gt;
&lt;li&gt;Experienced developers often spend &lt;strong&gt;fewer&lt;/strong&gt; tokens, because small steps, tests and saved context beat "fix it" loops.&lt;/li&gt;
&lt;li&gt;Sustainable AI-written code has &lt;strong&gt;tests, small diffs, written decisions and boring choices&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Give the AI &lt;strong&gt;tools&lt;/strong&gt;, not copy-paste.&lt;/li&gt;
&lt;li&gt;Experience isn't about typing faster anymore. It's about knowing what to ask for, and knowing when the answer is wrong.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://ayushwing.com/blog/same-ai-two-developers" rel="noopener noreferrer"&gt;ayushwing.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>softwareengineering</category>
      <category>career</category>
    </item>
  </channel>
</rss>
