<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Azam Akram</title>
    <description>The latest articles on DEV Community by Azam Akram (@azam-akram).</description>
    <link>https://dev.to/azam-akram</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3974919%2Fe4b398b8-3571-4282-b0f4-2d73a6e38962.jpg</url>
      <title>DEV Community: Azam Akram</title>
      <link>https://dev.to/azam-akram</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/azam-akram"/>
    <language>en</language>
    <item>
      <title>Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained the Hard Way</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sat, 12 Sep 2026 12:09:31 +0000</pubDate>
      <link>https://dev.to/azam-akram/kafka-fundamentals-topics-partitions-and-consumer-groups-explained-the-hard-way-295c</link>
      <guid>https://dev.to/azam-akram/kafka-fundamentals-topics-partitions-and-consumer-groups-explained-the-hard-way-295c</guid>
      <description>&lt;p&gt;Most developers can wire up a &lt;a href="https://kafka.apache.org/" rel="noopener noreferrer"&gt;Kafka&lt;/a&gt; producer and consumer, get a message flowing end to end,&lt;br&gt;
and call it done. But there are much more into it, for example, why do we go with 6 partitions instead of&lt;br&gt;
3, how partitions are created, what is relationship between consumers and topic partitions etc - they seem to have no real answer.&lt;/p&gt;

&lt;p&gt;In this blog I will cover some of very fundamentatl elements of kafka, such as, what a partition actually is, &lt;br&gt;
how a message ends up on one, and how consumer groups split the work. Then I will set up a kafka broker in Docker and &lt;br&gt;
break it on purpose, which will explain something very useful. Along the way I also pick up a few patterns that keep showing up once you&lt;br&gt;
start building real things on top of Kafka.&lt;/p&gt;
&lt;h2&gt;
  
  
  The Basic Terms, in Plain English
&lt;/h2&gt;

&lt;p&gt;Before any commands, it helps to get the terminologies straight. Most blogs use them loosely, and that's usually where the&lt;br&gt;
confusion starts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cluster&lt;/strong&gt;: a bunch of Kafka servers working as one system. You talk to "the cluster", not to any one machine in it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Broker&lt;/strong&gt;: a single Kafka server. It stores partitions and handles reads/writes for whatever it owns. Real clusters&lt;br&gt;
run at least 3 brokers so they can survive one going down; on your laptop you'll usually just run one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Producer / Consumer&lt;/strong&gt;: not separate infrastructure, just code in your app. A producer sends messages, a consumer&lt;br&gt;
reads them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Topic&lt;/strong&gt;: a named stream, like &lt;code&gt;orders&lt;/code&gt;. Here's the twist that takes a while to understand: a topic isn't one&lt;br&gt;
queue. It's a &lt;strong&gt;set of partitions&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Partition&lt;/strong&gt;: the concept that takes the longest to grasp, so it gets its own section below.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consumer group&lt;/strong&gt;: a name you give a bunch of consumers so they can split up the work of reading a topic. More on&lt;br&gt;
this further down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Offset&lt;/strong&gt;: just a number that goes up by one for every message, marking its position &lt;em&gt;inside a partition&lt;/em&gt;. Offsets&lt;br&gt;
are per-partition, not shared across the whole topic. There's no per-message ack, no visibility timeout like AWS SQS -&lt;br&gt;
each consumer just remembers how far it's read.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why Partitions Matter
&lt;/h2&gt;

&lt;p&gt;A lot of intro material treats a topic like a plain queue: stuff goes in one end, a consumer takes it out the other,&lt;br&gt;
done. That picture is wrong enough to cause real problems later. A Kafka topic is actually a durable log split into&lt;br&gt;
several independent pieces. Nothing gets deleted when it's read, several consumers can each replay the whole thing at&lt;br&gt;
their own pace, and how it's split up is something you decide and have to live with. That splitting is the partition,&lt;br&gt;
and it's where most of the early confusion comes from - so that's next.&lt;/p&gt;
&lt;h2&gt;
  
  
  Partitions Aren't Copies
&lt;/h2&gt;

&lt;p&gt;A very common mistake is, assuming partitions as copies of the same stream, like read replicas of a database. They&lt;br&gt;
aren't. That's what &lt;strong&gt;replicas&lt;/strong&gt; are - copies of one partition spread across brokers, purely for fault tolerance,&lt;br&gt;
which is a different topic I'll get to another time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Partitions are shards.&lt;/strong&gt; Each one holds a different slice of the topic, with no overlap. It's the same idea as&lt;br&gt;
sharding a database table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Topic: "orders" with 3 partitions

Partition 0:  [order-101][order-104][order-109] ...   ← different events
Partition 1:  [order-102][order-105][order-107] ...   ← different events
Partition 2:  [order-103][order-106][order-108] ...   ← different events
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;order-101&lt;/code&gt; only ever lives in one partition. Put all the partitions together and you get the full topic; on their&lt;br&gt;
own, none of them looks like any of the others. Two things fall out of this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Order is only guaranteed inside a single partition.&lt;/strong&gt; Messages in partition 0 stay in order relative to each
other. A message in partition 0 and one in partition 1 have no ordering relationship at all - Kafka makes no
promises there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Different consumers can read different partitions at the same time.&lt;/strong&gt; This is really the whole reason Kafka can
scale the way it does.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  How Kafka Decides Which Partition a Message Goes To
&lt;/h2&gt;

&lt;p&gt;This comes down to the &lt;strong&gt;partition key&lt;/strong&gt;, something you choose when you send a message. Kafka hashes it and always&lt;br&gt;
maps that hash to the same partition. Same key in, same partition out, every time - as long as the partition count&lt;br&gt;
doesn't change.&lt;/p&gt;

&lt;p&gt;Say you've got a &lt;code&gt;customer-orders&lt;/code&gt; topic. A few ways to key it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;By &lt;code&gt;customer_id&lt;/code&gt;&lt;/strong&gt;: every event for one customer lands on the same partition, so ordering within a customer (say
&lt;code&gt;CREATED&lt;/code&gt; before &lt;code&gt;SHIPPED&lt;/code&gt;) is guaranteed. Different customers still spread out across partitions, so you keep the
parallelism.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;By &lt;code&gt;order_id&lt;/code&gt;&lt;/strong&gt;: good when a single order's events need to stay in order, but you don't care how one customer's
different orders relate to each other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No key at all&lt;/strong&gt;: Kafka spreads messages around over time for max throughput, with no ordering guarantee (more on
the batching behaviour behind this a bit further down). Fine for something like clickstream events where order
genuinely doesn't matter.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The question to actually ask yourself: which entity's events need to stay in order relative to each other? Whatever&lt;br&gt;
that is, that's your key.&lt;/p&gt;

&lt;p&gt;One thing worth knowing upfront: if you key by &lt;code&gt;customer_id&lt;/code&gt; and one customer is way more active than the rest, their&lt;br&gt;
partition turns into a hot spot - it's pinned to one broker and one consumer thread, so it takes all the load while&lt;br&gt;
everyone else's partition sits comfortably. Real production headache, and the only way around it is thinking about key&lt;br&gt;
design before you hit it.&lt;/p&gt;
&lt;h2&gt;
  
  
  How Many Partitions Should You Actually Use
&lt;/h2&gt;

&lt;p&gt;This is a separate call from the key, and mixing the two up is a common next mistake. The key decides &lt;em&gt;where a given&lt;br&gt;
message goes&lt;/em&gt;. The count decides &lt;em&gt;how many shards the topic has, full stop&lt;/em&gt; - it's about capacity and parallelism,&lt;br&gt;
not about your business domain.&lt;/p&gt;

&lt;p&gt;The rule that actually matters: &lt;strong&gt;only one consumer in a group can read a given partition at a time.&lt;/strong&gt; So the&lt;br&gt;
partition count is a hard cap on how much you can parallelise. A topic with 4 partitions will never be processed by&lt;br&gt;
more than 4 consumers in the same group - deploy a fifth and it just sits there doing nothing.&lt;/p&gt;

&lt;p&gt;A rough way to size it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="n"&gt;target_throughput&lt;/span&gt; = &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="n"&gt;MB&lt;/span&gt;/&lt;span class="n"&gt;sec&lt;/span&gt;
&lt;span class="n"&gt;single_partition_throughput&lt;/span&gt; ≈ &lt;span class="m"&gt;10&lt;/span&gt; &lt;span class="n"&gt;MB&lt;/span&gt;/&lt;span class="n"&gt;sec&lt;/span&gt; (&lt;span class="n"&gt;hardware&lt;/span&gt;/&lt;span class="n"&gt;network&lt;/span&gt; &lt;span class="n"&gt;dependent&lt;/span&gt;)

&lt;span class="n"&gt;partitions&lt;/span&gt; &lt;span class="n"&gt;needed&lt;/span&gt; ≈ &lt;span class="m"&gt;100&lt;/span&gt; / &lt;span class="m"&gt;10&lt;/span&gt; = &lt;span class="m"&gt;10&lt;/span&gt; (&lt;span class="n"&gt;minimum&lt;/span&gt;, &lt;span class="n"&gt;then&lt;/span&gt; &lt;span class="n"&gt;round&lt;/span&gt; &lt;span class="n"&gt;up&lt;/span&gt; &lt;span class="n"&gt;for&lt;/span&gt; &lt;span class="n"&gt;headroom&lt;/span&gt;)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few other things feed into the number: how slow your per-message processing is (slower downstream calls need more&lt;br&gt;
partitions to keep up), how much you expect to grow (bumping the partition count later changes the key-to-partition&lt;br&gt;
mapping and breaks ordering guarantees you already had, so most people just over-provision early), and broker&lt;br&gt;
overhead (more partitions means more file handles and replication traffic per broker, and there's a real ceiling).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;There's no "business" answer to partition count&lt;/strong&gt;, and that's the thing to unlearn. The domain logic belongs in the&lt;br&gt;
key. The count is just infra sizing, decided once, and annoying to change later.&lt;/p&gt;
&lt;h2&gt;
  
  
  Consumer Groups: How the Work Actually Gets Split Up
&lt;/h2&gt;

&lt;p&gt;A consumer group is just a name you give a set of consumers that want to share the work of reading a topic. Any&lt;br&gt;
consumer that starts up with the same group ID joins that group, and Kafka handles splitting the topic's partitions&lt;br&gt;
between them.&lt;/p&gt;

&lt;p&gt;This is where the partition-count rule from above actually bites: Kafka gives each partition to exactly one consumer&lt;br&gt;
in the group at a time. More consumers than partitions and some sit idle. More partitions than consumers and some&lt;br&gt;
consumers end up handling several.&lt;/p&gt;

&lt;p&gt;Every time a consumer joins or drops out, Kafka triggers a rebalance - it works out a fresh split of the partitions&lt;br&gt;
across whoever's left and hands out ownership again. That's the whole trick behind scaling up: start another instance&lt;br&gt;
of your consumer with the same group ID, and Kafka does the rest. No code change needed.&lt;/p&gt;

&lt;p&gt;There's a less obvious perk to grouping this way too: broadcasting. Two different consumer groups reading the same&lt;br&gt;
topic each get their own full copy of every message. Say &lt;code&gt;billing-service&lt;/code&gt; is one group and &lt;code&gt;analytics-service&lt;/code&gt; is&lt;br&gt;
another - both see everything, both track their own offsets, and neither one affects the other. Inside a group, Kafka&lt;br&gt;
acts like a queue: one message, one consumer. Across groups, it acts like pub/sub: every group gets everything. Same&lt;br&gt;
underlying log, both patterns at once.&lt;/p&gt;
&lt;h2&gt;
  
  
  Trying It on a Local Broker
&lt;/h2&gt;

&lt;p&gt;Got a Kafka container running, with the scripts under &lt;code&gt;/opt/kafka/bin&lt;/code&gt; added to &lt;code&gt;PATH&lt;/code&gt; so I didn't have to type the&lt;br&gt;
full path every time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;:/opt/kafka/bin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Created two topics so I could compare them side by side:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-topics.sh &lt;span class="nt"&gt;--create&lt;/span&gt; &lt;span class="nt"&gt;--topic&lt;/span&gt; orders-1p &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--bootstrap-server&lt;/span&gt; localhost:9092 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--partitions&lt;/span&gt; 1 &lt;span class="nt"&gt;--replication-factor&lt;/span&gt; 1

kafka-topics.sh &lt;span class="nt"&gt;--create&lt;/span&gt; &lt;span class="nt"&gt;--topic&lt;/span&gt; orders-12p &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--bootstrap-server&lt;/span&gt; localhost:9092 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--partitions&lt;/span&gt; 12 &lt;span class="nt"&gt;--replication-factor&lt;/span&gt; 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--describe&lt;/code&gt; on &lt;code&gt;orders-12p&lt;/code&gt; shows all 12 partitions, all led by broker 1 (single-broker setup, so nothing interesting&lt;br&gt;
happening on the replication side yet):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;Topic: orders-12p   PartitionCount: 12   ReplicationFactor&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
    &lt;span class="na"&gt;Partition: 0   Leader: 1   Replicas: 1   Isr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
    &lt;span class="na"&gt;Partition: 1   Leader: 1   Replicas: 1   Isr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
    &lt;span class="s"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Watching the Consumer Limit Play Out
&lt;/h3&gt;

&lt;p&gt;Started three console consumers, all in the same group, pointed at the &lt;strong&gt;1-partition&lt;/strong&gt; topic:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-console-consumer.sh &lt;span class="nt"&gt;--topic&lt;/span&gt; orders-1p &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--bootstrap-server&lt;/span&gt; localhost:9092 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--group&lt;/span&gt; test-group-1p &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--formatter-property&lt;/span&gt; print.partition&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then sent a few messages. Only &lt;strong&gt;one&lt;/strong&gt; of the three consumers got anything. The other two just sat there, connected&lt;br&gt;
and doing nothing - which is exactly the point. Checking the group directly backs it up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-consumer-groups.sh &lt;span class="nt"&gt;--bootstrap-server&lt;/span&gt; localhost:9092 &lt;span class="nt"&gt;--describe&lt;/span&gt; &lt;span class="nt"&gt;--group&lt;/span&gt; test-group-1p
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="go"&gt;GROUP           TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
test-group-1p   orders-1p   0          22              22              0   console-consumer-...
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One row. One partition, one consumer doing anything. The other two are members of the group, but they own nothing.&lt;/p&gt;

&lt;p&gt;Ran the same test against the &lt;strong&gt;12-partition&lt;/strong&gt; topic, giving all three consumers a few seconds to settle before&lt;br&gt;
producing anything, and this time the split was clean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;console-consumer-A → partitions 0, 1, 2, 3
console-consumer-B → partitions 4, 5, 6, 7
console-consumer-C → partitions 8, 9, 10, 11
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exactly what you'd expect: 12 partitions, 3 consumers, 4 each.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why 10 Unkeyed Messages All Landed in One Partition
&lt;/h3&gt;

&lt;p&gt;This one threw me for a minute. With all three consumers sitting evenly assigned, I sent 10 messages with no key and&lt;br&gt;
expected them to spread out. Instead, &lt;strong&gt;all 10 landed in the same partition&lt;/strong&gt;, and only one consumer terminal lit up.&lt;/p&gt;

&lt;p&gt;My first thought was that the rebalance was broken. It wasn't - &lt;code&gt;--describe&lt;/code&gt; still showed a perfectly even split. What&lt;br&gt;
was actually happening: when you produce without a key, Kafka doesn't round-robin each message individually. The&lt;br&gt;
producer uses what's called a sticky partitioner - it picks one partition and sticks with it for a whole batch (until&lt;br&gt;
the batch fills up or &lt;code&gt;linger.ms&lt;/code&gt; runs out), then switches to a different one for the next batch. I'd typed those 10&lt;br&gt;
lines quickly into the console producer, so they got batched together and shipped as one unit to a single partition.&lt;/p&gt;

&lt;p&gt;That's not a quirk, it's a real thing to know about before you hit it in production. Sticky partitioning trades&lt;br&gt;
perfectly even spread for fewer, bigger batches and better throughput - it evens out over many batches, not&lt;br&gt;
necessarily over a handful of messages typed by hand. Key your messages explicitly, though, and you get the same&lt;br&gt;
result every time, no batching surprises involved.&lt;/p&gt;

&lt;h2&gt;
  
  
  Patterns That Show Up Once You Build on Top of This
&lt;/h2&gt;

&lt;p&gt;Once partitions, keys, and consumer groups make sense, most of what you see in real Kafka systems turns out to be a&lt;br&gt;
small handful of patterns built on top of them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dead letter queue (DLQ).&lt;/strong&gt; A message that keeps failing after a few retries goes to a separate &lt;code&gt;&amp;lt;topic&amp;gt;-dlt&lt;/code&gt;
topic instead of blocking the partition or getting silently dropped. The main consumer keeps moving, and the failed
messages sit somewhere you can actually go look at them. I walk through building one in
&lt;a href="https://dev.to/blog/golang-kafka-producer-consumer-with-docker/"&gt;Building Kafka Producer-Consumer Using Go and Docker&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retry with backoff.&lt;/strong&gt; The simple version: retry a failed message a few times with a short delay before giving up
and sending it to the DLQ (also in the post above). At bigger scale, that retry often moves to its own topic
(&lt;code&gt;&amp;lt;topic&amp;gt;-retry-30s&lt;/code&gt;, &lt;code&gt;&amp;lt;topic&amp;gt;-retry-5m&lt;/code&gt;) so a slow downstream doesn't hold up the main partition while messages
wait around.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Idempotent consumer.&lt;/strong&gt; Kafka's default guarantee is at-least-once, so every consumer has to assume a message might
show up twice - say, after a rebalance like the one above, or a commit that got retried. The fix lives on the
consumer side, not the broker: keep track of a message's unique key (or a hash of its contents) somewhere, and skip
anything you've already handled.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transactional outbox.&lt;/strong&gt; The problem: a service can't save to its own database &lt;em&gt;and&lt;/em&gt; publish a Kafka event as one
all-or-nothing step. One can succeed while the other fails, and now the database and Kafka disagree - either the
order got saved but nobody heard about it, or Kafka says the order exists when it was never actually saved.
The fix: don't publish to Kafka directly. Save the event as a row in an &lt;code&gt;outbox&lt;/code&gt; table instead, in the same
database transaction as the real write. That part is now truly atomic - both rows are saved, or neither is.
Getting the event into Kafka happens afterward, as a separate step:

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Polling&lt;/strong&gt; - a background job checks the outbox table now and then, publishes anything new, and marks it done.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CDC via Kafka Connect&lt;/strong&gt; - watches the database's own internal change log for new rows and streams them to
Kafka automatically, no polling needed.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On cleanup: don't delete a row the moment it's published - just mark it &lt;code&gt;published&lt;/code&gt;, and let a separate scheduled&lt;br&gt;
  job purge old published rows later (say, once a day). That way a crash right after publishing doesn't lose the&lt;br&gt;
  record before you're sure Kafka got it, and the table doesn't grow forever either.&lt;/p&gt;

&lt;p&gt;The database write stays the single source of truth and stays atomic; publishing becomes a "happens shortly after"&lt;br&gt;
  step instead of a "must happen right now" one - &lt;strong&gt;the database and Kafka are eventually consistent, not instantly&lt;br&gt;
  consistent&lt;/strong&gt; - and the event can't be lost since it sits safely in the outbox table until it's confirmed published.&lt;br&gt;
  The cost: it's at-least-once, not exactly-once, so consumers still need to be&lt;br&gt;
  idempotent (as above), there's a small delay before events show up, and you're now running an extra table plus a&lt;br&gt;
  poller or CDC pipeline.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fan-out through separate consumer groups.&lt;/strong&gt; Already covered above - every group reading a topic gets its own
full copy of the stream. It's what lets &lt;code&gt;billing-service&lt;/code&gt; and &lt;code&gt;analytics-service&lt;/code&gt; both read &lt;code&gt;orders&lt;/code&gt; independently,
without either one's speed or downtime touching the other.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these are switches you flip in Kafka itself - they're just conventions you build into your producers and&lt;br&gt;
consumers on top of the same partitions, keys, and groups covered above.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Actually Walked Away With
&lt;/h2&gt;

&lt;p&gt;The mental model that finally stuck after all this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concept&lt;/th&gt;
&lt;th&gt;What it actually is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Partition&lt;/td&gt;
&lt;td&gt;A shard - a different, non-overlapping slice of the topic's events&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Partition key&lt;/td&gt;
&lt;td&gt;Decides &lt;em&gt;which&lt;/em&gt; partition a given message lands on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Partition count&lt;/td&gt;
&lt;td&gt;A capacity and parallelism decision, fixed at topic creation, expensive to change later&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consumer group&lt;/td&gt;
&lt;td&gt;A set of consumers sharing the work of a topic; one partition → one active consumer at a time within a group&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Offset&lt;/td&gt;
&lt;td&gt;Per-partition read position, tracked separately by each consumer group&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;At-least-once delivery&lt;/td&gt;
&lt;td&gt;The default guarantee - consumers need to be idempotent, not assume single delivery&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Next up, when I get to it: multi-broker replication and leader election, and how retention and compaction actually&lt;br&gt;
work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://dev.to/blog/golang-kafka-producer-consumer-with-docker/"&gt;Building Kafka Producer-Consumer Using Go and Docker&lt;/a&gt; - a hands-on Go implementation of the DLQ and retry patterns described above&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://dev.to/blog/spring-boot-kafka-producer-consumer-with-docker/"&gt;Spring Boot Kafka Producer-Consumer with Docker&lt;/a&gt; - the same producer-consumer shape built with Spring Boot&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://dev.to/blog/getting-started-with-docker-and-go-lang/"&gt;Getting Started with Docker and Go Lang&lt;/a&gt; - containerizing the services these patterns run in&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>kafka</category>
      <category>microservices</category>
      <category>architecture</category>
      <category>docker</category>
    </item>
    <item>
      <title>SQL Formatter</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Fri, 03 Jul 2026 15:47:45 +0000</pubDate>
      <link>https://dev.to/azam-akram/sql-formatter-2gbj</link>
      <guid>https://dev.to/azam-akram/sql-formatter-2gbj</guid>
      <description>&lt;p&gt;&lt;a href="https://www.solutiontoolkit.com/tools/sql-formatter" rel="noopener noreferrer"&gt;SQL formatter&lt;/a&gt; instantly formats and beautifies the SQL queries for any database. It supports MySQL, PostgreSQL, T-SQL, SQLite, BigQuery, and more.&lt;/p&gt;

&lt;h2&gt;
  
  
  Formatting options explained
&lt;/h2&gt;

&lt;p&gt;Dialect: Controls dialect-specific syntax rules. For example, PostgreSQL supports :: casting, T-SQL uses TOP instead of LIMIT, and BigQuery uses backtick identifiers.&lt;br&gt;
Keywords UPPERCASE: The most common SQL style guide convention. Makes reserved words like SELECT, FROM, and WHERE visually distinct from column and table names.&lt;br&gt;
Commas before: Places commas at the start of each line (,column_name). Popular in data warehousing teams and dbt style guides - easier to comment out a column without breaking the query.&lt;br&gt;
Commas after: Standard convention in most SQL editors and ORMs (column_name,).&lt;/p&gt;

&lt;h2&gt;
  
  
  Common use cases
&lt;/h2&gt;

&lt;p&gt;Readable queries from ORMs, query builders, or database console output.&lt;br&gt;
Enforce a consistent style across team SQL files before committing.&lt;br&gt;
Unminify hand-written one-liners for debugging in production logs.&lt;br&gt;
Format EXPLAIN queries before sharing with a teammate.&lt;br&gt;
Prepare SQL snippets for documentation, blog posts, or code review.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Use This SQL Formatter
&lt;/h2&gt;

&lt;p&gt;SQL formatter is designed for quick browser-based work when you need to format sql for 10+ database dialects. Paste or select your input in the tool area above, run the conversion or formatting step, then review the result before copying it into code, documentation, tickets, test data, or an API client.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example input&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;select id,name from users where active=true order by created_at desc&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example output&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;SELECT id, name&lt;br&gt;
FROM users&lt;br&gt;
WHERE active = true&lt;br&gt;
ORDER BY created_at DESC&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Use This Tool
&lt;/h2&gt;

&lt;p&gt;Make a long query readable before code review&lt;br&gt;
Normalize copied SQL from logs or dashboards&lt;br&gt;
Prepare database examples for docs, tickets, or migration notes&lt;/p&gt;

&lt;h2&gt;
  
  
  Accepted Input Formats
&lt;/h2&gt;

&lt;p&gt;SQL queries from MySQL, PostgreSQL, SQLite, BigQuery, T-SQL, and similar dialects&lt;br&gt;
Single statements, multi-statement scripts, and copied query logs&lt;br&gt;
Minified SQL that needs consistent indentation and keyword casing&lt;/p&gt;

&lt;h2&gt;
  
  
  Output Details
&lt;/h2&gt;

&lt;p&gt;The output keeps the same query logic while improving layout&lt;br&gt;
Keyword case and indentation follow the options selected in the tool&lt;/p&gt;

&lt;p&gt;Related Tools&lt;br&gt;
&lt;a href="https://www.solutiontoolkit.com/tools/javascript-beautifier" rel="noopener noreferrer"&gt;JavaScript Beautifier and Formatter&lt;/a&gt;&lt;br&gt;
Beautify or format JavaScript online instantly. Paste minified or messy JS and get clean, readable, indented code. Free JS beautifier and formatter, no sign-up.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.solutiontoolkit.com/tools/css-beautifier" rel="noopener noreferrer"&gt;CSS Beautifier and Formatter&lt;/a&gt;&lt;br&gt;
Beautify, format, or prettify CSS online for free. Unminify compressed stylesheets into clean, readable code. Free CSS beautifier and formatter, no sign-up.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.solutiontoolkit.com/tools/yaml-formatter" rel="noopener noreferrer"&gt;YAML Formatter &amp;amp; Validator&lt;/a&gt;&lt;br&gt;
Format, beautify, and validate YAML online. Supports Kubernetes, Docker Compose, GitHub Actions, Ansible, and Helm. Syntax errors reported with line and column numbers.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>programming</category>
      <category>sql</category>
      <category>database</category>
    </item>
    <item>
      <title>Legacy system needs XML but you're living in JSON world? One paste, done. https://www.solutiontoolkit.com/tools/json-to-xml-converter
#xml #json #validate</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Tue, 30 Jun 2026 19:55:58 +0000</pubDate>
      <link>https://dev.to/azam-akram/legacy-system-needs-xml-but-youre-living-in-json-world-one-paste-done-40pd</link>
      <guid>https://dev.to/azam-akram/legacy-system-needs-xml-but-youre-living-in-json-world-one-paste-done-40pd</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/tools/json-to-xml-converter" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
      <category>xml</category>
      <category>json</category>
      <category>webdev</category>
      <category>programming</category>
    </item>
    <item>
      <title>Gzip Base64 encoder / decoder</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sun, 28 Jun 2026 15:44:55 +0000</pubDate>
      <link>https://dev.to/azam-akram/gzip-base64-encoder-decoder-12hc</link>
      <guid>https://dev.to/azam-akram/gzip-base64-encoder-decoder-12hc</guid>
      <description>&lt;p&gt;Compress text with Gzip and encode the result as Base64 in a single step — or go the other way and decode then decompress a Base64 payload back to the original string. Everything runs in your browser.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.solutiontoolkit.com/tools/gzip-base64-encoder-decoder" rel="noopener noreferrer"&gt;https://www.solutiontoolkit.com/tools/gzip-base64-encoder-decoder&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  When is this useful?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;API payloads: some services accept compressed, Base64-encoded bodies to reduce bandwidth. Decode them here to inspect the original JSON or XML.&lt;/li&gt;
&lt;li&gt;Environment variables: large config blobs are often Gzip+Base64 encoded before being stored as env vars or Kubernetes secrets.&lt;/li&gt;
&lt;li&gt;CloudWatch / logging: AWS Lambda logs shipped via Kinesis are Gzip+Base64 encoded. Paste the payload here to read the raw log lines.&lt;/li&gt;
&lt;li&gt;Cookie compression: session cookies containing JSON are sometimes compressed to stay under browser size limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Compress &amp;amp; Encode: runs Gzip compression on your text using the pako library, then encodes the binary output as a Base64 string safe for transport in JSON, URLs, and headers.&lt;/li&gt;
&lt;li&gt;Decompress &amp;amp; Decode: validates the Base64 format, decodes the bytes, then runs Gzip inflate to recover the original text.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Short strings often produce a larger Base64 output than the original because Gzip's header and Base64 expansion (x1.33) outweigh the compression gain. Gzip shines on repetitive text over ~1 KB.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Use This Gzip Base64 Decoder and Encoder
&lt;/h2&gt;

&lt;p&gt;gzip base64 decode is designed for quick browser-based work when you need to gzip compress text. Paste or select your input in the tool area above, run the conversion or formatting step, then review the result before copying it into code, documentation, tickets, test data, or an API client.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example input:&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;Gzip-compressed Base64 API payload&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example output:&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;H4sIAAAAAAAAA3OvyizQTc7PLShKLS5OTVFwSixONTNRcAzwVChIrMzJT0wBAOnU+sEiAAAA&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Common Issues to Check&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A normal Base64 string will not decompress unless the decoded bytes are gzip data&lt;/li&gt;
&lt;li&gt;Copied payloads can fail when quotes, escaping, or whitespace are included&lt;/li&gt;
&lt;li&gt;Very large payloads can take longer because compression happens in the browser&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>gzip</category>
      <category>webdev</category>
      <category>base64</category>
      <category>encode</category>
    </item>
    <item>
      <title>Stop rewriting configs by hand. Paste JSON, get YAML (or the other way around) in one click. 

https://www.solutiontoolkit.com/tools/json-yaml-converter

#json #yaml</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sun, 28 Jun 2026 15:36:03 +0000</pubDate>
      <link>https://dev.to/azam-akram/stop-rewriting-configs-by-hand-paste-json-get-yaml-or-the-other-way-around-in-one-click-407a</link>
      <guid>https://dev.to/azam-akram/stop-rewriting-configs-by-hand-paste-json-get-yaml-or-the-other-way-around-in-one-click-407a</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/tools/json-yaml-converter" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
    </item>
    <item>
      <title>Your API wants JSON. Your config is YAML. Convert it here, no signup needed. https://www.solutiontoolkit.com/tools/yaml-to-json-converter
#json #yaml</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sat, 27 Jun 2026 23:42:12 +0000</pubDate>
      <link>https://dev.to/azam-akram/your-api-wants-json-your-config-is-yaml-convert-it-here-no-signup-needed-3n9m</link>
      <guid>https://dev.to/azam-akram/your-api-wants-json-your-config-is-yaml-convert-it-here-no-signup-needed-3n9m</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/tools/yaml-to-json-converter" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
    </item>
    <item>
      <title>Stop rewriting configs by hand. Paste JSON, get YAML (or the other way around) in one click. https://www.solutiontoolkit.com/tools/json-yaml-converter
#json #yaml #converter</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sat, 27 Jun 2026 23:40:03 +0000</pubDate>
      <link>https://dev.to/azam-akram/stop-rewriting-configs-by-hand-paste-json-get-yaml-or-the-other-way-around-in-one-click-31d</link>
      <guid>https://dev.to/azam-akram/stop-rewriting-configs-by-hand-paste-json-get-yaml-or-the-other-way-around-in-one-click-31d</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/tools/json-yaml-converter" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
    </item>
    <item>
      <title>Tired of pasting JSON into random sites? Here's a free in-browser validator - nothing leaves your machine. 
https://www.solutiontoolkit.com/tools/validate-and-prettify-json
#json #pretty #validate</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Sat, 27 Jun 2026 23:37:58 +0000</pubDate>
      <link>https://dev.to/azam-akram/tired-of-pasting-json-into-random-sites-heres-a-free-in-browser-validator-nothing-leaves-your-1fc6</link>
      <guid>https://dev.to/azam-akram/tired-of-pasting-json-into-random-sites-heres-a-free-in-browser-validator-nothing-leaves-your-1fc6</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/tools/validate-and-prettify-json" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
    </item>
    <item>
      <title>Check live scores, full group stage schedule, and knockout fixtures all in one place: 
👉 https://www.solutiontoolkit.com/world-cup-2026/schedule
#WorldCup2026 #FIFA2026 #NextJS #SolutionToolkit #WebDev #Football</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Thu, 25 Jun 2026 23:18:46 +0000</pubDate>
      <link>https://dev.to/azam-akram/check-live-scores-full-group-stage-schedule-and-knockout-fixtures-all-in-one-place-1jb8</link>
      <guid>https://dev.to/azam-akram/check-live-scores-full-group-stage-schedule-and-knockout-fixtures-all-in-one-place-1jb8</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
      &lt;div class="c-embed__body flex items-center justify-between"&gt;
        &lt;a href="https://www.solutiontoolkit.com/world-cup-2026/schedule" rel="noopener noreferrer" class="c-link fw-bold flex items-center"&gt;
          &lt;span class="mr-2"&gt;solutiontoolkit.com&lt;/span&gt;
          

        &lt;/a&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


</description>
    </item>
    <item>
      <title>Debugging GZip Base64 compressed payloads from AWS API Gateway / Lambda</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Mon, 22 Jun 2026 17:40:12 +0000</pubDate>
      <link>https://dev.to/azam-akram/debugging-gzip-base64-compressed-payloads-from-aws-api-gateway-lambda-3ehm</link>
      <guid>https://dev.to/azam-akram/debugging-gzip-base64-compressed-payloads-from-aws-api-gateway-lambda-3ehm</guid>
      <description>&lt;p&gt;You call an API. The response comes back and one of the fields looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;H4sIAAAAAAAAE6tWKkktLlGyUlIqS04sLknMSwUAAAD//wMAAAD//w==
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you have ever looked at something like that wondering what is hiding inside, you are looking at a GZip-compressed payload encoded in Base64.&lt;/p&gt;

&lt;p&gt;This pattern appears more often than you might expect — in AWS, Azure, Kafka, webhook systems, and internal microservices. Understanding why it exists and how to quickly decode it will save you time every time you hit it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Just need to decode one now?&lt;/strong&gt; Paste the string into the &lt;a href="https://www.solutiontoolkit.com/tools/gzip-base64-encoder-decoder" rel="noopener noreferrer"&gt;GZip Base64 Encoder / Decoder&lt;/a&gt; and get the result instantly — no terminal required.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The Problem: Binary Data in Text Channels
&lt;/h2&gt;

&lt;p&gt;GZip compression produces binary output — raw bytes that can include any value from 0 to 255. Most text-based protocols (HTTP headers, JSON fields, environment variables, message queues) are designed for printable characters. If you try to embed raw binary bytes into a JSON string, you will get parsing errors or silent data corruption.&lt;/p&gt;

&lt;p&gt;Base64 solves this by converting binary data into a safe alphabet of 64 printable ASCII characters. The trade-off is a roughly 33% increase in size — but after GZip compression, the total payload is still significantly smaller than the original.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why APIs Combine GZip and Base64
&lt;/h2&gt;

&lt;p&gt;The two techniques solve different problems and work well together:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GZip&lt;/strong&gt; reduces size. For repetitive, structured data like JSON or XML, GZip typically achieves 60–80% compression. A 100 KB JSON payload can compress to 15–20 KB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Base64&lt;/strong&gt; makes binary data safe to transport. It guarantees the compressed bytes can be embedded in a JSON field, stored in a database column, passed through an environment variable, or included in a URL query string without corruption.&lt;/p&gt;

&lt;p&gt;Together they allow a system to transmit large, structured payloads efficiently through channels that only support text.&lt;/p&gt;

&lt;h3&gt;
  
  
  Real-World Examples
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;AWS Lambda and API Gateway&lt;/strong&gt; — response bodies larger than a certain threshold can be returned as Base64-encoded compressed content, with a flag telling the client to decompress&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kafka messages&lt;/strong&gt; — event payloads are often GZip + Base64 encoded before publishing so they fit within message size limits and serialize cleanly&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Webhook events&lt;/strong&gt; — third-party services sometimes send compressed event data to reduce egress costs&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Distributed tracing / logging&lt;/strong&gt; — trace context or structured log data embedded in HTTP headers&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How to Debug a Compressed Payload
&lt;/h2&gt;

&lt;p&gt;When you encounter one of these strings during development or debugging, the workflow is straightforward:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Copy the encoded string from the API response, log line, or message payload&lt;/li&gt;
&lt;li&gt;Decode the Base64 layer to recover the GZip binary&lt;/li&gt;
&lt;li&gt;Decompress the GZip to recover the original data&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In a terminal you can do this with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"H4sIAAAAAAAAE6tWKkktLlGyUlIqS04sLknMSwUAAAD//wMAAAD//w=="&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;base64&lt;/span&gt; &lt;span class="nt"&gt;--decode&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;gunzip&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That works fine in a Linux or macOS shell. On Windows, or when you just want a quick result without opening a terminal, an online tool is faster.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decode It Instantly
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://www.solutiontoolkit.com/tools/gzip-base64-encoder-decoder" rel="noopener noreferrer"&gt;GZip Base64 Encoder / Decoder&lt;/a&gt; tool on this site handles both directions — paste your encoded string to decode it, or paste plain text and JSON to compress and encode it. No installation, no terminal, works in the browser with no data leaving your machine.&lt;/p&gt;

&lt;p&gt;It is useful when you need to inspect a payload quickly during an incident, verify what a service is actually sending before writing decoding logic in your application, or compress test data to embed in a config or environment variable.&lt;/p&gt;

&lt;p&gt;The next time you hit an unreadable string in an API response, you will know exactly what it is and how to open it up.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>devops</category>
      <category>api</category>
      <category>aws</category>
    </item>
    <item>
      <title>Generate MD5, SHA-1, SHA-256, and SHA-512 Hashes Instantly in Your Browser</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Thu, 18 Jun 2026 17:32:40 +0000</pubDate>
      <link>https://dev.to/azam-akram/generate-md5-sha-1-sha-256-and-sha-512-hashes-instantly-in-your-browser-2o10</link>
      <guid>https://dev.to/azam-akram/generate-md5-sha-1-sha-256-and-sha-512-hashes-instantly-in-your-browser-2o10</guid>
      <description>&lt;p&gt;A free, client-side tool for hashing text — no installation, no account, no server.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is a Hash and When Do You Need One?
&lt;/h2&gt;

&lt;p&gt;A hash function takes any input — a word, a sentence, a JSON blob — and produces a fixed-length string of characters. The same input always produces the same output. Change even a single character and the output changes completely.&lt;/p&gt;

&lt;p&gt;Developers reach for hashing in a surprisingly wide range of situations:&lt;/p&gt;

&lt;p&gt;Verifying data integrity — confirm a file or string has not been tampered with&lt;br&gt;
Checksums — compare two pieces of data without storing or transmitting the originals&lt;br&gt;
Password storage — store a hash instead of a plaintext password (with a proper algorithm)&lt;br&gt;
Caching and deduplication — use a hash as a unique key for a piece of content&lt;br&gt;
Debugging — quickly fingerprint a string to check if two values are identical across environments&lt;br&gt;
The &lt;a href="https://www.solutiontoolkit.com/tools/text-hash-generator" rel="noopener noreferrer"&gt;Text Hash Generator&lt;/a&gt; lets you generate hashes across all four commonly used algorithms in one step, directly in your browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Four Algorithms
&lt;/h2&gt;

&lt;p&gt;MD5–128-bit MD5 is fast and widely recognised. It is no longer considered cryptographically secure for security-sensitive use cases, but remains practical for checksums, non-security fingerprinting, and working with legacy systems that expect MD5 output.&lt;/p&gt;

&lt;p&gt;SHA-1–160-bit SHA-1 is similarly deprecated for security use but still appears in older protocols, version control systems, and certificate chains. Useful when you need to match or verify a SHA-1 value produced by another system.&lt;/p&gt;

&lt;p&gt;SHA-256–256-bit SHA-256 is the workhorse of modern cryptography. It is part of the SHA-2 family and is widely used in TLS certificates, code signing, blockchain, and general-purpose integrity verification. When in doubt, use SHA-256.&lt;/p&gt;

&lt;p&gt;SHA-512–512-bit SHA-512 produces a longer digest and offers a higher security margin. It is preferred in high-security contexts and, on 64-bit processors, can actually be faster than SHA-256 due to how modern CPUs handle 64-bit operations.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the Tool Does
&lt;/h2&gt;

&lt;p&gt;Open the &lt;a href="https://www.solutiontoolkit.com/tools/text-hash-generator" rel="noopener noreferrer"&gt;Text Hash Generator&lt;/a&gt;, type or paste any text, and click Generate Hashes (or press Ctrl+Enter). All four hashes are computed and displayed simultaneously.&lt;/p&gt;

&lt;p&gt;Uppercase toggle Switch between lowercase and uppercase hex output with a single checkbox. Some systems expect uppercase hash strings — this saves you a manual conversion step.&lt;/p&gt;

&lt;p&gt;Copy individual or all Each hash has its own Copy button. There is also a Copy all hashes option that puts all four results on the clipboard in a labelled format, ready to paste into documentation or a ticket.&lt;/p&gt;

&lt;p&gt;Character count The input field shows a live character count as you type, useful when you need to know the exact length of the string you are hashing.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Works
&lt;/h2&gt;

&lt;p&gt;SHA-1, SHA-256, and SHA-512 are computed using the browser’s built-in Web Crypto API via crypto.subtle.digest. MD5 is handled by the js-md5 library, since MD5 is not included in the Web Crypto API (intentionally — browsers do not endorse it for cryptographic use). All four run client-side. Open DevTools, go to the Network tab, type some text, and generate — you will see zero outbound requests.&lt;/p&gt;

&lt;p&gt;Try It&lt;br&gt;
Open the &lt;a href="https://www.solutiontoolkit.com/tools/text-hash-generator" rel="noopener noreferrer"&gt;Text Hash Generator&lt;/a&gt; — paste any text and get all four hashes in one click.&lt;/p&gt;

&lt;p&gt;The Text Hash Generator is one of several developer tools at &lt;a href="https://www.solutiontoolkit.com/" rel="noopener noreferrer"&gt;SolutionToolkit&lt;/a&gt;. All tools run client-side with no server-side processing.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>security</category>
      <category>web</category>
      <category>browser</category>
    </item>
    <item>
      <title>A Client-Side JWT Debugger That Runs Entirely in Your Browser</title>
      <dc:creator>Azam Akram</dc:creator>
      <pubDate>Thu, 18 Jun 2026 13:08:27 +0000</pubDate>
      <link>https://dev.to/azam-akram/a-client-side-jwt-debugger-that-runs-entirely-in-your-browser-4ha4</link>
      <guid>https://dev.to/azam-akram/a-client-side-jwt-debugger-that-runs-entirely-in-your-browser-4ha4</guid>
      <description>&lt;p&gt;Why Another JWT Tool?&lt;br&gt;
Most JWT debugging workflows involve copying a token into an online tool, inspecting the payload, and moving on. That works fine for simple use cases, but falls short when you need to do more — sign a token with a specific algorithm, verify a signature against a public key, or quickly check whether a token has expired.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://www.solutiontoolkit.com/tools/jwt-debugger" rel="noopener noreferrer"&gt;JWT Debugger&lt;/a&gt; is a free, browser-based tool that covers the full JWT workflow: decode, inspect, sign, and verify — all in one place, with no installation and no account required.&lt;br&gt;
What It Does&lt;/p&gt;

&lt;p&gt;Decode and inspect Paste any JWT and the tool splits it into header, payload, and signature — colour-coded so each part is visually distinct. Timestamp claims (iat, exp, nbf) are automatically converted to human-readable dates with a relative time display ("expires in 2h", "3 days ago").&lt;/p&gt;

&lt;p&gt;Status at a glance A badge shows whether the token is Active, Expired, or has No Expiry set. If a token is expired, you see exactly how long ago — no manual Unix timestamp conversion needed.&lt;/p&gt;

&lt;p&gt;Sign tokens Build a header and payload from scratch and sign them. All twelve standard algorithms are supported:&lt;/p&gt;

&lt;p&gt;FamilyAlgorithmsHMACHS256, HS384, HS512RSA PKCS#1RS256, RS384, RS512RSA-PSSPS256, PS384, PS512ECDSAES256, ES384, ES512&lt;/p&gt;

&lt;p&gt;For HMAC algorithms, enter your shared secret. For asymmetric algorithms, paste your PKCS#8 private key. The tool signs the token locally using crypto.subtle.sign and gives you the complete JWT.&lt;/p&gt;

&lt;p&gt;Verify signatures Paste a JWT and your key, click Verify, and the tool confirms whether the signature is valid. For RSA and ECDSA, paste the public key in SPKI PEM format. Verification runs entirely in crypto.subtle.verify.&lt;/p&gt;

&lt;p&gt;Edit and re-encode Modify the header or payload JSON and the encoded token updates in real time. Useful for quickly testing how a claim change affects the token structure, or for building a test token before signing it.&lt;/p&gt;

&lt;p&gt;How It Works Under the Hood&lt;br&gt;
The tool is built on the Web Crypto API — a browser-native cryptography interface available in all modern browsers. There are no third-party cryptography libraries. Every signing and verification operation calls crypto.subtle directly.&lt;/p&gt;

&lt;p&gt;Base64url encoding and decoding are handled with TextEncoder and atob/btoa. PEM keys are stripped of their headers and decoded from base64 before being passed to crypto.subtle.importKey. The signing input follows the JWT spec — base64url(header) + "." + base64url(payload) — and the resulting signature bytes are base64url-encoded and appended as the third segment.&lt;/p&gt;

&lt;p&gt;If you want to verify the behaviour yourself: open DevTools, go to the Network tab, and paste a token. You will see zero outbound requests.&lt;/p&gt;

&lt;p&gt;Try It&lt;br&gt;
Open the &lt;a href="https://www.solutiontoolkit.com/tools/jwt-debugger" rel="noopener noreferrer"&gt;JWT Debugger&lt;/a&gt; and paste any JWT into the top field. The decoded header and payload appear immediately. No account, no signup.&lt;/p&gt;

&lt;p&gt;If you want to understand the theory behind what you are looking at — how the header, payload, and signature fit together, what RS256 means, and how the private/public key split works — the companion article JWT Shared Secret: How JWTs are Signed and Shared Across Services covers the fundamentals.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://www.solutiontoolkit.com/tools/jwt-debugger" rel="noopener noreferrer"&gt;JWT Debugger&lt;/a&gt; is one of several developer tools at &lt;a href="https://www.solutiontoolkit.com" rel="noopener noreferrer"&gt;SolutionToolkit&lt;/a&gt;. All tools run client-side with no server-side processing.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>security</category>
      <category>web</category>
      <category>jwt</category>
    </item>
  </channel>
</rss>
