DEV Community: Piyush Gupta

Mastering Schema Evolution: Why Apache Avro is the King of Big Data (Part 2)

Piyush Gupta — Sun, 05 Apr 2026 14:03:38 +0000

The Evolution Nightmare

In Part 1, we saw how Thrift and Protocol Buffers use numeric tags to shrink data. But in a real-world distributed system, you can’t upgrade every microservice at the same time. You will always have Old Code talking to New Code.

This creates a massive problem: Schema Evolution. If Service A adds a new field to its database, will Service B (running the old code) crash when it tries to read that data?

Forward vs. Backward Compatibility

To build a resilient system, you must understand two concepts:

Backward Compatibility: New code can read data written by old code. (Essential when you update your "Readers" first).
Forward Compatibility: Old code can read data written by new code. (Essential when you update your "Writers" first).

In Thrift and Protobuf, this is managed by Tags. If a reader sees a tag it doesn't recognize, it simply ignores it. But what if you want to avoid tags entirely?

3. Apache Avro: The "No Tag" Evolution

Apache Avro was created within the Hadoop ecosystem because Thrift wasn't a perfect fit for massive data files. Unlike Protobuf, Avro does not store tag numbers or field types in the binary data. It only stores the raw values.

How it works: The Pairwise Resolution

How does the reader know what the data is if there are no tags?

Avro uses two schemas:

Writer’s Schema: The schema the application used when it sent the data.
Reader’s Schema: The schema the receiving application expects.

When data is read, the Avro library looks at both schemas side-by-side. If the field order changed or a field was renamed, Avro "resolves" the difference by looking at the field names.

![Diagram: Avro Reader and Writer Schema Resolution Logic]

Implementation Example (Avro JSON Schema)

Avro schemas are written in simple JSON, making them much easier to generate dynamically than the IDLs we saw in Part 1.

user_schema.avsc

{
  "type": "record",
  "name": "User",
  "namespace": "com.piyush.devto",
  "fields": [
    {"name": "username", "type": "string"},
    {"name": "age", "type": ["int", "null"], "default": 0},
    {"name": "email", "type": ["string", "null"]}
  ]
}

Python Implementation:

import avro.schema
from avro.datafile import DataFileWriter
from avro.io import DatumWriter

# 1. Load the schema from a file
schema = avro.schema.parse(open("user_schema.avsc", "rb").read())

# 2. Write binary data to an Avro Object Container File
with DataFileWriter(open("users.avro", "wb"), DatumWriter(), schema) as writer:
    writer.append({
        "username": "Piyush", 
        "age": 25, 
        "email": "piyush@example.com"
    })

print("Data serialized to users.avro")

The Killer Feature: Database Integration

One reason Avro is the "gold standard" for Kafka and Big Data is its relationship with relational databases.

Because Avro schemas are JSON, you can write a script to automatically convert a SQL table into an Avro schema.

SQL Column → Avro Field Name
SQL Data Type → Avro Type
Nullable Column → Avro Union ["type", "null"]

This makes Avro perfect for Change Data Capture (CDC), where you stream every single update from your Postgres or MySQL database into a Data Lake like S3 or Snowflake.

Summary: Choosing Your Weapon

Which encoding should you use for your next project?

Feature	JSON	Protobuf / Thrift	Apache Avro
Best For	Public APIs	Internal Microservices	Big Data / Pipelines
Speed	Slowest	Fast	Fastest (no tags)
Schema	Optional	Required (IDL)	Required (JSON)
Logic	Human-readable	Tag-based	Resolution-based

Final Thoughts

Encoding isn't just about saving bytes; it's about defining the contract between your services.

Use JSON when you need ease of use.
Use Protobuf for gRPC and internal speed.
Use Avro when your data is massive and your schemas are constantly evolving.

What are you using in production? Let's discuss in the comments!



---

### Why this works for dev.to:
* **Series Link:** The `series` tag in the metadata automatically links Part 1 and Part 2 on the platform.
* **Liquid Tags:** I used blockquotes and bolded text to highlight "Forward/Backward" compatibility—a common interview question for senior devs.
* **Conclusion Table:** Dev.to readers love a quick "Cheat Sheet" or comparison table to wrap up a long-form post.

Beyond JSON: A High-Performance Guide to Thrift & Protocol Buffers (Part 1)

Piyush Gupta — Sun, 05 Apr 2026 14:02:03 +0000

The "Text-Based" Performance Tax

Most of us start our careers in the land of JSON. It’s human-readable, it’s the language of the web, and it’s incredibly easy to debug. But as your system scales from a few hundred requests to hundreds of thousands per second, JSON starts to reveal its "tax."

Why JSON/XML is Killing Your Throughput:

Redundancy: Every single message repeats the keys. If you send a list of 1,000 users, you are sending the string "username" 1,000 times. That’s pure overhead.
Parsing Overhead: Converting a string like "12345.67" into a 64-bit float is a CPU-intensive operation. In high-performance systems, the time spent parsing JSON often exceeds the time spent processing the actual business logic.
Binary Inefficiency: If you need to send binary data (like a profile picture or a byte array), you must use Base64 encoding, which increases the data size by approximately 33%.

The Solution: Binary Encoding. By using a schema-based binary format, we can strip away the field names and focus entirely on the data.

1. Apache Thrift: The Multi-Protocol Powerhouse

Originally developed at Facebook to solve cross-language service communication, Apache Thrift is both an encoding format and a full RPC (Remote Procedure Call) framework.

The Magic of the IDL

Thrift uses an Interface Definition Language (IDL). Instead of defining your data in code, you define it in a .thrift file. This acts as the single source of truth for all your services, whether they are written in Python, Go, or Java.

Binary Protocol vs. Compact Protocol

Thrift offers different ways to "pack" your data:

Binary Protocol: A simple, fast approach that encodes data in a straightforward binary format without much compression.
Compact Protocol: This is where the efficiency shines. It uses Variable-length integers (Varints). For example, the number 7 only takes 1 byte, while 7,000,000 takes significantly more. It also packs field IDs and data types into a single byte whenever possible.

Implementation Example (Thrift IDL)

// user_profile.thrift
namespace java com.piyush.devto
namespace py devto.piyush

struct UserProfile {
  1: required string username,
  2: optional i32 age,
  3: optional bool is_active = true,
  4: optional list<string> tags
}

Python Serialization Code:

from thrift.protocol import TCompactProtocol
from thrift.transport import TTransport
from devto.piyush.ttypes import UserProfile

# Creating a sample object
user = UserProfile(username="Piyush", age=25, tags=["distributed-systems", "backend"])

# Step 1: Initialize a memory buffer
transport = TTransport.TMemoryBuffer()

# Step 2: Use the Compact Protocol
protocol = TCompactProtocol.TCompactProtocol(transport)

# Step 3: Write (Serialize) the object to the buffer
user.write(protocol)

# Get the raw bytes
payload = transport.getvalue()
print(f"Total encoded size: {len(payload)} bytes")

2. Protocol Buffers (Protobuf): The Google Standard

If you’ve ever looked into gRPC, you’ve encountered Protocol Buffers. Developed by Google, it is arguably the most popular binary encoding format in the industry today.

The "Tag" System: Why Order Matters

In Protobuf, the name of the field (username) is never sent over the wire. Instead, Protobuf uses Tags (unique numbers assigned to each field).

When the encoder sees string username = 1;, it simply writes: [Field Tag 1] [Data Length] [Value].

Critical Rule: Once you assign a tag number (like 1), you can never change it. If you change a tag, you break the ability for your services to understand each other.

Visualizing the Binary Layout

While JSON looks like a mess of curly braces and quotes, a binary message looks like a streamlined stream of bits.

Tag & Type	Length/Value	Meaning
`0x12`	`0x06`	Field #1 is a String of 6 bytes
`0x50 0x69 0x79...`	"Piyush"	The actual data
`0x18`	`0x19`	Field #2 is an Integer (25)

Implementation Example (Protobuf)

syntax = "proto3";

package devto.piyush;

message UserProfile {
  string username = 1;
  int32 age = 2;
  bool is_active = 3;
  repeated string tags = 4;
}

Java Implementation:

// Building the object
UserProfile user = UserProfile.newBuilder()
    .setUsername("Piyush")
    .setAge(25)
    .setIsActive(true)
    .addTags("java")
    .addTags("grpc")
    .build();

// Serialization to byte array
byte[] binaryData = user.toByteArray();

// Deserialization on the other end
UserProfile receivedUser = UserProfile.parseFrom(binaryData);
System.out.println("User: " + receivedUser.getUsername());

Summary of Part 1

By moving from JSON to a format like Thrift or Protobuf, you aren't just saving a few bytes. You are:

Lowering Latency: Binary data is parsed up to 10x faster than text.
Reducing Infrastructure Costs: Less bandwidth means lower cloud egress bills.
Enforcing Type Safety: Your API becomes a contract that cannot be easily broken.

Coming up in Part 2: We will dive into Schema Evolution (how to update your data without crashing your app) and why Apache Avro is the undisputed king of Big Data and Kafka pipelines.

B-Trees, Clustered Indexes, and the OLAP Revolution (Part 2) 📊

Piyush Gupta — Sun, 05 Apr 2026 13:52:02 +0000

In Part 1, we looked at LSM Trees—the write-heavy champions found in NoSQL databases. But if you’re using PostgreSQL, MySQL, or Oracle, you’re likely interacting with a different beast: the B-Tree.

Today, we’ll explore why B-Trees still dominate the relational world and how the "Big Data" era forced us to rethink how we store rows entirely.

1. The King of RDBMS: B-Trees
2. Clustered vs. Non-Clustered Indexes
3. OLTP vs. OLAP: The Great Divide
4. Why Column-Oriented Storage Wins at Scale

1. The King of RDBMS: B-Trees

B-Trees are the most widely used indexing structure in history. Unlike the variable-size segments in LSM Trees, B-Trees break the database down into fixed-size pages (usually 4KB to 16KB).

How it Works:

A B-Tree is a balanced tree where each node contains multiple keys and pointers to child pages.

The Root: The entry point for every query.
Leaf Nodes: These contain the actual data or a reference to where the data lives on disk.

Because the tree is always balanced, a B-Tree with a branching factor of 500 and just 4 levels can store 256 billion rows! This makes lookups incredibly consistent at $O(\log n)$.

B-Trees vs. LSM Trees: The Trade-off

Feature	B-Trees	LSM Trees
Best for	Read-heavy workloads	Write-heavy workloads
Fragmentation	High (due to empty page space)	Low (sequential background merges)
Throughput	Lower (multiple disk seeks)	Higher (sequential appends)

2. Clustered vs. Non-Clustered Indexes

Where does the actual row live? This is a common interview question that boils down to index design:

Clustered Index: The index is the data. The leaf nodes of the B-Tree contain the actual row values. You can only have one clustered index per table (usually the Primary Key).
Non-Clustered (Secondary): The index contains a "pointer" (like a Row ID) to the data's location. This allows for multiple indexes but requires an extra "hop" to fetch the full row.

3. OLTP vs. OLAP: The Great Divide

Most web developers spend their time in OLTP (Online Transaction Processing). You handle thousands of small queries: "Update this user’s bio" or "Add this item to the cart."

However, businesses eventually need OLAP (Online Analytical Processing). This involves massive aggregate queries like: "What was the total revenue in Q4 across all Asian markets?"

Running these on your production database will cause it to crawl. Instead, we move data to a Data Warehouse using ETL (Extract, Transform, Load).

4. Why Column-Oriented Storage Wins at Scale

Traditional databases store data Row-by-Row. To calculate an average age, the DB loads the entire row (Name, Email, Bio, Password) just to access one tiny "Age" integer.

Column-Oriented Storage (BigQuery, Snowflake, ClickHouse) stores each column in its own file.

The Code Reality:

Imagine a table with 100 columns and 1 billion rows.

-- Row-oriented: Reads 100 columns from disk per row.
SELECT AVG(age) FROM users;

-- Column-oriented: Reads ONLY the 'age' file. 
-- It ignores the other 99 columns completely.

By only reading the necessary bytes, analytical queries that used to take hours now take seconds. Furthermore, because column data is often repetitive (e.g., many users living in the same "City"), these files compress significantly better than row-based data.

💬 Engineering Challenge

Most modern architectures use Polyglot Persistence—an RDBMS (B-Trees) for user transactions and a Data Warehouse (Columnar) for analytics.

Join the conversation:

Have you ever crashed a production DB by running a heavy "Analytical" query during peak hours?
What’s your "Big Data" tool of choice—Snowflake, BigQuery, or maybe self-hosted ClickHouse?

Drop a comment below with your horror stories or favorite setups! 🛠️✨

How Databases Actually Work: From Log Files to LSM Trees (Part 1) 🚀

Piyush Gupta — Sun, 05 Apr 2026 13:49:52 +0000

We often treat databases like PostgreSQL, MySQL, or MongoDB as magic black boxes. We send a query, and data comes back. But what is actually happening on the disk?

If you've ever wondered why some databases are "fast for writes" while others are "fast for reads," the answer lies in the Storage Engine.

In this post, we’re going to build a database from scratch and evolve it into a production-grade LSM Tree.

1. The World’s Simplest Database
2. Adding an Index (The Hash Map)
3. Solving the Space Crisis: Compaction
4. The Power of SSTables and LSM Trees

1. The World’s Simplest Database

The simplest way to store data is to just append it to a text file. No complex schemas, just raw speed.

# Simple Key-Value Store in Bash
db_set() {
  echo "$1,$2" >> database.db
}

db_get() {
  # We use 'tail -n 1' to get the most recent update for that key
  grep "^$1," database.db | sed "s/^$1,//" | tail -n 1
}

The Verdict:

Writes: O(1) — Extremely fast. You just append to the end of the file.
Reads: O(n) — Terrible. To find one key, you have to scan the entire file from start to finish.

2. Adding an Index (The Hash Map)

To fix the $O(n)$ read problem, we use a Hash Index. Think of this as a "Table of Contents" kept in your server's RAM.

# Conceptual In-Memory Index
index = {
  "user_123": 0,    # Byte offset in the file
  "user_456": 64,
  "user_789": 128
}

def get_data(key):
    offset = index[key]
    file.seek(offset)
    return file.read_line()

This is how Bitcask (the default storage engine for Riak) works. It’s incredibly fast, but there’s a catch: All your keys must fit in RAM. If you have billions of keys, your server will crash.

3. Solving the Space Crisis: Compaction

Since we only append to our log, the file grows forever even if we're just updating the same key. Databases solve this via Compaction.

We break the log into segments. Once a segment reaches a certain size, we close it and start a new one. A background process then merges these segments, throwing away old, overwritten values.

4. The Power of SSTables and LSM Trees

What if we store our data files sorted by key? This is a Sorted String Table (SSTable). Sorting allows us to merge segments efficiently (like Merge Sort) and perform range queries.

The Modern Architecture:

Memtable: All writes go to a balanced tree in memory (AVL or Red-Black Tree).
SSTable: When the Memtable gets too big, we flush it to disk as a sorted file.
WAL (Write-Ahead Log): Before writing to the Memtable, we append the operation to a "crash-recovery" log on disk.

class StorageEngine:
    def write(self, key, value):
        # 1. Append to WAL (for crash recovery)
        self.wal.append(key, value)

        # 2. Add to Memtable
        self.memtable.add(key, value)

        if self.memtable.is_full():
            self.memtable.flush_to_sstable_on_disk()

LSM Trees (Log-Structured Merge Trees) are used by Cassandra, RocksDB, and LevelDB. They are the kings of write-heavy workloads!

💬 Let's Discuss!

Have you ever run into a situation where your database writes were lagging? Did you realize your storage engine might be the bottleneck?

Question for the comments: If you were building a high-frequency trading app with millions of updates per second, would you choose an LSM-based store or a traditional RDBMS? Why?

Master-Class: Understanding Database Replication (Single, Multi, and Leaderless)

Piyush Gupta — Sun, 05 Apr 2026 13:39:36 +0000

Database Replication is the process of keeping a copy of the same data on multiple nodes. Whether you are aiming for high availability, reduced latency, or horizontal scalability, choosing the right replication algorithm is critical.

In this guide, we will explore the three primary algorithms used in modern distributed systems: Single Leader, Multi-Leader, and Leaderless.

Single Leader Replication
Multi-Leader Replication
Leaderless Replication
The Replication Lag Problem
Summary Comparison

1. Single Leader Replication

This is the most common approach (used by MySQL, PostgreSQL, and MongoDB). One node is designated as the leader (master), and all other nodes are followers (read replicas).

How it Works

Writes: All write requests must be sent to the leader. The leader writes the data locally and sends the change to all followers.
Reads: Clients can read from the leader or any follower.

Synchronous vs. Asynchronous

Synchronous: The leader waits for followers to confirm the write.
- Pros: Guaranteed consistency.
- Cons: High latency; if one node fails, the whole write pipeline blocks.
Asynchronous: The leader confirms the write immediately.
- Pros: High performance.
- Cons: Risk of data loss if the leader fails before followers sync.

Handling Failures

Follower Failure: A follower "catches up" by using its local log to request missing data from the leader.
Leader Failure (Failover): Requires detecting failure via timeouts, electing a new leader, and reconfiguring the system.

2. Multi-Leader Replication

In this setup, more than one node can accept writes. This is typically used for applications spread across multiple geographic data centers.

Use Cases

Multi-Data Center Operation: Users write to the nearest data center to reduce latency.
Offline Operation: Apps like calendars or note-taking tools act as local "leaders" that sync with a server later.

The Challenge: Conflict Resolution

If two users edit the same data in different data centers simultaneously, a conflict occurs.

Conflict Avoidance: Routing all writes for a specific record to the same leader.
Convergence: Using Last Write Wins (LWW) or Conflict-free Replicated Data Types (CRDTs) to merge changes.

3. Leaderless Replication

Popularized by Amazon’s Dynamo, this approach allows any node to accept writes and reads. Systems like Cassandra and Riak use this model.

Quorums ($n, w, r$)

To maintain consistency without a leader, these systems use quorums:

$n$: Total number of replicas.
$w$: Nodes that must confirm a write.
$r$: Nodes that must be queried for a read.
The Rule: For a successful read of the latest data, $w + r > n$.

Fixing Stale Data

Since nodes can go down, systems fix stale data via:

Read Repair: When a client detects an old version during a read, it pushes the newer value back to that node.
Anti-Entropy: A background process that constantly syncs data between replicas.

The Replication Lag Problem

Regardless of the algorithm, asynchronous replication often results in "replication lag." To maintain a good user experience, developers should implement:

Read-Your-Own-Writes: Ensures a user always sees the updates they just made.
Monotonic Reads: Ensures a user doesn't see data "disappear" when querying different replicas.
Consistent Prefix Reads: Guarantees that if writes happen in a specific order, they are read in that same order.

Summary Comparison

Algorithm	Best For	Main Downside
Single Leader	Read-heavy apps, general simplicity	Leader is a single point of failure for writes
Multi-Leader	Multi-region apps, offline capabilities	Extremely complex conflict resolution
Leaderless	High write throughput, high availability	Complexities in eventual consistency