DEV Community: Divyanshu Deepam

Beyond the Cheat Sheets: How to Actually Reason About Partitioning VS Sharding in System Design Interview

Divyanshu Deepam — Thu, 28 May 2026 14:43:29 +0000

You are mid-way through a system design interview, confidently whiteboarding your database architecture. You casually drop the word: “…and then we’ll just shard the database.”
The interviewer leans forward, smiles, and asks a devastatingly simple follow-up: “Why? Why can’t we just use database partitioning here?”. Suddenly, you freeze.

Without a crisp, production-grade mental model of the difference between the two, even experienced engineers get caught off guard. Whiteboard cheat sheets give you clean, sanitized definitions: Partitioning is local; sharding is distributed.
But when your system is melting down, textbook definitions won’t save you. Let’s look at the actual operational differences, the hidden bottlenecks, and how to choose between them.

The Golden Rule of Database Splits

Before writing a single line of DDL, anchor your brain to this fundamental truth:

All sharding is partitioning, but not all partitioning is sharding.

Partitioning is the splitting of data into smaller parts for manageability or performance. Partitioning is mainly for manageability and improving the performance, like if you want to have faster queries.

Partitioning does not necessarily mean distributed systems.

Sharding is a special type of partitioning where partitions are on different machines. It is horizontal scaling in its true sense.

The main goal of sharding is to have:

more throughput
more storage capacity

Scenario 1: The Query Trap

Imagine you built an ecommerce platform. Your Orders table has ballooned to 500 million rows, and your latency graphs look terrible. Queries fetching recent orders are crawling.

What do you do ?

The best approach is to partition by date or month. Queries become faster , old partitions can be archived. This is easy on maintenance and no extra machine cost.

The Engineering Benefit:

Partition Pruning: When a user checks their recent orders, the database query planner instantly ignores 95% of the table and searches only the specific month’s partition file.
Zero-Cost Data Retention: When data gets old, you don’t run a massive, CPU-locking Delete query. You simply drop or archive the entire historical partition file instantly
At this point, partitioning solves query efficiency - but not machine capacity.

Scenario 2: The Infrastructure Ceiling

Now imagine your platform explodes in popularity. You hit 50 million active users, and your primary database machine is choking on write throughput. The CPU is pinned, disk I/O is saturated, and your connection pool is exhausted.

What do you do now ?

Now , partitioning alone is not enough because the bottleneck is no longer Query Optimisation. So now you must do Sharding.

Once CPU, RAM, storage or write throughput starts becoming bottlenecks for DB machine , you think of Sharding.

You can map User Id 1-10M to Shard A
10M-20M to Shard B , and so on.

What did you gain by this and how does it improve write throughput in your scenario ?

The write requests will get distributed among different shards. The throughput increases because suppose earlier you had 1 machine with ability to support 10,000 QPS now you have 10 such machines (shards) so you can simultaneously process 100,000 Queries each second.

But like everything in life , Sharding is not free optimisation , its a tradeoff. You did gain on throughput , memory , storage but there are downsides as well. This leads us to…

The Senior Engineer Reality Check: Sharding is a Tax
(1) The Rebalancing Nightmare: If one shard becomes a hotspot due to a poorly chosen shard key, re-sharding and moving live production data across network boundaries is a cumbersome, high-risk operations project.

(2) Costly Joins: If you have to run queries which have lot of joins , that becomes really costly or flat-out unsupported. You are forced to handle complex join logic in your application layer.

(3) Shard Hotspot: If you dont choose your shard key carefully based on your query pattern then one shard can become hotspot and you come back to square one, dealing with same problem which you were trying to solve via Sharding.

Hence when beginners think "Sharding increases scalability" , Senior Engineers have a different mindset :

Sharding is expensive and should be avoided until necessary

Partitioning helps you manage data better. Sharding helps you survive scale.

Good engineers don’t shard because it sounds advanced.
They shard only when a single machine becomes the bottleneck.

10 Avro Schema Mistakes Even Experienced Developer Do

Divyanshu Deepam — Tue, 26 May 2026 18:55:19 +0000

Avro schemas widely used with messaging systems like Apache Kafka to serialize messages into a compact binary format. This dramatically reduces bandwidth and storage overhead compared to sending verbose formats like JSON or XML.
They look deceptively simple until a tiny mistake breaks your serializer, schema registry validation, or downstream consumers.

Some mistakes are obvious rookie errors. Others are subtle enough that even experienced developers make them when moving fast.

I built Dev Suite Avro Schema Validator ( https://devsuite.tools/avro-schema-validator ) to automate the painful part. Paste your schema, and it validates, analyzes, and flags production-breaking issues in seconds.

Here are 10 real Avro schema mistakes that can quietly break production pipelines:

(1) Referencing a Named Type with an Incorrect Fullname

Rule: Subsequent references to a declared named type MUST be made by its full name.

Test Case

{
  "type": "record",
  "name": "Order",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "shippingAddress",
      "type": {
        "type": "record",
        "name": "Address",
        "fields": [
          { "name": "city", "type": "string" }
        ]
      }
    },
    {
      "name": "billingAddress",
      "type": "com.Address"
    }
  ]
}

Why this fails
Named type references containing a dot are treated by Avro as explicit full names, not partially qualified names. Since com.Address does not match the declared full name com.devsuite.Address, schema resolution fails.

(2) Referencing a Named Type Before It Is Declared

Rule: All named types used within a schema MUST be declared where they are first used.

Test Case

{
  "type": "record",
  "name": "Customer",
  "namespace": "com.devsuite.core",
  "fields": [
    {
      "name": "primaryAccount",
      "type": "com.devsuite.core.Account"
    },
    {
      "name": "secondaryAccount",
      "type": {
        "type": "record",
        "name": "Account",
        "fields": [
          { "name": "routingNumber", "type": "string" }
        ]
      }
    }
  ]
}

Why this fails
The primaryAccount field references com.devsuite.core.Account before it is inline-declared in the secondaryAccount field. Avro parses top-down, left-to-right; a type cannot be referenced by full name until the parser has actually encountered its full structural declaration.

Declare the named type before the first reference that uses it.

(3) Alias Contains the Type’s Own Name

Rule: The aliases attribute MUST NOT contain the name attribute of the named type.

Test Case

{
  "type": "record",
  "name": "Account",
  "namespace": "com.devsuite",
  "aliases": ["Account", "OldAccount"],
  "fields": [
    { "name": "accountId", "type": "string" }
  ]
}

Why this fails
Aliases in Avro exist to support schema evolution, particularly when a type has been renamed but older producers or consumers may still refer to the previous name. Including the current schema name inside its own aliases list defeats the purpose entirely because the alias mechanism is meant to represent alternate historical identities, not duplicate the current one.

Account should be removed from the aliases array

(4) Duplicate Equivalent Types Inside a Union

Rule: Any primitive type MUST be included at most once, which also applies to logical type annotations. A UUID logical type, which annotates string, and a string primitive type therefore MUST NOT appear in the same type union.

Test Case

{
  "type": "record",
  "name": "Event",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "identifier",
      "type": [
        "string",
        {
          "type": "string",
          "logicalType": "uuid"
        }
      ]
    }
  ]
}

Why this fails
Avro unions distinguish branches by schema type category, not semantic meaning. Since a UUID logical type is still fundamentally a string, combining both creates ambiguity during resolution.

(5) Multiple Arrays or Maps Inside the Same Union

Rule: A union MUST NOT contain more than one array type and NOT more than one map type.

Test Case

{
  "type": "record",
  "name": "DataPayload",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "collections",
      "type": [
        { "type": "array", "items": "string" },
        { "type": "array", "items": "int" }
      ]
    }
  ]
}

Why this fails
The issue is that Avro distinguishes union members by top-level schema category, not by their internal configuration details. Two arrays are still both arrays, regardless of whether their item definitions differ. The same applies to maps. During deserialization, Avro cannot reliably determine which union branch should be selected purely based on the fact that both branches are structurally the same top-level type. This makes the schema ambiguous and therefore invalid.

Tip - Wrap structurally different meanings inside named records instead of directly placing multiple arrays or maps in the same union.

(6) Union Default Value Does Not Match the First Type

Rule: The default value of a union field MUST match the structure of the first type declared in the union array.

Test Case

{
  "type": "record",
  "name": "UserStatus",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "state",
      "type": ["null", "string"],
      "default": "ACTIVE"
    }
  ]
}

Why this fails
The union allows null or a string. However, because “null” is the first element in the array, the default value must be null.
If a developer wants the default value to be “ACTIVE”, they must reorder the union to [“string”, “null”]

(7) Enum Default Not Present in Symbols

Rule: The default value for an enum must be a string that exactly matches one of the values defined in the symbols array.

Test Case

{
  "type": "record",
  "name": "UserStatus",
  "namespace": "com.devsuite.test",
  "fields": [
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "StatusEnum",
        "symbols": ["ACTIVE", "INACTIVE", "BANNED"]
      },
      "default": "PENDING"
    }
  ]
}

Why this fails
Enum defaults are not arbitrary fallback strings chosen by business meaning; they must correspond exactly to one of the enum’s declared symbols. Developers frequently make this mistake when renaming enum values during schema evolution or when choosing a semantically meaningful default that feels right but no longer exists in the actual symbols array.

(8) Default Object Missing Required Fields

Rule: The default attribute value MUST be a structurally valid instance representation of that specific type.

Test Case

{
  "type": "record",
  "name": "MapData",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "coordinates",
      "type": {
        "type": "record",
        "name": "Point",
        "fields": [
          { "name": "x", "type": "int" },
          { "name": "y", "type": "int" }
        ]
      },
      "default": {
        "x": 100
      }
    }
  ]
}

Why this fails
Default values in Avro are not placeholders or partially descriptive hints; they must be fully valid structural representations of the declared schema. A common mistake happens when developers define a nested record default and provide only some of the fields, assuming omitted values will somehow be inferred or auto-filled. Avro does not do that unless those omitted fields themselves define valid defaults. If a nested record requires fields x and y , providing only x makes the entire default structurally incomplete. This becomes especially easy to miss in large schemas where nested records span many fields and developers manually craft defaults under time pressure.

(9) Invalid Duration Logical Type Definition

Rule: The duration logical type extends the fixed type and must annotate a fixed size of exactly 12 bytes.

Test Case

{
  "type": "record",
  "name": "Timeline",
  "namespace": "com.devsuite",
  "fields": [
    {
      "name": "windowSize",
      "type": {
        "type": "fixed",
        "name": "Interval",
        "size": 8,
        "logicalType": "duration"
      }
    }
  ]
}

Why this fails
In Avro, duration is extremely specific. It represents three unsigned 32-bit integers corresponding to months, days, and milliseconds, which together require exactly 12 bytes of fixed storage. If duration is attached to anything other than a fixed type of size 12, the binary representation no longer matches Avro’s expected encoding contract.

(10) Fixed Type Without Valid Size

Rule: A fixed type must have a size attribute that is an integer strictly greater than zero.

Test Case

{
  "type": "record",
  "name": "HashRecord",
  "namespace": "com.devsuite.test",
  "fields": [
    {
      "name": "md5",
      "type": {
        "type": "fixed",
        "name": "MD5"
      }
    }
  ]
}

Why this fails
A fixed type exists specifically to represent a binary blob of exact known size. Without a valid positive size, Avro has no idea how much memory should be allocated or how many bytes should be read and written during serialization. Developers often scaffold fixed types quickly for hashes, binary identifiers, or protocol payloads and forget to define the actual size, treating it as metadata to fill later. But for Avro, size is fundamental to the schema’s structural definition.

Documentation is Debt: Why Your Class Diagrams Should Be Executable

Divyanshu Deepam — Mon, 27 Apr 2026 16:40:35 +0000

In high-stakes system design sessions or SDE-2 interview prep, the "Class Diagram" is the hero of the hour. We map out domain logic, define inheritance hierarchies, and establish access modifiers.
Then the meeting ends.

The diagram is pasted into a Confluence page or a README, and there it sits—a static relic of a moment in time. As implementation begins, the developer is forced to pay the "Manual Translation Tax": manually creating files, defining boilerplate fields, implementing getters/setters, and ensuring the inheritance links match the drawing.

This is where architectural entropy starts. The moment code diverges from the diagram, the documentation becomes debt.

I wanted to change that. I wanted the diagram to be the Single Source of Truth for the initial implementation.

Introducing: Executable Architecture in Dev Suite

I’ve integrated a Mermaid Class Diagram to Code Generator into Dev Suite.
The philosophy is simple: If the structure is already defined, the code should be a byproduct, not a chore.

Why Mermaid?

Mermaid has become the industry standard for "Diagrams as Code" because it is:

Versionable: It lives in Git alongside your source code.
Text-Based: No proprietary binary files or drag-and-drop UI lag.
Ubiquitous: Native support in GitHub, GitLab, and most modern wikis.

The Real Friction: Moving Beyond Visualization

Imagine this Mermaid snippet:

classDiagram
    class User {
        +String name
        +int age
        +login() bool
    }
    class Admin {
        +String role
        +deleteUser(User u)
    }
    User <|-- Admin

Most tools simply render a pretty SVG. But for a developer, the "real work" is just starting. You still have to write:

User.java with private fields and public accessors.

Admin.java with the extends keyword.

Method stubs that match the diagram signatures.

Dev Suite automates this transition. You paste the Mermaid syntax and instantly get a production-ready project skeleton.

Under the Hood: Building a Compiler Pipeline

As an engineer, I knew that simple string replacement wouldn't scale. To support the nuances of different languages, I built the generator as a mini-compiler pipeline:
1. The Parser Layer
We use a robust parser to transform the Mermaid string into an Abstract Syntax Tree (AST). This allows the tool to understand relationships like Composition, Inheritance, and Visibility without getting tripped up by formatting or whitespace.

2. The Intermediate Representation (IR)
The AST is converted into a language-agnostic class model. This "source of truth" represents the classes, their members, and their connections.

3. The Language-Specific Emitter
The Intermediate Representation (IR) is piped into specialized emitters. This decoupled architecture ensures that adding support for Python, TypeScript, or C# is a simple matter of writing a new emitter, not rebuilding the core engine.

Why This Matters for Your Workflow

Faster Prototyping: Go from a whiteboard sketch to a compilable project structure in seconds.
Reduced Cognitive Load: Spend your energy on solving business logic, not typing public String getUsername().
Architectural Parity: Ensure the code actually reflects the design approved during the review phase.
Learning Tool: For students and junior devs, seeing the immediate translation of visual OOP concepts into code is a powerful educational feedback loop.

I want to hear from you:
What is the most tedious part of your design-to-dev workflow? What "mechanical" task do you wish was automated?

Try the generator now:
Mermaid Class Diagram to Java
Mermaid Class Diagram to C++

How I Built a React JSON Tool That Handles 150k+ Lines Without Freezing the Browser

Divyanshu Deepam — Thu, 23 Apr 2026 19:13:27 +0000

Most JSON tools work fine… until they meet real-world production data.

We’ve all been there: You paste a massive payload into a web-based formatter, and suddenly:

The page stops responding.
Scrolling becomes a stuttering mess.
The "Page Unresponsive" popup appears.
I encountered this while debugging complex automation flows and deeply nested configurations. I needed a tool that could handle 150k+ lines of JSON without falling apart, but I didn't want to compromise on privacy by sending that data to a backend.

So I built Dev Suite - a client-side toolkit designed for performance-first JSON manipulation.

Why Large JSON is a "Systems Engineering" Challenge

Performance in the browser isn't just about a fast for loop. It’s a multi-layered bottleneck involving:

Parsing Cost: Converting a 10MB+ string into a JavaScript object.
Memory Pressure: Storing that object and its associated metadata.
DOM Complexity: The browser struggling to calculate layout for 5,000+ nodes.
React Overhead: Re-renders triggered by state changes during heavy interactions. Here is how I tackled the specific challenges of scaling the tool from 50k to 150k+ lines.

1. JSON Path Finder: Reducing the Noise

The Path Finder returns dot-notation paths (e.g., payment.gateway.retry.maxAttempts). While the logic seems simple, iterating through a massive tree structure can generate significant "noise."

The Challenge: Array Explosion
If you search for a key like feeTypesand it’s an array of 1,000 objects, a naive search might return: feeTypes[0], feeTypes[1], feeTypes[2]...

The Fix: AST-Based Traversing
I implemented a search algorithm that walks the Abstract Syntax Tree (AST) of the JSON. Instead of a flat search, I added conditional result filtering. This ensures users see the meaningful parent matches first. By intelligently pruning the traversal, we keep the UI clean and the search execution time sub-millisecond.

2. Solving the "5,000 Element" Freeze

If a search returns 5,000 matches, and React tries to render 5,000 buttons (each with icons, hover states, and tooltips), the main thread will lock up for several seconds while the browser paints.

The Solution: Virtualization
I implemented DOM Virtualization. Instead of rendering all 5,000 results, the tool only renders the ~20 items currently in the user's viewport. As the user scrolls, DOM nodes are recycled and updated with new data.

3. Bypassing the React Render Cycle

React is amazing for state management, but it can be a bottleneck for text editors. If every keystroke in a 100k-line file triggers a React state update and a virtual DOM diff, the latency (typing lag) becomes unbearable.

The Fix: Decoupling the Editor Engine
I moved the "hot path" away from React's state. By using an uncontrolled component approach with the underlying editor engine (CodeMirror/Monaco), I severed the connection between the typing engine and React’s render cycle.

4. JSON Diffing

A standard text diff is useless for JSON because key order often doesn't matter. JSON Diff Checker helps to check differences between two JSONs semantically.

The Solution: Order-Invariant Comparison
I built a recursive diffing algorithm using Map and Set data structures.

It identifies Added, Removed, keys with modified value and keys with value type changed.
It ignores object key order (semantic equality).
It handles nested structures recursively.

Navigating the Diffs
In a 150k line file, you can't just "jump" the user around. I implemented Binary Search over the sorted array of diff locations. This allows the "Next" and "Previous" buttons to find the closest difference relative to the user's current scroll position instantly.

The Result

Dev Suite now handles 150k+ lines of JSON with ease. It’s been an insightful journey into the limits of browser performance and the importance of choosing the right tool for the right job (even if that tool is "Not React" for certain specific tasks).

There are more tools at Dev Suite , you can explore
👉 https://devsuite.tools/mermaid-class-diagram-to-java
👉 https://devsuite.tools/mermaid-class-diagram-to-cpp
👉 https://devsuite.tools/yaml-diff
👉 https://devsuite.tools/yaml-pathfinder

I'd love to hear from you:
What’s the largest JSON payload you’ve had to debug in a browser? Did your existing tools survive, or did you have to reach for the terminal?

Let's discuss in the comments!