DEV Community: Anusha Mukka

The Illusion of Scale, Part 2: When Your Data Model Becomes Your Bottleneck

Anusha Mukka — Sun, 17 May 2026 06:19:41 +0000

I want to talk about the cruelest kind of technical debt. Not the kind where someone wrote bad code, and you can see it. The kind where the code is clean, the tests pass, the results are correct, and you're still screwed.

Data model debt.

It hides. For months, sometimes years. It doesn't announce itself. It just sits inside perfectly functional code, returning correct results, passing every test. And then one day, you realize everything else is built on top of it, and you cannot move it without moving everything.

This is Part 2 of a series about assumptions that quietly break systems at scale.

The customer who broke our schema

A few years into working on a multi-tenant system, we onboarded a large enterprise customer. System had been running great for over a year at that point. Hundreds of tenants, smooth operations, no major incidents. We were feeling pretty good about ourselves.

This customer had fifty million records in a table where our typical tenant had maybe fifty thousand.

Same schema. Same queries. Same everything. But queries that ran in 200ms for every other tenant were running in 45 seconds for them.

Nobody had designed a bad system. The schema had just quietly encoded a belief: that tenants would be roughly similar in size. That belief had never been written down anywhere. Never tested. Never questioned. It was just... assumed, the way you assume things that have always been true until they suddenly aren't.

The fix was conceptually simple -- partition the data, route large tenants differently. The implementation took months, because everything else had been built around that original schema. Every query, every index, every join had opinions about how the data was structured. We ended up running two schemas simultaneously for six weeks to migrate without downtime.

It was the most expensive technical debt I've ever watched get paid off. And I'm including the time someone accidentally dropped a production table (different story, different company, different bottle of wine).

Why "good design" has an expiration date

Here's the thing about data models: they're designed for the use cases the team can see right now. That's almost always the wrong frame, because the use cases that matter at scale are the ones nobody anticipated when the schema was first written.

The pattern is incredibly consistent. System starts with a well-normalized schema. Foreign keys everywhere. Third normal form. At moderate load, it's fine. Correct, even. Textbook stuff.

Then volume grows. Queries that touched thousands of rows now touch millions. Joins that were fast become table scans. The query planner starts making choices that surprise you, and suddenly you're reading execution plans at midnight -- midnight -- trying to understand why a query that used to take 80ms now takes 12 seconds.

Normalization optimizes for write correctness and storage efficiency. Not read performance at volume. When your read load is enormous relative to your write load -- which it is in basically every user-facing system -- those goals pull in opposite directions. You find out which one your schema actually prioritized the hard way. Usually on a Friday.

The cardinality time bomb

Okay, this one's personal because I've made this exact mistake.

A permissions table with one row per user-resource pair. Fine when users have tens of permissions. Completely reasonable design. Then fine-grained access becomes a product requirement and users can have thousands of them. Table gets enormous fast.

Every permission check is now a large query. Every access decision slows down. And because authorization sits in the critical path of almost everything, a slow permissions table makes the whole system feel sluggish in ways that are incredibly hard to diagnose. You end up chasing phantom performance issues across half the codebase before someone finally traces it all the way back to a table that's just too big to query efficiently anymore.

The schema wasn't badly designed. It was designed for a world where users had 10-20 permissions. Then the product team said "actually, we need thousands" and the schema didn't get the memo.

When you design a schema, there are two questions: "what cardinality do I expect?" and "what cardinality could this legitimately reach?" They're not the same question. The first one is optimistic. The second one saves you.

When being correct gets too expensive

If producing the accurate answer requires joining five tables and aggregating across millions of rows... correctness has a real cost. A cost you pay on every single request.

Your options at that point are denormalization, pre-computation, materialized views, or derived tables. They all work. They all introduce consistency challenges that the normalized schema never had. That's the actual tradeoff, and it's worth naming clearly: not "normalization vs. performance" but "easy to get right" vs. "fast under real load."

Choosing consciously is very different from discovering the tradeoff at 3am during an incident. Trust me on this.

Migrations: where you pay the real price

A migration that takes 30 seconds in development can take three weeks in production. Not because the operation changed. Because the table grew from thousands of rows to billions, and suddenly every part of the process has consequences you never thought about.

Locking is the first problem. DDL operations on large tables can block reads or writes even briefly. "Briefly" on a hot table cascades into timeouts across the entire system within seconds.

Backfill is the second. Writing a new column's default value to a billion rows is a lot of I/O competing directly with live traffic.

And then there's the dual-write period -- running old and new schemas simultaneously so you can migrate without downtime. This is the right approach. It's also the approach that reveals every single implicit assumption in your application code. Things you didn't know your code believed about the schema. Fun.

It almost always happens under pressure too. Nobody says "let's do a major schema migration" when things are going well. They say it when things are on fire. Plan for it before you're in that situation. You won't, but you should.

What I'd tell my past self

Design for your read patterns, not just your write patterns. Know which queries are on your critical path and whether your schema serves them cheaply or with heroics.

Write down your cardinality assumptions explicitly before you ship. Explicitly. In a document. "This table is expected to have X rows per tenant. At Y rows, query Z will degrade." If you can't fill in those numbers, the answer to "will this hold at scale?" is also unclear.

Separate your operational and analytical models early. The schema optimized for transactional correctness is rarely the schema optimized for reporting. Trying to serve both from one schema is a compromise that satisfies neither at volume.

And treat major schema changes as an operational project, not a technical task. They need a plan, a rollback strategy, a communication plan, and ideally someone who has done it before and can warn you about the part you haven't thought of. There's always a part you haven't thought of.

Next up: why access control is one of the most quietly expensive places for schema assumptions to go wrong at scale. Spoiler: 15 roles became 340.

What data model decision have you had to undo the hard way? I want the painful stories. The "we ran two schemas for six weeks" stories. The more awful, the more I want to hear it.

The Illusion of Scale, Part 1: When Your "Scalable" System Isn't

Anusha Mukka — Mon, 11 May 2026 00:48:03 +0000

I want to talk about something that's been bugging me for a while.

There's this moment -- and if you've been in this industry long enough you know exactly what I mean -- where a system that looked rock solid just... stops working. Not dramatically. Not with a big crash and a SEV page at 3am (though sometimes that too). It's more like a slow suffocation. Latencies creep up. Queues get deeper. Someone opens a ticket that says "it feels slow" and you roll your eyes because everything feels slow to users, but then you look at the graphs and oh. Oh no.

I've been on both sides of this. I spent years working on public-sector infrastructure -- criminal justice workflows that had to work across 87 counties in a state, which sounds boring until you realize that "87 counties" means 87 different usage patterns, 87 different peak hours, and at least 12 counties who will absolutely hammer your API in ways you never anticipated. More recently I've been in enterprise AI infrastructure, where the fun game is "this API call costs $0.003 and we make it 40 million times a month, do the math."

Both times, the system didn't fail because we forgot to add servers. It failed because of something dumber.
This is the first in a series I'm writing about scale assumptions. I don't have a clever acronym for it. It's basically: the decisions that seem fine when you're small and make you want to quit your job when you're big.

Linear thinking will absolutely wreck you

Here's the thing nobody tells you early in your career: scaling is not a linear problem, and your intuition about it is almost certainly wrong.

A system handles 1,000 req/s. So 100,000 is just... more machines, right? Tune some indexes, maybe bump the connection pool, call it a day?

Sometimes, honestly, yes. I've had that experience and it's great. You feel like a genius. "We just horizontally scaled it." High fives all around.

But more often -- and this is the part that took me embarrassingly long to internalize -- the bottleneck isn't compute. It's a design choice someone made in week 2 of the project that seemed totally reasonable at the time.

I'll give you a specific example because I think abstractions are useless here.

We had a system running in pilot with one county agency. Worked beautifully. Fast, stable, everyone's happy. We expand to three agencies. Same code. Literally the same code, no changes. System slows down noticeably.

I remember staring at the metrics genuinely confused. Nothing changed! What is it then?

What changed was width. Three agencies meant three times the concurrent load on shared workflow components. Database access patterns that were totally fine with one agency's usage started colliding. Integration points that had been sized for one agency's volume were now contested. It wasn't a bug. It was an assumption -- that the system would scale linearly with tenants -- that nobody had written down because nobody had thought to question it.

That was the week I started losing sleep about the statewide rollout. Not because the architecture was bad -- it was actually pretty solid for what it was designed for -- but because "what it was designed for" and "what it was about to face" were diverging fast.

The synchronous call in the hot path (a.k.a. my nemesis)

Okay, pet peeve time.
A 50ms synchronous call to a downstream service. Totally fine at low traffic. You barely notice it. It's in the critical path but hey, 50ms, who cares.

Then traffic goes 10x and suddenly that 50ms dependency is your ceiling. Every request is waiting on it. When it has a bad day, you have a bad day. When it times out, you time out. And the really fun part: by the time you realize this is the problem, it's woven into everything. You can't just "make it async" without rearchitecting half the request flow.

I don't have a clean solution here. I just have scar tissue.

Data models: where optimism goes to die ####

I need to rant about schemas for a second.
Every bad scaling story I have eventually comes back to the data model. Not because anyone designed a bad schema -- usually the schema was perfectly sensible for the requirements as understood at the time. The problem is that schemas encode beliefs about the future, and we are terrible at predicting the future.

Beliefs like:
●      "We'll only have a handful of roles" (we now have 47)
●      "This workflow has 4 states" (it has 11, plus 3 that are technically illegal but exist in prod)
●      "This lookup will always be fast" (it was, until someone added a tenant with 2M records)
These aren't mistakes. They're reasonable bets that didn't pan out. But the wreckage is the same either way.

The logging bill

This one still makes me laugh in a pained way.
You start a project. Good engineering culture. "Let's log everything, we'll need it for debugging." Absolutely correct instinct! Gold star.
Fast forward 14 months. Someone pulls up the infrastructure bill and goes "uh, why is our logging pipeline costing more than our actual application?" And everyone looks at each other. Nobody planned for this. Nobody put "the audit trail will eventually need its own architecture team" on any roadmap. It just... happened. Slowly, and then all at once.

p99 is not a rounding error

I used to think about p99 the way most people do: as an edge case. The unlucky 1%.

Then I did the math on a system doing 100k req/s and realized that 1% is a thousand requests every second getting a bad experience. Those aren't theoretical users. They're filing support tickets. They're hitting retry. Their retries are making other requests slower. The p99 tail is generating its own secondary workload that feeds back into the system.

Your unhappy path, at scale, is a system unto itself. That realization changed how I think about optimization priorities pretty fundamentally.

What actually breaks (spoiler: it's never what you tested)

Look. I have never -- not once in my career -- seen a system fail in production the same way it failed in load testing. The tests always pass because test traffic is polite. Real traffic is feral.
Real traffic is: retries stacking on retries. One tenant with 10x everyone else's data volume. A permissions edge case that only fires for one specific role combination that nobody on the QA team had. Duplicate events from an upstream that swore they'd deduplicate on their end. Events arriving out of order because someone's clock is wrong.

The thing I got most wrong, personally: I assumed a decision-making component would maintain consistent latency as we onboarded more systems. In isolation, it was fast. Really fast. What I didn't think about was what happens when multiple systems are doing concurrent writes to the shared database underneath it. The component was fine. The contention was the problem. And you can't see contention in a single-system test. By definition.

I think the broader lesson -- and sorry if this sounds hand-wavy but I genuinely believe it -- is that at scale, failures happen in the interactions between components. Not in the components. A retry policy that's totally safe in isolation starts amplifying failures when combined with another service's retry policy. Cache invalidation creates cascading churn nobody modeled. A permission check that's microseconds alone shows up on flame graphs when it's called 50,000 times per second.

There's one debugging session that broke my brain a little. Access control issue. Could not figure out where to even look. Turned out we had multiple sources of truth for permissions and they'd drifted apart. The system was just... checking whichever source it hit first. There was no canonical answer to "does this user have access." I had to reconstruct the state of three different systems at a specific timestamp to understand one decision the system had made.
That was when I realized: past a certain scale, you stop debugging code and start debugging emergent behavior. And that's a fundamentally different skill.

So what do you do about it?

I'm not going to tell you to design for massive scale on day one. That's almost always wrong. YAGNI is real. Premature optimization makes systems worse, not better.

But.
Some decisions are genuinely hard to reverse. And you should at least know which ones they are:
●      Your data model (migration under load is hell)
●      Sync vs. async boundaries (you can't easily untangle these later)
●      Consistency vs. availability tradeoffs (distributed systems don't let you change your mind cheaply)
●      Authorization architecture (this one always comes back to haunt you)
●      Audit and retention strategy (see: logging bill, above)

Get any of these wrong and the rewrite happens under pressure, in production, while users are affected, with half the team arguing about the approach and the other half on PTO. It's never the calm six-month project you pitch to leadership.

Next time I'll write about the one that's cost me the most career stress: data modeling decisions that look totally fine on day one and become load-bearing walls by year three. I have stories.

Genuinely curious -- what's the scaling assumption that burned you worst? The one where you looked at the system and went "oh no, this was baked in from the start"? Drop it in the comments, I collect these like trading cards at this point.

When the Cloud is Too Slow: Enter Fog Computing

Anusha Mukka — Sat, 11 Apr 2026 17:38:01 +0000

You know that feeling when you're waiting for a response from your cloud service, and it feels like forever? Now imagine that same delay happening for a self-driving car making a split-second decision, or a smart factory robot on an assembly line. Yeah, not great.

I've been digging into this problem lately, and I wanted to share what I've learned about a pretty cool approach that's gaining traction: hierarchical fog computing combined with some clever optimization tricks.

The Problem: Everything Lives in the Cloud (And That's a Problem)

Here's the thing. We've gotten really good at building cloud infrastructure. AWS, Azure, GCP—they're incredible. But as we add more IoT devices everywhere (smart homes, industrial sensors, autonomous vehicles), we're running into a fundamental issue:

The cloud is physically far away.

When your smart thermostat needs to process data, that packet has to travel potentially hundreds or thousands of miles to a data center and back. For simple tasks, that round-trip can take 50-200 milliseconds. For real-time applications? That's an eternity.

Plus, you're sending everything to the cloud:

Burning through bandwidth 💸
Draining device batteries 🔋
Creating potential privacy issues 🔒
Wasting cloud resources on trivial tasks

There has to be a better way.

Enter Fog Computing: The Middle Ground

Fog computing is basically the answer to "what if we put mini data centers closer to where the action is happening?"

Think of it like this:

Traditional Model:
IoT Device → (hundreds of miles) → Cloud → (hundreds of miles back) → Response

Fog Model:
IoT Device → (few feet) → Fog Node → Decision made locally
→ Only important stuff goes to cloud
The fog layer sits between your devices and the cloud—on routers, gateways, local servers. It handles the time-sensitive stuff locally and only sends the heavy lifting or long-term storage to the cloud.

But Here's Where It Gets Tricky

Okay, so fog computing sounds great. But now you have a new problem: how do you decide what runs where?

Imagine you're managing thousands of IoT devices, and each one is generating tasks that need to be processed. Some tasks are urgent (like collision detection), others are less critical (like uploading historical temperature data). You have:

Edge devices with limited CPU and battery
Fog nodes with medium computing power
Cloud with unlimited power but high latency

The million-dollar question: For each task, where should it run?

This is called the task offloading problem, and it's harder than it sounds because you're trying to optimize multiple things at once:

Minimize latency (keep things fast)
Minimize energy consumption (save battery)
Minimize costs (use resources efficiently)
Respect deadlines (urgent tasks can't wait)

Hierarchical Architecture: Think in Layers

What I've been researching is a three-tier hierarchical approach:

Layer 1: The Edge (Your Devices)

Smartphones, sensors, smart cameras
Super limited resources
Makes quick decisions: "Can I handle this myself?"

Layer 2: The Fog (Local Processing)

Routers, gateways, local servers
Moderate computing power
Handles most of the real-time processing
Coordinates with nearby fog nodes
Only escalates to cloud when necessary

Layer 3: The Cloud (The Big Guns)

Massive data centers
Heavy computations, machine learning training
Long-term storage and analytics
The beauty is that each layer knows its role and passes work up only when needed. It's like having a good manager who doesn't escalate every little thing to the CEO.

The Optimization Challenge: Grey Wolf to the Rescue

So how do you actually decide where tasks should run? You need an algorithm that can:

Make decisions fast (no time for complex calculations)
Handle changing conditions (devices come and go)
Optimize multiple objectives at once
Scale to thousands of devices This is where Grey Wolf Optimization (GWO) comes in. And yes, it's literally inspired by how wolves hunt.

How Wolves Hunt (Seriously)

Grey wolves have a pack hierarchy:

Alpha (α): The leader, makes final decisions
Beta (β): The advisor, second in command
Delta (δ): Scouts, soldiers, elders
Omega (ω): The rest of the pack
When hunting, the pack uses a coordinated strategy:

Track and approach the prey (exploring solutions)
Surround the prey (narrowing down options)
Attack when the time is right (converge on optimal solution)
The algorithm mimics this: you start with a bunch of random solutions (the pack), identify the best ones (alpha, beta, delta), and have the rest follow their lead while still exploring. Over time, everyone converges on the best solution.

Why This Works for Fog Computing

In our case:

Prey = Optimal task distribution across edge/fog/cloud
Pack = Different possible ways to allocate resources
Hunting = Iteratively finding the best solution
The algorithm runs fast (critical for real-time decisions), avoids getting stuck in local optima, and handles the complexity of balancing latency, energy, and cost.

Adding Deep Learning to the Mix

Here's where it gets even better. We can combine GWO with deep learning to make smarter predictions:

Step 1: Predict the Future (kinda)

Use LSTM networks to predict incoming workload patterns:

"Oh, it's 5 PM, traffic pattern analysis requests are about to spike"
"Battery on this device is at 20%, we should offload more"

Step 2: Classify Tasks

Use a feedforward neural network to classify tasks:

Compute-heavy vs. latency-sensitive
High-priority vs. can-wait
Local-capable vs. needs-cloud-power

Step 3: Optimize with GWO

Feed all this info into the GWO algorithm to find the best task distribution in real-time.

Step 4: Learn and Adapt

Use reinforcement learning to improve over time based on actual results.

The Results (Why This Matters)

Early research shows some pretty impressive numbers:

Latency reduction: 40-70% compared to cloud-only approaches
Energy savings: Up to 80% by processing locally when possible
Throughput increase: 80%+ by distributing load efficiently
Faster convergence: 20-30% quicker than traditional genetic algorithms
Real-World Applications

Where does this actually help?

Smart Cities:

Traffic light coordination (can't wait for cloud round-trip)
Emergency response systems
Public safety monitoring

Industrial IoT:

Manufacturing robots (milliseconds matter)
Predictive maintenance
Quality control systems

Healthcare:

Patient monitoring (life-critical response times)
Wearable health devices
Remote surgery assistance

Autonomous Vehicles:

Real-time obstacle detection
Cooperative driving (vehicle-to-vehicle)
Edge-based navigation
Why I Find This Fascinating

I've spent the last decade building distributed systems at scale—from nation-wide law enforcement infrastructure to Meta's monetization platform handling billions of requests. Here's what strikes me about this approach:

It's Practical: This isn't just academic theory. These are real problems I've encountered: how do you reduce latency from hours to minutes? How do you optimize resource allocation when you have millions of users?
It Scales: The hierarchical model mirrors how we build microservices—each layer has a specific job, clear boundaries, and knows when to escalate.
It's Adaptive: Systems that can learn and optimize themselves are way more resilient than static configurations. I've seen this firsthand—adaptive systems survive conditions you never planned for.
It Solves: Multi-Objective Problems In production systems, you're never optimizing just one thing. It's always latency AND cost AND reliability AND user experience. GWO handles this gracefully.

The Challenges (Let's Be Real)

Nothing's perfect. Here are the hard parts:

Complexity: Managing three tiers is harder than managing one. You need coordination, monitoring, fallback strategies.

Edge Heterogeneity: Your edge devices aren't uniform. Different CPUs, memory, network capabilities. The algorithm has to handle this diversity.

Network Reliability: What happens when a fog node goes down? You need fast failover and re-optimization.

Privacy & Security: Distributing processing means distributing attack surface. Need end-to-end security across all layers.

Debugging: Ever try debugging a distributed system? Now add "distributed across thousands of devices in the real world." Fun times.

What I'm Working On Next:

I'm currently diving deeper into:

Reinforcement learning integration: Making the system continuously improve from real traffic patterns

Multi-agent coordination: How fog nodes can collaborate without central control

Fault tolerance: Graceful degradation when nodes fail

Real-world deployment considerations: Because simulations are one thing, production is another

I'm also exploring how this applies to edge AI scenarios—running ML models across the hierarchy, where each layer handles what it can and passes up only what it must.

Try It Yourself - If you want to experiment with fog computing concepts:

Simulation Tools:

iFogSim: Java-based fog computing simulator
EdgeCloudSim: Simulates edge computing scenarios
Python + NetworkX: Build your own simple model

Start Small:

Model a simple 3-tier architecture
Create synthetic tasks with different requirements
Implement a basic task scheduler
Compare random vs. optimized offloading

Read More: The research in this space is moving fast. Look for papers on:

Task offloading strategies
Deep reinforcement learning in edge computing
Optimization algorithms for distributed systems

Final Thoughts

We're at an interesting inflection point. IoT devices are everywhere and getting smarter, but the old "send everything to the cloud" model is hitting physical limits.

Fog computing isn't going to replace the cloud—it's going to make it better by handling what it does best and letting the cloud focus on what it does best.

And optimization algorithms like GWO combined with deep learning? They're giving us tools to manage this complexity at scale, in real-time, with multiple competing objectives.

If you're building IoT systems, industrial automation, edge AI, or anything where latency really matters—it's worth understanding these concepts. The architecture patterns and optimization techniques apply to a lot more than just academic papers.

What do you think? Are you working with fog/edge computing? Running into latency issues with your IoT systems? I'd love to hear your experiences in the comments.

And if you're interested in the full technical details, I'm working on a research paper diving deep into the hierarchical GWO approach. Happy to chat about it!

P.S. - Yes, I did just spend several paragraphs explaining computer science concepts using wolf hunting analogies.. 🐺