DEV Community: Malay Mehta

System Design for Working Engineers, Not Interview Prep

Malay Mehta — Fri, 26 Jun 2026 18:22:31 +0000

Originally published at malaymehta.com

The Interview Trap

If you look at most system design tutorials, you get an extreme use case. Design Twitter. Design YouTube. Scale it to a billion users. Draw boxes on a whiteboard for 45 minutes.

Do you think your app will be used by a billion users on day one? The answer is almost always no. But the tutorials don't teach you what to do when you have 500 users, unclear requirements, a team of four, and a quarter to ship something that works.

Real system design is nothing like a whiteboard interview. You don't get clean requirements, you don't design from scratch, and nobody asks you to handle a billion requests per second on day one.

Real System Design Starts with Questions, Not Diagrams

The very first thing that matters in system design is something most tutorials skip entirely: unclear and chaotic requirements. In the real world, requirements don't come as a clean problem statement. They come from non-technical business teams, and you need to navigate through cross-questions to get all the clarity you need.

Ask as many questions as possible. Understand your functional and non-functional requirements. Which features need to be synchronous and which can be async? What are the read and write load patterns? What is the maximum and average number of concurrent users right now? What does authentication look like? Do you need role-based access control?

These questions drive your choices. You don't always need an axe where a knife will do. Being minimalist with a reasonable growth prediction and a 3, 6, 9 month plan will take you in the right direction.

There will be things the situation demands immediately but would take more time than expected. Taking a predictable hit now and fixing it at the right future time without missing that balance is truly important. Weighing what will be expensive to change later, in terms of dollar cost or human effort, is how real architectural decisions get made.

Pushing Back on Bad Requirements

Many times requirements come from non-technical business teams and you need to push back on why certain things should not be done the way they expect.

Here is a real example. A business person once asked to duplicate data into another Kafka topic because their prediction was that the existing topic would not handle more load from a new subscriber. The technical reality? Kafka is built for exactly this. A new consumer group on the same topic would work without impacting existing consumers at all. If you don't push back, you end up creating tech debt with support and maintenance costs forever, just for replicating data that never needed to be replicated.

Trade-off Decisions Nobody Teaches

Monolith vs Microservices

Typically the very first thing engineers want to talk about is microservices and how they can help. But do you realistically have even 100 users on the product? Why do you need K8s, Docker, distributed tracing, cross-cutting async messaging, and service mesh? Do you really need that scale, or are you doing it to make your resume look better?

If you have no real users in the thousands, a modular monolith is the best choice. Deploy everything as one server on Linux with a reverse proxy and a CNAME record. That simple. You need a database, sure. But you don't need Kafka, distributed tracing, auto-scaling, or any complex distributed computing to begin with.

When predictable growth comes, add monitoring and observability to understand which requests are hitting hardest. Decouple the modules doing heavy work into independent microservices. Then pivot. That is the right sequence.

Synchronous vs Async

If you don't need to process something immediately, decoupling via async helps. If it is fire-and-forget, use a simple queue. If you need multiple services to consume the same event with highly scalable producers and consumers, use Kafka. If the user is waiting for a response, keep it synchronous via a RESTful API because it needs to happen right now.

Build vs Buy

Rule of thumb: never reinvent the wheel. If something already exists at low cost and does the job, buy it. If companies like OpenAI and Anthropic are not building their own payment systems and instead use established financial integrators, then you should trust that. If giants are not building everything from scratch, why should you? Building only makes sense when no existing solution fits your needs.

Consistency vs Availability: Real-World CAP

When you are dealing with transactions that need ACID guarantees, use SQL. Ticket booking, inventory updates, financial debits and credits. These cannot tolerate stale reads or lost writes.

If you need consistency and partition tolerance where stale reads must be errored out, NoSQL works better. Social media feeds, messaging, analytics, and streaming. If you need availability and partition tolerance with tolerance for eventual consistency, columnar databases like Cassandra fit well. IoT data, time series, high write throughput with low read frequency.

Perfect Architecture vs Shipping This Quarter

Perfect architecture is always the goal, but if you can balance it with shipping this quarter, that brings real business value and revenue. Find the healthy mix. Build a base that requires very little change even if the actual decision evolves later.

For example, tightly couple your audit logging service synchronously because you don't have async processing yet. It ships real business value now. Later, when async communication is added, you decouple it without changing how the end user experience works.

Analytics is another one. You might not have the full setup of MySQL CDC to Debezium to ClickHouse yet. But you can start by ingesting specific tables into ClickHouse directly for analytics. Solve it elegantly later when DevOps capacity allows the full event streaming pipeline.

When to Scale and When Not To

The time to scale is based on observability data and predictive customer expansion patterns. Your business understanding combined with analytical thinking will surface the signals that tell you when scaling is actually needed.

Before jumping to horizontal or vertical scaling, check the basics first. Does your database have optimal indexing? Is your application connection pool configured properly? Are there N+1 queries firing hundreds of calls where one would do? These are high-level checks. Deeper concepts like partitioning and sharding are problems you encounter with billions of records, not a few million.

Horizontal scaling is generally the better approach because it guarantees higher throughput with the ability to scale up or down without downtime. But only when you actually need it.

A Real Story: Premature Scaling Gone Wrong

I worked with a company that had fewer than 50-100 customers and less than 5,000 business transactions total. They had already added Docker, Kubernetes, Kafka, distributed monitoring, and auto-scaling. Now they had two problems instead of one: the real business problem and a tech problem.

Very few developers on the team understood microservices as a whole. Nobody knew DevOps practices well enough to manage how scaling actually works. It was not just slowing business delivery but also burning cloud costs because nobody knew how to optimize the infrastructure bill.

A double-edged sword. Premature scaling without proper architectural guidance creates more problems than it solves.

Every Architecture Decision Is a Cost Decision

Will you use managed Kubernetes or bare K8s? New Relic or Dynatrace or open-source alternatives? Managed database or self-hosted? It all depends on who is owning what. If you have DevOps engineers who can manage the nightmare of persistent storage, networking, constant upgrades, and maintenance, then self-hosted can work. If the answer is no, managed is better but it comes with a higher price tag.

It is equally important to monitor your cloud costs and understand the incremental bills. Is your Docker image lifecycle policy set to delete old images within a few days? Is your S3 storage persistent forever or only for a retention period? Have you optimized or dropped high-cardinality metrics in your distributed tracing to save cost? How about networking costs for transporting data across regions? It all adds up.

Here is a question I ask teams all the time: will you optimize your MySQL queries and indexing, or will you throw more money at bigger database instances so the app functions at a dollar cost that keeps increasing? Unless the root cause is identified and fixed, you are just burning money.

Small teams with few users almost always face overly expensive microservices hosting and management. The operational overhead, debugging complexity, and cognitive load on the team need to be balanced against the actual benefits.

Database Design Is Architecture

Schema design decisions haunt you for years. The table structure you choose in month one determines how painful your queries are in year two. Foreign keys, indexes, data types, normalization vs denormalization. These are architecture decisions, not database admin tasks.

Pages taking 10+ seconds to load because nobody thought about indexing. N+1 queries firing hundreds of database calls. Unused columns bloating tables. No caching layer. Complex business logic with if-else ladders that nobody can follow.

For SQL vs NoSQL, the real answer is simpler than the blog posts make it: if you need transactions and relationships, use SQL. If you need flexible schema with high write throughput and can tolerate eventual consistency, use NoSQL. Most applications should start with SQL and add NoSQL for specific use cases when needed.

Caching strategy is another design decision that gets treated as an afterthought. Cache the data that is read frequently but changes rarely. Product catalogs, user profiles, configuration data. Invalidate on write. Start with a simple TTL-based approach and add event-driven invalidation when your system complexity demands it.

Observability Is a Design Decision, Not an Afterthought

Will your app monitoring give you real analytical insights, or is it just collecting logs nobody reads? P99 and P95 latency metrics, structured logs, alerts that tell you production broke at 12 PM instead of finding out at 2 PM when customers start calling.

The real power of observability is the transactional breakdown: cache hit vs miss ratio, time spent in business logic, time spent in SQL queries, external API call latency, queue processing time. When you can see all of this in one trace, debugging goes from guesswork to precision.

Being preventive rather than reactive is what separates teams that sleep well from teams that get paged at 2 AM.

The Architecture Review Nobody Does

Most teams never review their architecture after the initial design. Not a priority. Time constraints. Always something more urgent. But it does not take long for a system to become legacy if it does not keep up with the right patterns.

Consider how fast things move in our industry. Teams still running Java 8 in 2026 because nobody reviewed the stack. Apps built on deprecated frameworks because there was never a checkpoint to evaluate alternatives. It takes no time for things to become stale if nobody reviews how they are done.

In many architecture review meetings, I see people not coming with homework. They did not state why they considered alternatives. They have no POC results to support their design. If you only made a choice instead of evaluating options, it is a pure assumption that you struck gold. In reality it might be silver, and you would not know without backed proof.

The best architecture reviews are the ones where someone walks in and says "here is what I considered, here is what I ruled out, and here is why this approach wins." That is the difference between engineering and guessing.

System Design Is Not a Diagram. It Is a Series of Decisions.

Every section in this post is a decision. Monolith or microservices. Sync or async. SQL or NoSQL. Build or buy. Scale now or scale later. Spend on managed services or invest in DevOps. Ship the perfect version or ship what works this quarter.

The tutorials teach you to draw boxes and arrows. Real system design is about making the right call at the right time with incomplete information and real constraints. That is a skill you build through experience, not through watching YouTube videos about designing Netflix.

I'm Malay Mehta, a Software Architect and Engineering Mentor with 726+ mentoring sessions and a 5.0 rating. I mentor working engineers on real-world architecture decisions, not interview prep. Read more on my blog or get in touch.

Why Your Vibe Coded App Breaks in Production

Malay Mehta — Wed, 24 Jun 2026 14:45:24 +0000

Originally published at malaymehta.com

It Starts with an Idea

Let me walk you through a story that's playing out thousands of times right now.

It starts with an idea. A real one. Something that cracks a genuine problem. In the past, turning that idea into software required a developer squad, a QA team, and a bit of DevOps to get an MVP out the door.

But things changed fast. Now someone with an idea and no technical background can build it themselves. We call it vibe coding. Hand the idea to an AI agent, iterate on the output, verify the end state, and ship it. If you mastered AI looping and prompt chaining, hats off. That part is genuinely impressive.

The demo works. Early users sign up. You feel like you cracked it.

The Wall

Then comes the next logical thought: let me verify code quality. So you put another AI agent on it. The agent reviews the code, says it looks good, maybe suggests some minor refactors. Feels reassuring.

Until you realize what just happened: AI wrote the code, and AI reviewed the code.

Nobody with actual engineering judgment ever looked at it. Nobody walked through the code themselves. Nobody asked whether the architecture makes sense for what comes next.

And I haven't even gotten to code security and vulnerabilities yet.

What I Actually See in These Codebases

I review AI-generated and vibe coded repos regularly. Here is what shows up every single time:

Users can see each other's data by changing an ID in the URL.

This is the most common and most dangerous pattern. The app works perfectly in testing because you only ever test with one account. The moment two real users are on the system, one can access the other's data just by modifying a request parameter. Data isolation was never designed because the AI never thought about it, and nobody asked.

Authentication and token management full of holes.

Modified tokens are accepted. Privilege escalation is possible. The auth layer looks correct on the surface but has no real validation depth. An experienced attacker would walk through it in minutes.

Secrets floating in the code like public property.

API keys, database credentials, third-party tokens hardcoded in source files. Open invitations for anyone who gets access to the repo. AI generates these inline because it doesn't understand the difference between making something work locally and making something safe for production.

Messy business logic with unnecessary abstractions everywhere.

AI loves abstracting things. Layers upon layers of patterns that add complexity without value. When something breaks, nobody knows where to start looking because the logic is spread across files that reference each other in ways that don't map to anything in the business domain.

Nobody can do root cause analysis because nobody walked the code.

This is the fundamental problem. When a production issue hits, the team stares at code they didn't write, don't understand, and can't trace. Debugging becomes guesswork. Fixes introduce new bugs. The codebase fights you at every turn.

Your App Works with 100 Users. It Won't Work with 10,000.

The app that worked fine in development starts showing cracks under real load. Here is what actually happens:

Your database isn't functioning properly under heavy load because nobody thought about connection pooling, proper indexing, or slow query monitoring. API latency P99 and P95 are spiking, giving you sleepless nights. Is it a microservices mess, a debugging hell, or everything together?

Do you have the right observability in place to even know what's happening? Are you following OWASP security principles? For most vibe coded apps, the answer is no.

The Numbers Back This Up

This is not just my observation. 81% of enterprise tech leaders report increased production issues from AI-generated code. 43% of AI code changes need manual debugging in production even after passing QA. AI-generated code introduces 1.7x more issues than human-written code across production systems.

A Ship Without a Sailor

If all of this sounds like afterthoughts, then it's the right time to think. You may have gained moderate to good technical knowledge along the way, but is it enough to choose the right things now? To prioritize what to fix first? To know which shortcuts will cost you and which are fine?

Without guidance, it's like a ship without a sailor. It goes wherever the wind takes it. In your case, the wind is a hallucinating AI agent that's only as good as the direction you give it.

Does it seem too late? It can feel that way. You already have users growing and production burning with issues. But this is exactly the point where the right guidance makes the biggest difference.

What to Look for Before It's Too Late

If you're running a vibe coded app in production right now, check these in order:

Data isolation - can one user access another's data by modifying request parameters? Test with two accounts. If yes, stop everything and fix this first.
Auth and tokens - try modifying a JWT token. Does your API still accept it? Can you escalate privileges?
Secrets in code - search your repo for API keys, database passwords, and tokens. Move them to environment variables.
Database under load - check connection pooling, add indexes to your most queried columns, set up slow query logging.
Observability - if you can't see what's happening in production without reading logs manually, you don't have observability. Set up structured logging, metrics, and alerting.
Walk the code yourself - the single most important thing. Read through the critical paths. Understand how a request flows from the user to the database and back. If you can't trace it, neither can anyone you hire to fix it.

The Right Guidance Changes Everything

This is the vibe coded production mess. And it's fixable. An experienced mentor who has seen these patterns before can help you steer production, rescue catastrophic issues, and fix things in a way that actually stays fixed.

The goal is not to throw away what you built. The goal is to save the momentum you have and build a real foundation under it.

AI will keep getting better at generating code. But right now, the gap between code that works on a demo and code that works in production is where the real engineering happens.

The typing was never the hard part. The thinking is.

I'm Malay Mehta, a Software Architect and Engineering Mentor with 726+ mentoring sessions and a 5.0 rating. I help teams fix AI-generated software and mentor engineers on building the judgment AI can't replace. Read more on my blog or get in touch.