DEV Community: Siddhartha Reddy

Handling Failure: The Most Important Part of AI Systems

Siddhartha Reddy — Fri, 29 May 2026 15:08:10 +0000

Every AI system will fail.

The question isn't whether it will happen.

The question is:

What happens next?

🚨 The Biggest Difference Between Demos and Products

In demos:

Success is showcased
Failure is hidden

In production:

Failure is inevitable
Failure is visible

The systems that succeed aren't the ones that never fail.

They're the ones that:

Fail gracefully.

🧠 The Dangerous Assumption

Many teams build AI systems as if:

Input → Model → Correct Output

But reality looks more like:

Input → Model → Sometimes Correct
                Sometimes Wrong
                Sometimes Uncertain

And that's completely normal.

⚠️ Failure is Not a Bug

This is one of the hardest lessons in AI.

Traditional software often follows deterministic rules.

Given the same input:

You expect the same output.

AI systems are different.

They operate on probabilities.

That means:

Wrong predictions happen
Edge cases happen
Unexpected behavior happens

Failure isn't exceptional.

It's built into the system.

🧩 Example: Fraud Detection

Imagine a fraud detection system.

Scenario A

The system flags a legitimate transaction as fraud.

Result:

Frustrated customer
Lost trust

Scenario B

The system misses a fraudulent transaction.

Result:

Financial loss
Security concerns

Neither outcome is ideal.

The goal isn't perfection.

The goal is:

Managing the consequences of being wrong.

🔄 Designing for Uncertainty

Strong AI systems don't pretend to know everything.

Instead they ask:

"What should happen when confidence is low?"

Possible responses:

Escalate to a human
Request more information
Delay action
Use fallback rules

👨‍💻 The Human-in-the-Loop Pattern

One of the most effective approaches is:

AI Prediction
      ↓
Confidence Check
      ↓
High Confidence → Automatic Action

Low Confidence → Human Review

This combines:

Speed
Automation
Reliability

📊 Monitor Failure, Not Just Success

Many teams track:

Accuracy
Precision
Recall

But forget to track:

Failure rates
User complaints
Escalations
Recovery time

The most valuable data often comes from:

The mistakes.

🛡️ Build Fallback Systems

Every critical AI system should have:

✅ Backup logic

Simple rules when the model fails.

✅ Human review paths

For high-risk decisions.

✅ Safe defaults

Actions that minimize harm.

✅ Alerting systems

To detect unusual behavior quickly.

🚀 What Great AI Systems Do Differently

Weak systems ask:

"How do we prevent failure?"

Strong systems ask:

"How do we recover from failure?"

Because prevention is never perfect.

Recovery can be.

🔁 Failure Creates Better Systems

Ironically:

The systems that improve fastest are often the ones that:

Capture failures
Analyze failures
Learn from failures

Failure isn't just a problem.

It's a source of learning.

🧠 Key Insight

AI systems are not defined by how often they succeed.

They're defined by how they behave when they fail.

🚀 Final Take

Most teams spend months improving models.

Very few spend time designing failure handling.

Yet failure handling often matters more.

Because users remember:

Unexpected errors
Broken experiences
Lost trust

Far more than a small increase in accuracy.

🧠 If You Take One Thing Away

Don't design AI systems for perfect predictions.

Design them for imperfect reality.

💬 Closing Thought

Anyone can build a system that works when everything goes right.

Very few can build one that:

Works when everything goes wrong.

That's where real AI engineering begins.

Designing Feedback Loops That Actually Improve AI Systems

Siddhartha Reddy — Wed, 27 May 2026 16:44:48 +0000

Training a model once is easy.

Keeping an AI system useful over time is the hard part.

🚨 The Biggest Mistake in AI Systems

Most teams build AI like this:

Train → Deploy → Done

But real-world systems don’t work that way.

Why?

Because:

Users change
Data changes
Behavior changes

👉 Your system slowly becomes outdated.

🧠 What Real AI Systems Need

They need:

Feedback loops

A feedback loop allows the system to:

Observe outcomes
Learn from failures
Improve over time

Without this:

Performance silently degrades.

⚙️ What is a Feedback Loop?

Simple version:

Prediction → Outcome → Feedback → Improvement

The system:

Makes a prediction
Sees what actually happened
Learns from the difference

👉 Then adapts.

🧩 Example: Recommendation System

Imagine a content recommendation engine.

System predicts:

“User will like this article”

Reality:

User ignores it

That interaction becomes:

Feedback data

Over time:

The system learns preferences
Recommendations improve

📉 Without Feedback Loops

Your system slowly drifts away from reality.

Because:

User behavior evolves
Patterns change
Old assumptions stop working

👉 Static AI systems decay over time.

🔄 Types of Feedback Loops

✅ Explicit feedback

Users directly respond:

Likes
Ratings
Reviews

✅ Implicit feedback

System observes behavior:

Clicks
Watch time
Purchases
Skips

👉 Most modern systems rely heavily on implicit feedback.

⚠️ Feedback Loops Can Also Break Systems

Bad feedback loops create:

Bias amplification
Echo chambers
Reinforced mistakes

Example:

Recommending only similar content
Narrowing user exposure over time

👉 Feedback loops must be designed carefully.

🧱 What Strong Feedback Systems Include

✅ Monitoring

Track:

Accuracy
User behavior
System performance

✅ Retraining pipelines

Continuously update models with new data

✅ Failure tracking

Capture:

Incorrect outputs
Edge cases
User complaints

✅ Evaluation layers

Measure:

Whether changes actually improve outcomes

🚀 The Real Shift

The future of AI is not:

Static models

It’s:

Adaptive systems

Systems that:

Learn continuously
Improve continuously
Respond to real-world behavior

🧠 Key Insight

The model is not the intelligence.

The learning loop is.

🚀 Final Take

AI systems don’t stay good because:

The model was trained well

They stay good because:

The system keeps learning

🧠 If You Take One Thing Away

AI systems are not products you finish.

They are systems you continuously evolve.

💬 Closing Thought

Anyone can deploy a model.

Very few can:

Design systems that improve themselves over time

👉 That’s where real AI engineering begins.

Choosing the Right Model (Not the Best One)

Siddhartha Reddy — Tue, 28 Apr 2026 05:56:34 +0000

The best model is rarely the right model.

And chasing it is one of the biggest mistakes in AI.

🚨 The Default Thinking

Most people ask:

“What’s the best model for this problem?”

So they:

Look for highest accuracy
Pick the most advanced architecture
Optimize benchmarks

👉 This works in research.

👉 It fails in production.

🧠 The Real Question

Instead of asking:

“What’s the best model?”

Ask:

“What’s the right model for this system?”

⚙️ Why “Best” Doesn’t Work in Real Systems

Real-world systems have constraints:

Latency (how fast it responds)
Cost (compute + infra)
Scale (number of users)
Reliability (consistency)

👉 The “best” model often violates these.

⚖️ The Tradeoff Triangle

Every model choice is a tradeoff between:

Accuracy ↔ Latency ↔ Cost

You can’t maximize all three.

High accuracy → slower, expensive
Low latency → simpler models
Low cost → compromises elsewhere

👉 Choosing a model = choosing a tradeoff

🧩 Example (Real-World Scenario)

Imagine a recommendation system:

Option A:

Complex deep learning model
High accuracy
Slow response

Option B:

Simpler model
Slightly lower accuracy
Fast + cheap

👉 In production, Option B often wins.

Why?

Because users care about speed and consistency.

⚠️ Overengineering is a Real Problem

Many systems fail because:

Model is too complex
Hard to deploy
Hard to debug
Expensive to scale

👉 Complexity increases cost without proportional value.

🧱 Fit the Model to the System

Your model should match:

✅ Use case

Real-time vs batch

✅ Data availability

Small vs large datasets

✅ Infrastructure

What you can actually run

✅ Business goals

Speed vs accuracy vs cost

🔄 Iteration Matters More Than Perfection

Instead of:

Building the “perfect” model

Focus on:

Deploying a working model
Measuring performance
Improving iteratively

👉 Speed of iteration > initial perfection

🚀 What Actually Works

Start simple:

Baseline model
Measure
Improve

Then:

Increase complexity only if needed

🧠 Key Insight

A slightly worse model in a strong system

beats

A perfect model in a weak system

🚀 Final Take

AI systems don’t succeed because:

They use the best models

They succeed because:

They use the right models for the system

🧠 If You Take One Thing Away

Don’t chase the best model.

Choose the model that fits your constraints.

💬 Closing Thought

Anyone can benchmark models.

Very few can:

Choose models that actually work in production

👉 That’s where real engineering happens.

Data First, Model Later: The Right Way to Build AI Systems

Siddhartha Reddy — Thu, 23 Apr 2026 11:12:39 +0000

Most AI systems fail not because of bad models, but because of bad data. Here’s why data should come first.

Most AI systems don’t fail because of bad models.

They fail because of bad data.

🚨 The Common Mistake

Most teams start like this:

Choose a model
Train it
Then figure out the data

👉 This is backwards.

🧠 The Reality

Models don’t create intelligence.

Data does.

The model just:

Learns patterns
From the data you give it

If your data is:

Incomplete
Noisy
Misaligned

👉 Your system will fail no matter how good the model is.

📊 Why Data Matters More Than Models

A simple rule:

Better data + simple model

beats

Bad data + complex model

🧩 What “Good Data” Actually Means

Not just:

Large datasets

But:

✅ Relevant

Matches real-world use cases

✅ Clean

Minimal errors and inconsistencies

✅ Representative

Covers actual production scenarios

✅ Updated

Reflects current patterns (not outdated)

⚠️ The Biggest Problem: Training ≠ Production Data

In training:

Clean datasets
Structured inputs

In production:

Missing values
Noise
Unexpected formats

👉 This mismatch is where systems break.

🔄 Data is Not Static

Most people think:

Collect data → Train → Done

Reality:

Collect → Clean → Use → Monitor → Update → Repeat

👉 Data is a continuous process, not a one-time step.

🧪 Example (Simple but Real)

Imagine a spam detection system:

Training data:

Clean emails
Proper grammar

Production data:

Slang
Typos
Mixed languages

👉 Your model suddenly performs worse.

Not because:

The model is bad

But because:

The data changed

⚙️ What You Should Do Instead

Before choosing a model:

1. Audit your data

What do you actually have?
Is it usable?

2. Simulate production inputs

Test real-world scenarios

3. Build data pipelines

Collection
Cleaning
Transformation

4. Plan for updates

How will new data be added?

🧱 Data Pipelines Are the Real Foundation

Your system should look like:

Data Sources → Cleaning → Transformation → Storage → Model

👉 If this pipeline is weak:

The system collapses

🚀 Final Take

AI systems don’t improve because:

You switch models

They improve because:

You improve the data

🧠 If You Take One Thing Away

Don’t ask “Which model should we use?”

Ask: “Do we have the right data?”

💬 Closing Thought

Anyone can download a model.

Very few can:

Build and maintain high-quality data systems

👉 That’s where real advantage lies.

Designing an AI System: Where Do You Even Start?

Siddhartha Reddy — Tue, 21 Apr 2026 07:30:00 +0000

"Most people start AI projects with models. That’s the wrong place to begin. Here’s how to think about designing AI systems correctly."

Most AI projects fail before they even start.

Not because of bad models but because of bad starting points.

🚨 The Default (Wrong) Approach

Most people start like this:

Problem → Model → Data → Deployment

Pick a model
Train it
Hope it works

👉 This leads to:

Misaligned systems
Poor performance
Failed products

🧠 The Real Question

Before you touch a model, ask:

What problem are we actually solving?

Not:

“Can we use AI here?”
“Which model should we use?”

But:

Who is the user?
What decision are we supporting?
What does success look like?

👉 AI is not the goal.

👉 The system solving a real problem is.

⚙️ The Right Starting Point

Real AI systems start like this:

Problem → Data → Constraints → System → Model

Let’s break this down.

🧩 1. Problem Definition

Be specific.

Bad:

“Build a recommendation system”

Good:

“Increase user retention by recommending relevant content within 200ms”

👉 Constraints and goals matter more than the model.

📊 2. Understanding Data

Ask:

What data do we have?
Is it reliable?
Does it reflect real-world usage?

👉 No data → no system

👉 Bad data → bad system

⚠️ 3. Constraints (Most Ignored Step)

Every system has constraints:

Latency (real-time vs batch)
Cost (compute limits)
Scale (number of users)
Accuracy requirements

👉 These decisions shape EVERYTHING.

🧱 4. System Design

Now you design:

Data pipeline
Processing layers
Model integration
Serving layer
Monitoring

👉 This is where most real engineering happens.

🤖 5. Model Selection (Finally)

Only now do you ask:

What model fits the constraints?
What tradeoffs are acceptable?

👉 Not:

“What’s the best model?”

But:

“What’s the right model for this system?”

🔁 Putting It Together

The correct mindset:

Start with the problem  
Design the system  
Then choose the model

⚠️ Why Most Teams Get This Wrong

Because:

Models are exciting
Systems are complex
Data is messy

So people skip to:

The easiest visible part, The Model

🚀 Final Take

AI projects don’t fail because:

Models are bad

They fail because:

The system was never designed properly

🧠 If You Take One Thing Away

Don’t start with the model.

Start with the problem and constraints.

💬 Closing Thought

Anyone can train a model.

Very few can:

Design a system that actually works

👉 That’s the difference.

The Future: Engineers as AI System Architects

Siddhartha Reddy — Mon, 20 Apr 2026 07:30:00 +0000

AI is not replacing engineers.

It’s redefining what it means to be one.

🚨 The Narrative is Wrong

The common belief:

“AI will replace programmers”

But what’s actually happening is more subtle:

AI is replacing parts of programming not the role of engineers

🧠 What’s Actually Changing

Traditionally, engineers:

Wrote code
Debugged logic
Built features

Now, increasingly, they:

Guide AI systems
Design workflows
Validate outputs

👉 The shift is not from engineer → obsolete

👉 It’s from engineer → architect

⚙️ From Coding to Orchestration

Old workflow:

Write → Debug → Deploy

New workflow:

Define → Generate → Evaluate → Refine → Deploy

Engineers are no longer just writing code.

They are:

Orchestrating systems that produce code

🤖 The Rise of AI Systems

Modern systems include:

LLMs
Tool integrations
Feedback loops
Memory layers

These systems:

Generate solutions
Test them
Improve over time

👉 The intelligence is in the system, not just the model.

🧩 The New Skill Stack

The engineers who will thrive understand:

✅ Systems thinking

How components interact and fail

✅ Data flows

Where inputs come from and how they change

✅ Evaluation

How to measure correctness beyond accuracy

✅ Tooling

How to integrate models into real workflows

✅ Feedback loops

How systems improve over time

❌ What Becomes Less Important

Not irrelevant — but less dominant:

Writing boilerplate code
Memorizing syntax
Manual implementation of standard patterns

👉 These are increasingly handled by AI.

🧑‍💻 What Becomes More Important

Designing robust systems
Handling uncertainty
Managing failure cases
Making trade-offs (speed vs cost vs accuracy)

👉 These cannot be automated easily.

⚠️ The Risk: Shallow Engineers

There’s a danger emerging:

Engineers who can generate code…

but don’t understand systems

This leads to:

Fragile products
Poor debugging ability
Hidden technical debt

👉 AI amplifies both skill and ignorance.

🚀 The Opportunity

The engineers who win won’t be:

The best coders

They’ll be:

The best system designers

🔁 The Future of Software

We’re moving toward systems that:

Generate code
Test themselves
Adapt to new data
Improve continuously

Software becomes:

A living system, not a static artifact

🧠 If You Take One Thing Away

Learn how systems work.

Not just how to write code.

💬 Final Thought

Everyone is learning how to use AI.

Very few are learning how to:

Design AI systems

👉 That gap defines the next generation of engineers.

From Model to Product: Where AI Projects Actually Break

Siddhartha Reddy — Sun, 19 Apr 2026 07:30:00 +0000

Most AI projects don’t fail in training.

They fail when you try to turn them into products.

🚨 The Illusion: “The Model Works”

You trained a model:

Good accuracy
Clean evaluation metrics
Solid results

So you think:

“We’re ready to ship.”

But this is where most teams hit a wall.

🧠 The Real Problem

A working model ≠ a working product

AI products require:

Reliability
Consistency
Usability
Trust

👉 None of which are guaranteed by a model.

❌ 1. The “Demo Trap”

In demos:

Controlled inputs
Best-case scenarios
Clean outputs

In production:

Messy inputs
Edge cases
Unpredictable behavior

👉 What worked in a demo often breaks immediately in real usage.

❌ 2. UX is an Afterthought

Most AI systems are built like this:

Model first
UX later

But users care about:

Response time
Clarity
Consistency

Not:

Your model architecture

👉 A powerful model with poor UX feels broken.

❌ 3. No Handling of Failure Cases

AI systems WILL fail.

But most products don’t plan for:

Incorrect outputs
Uncertain predictions
Edge cases

Good products:

Detect failure
Handle it gracefully
Communicate clearly

👉 This is product thinking, not model thinking.

❌ 4. Latency Kills Experience

Your model might be accurate…

But if it takes:

2–3 seconds to respond

Users feel:

“This is slow”

👉 Perception matters more than accuracy.

❌ 5. Lack of Trust

Users don’t trust AI by default.

They need:

Predictability
Transparency
Consistency

If your system:

Sometimes works
Sometimes doesn’t

👉 Users stop relying on it.

❌ 6. Integration is Harder Than Expected

AI rarely exists alone.

It must integrate with:

Databases
APIs
Existing systems
Business workflows

👉 Most failures happen here, not in the model.

❌ 7. Misaligned Expectations

Stakeholders expect:

“Human-level intelligence”

Reality:

Probabilistic outputs
Imperfect predictions

👉 This gap kills projects.

🧩 The Missing Layer

Most teams focus on:

Model performance

But ignore:

Product design

🧑‍💻 What Actually Works

Successful AI products focus on:

✅ UX first

Design around user experience

✅ Failure handling

Expect and manage errors

✅ Speed optimization

Balance latency vs accuracy

✅ Trust building

Consistent behavior

✅ System integration

Fit into real workflows

🚀 Final Take

A model answers:

“Can this work?”

A product answers:

“Will people actually use it?”

🧠 If You Take One Thing Away

A great model doesn’t make a great product.

Great systems + UX do.

💬 Closing Thought

Everyone is building smarter models.

Very few are building:

Better AI products

👉 That’s where the real impact is.

Why 90% of ML Engineers Struggle in Real-World Systems

Siddhartha Reddy — Sat, 18 Apr 2026 07:30:00 +0000

Most ML engineers don’t fail because they lack knowledge.

They fail because they’re solving the wrong problem.

🚨 The Hard Truth

Most ML engineers are trained to:

Optimize models
Improve accuracy
Tune hyperparameters

But real-world systems don’t fail because of bad models.

They fail because of:

Bad system design

🧠 The Root Problem

ML education focuses on:

Dataset → Model → Accuracy

But real-world systems look like:

Data → Pipeline → System → Monitoring → Feedback → Iteration

👉 The model is just one part of a much bigger system

❌ 1. Too Much Focus on Accuracy

Engineers obsess over:

92% → 94% accuracy

But ignore:

Data quality
Pipeline reliability
System latency

👉 A slightly worse model in a solid system

will outperform a perfect model in a broken one.

❌ 2. No Understanding of Data in Production

In training:

Clean datasets
Well-structured inputs

In production:

Missing values
Noisy inputs
Changing distributions

👉 Many engineers don’t design for this reality.

❌ 3. Weak System Design Skills

ML engineers often struggle with:

APIs
Scalability
Distributed systems
Fault tolerance

👉 Because these aren’t taught in most ML paths.

❌ 4. Ignoring the Pipeline

They think:

“The model is the product”

But in reality:

The pipeline is the product

Problems appear in:

Preprocessing mismatch
Feature inconsistency
Data leakage

❌ 5. No Monitoring Mindset

After deployment:

Train → Deploy → Done

This is a mistake.

Real systems require:

Monitor → Evaluate → Improve → Repeat

👉 Without this, systems degrade silently.

❌ 6. Poor Debugging Skills

When models fail:

It’s not obvious why
It’s not reproducible
It’s not localized

Debugging AI systems requires:

Data tracing
Experiment tracking
System-level thinking

👉 This is very different from traditional debugging.

❌ 7. No Product Thinking

ML engineers often optimize for:

Metrics

But products require:

User experience
Latency
Reliability
Business impact

👉 A high-accuracy model that users don’t trust is useless.

🧩 The Real Skill Gap

It’s not:

“ML knowledge”

It’s:

Systems thinking

🧑‍💻 What Actually Makes a Strong ML Engineer

The best engineers understand:

✅ Data systems

How data flows and breaks

✅ Pipelines

End-to-end consistency

✅ Infrastructure

Serving, scaling, latency

✅ Monitoring

Real-world performance

✅ Feedback loops

Continuous improvement

🚀 Final Take

If you focus only on models:

You’ll stay stuck in notebooks

If you learn systems:

You’ll build real products

🧠 If You Take One Thing Away

ML is not just about models.

It’s about building reliable systems.

💬 Closing Thought

Most people are trying to become better at machine learning.

Very few are trying to become:

Better at building AI systems

👉 That’s the difference.

The Hidden Cost of AI Systems Nobody Talks About.

Siddhartha Reddy — Fri, 17 Apr 2026 07:30:00 +0000

AI isn’t expensive.

Bad AI systems are.

💸 The Illusion: “AI is Cheap Now”

With APIs and open-source models, it feels like:

Spin up a model
Plug in an API
Ship a product

👉 Done.

But that’s the demo illusion.

🚨 The Reality: Cost Starts After Deployment

The real cost of AI systems doesn’t show up when you build them.

It shows up when you:

Run them continuously in production

⚙️ 1. Infrastructure Costs (The Silent Drain)

Running AI at scale requires:

GPUs / high-performance CPUs
Memory-heavy systems
Distributed infrastructure

Even simple systems:

Handle thousands of requests
Run models repeatedly

👉 Costs scale with usage, not development

⏱️ 2. Latency vs Cost Tradeoff

You want:

Fast responses
High accuracy

But:

Faster models = more compute
Cheaper models = worse performance

👉 You’re constantly balancing:

Speed ↔ Cost ↔ Accuracy

You can’t optimize all three.

🔁 3. Continuous Retraining

Your model doesn’t stay good.

You need:

New data pipelines
Regular retraining
Validation cycles

This means:

More compute
More engineering time
More complexity

👉 AI systems are never “done”

🧑‍💻 4. Engineering Overhead

The hidden cost isn’t just infra.

It’s people.

You need:

ML engineers
Data engineers
Backend engineers
DevOps / MLOps

👉 The model is 10% of the effort

👉 The system is 90%

🐛 5. Debugging is Expensive

When AI systems fail:

It’s not obvious why
It’s not reproducible
It’s not localized

Debugging requires:

Logs
Data tracing
Experiment tracking

👉 This takes serious time.

📊 6. Monitoring & Observability

To keep systems reliable, you need:

Drift detection
Performance tracking
Alerting systems

Without this:

Your system degrades silently.

With this:

You pay in infrastructure + engineering.

🔒 7. Risk & Reliability Costs

AI introduces new risks:

Incorrect predictions
Bias issues
Hallucinations
Edge-case failures

To handle this, you need:

Safeguards
Human-in-the-loop systems
Validation layers

👉 More complexity = more cost

🧩 The Real Insight

Most teams think:

“We need a better model”

But the real problem is:

We need a better system

🚀 Final Take

AI doesn’t become expensive because of:

Model size
Training cost

It becomes expensive because:

You have to run, maintain, and evolve the system

🧠 If You Take One Thing Away

AI is not a one-time cost.

It’s a continuous system expense.

💬 Closing Thought

Anyone can build an AI demo.

Very few can afford to:

Run it reliably in production

👉 That’s the real challenge.

Inside an AI Pipeline: What Actually Happens After You Train a Model

Siddhartha Reddy — Thu, 16 Apr 2026 20:45:28 +0000

Training a model is the easiest part of AI.

Building the system around it is where things get real.

🧠 The Biggest Misunderstanding in AI

Most people think AI looks like this:

Data → Model → Predictions

That’s a toy version.

Real-world AI systems look like this:

Data → Validation → Preprocessing → Feature Engineering → Model → Post-processing → Serving → Monitoring → Feedback → Retraining

👉 The model is just one step in a long pipeline

⚙️ Step 1: Data Ingestion

Your system starts with:

Databases
APIs
Logs
User input

Problems:

Missing data
Inconsistent formats
Delayed updates

👉 If your data is bad, everything downstream is broken.

🧹 Step 2: Data Validation & Cleaning

Before anything else:

Null checks
Schema validation
Outlier detection

Example:

Age = -5
Salary = 999999999

👉 Garbage in → garbage out

🧪 Step 3: Preprocessing

Transform raw data:

Normalization
Encoding
Tokenization

⚠️ Critical issue:

Training preprocessing ≠ Production preprocessing

🧩 Step 4: Feature Engineering

This is where:

Domain knowledge meets ML

Examples:

Aggregations
Time-based features
Derived metrics

🤖 Step 5: Model Training

Train
Tune
Evaluate

A great model inside a bad system still fails.

🔄 Step 6: Post-processing

Thresholding
Ranking
Business rules

🚀 Step 7: Model Serving

APIs
Batch jobs
Streaming

Challenges:

Latency
Scaling

📊 Step 8: Monitoring

Track:

Accuracy
Input drift
Latency

Without monitoring, you’re flying blind.

📉 Step 9: Feedback Loop

Collect:

User feedback
Errors
Edge cases

Feed into retraining.

🔁 Step 10: Continuous Retraining

New Data → Retrain → Deploy → Repeat

🧩 Full Pipeline

Data Sources
     ↓
Validation
     ↓
Preprocessing
     ↓
Feature Engineering
     ↓
Model
     ↓
Post-processing
     ↓
Serving
     ↓
Monitoring
     ↓
Feedback
     ↓
Retraining

⚠️ Where Systems Fail

Data quality
Pipeline mismatch
No monitoring
No feedback

🚀 Final Take

If you focus only on models:

You build demos

If you focus on pipelines:

You build products

🧠 Key Insight

The model is just a component.

The pipeline is the product.

🔗 Series

AI Doesn’t Write Code, Systems Do
Why Most AI Systems Fail in Production

Next:
👉 The Hidden Cost of AI Systems Nobody Talks About

"Why Most AI Systems Fail in Production (And No One Talks About It)"

Siddhartha Reddy — Thu, 16 Apr 2026 13:31:12 +0000

"AI demos look perfect production systems don’t. Here’s why most AI systems fail in the real world."

AI demos look magical.

Production systems look broken.

And the gap between them is where most teams fail.

🚨 The Truth Nobody Likes to Admit

Most AI systems don’t fail in training.

They fail in production.

Not because:

The model is bad
The accuracy is low

But because:

Real-world systems are messy, unpredictable, and constantly changing

🧠 The “Demo vs Reality” Problem

In demos:

Clean datasets
Controlled inputs
No edge cases

In production:

Noisy data
Missing values
Unexpected inputs
Changing distributions

👉 Your model isn’t solving the same problem anymore.

📉 1. Data Drift (Silent Killer)

Your model was trained on past data.

Production gives you:

New data, new patterns, new behavior

Types of drift:

Feature drift (input changes)
Concept drift (relationship changes)

Example:

Fraud model trained on 2023 data
Used in 2025 → patterns completely different

👉 Accuracy drops silently.

⚙️ 2. The Pipeline is the Real System

Most people focus on the model.

But the real system is:

Data → Preprocessing → Model → Post-processing → API → Monitoring

Failure can happen anywhere:

Wrong preprocessing
Feature mismatch
Data leakage
Version mismatch

👉 The model is just one piece.

🐛 3. Edge Cases Destroy Everything

AI works well on:

“Common cases”

But production is full of:

Rare inputs
Unexpected formats
Adversarial cases

Example:

NLP model trained on clean text
Production input = slang + emojis + typos

👉 System breaks instantly.

⏱️ 4. Latency & Cost Constraints

Your model works great…

Until:

It takes 2 seconds per request
Or costs too much to run

Production requires:

Low latency
High throughput
Cost efficiency

👉 A perfect model that’s slow is useless.

🔁 5. No Feedback Loop = Slow Death

Most systems are deployed like this:

Train → Deploy → Forget

That’s a mistake.

Real systems need:

Monitor → Evaluate → Retrain → Improve

Without feedback:

Performance degrades
Errors accumulate
Users lose trust

🧩 6. Observability is Missing

Most teams don’t track:

Model performance in real-time
Input distributions
Failure cases

So when things break:

You don’t even know why.

🤖 The Real Problem

The biggest mistake teams make:

Treating AI as a model problem

Instead of a systems problem

🧑‍💻 What Actually Works

Successful AI systems focus on:

✅ Data pipelines

Clean, versioned, monitored

✅ Continuous evaluation

Not just offline metrics

✅ Feedback loops

Real-world learning

✅ System design

Not just model tuning

🚀 Final Take

AI doesn’t fail because models are bad.

It fails because:

Systems are incomplete

🧠 If You Take One Thing Away

Building the model is easy.

Building the system is the real challenge.

💬 Closing Thought

Everyone is building AI models.

Very few are building:

Reliable AI systems

👉 That’s where the real opportunity is.

AI Doesn’t Write Code, Systems Do (And Most People Are Missing This)

Siddhartha Reddy — Tue, 14 Apr 2026 19:03:23 +0000

"AI isn’t writing your code, systems are. Here’s what most developers are missing about AI coding."

Everyone says AI is replacing programmers.

That’s not what’s happening.

Something much more interesting is.

⚠️ The Biggest Misconception in AI Right Now

People think:

“AI writes code.”

What’s actually happening:

Systems built around AI are writing software.

And if you don’t understand that difference, you’re going to fall behind fast.

🧠 The Illusion of Intelligence

When you use AI to generate code, it feels like you're working with something intelligent.

But under the hood, it’s just:

Predicting the next token
Based on patterns
Without understanding correctness

It doesn’t “know” your code works.

It only knows:

“This looks like code that usually follows this prompt.”

So why does it work so well?

⚙️ Why AI Code Actually Works (Surprisingly Well)

AI works in coding not because it's smart

but because software engineering is structured for it to succeed.

1. Code is predictable

Repeated patterns
Standard libraries
Known structures

2. Feedback loops are instant

Compile → fail → fix
Test → fail → fix

3. “Good enough” wins

Companies don’t need perfect code. They need:

Faster shipping
Lower costs
Acceptable reliability

👉 AI fits perfectly into this system.

💥 Where AI Completely Breaks

AI is great at local problems, but fails at system-level thinking.

❌ Long-term architecture

Doesn’t plan systems
Doesn’t maintain consistency

❌ State & memory

No real awareness of past decisions
No persistent understanding

❌ Debugging complex systems

AI can fix:

Syntax errors
Small bugs

But fails at:

Distributed failures
Race conditions
Deep system issues

👉 Because these require causal reasoning, not pattern matching.

🧩 The Real Architecture of “AI Coding”

AI coding tools are NOT just models.

They are systems.

🧠 What’s actually happening:

User Prompt
     ↓
LLM (generates code)
     ↓
Tooling Layer
 (compiler / tests / linters)
     ↓
Feedback Loop
 (errors, logs, outputs)
     ↓
Iteration Engine
 (fix → retry → improve)
     ↓
Final Output

👉 The intelligence is NOT in the model

👉 The intelligence is in the loop

🤖 Agentic Systems: The Real Shift

We’re moving from:

Prompt → Output

To:

Goal → Plan → Execute → Evaluate → Iterate

This is agentic coding.

These systems:

Write code
Run it
Analyze failures
Fix it
Repeat

Until it works.

👉 The model is just a component

👉 The system does the thinking

🧑‍💻 What Happens to Engineers?

You’re not being replaced.

Your role is being redefined.

Old role:

Write code
Debug manually
Build features

New role:

Design systems
Orchestrate AI workflows
Validate outputs
Own complexity

👉 The best engineers won’t be:

“The fastest coders”

👉 They’ll be:

The best system designers

⚠️ The Hidden Problem Nobody Talks About

AI introduces a dangerous shift:

You didn’t write the code…

But you’re still responsible for it.

This leads to:

Shallow understanding
Fragile systems
Hidden technical debt

If you rely blindly on AI, you lose:

Code intuition

And that’s where things break.

🔁 The Future: Software as a Feedback Loop

We’re moving toward:

Generate → Test → Fix → Deploy → Monitor → Repeat

Continuously.

Software won’t be written once.

It will be:

Continuously generated and refined by systems

🚀 Final Take

The narrative is wrong.

It’s not:

“AI writes code”

It’s:

AI generates possibilities

Systems validate them

Engineers make them meaningful

🧠 If You Take One Thing Away

Don’t focus on prompts.

Focus on systems.

That’s where the real leverage is.

💬 Closing Thought

Most people are learning how to use AI.

Very few are learning how AI systems actually work.

👉 That gap is your opportunity.