DEV Community: Kunwar Jhamat

AI Technical Debt: The Hidden Cost of AI Coding Tools

Kunwar Jhamat — Tue, 10 Mar 2026 08:28:19 +0000

AI technical debt is the hidden cost that accumulates when AI coding tools generate code faster than teams can understand, review, and maintain it. Traditional technical debt builds up over months and years as developers take shortcuts under deadline pressure. AI technical debt is fundamentally different. It accumulates at the speed of code generation — which means a team can create months worth of traditional technical debt in a single week.

I have been using AI coding tools daily for over three years. They are genuinely transformative for productivity. But after watching multiple projects — my own and others — I have noticed a pattern: the faster AI generates code, the faster the codebase becomes something nobody fully understands. This is not a reason to stop using AI tools. It is a reason to understand the specific type of debt they create so you can manage it deliberately instead of discovering it during a production incident at 3 AM.

What Is AI Technical Debt?

Traditional technical debt happens when developers knowingly take shortcuts — using a quick fix instead of a proper solution, skipping tests, or choosing a simpler architecture that will not scale. The key word is "knowingly." The developer understands the trade-off. They know they are borrowing against future effort. And they can explain the debt to the next person who works on the code.

AI technical debt is different in three critical ways:

It is unintentional. The developer did not choose to take a shortcut. The AI generated code that happened to contain architectural compromises the developer did not notice.
It is invisible. AI-generated code looks professional — clean formatting, proper comments, modern patterns. The debt is hidden behind a polished surface.
It is undocumented. With traditional debt, the developer who created it usually knows where it is and why. With AI technical debt, nobody knows — not the developer, not the AI, and certainly not the next person who inherits the codebase.

Dimension	Traditional Technical Debt	AI Technical Debt
Speed of creation	Slow — limited by human typing speed	Fast — limited only by API response time
Awareness	Developer usually knows the debt exists	Developer often does not know
Visibility	Often visible in messy code	Hidden behind clean formatting
Documentation	Developer can explain why	Nobody can explain why
Location	Usually concentrated in specific areas	Spread across the entire codebase
Resolution	Developer who created it can fix it	Requires full code audit to find it

The Speed vs Quality Trade-Off: Real Numbers

Let us look at what happens when you increase code generation speed by 10 to 25 times without proportionally increasing review capacity.

A developer manually writing code produces roughly 200 to 400 lines per day. These lines are written with understanding — the developer knows what each line does and why it exists. The defect rate is typically around 5 to 15 defects per 1,000 lines of code, depending on the complexity.

The same developer using AI tools can generate 2,000 to 10,000 lines per day. Even if we assume the AI produces code with a similar defect rate — say 10 defects per 1,000 lines — the absolute number of defects introduced per day increases dramatically:

Scenario	Lines Per Day	Defect Rate	Defects Per Day	Defects Per Month
Manual coding	300	10 per 1,000	3	60
AI-assisted (careful)	2,000	10 per 1,000	20	400
AI-assisted (fast)	5,000	15 per 1,000	75	1,500
Vibe coding	10,000	20 per 1,000	200	4,000

The numbers tell a clear story. Even with a conservative defect rate, AI-assisted development at high speed can introduce 1,500 defects per month. Many of these are not bugs that crash the application — they are architectural compromises, performance issues, security gaps, and maintainability problems that compound over time. They are the kind of issues that make a codebase progressively harder to work with.

This is not an argument against AI tools. It is an argument for matching your review capacity to your generation speed. If you generate code 10 times faster but review at the same speed, you are creating a review deficit that turns directly into AI technical debt.

The AI Spaghetti Code Problem

There is a specific pattern of AI technical debt that I see repeatedly, and it deserves its own name: AI spaghetti code. Unlike traditional spaghetti code — which is messy and obviously poorly structured — AI spaghetti code looks clean on the surface but has tangled dependencies and inconsistent patterns underneath.

Here is how it happens. You ask the AI to build Feature A. It generates clean code using Pattern X. A week later, you ask for Feature B. The AI generates clean code using Pattern Y. Both features work perfectly. But Pattern X and Pattern Y are fundamentally different approaches to similar problems. Your codebase now has two competing architectures that will eventually conflict.

This is because AI does not remember your previous sessions. Each prompt gets a fresh response optimized for that specific request. The AI does not think about consistency across your entire codebase — it thinks about the best answer for the current prompt. Over time, this produces a codebase that is locally correct but globally incoherent.

Week 1 — Feature A: User Authentication
  AI uses: Express middleware + JWT + cookie-based sessions
  Pattern: Middleware chain with req.user populated early
  Works perfectly ✓

Week 2 — Feature B: API Rate Limiting
  AI uses: Express middleware + Redis + custom headers
  Pattern: Different middleware chain, checks req.headers directly
  Works perfectly ✓

Week 3 — Feature C: Admin Dashboard
  AI uses: Different auth check (reads JWT directly instead of req.user)
  Pattern: Bypasses the middleware chain from Feature A
  Works perfectly ✓

Week 6 — Bug Report:
  "Admin users are rate-limited but should not be"

  Root cause: Three features built by AI, three different auth patterns.
  Rate limiter does not know about the auth middleware.
  Admin check does not use the same auth path.

  Time to debug: 8 hours
  Time to fix properly: Refactor all three to use consistent patterns
  Time it would have taken with consistent architecture: 30 minutes

The expensive part is not the bug itself. It is the refactoring required to create a consistent architecture after three different AI sessions produced three different approaches to the same underlying problem. This is AI technical debt in its purest form — the cost of local optimization without global coherence.

5 Barriers AI Creates in Software Development

AI technical debt does not exist in isolation. It is a symptom of deeper barriers that AI introduces into the software development process. Understanding these barriers helps you anticipate where AI technical debt will accumulate in your projects.

Barrier 1: Hallucinated Code. AI models sometimes generate code that references libraries, APIs, or functions that do not exist. This happens because the model was trained on patterns and can extrapolate patterns that look plausible but are not real. A hallucinated API call might compile if the function name happens to match something in your dependencies, but it will behave in unexpected ways. Catching hallucinated code requires the developer to verify that every import, every function call, and every library reference actually exists and does what the AI claims it does.

Barrier 2: Lack of System Understanding. AI generates code for the prompt it receives, not for the system the code will live in. It does not know about your deployment constraints, your team's skill level, your scaling requirements, or your operational procedures. Code that is technically correct in isolation can be operationally wrong in your specific context. A perfectly written database query that works in development might cause lock contention in your specific production configuration.

Barrier 3: Context Limitations. Even the most advanced AI models have context window limitations that prevent them from understanding your entire codebase at once. This means the AI is always working with incomplete information. It cannot see how its code will interact with the 50 other files in your project. It cannot verify that its approach is consistent with the patterns established in modules it has not seen. Every piece of AI-generated code is potentially inconsistent with parts of the codebase the AI was not shown.

Barrier 4: Security Vulnerabilities. AI models generate code based on patterns in their training data. If common patterns include security vulnerabilities — and they do, because a significant amount of open-source code contains security issues — the AI will reproduce those vulnerabilities. SQL injection patterns, insecure authentication flows, improper input validation, and exposed secrets in configuration files all appear in AI-generated code. The AI does not think about security; it thinks about pattern completion.

Barrier 5: Massive Technical Debt Velocity. This is the meta-barrier. All four previous barriers create AI technical debt, and they do so at the speed of AI code generation. Traditional development creates debt at human speed. AI creates debt at machine speed. A team of five developers using AI aggressively can create more architectural inconsistency in one month than the same team would create manually in a year.

Barrier	What Goes Wrong	How to Detect It	How to Prevent It
Hallucinated Code	References to non-existent libraries or APIs	Verify every import and function call	Use AI with documentation context, verify outputs
No System Understanding	Code correct in isolation, wrong in context	Integration testing, architecture review	Provide system context in prompts (CLAUDE.md)
Context Limitations	Inconsistent patterns across codebase	Cross-module code review	Establish patterns before AI generates code
Security Vulnerabilities	Reproduced common vulnerability patterns	Security scanning, manual audit	Security-focused review of all AI output
Debt Velocity	Debt accumulates faster than it can be resolved	Track code quality metrics weekly	Match review speed to generation speed

How AI Technical Debt Accumulates Under the Hood

Let me trace exactly how AI technical debt builds up in a real project. This is a simplified but realistic example based on patterns I have seen.

Imagine a team building an e-commerce API. They use AI coding tools for most of the implementation. Here is what happens week by week:

Week 1: Product catalog API
  Generated: 3,000 lines
  AI chose: Repository pattern with direct SQL queries
  Debt created: None visible yet
  Total debt: LOW

Week 2: Order management API
  Generated: 4,000 lines
  AI chose: Active Record pattern with ORM
  Debt created: Two different data access patterns in one codebase
  Total debt: MEDIUM (inconsistent architecture)

Week 3: Payment integration
  Generated: 2,500 lines
  AI chose: Service layer with third-party SDK
  Debt created: Error handling inconsistent with weeks 1-2
  Total debt: MEDIUM-HIGH

Week 4: User authentication
  Generated: 2,000 lines
  AI chose: JWT with middleware (different from order API's session approach)
  Debt created: Two auth patterns, order API has session leaks
  Total debt: HIGH

Week 5: Admin dashboard
  Generated: 5,000 lines
  AI chose: Mix of all previous patterns (pulled from different contexts)
  Debt created: Admin bypasses auth middleware, direct SQL in some routes
  Total debt: CRITICAL

Week 6: First production bug
  Customer charged twice. Root cause: Order API's Active Record and
  Payment API's service layer handle transactions differently.
  No consistent transaction boundary across the two modules.

  Time to understand the bug: 2 days
  Time to fix properly: 1 week (need to unify transaction handling)
  Time it would have taken with consistent architecture: 2 hours

Notice the pattern. Each week's code works perfectly in isolation. The AI generated correct, functional code every time. The AI technical debt is not in any individual module — it is in the spaces between modules. It is in the inconsistent patterns, the conflicting approaches, and the missing architectural coherence that a human architect would have maintained.

Hallucinated Libraries and APIs: A Special Category of AI Technical Debt

One of the most dangerous forms of AI technical debt comes from AI hallucination in code generation. The AI generates code that references packages, functions, or API endpoints that do not actually exist. Sometimes these hallucinations are obvious — the package name is clearly made up. But sometimes they are subtle and dangerous.

For example, the AI might generate an import for a package that existed in an older version of a framework but was removed. Or it might reference an API method that exists in a different library with a similar name. Or it might create a function call that looks correct based on naming conventions but has different parameters than the actual implementation.

The particularly insidious case is when the hallucinated package name actually exists in a package registry but is a different package entirely. Security researchers have demonstrated that attackers can register package names that AI commonly hallucinates, turning AI code generation into a supply chain attack vector. This transforms hallucinated code from a bug into a security vulnerability.

The defense is straightforward but requires discipline: verify every import, every package reference, and every API call in AI-generated code. Do not assume that because the code compiles and the tests pass, all the dependencies are legitimate. This is one area where AI technical debt can have immediate security consequences, not just long-term maintenance costs.

How to Measure AI Technical Debt in Your Project

You cannot manage what you cannot measure. Here are concrete signals that indicate AI technical debt is accumulating in your project:

Signal	What It Means	Severity
Multiple patterns for the same concern	AI generated different solutions in different sessions	High — architecture is fragmenting
Debugging takes longer than expected	Developer is learning the codebase during debugging	High — understanding gap is growing
Bug fixes introduce new bugs	The infinite refactor loop has started	Critical — stop and audit
Nobody can explain why a pattern was chosen	Architectural decisions were delegated to AI	Medium — document decisions now
Duplicated logic across modules	AI generated similar code independently	Medium — refactor into shared utilities
Tests pass but coverage is shallow	AI wrote tests for happy paths only	High — real defects are hiding
Code review becomes rubber-stamping	Team trusts AI output too much	Critical — review standards must increase

Track these signals weekly. If three or more appear simultaneously, your project has significant AI technical debt that needs attention before it compounds further.

How to Manage AI Technical Debt

Managing AI technical debt requires a different approach than managing traditional technical debt. With traditional debt, you know where it is because the developer who created it can point to it. With AI technical debt, you need to actively discover it through systematic review.

Strategy 1: Architecture-First Development. Define your architectural patterns before AI generates any code. Write down your data access pattern, your error handling strategy, your authentication approach, and your naming conventions. Give this to the AI as context. This prevents the most common source of AI technical debt: inconsistent patterns across modules.

Strategy 2: Weekly Architecture Reviews. Set aside time every week to look at the codebase as a whole, not just individual features. Ask: are we still using consistent patterns? Has the AI introduced approaches that conflict with our established architecture? Are there duplications that should be consolidated? This is the single most effective practice for catching AI technical debt early.

Strategy 3: Incremental Understanding. For every piece of AI-generated code, the developer who accepted it should be able to explain it. Not in general terms — in specific terms. What does this function do? Why was this pattern chosen? What happens when this input is null? If the developer cannot answer these questions, the code is a liability, not an asset.

Strategy 4: Debt Budgeting. Accept that some AI technical debt is inevitable and budget time to address it. A good ratio is to spend 20 percent of development time on debt reduction — reviewing AI-generated code, consolidating patterns, improving test coverage, and documenting architectural decisions. This prevents debt from compounding to the point where it blocks feature development.

Strategy 5: Context Engineering. The better context you give AI tools, the less debt they create. Use project configuration files like CLAUDE.md and .cursorrules to communicate your architectural patterns, coding standards, and constraints. As we discussed in Why AI Needs Better Memory, context quality directly determines output quality. Good context engineering is the most effective preventive measure against AI technical debt.

Anti-Patterns That Accelerate AI Technical Debt

Anti-Pattern	What Happens	Why It Is Dangerous	What to Do Instead
Generate and Forget	Accept AI code, never review architecture	Inconsistencies compound silently	Review architecture weekly, not just code
Speed Over Understanding	Prioritize feature velocity over code comprehension	Team loses ability to debug their own system	Ensure every developer can explain every module they own
AI-to-AI Debugging	Use AI to fix bugs in AI-generated code	Surface-level patches instead of root cause fixes	Understand the bug yourself before asking for a fix
No Architectural Blueprint	Let AI decide patterns for each feature independently	Codebase becomes a collection of disconnected approaches	Define patterns upfront, constrain AI to follow them
Test Trust	AI writes tests that pass, team assumes code is correct	Tests cover happy paths, miss edge cases and integration issues	Define test scenarios based on requirements, not AI output
Metric Illusion	Track lines of code or features shipped as productivity	Velocity metrics hide debt accumulation	Track maintainability, debug time, and architecture consistency

Key Takeaways

AI technical debt is different from traditional debt: It is unintentional, invisible behind clean formatting, and undocumented. Nobody knows where it is or why it was created, making it harder to find and fix.
Code generation speed without review speed creates a deficit: If you generate 10 times faster but review at the same speed, the gap becomes AI technical debt. Match your review capacity to your generation speed.
AI spaghetti code looks clean but has tangled architecture: Each AI session produces locally correct code, but across sessions, the patterns conflict and create global incoherence. Weekly architecture reviews catch this early.
The five barriers compound each other: Hallucinated code, lack of system understanding, context limitations, security vulnerabilities, and debt velocity all feed into each other. Address them as a system, not individually.
Architecture-first development is your best prevention: Define patterns before AI generates code. Give AI your constraints through context files. This single practice eliminates the most common source of AI technical debt.
Budget 20 percent of time for debt reduction: Accept that AI technical debt is inevitable and allocate time to address it systematically. Review, consolidate, document, and test every week.
Context engineering directly reduces AI technical debt: The better context your AI tools have about your system, the more consistent and appropriate their output will be. Invest in context infrastructure.

Originally published at DECYON — Clarity for complex systems in the age of AI.

Iterator Patterns: How to Process Millions of Records Without Running Out of Memory

Kunwar Jhamat — Thu, 05 Mar 2026 10:42:33 +0000

You have 100,000 rows in your database. You need to process each one. The obvious approach loads everything into an array, loops through it, and writes the results. This works fine with 1,000 records. With 100,000, your server hits the memory limit and crashes.

This is not just a tutorial on PHP iterators. This is about understanding what actually happens in memory and why iterator patterns keep memory usage constant regardless of dataset size.

What Actually Happens in Memory

Let us start with the simplest case: three database records, each 1KB in size.

The Array Approach

// Assuming $database is a PDO instance or similar wrapper
$rows = $database->query("SELECT * FROM users")->fetchAll();
// At this point: ALL rows are loaded into memory

Step-by-step memory allocation:

Step 1: fetchAll() is called → PHP allocates memory for an empty array (~200 bytes).
Step 2: First row arrives (1KB) → Memory: ~1.2KB.
Step 3: Second row arrives (1KB) → Memory: ~2.2KB.
Step 4: Third row arrives (1KB) → Memory: ~3.2KB.

With 100,000 rows at 1KB each, that is 100MB. If your PHP memory_limit (e.g., in php.ini) is 64MB, the process crashes before you even process the first row.

Key insight: fetchAll() waits until every single row is loaded from the result set before returning control to your code.

Approach	Memory (1K Rows)	Memory (100K Rows)	Memory (1M Rows)
Array (fetchAll)	~1 MB	~100 MB	~1 GB (Crash)
Iterator (cursor)	~1 KB	~1 KB	~1 KB
Batched Iterator	~1 MB	~1 MB	~1 MB

The Iterator Approach

Iterators use a "cursor" concept. They load one record, hand it to you, and then move on, minimizing the "resident" memory required.

// $cursor is typically an unbuffered PDOStatement or an iterator
$cursor = $database->query("SELECT * FROM users");
foreach ($cursor as $row) {
    // Do your work here — transform, validate, insert, etc.
    $name = trim($row['name']);
    $db->insert('clean_users', ['name' => $name, 'email' => $row['email']]);
    // Memory used is just for this ONE row
}

The Step-by-Step:

Step 1: PHP asks the cursor/stream for the first row → 1KB allocated.
Step 2: Your code processes the row (transform, insert, etc.) → Memory remains ~1.2KB.
Step 3: Loop moves to the next iteration → PHP overwrites/releases row 1 and fetches row 2.
Step 4: Memory remains ~1.2KB.

Whether you have 3 rows or 3 million, your memory footprint stays flat.

What `yield` Actually Does Internally

In PHP, generators use the yield keyword. Understanding its internal mechanics explains the magic. A generator function looks like a standard function but behaves like an iterator.

// A simple generator function
function getRows($pdo) {
    // We assume the PDO driver is configured for unbuffered queries
    $stmt = $pdo->query("SELECT * FROM users");
    while ($row = $stmt->fetch()) {
        yield $row; // Execution pauses HERE
    }
}

Under the hood, yield performs three main actions:

Returns the value: The current $row is sent back to the loop/caller (e.g., foreach ($rows as $row)).
Pauses execution: PHP "freezes" the function, remembering exactly which line it was on and the state of all local variables ($stmt, $row).
Waits for next(): Execution stays paused until the loop asks for another item (the next iteration). No CPU is consumed while waiting, and only the small state context is kept in memory.

When the loop moves to the next iteration, PHP resumes the function immediately after the yield, continues the loop, fetches the next $row, and yields again.

Combining Iterators with Batch Processing

Processing one record at a time is memory-safe but can be slow due to the overhead of many sequential operations, like database single-inserts. The "pro" move is batch processing with bounded memory.

/**
 * Processes data in memory-efficient batches.
 *
 * @param iterable $iterator A Generator or Iterator of records.
 * @param int $batchSize How many records to process at once.
 */
function processBatched(iterable $iterator, int $batchSize = 1000) {
    $batch = [];
    foreach ($iterator as $record) {
        $batch[] = $record;

        // When the batch is full, process and clear it
        if (count($batch) >= $batchSize) {
            bulkInsert($batch); // One DB call for 1000 records
            $batch = [];        // Crucial: Release the memory
        }
    }

    // Process any remaining records (the last partial batch)
    if (!empty($batch)) {
        bulkInsert($batch);
    }
}

Maximum memory usage is always (batch size x average record size). With 1,000 records at 1KB each, you never use more than ~1MB, whether processing 10,000 or 10,000,000 records.

Choosing the Right Batch Size

Batch Size	Memory	DB Efficiency	Error Recovery	Best For
100	~100KB	Moderate	Easy (Small scope)	Strict memory limits
1,000	~1MB	Good	Acceptable	General-purpose ETL
5,000	~5MB	Excellent	Coarse	Fast networks/Large records

Pro Tip: High batch sizes (e.g., >10,000) often show diminishing returns in database performance and make error recovery harder. If a batch of 10,000 fails, finding the single "bad" record that caused the constraint violation is like finding a needle in a haystack.

Chaining Transformations (Lazy Evaluation)

What happens when you need to normalize, filter, and enrich the data?

// (Pseudocode concept)
$stream = getRows($pdo); // Generator (0 memory)

$processedStream = $stream
    ->map(fn($row) => normalize($row))
    ->filter(fn($row) => $row['active'])
    ->map(fn($row) => enrich($row));

Each operation does not run yet; it returns a new iterator that wraps the previous one. This is lazy evaluation. The chain is a blueprint, not execution. When you finally iterate on $processedStream, each record flows through all transformations one at a time. Chaining ten transformations together still only holds ONE active record in memory at any given point in the pipeline.

Common Anti-Patterns

Anti-Pattern	Why It Fails	Solution
`iterator_to_array()`	Converts the efficient stream back into a giant, memory-crashing array.	Stay inside the `foreach` loop.
Logging every record	If you store detailed logs in a local array, that array grows unbounded.	Log summaries per batch or stream directly to a file/service.
Accumulating errors	Storing every "failed" row in a `$failedRows` array for later processing.	Write errors to a dedicated file or 'error' database table immediately.

When Iterator Patterns "Break"

Some operations require state — meaning they need to see the whole dataset to work. You cannot stream these.

Sorting: You cannot know the mathematically first item until you have seen the very last one. (Solution: Push this to the database with ORDER BY).
Deduplication: To know if record #1,000,000 is a duplicate, you have to remember the unique keys of all previous 999,999 records. (Solution: Use database constraints or a bloom filter/LRU cache for very large sets).
Aggregations: Sums, averages, and counts usually need the full set. (Solution: Use database functions like SUM() or COUNT()).

The general rule: if an operation needs to see more than one record at a time to complete, it belongs in the database layer.

Key Takeaways

Never load all data at once: Stop using fetchAll() for large datasets; switch to unbuffered queries/cursors or generators.
Understand yield internally: It pauses, returns one value, and releases the previous value.
Batch your writes: Stream data one record at a time to keep memory safe, but execute expensive operations (like DB inserts) in chunks of 500-2,000.
Watch for anti-patterns: iterator_to_array() and error accumulation silently undo all your memory savings.
Push stateful operations: Let the database handle sorting, deduplication, and aggregation (SQL).

This is not just a PHP thing. Whether it is Python Generators, Java Streams, or Go Channels, the principle is identical: process and release, process and release.

This is part of the ETL Pipeline Series on DECYON — real engineering patterns from 20+ years of building production systems.

Read the full version with interactive diagrams and benchmarks at decyon.com

ETL Pipeline: The 6-Phase Pattern That Cuts Debugging From Hours to Minutes

Kunwar Jhamat — Wed, 04 Mar 2026 11:11:44 +0000

You have a customer record from a legacy database. The name field contains "JOHN SMITH " with extra spaces. The phone field has "(555) 123-4567" in a format your system does not accept. The email field is "NULL" as a literal string. The birth date is "0000-00-00".

You need to extract this record, fix all these issues, and load it into your target system. The question is: where in your pipeline does each fix happen? And when something breaks, how do you know which fix failed?

This is where the traditional 3-phase ETL model fails. "Extract, Transform, Load" bundles too much into "Transform." The 6-phase pattern unbundles it into distinct responsibilities, so when something breaks at 3 AM, you know exactly where to look.

Why 3 Phases Are Not Enough

The classic ETL model looks simple:

Extract → Transform → Load

But "Transform" is doing too much work. It handles field renaming, type conversion, data cleaning, business logic, and enrichment. When the pipeline fails with "Invalid date format," you are left asking: Was it a mapping issue? A type conversion? A business rule? A data quality problem?

3-Phase ETL	Problem
Extract	Clear responsibility — no issue here
Transform	Field renaming + type conversion + cleaning + business logic + enrichment — all bundled together
Load	Clear responsibility — no issue here

The problem is not that "Transform" does too many things. The problem is that when it fails, you cannot tell which thing failed.

The 6-Phase ETL Pipeline Pattern

Let us trace what actually happens to that messy customer record as it flows through all six phases.

Extract → Map → Transform → Clean → Refine → Load

Starting record from source:

{
  "cust_id": 12345,
  "cust_nm": "JOHN   SMITH  ",
  "cust_phone": "(555) 123-4567",
  "cust_email": "NULL",
  "birth_dt": "0000-00-00"
}

Phase 1: Extract — Get Raw Data

The extractor pulls the record exactly as it exists in the source. No modifications. No cleaning. Just faithful extraction.

{
  "cust_id": 12345,
  "cust_nm": "JOHN   SMITH  ",      // Extra spaces? Still there.
  "cust_phone": "(555) 123-4567",   // Parentheses? Still there.
  "cust_email": "NULL",             // Literal string? Still there.
  "birth_dt": "0000-00-00",         // Invalid date? Still there.
  "_meta": {
    "extracted_at": "2024-01-15T10:30:00Z",
    "source": "legacy_crm"
  }
}

Phase 2: Map — Rename and Restructure

Field names change to match the target schema. No data values change, only the structure.

Mapping rules:
  cust_id    → customer_id
  cust_nm    → full_name
  cust_phone → phone
  cust_email → email
  birth_dt   → birth_date

If this phase fails, you know immediately: the source schema changed.

Phase 3: Transform — Convert Types

Data types change. Strings become integers. Dates get parsed into proper date objects. No business logic yet.

{
  "customer_id": 12345,
  "full_name": "JOHN   SMITH  ",
  "phone": "(555) 123-4567",
  "email": "NULL",
  "birth_date": null    // "0000-00-00" → null (unparseable date)
}

If this phase fails, the source sent data in an unexpected format.

Phase 4: Clean — Fix Data Quality

Data quality issues get fixed. Extra whitespace trimmed. Invalid phone formats normalized. Placeholder values like "NULL" become actual nulls.

{
  "customer_id": 12345,
  "full_name": "JOHN SMITH",    // Extra spaces removed
  "phone": "5551234567",        // Normalized to digits only
  "email": null,                // "NULL" string → actual null
  "birth_date": null
}

Phase 5: Refine — Apply Business Logic

Business rules and enrichment. Calculated fields. Lookups from reference tables.

{
  "customer_id": 12345,
  "full_name": "JOHN SMITH",
  "phone": "5551234567",
  "phone_formatted": "(555) 123-4567",
  "email": null,
  "email_status": "missing",       // Business rule: flag missing emails
  "birth_date": null,
  "age_verified": false,           // Business rule: needs birth date
  "customer_tier": "standard"      // Lookup from tier rules
}

Phase 6: Load — Write to Destination

The final record gets inserted or updated in the target system with proper transaction handling.

Why This Pattern Matters

When a pipeline fails in production, the error message tells you which phase failed. This is the difference between a 5-minute fix and a 3-hour investigation.

Phase	Error Example	Root Cause	Fix Time
Extract	"Connection refused"	Source system down	Minutes
Map	"Unknown field 'customer_name'"	Schema changed at source	Minutes
Transform	"Cannot parse '2024/13/45' as date"	Unexpected format	Minutes
Clean	"Phone validation: value is all dashes"	Data quality problem	Minutes
Refine	"No tier for 'PREMIUM_PLUS'"	New business tier type	Minutes
Load	"FK constraint violation"	Dependency not loaded	Minutes

With a 3-phase pipeline, all of these errors would say "Transform failed." You would spend hours reading through code trying to figure out what went wrong.

The Assembly Line Mental Model

Think of a car manufacturing plant. Raw materials pass through stations, each with a single responsibility.

Station	Phase	What Happens	Failure Means
Receiving	Extract	Raw materials arrive	Supplier did not deliver
Sorting	Map	Parts sorted into bins	Part labels changed
Machining	Transform	Parts cut to spec	Wrong dimensions
QC	Clean	Defects caught	Material quality declined
Assembly	Refine	Parts become components	Design spec has a gap
Delivery	Load	Car rolls off the line	Customer garage is full

When a car has a problem, you know exactly which station to investigate.

Common Anti-Patterns to Avoid

Anti-Pattern	Why It Fails	Do This Instead
Cleaning in Extract	You lose raw source data for comparison	Extract faithfully, clean in Phase 4
Business logic in Clean	Different change frequencies and owners	Clean in Phase 4, business rules in Phase 5
Mapping in Transform	Schema errors look like format errors	Map in Phase 2, convert types in Phase 3
Skipping phases	When source changes, you edit transform code instead of config	Keep all 6 phases
Phase coupling	Phases cannot be tested independently	Each phase depends only on record structure

Implementation

Each phase is a function that takes a record in and returns a record out:

for record in extract(source):
    mapped      = map_fields(record, config.mappings)
    transformed = convert_types(mapped, config.types)
    cleaned     = clean_fields(transformed, config.cleaners)
    refined     = apply_rules(cleaned, config.rules)
    load(refined, destination)

Each function:

Takes one record as input
Returns one record as output (or null to skip)
Has no side effects on other records
Emits events for observability
Can be tested with a single record

Key Takeaways

Extract faithfully: Never modify data during extraction
Map separately: Field renaming is configuration, not code
Transform types explicitly: Type conversion has its own failure mode
Clean with reporting: Track what changed so you detect upstream problems
Refine for business: Business rules change independently from data quality rules
Load with transactions: Use upserts, batch inserts, proper error handling
Keep phases independent: Testable, skippable, debuggable
Emit events at every phase: The event trail turns 3-hour investigations into 5-minute fixes

Six phases might seem like overhead until you are debugging a production failure. Then it seems like the bare minimum.

This is part of the ETL Pipeline Series on DECYON — real engineering patterns from 20+ years of building production systems.

Read the full version with interactive diagrams at decyon.com

Understanding ETL Pipelines: The Philosophy Behind Reliable Data Integration

Kunwar Jhamat — Wed, 04 Mar 2026 10:56:12 +0000

Every ETL pipeline addresses the same fundamental challenge: data exists in one system, needs to exist in another system, and something must change along the way. That sounds simple. It is not. Behind that sentence sits decades of engineering complexity, failed 3 AM runs, and hard-won lessons about what actually works in production.

This is not a tutorial. This is how I think about ETL pipeline design after building data integration systems for years. The philosophy first. Then the mechanics. Then the patterns that emerge when you combine both.

What is an ETL Pipeline? Understanding the Core Problem

An ETL pipeline is a data integration process that extracts data from source systems, transforms it according to specific rules, and loads it into a destination system. The term stands for Extract, Transform, Load. But that clean three-word definition hides the real complexity underneath.

The core tension in data engineering stems from a reality that never changes: different systems optimize for different objectives. They are built for different jobs, and they cannot serve each other directly.

System Type	Optimized For	Example
Source Databases	Write performance, transaction processing	PostgreSQL, MySQL
Analytics Warehouses	Read performance, query speed	BigQuery, Snowflake
APIs	Low latency, real-time responses	REST, GraphQL endpoints
Data Lakes	Storage capacity, schema flexibility	S3, Azure Data Lake

Your source database is built to handle thousands of writes per second. Your analytics warehouse is built to scan millions of rows in a single query. These are fundamentally different machines with fundamentally different priorities. An ETL pipeline is the bridge between them. It translates between worlds that speak different languages and care about different things.

Why the Traditional Extract-Transform-Load Model Falls Short

Extract-Transform-Load sounds clean. Three phases. Three responsibilities. In practice, "Transform" becomes a dumping ground for everything that happens between extraction and loading. Type conversions live there. Business rules live there. Data cleaning lives there. Enrichment lives there.

When your ETL pipeline breaks at 3 AM, you are digging through a monolith trying to figure out which of these completely different operations failed.

The insight that changed how I design pipelines: transformation is not one thing. It is several distinct operations that happen to occur between extraction and loading.

Phase	What It Does	How It Can Fail
Map	Changes structure without changing meaning	Schema mismatches, missing fields
Type Convert	Changes representation (string → integer)	Invalid formats, precision loss
Clean	Improves quality (invalid values → null)	Excessive cleaning means upstream problems
Enrich	Adds information (lookups, calculations)	Missing reference data, calculation errors

Each of these can fail independently. Each has different failure modes. Each requires different debugging approaches. When they are bundled together under "Transform," every problem is harder to diagnose.

How Streaming Architecture Solves the Memory Problem

Most ETL pipeline code follows a pattern that works until it does not: load all data into memory, process it, write it out. The question is not whether your dataset will exceed available memory. The question is when.

# Anti-Pattern: Load Everything
data = extract_all_records()      # 10M records loaded → Memory: 4GB
transformed = transform_all(data) # New copy created   → Memory: 8GB
load_all(transformed)             # Memory crashes before this

The solution is streaming: never hold the entire dataset. Process one record at a time. Chain operations together. Memory usage stays constant regardless of dataset size.

# Streaming: Constant Memory
for record in extract_stream():     # One record at a time
    mapped = map_record(record)      # Transform in place
    converted = convert_types(mapped)
    cleaned = clean_record(converted)
    enriched = enrich_record(cleaned)
    load_record(enriched)            # Write immediately
    # Previous record released from memory

This is not optimization. This is fundamental architecture. The difference between an ETL pipeline that handles 10,000 records and one that handles 10 million is not speed. It is whether the pipeline completes at all.

Think of it like a factory assembly line. Workers do not pile up all the parts on the floor, process them, then move everything at once. One part enters, gets processed at each station, and exits. The factory floor stays clear regardless of how many parts flow through.

What Makes an ETL Pipeline Observable?

An ETL pipeline that runs silently is a pipeline you cannot trust. But the answer is not more logging. The answer is observability, and the two are not the same thing.

Approach	What It Tells You	When It Helps
Logging	What happened after the fact	Post-mortem debugging
Observability	What is happening right now, and what normally happens	Detecting anomalies before they become failures

Here is the scenario that made this click for me. Your pipeline processes 100,000 customer records every night. One morning it processes 50,000 records and reports "SUCCESS." Logging says: task completed successfully. Observability says: volume anomaly detected, investigate source system.

The pattern I use in every production pipeline: emit events at every significant point. Pipeline started. Phase completed. Record failed validation. Batch processed. Let monitoring systems decide what matters.

How to Design ETL Pipelines That Recover Gracefully

When your ETL pipeline fails halfway through processing 10 million records, what happens? The answer determines whether you lose hours or minutes.

Recovery Strategy	How It Works	Pros	Cons
Start Over	Delete partial results, run from beginning	Simple, safe	Wasteful, time-consuming
Resume	Continue from the failure point	Efficient	Complex, requires state tracking
Idempotent	Rerun safely, get identical results	Simple AND efficient	Requires careful design upfront

Idempotency means running an operation multiple times produces the same result as running it once. If your pipeline is idempotent, recovery is trivial: just run it again.

-- Non-idempotent: Running twice creates duplicates
INSERT INTO customers (id, name) VALUES (1, 'John');

-- Idempotent: Running twice produces same result
INSERT INTO customers (id, name) VALUES (1, 'John')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name;

Why Configuration-Driven Design Changes Everything

Code tells you how. Configuration tells you what. When these two are mixed together, every change requires a developer. When they are separated, domain experts can modify pipeline behavior without touching implementation code.

# config/customer_pipeline.yaml
source:
  type: postgres
  table: customers

mappings:
  - source: customer_id
    destination: id
    type: integer

  - source: full_name
    destination: name
    cleaner: trim_whitespace

  - source: email_address
    destination: email
    validators:
      - email_format
      - not_null

This connects directly to what I call the 80/20 insight:

Category	Percentage	What It Includes
Framework (stable)	80%	Streaming engine, transaction management, error recovery, monitoring
Business Logic (changes)	20%	Your field mappings, validation rules, enrichment logic

Focus your time on the 20% that makes your pipeline unique. Let a framework handle the infrastructure.

The Data Quality Reality Nobody Talks About

Source data is never clean. This is not a bug. It is a universal constant. Systems store "NULL" as a literal string. Dates arrive as "0000-00-00". Phone numbers are just dashes.

Approach	What Happens	Why It Matters
Implicit Cleaning	Silently fixes bad data, no record of what changed	Problems are hidden
Explicit Cleaning	Reports every change, tracks what was cleaned and why	Problems are visible

Build cleaners that report what they changed. Track how much data fails each rule. If phone number cleaning suddenly doubles from 5% to 10%, that is not a cleaning problem. That is a signal that something changed upstream.

Key Takeaways: The Philosophy of Boring Pipelines

ETL pipeline work is not glamorous. It is plumbing. But plumbing done wrong floods the building. Plumbing done right is invisible. The best ETL pipelines are boring.

Separate concerns: Break "Transform" into distinct phases so you can debug each independently
Stream, do not batch: Process one record at a time for constant memory usage
Design for observability: Emit comprehensive metrics so anomalies surface before they become outages
Build idempotent operations: Design pipelines that produce identical results whether run once or ten times
Configure, do not code: Separate business logic from infrastructure
Track data quality explicitly: Report every cleaning operation so you can detect upstream changes
Respect dependencies: Declare table relationships and let topological sorting determine load order
Focus on the 20%: Use frameworks for infrastructure, invest your time in domain-specific logic

Boring means reliable. Reliable means the pipeline runs at 3 AM, processes millions of records, fails gracefully when sources change, recovers without intervention, and produces the same results every time.

This is part of the ETL Pipeline Series on DECYON, where I write about architecture decisions, data engineering patterns, and lessons from 20+ years of building production systems.

Read the full version with diagrams and code walkthroughs at decyon.com

DEV Community: Kunwar Jhamat

AI Technical Debt: The Hidden Cost of AI Coding Tools

What Is AI Technical Debt?

The Speed vs Quality Trade-Off: Real Numbers

The AI Spaghetti Code Problem

5 Barriers AI Creates in Software Development

How AI Technical Debt Accumulates Under the Hood

Hallucinated Libraries and APIs: A Special Category of AI Technical Debt

How to Measure AI Technical Debt in Your Project

How to Manage AI Technical Debt

Anti-Patterns That Accelerate AI Technical Debt

Key Takeaways

Iterator Patterns: How to Process Millions of Records Without Running Out of Memory

What Actually Happens in Memory

The Array Approach

The Iterator Approach

What yield Actually Does Internally

Combining Iterators with Batch Processing

Choosing the Right Batch Size

Chaining Transformations (Lazy Evaluation)

Common Anti-Patterns

When Iterator Patterns "Break"

Key Takeaways

ETL Pipeline: The 6-Phase Pattern That Cuts Debugging From Hours to Minutes

Why 3 Phases Are Not Enough

The 6-Phase ETL Pipeline Pattern

Phase 1: Extract — Get Raw Data

Phase 2: Map — Rename and Restructure

Phase 3: Transform — Convert Types

Phase 4: Clean — Fix Data Quality

Phase 5: Refine — Apply Business Logic

Phase 6: Load — Write to Destination

Why This Pattern Matters

The Assembly Line Mental Model

Common Anti-Patterns to Avoid

Implementation

Key Takeaways

Understanding ETL Pipelines: The Philosophy Behind Reliable Data Integration

What is an ETL Pipeline? Understanding the Core Problem

Why the Traditional Extract-Transform-Load Model Falls Short

How Streaming Architecture Solves the Memory Problem

What Makes an ETL Pipeline Observable?

How to Design ETL Pipelines That Recover Gracefully

Why Configuration-Driven Design Changes Everything

The Data Quality Reality Nobody Talks About

Key Takeaways: The Philosophy of Boring Pipelines

What `yield` Actually Does Internally