DEV Community: RajeshRenato

How I Solved the Flooded Icons Crisis in Our React Codebase

RajeshRenato — Thu, 18 Sep 2025 18:29:37 +0000

The Problem: Icon Chaos in a Growing SaaS Platform

Picture this: You're working on a fast-growing SaaS platform with over 400+ React components. Your design system has evolved organically over 5+ years, and suddenly you realize you have 600+ icon files scattered across your codebase like digital confetti.

Sound familiar? Here's what our icon structure looked like:

src/common/icons/
├── IconAdd.tsx
├── IconAddCircle.tsx
├── IconAddNoOutline.tsx
├── IconPlus.tsx        # Wait, isn't this the same as IconAdd?
├── IconPlusCircle.tsx  # And this too?
├── QuickAction/
│   ├── IconEmail.tsx   # Different from main IconMail.tsx
│   └── IconLinkedIn.tsx # Different from IconLinkedin.tsx
├── NewIcons/
│   ├── IconCall.tsx    # Yet another call icon variant
│   └── IconEmail.tsx   # Another email icon!
├── Leads/
│   └── IconLinkedIn.tsx # The third LinkedIn icon!
└── ... 300+ more files

The Pain Points

1. Developer Confusion

// Which one should I use? 🤔
import { IconCall } from '../../common/icons/IconCall';
import { IconCall } from '../../common/icons/NewIcons/IconCall';
import { IconCallAlt } from '../../common/icons/IconCallAlt';
import { IconCallHollow } from '../../common/icons/IconCallHollow';

2. Duplicate Icons Everywhere

We had:

6 different "Add" icons (IconAdd, IconPlus, IconAddCircle, etc.)
4 different "LinkedIn" icons across different folders
8 different "Call" icon variations
Multiple email icons with slight variations

3. Inconsistent Props & Types

// Icon A
interface IconProps {
  size?: string;
  color?: string;
}

// Icon B
interface IconCallProps {
  size?: string;
  color?: string;
  type?: IconCallType; // Extra prop!
}

// Icon C
interface IconWarningProps {
  size: string; // Required, not optional!
  color?: string;
  variant?: ICON_WARNING_TYPE;
}

4. No Visual Discovery

Developers couldn't easily see what icons were available. They'd either:

Create new icons instead of reusing existing ones
Spend ages searching through folders
Ask designers "do we have an icon for X?"

The Solution: Icon Browser + Automated Discovery

I built a comprehensive Icon Browser that dynamically discovers, loads, and displays all icons in our codebase. Here's how:

1. Automated Icon Discovery

First, I created a script that scans the entire codebase and generates a JSON manifest:

// iconList.json (generated automatically)
{
  "iconPaths": [
    {
      "path": "common/icons/QuickAction/IconClock.tsx",
      "importPath": "common/icons/QuickAction/IconClock",
      "pathWithoutExtension": "common/icons/QuickAction/IconClock",
      "name": "IconClock",
      "category": "Actions",
      "extension": ".tsx",
      "directory": "common/icons/QuickAction",
      "size": 751,
      "modifiedAt": "2025-02-16T07:38:13.215Z"
    }
    // ... 600+ more icons
  ]
}

2. Dynamic Icon Loading System

The tricky part was dynamically importing icons at runtime. Webpack needs static analysis, so I created a smart import system:

const importModule = async (iconPath: IconPath) => {
  const { importPath, directory, extension } = iconPath;

  // Extract filename once
  const fileName = importPath.split('/').pop();

  // For SVG files, use different base path
  if (extension === '.svg') {
    return await import(`../../assets/bigSvgs/${fileName}`);
  }

  // For icon files, directly use the directory as target path
  const targetPath = directory; // directory already contains "common/icons/..."

  return await import(`../../${targetPath}/${fileName}`);
};

3. Intelligent Component Extraction

Different icons export their components differently. I built a system to handle all patterns:

const extractComponent = (module: any, iconName: string) => {
  // Handle default exports
  if (module.default && isValidReactComponent(module.default)) {
    return module.default;
  }

  // Handle named exports (exact match)
  if (module[iconName] && isValidReactComponent(module[iconName])) {
    return module[iconName];
  }

  // Handle named exports (without Icon prefix)
  const nameWithoutIcon = iconName.replace(/^Icon/, '');
  if (
    module[nameWithoutIcon] &&
    isValidReactComponent(module[nameWithoutIcon])
  ) {
    return module[nameWithoutIcon];
  }

  // Try all exported functions
  for (const key in module) {
    if (isValidReactComponent(module[key])) {
      return module[key];
    }
  }

  return null;
};

4. Safe Rendering with Error Boundaries

Icons can fail in many ways, so I built robust error handling:

const RenderIcon = React.memo(({ component: IconComponent, ...props }) => {
  try {
    if (!IconComponent) {
      return <div className="error-icon">🚫</div>;
    }

    // Sanitize props to prevent crashes
    const safeProps = {
      ...props,
      size: String(props.size || '32').replace(/[^0-9]/g, '') || '32',
      color: props.color || '#080C2B',
      iconType: props.iconType || ICON_TYPE.SOLID,
    };

    const element = React.createElement(IconComponent, safeProps);

    return React.isValidElement(element) ? (
      element
    ) : (
      <div className="error-icon">⚠️</div>
    );
  } catch (error) {
    console.error('Error rendering icon:', error);
    return <div className="error-icon">❌</div>;
  }
});

5. The Final Icon Browser UI

const IconBrowser = () => {
  const { icons, loading, error } = useIconLoader();
  const [searchTerm, setSearchTerm] = useState('');
  const [selectedCategory, setSelectedCategory] = useState('all');

  const filteredIcons = icons.filter((icon) => {
    const matchesSearch = icon.name
      .toLowerCase()
      .includes(searchTerm.toLowerCase());
    const matchesCategory =
      selectedCategory === 'all' || icon.category === selectedCategory;
    return matchesSearch && matchesCategory;
  });

  return (
    <div className="icon-browser">
      <div className="controls">
        <input
          placeholder="Search icons..."
          value={searchTerm}
          onChange={(e) => setSearchTerm(e.target.value)}
        />
        <select
          value={selectedCategory}
          onChange={(e) => setSelectedCategory(e.target.value)}
        >
          <option value="all">All Categories</option>
          <option value="Actions">Actions</option>
          <option value="Navigation">Navigation</option>
          {/* ... more categories */}
        </select>
      </div>

      <div className="icon-grid">
        {filteredIcons.map((icon) => (
          <IconErrorBoundary key={icon.uniqueId} iconName={icon.name}>
            <div className="icon-item" onClick={() => copyToClipboard(icon)}>
              <RenderIcon
                component={icon.component}
                size="32"
                color="#080C2B"
              />
              <span className="icon-name">{icon.name}</span>
              <span className="icon-path">{icon.path}</span>
            </div>
          </IconErrorBoundary>
        ))}
      </div>
    </div>
  );
};

The Results: From Chaos to Clarity

✅ Immediate Benefits

100% Icon Visibility: Developers can now see all 383+ icons in one place
Smart Search: Find icons by name, category, or functionality
Copy-Paste Ready: Click any icon to copy the import statement
Duplicate Detection: Easily spot similar icons and consolidate
Zero Maintenance: Automatically updates when new icons are added

🎯 Developer Experience Impact

Before:

// 😫 The old way
// 1. Search through folders for 10 minutes
// 2. Guess which IconCall to use
// 3. Import and hope it works
// 4. Realize it's the wrong one
// 5. Repeat process

import { IconCall } from '../../common/icons/IconCall'; // Maybe?

After:

// 😍 The new way
// 1. Open Icon Browser
// 2. Search "call"
// 3. See all call-related icons visually
// 4. Click to copy import
// 5. Done in 30 seconds!

import { IconCall } from '../../common/icons/NewIcons/IconCall';

📊 Measurable Improvements

Icon Discovery Time: 10+ minutes → 30 seconds
Duplicate Icon Creation: Reduced by ~80%
Developer Onboarding: New devs can find icons immediately
Designer Collaboration: Designers can quickly see existing icons before creating new ones
Design System Consistency: Much easier to spot and fix inconsistencies

Key Technical Challenges & Solutions

1. Webpack Dynamic Import Limitations

Problem: Webpack can't analyze completely dynamic imports

Solution: Leverage existing directory structure data to build import paths dynamically

2. Inconsistent Icon Interfaces

Problem: 600+ icons with different prop interfaces

Solution: Built a prop sanitization layer that handles all variations

3. Runtime Failures

Problem: Some icons crash when rendered

Solution: Error boundaries around every icon with fallback UI

4. Performance with 600+ Icons

Problem: Loading hundreds of icons at once

Solution: Batch loading + lazy rendering + virtual scrolling

Lessons Learned

1. Tooling Prevents Technical Debt

Without visibility into our icon mess, the problem just kept growing. The Icon Browser made the chaos visible and actionable.

2. Developer Experience = Product Quality

When developers can easily find and use the right icons, the product becomes more consistent. UX wins when DX wins.

3. Automation > Documentation

Instead of maintaining docs about "which icons to use," I built a tool that shows them all. Self-documenting systems are better than documented systems.

4. Small Improvements, Big Impact

This wasn't a major architectural change, but it dramatically improved daily developer productivity.

What's Next?

Now that we have the Icon Browser, our next steps are:

Icon Consolidation: Merge duplicate icons using the browser to identify them
Design System Integration: Work with designers to standardize our icon library
Automated Linting: Add ESLint rules to prevent duplicate icon creation
Icon Analytics: Track which icons are actually used vs. abandoned

Want to Build This for Your Team?

The core pattern is:

Discover: Use custom scripts to scan your codebase for icon files
Load: Dynamically import them at runtime
Display: Build a searchable, visual interface
Handle Errors: Gracefully handle all the edge cases

The investment in developer tooling pays compound dividends. When developers can easily discover and reuse existing assets, your codebase becomes more consistent and your team becomes more productive.

Have you solved similar design system challenges in your codebase? I'd love to hear about your approach in the comments!

Solving the Nested Try-Catch Hell in JavaScript with Trywrap

RajeshRenato — Thu, 01 May 2025 18:17:41 +0000

Are you tired of writing nested try-catch blocks in your async JavaScript code? Do you find yourself repeating error handling logic across your codebase? Let me introduce you to Trywrap - a lightweight utility module that makes error handling in asynchronous functions clean, flexible, and maintainable.

The Problem with Traditional Error Handling

Let's look at a typical scenario where we need to handle errors in multiple async operations:

async function complexOperation() {
    try {
        // Operation 1
        try {
            await fetchUserData();
        } catch (error) {
            console.error("Failed to fetch user data:", error);
            // Handle error...
        }

        // Operation 2
        try {
            await processData();
        } catch (error) {
            console.error("Failed to process data:", error);
            // Handle error...
        }

        // Operation 3
        try {
            await saveResults();
        } catch (error) {
            console.error("Failed to save results:", error);
            // Handle error...
        }
    } catch (error) {
        console.error("Unexpected error:", error);
        // Handle any uncaught errors...
    }
}

This code is:

😫 Verbose and repetitive
😕 Hard to maintain
🤔 Difficult to standardize error handling
😬 Prone to inconsistencies

Enter Trywrap

Trywrap provides a elegant solution to this problem. Here's how you can transform the above code:

const trywrap = require('trywrap');

async function complexOperation() {
    const onError = ({ error, methodName }) => {
        console.error(`Error in ${methodName}:`, error.message);
    };

    await trywrap(fetchUserData, [], { 
        onError, 
        fallback: defaultUserData 
    });

    await trywrap(processData, [], { 
        onError, 
        fallback: [] 
    });

    await trywrap(saveResults, [], { 
        onError,
        fallback: false 
    });
}

🚀 Key Features

Centralized Error Handling: Define error handling logic once and reuse it across your application.
Fallback Values: Specify default values to return when operations fail.
Flexible Configuration: Customize error handling behavior with options like rethrow.
Detailed Error Context: Access method names and arguments in error handlers for better debugging.

📦 Installation

npm install trywrap

🛠️ How to Use Trywrap

Basic Usage

const trywrap = require('trywrap');

async function riskyOperation(value) {
    if (value < 0) throw new Error("Invalid value");
    return value * 2;
}

// Define an error handler
const onError = ({ error, methodName, args }) => {
    console.error(`Error in ${methodName}:`, error.message);
};

// Use trywrap
const result = await trywrap(riskyOperation, [10], {
    onError,
    fallback: 0
});

Advanced Usage: API Calls

async function fetchUserProfile(userId) {
    const response = await fetch(`/api/users/${userId}`);
    if (!response.ok) throw new Error('User not found');
    return response.json();
}

const userProfile = await trywrap(fetchUserProfile, ['123'], {
    onError: ({ error }) => {
        // Log to monitoring service
        logError(error);
    },
    fallback: { 
        id: '123',
        name: 'Guest User',
        isGuest: true
    }
});

🎯 When to Use Trywrap

Trywrap is perfect for:

API Integration: Handle network requests and response parsing
Database Operations: Manage database query errors gracefully
File System Operations: Handle file read/write errors
Any async operations where you want consistent error handling

💡 Pro Tips

Create Reusable Error Handlers:

const defaultErrorHandler = ({ error, methodName }) => {
    logger.error(`${methodName} failed:`, error);
    metrics.incrementError(methodName);
};

Use Type-Specific Fallbacks:

const arrayFallback = [];
const objectFallback = null;
const booleanFallback = false;

// Use appropriate fallbacks
await trywrap(fetchUsers, [], { fallback: arrayFallback });
await trywrap(fetchUser, [id], { fallback: objectFallback });
await trywrap(validateUser, [data], { fallback: booleanFallback });

Chain Operations:

const processUser = async (userId) => {
    const user = await trywrap(fetchUser, [userId], {
        onError,
        fallback: null
    });

    if (!user) return;

    const processed = await trywrap(processUserData, [user], {
        onError,
        fallback: user
    });

    return processed;
};

🎉 Benefits

✅ Cleaner Code: No more nested try-catch blocks
✅ DRY Principle: Define error handling once, use everywhere
✅ Better Maintainability: Standardized error handling across your codebase
✅ Flexible: Customize behavior with options and fallbacks
✅ Lightweight: Minimal overhead, maximum impact

🔍 Real-World Example

Here's a practical example showing how Trywrap can improve your code organization:

// Before Trywrap
async function getUserData(userId) {
    try {
        const user = await fetchUser(userId);
        try {
            const posts = await fetchUserPosts(userId);
            try {
                const analytics = await fetchUserAnalytics(userId);
                return { user, posts, analytics };
            } catch (error) {
                console.error('Analytics error:', error);
                return { user, posts, analytics: null };
            }
        } catch (error) {
            console.error('Posts error:', error);
            return { user, posts: [], analytics: null };
        }
    } catch (error) {
        console.error('User error:', error);
        return null;
    }
}

// After Trywrap
async function getUserData(userId) {
    const onError = ({ error, methodName }) => {
        console.error(`${methodName} failed:`, error);
    };

    const user = await trywrap(fetchUser, [userId], {
        onError,
        fallback: null
    });

    if (!user) return null;

    const posts = await trywrap(fetchUserPosts, [userId], {
        onError,
        fallback: []
    });

    const analytics = await trywrap(fetchUserAnalytics, [userId], {
        onError,
        fallback: null
    });

    return { user, posts, analytics };
}

🌟 Conclusion

Trywrap brings elegance and simplicity to error handling in async JavaScript. It helps you write cleaner, more maintainable code while ensuring robust error handling across your application.

Give it a try in your next project:

npm install trywrap

🔗 Useful Links

What's your take on error handling in async JavaScript? Have you faced similar challenges? Share your thoughts and experiences in the comments below! 👇

Docker Compose Explained Like You're Five

RajeshRenato — Sun, 27 Apr 2025 10:59:31 +0000

Imagine you're making your favorite sandwich. You need:

Bread and fillings
Tools (knife, plate)
A clean workspace

Now imagine if every time you wanted to make a sandwich, you had to:

Set up your workspace and tools
Prepare all ingredients
Make and serve the sandwich

Sounds simple, right? What if you could write down these steps once, and just say "Make my sandwich" and everything happens automatically? That's exactly what Docker Compose does for your applications!

What is Docker Compose? 🥪

Docker Compose is like a recipe book that tells your computer how to set up all the parts of your application. Instead of a sandwich, think of a web application that needs:

A web server (like your plate to serve)
A database (like your ingredient shelf)
Some storage (like your fridge)

The Magic Recipe (docker-compose.yml) 📝

version: '3'
services:
  web:
    image: nginx
    ports:
      - "80:80"

  database:
    image: postgres
    environment:
      - POSTGRES_PASSWORD=secret

This is like your sandwich recipe that says:

Get your plate ready (web server)
Organize your ingredients (database)

Why Use Docker Compose? 🤔

One Command Setup: Just say docker-compose up and everything starts in the right order!
Always the Same: Like following a recipe, you get the same result every time
Easy to Share: Share your recipe, and others can make the exact same setup

Prerequisites 🛠️

Before we start making our sandwich (I mean, running our app), let's make sure we have all the tools:

Docker Desktop (Download)
Docker Compose (Don't worry, it comes with Docker Desktop!)

You can check if everything's ready by running:

docker --version
docker-compose --version

Common Commands 🔪

docker-compose up    # Start making
docker-compose down  # Clean up
docker-compose ps    # Check what's running

Best Practices 🌟

Keep It Simple
- Only add what you need
- Start small, grow as needed
Stay Organized
- Label everything clearly
- Keep related things together

Common Problems and Solutions 🔧

"My Containers Won't Talk!"
- Like ingredients that need to mix, containers need to connect
- Docker Compose handles this automatically
"My Data Disappeared!"
- Use volumes to save data:

   volumes:
     - ./my_data:/app/data

Real Example 🚀

Check out our demo app:

A webpage to show messages
An API to handle requests
MongoDB to store everything

Visit http://localhost:8080 to try it out!

UI 🖥️

What's Next? 💭

We'd love to hear from you! Tell us:

What topics would you like to learn more about?
Which parts need more explanation?
What real-world problems are you trying to solve?

Drop your comments below or open an issue in our GitHub repository!

Want to see this in action? Check out the companion repository with working examples!

Building a Chat App with MCP(Model Context Protocol) and Claude in Node js

RajeshRenato — Sun, 13 Apr 2025 11:42:59 +0000

Building a Chat App with MCP(Model Context Protocol) in Node.js

I recently built a simple chat application that lets Claude call external tools using the Model Context Protocol (MCP). Here's how you can build one too.

What is MCP?

The Model Context Protocol (MCP) is a standardized way for AI models like Claude to interact with external tools. It provides a structured format for:

Defining tools the AI can use
Sending requests from the AI to tools
Returning results back to the AI

Project Structure

The application has three main parts:

Express Server (server.js): Handles web requests and user sessions
MCP Client (mcpClient.js): Connects to Claude and the MCP server
MCP Server (mcpServer.js): Defines and implements the tools

Chat UI

Architecture

How to Add Tools to Your MCP Server

The MCP server is where you define tools that Claude can use. Here's a basic example of how to create a weather tool:

// In mcpServer.js
import { Server } from "@modelcontextprotocol/sdk/server/index.js";

const server = new Server({ name: "mcp-weather-server", version: "1.0.0" });

// Define a weather tool
server.defineTool({
  name: "getCurrentWeather",
  description: "Get the current weather for a location",
  inputSchema: {
    type: "object",
    properties: {
      location: {
        type: "string",
        description: "The city and state, e.g. San Francisco, CA",
      },
      unit: {
        type: "string",
        enum: ["celsius", "fahrenheit"],
        description: "The unit of temperature to use",
      },
    },
    required: ["location"],
  },
  handler: async function (args) {
    const { location, unit = "celsius" } = args;

    // Here you would typically call a weather API
    // For demo purposes, we're returning mock data
    return {
      location: location,
      temperature: unit === "celsius" ? 22 : 72,
      conditions: "Sunny",
      humidity: "45%",
      windSpeed: "10 km/h",
    };
  },
});

// Start the server
server.start();

Available Tool Types

You can create different types of tools for Claude to use:

Data Retrieval Tools: Get weather, news, stock prices, etc.
Calculation Tools: Perform complex calculations or data analysis
Database Tools: Query or update databases
API Integration Tools: Connect to external services
File Processing Tools: Read, write, or analyze files

How the MCP Client Works

The MCP client connects Claude to your tools:

async processQuery(query) {
  // Add user message to history
  this.chatHistory.push({ role: 'user', content: query });

  // Send to Claude with tool definitions
  const response = await this.anthropic.messages.create({
    model: "claude-3-5-sonnet-20241022",
    max_tokens: 1000,
    messages: this.chatHistory,
    tools: this.tools,
  });

  // Process the response
  for (const content of response.content) {
    if (content.type === "tool_use") {
      // Claude wants to use a tool
      const result = await this.mcp.callTool({
        name: content.name,
        arguments: content.input,
      });

      // Send tool result back to Claude
      this.chatHistory.push({
        role: "user",
        content: JSON.stringify(result.content),
      });
    }
  }
}

Setting Up Your Project

To build your own MCP chat app:

Clone the repository: git clone https://github.com/RajeshRenato/mcp-node
Install dependencies: npm install
Add your Anthropic API key to a .env file
Create your tools in mcpServer.js
Start the server: node server.js

Example Tools You Can Build

Here are some ideas for tools you could add:

News Search: Fetch recent news articles on a topic
Wikipedia Lookup: Search and summarize Wikipedia content
Calendar Integration: Check or create calendar events
Language Translation: Translate text between languages
Image Generation: Generate images based on text descriptions (using DALL-E or similar)

Conclusion

The Model Context Protocol opens up exciting possibilities for AI applications. By giving Claude access to external tools, you can build powerful, interactive applications that combine AI with real-time data and functionality.

Want to try it yourself? Get the complete code on GitHub.

Got questions? Drop them below! 🚀

Understanding Node.js Streams: Readable, Writable, Transform (With Custom Examples!) 🚀

RajeshRenato — Sun, 13 Apr 2025 10:33:05 +0000

If you've ever worked with large files, network sockets, or real-time data processing in Node.js, you've probably come across Streams. But what exactly are Readable, Writable, and Transform streams? And how do you create custom ones?

This post will simplify Node.js streams and show you how to create your own from scratch 💡

📦 What Are Streams in Node.js?

Streams are a way to handle reading/writing data efficiently—especially large amounts of it—without loading everything into memory at once.

Memory Efficiency Example 🧠

Imagine processing a 1GB file:

// ❌ Without streams (loads entire file into memory)
const data = fs.readFileSync('large-file.csv');
processData(data);

// ✅ With streams (processes chunks at a time)
fs.createReadStream('large-file.csv')
  .pipe(processData);

Node.js provides 4 types of streams:

Readable – You can read data from it.
Writable – You can write data to it.
Duplex – Both readable and writable (like a socket).
Transform – Duplex stream that modifies the data.

🧪 Let's Build Some Streams!

We'll build each stream step-by-step:

🟢 Custom Readable Stream

const { Readable } = require('stream');

class MyReadable extends Readable {
  constructor(options) {
    super(options);
    this.currentCharCode = 65; // ASCII A
  }

  _read(size) {
    if (this.currentCharCode > 90) {
      this.push(null); // end of stream
    } else {
      this.push(String.fromCharCode(this.currentCharCode++));
    }
  }
}

const readable = new MyReadable();
readable.on('data', chunk => {
  console.log(`Received: ${chunk}`);
});

🧠 Explanation:

We start from character 'A' and push till 'Z'
this.push(null) tells Node we're done

🟡 Custom Writable Stream

const { Writable } = require('stream');

class MyWritable extends Writable {
  _write(chunk, encoding, callback) {
    console.log(`Writing: ${chunk.toString()}`);
    callback(); // indicate ready for next write
  }
}

const writable = new MyWritable();
writable.write('Hello ');
writable.write('World!');
writable.end();

🧠 Explanation:

Every chunk passed to write() gets printed
callback() is important – it tells Node we're done writing the current chunk

🟣 Custom Transform Stream

const { Transform } = require('stream');

class MyTransform extends Transform {
  _transform(chunk, encoding, callback) {
    const upper = chunk.toString().toUpperCase();
    this.push(upper);
    callback();
  }
}

const transform = new MyTransform();

process.stdin.pipe(transform).pipe(process.stdout);

🧠 Explanation:

We take input from terminal, transform it to uppercase, and pipe it to output

Try running this in terminal and type something — it'll be converted on the fly!

🛠 Real World Use Case: File Piping with Transform

const fs = require('fs');
const { Transform } = require('stream');

const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

const transformStream = new Transform({
  transform(chunk, enc, cb) {
    const updated = chunk.toString().replace(/foo/g, 'bar');
    cb(null, updated);
  }
});

readStream.pipe(transformStream).pipe(writeStream);

👆 This replaces "foo" with "bar" while streaming data between files.

🚀 Advanced Stream Features

🔄 Stream Pipeline (Better Error Handling)

The pipeline function properly handles errors and resource cleanup:

const { pipeline } = require('stream');
const fs = require('fs');

const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

pipeline(
  readStream,
  transformStream,
  writeStream,
  (err) => {
    if (err) {
      console.error('Pipeline failed:', err);
    } else {
      console.log('Pipeline succeeded!');
    }
  }
);

🔄 Error Handling with Streams

Always handle errors in your streams to prevent crashes:

readStream.on('error', (err) => {
  console.error('Read error:', err);
});

writeStream.on('error', (err) => {
  console.error('Write error:', err);
});

🔄 Object Mode Streams

Streams can process objects instead of just buffers/strings:

const { Transform } = require('stream');

const objectTransform = new Transform({
  objectMode: true,
  transform(chunk, encoding, callback) {
    // Process objects instead of buffers
    const transformedObject = { 
      ...chunk, 
      processed: true,
      timestamp: Date.now()
    };
    this.push(transformedObject);
    callback();
  }
});

🔄 Async Iteration with Streams (Modern JS)

ES2018 introduced a cleaner way to consume streams:

const fs = require('fs');

async function processFile() {
  const stream = fs.createReadStream('data.txt');

  // Process each chunk using for-await-of
  for await (const chunk of stream) {
    console.log('Chunk:', chunk.toString());
  }

  console.log('Stream processing complete!');
}

processFile().catch(console.error);

🔄 Understanding Backpressure

Backpressure happens when data is produced faster than it can be consumed.
Stream's pipe handles this automatically:

// ✅ Handles backpressure (waits if destination is slow)
sourceStream.pipe(destinationStream);

// ❌ Ignores backpressure (could overwhelm destination)
sourceStream.on('data', (chunk) => {
  destinationStream.write(chunk); // No flow control!
});

🧵 TL;DR

Readable Stream: You push data out.
Writable Stream: You consume data being written.
Transform Stream: You take input, modify it, and push it forward.
Pipeline: Better error management and resource cleanup.
Backpressure: Automatic flow control between fast producers and slow consumers.

💡 Tips

Always call callback() in _write() or _transform()
Use .push(null) to signal end of readable stream
Prefer streams when working with large data (files, network, etc.)
Don't forget to handle errors
Use pipeline() for safer piping

📚 One More Example?

One final version of custom Readable, Writable and Transform Stream


const { pipeline } = require('stream');

const { Readable, Writable, Transform } = require('stream');

class MyReadable extends Readable {
  constructor(options) {
    super(options);
    this.currentCharCode = 65; // ASCII A
  }

  _read(size) {
    if (this.currentCharCode > 90) {
      this.push(null); // end of stream
    } else {
      this.push(String.fromCharCode(this.currentCharCode++));
    }
  }
}

class MyWritable extends Writable {
  _write(chunk, encoding, callback) {
    console.log(`Writing: ${chunk.toString()}`);
    callback(); // indicate ready for next write
  }
}

class MyTransform extends Transform {
  _transform(chunk, encoding, callback) {
    const upper = chunk.toString().toUpperCase();
    this.push(upper);
    callback();
  }
}

const readable = new MyReadable();
const writable = new MyWritable();
const transform = new MyTransform();

const { pipeline } = require('stream');

pipeline(
  readable,
  transform,
  writable,
  (err) => {
    if (err) {
      console.error('Pipeline failed:', err);
    } else {
      console.log('Pipeline succeeded');
    }
  }
);

Got questions? Drop them below! 🚀