DEV Community: Rishi Kumar

Breakdown: Who is Responsible?

Rishi Kumar — Wed, 24 Dec 2025 12:35:28 +0000

In the vast majority of commercial system designs (web apps, e-commerce, banking, SaaS), the hierarchy of "slowness" is almost always the same.

For a typical application, the ranking of importance (and performance bottleneck) looks like this:

1. Database Queries (The bottleneck)
2. Network Calls (The delay)
3. Code Algorithms (The easy part)

Here is the breakdown of why, including the estimated impact percentages for a standard web application.

The Breakdown: Who is Responsible?

1. Database Queries (~60% of Performance Issues)

In 9 out of 10 system design interviews (and real-world outages), the database is the first thing to break.

Why: Reading data from a hard drive (Disk I/O) is physically slow. Even with SSDs, searching through millions of rows, joining tables, and waiting for "locks" takes millions of CPU cycles.
The Cost: A bad SQL query can freeze a system for seconds.
System Design Fix: This is why we use Caching (Redis/Memcached). We try desperately to avoid hitting the database.

2. Network Calls (~30% of Performance Issues)

This is the time it takes for data to travel from Server A to Server B (latency).

Why: Even if your code is instant, the speed of light is a limit. If your app has "Microservices" where Service A calls Service B, which calls Service C, you are simply waiting for the data to travel wires.
The Cost: A network call usually takes milliseconds (ms). In CPU time, 1ms is an eternity.
System Design Fix: This is why we use Batching (sending 100 items in 1 request instead of 100 requests) and CDNs.

3. Algorithms / Code Logic (~10% of Performance Issues)

Unless you are doing AI, Video Processing, or Crypto Mining, your algorithm is rarely the problem.

Why: Modern CPUs can do billions of calculations per second. A loop that iterates 10,000 times to sum a shopping cart takes microseconds (0.000001 seconds).
The Cost: Negligible in comparison to waiting for a database.
System Design Fix: Keep code clean, but don't obsess over micro-optimizations. Focus on readability.

The "Scale of Latency" (Why Algo loses)

To understand why Algorithms matter so little compared to Network/DB, look at the time scale differences.

Let's imagine 1 CPU Cycle = 1 Second.

Action	Actual Time	Relative Time (If 1 cycle = 1 sec)
Algo: Simple Loop / Variable access	~1 nanosecond	1 second
Network: Sending 2KB data over 1Gbps	~20,000 ns	5.5 hours
Network: Packet Round Trip (Ping)	~150,000 ns	1.7 days
Disk/DB: Reading from SSD	~1,000,000 ns	11.5 days
Network: Packet sent from US to Europe	~150,000,000 ns	4.8 years

Conclusion: Optimizing your code to save 5 CPU cycles is useless if your database query takes "11 days" (relative time) to return.

When does the Algorithm become #1?

There are specific cases where the Algorithm becomes the most important factor (80%+ impact):

Video Compression/Streaming: (Netflix/YouTube encoding) - Heavy CPU math.
High-Frequency Trading: (Stock markets) - Every nanosecond counts; network is optimized via hardware, so code speed wins.
Encryption/Hashing: (Blockchain/Crypto) - Pure mathematical processing.
AI/ML Inference: Running neural networks is pure matrix multiplication.

Summary

For a standard company system (like an e-commerce store or social network):

Database (60%): Optimize your SQL, add Indexes, use Redis.
Network (30%): Reduce the number of calls (Chatty I/O), keep servers close to users.
Algo (10%): Just avoid obvious mistakes (like nested loops on massive arrays).

Ultimate Logging Guide for Node.js, Bun, and Next.js SSR in 2025

Rishi Kumar — Tue, 22 Apr 2025 10:16:38 +0000

If you’ve been building Node.js apps for a while, you’ve probably used Winston, Pino, or maybe Bunyan for logging. And they’ve served us well — structured logs, log levels, file transport — all good stuff.

But here's the thing: logs alone aren't enough anymore.

Modern applications need more than just lines in the terminal or log files. You need to see logs in a dashboard, filter them, search by word, errors with meta details, get alerts, and sometimes even dig into logs from multiple environments — local, staging, prod — without relying on a third-party cloud provider.

And that’s exactly why Errsole exists.

It’s an open-source logging system built specifically for Node.js. It’s not just a logger — it’s a complete log management tool, with:

A web-based log viewer
Alerts (email, Slack)
Support for structured logging, metadata, and JSON payloads
Built-in support for Express, Fastify, NestJS, Koa, Hapi, Next.js (SSR)
Storage flexibility (SQLite, MySQL, Postgres, MongoDB)
Production-grade performance that beats Elasticsearch and Cloud loggers by 70,000–90,000 RPM
All self-hosted, no vendor lock-in

Let’s break this down properly and explain why Errsole is different than every other Node.js logger — and why it might be the only logging tool you’ll need.

1. Built-in Log Viewer That Runs in Your App

With Pino or Winston, you get log lines. If you want to view them visually, you have to set up something like Kibana, Grafana Loki, or a third-party tool like Better Stack or Logtail. That takes time, config, and often... monthly bills.

With Errsole, the log viewer is part of the package.

It runs as a lightweight web dashboard (by default on port 8001), where you can:

Search logs by keyword, user ID, path, or any custom field
Filter logs by level (error, warn, debug, info, alert)
See metadata like HTTP headers, DB queries, request bodies
View logs in full-screen JSON mode
Inspect logs across environments and apps

And it’s super easy to embed. Just add the Express proxy middleware and you can serve the log viewer at /errsole inside your own app — no need to open ports or run extra containers.

2. Instant Setup with Built-in Console Capture

Errsole works out of the box. As soon as you call errsole.initialize(), it starts capturing all logs, including:

console.log, console.warn, console.error
Uncaught exceptions
Unhandled rejections
Stack traces
Errors thrown inside async functions or routes

No need to rewrite your existing log statements or wrap everything in custom logic. Errsole handles it all for you.

You can also disable console output, or configure which log levels to capture using collectLogs option in its config.

3. Blazing Fast Performance (Benchmarks Don't Lie)

Let’s talk real numbers.

When benchmarking Errsole vs popular setups like Pino + Elasticsearch or Winston + CloudWatch, the difference was massive.

In tests using AWS EC2 instances and a simple Node.js app under K6 load testing, Errsole consistently handled 70,000–90,000 more requests per minute than Elasticsearch-based loggers.

Why?

Because:

Elasticsearch, CloudWatch, and Better Stack all use HTTP. Logs are sent over the network — which adds latency, risk of log loss, and slowdowns under load.
Errsole writes to local databases, like SQLite, MySQL, MongoDB, or Postgres. That’s faster, more reliable, and 100% under your control.

4. Powerful Advanced Logging Features

Errsole isn’t just about catching logs — it helps you understand them deeply.

It supports:

Log Levels: log, info, warn, error, alert, debug
Alerts: Send critical logs to Slack or Email with errsole.alert()
Structured Logs: Attach metadata (like req.body, query params, user session info)
JSON-friendly view: Inspect logs in a clean, collapsible format
Search + Filter UI: Find logs by route, level, words, or custom tags

5. Storage Options for Every Use Case

Errsole doesn’t force you into one database or logging stack. You choose how and where to store your logs.

It supports:

SQLite for local development, testing, or small apps
MySQL for shared logs in production APIs
PostgreSQL for scalable, transactional logging
MongoDB for flexible, schema-less logs and large-scale data

_You can even run multiple apps logging into the same DB, using tablePrefix or collectionPrefix to separate logs by app name.

_

6. Runs Anywhere — Local, Server, Docker, Cloud

Whether you’re developing locally, deploying to EC2, running containers, or using serverless platforms, Errsole fits right in.

You can:

Run it as a standalone app with its own viewer UI
Mount the dashboard inside your main app at any route
Use it in dev, staging, or prod — with separate DBs or shared ones
Use appName, environmentName, serverName to organize logs by source

7. Works with Your Favorite Frameworks

Errsole integrates smoothly with popular Node.js frameworks:

Express (zero-config)
Fastify (full guide available)
NestJS (supports interceptors, services, DI, etc.)
Next.js SSR apps (server-side logging with SQLite or MySQL)

It even works as a transport for Winston via winston-errsole, so you can upgrade your existing logger setup without a rewrite.

8. Safer Than Remote Logging

Let’s be honest — cloud loggers are fast to set up, but they come with risks:

Privacy issues: Your logs go to someone else’s server
Downtime risk: If the logging service goes down, your logs vanish
App crashes: Under high load, HTTP log calls can block or fail
High cost: You’ll pay per GB or per log line as you scale

With Errsole, none of that happens. Your logs are on your infra, you pay $0/month, and your app stays safe — even under pressure.

9. Real-World Proof & Developer Stories

Developers have already shared their stories of switching to Errsole:

One team saw 60% throughput improvement after replacing Better Stack with Errsole
Another dev caught an error that almost wiped their app’s production data — and Errsole helped them debug it in seconds
Multiple teams use Errsole daily in Fastify, Next.js, and NestJS apps running at scale

10. Open Source, Free Forever

Errsole is fully open-source, with permissive licensing, and runs under your full control. You can:

Fork it
Self-host it
Contribute to it

No API limits. No tracking. No hidden costs. Just logs you control, in a viewer you own.

Final Thoughts

Winston and Pino will give you log lines. Cloud loggers give you dashboards — for a price. But Errsole gives you the best of both worlds:

Easy to set up
Fast to scale
Beautiful to view
Safe for production
100% free and self-hosted

Whether you’re a solo dev building side projects, or a team deploying production microservices, Errsole is the logging tool you didn’t know you needed.

Ready to get started?
👉 npm install errsole
👉 Visit: https://github.com/errsole/errsole.js

Errsole Live on Product Hunt

Rishi Kumar — Wed, 16 Apr 2025 17:21:57 +0000

Hi, we've just launched a free, open-source Node.js logger for developers on Product Hunt!

If you find it interesting, support us, your support would mean a lot.

Link: https://www.producthunt.com/posts/errsole

Would also be lovely if your team gets a chance to check it out too

Worker Threads in Node.js and Their Impact on Logging Performance

Rishi Kumar — Tue, 25 Feb 2025 11:50:23 +0000

Node.js is known for its non-blocking, event-driven architecture that makes it efficient for handling asynchronous operations. However, when it comes to CPU-bound tasks, the single-threaded nature of Node.js can become a bottleneck. This is where worker threads come into play. They allow developers to offload CPU-intensive tasks from the main event loop to separate threads within the same process.

What Are Worker Threads?

Worker threads in Node.js enable parallelism by running JavaScript code concurrently. Instead of executing everything on the main thread—which could block incoming requests and slow down performance—worker threads let you perform heavy computations in the background. This design helps keep the main event loop free to handle I/O operations and respond to client requests.

How Worker Threads Affect System Performance

Worker threads offer significant benefits on multi-core systems where true parallelism can be achieved. When the operating system distributes the load across multiple cores, the main thread can focus on handling I/O-bound tasks, while worker threads manage CPU-intensive operations.

On Single-Core Systems

No True Parallelism: With only one core, both the main thread and any worker threads must share the same CPU time. This means that the tasks are interleaved rather than executed concurrently.
Overhead of Context Switching: Offloading work to a worker thread introduces overhead due to inter-thread communication and context switching. On a single-core machine, these overheads can diminish or even negate the benefits of offloading tasks.
Scheduling Delays: The operating system must alternate between executing the main thread and the worker thread, potentially causing delays under heavy computational or logging loads.

On Multi-Core Systems Fully Occupied by Node.js Processes

Even on multi-core machines, if all cores are dedicated to running Node.js processes (for example, in a clustering setup to handle high traffic), similar performance constraints can emerge:

Resource Contention: When every core is fully utilized by Node.js processes, offloading work to a worker thread can introduce additional context switching and inter-thread communication overhead. This may limit performance gains, especially for lightweight tasks.
Marginal Gains for Lightweight Tasks: For logging tasks that are designed to be lightweight and asynchronous, the extra overhead associated with worker threads might not translate into significant performance improvements compared to a well-optimized main-thread solution.

Worker Threads in Logging

Logging is a critical aspect of many applications, and its implementation can significantly influence overall performance. There are two common approaches:

Main Thread Logging: This traditional approach performs asynchronous I/O directly on the main thread. With proper optimization—such as using non-blocking I/O and connection pooling—this method can efficiently handle high volumes of log messages without causing significant delays.
Worker Thread Logging: In this approach, logging operations are offloaded to a worker thread. This can be especially beneficial when the logging process involves heavy processing, such as complex formatting, data transformation, or buffering. Offloading these tasks helps to ensure that the main thread remains free to manage incoming requests.

When Worker Threads May Not Offer Benefits

On Single-Core Systems: Without true parallelism, the benefits of offloading work to a worker thread are reduced. The overhead from inter-thread communication and context switching can negate any potential advantages, particularly under heavy logging or CPU-intensive conditions.
On Multi-Core Systems with Full Node.js Utilization: Even in multi-core environments, if all cores are already occupied by Node.js processes handling application traffic, the extra overhead introduced by worker threads might lead to performance bottlenecks for lightweight tasks. In such cases, a well-tuned main-thread logging approach could perform better.

Worker threads in Node.js provide a powerful mechanism for offloading CPU-intensive tasks and keeping the main event loop responsive. Their benefits are most pronounced in scenarios where heavy computational tasks can run concurrently on multi-core systems. However, on single-core machines—or even on multi-core systems where all cores are fully utilized by Node.js processes—the overhead associated with worker threads may outweigh their advantages, particularly for lightweight logging tasks.

Ultimately, the decision to use worker threads for logging should be based on a careful analysis of your workload and the system’s resource utilization. Profiling your application under realistic conditions is essential to determine whether offloading to a worker thread offers tangible benefits over a well-optimized main-thread approach.

To work best, a worker thread ideally should have a dedicated core so that under any condition, it can perform better without contending for resources with the main thread.

Why Paid Logging Services Like BetterStack Can Hurt Your Throughput: Node.js

Rishi Kumar — Fri, 21 Feb 2025 06:44:36 +0000

I’ve been on a journey—one that started with a simple idea: build a free open-source Node.js logger. (No, I’m not going to name it because this isn’t a promotional story—just an honest one.)

I tested many logging tools—both open-source and paid. Each had its pluses and minuses, but let me share my experience with one paid service: Better Stack.

It's not just about Better Stack. It's about many paid logging services.

The results of Better Stack vs Open-source logging services:

Better Stack (HTTP endpoint): ~110k requests/min
Open-source (LocalDB): ~300k requests/min

Why this performance discrepancy happens and why a paid logging service that relies on HTTP/HTTPS can bring your system’s throughput down.

Paid Logging Services and Their Drawbacks

Paid logging services often provide many other services with logging services. But if you into only logging service feature then this may help you to decide what to chose between paid service / open source logger.

Take Better Stack’s logging feature, for instance. In a recent performance test, using winston-betterstack to ship logs to Better Stack’s remote endpoint yielded only around 110k requests/min throughput. This performance is notably lower than many self‑hosted or local logging setups, which often reach upwards of 300k requests/min.

So, why the discrepancy?

HTTP/HTTPS Overhead: Logging to a remote service requires creating outbound HTTP requests for every log entry. This is inherently slower than writing directly to a local or same‑network database, where you can use efficient protocols and avoid extra layers of latency.
Extra Network Hops: Sending logs across the internet means more points of failure and potential congestion. You rely on your connection stability, the remote provider’s ingestion pipeline, and everything in between.
Rate Limiting & Throttling: Paid logging services often enforce ingestion or rate limits to protect their shared infrastructure. During traffic spikes, you risk losing critical logs or facing significant ingestion delays. For example, if you push logs via pino-betterstack at high volumes, you might suddenly hit throttling thresholds—and lose logs right when you need them the most.
Added Cost: Beyond slower performance, you’re also locked into monthly fees or usage‑based billing. If your application scales, you may find that your logging bills become larger and less predictable.

In short, paid services that rely on remote endpoints inherently add latency and can limit throughput. Plus, they’re multi-tenant, so you share resources and bandwidth with countless other users. All it takes is one usage surge—yours or another tenant’s—to stall or throttle your logs.

The Benefits of Open-Source Logging Solutions

If the drawbacks of paid services concern you, the good news is that open-source and self-hosted alternatives can address most of these pain points. Whether you’re storing logs in MySQL, Postgres, or even using the ELK stack (Elasticsearch, Logstash, and Kibana), open-source systems put you in control.

Here’s why that’s a game-changer:

Faster and More Direct
- When you log to a locally hosted or network‑close database, you bypass the overhead of HTTP requests and external networks. In many cases, the data flows directly through a binary protocol (e.g., MySQL’s native protocol), significantly reducing latency.
- Typical throughput comparisons can show 300k+ requests/min on self‑hosted systems, which outperforms remote logging to Better Stack or similar paid providers.
Full Control Over the Pipeline
- With open-source tools, you control your hardware, network settings, and data storage configurations. You can tune MySQL, for instance, to optimize write performance or replicate data across multiple nodes without third‑party constraints.
- No arbitrary rate limits or forced throttling. You can choose to scale horizontally, add caching layers, or adjust ingestion queues according to your application’s needs.
Cost‑Effectiveness
- Open‑source solutions are free to use—your main costs are infrastructure and time. That can lead to significant savings once you pass a certain scale, especially if you’re using existing servers or shared resources.
- You won’t be blindsided by usage‑based pricing or sudden overage fees.
Avoid Vendor Lock-In
- Proprietary platforms make it harder to switch providers because your logs and integrations live behind their paywall. In contrast, open-source solutions store your data in standard formats you control. If you need to migrate or pivot, you can do so without being chained to a provider’s platform.
Community-Driven Development
- Open-source projects benefit from a community of contributors who actively improve performance, fix bugs, and release new features. As a user, you can file issues, submit pull requests, or just benefit from the collective expertise of a global user base.

In summary, open-source logging solutions give you the freedom to manage your logs at scale without worrying about throttling, arbitrary limits, or hidden fees. While paid services might be convenient for smaller shops or teams that need quick deployment and can tolerate lower throughput, high‑performance applications will likely find more reliability, control, and cost‑savings in open-source, self‑hosted setups.

Final Word

Paid logging services can provide a slick user experience—but if you value raw performance, cost control, and having complete ownership over your logs, they often fall short. As shown in our comparison with Better Stack, relying on an external HTTP endpoint can cap your throughput significantly and impose usage or rate limits that lead to dropped logs.

Open-source solutions, on the other hand, empower you with direct control over your infrastructure, letting you optimize for both speed and reliability. For teams that prioritize performance, scalability, and cost‑savings, self‑hosting is often the superior choice—and it avoids the headaches and constraints imposed by remote, multi-tenant providers.

I know—setting up open-source or self-hosted solutions can be a hassle, while paid services are often simpler. Many open-source loggers require a complicated setup and some tech know-how. That’s why I built a Node.js logger that can be up and running in under a minute—so easy even an intern can do it!

Also, there are lots of open-source logging tools out there. Some are old and solid, others are new with great speed and simple setup. There’s no absolute “best” or “worst” option—it really depends on what you need for your project.

The Magical World of Garbage Collection in Node.js

Rishi Kumar — Wed, 19 Feb 2025 12:38:29 +0000

How Your Code Stays Clean Without Lifting a Finger

Imagine if your room could magically clean itself up every time you left a mess—no more piles of dirty dishes or clothes strewn everywhere. Well, in the world of Node.js, there’s something almost as awesome working behind the scenes: Garbage Collection (GC). And yes, it’s as cool as it sounds!

Let’s dive into this topic with a fun twist. Here’s a down-to-earth, no-fluff guide that explains how Node.js’ GC works, why it’s your unsung hero, and a few tips that will actually help you improve your code’s performance. Let’s get started!

What’s Garbage Collection Anyway?

Picture your Node.js application as a bustling kitchen. Every time you whip up a meal (i.e., create an object), you use ingredients (memory). Eventually, you finish cooking, but the ingredients and tools remain on the counter, cluttering your workspace. Garbage Collection is like that magical helper who comes in, tidies up, and throws away the leftovers you no longer need—so you have plenty of room to keep cooking without tripping over a mess.

The Secret Life of GC in Node.js (Powered by V8)

Node.js rides on the back of the V8 engine, which has its own nifty way of handling garbage collection. Let’s break it down in a fun, relatable way:

1. Young vs. Old Generation: The Teenage Rebels and Seasoned Veterans

Young Generation (The Teenagers):
Think of these objects as rebellious teenagers—new, energetic, and likely to disappear quickly. V8 uses a copying algorithm here. It quickly sweeps through, moving the survivors (the few that manage to stick around) to a more mature area.
Old Generation (The Veterans):
Now, the objects that survive the wild teenage phase become the "old generation"—loyal, stable, and around for the long haul. Cleaning up here is a bit more involved, often using a mark-sweep or mark-compact method, but hey, these veterans deserve a careful cleanup!

2. Incremental & Concurrent GC: Cleaning Without the Drama

Imagine your magical cleaning helper working in small, almost invisible bursts instead of a massive all-day cleaning frenzy.

Incremental Marking: This breaks down the cleaning process into small chunks so your app doesn’t freeze up for a long time.
Concurrent Sweeping: Parts of the cleanup happen while your app is still busy doing its thing. No long pauses—just smooth, nearly invisible maintenance.

How to Peek Behind the Curtain: Detecting GC in Action

Even though GC works its magic quietly, sometimes you need to know what’s going on under the hood—especially when your app seems to lag or memory usage starts creeping up. Here are some tricks to help you spot GC in action:

1. Turn on the GC Radar with Runtime Flags

Fire up your Node.js app with these flags to see real-time GC events:

node --trace_gc your-app.js

This command prints GC events to your console. It’s like having a window into the cleaning crew’s schedule.

2. Use Chrome DevTools for a Closer Look

Start Node.js with the --inspect flag, and open Chrome DevTools to inspect memory usage. The Memory tab lets you:

Take heap snapshots.
Monitor object allocation.
Identify memory leaks.

3. Bring in the Heavy Machinery: Profiling Tools

For more detailed insights, try tools like:

clinic.js: Visualizes performance issues.
heapdump: Captures snapshots of your heap to find stubborn memory hogs.
node-memwatch: Alerts you about leaks and GC events.

Real-World Scenarios: When GC Can Trip You Up

Even with its magical prowess, GC can sometimes cause hiccups in your application. Here’s what to watch for and how to tackle it:

Memory Leaks: The Unwanted Houseguests

Symptoms: Gradually increasing memory usage, frequent long GC pauses, and, eventually, your app getting sluggish or crashing.
Pro Tip: Regularly profile your app with heap snapshots. Keep an eye on closures, global variables, and event listeners that might be holding onto memory longer than needed.

High GC Pauses: When the Cleaning Crew Overdoes It

Symptoms: Noticeable stutters or freezes during heavy object creation and deletion.
Workaround: Optimize your code by reducing unnecessary object creation. Think about reusing objects or using object pools for things that get created and destroyed frequently.

GC Overhead in High-Throughput Systems

Scenario: Your app is handling thousands of requests per second, and the GC is working overtime to clean up all those ephemeral objects.
Solution: Consider refactoring critical paths to minimize short-lived allocations. Use performance monitoring (APM tools like New Relic or Datadog) to spot and address these issues before they impact users.

Embrace the Magic!

Garbage Collection in Node.js isn’t just a background process—it’s a sophisticated, finely-tuned system that keeps your app running smoothly without you having to sweat the small stuff. By understanding the basics of how GC works (and having some fun with it), you can write cleaner, more efficient code and avoid common pitfalls like memory leaks and performance lags.

So next time you marvel at how your Node.js app manages to stay performant despite a chaotic codebase, give a nod to the unsung hero: Garbage Collection. It’s working hard behind the scenes, so you can focus on building awesome features without getting bogged down in manual memory management.

Below is your original "Magical World of Garbage Collection" article with additional details appended. The original text remains unchanged, and the extra sections provide more insights for those who crave a bit more technical magic!

Bonus: Extra Magical Tidbits for the Adventurous Developer

While we’ve journeyed through the enchanting basics of GC, here are some extra details and tools to help you master the art of memory management in Node.js:

Dive Deeper into the V8 Engine

Major vs. Minor GC: Remember, the young generation’s minor GCs are quick sprinkles of magic, while the old generation’s major GCs are like a deep cleaning session. These full GCs scan the entire heap and are more performance-intensive—so watch out when your app starts to slow down.
Tuning Your Memory: Experiment with Node.js runtime flags like --max-old-space-size to control how much memory your app can use. Adjusting these settings can help manage GC behavior during heavy loads.

Unleash the Power of Profiling Tools

Chrome DevTools & Heap Snapshots: Launch your app with the --inspect flag and connect via Chrome DevTools. The Memory tab isn’t just for pretty pictures—it lets you take snapshots of your heap, track object lifetimes, and pinpoint those sneaky memory leaks.
Advanced Tools: Give tools like Clinic.js a whirl to visualize performance bottlenecks, or use heapdump to capture the state of your memory during runtime. These tools add an extra layer of magic to your debugging process.

Modern Spells: WeakRef and FinalizationRegistry

Harnessing New ECMAScript Features: With WeakRef and FinalizationRegistry, you can create weak references to objects. This means your objects can vanish when they’re no longer needed, without holding up the GC process. It’s like setting up a farewell party for your objects!

Watch Out for Memory Gremlins

Memory Leaks & Lingering Closures: Sometimes, those pesky global variables, event listeners, or closures can hold onto memory longer than they should. Keep your code tidy by reviewing these potential trouble spots and running regular memory profilers.

Experiment, Benchmark, and Learn

Iterative Magic: The realm of garbage collection is ever-evolving. Try different coding patterns, benchmark your changes, and learn from each magical tweak you make. With practice, you’ll find the perfect balance for your Node.js applications.

Stream and HighWatermark: Node.js

Rishi Kumar — Fri, 31 Jan 2025 06:54:59 +0000

Understanding the interplay between objectMode and highWaterMark is essential for optimizing stream performance in Node.js. Let's delve into what each term means, how they interact, and whether objectMode improves highWaterMark.

Understanding highWaterMark
Understanding objectMode
Interaction Between objectMode and highWaterMark
Does objectMode Improve highWaterMark?
Best Practices

Understanding `highWaterMark`

What is `highWaterMark`?

highWaterMark is a configuration option in Node.js streams that specifies the threshold at which the stream will apply backpressure. It essentially determines how much data can be buffered in the stream before it starts to signal that it's "full" and needs to pause or slow down the incoming data.

Readable Streams: For readable streams, highWaterMark defines the maximum number of bytes (in binary mode) or objects (in object mode) to store in the internal buffer before ceasing to read from the underlying resource.
Writable Streams: For writable streams, it indicates the maximum number of bytes or objects that can be buffered while writing data before applying backpressure to the source.

Default Values

Binary Mode:
- Readable Streams: 16 KB (16 * 1024 bytes)
- Writable Streams: 16 KB (16 * 1024 bytes)
Object Mode:
- Both readable and writable streams default to a highWaterMark of 16 objects.

The above defaults are stable in modern Node.js versions (e.g., Node 14+). If you’re on an older version of Node, be sure to check the docs to confirm these defaults.

Why is `highWaterMark` Important?

Memory Management: Prevents excessive memory usage by limiting the amount of data buffered.
Flow Control: Ensures smooth data flow between producers and consumers without overwhelming either side.
Performance Optimization: Balances throughput and latency based on the application's requirements.

Understanding `objectMode`

What is `objectMode`?

objectMode is a stream option in Node.js that allows streams to handle arbitrary JavaScript objects instead of raw binary data (Buffer) or strings. When objectMode is enabled, the stream operates in "object mode," treating each chunk as a distinct object.

Enabling `objectMode`

To enable objectMode, set the objectMode option to true when creating a stream:

const { Readable } = require('stream');

const objectReadable = new Readable({
  objectMode: true,
  read() {}
});

Benefits of `objectMode`

Structured Data Handling: Ideal for streaming complex data structures like JSON objects, database records, etc.
Simplified Processing: Easier to work with objects directly without manual serialization/deserialization.
Flexibility: Can handle varying data types and structures seamlessly.

Keep in mind that objectMode can add a little overhead compared to binary mode, because the stream must handle discrete JavaScript objects rather than raw bytes. In most real-world use cases (like streaming database records), this overhead is negligible compared to the clarity gained when working with structured data.

Interaction Between `objectMode` and `highWaterMark`

How They Work Together

When objectMode is enabled, the interpretation of highWaterMark changes:

Binary Mode: highWaterMark is measured in bytes.
Object Mode: highWaterMark is measured in the number of objects.

For example, a highWaterMark of 16 in object mode means the internal buffer can hold up to 16 objects, regardless of their size.

Implications

Granularity: Object mode provides a more granular control based on the number of discrete objects rather than their byte size.
Consistency: Each object is treated equally, making buffer management predictable regardless of individual object sizes.
Backpressure Signaling: The stream can more accurately signal backpressure based on object count, which is particularly useful when object sizes vary significantly.

Does `objectMode` Improve `highWaterMark`?

Clarifying "Improve"

The term "improve" can be subjective, but in this context, it likely refers to whether using objectMode enhances the effectiveness or efficiency of highWaterMark in managing stream buffers and backpressure.

How `objectMode` Affects `highWaterMark`

Consistency in Buffer Management:
- Object Mode: Each object, regardless of size, is counted uniformly. This consistency is beneficial when dealing with objects of varying sizes, ensuring that buffer limits are based on the number of discrete items rather than their potentially unpredictable byte sizes.
- Binary Mode: Buffer limits are based on byte sizes, which can be problematic if object sizes vary widely, leading to inefficient memory usage or unexpected backpressure triggers.
Predictable Backpressure:
- Object Mode: By limiting the number of objects, developers can predict and manage backpressure more accurately, especially when object sizes are inconsistent.
- Binary Mode: Backpressure is tied to byte counts, which may not directly correlate with processing capabilities, especially when dealing with large or small objects.
Ease of Configuration:
- Object Mode: Setting highWaterMark based on the number of objects is often more intuitive when dealing with streams of objects.
- Binary Mode: Requires careful consideration of byte sizes, which may vary and complicate buffer management.

Does It "Improve" `highWaterMark`?

In scenarios where streams handle discrete, structured objects with varying sizes, objectMode enhances the effectiveness of highWaterMark by providing a more logical and consistent buffer management strategy. It allows developers to control the flow based on the number of objects, leading to more predictable backpressure behavior.

However, it's essential to note that:

Use Case Dependent: The benefits are most pronounced when dealing with object streams. For binary or text streams, objectMode is not applicable.
Performance Considerations: While objectMode offers better control for object streams, it may introduce some overhead compared to binary streams due to object handling. However, this is generally negligible compared to the benefits in appropriate use cases.

Summary

Yes, objectMode can improve the effectiveness of highWaterMark when dealing with streams of JavaScript objects by providing a more intuitive and consistent way to manage buffer limits and backpressure.
No, in the sense that objectMode doesn't inherently make highWaterMark "better" for all types of streams; its benefits are context-specific.

Best Practices

Choose the Right Mode:
- Use objectMode when streaming JavaScript objects.
- Use binary or text modes for streaming raw data like files, network packets, etc.
Set Appropriate highWaterMark Values:
- Object Mode: Set based on the number of objects that can be comfortably buffered without exhausting memory or causing delays.
- Binary Mode: Set based on byte sizes that align with processing capabilities and memory constraints.
Monitor and Adjust:
- Use tools and logging to monitor stream performance and adjust highWaterMark as needed.
- Be mindful of the nature of your data; for instance, if objects are large, a lower highWaterMark may be appropriate.
Handle Backpressure Properly:
- Respect the write() method's return value.
- Listen for the drain event to resume data flow when appropriate.

In Node.js, when a writable stream's internal buffer exceeds highWaterMark, the write() method will return false. That signals backpressure. You should then stop writing until the stream emits the drain event, which tells you it’s ready for more data. For readable streams, the flow can be paused automatically or manually (via pause() and resume()).

Optimize Stream Pipelines:
- Use stream.pipeline for better error handling and cleaner stream composition.
- Ensure all streams in the pipeline are correctly configured for objectMode if needed.
Avoid Unnecessary Object Mode:
- Enabling objectMode when not needed can lead to increased memory usage and processing overhead.

Practical Example

Object Mode vs. Binary Mode with `highWaterMark`

Let's illustrate how objectMode affects highWaterMark with a practical example.

const { Readable, Writable } = require('stream');

// Binary Mode Example
const binaryReadable = new Readable({
  highWaterMark: 1024, // 1 KB
  read() {}
});

const binaryWritable = new Writable({
  highWaterMark: 2048, // 2 KB
  write(chunk, encoding, callback) {
    console.log(`Writing ${chunk.length} bytes`);
    callback();
  }
});

binaryReadable.pipe(binaryWritable);

// Push data
binaryReadable.push(Buffer.alloc(500)); // Within highWaterMark
binaryReadable.push(Buffer.alloc(600)); // Exceeds highWaterMark, triggers backpressure
binaryReadable.push(null); // End the stream

// Object Mode Example
const objectReadable = new Readable({
  objectMode: true,
  highWaterMark: 2, // 2 objects
  read() {}
});

const objectWritable = new Writable({
  objectMode: true,
  highWaterMark: 3, // 3 objects
  write(obj, encoding, callback) {
    console.log(`Writing object:`, obj);
    callback();
  }
});

objectReadable.pipe(objectWritable);

// Push objects
objectReadable.push({ id: 1, name: 'Alice' }); // Within highWaterMark
objectReadable.push({ id: 2, name: 'Bob' });   // Within highWaterMark
objectReadable.push({ id: 3, name: 'Charlie' }); // Exceeds highWaterMark, triggers backpressure
objectReadable.push(null); // End the stream

Explanation:

Binary Mode:
- The binaryReadable stream has a highWaterMark of 1 KB.
- Pushing 500 bytes is fine, but pushing an additional 600 bytes exceeds the highWaterMark, triggering backpressure.
Object Mode:
- The objectReadable stream has a highWaterMark of 2 objects.
- Pushing three objects exceeds the highWaterMark, triggering backpressure based on the number of objects, not their size.

Conclusion

objectMode and highWaterMark are powerful configurations in Node.js streams that, when used together appropriately, can significantly enhance stream performance and reliability:

objectMode allows streams to handle JavaScript objects seamlessly, providing a natural way to work with structured data.
highWaterMark controls the buffering strategy, ensuring that streams do not overwhelm system resources by limiting the amount of data buffered.

By enabling objectMode when dealing with object streams, you can improve the effectiveness of highWaterMark by aligning buffer management with the nature of your data. This leads to more predictable backpressure behavior, efficient memory usage, and smoother data flow within your applications.

Final Tips

Assess Your Data: Determine whether your stream handles objects or binary data to choose the appropriate mode.
Configure Thoughtfully: Set highWaterMark based on the specific needs and constraints of your application.
Test and Iterate: Monitor stream behavior under different loads and adjust configurations to achieve optimal performance.

If you have more specific scenarios or further questions about objectMode and highWaterMark, feel free to comment!

Log Viewer for Next.js Server Side: Node.js

Rishi Kumar — Wed, 22 Jan 2025 10:01:22 +0000

Logging on the server side is vital when you’re building applications with Next.js. Server-side rendering (SSR) introduces unique challenges and potential errors that client-side logs can’t capture.

Let's walk through how to integrate Errsole into your Next.js application for a clean, insightful logging experience. You’ll see how to start logging quickly using SQLite for local storage and also learn about options for more centralized solutions such as MySQL or PostgreSQL.

1. Why Server-Side Logging Is Important

When Next.js renders pages on the server, it’s crucial to have a clear view of:

Application Health: Identify performance bottlenecks or errors impacting server-side rendering.
Debugging: Pin down issues that never surface on the client side.
Security and Auditing: Track suspicious or unexpected behaviors in your server logs.

A well-organized logging system makes these tasks straightforward, and Errsole provides a user-friendly way to capture and view these logs.

2. Install Errsole and Errsole-SQLite

Let’s start by installing Errsole and its SQLite storage plugin. SQLite offers a simple, file-based database—perfect for local development or small-scale projects.

npm install errsole errsole-sqlite

Tip: If you need a more centralized solution in production, Errsole also supports:

errsole-mysql for MySQL

errsole-postgres for PostgreSQL

errsole-mongodb for MongoDB

3. Create a Logger File (`logger.js`)

To keep your logging setup organized, we’ll create a dedicated file for initializing Errsole. Place this file in a lib folder. By doing this, you can import your logger into all your server-side routes under pages/api.

// lib/logger.js
import errsole from 'errsole';
import ErrsoleSQLite from 'errsole-sqlite';

// Initialize Errsole
errsole.initialize({
    storage: new ErrsoleSQLite('path/to/database.sqlite')
});

// Export errsole to use throughout your app
export default errsole;

Here’s what’s happening:

Importing Errsole: We bring in the core errsole module and the errsole-sqlite plugin.
Initialize: We configure Errsole to store logs in an SQLite database.
Export: We export our initialized logger so it can be reused across your project.

4. Use the Logger in API Routes

Now that we have our logger set up, let’s see how to log events in a Next.js API route. Any file in the pages/api directory runs on the server side, making it the perfect place for server logs.

// pages/api/hello.js
import logger from '../../lib/logger';

export default function handler(req, res) {
  // Log an info message
  logger.info('API /hello was called');

  // Simulate an error scenario
  const errorHappened = true;
  if (errorHappened) {
    logger.error('Something went wrong in /hello!');
  }

  res.status(200).json({ message: 'Hello from Next.js + Errsole!' });
}

Logging Levels

Errsole supports various log levels (info, error, warn, etc.), so you can categorize logs by severity:

info: General operational messages.
warn: Something unexpected but not breaking.
error: An error that needs immediate attention.

Using these levels consistently makes filtering and searching through logs much easier down the line.

5. Viewing Logs in the Errsole Dashboard

One of Errsole’s standout features is its Web Dashboard, which displays real-time logs and error insights in a central location.

After completing the setup, you can access the Errsole Web Dashboard through the following methods:

Local Environment: Open your web browser and visit http://localhost:8001/.
Remote Server: If you have deployed Errsole on a remote server, use the server's IP address or domain name followed by the port number (e.g., YourServerIP:8001 or YourDomain:8001).

Once done, you’ll see:

Timeline and Filtering: Search logs by date, severity, or message content.
Error Stacks: Detailed error context, including stack traces.
Real-time Updates: Watch logs appear as they happen.
Collaboration: Invite your teammates to join and work together to monitor logs effectively.

Advanced Tip: For multi-app setups, you can assign different table prefixes or database files for each project to keep logs separate.

6. Centralized Logging (MySQL, PostgreSQL, etc.)

If your application is at scale or you need advanced reporting capabilities, consider integrating a more robust database.

Example: Using MySQL

// lib/logger.js
import errsole from 'errsole';
import ErrsoleMySQL from 'errsole-mysql';

errsole.initialize({
  storage: new ErrsoleMySQL({
    host: 'mysql-host',
    user: 'database-user',
    password: 'database-password',
    database: 'database-name'
  })
});

export default errsole;

Similarly, you can use errsole-postgres, errsole-mongodb, or other supported databases for comprehensive log management.

7. Configure NGINX

If your app is behind an NGINX reverse proxy, you can configure access to the Errsole Web Dashboard by adding the following lines to your NGINX configuration file:

location = /helloworld/logs {
  return 301 /helloworld/logs/;
}
location /helloworld/logs/ {
  proxy_pass http://localhost:8001/;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
}

After updating the configuration, reload NGINX:

sudo nginx -s reload

You can now access the dashboard through your domain:

For HTTP: http://YourDomain/helloworld/logs/
For HTTPS: https://YourDomain/helloworld/logs/

Note: Replace /helloworld/logs with your desired log path.

8. Advanced Configuration

Option	Type	Description
storage	ErrsoleSQLite ErrsoleMongoDB ErrsoleMySQL ErrsolePostgres	Required. Specify the storage backend along with connection details.
collectLogs	Array of Strings	Optional. Default: `['error', 'info']`. By default, Errsole collects both error and info logs. To collect only error logs, set this to `['error']`. To disable log collection entirely, set this to an empty array, `[]`.
enableConsoleOutput	Boolean	Optional. Control whether log output is shown in the console.
exitOnException	Boolean	Optional. Default: `true`. By default, Errsole exits the process after capturing an uncaught exception. To disable this behavior, set exitOnException to `false`.
enableDashboard	Boolean	Optional. Default: `true`. Enable or disable the web dashboard feature.
port	Number	Optional. Default: `8001`. Specify the network port for the web dashboard.
path	String	Optional. Default: `/`. Define the base path for accessing the web dashboard.
appName	String	Optional. Specify the name of the app.
environmentName	String	Optional. Default: `process.env.NODE_ENV`. Specify the deployment environment.
serverName	String	Optional. Default: the hostname of the machine. Specify the name of the server.

9. Best Practices for Logging

Use Descriptive Messages: Make sure your log messages provide clear context—this helps when revisiting them in the future.
Avoid Over-Logging: Log only what’s necessary. Too many logs can obscure important information and increase storage costs.
Rotate or Archive: In production, make sure to rotate logs periodically to keep your storage in check.

Conclusion

By integrating Errsole into your Next.js server, you’ll gain real-time visibility into your application’s health and performance. Whether you choose SQLite for local development or a more advanced database like MySQL or PostgreSQL for production, Errsole offers a user-friendly solution to capture, store, and review all your server-side logs in one place.

Key Takeaways:

Initialize Errsole in a dedicated logger.js file under lib/ to keep things organized.
Import this logger throughout your server routes for consistent, structured logging.
Leverage the Errsole dashboard to view logs, filter them, and troubleshoot issues quickly.
Scale to centralized logging options (MySQL, PostgreSQL, etc.) as your application grows.

With this setup, you can confidently develop and maintain a robust Next.js application, keeping a close eye on everything happening behind the scenes! Enjoy the enhanced clarity and peace of mind that come with a proper logging strategy.

A Next.js application with server-side logging using Errsole and MySQL. Clone or download the repository to experience it in action.

GitHub Code: nextjs-server-log-viewer

Common Errors in Node.js and How to Fix Them

Rishi Kumar — Thu, 09 Jan 2025 09:12:39 +0000

Common Errors in Node.js and How to Fix Them

Node.js, celebrated for its highly scalable runtime environment, provides developers with a robust framework to build efficient, high-performance server-side applications. Nevertheless, its non-blocking, event-driven architecture introduces unique challenges that require a sophisticated understanding and systematic resolution.

Syntax Errors
Reference Errors
Type Errors
Module Not Found Errors
EventEmitter Memory Leaks
Unhandled Promise Rejections
Asynchronous Programming Issues
Error: Cannot Find Module
Networking Errors
Performance Issues

Syntax Errors

Definition

Syntax errors arise from deviations in the structural rules of JavaScript, such as unbalanced braces, incorrect punctuation, or misuse of keywords. These errors prevent the interpreter from executing the code.

Example

function helloWorld() {
    console.log("Hello, World!";
}

This code snippet demonstrates a mismatch between the parentheses and curly braces.

Resolution

✅ Debugging syntax errors involves reviewing the error messages provided by the interpreter, pinpointing the location of the issue, and resolving it. The corrected code is as follows:

function helloWorld() {
    console.log("Hello, World!");
}

Reference Errors

Definition

Reference errors occur when an undeclared or out-of-scope variable is accessed. These errors commonly arise from programming oversights or scope mismanagement.

Example

console.log(myVar);

In this instance, myVar is referenced without prior declaration, leading to an error.

Resolution

✅Ensure that variables are appropriately declared within their intended scope before use:

let myVar = "Hello";
console.log(myVar);

Type Errors

Definition

Type errors occur when an operation is performed on a data type that does not support it. These errors often reveal logical flaws in the program.

Example

let num = 5;
num.toUpperCase();

Numbers do not have a toUpperCase method, resulting in a type error.

Resolution

✅ Align operations with compatible data types:

let str = "hello";
console.log(str.toUpperCase());

Module Not Found Errors

Definition

Module Not Found errors occur when Node.js fails to locate a module specified in a require or import statement. This issue often stems from incorrect paths or missing dependencies.

Example

const express = require('express');

If express is not installed, the interpreter will throw an error.

Resolution

✅ Use the Node.js package manager to install the missing module:

npm install express

Additionally, verify the module path and its existence within the project.

EventEmitter Memory Leaks

Definition

The EventEmitter class in Node.js facilitates event-driven programming by allowing objects to emit events and handle listeners. Memory leaks occur when excessive listeners are attached to an EventEmitter instance without proper management, leading to resource exhaustion.

The Problem

Each time a listener is registered using .on() or .addListener(), a reference is retained, which can accumulate indefinitely. Node.js issues a warning if the number of listeners exceeds the default threshold of 10:

(MaxListenersExceededWarning: Possible EventEmitter memory leak detected.)

Example

const EventEmitter = require('events');
const emitter = new EventEmitter();

for (let i = 0; i < 20; i++) {
    emitter.on('data', () => console.log('data event'));
}

Resolution

✅ Increase the listener limit:

emitter.setMaxListeners(50);

✅ Remove listeners when they are no longer needed:

emitter.off('data', listener);

✅ For one-time listeners, use .once():

emitter.once('data', () => console.log('data event'));

Unhandled Promise Rejections

Definition

An unhandled promise rejection occurs when a promise is rejected without a corresponding .catch() handler. This omission can destabilize applications, particularly in production environments.

Example

Promise.reject("Error");

Resolution

✅ Attach .catch() handlers to all promises:

Promise.reject("Error").catch(err => console.error(err));

✅ Employ try...catch blocks with async/await:

async function fetchData() {
    try {
        const data = await someAsyncOperation();
        console.log(data);
    } catch (err) {
        console.error("Error fetching data:", err);
    }
}

✅ Set up a global error handler:

process.on('unhandledRejection', (reason, promise) => {
    console.error('Unhandled Rejection:', promise, 'Reason:', reason);
});

Networking Errors

Definition

Networking errors arise from failed interactions between the application and external services. These issues include connection timeouts, DNS errors, and malformed HTTP requests.

Example

const http = require('http');
http.get('http://invalid-url', res => {
    console.log(res);
});

Resolution

✅ Incorporate error handling:

http.get('http://invalid-url', res => {
    console.log(res);
}).on('error', err => {
    console.error(`Error: ${err.message}`);
});

✅ Address timeouts explicitly:

req.setTimeout(5000, () => {
    req.abort();
    console.error('Request timed out');
});

✅ Validate input URLs:

if (!validUrl.isUri('http://example.com')) {
    console.error('Invalid URL');
}

Performance Issues

Definition

Performance degradation in Node.js often arises from blocking operations, suboptimal queries, and excessive resource consumption, affecting response times and scalability.

Example

const data = fs.readFileSync('/file.txt');

Resolution

✅ Favor asynchronous operations:

fs.readFile('/file.txt', (err, data) => {
    if (err) throw err;
    console.log(data.toString());
});

✅ Implement caching with tools like Redis:

client.get('key', (err, value) => {
    if (value) {
        console.log('Cache hit:', value);
    }
});

✅ Monitor performance using tools like clinic.js and pm2.

Conclusion

Node.js, while powerful, requires meticulous handling of errors to ensure reliability and performance. Addressing syntax inconsistencies, unhandled promises, and networking failures through best practices fosters robust, scalable applications. Through deliberate debugging and optimization, developers can harness the full potential of Node.js to build sophisticated systems.

When Winston with MySQL or CloudWatch Go Wrong: Node.js

Rishi Kumar — Tue, 07 Jan 2025 11:06:48 +0000

A Fleeting Glimpse of Smooth Sailing

Sometimes, everything looks perfect on the surface. The Node.js app is snappy, logs are going exactly where you want them, and the performance metrics appear respectable—even under a healthy testing load. That was how my recent experiment began. I hooked up Winston to handle logs, fired up k6 to generate traffic, and kicked back to see if everything would hold together.

The Winston + MySQL Puzzle

Initial Performance Numbers

About a 5% dip in app performance under load
Over 350,000 requests successfully processed

Hidden Problem

Only ~70,000 logs made it into the MySQL database
The rest were backed up in Node.js streams, waiting to be inserted

Why It’s Terrifying

A simple crash could wipe out tens of thousands of logs still stuck in memory
Critical debugging info would vanish, leaving developers in the dark

What This Means in Practice

Even though things looked okay on the front end, Winston-MySQL couldn’t keep up with the log volume. Winston+MySQL, by default, was processing logs one by one through a Node.js stream. Over time, the queue just ballooned, and if anything went wrong, a huge chunk of important log data could simply be lost.

Winston + CloudWatch Chaos

Decent Early Results

Around a 15% performance hit during moderate loads
Logs arrived consistently in CloudWatch at first

The Domino Effect

Under heavier traffic, log processing consumed too many resources
The logging setup itself crashed, taking the entire node.js service down

Why It’s Risky

A logging tool that can destabilize the core app is a serious flag
Production environments need stability—tolerating a logging-induced crash is out of the question

Lessons Learned

CloudWatch felt fine at lower volumes but quickly turned into a showstopper under heavier traffic. Instead of giving us reliable insights, it caused resource contention that brought the house down.

Rethinking the Logging Approach

Both Winston-MySQL and Winston-CloudWatch have their places, but these examples highlight how important it is to choose the right logging strategy for your specific load. Your logs should be fast, reliable, and never the source of crashes or major bottlenecks.

No one wants to stay up at night worrying about incomplete logs or random crashes. Logging might not be the flashiest part of the application, but it’s one of the most critical. And when it’s done poorly, it can have a bigger impact on performance and reliability than almost anything else in your stack.

The Logging Tool You Need for Logs, Errors, and Collaboration in Node.js

Rishi Kumar — Mon, 30 Dec 2024 07:18:07 +0000

Errsole is not just a logging library but an enhanced logger with built-in visualization and error management features. It stands out by combining logging, error notifications, and a web-based dashboard for managing and analyzing logs efficiently.

Why Use Errsole.js

Single-Module Simplicity:
- Errsole.js combines logging, error alert, and visualization into one package, reducing the need for multiple tools.
Web Dashboard:
- A built-in web dashboard allows you to view, filter, and search logs without third-party tools.
- Features like team management and secure access make it ideal for collaborative debugging.
Flexible Storage:
- Offers support for SQLite, MySQL, PostgreSQL, and MongoDB for storing logs.
- You can customize log retention policies, ensuring suitability for both local development and production environments.
Critical Error Notifications:
- Alerts for critical errors, sent through channels like email or Slack, help in reducing response times.
High Performance:
- Benchmarks show it handles 70,000-90,000 more requests per minute compared to traditional setups like Elasticsearch and CloudWatch.
Custom Logging Features:
- Enhanced logging capabilities include custom levels (e.g., error, info, debug) and metadata support for better context in logs.
Integration with Popular Tools:
- Works seamlessly with tools like Winston and offers an easy setup for integrating existing logging infrastructure.
Open Source and Free:
- Fully open-source with no licensing costs.

Errsole.js vs Other Libraries

Feature	Errsole.js	Winston	Pino
Built-in Dashboard	Yes	No (needs external tools)	No (needs external tools)
Error Notifications	Yes	No	No
Custom Logging	Advanced (metadata, levels)	Yes	Yes
Storage Options	Multiple DBs, SQLite, MongoDB	Custom	Custom
Performance	High	High	High
Integration	Built-in (e.g., Winston)	Extensive (setup required)	Setup required
Team Management	Yes	No	No

When to Use Errsole.js

Small Teams: Perfect for teams needing an all-in-one solution without setting up complex infrastructures.
High-Performance Apps: Ideal for handling high traffic while maintaining logging efficiency.
Collaborative Debugging: Offers team-specific access and metadata-rich logs for shared troubleshooting.
Production Monitoring: Combines real-time error notifications with robust storage options.

How to Get Started with Errsole.js

Installation:

npm install errsole

Setup:

const errsole = require('errsole');
const ErrsoleSQLite = require('errsole-sqlite');

errsole.initialize({
  storage: new ErrsoleSQLite('/path/to/logs.sqlite'),
});

console.log('Logging with Errsole.js!');

Access the Dashboard:

Open your browser and visit http://localhost:8001/ or your configured server URL.

Custom Logging:

errsole.log('info', 'Application started successfully');
errsole.meta({ userId: 1234 }).error('User action failed', { action: 'delete' });

In production environments, where centralized log storage is critical, Errsole offers multiple storage options to fit your needs:

Errsole with MongoDB
Errsole with MySQL
Errsole with PostgreSQL

Conclusion

Errsole.js is not just a logging library; it’s a comprehensive solution for error management and log visualization. It is particularly beneficial for modern Node.js applications where performance, collaboration, and reliability are key.

Building a Simple Email-Open Tracking System for Your Gmail

Rishi Kumar — Fri, 27 Dec 2024 13:21:04 +0000

In today’s email landscape, marketing platforms typically bundle “open” or “read” tracking by default. But what if you want a personal tracking pixel system—one where you can see open logs for certain one-off emails you send from Gmail?

GitHub Link : Mail Tracker

Roll your own Node.js + Express + EJS + SQLite app:

Generates 1×1 pixels tied to unique IDs,
Logs open events (date/time, IP, user agent),
Serves a real .png file so Gmail/other clients see a standard image.

You’ll get a lightweight dashboard showing which pixels were created, plus the logs of open events. While this won't be 100% foolproof (due to image blocking or proxies like GoogleImageProxy), it’s a fun, educational way to see basic open-tracking in action for your personal emails.

1. Prerequisites

Node.js installed locally.
Some familiarity with Express (for routing).
A willingness to tinker with EJS (template engine) and SQLite for storing data.

Tip: If you want other people (external recipients) to open your pixel, your server must be accessible to the outside world. Hosting on a platform like Render, Heroku, or using a tunnel service (e.g. ngrok or localtunnel) can help.

2. Core Concepts

2.1. Tracking Pixel

A “tracking pixel” is just a hidden image in your email. If the recipient’s client loads images, your server sees an HTTP request for that 1×1 PNG (or GIF).

2.2. Logging Opens

When someone (or some proxy) fetches /tracker/<pixel_id>.png, you record:

Timestamp (when it was fetched),
IP address,
User agent (might show GoogleImageProxy for Gmail, indicating the real user’s IP is masked).

2.3. Gmail & Image Proxies

Modern Gmail proxies images (served from ggpht.com or similar). This means:

You won’t see the recipient’s real IP address.
Gmail often fetches the pixel once and caches it, so subsequent opens might not appear as a new log.

Despite these limitations, you can still see if/when the image was first fetched—often close to when the user opened your email for the first time.

3. Our Simple Node App

3.1. Project Setup

Create a folder (say mail-tracker), then:

cd mail-tracker
npm init -y
npm install express ejs sqlite3 uuid

You’ll end up with a package.json that references these dependencies. Next:

Create a public/images folder with a 1×1 transparent PNG named pixel.png.
Create two EJS files in a views folder: index.ejs (dashboard) and logs.ejs (show logs).

3.2. `app.js` (Main Server File)

Below is a simplified excerpt. It:

Stores pixel data in SQLite (mail-tracker.db).
Serves static files (so pixel.png can be loaded if we want).
Has a /tracker/:id.png route that logs an open and sends back the real pixel.png.

const express = require('express');
const path = require('path');
const { v4: uuidv4 } = require('uuid');
const sqlite3 = require('sqlite3').verbose();

const app = express();
app.set('view engine', 'ejs');
app.use(express.urlencoded({ extended: false }));

// Serve static files from /public
app.use(express.static(path.join(__dirname, 'public')));

// Initialize SQLite
const db = new sqlite3.Database(path.join(__dirname, 'mail-tracker.db'), (err) => {
  // Create tables if not existing
});

// Middleware to get baseUrl for EJS
app.use((req, res, next) => {
  const protocol = req.protocol;
  const host = req.get('host');
  res.locals.baseUrl = `${protocol}://${host}`;
  next();
});

// Dashboard: list all pixels
app.get('/', (req, res) => {
  // SELECT * FROM pixels ...
  res.render('index', { pixels });
});

// Create pixel
app.post('/create', (req, res) => {
  const pixelId = uuidv4();
  // Insert pixel into DB
  // redirect to '/'
});

// The tracker route
app.get('/tracker/:id.png', (req, res) => {
  // Check pixel ID, log open in DB
  // Serve the real 'pixel.png'
  res.sendFile(path.join(__dirname, 'public', 'images', 'pixel.png'));
});

// View logs
app.get('/logs/:id', (req, res) => {
  // SELECT logs from DB for that pixel
  res.render('logs', { pixel, logs });
});

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Listening on ${PORT}`));

3.3. `index.ejs` (Dashboard)

Shows a form to create a new pixel plus a table listing existing ones:

<!DOCTYPE html>
<html>
  <head>
    <title>Mail Tracker - Dashboard</title>
    <meta charset="UTF-8" />
  </head>
  <body>
    <h1>Mail Tracker - Dashboard</h1>

    <h2>Create a New Tracking Pixel</h2>
    <form action="/create" method="POST">
      <label>Pixel Name (optional):</label>
      <input type="text" name="name" />
      <button type="submit">Create Pixel</button>
    </form>

    <hr />

    <h2>Existing Pixels</h2>
    <!-- For each pixel, show tracker URL like: baseUrl/tracker/PIXEL_ID.png -->
    <!-- Link to logs page to see open events -->
  </body>
</html>

3.4. `logs.ejs` (Show Logs)

Lists each open event (time, IP, user agent). We can format the time, group rapid logs with color, etc.:

<!DOCTYPE html>
<html>
  <head>
    <title>Mail Tracker - Logs</title>
    <meta charset="UTF-8" />
  </head>
  <body>
    <h1>Logs for <%= pixel.name %></h1>
    <p>Created At: <%= pixel.createdAt %></p>

    <h2>Open Events</h2>
    <p style="font-style: italic;">
      You may see extra logs if bots (for example, GoogleImageProxy via ggpht.com) or the email client repeatedly load the image.
      Check each log’s timestamp to distinguish real user opens from proxy fetches.
    </p>

    <table>
      <thead>
        <tr><th>Time</th><th>IP</th><th>User-Agent</th></tr>
      </thead>
      <tbody>
        <% // For each log, show the local-time date, IP, userAgent, etc. %>
      </tbody>
    </table>
  </body>
</html>

4. Embedding into Gmail

4.1. Copy the Tracker URL

Once your server is running publicly, your pixel has a URL like:

https://myapp.example.com/tracker/1234abcd-....png

Or, if you’re using a local tunnel:

https://random.loca.lt/tracker/1234abcd-....png

4.2. Insert Photo by URL

Compose a new email in Gmail.
Click Insert photo → Web Address (URL).
Paste the Tracker URL.
If Gmail says “We can’t find or access that image,” you can still insert and send.
When the recipient opens (and images are enabled), Gmail loads (or caches) that image, and your logs record an event.

Note: Because Gmail proxies images, the request will likely come from Google servers (e.g., GoogleImageProxy/ggpht.com). You won’t get the real recipient’s IP. But you do see when the pixel was fetched—often correlating with a user open.

5. Understanding “Extra Logs” & Proxy Behavior

Multiple Logs: If Gmail or another client refreshes or re-fetches the image, or if the user reopens the message, you’ll see multiple entries. Some might be within seconds of each other, triggered by background processes or spam filters.
User Agent: You might see GoogleImageProxy instead of a real browser’s user agent. This is normal for Gmail. Other clients, like Apple Mail or Outlook, might show more direct info.
IP: Because of proxies, you’ll usually see a Google IP range, not the actual recipient’s IP. That’s a privacy measure.

6. Limitations & Future Enhancements

Image-Blocking: If the recipient’s mail client blocks images by default, your pixel is never loaded. So you won’t see an open event.
Caching: Gmail especially caches images. Subsequent opens may not trigger a new request.
No IP / Location: With proxies, you don’t see the real user’s IP or location.
Unique Query Params: If you want to track individual recipients, create a separate pixel for each person or append query strings like ?user=john@example.com.

7. Why Build a Personal Tracker?

Educational: Learn how open tracking works under the hood.
Privacy: You control your own data, rather than trusting a third-party marketing provider.
Debugging: If you want to see if a friend/client is reading your email, or simply confirm some one-time outreach was opened.

(Always be mindful of legal or ethical constraints in your region and among your contacts.)

8. Conclusion

Running your own open-tracking service can be a fun side project—especially to see how Gmail or other providers handle images. You’ll quickly discover that GoogleImageProxy can mask real IP addresses, and not every open is captured if images are off. Nonetheless, for personal use, it’s a neat way to see open events in real time.

Key Steps Recap:

Build a Node.js + Express server.
Generate unique IDs/pixels.
Log requests to /tracker/:id.png.
Serve a legitimate .png file.
Embed the pixel in your Gmail messages.
Check your logs for open events (and interpret carefully).

That’s it—your own personal email tracking system. Once you see it in action, you’ll have a deeper appreciation for how major mailing platforms do open tracking at scale and why they run into the same limitations of caching and image-blocking.

GitHub Link : Mail Tracker

DEV Community: Rishi Kumar

Breakdown: Who is Responsible?

The Breakdown: Who is Responsible?

1. Database Queries (~60% of Performance Issues)

2. Network Calls (~30% of Performance Issues)

3. Algorithms / Code Logic (~10% of Performance Issues)

The "Scale of Latency" (Why Algo loses)

When does the Algorithm become #1?

Summary

Ultimate Logging Guide for Node.js, Bun, and Next.js SSR in 2025

1. Built-in Log Viewer That Runs in Your App

2. Instant Setup with Built-in Console Capture

3. Blazing Fast Performance (Benchmarks Don't Lie)

4. Powerful Advanced Logging Features

5. Storage Options for Every Use Case

_

6. Runs Anywhere — Local, Server, Docker, Cloud

7. Works with Your Favorite Frameworks

8. Safer Than Remote Logging

9. Real-World Proof & Developer Stories

10. Open Source, Free Forever

Final Thoughts

Errsole Live on Product Hunt

Worker Threads in Node.js and Their Impact on Logging Performance

What Are Worker Threads?

How Worker Threads Affect System Performance

On Single-Core Systems

On Multi-Core Systems Fully Occupied by Node.js Processes

Worker Threads in Logging

When Worker Threads May Not Offer Benefits

Why Paid Logging Services Like BetterStack Can Hurt Your Throughput: Node.js

Paid Logging Services and Their Drawbacks

The Benefits of Open-Source Logging Solutions

Final Word

The Magical World of Garbage Collection in Node.js

What’s Garbage Collection Anyway?

The Secret Life of GC in Node.js (Powered by V8)

1. Young vs. Old Generation: The Teenage Rebels and Seasoned Veterans

2. Incremental & Concurrent GC: Cleaning Without the Drama

How to Peek Behind the Curtain: Detecting GC in Action

1. Turn on the GC Radar with Runtime Flags

2. Use Chrome DevTools for a Closer Look

3. Bring in the Heavy Machinery: Profiling Tools

Real-World Scenarios: When GC Can Trip You Up

Memory Leaks: The Unwanted Houseguests

High GC Pauses: When the Cleaning Crew Overdoes It

GC Overhead in High-Throughput Systems

Embrace the Magic!

Bonus: Extra Magical Tidbits for the Adventurous Developer

Dive Deeper into the V8 Engine

Unleash the Power of Profiling Tools

Modern Spells: WeakRef and FinalizationRegistry

Watch Out for Memory Gremlins

Experiment, Benchmark, and Learn

Stream and HighWatermark: Node.js

Table of Contents

Understanding highWaterMark

What is highWaterMark?

Default Values

Why is highWaterMark Important?

Understanding objectMode

What is objectMode?

Enabling objectMode

Benefits of objectMode

Interaction Between objectMode and highWaterMark

How They Work Together

Implications

Does objectMode Improve highWaterMark?

Clarifying "Improve"

How objectMode Affects highWaterMark

Does It "Improve" highWaterMark?

Summary

Best Practices

Practical Example

Object Mode vs. Binary Mode with highWaterMark

Conclusion

Final Tips

Log Viewer for Next.js Server Side: Node.js

1. Why Server-Side Logging Is Important

2. Install Errsole and Errsole-SQLite

Understanding `highWaterMark`

What is `highWaterMark`?

Why is `highWaterMark` Important?

Understanding `objectMode`

What is `objectMode`?

Enabling `objectMode`

Benefits of `objectMode`

Interaction Between `objectMode` and `highWaterMark`

Does `objectMode` Improve `highWaterMark`?

How `objectMode` Affects `highWaterMark`

Does It "Improve" `highWaterMark`?

Object Mode vs. Binary Mode with `highWaterMark`

3. Create a Logger File (`logger.js`)