DEV Community: 〽️ 𝙍𝙤𝙨𝙝𝙖𝙣

I Rewrote My Node.js/Express Backend in NestJS

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sun, 09 Aug 2026 16:05:08 +0000

I am currently working on a project where I have WebSockets, UDP and REST APIs, and although the V1 was almost ready and everything was working, the code had started to feel all over the place.

I didn't always understand where everything was, especially because you don't usually see many projects where multiple protocols like REST, WebSockets and UDP are being used together.

I wanted to reach a point where I could look at the code and immediately understand where things belong.

At the same time, there was a real need to add a new SUPER_ADMIN role and change some of the existing permission levels. So instead of continuing to patch the existing Express application, I decided to rewrite the project in NestJS.

The difference was visible in the structure

In V1, the backend had separate folders for HTTP, services, database, WebSockets and other utilities:

Nothing here is necessarily wrong, but as the project grew, following one feature could mean jumping between several different folders.

With NestJS, I moved toward a more feature-oriented structure:

That small change made a big difference for me. I could look at the project and have a much better idea of where something belongs.

Starting with the database

The first thing I worked on was the database.

I redesigned parts of the schema and moved the main database access to Prisma. What I liked about Prisma was that the schema became much easier to understand and the database models were connected more naturally with TypeScript.

model User {
  id    String   @id @default(uuid())
  email String   @unique
  role  UserRole
}

I didn't completely remove raw SQL either. There are still places where SQL makes more sense, especially when I need database-specific functionality or more direct control.

Rebuilding the APIs

After that, I started rewriting the APIs.

In the old Express version, it was very easy for routes to slowly become responsible for validation, business logic, database calls and responses all in one place.

With NestJS, I started separating these responsibilities:

@Controller("users")
export class UsersController {
  constructor(private readonly usersService: UsersService) {}

  @Post()
  create(@Body() dto: CreateUserDto) {
    return this.usersService.create(dto);
  }
}

The controller handles the request, the service handles the logic, and the database layer handles persistence.

NestJS's dependency injection also meant I could declare what a class needs instead of manually creating and passing everything around.

Authorization

Then I rebuilt the authorization because of the new SUPER_ADMIN role and the permission changes.

NestJS's guards and decorators made it possible to make permissions much more visible:

@Roles("ADMIN")
@Delete(":id")
remove(@Param("id") id: string) {
  return this.usersService.remove(id);
}

I could look at a route and understand who is allowed to access it without searching through different parts of the application.

WebSockets

Next came the WebSocket side.

NestJS provides WebSocket gateway support with Socket.IO, so I could keep the real-time communication in its own place.

For example, rooms could be handled inside the gateway:

@WebSocketGateway()
export class RealtimeGateway {
  @WebSocketServer()
  server: Server;

  @SubscribeMessage("join-room")
  joinRoom(
    @ConnectedSocket() client: Socket,
    @MessageBody() data: { room: string },
  ) {
    client.join(data.room);
  }
}

Now the real-time communication has a clear place, while the actual business logic can stay in services.

What NestJS gave me was a clear place to isolate it from the rest of the application and manage its lifecycle.

So the mental model became:

REST       → Controllers → Services → Database
WebSocket  → Gateway → Rooms → Services
UDP        → Transport → Services

I also used Bruno while rebuilding the APIs. I found it to be a great alternative to Postman, especially for keeping API requests and collections close to the project while rebuilding and testing the endpoints.

The biggest benefit

The biggest benefit I got from the rewrite wasn't performance.

It was that I could finally understand the project in layers.

I could look at the code and know where the API lives, where the business logic lives, where WebSockets are handled, where UDP is handled, and where permissions belong.

And this is probably what I like most about opinionated frameworks.

We don't have to decide between 10 or 20 different ways of structuring everything. NestJS gives us conventions, and we can follow them.

That doesn't mean NestJS is universally better than Express. Express was perfectly capable of running my original project.

For me, the rewrite was about making the architecture predictable and easier to understand.

Sometimes the best reason to rewrite something isn't that the old version doesn't work.

It's that you want the next version to be easier to understand.

What I Learned from Reading JavaScript: The Good Parts

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Wed, 05 Aug 2026 18:13:07 +0000

What I Learned from Reading JavaScript: The Good Parts

I recently finished reading JavaScript: The Good Parts by Douglas Crockford.

Even though the book was written in 2008, and javascript is much more capable now i really wanted to understand how things evolved with time and as engineer whose most time has been writing web development i really needed to know what's good and bad about the language i use

Here are the biggest lessons I took away.

1. Functions are first-class values

This was probably the biggest mindset shift.

Functions aren't special syntax—they're values.

You can:

store them in variables
pass them to other functions
return them from functions
create functions that create other functions

This idea explains callbacks, higher-order functions, currying, modules, and closures.

2. Closures are one of JavaScript's superpowers

An inner function remembers variables from its outer scope even after the outer function has finished executing.

Once this clicked, concepts like private state, factories, memoization, and modules became much easier to understand.

3. Objects inherit through prototypes

Objects don't inherit from classes.

Every object has an internal prototype link, and property lookup walks up the prototype chain until it finds a match.

Modern JavaScript gives us tools like:

Object.create()
Object.getPrototypeOf()

to work with prototypes directly.

4. `this` isn't determined by where a function is written

It's determined by how the function is called.

That single rule explains most of the confusing behavior around this.

5. JavaScript has some dangerous features

Some language features are better avoided:

eval()
with
wrapper objects like new Boolean() or new String()
implicit global variables
== when === is what you actually want

Just because something exists doesn't mean it should be used.

6. JavaScript was designed with some historical baggage

Understanding these quirks makes many "weird" behaviors less mysterious.

Examples:

typeof null === "object"
numbers are IEEE-754 floating point values
floating-point precision issues (0.1 + 0.2)
strings are UTF-16, so emojis can occupy two code units
NaN is contagious

7. Readability matters more than cleverness

One quote that stuck with me was the idea that:

If you can't easily read the program, you can't confidently predict what it will do.

Many of Crockford's recommendations ultimately come down to writing code that's easy for humans to understand.

8. Simplicity beats feature count

One analogy from the book that I liked:

My microwave has dozens of buttons, but I only ever use Cook and Clock.

The point wasn't about microwaves—it was about software design.

A language or API doesn't become better by adding more features.

It becomes better when the core features are well designed.

9. JavaScript is much simpler than it first appears

The language feels huge at first.

But most of it is built on a few fundamental ideas:

objects
functions
prototypes
closures
scope

Once those concepts make sense, many "advanced" topics feel much more approachable.

Things that are outdated today

The book is almost two decades old, so I wouldn't follow everything literally.

Today we'd use:

let / const instead of var
ESLint instead of JSLint
JSON.parse() instead of custom parsers
modules instead of IIFEs for encapsulation
Map and Set where appropriate

But the principles behind those recommendations are still valuable.

My biggest takeaway

I started reading this book expecting to learn more JavaScript syntax.

Instead, I finished it with a better understanding of why JavaScript behaves the way it does.

And that, more than any specific API or language feature, has already made me a better JavaScript developer.

I think this makes a stronger LinkedIn post because it focuses on insights rather than trying to cover every technical detail. People are more likely to engage with "here's how this book changed my understanding" than with a long list of language features.

Float64Array, TimSort and sort

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Wed, 05 Aug 2026 17:58:26 +0000

So in JavaScript, there are multiple ways to sort data. The native JavaScript Array.sort() uses an algorithm from the TimSort family. Modern V8 engines use PowerSort, which is an optimized version of TimSort.

I wanted to understand how different sorting approaches perform, so I compared them.

The interesting part is that there is no single "fastest" sorting algorithm — it depends on the type of data and the use case.

⚡ Float64Array.sort() — Fastest for pure numbers

When the data is only numbers, Float64Array.sort() performs extremely well.

new Float64Array(data).sort()

The reason it is faster is because:

It does not need a comparator function
The engine already knows every value is a number
Numbers are stored in a continuous block of memory
No extra type checks or object handling are needed

This allows the engine to directly work with the raw numeric representation in memory.

However, it only works for numeric data and cannot handle objects or custom sorting rules.

⚡ TimSort / PowerSort — Better for large and complex inputs

TimSort combines:

Insertion sort for small sections
Merge sort for efficiently combining sorted sections

It detects already sorted parts of the array (called runs) and uses that information to optimize sorting.

This makes it very effective for:

Large datasets
Partially sorted data
Real-world applications where data already has some order

For larger and more complex inputs, TimSort can perform better than simpler sorting approaches because it adapts to the structure of the data.

⚡ TimSort npm package

Even though it uses the same general algorithm, the npm implementation is usually slower because it runs completely in JavaScript and does not get the same low-level optimizations as the built-in engine.

The biggest lesson:

The fastest sorting method depends on the situation:

🔹 Pure numbers → Float64Array.sort() is usually the fastest
🔹 Large real-world datasets → TimSort/PowerSort is highly optimized
🔹 Custom objects and sorting rules → Array.sort() with a comparator

Performance is not only about the algorithm. It also depends on:

Memory layout
Data types
Native optimizations
Function call overhead

Understanding both the algorithm and the environment where it runs is what helps us write faster code. 🚀

Debugging Production Alerts Without Chasing The Wrong Problem

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sun, 17 May 2026 14:38:20 +0000

oday I worked through a production alert that looked simple at first:

“Services are unhealthy. Memory issue?”

But once we started digging, it turned out to be a mix of real issues, noisy errors, and one confusing timezone mismatch.

The First Assumption

The alert mentioned memory problems, worker restarts, database warnings, and a third-party integration error.

At first, it was tempting to group everything together and say:

“The system is running out of memory.”

But that would have been too broad.

So instead, we split the investigation by service.

What Was Actually Happening

One web service really was hitting memory pressure. Its process manager was killing workers after the total memory usage crossed the configured limit.

That part was real.

But another background worker was reported as “crash-looping.” When we checked the logs, it did not show the usual signs of a memory crash.

There were no lines like:

Process too large
Out of memory
Children dying rapidly
Exited with status 255

Instead, the logs showed graceful shutdowns and normal restarts:

Shutting down
Scheduler exiting
Bye
Booted application
Starting memory monitoring

That changed the conclusion. The worker may have restarted, but the logs did not prove it was crashing because of memory.

The Timezone Trap

Another confusing part was time.

One tool showed local time. Another exported logs in UTC.

So a window that looked like this in the UI:

20:30 - 21:05

was actually this in the logs:

16:30 - 17:05 UTC

That small difference can completely change an investigation.

If you query the wrong time window, you can easily miss the real event or accidentally blame the wrong one.

Other Noise

There were also database warnings. They were real, but they were related to a small set of records and not directly tied to the memory alert.

There was also a recurring third-party SMS error. It looked scary in the alert, but it was old background noise and not part of the main incident.

The Final Picture

After separating the signals, the situation looked more like this:

Web memory pressure: real
Background worker memory crash-loop: not proven
Database warnings: real, but separate
Third-party SMS error: noisy, unrelated
Timezone mismatch: caused confusion

Takeaway

Production debugging is not just about finding errors.

It is about separating related signals from unrelated ones.

Before acting on an alert, it helps to ask:

Which service actually produced this log?
Is this the correct time window?
Is the timestamp local time or UTC?
Do we see the real failure signature?
Is this new, or just recurring noise?

The biggest lesson from today:

Don’t fix the loudest symptom. First prove what is actually failing.

Bulk Emails from a chat input — without Redis, queues, or worker services

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sun, 17 May 2026 14:37:16 +0000

I built a feature into my portfolio site that lets me paste a list of recruiter emails and a job description directly into a chat box.

Once I type my unlock passphrase, it automatically:

generates a tailored email for each recruiter,
attaches my resume,
sends the emails,
and streams live progress back into the same chat bubble.

The fun part is the architecture behind it.

There’s no Redis.
No BullMQ.
No QStash.
No separate worker service running somewhere.

It’s just:

Next.js on Vercel
Neon Postgres
and after() from Next.js 15.

That’s it.

The idea

I originally started building it because I was tired of manually rewriting the same outreach emails over and over again while applying to jobs.

Most “AI job application tools” feel overly automated and spammy, so I wanted something smaller and more controlled:

I still choose the recruiters
I still provide the job description
I still manually trigger it
but the repetitive work disappears

The end result feels more like an assistant inside my portfolio chat than a mass-email bot.

Triggering the bulk flow

The detection logic lives directly inside the chat route.

If the message contains multiple email addresses and matches my existing outreach heuristic, the app switches into “bulk pipeline” mode.

const rows = buildRowsFromText(trimmed);

if (rows.length >= 2 && hasCurrentOutreachContext(trimmed)) {
  if (!authed) return lockResponse();

  const { runId } = await startBulkPipeline({
    rows,
    jobDescription: trimmed,
  });

  after(() => drainRun(runId));

  return NextResponse.json({
    reply: "📤 Sending applications...",
    pipelineRunId: runId,
  });
}

The important bit here is after().

In Next.js 15+, after() allows work to continue after the response has already been sent back to the client.

So the user instantly gets a response with a pipelineRunId, while the actual email processing continues in the same serverless invocation.

No worker queues.
No background containers.
No separate infrastructure.

Just the existing runtime continuing execution after the response flushes.

Honestly, this was the feature that made the whole architecture click for me.

Storage layer

I used two tables in Neon:

`pipeline_runs`

Stores batch-level state:

status
total jobs
sent count
failed count
original job description

`pipeline_jobs`

Stores one row per recipient:

email
derived company name
status
attempts
error messages

The processing loop atomically claims one queued job at a time:

UPDATE pipeline_jobs
   SET status = 'sending',
       attempts = attempts + 1
 WHERE id = (
   SELECT id
     FROM pipeline_jobs
    WHERE run_id = $1
      AND status = 'queued'
 ORDER BY created_at ASC
    LIMIT 1
 )
RETURNING *;

This makes retries safe and keeps the pipeline idempotent even if Vercel retries the invocation.

Generating the emails

For every claimed row, the flow is:

Extract the company name from the email domain
(hiring@stripe.com → “Stripe”)
Call Groq using llama-3.3-70b-versatile in structured JSON mode
Generate:

{
  "subject": "...",
  "body": "..."
}

Convert the plain text body into proper HTML:

paragraphs
bullet lists
clickable links
readable spacing

Send through Gmail SMTP using Nodemailer
Mark the row as sent

I also intentionally made the fallback behavior conservative.

If someone uses a Gmail or generic email address, the system falls back to “Hiring Team” instead of hallucinating fake company names.

Small detail, but it makes the emails feel way more natural.

Live progress in the chat UI

The response includes a pipelineRunId.

The frontend attaches that ID directly to the assistant message, and a <PipelineProgress /> component renders underneath the same chat bubble.

It polls:

GET /api/lab/pipeline/[runId]

every ~1.5 seconds until the run finishes.

I considered SSE/websockets, but honestly polling was simpler and more reliable for Vercel Hobby deployments.

Sometimes boring engineering decisions are the correct ones.

Tradeoffs

This setup definitely has limits.

Function timeout

Vercel Hobby gives ~60 seconds.

Each email takes roughly:

LLM generation
SMTP send
DB updates

About 3–4 seconds total per row.

So realistically one invocation comfortably handles ~10–15 emails.

For my use case (“apply to a few recruiters at a time”), that’s completely fine.

If I ever needed larger batches, I’d probably chunk the drain process and self-fanout recursively.

Gmail SMTP limits

Gmail caps daily sends.

Again, acceptable for personal usage.

Switching to Resend or SendGrid would basically be changing one file.

`after()` durability

This is probably the biggest tradeoff.

If the serverless invocation dies midway through processing, remaining rows simply stay in queued.

Right now there’s no recovery daemon.
No retry cron.
No dead-letter queue.

And honestly?

I haven’t needed one yet.

Why I skipped BullMQ, QStash, and Inngest

I actually started with BullMQ.

Then I remembered:

BullMQ wants Redis
Redis wants a worker process
worker processes don’t really fit Vercel well

I tried QStash too.
It worked.

But it also felt like I was introducing another service for a scale problem I didn’t actually have.

Same with Inngest.

Eventually I realized:

the runtime already exists
the database already exists
Next.js already provides after()

So I stopped overengineering it.

The entire system is roughly ~250 lines.

No orchestration layer.
No infra maze.
No queue dashboards.

Just a simple pipeline that solves the actual problem.

And honestly, that ended up being the right architecture.

Bulk Emails from a chat input — without Redis, queues, or worker services

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sat, 09 May 2026 12:44:14 +0000

I built a feature into my portfolio site that lets me paste a list of recruiter emails and a job description directly into a chat box.

Once I type my unlock passphrase, it automatically:

generates a tailored email for each recruiter,
attaches my resume,
sends the emails,
and streams live progress back into the same chat bubble.

The fun part is the architecture behind it.

There’s no Redis.
No BullMQ.
No QStash.
No separate worker service running somewhere.

It’s just:

Next.js on Vercel
Neon Postgres
and after() from Next.js 15.

That’s it.

The idea

I originally started building it because I was tired of manually rewriting the same outreach emails over and over again while applying to jobs.

Most “AI job application tools” feel overly automated and spammy, so I wanted something smaller and more controlled:

I still choose the recruiters
I still provide the job description
I still manually trigger it
but the repetitive work disappears

The end result feels more like an assistant inside my portfolio chat than a mass-email bot.

Triggering the bulk flow

The detection logic lives directly inside the chat route.

If the message contains multiple email addresses and matches my existing outreach heuristic, the app switches into “bulk pipeline” mode.

const rows = buildRowsFromText(trimmed);

if (rows.length >= 2 && hasCurrentOutreachContext(trimmed)) {
  if (!authed) return lockResponse();

  const { runId } = await startBulkPipeline({
    rows,
    jobDescription: trimmed,
  });

  after(() => drainRun(runId));

  return NextResponse.json({
    reply: "📤 Sending applications...",
    pipelineRunId: runId,
  });
}

The important bit here is after().

In Next.js 15+, after() allows work to continue after the response has already been sent back to the client.

So the user instantly gets a response with a pipelineRunId, while the actual email processing continues in the same serverless invocation.

No worker queues.
No background containers.
No separate infrastructure.

Just the existing runtime continuing execution after the response flushes.

Honestly, this was the feature that made the whole architecture click for me.

Storage layer

I used two tables in Neon:

`pipeline_runs`

Stores batch-level state:

status
total jobs
sent count
failed count
original job description

`pipeline_jobs`

Stores one row per recipient:

email
derived company name
status
attempts
error messages

The processing loop atomically claims one queued job at a time:

UPDATE pipeline_jobs
   SET status = 'sending',
       attempts = attempts + 1
 WHERE id = (
   SELECT id
     FROM pipeline_jobs
    WHERE run_id = $1
      AND status = 'queued'
 ORDER BY created_at ASC
    LIMIT 1
 )
RETURNING *;

This makes retries safe and keeps the pipeline idempotent even if Vercel retries the invocation.

Generating the emails

For every claimed row, the flow is:

Extract the company name from the email domain
(hiring@stripe.com → “Stripe”)
Call Groq using llama-3.3-70b-versatile in structured JSON mode
Generate:

{
  "subject": "...",
  "body": "..."
}

Convert the plain text body into proper HTML:

paragraphs
bullet lists
clickable links
readable spacing

Send through Gmail SMTP using Nodemailer
Mark the row as sent

I also intentionally made the fallback behavior conservative.

If someone uses a Gmail or generic email address, the system falls back to “Hiring Team” instead of hallucinating fake company names.

Small detail, but it makes the emails feel way more natural.

Live progress in the chat UI

The response includes a pipelineRunId.

The frontend attaches that ID directly to the assistant message, and a <PipelineProgress /> component renders underneath the same chat bubble.

It polls:

GET /api/lab/pipeline/[runId]

every ~1.5 seconds until the run finishes.

I considered SSE/websockets, but honestly polling was simpler and more reliable for Vercel Hobby deployments.

Sometimes boring engineering decisions are the correct ones.

Tradeoffs

This setup definitely has limits.

Function timeout

Vercel Hobby gives ~60 seconds.

Each email takes roughly:

LLM generation
SMTP send
DB updates

About 3–4 seconds total per row.

So realistically one invocation comfortably handles ~10–15 emails.

For my use case (“apply to a few recruiters at a time”), that’s completely fine.

If I ever needed larger batches, I’d probably chunk the drain process and self-fanout recursively.

Gmail SMTP limits

Gmail caps daily sends.

Again, acceptable for personal usage.

Switching to Resend or SendGrid would basically be changing one file.

`after()` durability

This is probably the biggest tradeoff.

If the serverless invocation dies midway through processing, remaining rows simply stay in queued.

Right now there’s no recovery daemon.
No retry cron.
No dead-letter queue.

And honestly?

I haven’t needed one yet.

Why I skipped BullMQ, QStash, and Inngest

I actually started with BullMQ.

Then I remembered:

BullMQ wants Redis
Redis wants a worker process
worker processes don’t really fit Vercel well

I tried QStash too.
It worked.

But it also felt like I was introducing another service for a scale problem I didn’t actually have.

Same with Inngest.

Eventually I realized:

the runtime already exists
the database already exists
Next.js already provides after()

So I stopped overengineering it.

The entire system is roughly ~250 lines.

No orchestration layer.
No infra maze.
No queue dashboards.

Just a simple pipeline that solves the actual problem.

And honestly, that ended up being the right architecture.

Bulk Emails from a chat input — without Redis, queues, or worker services

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sat, 09 May 2026 12:44:14 +0000

I built a feature into my portfolio site that lets me paste a list of recruiter emails and a job description directly into a chat box.

Once I type my unlock passphrase, it automatically:

generates a tailored email for each recruiter,
attaches my resume,
sends the emails,
and streams live progress back into the same chat bubble.

The fun part is the architecture behind it.

There’s no Redis.
No BullMQ.
No QStash.
No separate worker service running somewhere.

It’s just:

Next.js on Vercel
Neon Postgres
and after() from Next.js 15.

That’s it.

The idea

I originally started building it because I was tired of manually rewriting the same outreach emails over and over again while applying to jobs.

Most “AI job application tools” feel overly automated and spammy, so I wanted something smaller and more controlled:

I still choose the recruiters
I still provide the job description
I still manually trigger it
but the repetitive work disappears

The end result feels more like an assistant inside my portfolio chat than a mass-email bot.

Triggering the bulk flow

The detection logic lives directly inside the chat route.

If the message contains multiple email addresses and matches my existing outreach heuristic, the app switches into “bulk pipeline” mode.

const rows = buildRowsFromText(trimmed);

if (rows.length >= 2 && hasCurrentOutreachContext(trimmed)) {
  if (!authed) return lockResponse();

  const { runId } = await startBulkPipeline({
    rows,
    jobDescription: trimmed,
  });

  after(() => drainRun(runId));

  return NextResponse.json({
    reply: "📤 Sending applications...",
    pipelineRunId: runId,
  });
}

The important bit here is after().

In Next.js 15+, after() allows work to continue after the response has already been sent back to the client.

So the user instantly gets a response with a pipelineRunId, while the actual email processing continues in the same serverless invocation.

No worker queues.
No background containers.
No separate infrastructure.

Just the existing runtime continuing execution after the response flushes.

Honestly, this was the feature that made the whole architecture click for me.

Storage layer

I used two tables in Neon:

`pipeline_runs`

Stores batch-level state:

status
total jobs
sent count
failed count
original job description

`pipeline_jobs`

Stores one row per recipient:

email
derived company name
status
attempts
error messages

The processing loop atomically claims one queued job at a time:

UPDATE pipeline_jobs
   SET status = 'sending',
       attempts = attempts + 1
 WHERE id = (
   SELECT id
     FROM pipeline_jobs
    WHERE run_id = $1
      AND status = 'queued'
 ORDER BY created_at ASC
    LIMIT 1
 )
RETURNING *;

This makes retries safe and keeps the pipeline idempotent even if Vercel retries the invocation.

Generating the emails

For every claimed row, the flow is:

Extract the company name from the email domain
(hiring@stripe.com → “Stripe”)
Call Groq using llama-3.3-70b-versatile in structured JSON mode
Generate:

{
  "subject": "...",
  "body": "..."
}

Convert the plain text body into proper HTML:

paragraphs
bullet lists
clickable links
readable spacing

Send through Gmail SMTP using Nodemailer
Mark the row as sent

I also intentionally made the fallback behavior conservative.

If someone uses a Gmail or generic email address, the system falls back to “Hiring Team” instead of hallucinating fake company names.

Small detail, but it makes the emails feel way more natural.

Live progress in the chat UI

The response includes a pipelineRunId.

The frontend attaches that ID directly to the assistant message, and a <PipelineProgress /> component renders underneath the same chat bubble.

It polls:

GET /api/lab/pipeline/[runId]

every ~1.5 seconds until the run finishes.

I considered SSE/websockets, but honestly polling was simpler and more reliable for Vercel Hobby deployments.

Sometimes boring engineering decisions are the correct ones.

Tradeoffs

This setup definitely has limits.

Function timeout

Vercel Hobby gives ~60 seconds.

Each email takes roughly:

LLM generation
SMTP send
DB updates

About 3–4 seconds total per row.

So realistically one invocation comfortably handles ~10–15 emails.

For my use case (“apply to a few recruiters at a time”), that’s completely fine.

If I ever needed larger batches, I’d probably chunk the drain process and self-fanout recursively.

Gmail SMTP limits

Gmail caps daily sends.

Again, acceptable for personal usage.

Switching to Resend or SendGrid would basically be changing one file.

`after()` durability

This is probably the biggest tradeoff.

If the serverless invocation dies midway through processing, remaining rows simply stay in queued.

Right now there’s no recovery daemon.
No retry cron.
No dead-letter queue.

And honestly?

I haven’t needed one yet.

Why I skipped BullMQ, QStash, and Inngest

I actually started with BullMQ.

Then I remembered:

BullMQ wants Redis
Redis wants a worker process
worker processes don’t really fit Vercel well

I tried QStash too.
It worked.

But it also felt like I was introducing another service for a scale problem I didn’t actually have.

Same with Inngest.

Eventually I realized:

the runtime already exists
the database already exists
Next.js already provides after()

So I stopped overengineering it.

The entire system is roughly ~250 lines.

No orchestration layer.
No infra maze.
No queue dashboards.

Just a simple pipeline that solves the actual problem.

And honestly, that ended up being the right architecture.

How I Built a Production Observability Stack — And Broke It Twice Before It Worked

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sun, 05 Apr 2026 13:36:40 +0000

I used to dismiss monitoring as something you bolt on after the real engineering is done. Logs were noise. Metrics were "a later problem." Alerts were for teams with dedicated SREs, not a small startup running three service types on Render.

I was wrong. Badly wrong. And it took a self-inflicted incident — where my own monitoring system became the thing that needed monitoring — to understand why observability is engineering, not afterthought.

This is a detailed account of building our observability stack from scratch: what we built, what broke, why it broke, and what the architecture looks like today.

The starting point

We run three service types on Render:

A web service — the main API and frontend server
Background workers — async job processors (queuing, retries, scheduled tasks)
Key-value stores — Render's managed Redis-compatible service

Before this project, each service logged to Render's default log drain. If something broke in production, you'd open the Render dashboard, pick a service, and scroll. No correlation across services, no metrics, no alerting, no history beyond what Render retained.

The goal was simple to state: centralize everything into a self-hosted Grafana stack, set up alerting, and gain actual visibility into what the system was doing.

The stack

Before getting into what went wrong, here's what we ended up with:

Component	Role
Caddy	Reverse proxy and automatic TLS. All external traffic enters here.
syslog-proxy	Custom Python container. Enforces token validation on inbound syslog, strips the token, forwards clean RFC 5424 to Alloy.
Vector	Disk-backed telemetry buffer between syslog-proxy and Alloy. Prevents message loss during restarts or slowdowns.
Alloy	Grafana's telemetry collector. Receives logs, applies filters and transformations, routes to the right backend.
Loki	Log storage. Stores log lines compressed and indexed by labels. Queried with LogQL.
Mimir	Metrics storage. Long-term Prometheus-compatible time-series backend.
Tempo	Trace storage. Stores distributed traces with minimal indexing — cheap to run.
Grafana	The UI. Dashboards, alert rules, and unified querying across Loki, Mimir, and Tempo.

The full data flow:

Render services
    → syslog drain (TCP, RFC 5424)
    → syslog-proxy (token validation + strip)
    → Vector (disk-backed buffer)
    → Alloy (filter, transform, route)
    → Loki (logs) / Mimir (metrics) / Tempo (traces)
    → Grafana (dashboards + alerts)
    → Datadog + BetterStack (fan-out for external alerting)

All of this runs on a single 4 GB DigitalOcean droplet, currently at ~30% total memory usage. That number matters — it means the stack is right-sized, not just "scaled until it stopped crashing."

Problem 1: No token enforcement on the log stream

Render lets you configure a syslog log drain — a TCP endpoint that receives log lines in RFC 5424 format. You give Render a URL with a token embedded in it, and Render forwards logs there.

The problem: that token is in the URL, but there's no native mechanism to validate it before your collector ingests the message. Render's outbound IP range is also shared across tenants. Any service within that range — from any organisation — can technically send data to your syslog endpoint if they know the address. And since we were pointing directly at Alloy, anything that reached the port got ingested.

This is less of an active security threat and more of a correctness and isolation problem. You want to know that the logs you're querying are your logs.

The fix: we wrote a lightweight Python container — syslog-proxy — that sits between Render and Alloy. Every inbound TCP connection goes through it first. It reads the syslog message, extracts the token from the structured data field (RFC 5424 puts auth metadata in the [SD-ID key="value"] block), validates it against a shared secret, strips it from the message, and forwards clean RFC 5424 to Alloy's syslog listener.

If the token is missing or wrong, the connection is dropped. No entry.

The key distinction: the proxy isn't adding authentication to the stream. Render was already sending a token. The proxy is the enforcement layer that was missing — the thing that actually checks it before data moves downstream.

Problem 2: Connection drops under load

After deploying the proxy, we started seeing intermittent message loss. Not every time, not obviously, but present — we'd notice gaps in log sequences that shouldn't have gaps.

The root cause was straightforward: the proxy was synchronous and had no internal buffer. When Alloy was slow to accept a connection (startup, GC pause, momentary backpressure), the proxy would drop the TCP connection rather than queue the message. Lost message, no retry, no error surfaced to the user.

The fix: we inserted Vector between syslog-proxy and Alloy.

Vector is a high-performance, Rust-based telemetry agent built specifically for this kind of pipeline work. The relevant feature here is disk-backed buffering. Messages are written to disk immediately on receipt, then forwarded to the downstream destination. If Alloy is slow, Vector queues. If Alloy restarts entirely, Vector holds the messages and delivers once the connection is re-established.

The pipeline now looks like: syslog-proxy validates and strips → hands off to Vector → Vector buffers to disk → Vector forwards to Alloy.

Connection drops stopped immediately. We have not lost a log message to transport failure since.

The incident

With the pipeline stable, we validated it end-to-end on our highest-volume service — the web API. It processed tens of thousands of log lines without issue. Alloy filtered and routed correctly. Loki ingested cleanly. Grafana showed the data. Everything looked good.

So we did what seemed logical: we added all remaining services to the syslog drain simultaneously.

Within two hours, the system was on fire.

Both Datadog and BetterStack — external services we fan logs to in parallel for alerting and long-term retention — were overwhelmed. The Datadog exporter started returning unexpected EOF from their intake API. This is Datadog actively closing connections, not just timing out. They were rejecting us.

Alert rules fired across the board. The alert channel, which was supposed to tell us when our application was unhealthy, was now full of noise about the monitoring pipeline being unhealthy. The observer had become the thing being observed.

We rolled back all drain configurations to Render's defaults within minutes.

Post-mortem

The root cause was a combination of three things that compounded each other:

1. Unfiltered background worker logs. Background job processors emit a lot of lifecycle noise by default — job enqueued, job started, job completed, job retried, and so on. These events fire on every job. At our job volume, a single background worker service can generate thousands of log lines per minute. None of this is signal for ongoing observability. It's useful when debugging a specific job failure, not for dashboards.

2. Fan-out multiplication. We were routing logs to three destinations simultaneously: Loki, Datadog, and BetterStack. A 3x spike in log volume becomes a 9x spike in outbound pipeline throughput. Every destination gets hit at the same time, with the same burst.

3. All services at once. Testing on one service — even the highest-volume one — told us the pipeline could handle a single source. It told us nothing about what happens when five sources open simultaneously. The aggregate volume was an order of magnitude higher than anything we'd validated.

What we changed

Incremental rollout. After rolling back, we added services back one at a time with 24 hours of observation between each. If volume, error rates, and downstream health looked stable for 24 hours, we added the next one. The full rollout took five days instead of five minutes.

Log filtering in Alloy. We defined explicit filter rules per service type. Background worker INFO-level lifecycle events (enqueued, started, completed) are now dropped by Alloy before they reach any destination. Only WARNING and above, plus specific job failure patterns, pass through. This cut background worker log volume by roughly 80%.

Volume alerts on the ingestion pipeline. We added alert rules on Alloy's own metrics — ingestion rate, error rate, downstream write failures. If the pipeline itself starts showing stress, we know before it cascades.

The detail that almost cost us — the 8 KB syslog limit

After the system was stable, we noticed something odd: certain application errors were present in Render's own log view but missing from Loki. Not all of them. Just the large ones.

The ones that were missing were stack traces. Full Java-style stack traces can easily exceed 100 KB. Syslog has a default maximum message size of 8 KB (defined in RFC 5424). Messages that exceed it are silently truncated or dropped, depending on the implementation.

Silent. No error. No warning. No indication in any metric that a message was lost. The data simply didn't arrive.

We increased the max_message_len parameter in the syslog-proxy and Alloy's syslog listener to 256 KB. The missing stack traces appeared immediately.

The lesson here is broader than syslog: check the defaults of every tool in your pipeline. Buffer sizes, message limits, timeout values, retry caps — these are all set to something that was reasonable for a generic use case. They may not be reasonable for yours. And when they're wrong, most tools will not tell you.

Current state

The stack has been running stably for several weeks. Current metrics:

Droplet: 4 GB RAM, ~30% utilization under normal load
Log ingestion: all three service types, filtered, continuous
Storage: local filesystem (S3 planned for long-term retention)
External fan-out: Datadog and BetterStack, with volume alerts before we approach their intake limits
Alert coverage: application error rates, pipeline health, job failure patterns, infrastructure metrics via Mimir

The Grafana dashboards now show a real-time view of system health across all services. When something breaks, we know about it from an alert before a user reports it. That has happened twice since launch, and both times the alert fired before any user-facing degradation was detectable.

Key takeaways

Not all logs are worth storing. Define what matters per service before you start ingesting. Background worker lifecycle events are not observability — they're debug information that belongs in a trace, not a log aggregator. Decide at collection time, not retention time.

Buffer every I/O boundary. Silent drops are worse than backpressure. Vector saved us from losing data during restarts and slowdowns. Put a buffer anywhere data moves between two systems that can fail independently.

Fan-out multiplies everything. A 3x spike in log volume doesn't hit three destinations at 3x each — it hits three destinations at 3x simultaneously, and each downstream system now has to handle the same burst. Design for the aggregate.

Roll out incrementally. One source, observe for 24 hours, then next. Validating on a single service tells you almost nothing about aggregate behavior. The incident would not have happened if we had added services one at a time.

Check your defaults. The 8 KB syslog limit is a good example of a default that works for most cases and silently breaks for edge cases. Every tool in your pipeline has limits like this. Read the config reference. Set explicit values.

Monitor the monitor. Your observability pipeline is a production system. It needs its own health metrics, its own alerts, and its own runbook. If the pipeline goes down during an incident, you're blind exactly when you need visibility most.

Stack

Grafana · Loki · Mimir · Tempo · Alloy · Vector · Caddy · Render · DigitalOcean · S3 (planned)

If you're building something similar or have questions about any part of the architecture, feel free to reach out.

From Backend Engineer to Building Production Infrastructure in AWS

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Sun, 05 Apr 2026 13:30:01 +0000

Last year in September, I was laid off. By November, I'd landed a role at a startup with a responsibility I wasn't fully prepared for: build a reliable AWS infrastructure from scratch.
I'd only ever worked as a fullstack engineer. I knew the basics — VPCs, load balancers, containers — but not at the level where I'd feel confident deploying anything to production, let alone architecting an entire infrastructure with a real budget on the line.
But I took the job anyway. This is the story of what I did, what I got wrong, and what I learned along the way.

This is the story of every mistake, pivot, and small win along the way.

The Starting Point: Rewriting Before Deploying

Before I could deploy anything, there was a bigger problem. The existing backend was legacy JavaScript — no types, no structure, no clear separation of concerns. Deploying it as-is would just be putting broken code on expensive servers.

So I rewrote it in TypeScript.

I went with a 3-layer architecture: routes, controllers, and module functions. The goal was maintainability — whoever touches this codebase next should be able to find things without a treasure map. Most of the refactoring happened through Cursor, which made the migration significantly faster than doing it by hand.

With the codebase stable, it was time to ship.

First Deployment: The ECS Experiment

The first service that needed to go live was a CMS — a monorepo running a backend with SQLite. My initial plan was to standardise on ECS for everything. It felt like the "right" way to do containers on AWS.

I containerised the backend, learned how ECR works, pushed the image, mounted an EBS volume for the SQLite database, and put an ELB in front of it. It took two full weekends. But it worked.

Then I got the message from management: the cost of ECS is too high.

For a simple CMS serving internal traffic, they were right. I was over-engineering it.

The Pivot: Right-Sizing the Infrastructure

I tore everything down and went back to research — blogs, documentation, conversations with LLMs to pressure-test ideas. The answer was simple once I stopped thinking in terms of "what's the most modern approach" and started thinking about "what does this workload actually need."

A CMS doesn't need container orchestration. It needs a box that runs.

Here's what I landed on:

Backend: EC2 t3.small — cheap, sufficient, easy to manage
Frontend: AWS Amplify — fast deploys, built-in CDN, no server to maintain
Database: SQLite on EBS with weekly backups to S3

Total cost: a fraction of the ECS setup. Same reliability for the workload. That was my first real win — not building the most impressive architecture, but building the right one.

The Docker Build Problem

With the infrastructure sorted, a new bottleneck appeared: CI/CD.

Every push triggered a Docker build, and each build was taking around 5 minutes. We were running a monorepo with pnpm, and the issue was layer caching — or rather, the lack of it. Every build was reinstalling every dependency from scratch.

The fix was multi-stage builds. By separating the dependency installation layer from the application code layer, Docker could cache the expensive pnpm install step and only rebuild what actually changed.

Build times dropped from 5 minutes to 20 seconds. Across 4 repositories. That's the kind of improvement that compounds — every developer, every push, every day.

Nightly API Testing Without a Dedicated Server

The next requirement was daily test runs — a nightly job that would hit all our APIs, run the test suite with Jest and Supertest, and generate a report for the team.

My first instinct was to spin up another EC2 instance. But that meant paying for a machine that sits idle 23 hours a day, plus managing its uptime, patching, and monitoring.

Then I discovered that GitHub Actions provides temporary VMs for exactly this kind of job. No infrastructure to manage. No idle costs. The workflow spins up, runs 100+ API tests in about 5 minutes, generates an HTML report, uploads it to S3, and notifies the admins.

A scheduled cron job in a YAML file replaced an entire server.

What I Actually Learned

Looking back at these four months, the technical skills were the easy part. Docker, ECS, EC2, Amplify, GitHub Actions — you can learn any of these in a weekend. The harder lessons were about decision-making:

Cost awareness changes how you architect. When you're spending someone else's money, every decision carries weight. "Best practice" doesn't mean anything if it's 10x the cost for a workload that doesn't need it.

The first solution is rarely the right one. ECS was a fine technology. It was the wrong choice for the problem. Being willing to tear something down after two weekends of work is a skill in itself.

Boring infrastructure is good infrastructure. An EC2 instance with a cron job isn't exciting. It's also not going to wake you up at 3am.

Optimise the inner loop. The Docker build improvement saved 4+ minutes per push across 4 repos. Over a team and a quarter, that's days of developer time recovered.

I'm not a DevOps engineer. I'm not a backend engineer either. Honestly, it's better to let go of that mindset entirely. In the coming times, just doing one thing won't be enough. You're an engineer — you solve problems. The label doesn't matter; the willingness to figure it out does.

Months in and still going, I've learned that the gap between
"knowing the theory" and "running it in production" is mostly filled with wrong first attempts and the willingness to start over.

If you're in a similar position — thrust into infrastructure work without a roadmap — my advice is simple: deploy something, get it wrong, and iterate. The cloud makes it cheap to experiment, and every teardown teaches you more than any tutorial.

# Schema Evolution & Encoding: Building Future-Proof Data Systems 🚀

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Fri, 12 Sep 2025 11:35:41 +0000

How to design data-intensive applications that gracefully handle schema change

The Challenge: Data That Evolves 📈

In modern distributed systems, schema changes are inevitable. Your application will grow, requirements will shift, and your data structures must adapt. But here's the catch: you need to maintain compatibility with existing data and older versions of your application.

Understanding Compatibility Types 🔄

Backward Compatibility ⬅️

New code can read data written by old code
Essential when upgrading applications gradually

Forward Compatibility ➡️

Old code can read data written by new code
Critical for rolling back deployments safely

Encoding = Serialization 📦

Encoding is the process of converting in-memory data structures into a byte sequence that can be:

Stored in files
Transmitted over networks
Processed by different systems

Think of it as packaging your data for shipping! 📮

Traditional RPC Solutions 🌐

gRPC & Apache Thrift Approach

// Protocol Buffers example
message UserProfile {
  int32 user_id = 1;
  string name = 2;
  string email = 3;
  int32 age = 4 [optional]; // New field - backward compatible!
}

Key Strategies:

✅ Add new fields as optional
✅ Assign default/null values to new fields
✅ Generate classes from schema definitions
✅ Perfect for statically typed languages (Java, C++, Go)

The Dynamic Language Problem 🐍

Challenge: Traditional schema-based systems don't play well with dynamic languages like JavaScript, Python, or Ruby.

Why?

No compile-time type checking
Runtime schema validation needed
Class generation feels unnatural

Apache Avro 🌟

Apache Avro solves the dynamic language compatibility problem with a genius approach: dual schemas!

Avro's Dual Schema System 🎭

Writer Schema 📝

The schema used when encoding the data

Reader Schema 📖

The schema used when decoding the data

// Writer Schema (v1)
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
  ]
}

// Reader Schema (v2) - with new field
{
  "type": "record", 
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

How Avro Handles Compatibility ⚡

Field Resolution Rules:

Field in writer but not reader → Ignored 🙈
Field in reader but not writer → Default value used 📋
Field in both → Direct mapping ✅

This elegant system enables both forward and backward compatibility!

Schema Negotiation in Practice 🤝

When two processes communicate over a bidirectional network connection:

sequenceDiagram
    participant A as Service A
    participant B as Service B

    A->>B: Connection Request + Schema Version
    B->>A: Schema Negotiation Response
    A->>B: Agreed Schema for Session
    Note over A,B: Use negotiated schema for<br/>entire connection lifetime

Benefits:

✅ Both sides agree on schema version upfront
✅ Optimal performance (no per-message overhead)
✅ Clear compatibility guarantees

Best Practices for Schema Evolution 📚

Do's ✅

Always add new fields as optional
Provide sensible default values
Use semantic versioning for schemas
Test compatibility with real data
Document migration strategies

Don'ts ❌

Never remove required fields
Avoid changing field types drastically
Don't reuse field IDs/names for different purposes
Never skip compatibility testing

Real-World Impact 🌍

Companies using these patterns:

Netflix: Schema evolution for microservices
LinkedIn: Avro for data pipelines
Uber: Protocol Buffers for service communication
Airbnb: Thrift for cross-service APIs

The result? Zero-downtime deployments and seamless data migrations! 🎉

Conclusion: Future-Proof Your Data 🔮

Schema evolution isn't just a technical detail—it's a business enabler. By choosing the right encoding strategy:

Reduce deployment risks ⚡
Enable continuous delivery 🚀
Support diverse technology stacks 🌈
Maintain system reliability 🛡️

Key Takeaway: Whether you choose Protocol Buffers, Thrift, or Avro, the principles remain the same—design for change from day one!

Validating Environment Variables in Node.js with Zod

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Thu, 22 May 2025 20:00:01 +0000

Ever pushed a project to production only to realize an environment variable was missing or mistyped? I have—and it wasn’t fun.

Recently, I ran into a subtle but recurring issue in one of our Node.js projects: environment variables not loading correctly from the .env file. At first glance, this seemed trivial—just use dotenv, right? But as our configuration grew more complex, undefined values started creeping in silently, breaking features without clear errors.

The Problem with Plain .env + process.env

Like many teams, we began with this simple setup:

import dotenv from "dotenv";
dotenv.config();

const CORE_DB_URI = process.env.CORE_DB_URI;

But the issues quickly stacked up:
• We forgot to set some variables in staging or dev environments.
• Typos in variable names would go unnoticed until runtime.
• Some variables like PORT needed to be parsed (parseInt), while others had fallbacks—lots of boilerplate.

Worst of all, we had no way to know what was missing or invalid until something failed. That’s not a great developer experience.

The Breakthrough: Validating the Environment with Zod

After some research and trial-and-error, we discovered a better way—define a schema for environment variables using zod, and parse process.env against it before anything else runs.

Here’s what that looks like:

import z from "zod";

const envSchema = z.object({
  NODE_ENV: z.string().optional(),
  HOST: z.string().optional(),
  PORT: z.string().transform(val => parseInt(val)).optional(),

  MONGO_DB_URI: z.string().nonempty("MONGO_DB_URI is required"),
  // ... add other required vars

  AWS_ACCESS_KEY: z.string().nonempty("AWS_ACCESS_KEY is required"),
  SMTP_USERNAME: z.string().nonempty("SMTP_USERNAME is required"),
});

Then we updated the config loader like this:

import dotenv from "dotenv";
import { envSchema } from "./envSchema";

dotenv.config();

let parsedEnv;
try {
  parsedEnv = envSchema.parse(process.env);
  console.log("✅ Environment validation passed");
} catch (err) {
  console.error("❌ Environment validation failed:");
  console.error(JSON.stringify(err.format?.(), null, 2));
  process.exit(1);
}

Benefits We Gained

This refactor gave us immediate wins:
• Validation at startup: The app won’t even run unless all required env vars are present.
• Helpful errors: Each missing variable now shows a clear error message.
• No more implicit casting or null-checks: Everything is validated and transformed up front.
• Cleaner config code: No more process.env.XYZ || "fallback" everywhere.

Instead of this:
const CORE_DB_URI = process.env.CORE_DB_URI!;

We now use this:
const config = { CORE_DB_URI: parsedEnv.CORE_DB_URI, };

Fail Fast, Not in Production

One thing that really sold me on this pattern
[https://www.youtube.com/watch?v=eZBsBMf5zuQ]
is how it catches config issues at runtime—before your app misbehaves in production.

Without schema validation, a missing environment variable might not throw an error until it’s used—say, when connecting to a database or sending an email. That’s late. By then, users might already be affected.

With zod, your app won’t even boot if a required environment variable is missing or misconfigured. That’s intentional. You get a clear error in development, staging, or CI pipelines—not after you’ve shipped broken code.

It’s simple, safe, and clean.

Final Thoughts

It’s easy to underestimate the importance of config hygiene—until it breaks something. Using zod for schema validation gives you type safety, meaningful errors, and cleaner code. It’s now a pattern I’ll carry into every serious Node.js project.

If you’re still relying solely on .env + process.env, try wrapping it with validation. Your future self—and your team—will thank you.

From Storage to Stream: Delivering MongoDB Data Directly to Users

〽️ 𝙍𝙤𝙨𝙝𝙖𝙣 — Mon, 30 Dec 2024 20:16:48 +0000

Step 1: MongoDB Cursor

Here’s how we set up the cursor (reusing your snippet):

const cursor =
    userObject?.data?.serviceProviderName === 'ZYRO'
        ? zyroTransactionModel.find(query).cursor()
        : finoTransactionModel.find(query).cursor();

console.log("Cursor created successfully");

Step 2: Setting Up the ZIP File
Use the yazl library to stream CSV data into a ZIP file:

const yazl = require('yazl');
const zipfile = new yazl.ZipFile();

reply.raw.writeHead(200, {
    "Content-Type": "application/zip",
    "Content-Disposition": "attachment; filename=transactions.zip",
});

zipfile.outputStream.pipe(reply.raw);

const cleanup = async () => {
    console.log("Cleaning up resources...");
    zipfile.end(); // Finalize ZIP
    await cursor.close();
};
reply.raw.on("close", cleanup);
reply.raw.on("error", cleanup);

Step 3: Creating Dynamic CSV Streams
Generate CSV data dynamically and stream it into the ZIP file:

const createNewCSVStream = (headers) => {
    const csvStream = new Readable({ read() {} });
    csvStream.push(headers.join(",") + "\n"); // Add headers
    return csvStream;
};

const filteredHeaders = getHeaders(transactionDownloadFields, userObject?.state?.auth?.role);
const currentCSVStream = createNewCSVStream(filteredHeaders);

zipfile.addReadStream(currentCSVStream, "transactions_part_1.csv");

Step 4: Streaming MongoDB Data to CSV
Stream the data from MongoDB directly into the CSV:

cursor.on('data', (doc) => {
    const csvRow = filteredHeaders.map(header => doc[header.key] || '').join(',');
    currentCSVStream.push(csvRow + '\n'); // Write row
});

cursor.on('end', () => {
    currentCSVStream.push(null); // End the stream
    zipfile.end(); // Finalize the ZIP
});

Step 5: Processing Data from MongoDB Cursor
Stream documents from the MongoDB cursor, transform them as needed, and dynamically write rows to the CSV stream:

try {
    for await (const doc of cursor) {
        if (clientDisconnected) {
            console.log("Client disconnected. Stopping processing...");
            break;
        }

        streamedCount++;
        rowCount++;

        let row = "";
        const filteredHeaders = getHeaders(
            transactionDownloadFields,
            userObject?.state?.auth?.role
        );

        for (let i = 0; i < filteredHeaders.length; i++) {
            const field = filteredHeaders[i];

            // Fetch the corresponding field configuration from transactionDownloadFields
            const originalField = transactionDownloadFields.find((f) => f.value === field.value);

            // Get the value from the transaction document
            let value = getValueFromTransaction(doc, field.value);

            // Apply transformation if the field has a transform function
            if (originalField?.transform) {
                value = originalField.transform(value);
            }

            // Enclose the value in double quotes
            value = value !== undefined ? `"${value}"` : '"N/A"';
            row += (i > 0 ? "," : "") + value;
        }
        row += "\n";
        currentCSVStream.push(row);

        // Check if the row count has reached the threshold for the current CSV file
        if (rowCount >= MAX_ROWS_PER_FILE) {
            console.log(`Threshold reached for file ${fileIndex - 1}. Starting new file...`);
            currentCSVStream.push(null); // End the current CSV stream
            currentCSVStream = createNewCSVStream(); // Start a new stream
            rowCount = 0; // Reset the row count
        }
    }

    // Finalize the current CSV stream if it has data
    if (currentCSVStream) {
        currentCSVStream.push(null);
    }

    // Finalize the ZIP file
    zipfile.end();
    console.log(`Successfully streamed ${streamedCount} rows across ${fileIndex - 1} files.`);
} catch (error) {
    console.error("Error during processing:", error);
    if (!headersSent) reply.status(500).send({ error: "Failed to generate ZIP file" });
} finally {
    // Cleanup: Close the MongoDB cursor
    await cursor.close().catch((err) => console.error("Error closing cursor:", err));
}

Summary

Document Iteration Using for await...of:

Streams documents one by one from the MongoDB cursor efficiently.
Enables real-time processing without loading all data into memory.

Dynamic CSV Row Generation:

Constructs each row dynamically by iterating over filteredHeaders.
Applies transformations using a transform function, if defined in transactionDownloadFields.
Row Threshold and File Splitting:

Monitors the row count against the threshold (MAX_ROWS_PER_FILE).
Ends the current CSV stream and starts a new one when the threshold is reached.

Error Handling:

Logs and sends an error response if an issue occurs during processing.
Ensures proper cleanup by closing the MongoDB cursor in the finally block.

Finalizing Streams:

Pushes null to terminate the current CSV stream.
Completes the ZIP file once all rows are processed.

DEV Community: 〽️ 𝙍𝙤𝙨𝙝𝙖𝙣

I Rewrote My Node.js/Express Backend in NestJS

The difference was visible in the structure

Starting with the database

Rebuilding the APIs

Authorization

WebSockets

The biggest benefit

What I Learned from Reading JavaScript: The Good Parts

What I Learned from Reading JavaScript: The Good Parts

1. Functions are first-class values

2. Closures are one of JavaScript's superpowers

3. Objects inherit through prototypes

4. this isn't determined by where a function is written

5. JavaScript has some dangerous features

6. JavaScript was designed with some historical baggage

7. Readability matters more than cleverness

8. Simplicity beats feature count

9. JavaScript is much simpler than it first appears

Things that are outdated today

My biggest takeaway

Float64Array, TimSort and sort

Debugging Production Alerts Without Chasing The Wrong Problem

The First Assumption

What Was Actually Happening

The Timezone Trap

Other Noise

The Final Picture

Takeaway

Bulk Emails from a chat input — without Redis, queues, or worker services

The idea

Triggering the bulk flow

Storage layer

pipeline_runs

pipeline_jobs

Generating the emails

Live progress in the chat UI

Tradeoffs

Function timeout

Gmail SMTP limits

after() durability

Why I skipped BullMQ, QStash, and Inngest

Bulk Emails from a chat input — without Redis, queues, or worker services

The idea

Triggering the bulk flow

Storage layer

pipeline_runs

pipeline_jobs

Generating the emails

Live progress in the chat UI

Tradeoffs

Function timeout

Gmail SMTP limits

after() durability

Why I skipped BullMQ, QStash, and Inngest

Bulk Emails from a chat input — without Redis, queues, or worker services

The idea

Triggering the bulk flow

Storage layer

pipeline_runs

pipeline_jobs

Generating the emails

Live progress in the chat UI

Tradeoffs

Function timeout

Gmail SMTP limits

after() durability

Why I skipped BullMQ, QStash, and Inngest

How I Built a Production Observability Stack — And Broke It Twice Before It Worked

The starting point

The stack

Problem 1: No token enforcement on the log stream

Problem 2: Connection drops under load

The incident

Post-mortem

What we changed

The detail that almost cost us — the 8 KB syslog limit

Current state

Key takeaways

Stack

4. `this` isn't determined by where a function is written

`pipeline_runs`

`pipeline_jobs`

`after()` durability

`pipeline_runs`

`pipeline_jobs`

`after()` durability

`pipeline_runs`

`pipeline_jobs`

`after()` durability