DEV Community: Vishal Pandey

From | to SIGINT : How Linux Shells Build and Control Pipelines

Vishal Pandey — Sun, 20 Sep 2026 06:57:43 +0000

Have you ever looked at a command like this and wondered what the | is actually doing?

cat file.txt | grep error

We type pipelines so often that the symbol almost disappears. cat produces some text, grep filters it, and the terminal prints the result. Simple enough.

Linux has to build that connection before either program can do its work.

There is the shell, one process running cat, and another running grep. The two commands have different process IDs. A kernel pipe carries bytes between them, and a process group lets the shell control them as one job.

The | character never reaches cat or grep. Bash interprets it first. It creates the pipe, starts the commands, changes their standard file descriptors, groups the processes, and waits.

A tiny character is hiding a fair amount of operating-system machinery.

The finished pipeline

Start with the state Bash is trying to create.

cat and grep remain ordinary programs. cat writes to standard output. grep reads from standard input. Bash changes where those descriptors lead before either program begins running.

Two separate ideas are involved:

Pipe          moves bytes between processes
Process group lets the shell and terminal control related processes together

Keeping those ideas separate makes the rest much easier to follow.

Bash creates a kernel pipe

For a simple two-command pipeline, Bash needs one pipe. Conceptually, it asks the kernel for one using pipe() or pipe2().

The kernel returns two file descriptors:

FD 3 -> pipe read end
FD 4 -> pipe write end

The numbers 3 and 4 are only examples. The kernel returns available descriptor numbers in the calling process.

The pipe is a unidirectional byte stream maintained by the kernel. Bytes written through its write end can be read through its read end. It has no application-level idea of lines, JSON objects, or log records. Those meanings belong to the programs using it.

At this point, the shell might have a descriptor table like this:

Bash

FD 0 -> terminal input
FD 1 -> terminal output
FD 2 -> terminal error output
FD 3 -> pipe read end
FD 4 -> pipe write end

The pipe exists, but neither command exists yet.

Each command gets a process

The traditional model is fork() followed by execve().

Bash forks a child for cat. It also forks a child for grep:

Bash PID 1000
   |
   +-- child PID 1101
   |
   `-- child PID 1102

After fork(), a child inherits copies of its parent's open file descriptors. Those descriptors refer to the same underlying open-file descriptions as the parent's descriptors. The child therefore inherits access to both ends of the pipe.

The child still begins as a copy of the shell process. It has not become cat or grep yet.

That happens through execve().

Child PID 1101 --execve()--> cat PID 1101
Child PID 1102 --execve()--> grep PID 1102

execve() replaces the program running inside a process. The process keeps its PID, while its code, stack, heap, and other program state are replaced for the new executable. File descriptors normally remain open across execve() unless they carry the close-on-exec flag.

This distinction is easy to miss:

fork()   creates a new process
execve() replaces the program inside a process

Real shells have implementation details and optimizations. They may use posix_spawn() on some systems, and built-ins can follow different rules. The fork() plus execve() model still exposes the mechanics cleanly.

`dup2()` does the wiring

The children inherited the pipe, but their standard streams still point at the terminal. Bash must rearrange the descriptors before executing the programs.

The cat child needs its standard output connected to the write end of the pipe:

dup2(pipe_write, STDOUT_FILENO);

Using the example descriptor numbers, that is:

dup2(4, 1);

Afterward, descriptors 1 and 4 refer to the same pipe endpoint:

cat child

FD 1 --+
       +--> pipe write end
FD 4 --+

The child can close descriptor 4. Standard output already points to the required endpoint.

The grep child needs the other side:

dup2(pipe_read, STDIN_FILENO);

With the example numbers:

dup2(3, 0);

Its standard input now refers to the pipe's read end:

grep child

FD 0 --+
       +--> pipe read end
FD 3 --+

Once both children have been prepared and the unnecessary descriptors closed, their useful descriptor tables look like this:

cat
  FD 0 -> inherited input
  FD 1 -> pipe write end
  FD 2 -> terminal

grep
  FD 0 -> pipe read end
  FD 1 -> terminal
  FD 2 -> terminal

Bash can now execute the programs. cat writes to FD 1 as usual, except FD 1 leads into the pipe. grep reads FD 0, which leads to the other end.

No special pipeline-aware version of either program is needed.

Closing descriptors is part of correctness

Every process inherits descriptors during fork(). That inheritance creates copies which must be closed when a process does not need them.

After starting both commands, Bash still holds its own copies of the pipe descriptors:

Bash
  FD 3 -> pipe read end
  FD 4 -> pipe write end

Bash should close both. The children must also close every pipe end they do not use.

This affects observable behaviour.

A reader sees EOF only after every descriptor referring to the pipe's write end has been closed. Suppose cat finishes, but Bash accidentally keeps its copy of the write end open:

cat's write descriptor   -> closed
Bash's write descriptor  -> still open

From the kernel's perspective, a writer still exists. grep can remain blocked, waiting for bytes that will never arrive.

The other direction has its own failure mode. Once every read end is closed, another write generates SIGPIPE. If the writer ignores that signal, write() fails with EPIPE instead.

You can see this relationship with:

yes | head -n 5

head exits after five lines and closes its read side. yes keeps trying to write, so it encounters the broken pipe condition and stops.

Closing an unused descriptor is doing more than saving a small kernel resource. It changes when peers observe EOF and broken pipes.

One pipeline, several PIDs, one process group

The pipeline now moves data correctly. Bash still needs a way to treat its processes as one interactive job.

That is the role of a process group.

For example:

Bash
  PID  = 1000
  PGID = 1000

Pipeline
  cat   PID = 1101, PGID = 1101
  grep  PID = 1102, PGID = 1101

The group does not require an additional process. One member is the process-group leader, and its PID is used as the PGID. In this example, cat is the leader because its PID and the group's PGID are both 1101.

An interactive shell normally arranges this with setpgid(). The exact ordering is more careful than a straight-line diagram suggests because the parent and children run concurrently. The shell has to avoid races while the children start and execute their programs.

The useful mental model remains simple: every external command usually receives its own PID, while all processes in one pipeline join the same process group.

Bash builds a job around that pipeline. A job is shell state: it has a job number, status, and command text. The kernel knows about processes and process groups. It does not know about Bash job specifications such as %1.

Shell job        user-facing bookkeeping in Bash
Process group    kernel mechanism beneath that job

The terminal has a foreground process group

Process groups become especially useful once a terminal is involved.

A terminal tracks one foreground process group. tcsetpgrp() changes that group, while tcgetpgrp() reads it.

Before Bash runs the pipeline, the shell itself owns the terminal foreground:

Terminal foreground PGID = 1000
Bash PGID                = 1000

Bash creates the pipeline and gives terminal foreground control to PGID 1101:

Terminal foreground PGID = 1101

PGID 1101
  |- cat
  `- grep

Bash is still alive. It has simply moved outside the terminal's foreground group while it waits for the job.

This foreground relationship controls which job can normally read from the terminal. It also decides where keyboard-generated job-control signals go.

What `Ctrl+C` actually does

Pressing Ctrl+C does not ask Bash to choose a PID and kill it.

In a normal terminal session, the path looks more like this:

keyboard
   |
   v
terminal emulator
   |
   v
pseudoterminal master
   |
   v
pseudoterminal slave
   |
   v
N_TTY line discipline
   |
   v
SIGINT sent to the foreground process group

The terminal emulator commonly writes byte 0x03 for Ctrl+C to the PTY master. On the slave side, Linux's default N_TTY line discipline processes the input.

Terminal settings define an interrupt character named VINTR. Its usual value is ^C. When the ISIG setting is enabled, the line discipline interprets that character and generates SIGINT. The byte is consumed as terminal control input instead of being delivered to the foreground program as ordinary stdin data.

The target is the terminal's foreground process group.

In the example, both processes receive SIGINT:

SIGINT -> PGID 1101
           |- cat  PID 1101
           `- grep PID 1102

This explains why a whole pipeline usually reacts to one Ctrl+C. The pipe is not carrying the signal. Process-group membership gives the kernel the target set.

It also explains why Bash survives. Bash belongs to PGID 1000, while the terminal's foreground PGID is 1101.

`SIGINT` is a request the process can handle

The default action for SIGINT is process termination, which creates the familiar effect of Ctrl+C stopping a command.

A program can install a handler, block the signal temporarily, or ignore it. Python, for example, normally turns SIGINT into KeyboardInterrupt. Other programs use it to close files, restore terminal state, or ask the user to confirm an exit.

Every process in the foreground group receives the signal, but each one applies its own signal disposition. A pipeline can therefore react unevenly if one command catches SIGINT and another uses the default action.

SIGKILL has different rules. A process cannot catch, block, or ignore it. That is one reason kill -9 should not be the first way to stop a program: it removes any chance for application-level cleanup.

How Bash gets the prompt back

While the foreground job runs, Bash waits for its children. It needs to notice several possible outcomes:

a process exits normally;
a process terminates because of a signal;
a process stops, commonly after Ctrl+Z generates SIGTSTP.

Once the foreground job exits or stops, Bash makes its own process group the terminal foreground again. Then it updates the job's status and prints another prompt.

Pipeline running:
  terminal foreground PGID = 1101

Pipeline finishes or stops:
  terminal foreground PGID = 1000
  Bash prints the prompt

fg and bg use the same model. fg continues a stopped job and gives its process group the terminal. bg continues the job without transferring foreground terminal ownership.

Watch the IDs yourself

Start a pipeline that stays alive long enough to inspect:

sleep 100 | sleep 100

From another terminal, run:

ps -o pid,ppid,pgid,sid,tpgid,stat,comm -C sleep

An example result might look like:

  PID  PPID  PGID   SID TPGID STAT COMMAND
 2101  2000  2101  2000  2101 S+   sleep
 2102  2000  2101  2000  2101 S+   sleep

The exact values will differ. Look at their shape:

the PIDs are different;
the PGIDs match;
the TPGID matches the pipeline's PGID because this is the terminal's foreground job;
both processes belong to the same session.

Press Ctrl+C in the first terminal. Both sleep processes should disappear.

You can also inspect the terminal's configured special characters:

stty -a

Look for values similar to:

intr = ^C
susp = ^Z

These mappings are configurable. Ctrl+C is conventional rather than an unchangeable property of the keyboard.

A few boundaries around the model

The diagrams describe a normal interactive Bash session with job control enabled and external commands in the pipeline. They are a mental model, not Bash source code.

Shell built-ins complicate process placement. Bash usually runs pipeline components in subshell environments, while options such as lastpipe can change where the final component runs when job control is disabled.

Non-interactive shells commonly run without job control. They can still build pipes and execute multiple processes, but the foreground process-group handoff may not apply in the same way.

set -o pipefail changes how Bash calculates the pipeline's exit status. The processes stay separate.

And a longer pipeline simply extends the same wiring:

producer | transform | consumer

It normally needs two kernel pipes:

producer stdout
      |
      v
   pipe A -> transform -> pipe B -> consumer

Each process closes every pipe endpoint it does not need. The shell places the commands into a shared process group when interactive job control calls for it.

The whole path

The original command was short:

cat file.txt | grep error

Its working shape is larger:

Bash parses the pipeline
        |
        v
creates a kernel pipe
        |
        v
creates a process for each command
        |
        v
uses dup2() to connect stdout and stdin
        |
        v
closes unused descriptors
        |
        v
executes cat and grep
        |
        v
places them in one process group
        |
        v
gives that group the terminal foreground
        |
        v
waits for the job

Then Ctrl+C takes a different path through the same structure:

Ctrl+C
  -> PTY
  -> N_TTY line discipline
  -> SIGINT
  -> foreground process group
  -> cat and grep each decide how to respond

The pipe moves the data. The process group gives the job a boundary. The terminal uses that boundary when you ask the foreground work to stop.

Sources

Introducing limitngin — A Lightweight, Memory-Efficient & Scalable Rate Limiter for Express

Vishal Pandey — Tue, 03 Mar 2026 01:52:00 +0000

I recently built and open-sourced limitngin, a lightweight, zero-dependency, ESM-only rate limiter middleware for Express.

The goal was simple:

Build something that works efficiently for both small projects and larger systems, without forcing external infrastructure from day one.

Designed for Small Projects — Without Redis

For:

Personal projects
MVPs
Internal tools

You often don’t want to set up Redis just to enable basic rate limiting.

limitngin uses an optimized in-memory store that is:

Map-based for better memory behavior
Automatically cleaned up
Stable under high key churn
Zero external dependencies
Drop it in and it works.

Built With Scale in Mind

Even though it starts in-memory, the internal architecture is designed for extensibility.

It already supports:

Sliding Window Counter (default)
Token Bucket
IP-based limiting
Auth-token–based limiting
Standard RateLimit-* + Retry-After headers
Type-safe configuration using discriminated unions

And Redis (or external storage adapters) are planned for future releases.

So you can start simple — and scale later.

Stress & Memory Testing

Before publishing, I ran controlled stress tests:

~500,000 unique key simulations
Continuous request churn
Heap inspection under sustained load
Manual GC checks using node --expose-gc

Results:

Stable memory growth curve
No memory leaks
Proper cleanup of inactive keys
~10MB reduction in peak memory after migrating from Record → Map
Heap returning close to baseline after GC

The focus was to predict behavior under pressure.

If you’re building with Express and want:

A clean API
Zero dependencies
Algorithm flexibility
Efficient in-memory performance

limitngin is now available.

Open to discussions, feedback, and contributions.

How I Took a Slow Node.js API from 5.7 sec 11ms Using Real Load Tests (Beginner-Friendly)

Vishal Pandey — Mon, 24 Nov 2025 14:46:21 +0000

Most tutorials show you “fast Node.js code,” but they never show you what happens when you actually measure performance under load.

I wanted to learn real backend optimization, so I built a small URL redirect API:

GET /:slug → returns original URL

Simple idea.
But the first time I load-tested it, the API took 5.73 seconds to respond.

This article is a beginner → mid-level friendly breakdown of:

what slowed my API
how I optimized it
what actually improved performance
real k6 load test results at every step
and what beginners usually misunderstand about backend latency

🖥️ Environment Note

All benchmarks, optimizations, and load tests in this article were performed on a local machine, not a server or cloud instance.
Numbers will vary depending on hardware — but the optimization principles remain the same.

`Stage 1 — Baseline Test (No Index, Direct MySQL Connection)`

Avg latency: 5.73 seconds
Requests: ~68 RPS

This was the first k6 run.

Here are the baseline numbers:

Why it was this slow?
MySQL was doing a full table scan for every slug.
No index → MySQL compares slug against every row → worst possible plan.
At high concurrency, MySQL was simply overwhelmed.

`Stage 2 — Added a Slug Index`

Avg latency: 27.1 ms
Requests: ~18k RPS

Index command: CREATE INDEX idx_slug ON urls(slug);

After adding the index:
This alone brought latency from 5.73 seconds → 27 ms.
This is the single biggest improvement you can ever make in a read-heavy API.
Beginners underestimate how critical indexing is.
No amount of Node.js code optimization can beat a proper DB index.

`Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool`

Avg latency: ~18 ms
Before this, every request created a new MySQL connection.
That’s expensive under load.
So I switched to:
export const db = mysql2.createPool({ host, user, password, database, connectionLimit: 10, });

After switching to pooling:

Why this helps
Reuses open connections
Avoids handshake overhead
Much better concurrency
MySQL stays stable under load
If you're using MySQL + Node.js:
Connection pools are mandatory.

`Stage 4 — Added Redis Cache (Final Optimization)`

Avg latency: 11.75 ms
Requests: ~42k RPS
Redis turns your slug lookup into an O(1) in-memory lookup.
Updated handler:

app.get('/:slug', async (req, res) => {
const slug = req.params.slug;

const cached = await redis.get(slug);
if (cached) return res.redirect(cached);

const data = await db.query("SELECT url FROM urls WHERE slug = ?", [slug]);
if (!data[0].length) return res.status(404).send("Not found");

await redis.set(slug, data[0][0].url);
return res.redirect(data[0][0].url);
});

k6 results after Redis:

From 5.7 seconds → 11 ms.
This is a ~520x improvement, and nothing about the API changed except internal optimizations.

🔗 GitHub Repository (Full Source Code)

You can check out the entire project here:

👉 Github Link

🙏 Thanks for Reading
If you have questions, want to discuss backend engineering, or just want to say hi —
I’m always open to messages.

How I Took a Slow Node.js API from 5.7 sec 11ms Using Real Load Tests (Beginner-Friendly)

Vishal Pandey — Mon, 24 Nov 2025 14:46:21 +0000

Most tutorials show you “fast Node.js code,” but they never show you what happens when you actually measure performance under load.

I wanted to learn real backend optimization, so I built a small URL redirect API:

GET /:slug → returns original URL

Simple idea.
But the first time I load-tested it, the API took 5.73 seconds to respond.

This article is a beginner → mid-level friendly breakdown of:

what slowed my API
how I optimized it
what actually improved performance
real k6 load test results at every step
and what beginners usually misunderstand about backend latency

`Stage 1 — Baseline Test (No Index, Direct MySQL Connection)`

Avg latency: 5.73 seconds
Requests: ~68 RPS

This was the first k6 run.

Here are the baseline numbers:

Why it was this slow?
MySQL was doing a full table scan for every slug.
No index → MySQL compares slug against every row → worst possible plan.
At high concurrency, MySQL was simply overwhelmed.

`Stage 2 — Added a Slug Index`

Avg latency: 27.1 ms
Requests: ~18k RPS

Index command: CREATE INDEX idx_slug ON urls(slug);

`Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool`

After switching to pooling:

Why this helps
Reuses open connections
Avoids handshake overhead
Much better concurrency
MySQL stays stable under load
If you're using MySQL + Node.js:
Connection pools are mandatory.

`Stage 4 — Added Redis Cache (Final Optimization)`

Avg latency: 11.75 ms
Requests: ~42k RPS
Redis turns your slug lookup into an O(1) in-memory lookup.
Updated handler:

app.get('/:slug', async (req, res) => {
const slug = req.params.slug;

const cached = await redis.get(slug);
if (cached) return res.redirect(cached);

const data = await db.query("SELECT url FROM urls WHERE slug = ?", [slug]);
if (!data[0].length) return res.status(404).send("Not found");

await redis.set(slug, data[0][0].url);
return res.redirect(data[0][0].url);
});

k6 results after Redis:

From 5.7 seconds → 11 ms.
This is a ~520x improvement, and nothing about the API changed except internal optimizations.

🔗 GitHub Repository (Full Source Code)

You can check out the entire project here:

👉 Github Link

🙏 Thanks for Reading
If you have questions, want to discuss backend engineering, or just want to say hi —
I’m always open to messages.

DEV Community: Vishal Pandey

From | to SIGINT : How Linux Shells Build and Control Pipelines

The finished pipeline

Bash creates a kernel pipe

Each command gets a process

dup2() does the wiring

Closing descriptors is part of correctness

One pipeline, several PIDs, one process group

The terminal has a foreground process group

What Ctrl+C actually does

SIGINT is a request the process can handle

How Bash gets the prompt back

Watch the IDs yourself

A few boundaries around the model

The whole path

Sources

Introducing limitngin — A Lightweight, Memory-Efficient & Scalable Rate Limiter for Express

Designed for Small Projects — Without Redis

Built With Scale in Mind

Stress & Memory Testing

How I Took a Slow Node.js API from 5.7 sec 11ms Using Real Load Tests (Beginner-Friendly)

Stage 1 — Baseline Test (No Index, Direct MySQL Connection)

Stage 2 — Added a Slug Index

Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool

Stage 4 — Added Redis Cache (Final Optimization)

How I Took a Slow Node.js API from 5.7 sec 11ms Using Real Load Tests (Beginner-Friendly)

Stage 1 — Baseline Test (No Index, Direct MySQL Connection)

Stage 2 — Added a Slug Index

Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool

Stage 4 — Added Redis Cache (Final Optimization)

`dup2()` does the wiring

What `Ctrl+C` actually does

`SIGINT` is a request the process can handle

`Stage 1 — Baseline Test (No Index, Direct MySQL Connection)`

`Stage 2 — Added a Slug Index`

`Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool`

`Stage 4 — Added Redis Cache (Final Optimization)`

`Stage 1 — Baseline Test (No Index, Direct MySQL Connection)`

`Stage 2 — Added a Slug Index`

`Stage 3 — Switched from mysql.createConnection to MySQL2 Connection Pool`

`Stage 4 — Added Redis Cache (Final Optimization)`