DEV Community: Younes Merzouka

How to do work queuing?

Younes Merzouka — Wed, 19 Aug 2026 16:37:32 +0000

Introduction

One of the important things to consider for this CI/CD tool is how to execute the different jobs.
CI/CD jobs are long running and can take up to many hours depending on the complexity of the code
base, language, tooling, etc. This means that we cannot execute such tasks during the lifetime of a
request.

This would require having a job queue that multiple background workers read from to execute the
different jobs. After we are done with a job we can save its status to a database for later
querying. The user can poll the server to see the status of the CI/CD job.

For the actual implementation of such a job queue we have a few options. But we first need to know
what requirements we need the work queue to fulfill.

What do we need in a work queue?

There are a few things we need in a work queue:

At its most basic form a work queue needs to be able to store messages, and for those messages to be able to be queried by multiple workers.
Suppose a worker claims a job from the queue, then it crashes, or the message never reaches it because of a network partition. If we removed the message after the worker request, then that job is gone.

This means we cannot do anything to the job until we are sure of the delivery of the message. In
other words, we need an at-least-once delivery guarantee.

Suppose now that a worker receives the message and confirms the receipt, then crashes while executing the job. There are two potential issues in such a scenario:
- If we just deliver the job without persisting it, that job is lost when the worker crashes, we have no way of handling the failure.
- Even if we persisted the job, we need a specific retry mechanism.
We cannot keep retrying forever, we need a way to know when we retried a job too much. This is done by tracking the number of retries for a given job. In case we want to debug a potential issue in our system that might be causing job failure, we can save the job to a Dead Letter Queue.

For the specific case of a CI/CD pipeline, we don't need retries. CI/CD jobs run for long duration
and a job failure doesn't always translate to a system failure - it could be that the job is not
defined correctly. This is also what popular CI/CD tools do: GitHub, GitLab, etc. This means that we
don't need point 3 and 4.

Dedicated message queues

Now that we know what we need from our queue - or our implementation of one - we can start with the
first solution for a work queue, which is to use a dedicated message queue such as RabbitMQ or Kafka.

A message queue is difficult to get right, and using a dedicated tool for it allows us to move the
complexity of such a tool to the infrastructure, while delivering the guarantees we need. RabbitMQ
and Kafka have dedicated brokers that handle things such as persistence, acknowledgements, retries,
message tracking, and re-delivery.

They might, however, need some tuning to get the features needed - for example, RabbitMQ requires
enabling persistence to allow recovery after crashes, and you also need to configure consumers of
messages to acknowledge messages when they are done.

Kafka can process up to 1 million jobs/second and RabbitMQ around 10K/s.

Redis

Another common way of doing work queues - used by tools like Celery - is to use Redis. Redis is a
key-value store, which means it is not purposefully built to handle message queuing. However it has
a few data structures that allow implementing a message queue that can satisfy the requirements
listed above. We have two options: Lists and Streams.

A common way of implementing message queues in Redis is to use normal and sorted lists. The main
list holds the jobs. After claiming a job, the worker moves it from the main list to a sorted list
acting as the processing list - we'll get into why it needs to be sorted shortly. Once the job is
done, the worker removes it from the sorted list and either deletes it or writes the final result to
a database.

Since the worker explicitly moves the job between lists, we get an at-least-once delivery guarantee -
the worker has confirmed receipt. But if the worker fails mid-job, the job stays stuck in the
processing list. To handle this we need a background process that scans the processing list and
re-queues jobs that are past a time threshold. This is why the list needs to be sorted - it makes it
easier to retrieve the oldest stuck items.

We also don't want to rerun a buggy job forever, so we track failures using a retry field on the job.
Past a certain threshold, the job moves to a dead letter queue for later debugging.

The stream in Redis is also another way of implementing a queue. How it works is similar to Kafka in
that it is a log-based data structure that logs any incoming messages. The stream is a bit more
complicated than plain lists, however, and is more suitable when having a lot of workers that need to
be coordinated in different ways. Since it relies on a log-based mechanism for storage, it requires
compaction which needs to be done manually.

Redis queues can process up to 20K jobs/second.

Postgres & MySQL

In relational databases a job queue is simply another table in your database. This means that you
track many of the states of a given job using fields in your table schema (status), and you just
need to implement the logic around it:

Job claiming changes status to processing.
Finishing a job changes the status to done/failed depending on the result.
Only jobs that are queued/unprocessed will be picked up by idle workers.
A background job rechecks for timed out jobs and changes the status to queued/unprocessed/created in case a worker fails before completing a job.

The main challenge with relational databases is competing workers. If both worker A and worker B do
a select to get another job to execute and both get the same job (say with id 1), they will both
claim the job and start processing it.

You might ask can't that also be an issue for Redis? Well... not really, Redis can only execute one
operation at a time so no competing workers.

To solve this we can use SELECT ... FOR UPDATE which would select a row and lock it so that no
other transaction (run by a given worker) can read it. The worker would then update the row to
processing on the same transaction where it claimed it. The lock is then released when the
transaction commits.

There is one issue however: if another worker tries to look for a job to process in another
transaction, it would get blocked on the locked row - this is because relational databases block on
locks by default. In order to avoid this we can use SKIP LOCKED which would skip any locked rows.
So the full statement SELECT ... FOR UPDATE SKIP LOCKED would select a new unlocked row and apply a
lock on the row for the worker to update it.

For the actual implementation a few things are worth noting:

Locks can degrade the performance of the database, so it is best to keep transactions as short-lived as possible to release locks - this means you shouldn't process the jobs in the transactions.
Consider adding an index to speed up the querying of jobs.

Work queues based on Postgres process up to 12K requests/second. I don't have any information about MySQL.

Discussion

While many of the existing solutions are well suited for implementing a work queue, which one to
reach for depends on your specific use case, and on what infrastructure you already have or how much
you are willing to invest in new infrastructure.

For my current goal with KCX, which is the tool being a single deployable unit, none of the existing
solutions are suitable. They all require adding another component to the system in order to have a
queue. This is why I chose to use SQLite, which would require just an extra volume mounted when
deployed on a Kubernetes cluster.

Final implementation

While being an SQL database, SQLite has the same concurrency model as Redis, at least to some
degree. In its default mode, readers block writers and writers block each other.

This is fine for KCX's use case. SQLite lets me:

Have a single deployable unit without an external dependency
Keep write volume low, since CI/CD jobs are naturally long running

However it is still worthwhile to apply some optimizations to improve performance:

Enable Write-Ahead Logging (WAL) mode so that writers don't block readers. Writers still block each other, however.
In order for a writer to write to the database it must acquire a file (database) level lock. Other writers will receive a busy error when attempting to write in such a situation. In order to avoid the error, we must set a busy_timeout which would allow the writers to wait a bit before throwing an error. Another solution is to allow a single database connection. In order for writers to not block readers - in WAL mode - we need a separate connection for reading.
Another situation in which we could have a busy error is if a transaction promotes from a simple read transaction to a write transaction due to the execution of a write operation. The busy_timeout option can solve this but it is better to avoid it altogether. To avoid the error we must start a transaction as a write transaction using BEGIN IMMEDIATE.

I wasn't able to find a reliable source for how much SQLite can process, but one blog post reports
requests in the order of tens/second.

I will be considering moving to another queue system in the future to improve performance, but
SQLite suits my purposes for now.

References

The Start

Younes Merzouka — Wed, 12 Aug 2026 22:55:35 +0000

Introduction

Hello!!!

So I decided to learn a bit more about CI/CD pipelines, and what better way to learn about them than to create your own!!! However, I also wanted to add my own spin to it: integrating it with Kubernetes and maybe Docker, to build a Kubernetes-native CI/CD tool. I'm not sure if the idea already exists out there, but I think it's worthwhile exploring for learning purposes.

Why the Name

The goal is to make use of what Kubernetes natively provides and build on top of it to create a more "Kubernetes-native" solution to CI/CD. Hence the first letter in KCX is Kubernetes.

Since this is a CI/CD tool, I reduced both into CX.

So in a sense, it's named Kubernetes CI/CD.

Why This DevLog

The goal behind this devlog is to:

Share my progress
Share any interesting ideas I run into while working on the tool
Share my experience working with Go
Engage with the community and listen for feedback on things I can improve, which is always welcome

What Will I Use to Build It?

Since many tools in the cloud space already use Go, I decided to follow their lead. This also helps me get more familiar with the language, since I've wanted to learn it for a while.

Any Progress?

So far I've started exploring a bit around the core ideas:

Cloning a repository and executing a simple build script
Creating a webhook and wiring it up with GitHub

I also tried out a distributed work queue setup — multiple workers wired to an API service through a RabbitMQ broker. After testing it, I decided to keep the tool minimal for now, with workers running as local threads rather than their own services. Eventually, I plan to have the tool support a distributed setup too.

Will I Use LLMs?

Not as of yet.

How Will These DevLogs Be Structured?

I'll aim for weekly. Whether I actually stick to that depends on what else I have going on — we'll see!

What's Next

Next up, I want to get a full local build running end-to-end, with support for both public and private repositories. I also want to test this across multiple platforms — GitLab, Codeberg, and GitHub — to make sure the tool isn't tied to just one ecosystem.

Microservice Data Handling — Saga Pattern

Younes Merzouka — Wed, 05 Aug 2026 15:47:50 +0000

Introduction

In a microservice architecture, it is often recommended that each service has its own separate
data storage. This is because the nature of microservices — and why they are created —
mandates such design.

Implementing such architecture would allow for many things, such as: separate schema updates,
the use of proper storage technologies for the task the microservice is trying to achieve, ...
But how can we handle data operations (retrieval/update/...) that span multiple services? For example:

A transaction requires multiple updates: update balance and place order
A transaction requires joins across different services: what is a customer's most commonly ordered items?

These data operations can fall under one of two categories: either we are trying to retrieve data
from multiple services and aggregate them, or we are trying to make modifications/deletions/insertions
that span multiple services.

In this post, we will focus on the latter — or to be more specific:
how do we handle cross-service transactions?

Transactions

Before we look into how to solve cross-service transactions, we first need to understand:
what are transactions?

Suppose you want to update a single row in a database: a customer's address. You would need to
execute a single instruction/operation:

UPDATE customers SET address='***' WHERE id='***';

Since this is a single operation, it would either succeed or fail. This is what we call an
atomic operation since it satisfies the all-or-nothing behaviour, meaning it either
succeeds in its entirety or fails in its entirety.

Now suppose it is our product's anniversary and we are feeling generous. We want to give our loyal
customers (those that stayed for at least 10 years) a staggering 1 dollar!!! as in-app credits.
The SQL for that is:

UPDATE customers SET balance = balance + 1 WHERE created_at < ***;

Suppose now that we have this amounts to 1000 updates, and the database server fails mid-operation.
This would leave the system in an inconsistent state: some of the customers wouldn't receive our
generous offer. This is an example of a non-atomic operation. In such situation, it is more
desirable to at least go back to the previous state and retry later.

Transactions aim to address the problem of atomicity. They allow you to execute multiple
instructions that either succeed all together, or rollback — allowing the preservation of the
consistency of the database.

Transactions in distributed systems

Now that we know what transactions are, how do they look when the operations need to happen across
different nodes connected through a network? Or more formally: how do they work in distributed systems?

But wait... what do distributed systems have to do with microservices? In some sense,
microservices are a kind of distributed system where nodes are services connected through a network.

Let us consider a scenario as an example to understand why distributed transactions differ from
traditional ones: a customer makes a purchase, and as such we need to insert a new row and update his
balance in an atomic manner. Why atomic, you might say? Suppose that we are able to insert a
purchase row, but the update to the customer balance fails; then it is as if the customer has
made a free purchase.

For a single database, the atomicity of the operation can be insured using the traditional transaction we talked
about earlier, that either succeeds or rolls back. But what if the customers and purchases
table live on different nodes? How can we coordinate the two nodes to ensure the atomicity of the operation?

This problem is not new, and databases have come up with multiple solutions for what is called
distributed transactions. A common example is Two-Phase Commit (or **2PC* for short*),
but it is out of the scope of this blog post.

"Ok then, problem solved!!!" you might say... but there are a few things that prevent us from
using this solution directly when it comes to microservice:

Even though some databases already have support for distributed transactions, in a microservice architecture, different services might use different storage systems (for example, MongoDB and Postgres), which prevents implementing them at the database layer. Using a single storage backend across all services is also not an option, since the whole point of the architecture is to allow agility and the use of the proper tool for the job.
Storage systems sit behind the services themselves, which prevents direct access and requires that we handle such transactions at the level of the services rather than leave it out for storage systems.
Implementing a fully blown 2PC distributed transaction is complex even for a single service, let alone across dozens or hundreds.

So to solve for this, we need a different method that allows for the all-or-nothing capability of
transactions, but that is simpler and implementable at the service level.

This is where the Saga pattern comes in...

The Saga Pattern

Let us take a simple example of a banking application:
We have two services:

A transactions service to handle all transactions done by customers.
A customers service which stores information about customers and their banking information — most importantly, their account balance.

Now suppose in our example that the customer makes a purchase to buy an AI subscription
(because of all of what he has being hearing about its productivity benefits).

The bank's backend receives the request for the purchase. In a normal monolithic application,
the process is simple: you would insert a new row in the transactions table referencing the customer,
update the customer's balance, wrapping everything in a transaction to make sure this is applied atomically.

But in a microservice architecture, this isn't so straightforward as discussed earlier.

Let us consider the happy path. The transactions service inserts a new row with a PENDING status.
It then synchronizes with the customers service, which handles the update to the customer row.
Upon confirmation of the update, the transactions row would update its own newly inserted row to COMPLETE,
for example.

Let us suppose that the transactions service fails to synchronize the update with the customers service,
or the customers service fails to inform the transactions service about its successful update.
This is where the rollback part of the saga comes in, where each saga step has a
(compensating transaction) which works by rolling back all previous steps if a single one fails.
This is in part similar to how a database does transactions, but only ensures eventual consistency
rather than the atomicity insured by the database's ACID transactions.

There are two ways of implementing Sagas and coordinating services:

Choreography: a service processes its part of the saga and publishes events that trigger updates to other services.
Orchestration: a central saga orchestrator manages the full saga, saving its state and rolling back appropriately on failure.

One important caveat about Sagas is that the local write and the network call need to be atomic
— often resolved using the **Transactional Outbox Pattern** — or else the system
will become stuck in an inconsistent state.

We also ignored one part, which is the client waiting for his transaction to end (to start his Agentic life).
The most plausible way of solving this — other than keeping him waiting — would be to provide a
transactionId to the customer and a special endpoint for him to poll to see if his purchase succeeded or not.

Do you know of any other patterns?

Picking a Package Manager for Your CI/CD Pipeline

Younes Merzouka — Sun, 15 Mar 2026 22:12:22 +0000

If you want to ship fast then CI/CD is the most important thing for you to consider. But have you ever thought "Does the package manager I use really matter for CI/CD?". This might not be an issue for other languages like Python, but JavaScript does have a few options to choose from: some claiming to be the fastest, others prioritizing developer experience and disk usage, and then there's good old NPM.

In this blog post we'll look into different package managers, examples of how to use each, and how well each performs when it comes to CI/CD pipelines. Of course, with everybody's favorite: Benchmarks.

The package managers we will discuss are PNPM, NPM, Yarn and Bun.

Setup

The application used is the standard NestJS starter app, running on a GitLab CI pipeline. Below are the Dockerfile for each of the package managers with a brief explanation of the flags used and some extra info.

Note: We will not focus on securing the images for now, so the examples don't include any security measures. Distroless is used to minimize the size of the final container image.

NPM

The ci option ensures that package-lock.json is used, preventing any accidental changes to dependency versions. --omit=dev ensures only production dependencies are installed.

FROM node:22.22.1-slim AS base
WORKDIR /app

FROM base AS install
COPY ./package.json ./package-lock.json ./
RUN npm ci

FROM install AS build
COPY . .
RUN npm run build

FROM base AS deps
COPY ./package.json ./package-lock.json ./
RUN npm ci --omit=dev


FROM gcr.io/distroless/nodejs24-debian13 AS prod
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=deps /app/node_modules ./node_modules/
EXPOSE 3000
CMD [ "/app/dist/main.js" ]

PNPM

--frozen-lockfile has the same effect as npm ci — it locks the install to what's in the lockfile. The PNPM global store is placed at /pnpm inside the Docker image, configured via PNPM_HOME.

FROM node:22.22.1-slim AS base
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
RUN corepack enable
WORKDIR /app

FROM base AS install
COPY ./package.json ./pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile

FROM install AS build
COPY . .
RUN pnpm run build

FROM base AS deps
COPY ./package.json ./pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile --prod


FROM gcr.io/distroless/nodejs24-debian13 AS prod
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=deps /app/node_modules ./node_modules/
EXPOSE 3000
CMD [ "dist/main.js" ]

Yarn

The default Yarn configuration that comes with NestJS uses the old classic version rather than the modern one with Plug'n'Play. To upgrade, run yarn set version stable, then enable PnP with yarn config set nodeLinker pnp — this skips node_modules entirely. Then install with yarn install. For the rest of this post, "Yarn" refers to this latest Plug'n'Play version.

FROM node:22.22.1-slim AS base
WORKDIR /app
RUN corepack enable

FROM base AS build
COPY package.json yarn.lock .yarnrc.yml ./
RUN yarn install --immutable
COPY . .
RUN yarn build

FROM base AS deps
COPY package.json yarn.lock ./
RUN corepack enable \
&& yarn workspaces focus --production

FROM gcr.io/distroless/nodejs24-debian13:debug AS prod
WORKDIR /app
COPY --from=deps /root/.yarn/berry/cache /root/.yarn/berry/cache
COPY --from=deps /app/.pnp.cjs /app/.pnp.loader.mjs ./
COPY --from=deps /app/.yarn ./.yarn
COPY --from=build /app/dist ./dist
EXPOSE 3000
CMD [ "--require", "./.pnp.cjs", "dist/main.js" ]

Bun

FROM docker.io/oven/bun:1.3-alpine AS base
WORKDIR /usr/src/app

FROM base AS install
COPY ./bun.lock ./package.json ./
RUN bun install --frozen-lockfile

FROM install AS build
COPY . .
RUN bun run build

FROM docker.io/oven/bun:1.3-alpine AS prod
WORKDIR /app
COPY --from=build /usr/src/app/dist ./dist
COPY --from=install /usr/src/app/node_modules ./node_modules
EXPOSE 3000
ENTRYPOINT [ "bun", "dist/main.js" ]

How Package Managers Work — and Why It Matters for CI

Before jumping to the benchmarks, it's worth understanding what actually differentiates these package managers under the hood, since their design choices directly affect both install speed and image size.

PNPM

PNPM's main goal is solving the disk space problem that node_modules are notorious for. On a typical machine, every project gets its own full copy of every dependency — meaning if you have ten projects all using React, you have ten copies of React on disk.

PNPM solves this by maintaining a single global content-addressable store (pointed to by $PNPM_HOME), where every package file is stored exactly once. Each project's node_modules then contains hard links pointing back to that central store rather than actual copies of the files. To keep the store lean, PNPM also tracks file-level changes between package versions using hashes — only storing the differing files, not a full new copy of the package.

Beyond disk savings, PNPM also addresses a subtle but important issue called phantom packages. Here's the problem: NPM has historically hoisted dependencies — meaning if your dependency express depends on lodash, NPM would move lodash up into the root node_modules folder, making it accidentally importable in your own code. Your project would work even though you never listed lodash in package.json. If express later drops it or changes its version, your code silently breaks.

PNPM avoids this by keeping each package's dependencies isolated under node_modules/.pnpm/package@version/node_modules/, and exposing only your direct dependencies as symlinks at the root of node_modules. This makes the dependency tree accurate and predictable.

What this means for Docker

While PNPM's central store is extremely useful during development — especially in monorepos — it doesn't give you the same benefit inside a Docker container. Each container is its own isolated environment with its own dependencies, so the global store never gets reused across builds the way it would on a developer machine. In practice, PNPM ends up behaving similarly to NPM in a container context.

Bun

Bun takes a fundamentally different approach to performance: it treats package installation as a systems problem rather than a JavaScript problem.

The core of this is Bun being written in Zig, a compiled systems language, rather than JavaScript. This alone removes a significant layer of overhead. But Bun also targets three specific bottlenecks that slow down Node.js-based package managers:

System calls: Node.js uses libuv, a C library, to make OS-level calls for things like reading files and managing threads. Each call involves some compatibility overhead, and Node.js adds further overhead managing its thread pools. Bun calls the OS more directly and with fewer round-trips. You can see this concretely in the strace output below, from Bun's blog:

Benchmark 1: strace -c -f npm install
    Time (mean ± σ):  37.245 s ±  2.134 s [User: 8.432 s, System: 4.821 s]
    Range (min … max):   34.891 s … 41.203 s    10 runs

    System calls: 996,978 total (108,775 errors)
    Top syscalls: futex (663,158),  write (109,412), epoll_pwait (54,496)

  Benchmark 2: strace -c -f bun install
    Time (mean ± σ):      5.612 s ±  0.287 s [User: 2.134 s, System: 1.892 s]
    Range (min … max):    5.238 s …  6.102 s    10 runs

    System calls: 165,743 total (3,131 errors)
    Top syscalls: openat(45,348), futex (762), epoll_pwait2 (298)

Nearly 6× fewer system calls.

Manifest parsing: NPM reads package metadata from human-readable formats like JSON and YAML, which need to be parsed on every install. Bun stores this metadata in a binary format that can be loaded directly without parsing.

Decompression: When downloading packages (which are compressed tarballs), most tools decompress on the fly, guessing at buffer sizes and reallocating memory as data streams in. Bun instead downloads the full compressed file first, determines the exact buffer size needed using the gzip format, allocates it once, and then decompresses — eliminating unnecessary memory copies.

Bun also starts DNS lookups while reading package.json rather than waiting until after, trimming latency at the very start of the install process. There are a few more optimizations covered in the references if you want to go deeper.

Yarn

Yarn's Plug'n'Play mode works differently from both NPM and PNPM. Rather than populating a node_modules folder at all, Yarn downloads packages as .zip archives and stores them in a central cache folder (viewable via yarn config get cacheFolder — in our container, this is /root/.yarn/berry/cache).

At runtime, instead of Node.js resolving modules from node_modules, Yarn injects a custom loader (.pnp.cjs) that intercepts require() calls and loads modules directly from the .zip archives. The current install state is saved in .yarn/install-state.gz, and the package cache lives in .yarn/cache.

This design means no node_modules directory is created when you run yarn install — which also means no hoisting, no phantom packages, and significantly less I/O.

Why Yarn performed so well

The architecture above translates directly into install speed. Yarn only needs to download and archive packages, then generate the loader — no creating thousands of symlinks or hard links per file, simpler hoisting logic, no repeated I/O across a deep directory tree, and proper tracking of your project's dependencies. For a fresh Docker build where none of those steps can be cached, that's a meaningful advantage.

Result

Package Manager	Run 1 (f813c0c5)	Run 2 (ecb90ce2)	Run 3 (0938ee84)	Average Time	Average Size (MiB)
yarn	1:20	1:23	1:14	1:19	58.26
npm	1:49	2:06	1:51	1:55	57.32
pnpm	1:55	1:54	2:12	2:00	57.32
bun	2:06	2:01	2:14	2:07	72.14

Even though I personally expected Bun to win this, it ended up performing on par with NPM — and the extra few MiB don't really justify the switch. So the winner here is Yarn, by a clear margin.

I did some digging trying to explain why Bun underperformed, but couldn't find a satisfying answer. I'd be very interested in hearing your thoughts in the comments, and depending on the discussion, a deeper dive might be worth its own post.

References

What Rust Does Differently: A Beginner's Perspective

Younes Merzouka — Sat, 07 Mar 2026 22:11:29 +0000

Rust makes a lot of interesting decisions about how things work — what you can and cannot do, and how you do certain things. Coming from a mostly web development background, three concepts in particular caught my attention as a beginner: Mutability, Ownership, and Handling Null.

Mutability

Rust variables are immutable by default. So code like this in JavaScript:

let a = 0;
a += 1;

...wouldn't work in Rust:

let a = 0;
a += 1; // compiler error

To mutate a variable, you need to add the mut keyword:

let mut a = 0;
a += 1;

The reason is safety. This mechanism forces you to explicitly opt into mutability, which makes your code more deterministic and avoids problems — especially with concurrency, where two threads might otherwise modify the same variable without you realising it.

Mutable References

One thing that tripped me up early on was mutable references. Suppose we want to read from stdin:

let mut s = String::new();
io::stdin().read_line(&s); // passing a reference to s

This looks reasonable — similar to how you'd do it in C. But it actually results in a compiler error telling you that read_line expects a mutable reference. The correct version is:

let mut s = String::new();
io::stdin().read_line(&mut s);

You have to explicitly tell the compiler that the reference itself is mutable. Why this matters will become clearer once we talk about ownership.

Ownership

Managing allocated memory is a well-known challenge in systems programming, and there are a few common approaches:

Garbage collection — many high-level languages manage memory for you through a GC. This simplifies things for developers but can impact performance through "GC pauses", where the entire application halts while unused memory is freed. Not ideal for systems software.
Manual memory management — the approach C takes. You allocate memory and you're responsible for freeing it. This has historically led to many bugs and security vulnerabilities: null pointer dereferences, buffer overflows, use-after-free, and so on.

Rust takes a different approach through ownership.

Consider this example:

fn add_suffix(mut s: String, suf: &str) {
    s.push_str(suf);
    drop(s); // frees the heap memory
}

fn main() {
    let mut s = String::from("Hello, ");
    add_suffix(s, "World");
    println!("{}", s); // compiler error
}

The function add_suffix receives a String allocated on the heap, drops it (equivalent to free in C), and then main tries to access that freed memory — a classic use-after-free bug. Rust won't let this compile.

You might think the fix is simply to remove the drop:

fn add_suffix(mut s: String, suf: &str) {
    s.push_str(suf);
}

fn main() {
    let mut s = String::from("Hello, ");
    add_suffix(s, "World");
    println!("{}", s); // still a compiler error
}

But this still doesn't work. Passing s to add_suffix as an argument transfers ownership of s to that function.

So what does transferring ownership actually mean? In Rust, each heap-allocated value is owned by the scope in which it was created. When that scope ends, the memory is automatically freed. So when add_suffix returns, s is freed — and trying to use it in println! afterwards is accessing a dead memory region.

Here's another example of the same idea:

let p: &String;
{
    let s = String::from("Hello");
    p = &s;
} // s is dropped here, memory is freed
println!("{}", *p); // compiler error: p points to freed memory

The correct way to achieve what we originally wanted is through borrowing:

fn add_suffix(s: &mut String, suf: &str) { // takes a mutable reference
    s.push_str(suf);
}

fn main() {
    let mut s = String::from("Hello, ");
    add_suffix(&mut s, "World"); // lends s to the function
    println!("{}", s); // s is still valid here
}

By passing a reference (&mut s), main remains the owner of the memory. The function simply borrows it temporarily. When the function returns, the borrow ends — but the memory isn't freed, because main still owns it.

This same principle applies even without mutation. If you don't need to mutate the value, you use & instead of &mut. Now the reason you need &mut for mutable references (as we saw with read_line) should make a lot more sense.

Option

Rust doesn't have null. But there are situations where you genuinely want to express: "this function either returns a result, or nothing — not because of an error, but because there's simply nothing to return."

Consider this function:

fn get_slice_by_char(s: &str, needle: char) -> &str {
    for (i, c) in s.char_indices() {
        if c == needle {
            return &s[i..];
        }
    }
    return ?; // what do we return if the character isn't found?
}

null might seem useful here. For exactly these cases, Rust has Option<T>:

fn get_slice_by_char(s: &str, needle: char) -> Option<&str> {
    for (i, c) in s.char_indices() {
        if c == needle {
            return Some(&s[i..]);
        }
    }
    None
}

What makes Option<T> special is that it addresses the classic problem with null: forgetting to handle it. Option<T> forces you to explicitly deal with both cases. For example, this won't compile:

fn main() {
    let s: &str = get_slice_by_char("hello, there", 't');
    // compiler error: expected &str, found Option<&str>
}

The correct approach is to use match (similar to a switch statement) to handle both possibilities:

fn main() {
    let s: &str = match get_slice_by_char("hello, there", 't') {
        Some(slice) => slice,
        None => panic!("Character not found"),
    };
}

The compiler won't let you use the value inside an Option without first unpacking it — which means you can never accidentally forget to handle the "nothing" case.

These three concepts — mutability, ownership, and Option — felt strange to me at first coming from other languages. But once they click, you start to see how they all work together to make Rust code safe without needing a garbage collector. Pretty elegant, honestly.