DEV Community: Mohamed ABDELLANI

The Journey of a Request in a Raft-Based KV Store

Mohamed ABDELLANI — Wed, 01 Apr 2026 12:43:14 +0000

Originally published on:
https://abdellani.dev/posts/2026-04-01-the-journey-of-a-request-in-a-raft-based-kv-store/

Introduction:

This article comes after implementing labs 3 and 4 from the open MIT 6.584 course. We have a replicated KV server on top of the RAFT protocol, and we want to understand how the server works internally. To do so, I propose to follow all the steps that a request will pass through on the server before a client receives a response.

The objective of building the KV server on top of the RAFT protocol is to have a fault-tolerant system.
The system is composed of multiple KV server nodes, if some servers fall down, the system should continue serving the clients.
Raft is strongly consistent and can tolerate the failure of f nodes in a system composed of 2 * f + 1 nodes. For example, if we want the system to tolerate the failure of 2 nodes, we’ll need 5 nodes.

The labs are testing the implementation under different conditions:

an unreliable network that drops packets or delivers packets out of order
Creating network partitions
Crashing, restarting servers.

The system should work correctly under those conditions.

System design

Each server is composed of 3 layers:

1. Application layer

It’s responsible for handling the RPC calls coming from the clients, and for handling the logic related to the Key/Value store: read, write operations related to the keys. It communicates directly to the RSM layer and is not aware of the RAFT.
When the client sends a request, this layer will make a submit call to the RSM layer and wait for it to return before responding to the client.

2. RSM layer

The RSM layer is the glue between the Application layer and the raft layer.
When a request is submitted from the application layer, the request will be forwarded to the RAFT layer.
The RSM also monitors the apply Channel. When Raft decides to commit a command, the RSM layer forwards the command to the application layer to execute it.

3. Raft Layer

The Raft consensus protocol layer is responsible for leader elections and log replication.

When the client sends a request to the server, the server must be a leader or will reject the request.
The initial simple version of the follow consists of the following steps:

The application layer will submit the request to the RSM layer and wait for the function call to return before sending an answer to the client.
The RSM layer will add the client to its internal store and submit the request to the RAFT protocol. If the current server is not the leader, the command will be rejected at this level.
The Raft will append the command to its log and replicate the command to the other servers. The client must wait until the command is committed and applied before receiving a response.
If the command is replicated to a majority, and the current term condition is satisfied, the Raft protocol will decide to commit the command.
The RSM monitoring the apply channel, the channel used by the Raft layer to send the commands that are committed, will forward the command to the application layer by calling DoOp.
The application layer will update the Key/Value store state. For example, put(x,5) will set the key x to 5 at this stage.
The RSM receives the DoOp’s result from the application layer, checks its state to find the client waiting on step 2, and communicates the results to the client.

At first glance, this flow looks correct. In practice, it breaks in several ways.

1. The server loses the leadership before responding to a client request.

The client submits a request, the leader node receives it, submits it to the RSM, and then loses the leadership immediately. This can happen due to network issues between nodes.
In this situation, the client will be stuck waiting for a response while the command might be removed from the Raft logs since it’s not committed.
To solve this problem, the RSM needs to check periodically with the RAFT layer if the server is still the leader. If the server is no longer the leader, it’ll send an error code “ErrWrongLeader” to the application layer, which in its turn will forward it to the client.
This will tell the client that the targeted server is not the leader, and it has to try submitting its requests to another server.

2. The leader loses communication with the other nodes.

A client X sends its request to a leader. The leader receives the request and submits it to the Raft layer. The problem is that the leader is no longer able to reach the other nodes to replicate the command in the request.
The other nodes will notice the absence of the leader, run a new election, and select a new leader. The system will serve the other clients, but the initial leader will not notice the changes in the leadership and will assume that it’s still the current leader.
The client X will be stuck waiting for the old leader to replicate and commit the command.
To avoid this situation, different solutions can be implemented.
A leader can monitor its communications with the other nodes, and if it notices that none of the other nodes is reachable for a period of time, it’ll automatically increment the term and downgrade to a follower.
A simpler solution will be to implement a timer on the RSM level. After submitting a command to Raft, the RSM will use that timer to decide if enough time has passed to release the client and ask it to try with other servers.

3. The client sends a request, but the network fails.

The failure can happen before the server receives the requests, or when the server processes the request and tries to send a response.
The client might retry again and again, which can lead to overwriting the commands of the other clients.

One solution is to attach to every request two attributes: a client ID and a request ID.
The server will use those attributes to :

Track the requests' execution progress for each client, and
Implement a cache

The server will maintain two data structures.

latest request ID per client
latest response per client

In my implementation, I decided to implement the logic for checking and updating the request IDs tracking data structure and the cache on the DoOp function. This will keep RSM layer more flexible to be integrated with a different application layer. Also, we can not rely on the RPC handler to implement that logic. A follower node will only call the DoOp and will never use the RPC handler after processing a committed command. We want every node to keep track of those two data structures.

An optional optimization will be to check the cache on the RPC handler level before submitting the lower layers.

When the server receives a request from the client:

If the request ID is older than the last request ID per that client, the request will be ignored.
If the request ID equals the latest request ID per that client, return the result from the cache.
If the request ID is greater than the last request ID per that client, submit the request to RSM, then RAFT, update the two data structures, and then send the results to the client.

Also, it’s not necessary to persist the cache or the latest request IDs data structure, but they have to be part of the snapshots.
When a server restarts, after exchanging a few messages with the leader, it’ll learn about the commit index, replay the commands, and rebuild the cache and the request IDs.
If logs are compacted without adding the request IDs and the cache, there will be no way to rebuild them again.

Conclusion

What looks like a simple request flow might hide many complexities in practice. Most of the complexity is not in Raft itself, but in making the system behave correctly under failures.

Why Raft Can’t Safely Commit Old-Term Entries

Mohamed ABDELLANI — Mon, 23 Mar 2026 18:00:24 +0000

Originally published on:
https://abdellani.dev/posts/2026-03-23-why-raft-cant-safely-commit-old-term-entries/

Raft is a consensus algorithm that allows a group of servers to agree on the order of a sequence of commands.

The objective of this article is not to re-explain RAFT, as this work has already been done in several other places, and the RAFT paper gives enough details to implement it. Instead, my focus will be on two problems that I faced when I voluntarily studied the open distributed systems course from MIT (course code 6584, I didn’t enroll in the course officially, and I wasn’t a student of MIT).

The first issue:

The paper specifies that a node should not commit the commands from previous terms directly. Those commands should be committed indirectly by committing the commands from the leader’s current term.

At that level, I was not convinced by this constraint, especially after seeing that the initial tests were passing. A simple scenario might be:

Suppose we have 3 servers: A, B, and C.

In term 1, A becomes a leader.
A adds a command X and replicates it to B.
A crashes before committing the command.

The command X can’t be removed because the C server can’t become a leader in term 2. After all, its log isn’t as up to date as B's. The B server will refuse to vote for C.

The only valid next step is for B to become a leader and continue replicating the command X to C.

The answer was in a more complex scenario where the leader replicates its entries through several terms. In such a case, even if the entry is replicated on a majority, it still can be lost. This will break the safety rule that says: if a command is committed, it should not be lost.

Let’s take a concrete example and see how this can happen.

Suppose we have three servers: A, B, and C.

The A server is the leader in term 1
The A server receives the first command X, persists it, and crashes before replicating it.
The C server becomes a leader in term 2
The C server receives a command Y, persists it, and crashes before replicating it
The B server can’t make any progress until at least one server reconnects. The system tolerates the loss of one server at most at the same time.
The A server reconnects and takes the leadership in term 3 as its log is more updated than B’s log.
The A server replicates the command X to B. According to the paper, the command should not be committed.
The server A crashes again.
The C server reconnects.
At this stage, the B server will have command X from term 1, and the C server will have command Y from term 2. This means the C server will take the lead as its last log entry belongs to a more recent term.
The C server will force the B server to remove the command X, and replace it with command Y
The A server reconnects again. As a follower, it’ll be forced to remove the command X and replace it with Y

In term 3, the command X was replicated on the majority of servers (A & B). Yet in term 4, the command X was completely erased from logs. So the majority replication alone is not enough to guarantee safety

The following figures illustrate the example:

After understanding how a command replicated on a majority can still be lost after a few terms, that’s why the leader should never commit commands from previous terms.

Now, we can discuss the next question. Raft allows only a leader to consider an entry committed if it is in the current term and replicated on a majority. In term 4, from the previous figure, the server S3 has its entry replicated to a majority, but still can’t commit the command because the latest entry is from term 2.

As the protocol authorizes only to commit entries from the current term, one valid progress path will:

The C server receives a new command Z from the client. This command will be labeled term 4.
The C server replicates the new command to a majority (i.e., one additional node to reach a majority)
Once the C server confirms that the new command is replicated, it can commit the new command confidently. The first command will be automatically committed, as all the entries before the commit index will be considered as committed.

The question that arises at this stage will be: what if the client doesn’t submit a new command? The S3 server will not be able to commit, and the system will not make any progress.

To avoid falling into a situation where the system freezes for a long time, when a node is promoted to a leader, it adds a new command. A command that doesn’t do anything, a no-operation or ”NOOP”. It’ll be replicated as any other commands, it’ll have an index, it’ll be committed, and sent on the apply channel. The higher layer doesn’t need to send it to the application layer. The application layer doesn’t need to learn about it or need to know how to interpret it.

The figure shows how the leader replicates the NOOP to the other node. Since NOOP’s term is the same as the current term of the leader, the leader can commit without facing the risk of losing the command in the future.

In conclusion, this shows why RAFT restricts to the current term, and why the NOOP commands are a practical way for a leader to safely make progress.

References:

http://nil.csail.mit.edu/6.5840/2025/papers/raft-extended.pdf

Exploring Prometheus Code - part 1 (personal notes)

Mohamed ABDELLANI — Tue, 19 Dec 2023 08:20:27 +0000

I recently started learning the Go language and part of the process was to explore the code of popular open-source projects. Prometheus is one of the projects that I selected. It's a popular project with 848 contributors and more than 51k starts.

The objectives of this article:

understand how the code is organized in the filesystem,
explore some of the popular packages used on this project,
understand how the different modules are loaded
understand how the communication between the modules is implemented.

I started with the /cmd folder, as it stores the main function that will be called when we run Prometheus from the CLI.

Before reading that function, it'll be useful to learn about the following packages:

kingpin: to parse the command line arguments
run: A universal mechanism to manage goroutine lifecycles (from the repo description). check reference
context: to share state between function/goroutines. check reference

The main function does the following things:

parsing arguments and load configuration and populate the data structure flagConfig\ (using kingpin package)

    a.Flag("config.file", "Prometheus configuration file path.").
        Default("prometheus.yml").StringVar(&cfg.configFile)

    a.Flag("web.listen-address", "Address to listen on for UI, API, and telemetry.").
        Default("0.0.0.0:9090").StringVar(&cfg.web.ListenAddress)

checking the consistency between the configurations

    if agentMode && len(serverOnlyFlags) > 0 {
        fmt.Fprintf(os.Stderr, "The following flag(s) can not be used in agent mode: %q", serverOnlyFlags)
        os.Exit(3)
    }

Launching the different modules (using run package)

var g run.Group
...
    {
        // Scrape manager.
        g.Add(
            func() error {
                <-reloadReady.C
                err := scrapeManager.Run(discoveryManagerScrape.SyncCh())
                level.Info(logger).Log("msg", "Scrape manager stopped")
                return err
            },
            func(err error) {
                level.Info(logger).Log("msg", "Stopping scrape manager...")
                scrapeManager.Stop()
            },
        )
    }
...

the project is composed of several modules, each module is stored in a folder at the root level of the project. (internal architecture)

I started with the two modules: discovery and scrape.

discovery implements the service discovery (ScrapeDiscoveryManager) that collects the target groups and sends them to scrape

scrape implement the scrape manager (ScrapeManager) that controls the scraper pools (check internal architecture for details).

Each of those two managers will run in a different go-routine.

ScrapeDiscoveryManager will send the target groups to the ScrapeManager through a sync channel (SyncCh). This channel will be passed on the creation time in the main function.

        g.Add(
            func() error {
                // When the scrape manager receives a new targets list
                // it needs to read a valid config for each job.
                // It depends on the config being in sync with the discovery manager so
                // we wait until the config is fully loaded.
                <-reloadReady.C

                err := scrapeManager.Run(discoveryManagerScrape.SyncCh())
                return err
            },

the ScrapeManager#Run runs a loop that will watch for updates from the ScrapeDiscoveryManager and update the scraper pools accordingly.

// tsets is returned channel from discoveryManagerScrape.SyncCh()
func (m *Manager) Run(tsets <-chan map[string][]*targetgroup.Group) error {
    go m.reloader()
    for {
        select {
        case ts := <-tsets: // New Updates!!
            m.updateTsets(ts)

            select {
            case m.triggerReload <- struct{}{}: // trigger reload
            default:
            }

        case <-m.graceShut:
            return nil
        }
    }
}

the reloader will reload that will create/update the scraper pools

func (m *Manager) reloader() {
...
    ticker := time.NewTicker(time.Duration(reloadIntervalDuration))

    defer ticker.Stop()

    for {
        select {
        case <-m.graceShut:
            return
        case <-ticker.C:
            select {
            case <-m.triggerReload:
                m.reload() // Update the scraper pools
            case <-m.graceShut:
                return
            }
        }
    }
}

Now, let's return to ScrapeDiscoveryManager , to understand how it works we need to explore the data structure (some details are omitted).

//discovery/manager.go
type Manager struct {
    targets //  contains the list of all the targets
    providers []*Provider 
    syncCh // the communication channel to send targets to ScrapeManager
}

type Provider struct {
    name   string
    d      Discoverer
//....
}
type Discoverer interface {
    // Run hands a channel to the discovery provider (Consul, DNS, etc.) through which
    // it can send updated target groups.
    Run(ctx context.Context, up chan<- []*targetgroup.Group)
}

The manager has a list of providers, each one includes a discoverer that'll periodically create lists of targets and send them to the discovery manager.

We can see that on the ScrapeDiscoveryManager\'s startProvider an update channel is created and passed to the discoverer and the updater .

func (m *Manager) startProvider(ctx context.Context, p *Provider) {
    level.Debug(m.logger).Log("msg", "Starting provider", "provider", p.name, "subs", fmt.Sprintf("%v", p.subs))
    ctx, cancel := context.WithCancel(ctx)
    updates := make(chan []*targetgroup.Group)

    p.cancel = cancel

    go p.d.Run(ctx, updates)
    go m.updater(ctx, p, updates)
}

Each discoverer implements the function refresh function that will collect data and send it. (Some discoverers interact with the provider through refresh/refresh ).

The discoverer's logic can be summarized as follows:

 func (d *Discovery) Run(ctx context.Context, ch chan<- []*targetgroup.Group) {
    ticker := time.NewTicker(d.interval)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            tgs, err := d.refresh(ctx)
            if err != nil {
                if !errors.Is(ctx.Err(), context.Canceled) {
                    level.Error(d.logger).Log("msg", "Unable to refresh target groups", "err", err.Error())
                }
                continue
            }

            select {
            case ch <- tgs:
            case <-ctx.Done():
                return
            }
        case <-ctx.Done():
            return
        }
    }
}

Now on ScrapeDiscoveryManager, when updater receives new updates from the discoverer, it'll

update the statistics
update the list of targets
trigger send: unblock the sender (check ScrapeDiscoveryManager'Run below) loop to send the update to the ScrapeManager

func (m *Manager) updater(ctx context.Context, p *Provider, updates chan []*targetgroup.Group) {
    // Ensure targets from this provider are cleaned up.
    defer m.cleaner(p)
    for {
        select {
        case <-ctx.Done():
            return
        case tgs, ok := <-updates:
            receivedUpdates.WithLabelValues(m.name).Inc()
            //...
            p.mu.RLock()
            for s := range p.subs {
                m.updateGroup(poolKey{setName: s, provider: p.name}, tgs)
            }
            p.mu.RUnlock()

            select {
            case m.triggerSend <- struct{}{}:
            default:
            }
        }
    }
}

The ScrapeDiscoveryManager 's Run method will run the sender function which is responsible for sending updates to the ScrapeManager

func (m *Manager) Run() error {
    go m.sender() // send updates to ScrapeManager
    <-m.ctx.Done()
    m.cancelDiscoverers()
    return m.ctx.Err()
}

func (m *Manager) sender() {
    ticker := time.NewTicker(m.updatert)
    defer ticker.Stop()

    for {
        select {
        case <-m.ctx.Done():
            return
        case <-ticker.C: // Some discoverers send updates too often, so we throttle these with the ticker.
            select {
            case <-m.triggerSend:
                sentUpdates.WithLabelValues(m.name).Inc()
                select {
                case m.syncCh <- m.allGroups():
                  //...
            default:
            }
        }
    }
}

The diagram below summarizes the communication flow between the discoverer to the scrape manager

Github Actions to run Rubocop and RSpec tests on Rails with Postgres

Mohamed ABDELLANI — Mon, 27 Jan 2020 09:29:56 +0000

Hi,

As my first post in this community, I decided to share my experience when I was testing Github's actions. I hope that you'll find it useful.

Objectif

My goal was to define a workflow composed of three steps for testing a rails application. These steps are:

Using Rubocup, the code will be statically analyzed against the Ruby style guide.
If the code passes the previous step, the RSpec tests are triggered to check if the new committed code didn't break anything somewhere else.
Finally, if all tests pass, you will get a report generated using SimpleCov to see how well the tests covered the application's code

The demo application

You can find the code that I used for the demo in this github repository. The application is a pretty simple, one "Post" model with two attributed "title" and "content". Postgres is the database to store the data.

if you check config/database.yml, you'll find that I have solely edited the section related to the testing database as I didn't need to run the application in the development environment.

#config/database.yml
test:
<<: *default
host: localhost
username: postgres
database: rails-github-action_test

Setting up the workflow

Create `.github/workflows`

The first step is to create a folder called .github/workflows in the root of your project folder.

Create the file that will describe the workflow

In this file, you write all the details need to run the workflow. The extension must .yml or .yaml.

Define the environment variables

This section is not required but it makes the reconfiguration of the workflow for other contexts easier. This section should start with the keyword env.

env:
  RUBY_VERSION: 2.4
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: ""
  POSTGRES_DB: postgres

RUBY_VERSION will used to specify which version of ruby to use in the environment.
POSTGRES_USER, POSTGRES_PASSWORD and POSTGRES_DB are the database related information.

Define the name of the workflow and when it should be triggered

name: Rails tests
on: [push,pull_request]
jobs:
  job1:
   ...
  job2:
   ...

name is the name of the workflow that will appear in the Github Actions UI.
on specify what are the events that will trigger the workflow. In this example, it'll be triggered on push and pull_request events.
jobs will contain all the tasks to be defined in the workflow.

Describe jobs

Rubocop test

jobs:
  rubocop-test:
    name: Rubocop
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v1
      - uses: actions/setup-ruby@v1
        with:
          ruby-version: ${{ env.RUBY_VERSION }}
      - name: Install Rubocop
        run: gem install rubocop
      - name: Check code
        run: rubocop

name is the name of the task.
runs-on specifies which operating system will be used to run the task. In addition to ubuntu-18.04, you can use macOS and Windows Server 2019. For more details, you can check this link.
In the section steps, we'll define the instructions that will run in the operating system in order to prepare it, and make it run the tests. In this example :
- We call the action actions/checkout@v1 which necessary to check out the repository.
- We call the action actions/setup-ruby@v1 to install ruby,
- We use run to run the command to install rubocop gem install rubocop,
- And finally, we run the test using rubocop. you can learn more about actions here.

If the code passes the test, you'll get something like this :

RSpec tests

For the RSpect test, We use a very similar code. The first difference is that we used needs to precise that the task should run only after the task rubocop-test is finished and passed successfully.

job:
 ...
   rspec-test:
    name: RSpec
    needs: rubocop-test
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v1
      - uses: actions/setup-ruby@v1
        with:
          ruby-version: ${{ env.RUBY_VERSION }}
      - name: Install postgres client
        run: sudo apt-get install libpq-dev
      - name: Install dependencies
        run: |
          gem install bundler
          bundler install
      - name: Create database
        run: |
          bundler exec rails db:create RAILS_ENV=test
          bundler exec rails db:migrate RAILS_ENV=test
      - name: Run tests
        run: bundler exec rake

As libpq-dev is a required package to allow the rails application to communicate with Postgres. We added a step to install it using apt-get install libpq-dev.
To create the database and the tables, we defined the step Create database
To run the test, we used the command bundler exec rake

At this level, if we run the tests they'll fail. The reason is that we didn't create database service yet. We need to define a new section services inside the rspec-test job.

    services:
      postgres:
        image: postgres:latest
        ports:
        - 5432:5432

image is the docker image name.
port maps the local ports to the container's ports.

 rspec-test:
    name: RSpec
    needs: rubocop-test
    runs-on: ubuntu-18.04
    services:
      postgres:
        image: postgres:latest
        ports:
        - 5432:5432
    steps:
      - uses: actions/checkout@v1
...

SampleCov report

Now, we add the following step in order to upload the SimpleCov report from the testing environment and make it available to developers.

      - name: Upload coverage results    
        uses: actions/upload-artifact@master
        if: always()
        with:
          name: coverage-report
          path: coverage

we are using the action actions/upload-artifact@master to upload the report.
path is the path of the folder/file to upload
name is the name of the downloadable file.

If your code passes the tests, you'll be able to download the report from the link in the upper-right corner.

You can check the full code in this link.