DEV Community: cppchedy

Writing your first LLVM Transformation Passes - Dead Code Elimination and Constant Propagation

cppchedy — Sun, 16 Nov 2025 22:40:52 +0000

Introduction

In the previous article, we learned about LLVM passes and wrote our first analysis pass, which counted the number of add instructions inside a module. In this new post, I will explore more of LLVM by implementing transformation passes that make use of several LLVM APIs. This will expose us to different ways of processing the IR and highlight the facilities LLVM provides to its users.

The good news is that this article does not require much theory so we will dive directly into the code. Note that I will not be showing all of the pass code (specifically, the registration boilerplate), but I will provide a Gist link for the sake of completeness.

Dead Code Elimination

Dead Code Elimination (DCE) is one of the first optimizations a compiler applies to remove unused results and regions of code that are never reached. For example, expressions that appear after a return statement or conditions that always evaluate to false.

To keep things simple, I am redefining DCE for our beginner-level purposes. Our pass will remove only unused instructions inside a basic block and will not handle unreachable code.

Concretely, consider the following IR code:

define i32 @multip(i32 %a) {
entry:
  %unused = add i32 %a, %a
  %res = mul i32 %a, %a
  ret i32 %res
}

where we take an i32, add it to itself, and then multiply the original value (not the result of the addition) by itself before returning the result of that multiplication. When we apply our simplified DCE pass, the resulting code contains only the multiplication instruction, as shown in the following listing:

define i32 @multip(i32 %a) {
entry:
  %res = mul i32 %a, %a
  ret i32 %res
}

Our Downsized DCE Pass

As mentioned in the introduction, I will only be showing the pass class and not the registration code. Following what we discussed in the previous subsection, here is what our pass looks like:

struct DCECustomPass : PassInfoMixin<DCECustomPass> {
  void removeUnusedResultsIfAny(BasicBlock* BB) {
    for (auto it = BB->begin(); it != BB->end();++it) {
        Instruction *It = &*it;
        if(!It->isTerminator() && It->use_empty()) {
          It->eraseFromParent();
        }
    }
  }

  PreservedAnalyses run(Function &F, FunctionAnalysisManager &) {
    errs() << "Analyzing function: " << F.getName() << "\n";
    for (auto &BB : F) {
        removeUnusedResultsIfAny(&BB);
      }
    }
    for (auto &BB : F) {
      errs() << "  BasicBlock:\n";
      for (auto &I : BB) {
        errs() << "    Instruction: " << I << "\n";
      }
    }
    return PreservedAnalyses::none();
  }
};

Let’s break it down. First, we note two member functions: our usual run and removeUnusedResultsIfAny. The run method iterates over the basic blocks of a function and delegates the transformation work to removeUnusedResultsIfAny. Afterwards, it prints the updated IR and informs the pass system that none of the analyses have been preserved.

The removeUnusedResultsIfAny method checks each instruction in the given basic block and removes it if it is unused. It also checks for terminator instructions to avoid removing return statements and similar control-flow terminators.

You can refer to one of the previous articles for instructions on how to compile the pass. Next, we run it with:

opt -load-pass-plugin=./DCECustomPass.so -passes=dce-pass -disable-output < test.ll

and the result is :

💥💥💥💥

Yeah, we segfault marvelously! 😀

This bug is a common one you often see in beginner’s code. It occurs because we remove instructions while iterating over the basic block. When the current instruction is erased, the iterator becomes invalid, and advancing it afterwards can trigger a null-pointer dereference.

Fortunately, the fix is simple: we just need to advance the iterator to the next element before removing the current instruction, as shown below:

void removeUnusedResultsIfAny(BasicBlock* BB) {
  for (auto it = BB->begin(); it != BB->end();/*removed ++it*/) {
    Instruction *It = &*it;
    ++it;// move to here. we go to the next before we remove
    if(!It->isTerminator() && It->use_empty()) {
      It->eraseFromParent();
    }
  }
}

As you can see, when your transformation involves removing instructions, you must ensure that the iterator is advanced to the next instruction before performing any processing on the current one.

Running opt with the patched code yields:

Analyzing function: multip
  BasicBlock:
    Instruction:   %res = mul i32 %a, %a
    Instruction:   ret i32 %res

Constant Propagation Pass

This is an optimization pass that aims to replace computations involving known values with their results at compile time. In other words, if a binary operation uses two hard-coded constants, the compiler can evaluate it immediately and propagate the resulting value to any instructions that depend on it.

For example, consider the following IR listing:

define i32 @testConstProp(i32 %a) {
entry:
  %x = add i32 1, 2
  %y = mul i32 %x, 3
  %z = sub i32 %y, 6
  %w = udiv i32 %z, 3
  %res = add i32 %a, %w
  ret i32 %res
}

Where we are adding, multiplying, subtracting, and dividing constant operands, and finally adding the parameter %a to the result of those computations.

Our constant propagation pass transforms the IR above into:

define i32 @testConstProp(i32 %a) {
entry:
  %res = add i32 %a, 1
  ret i32 %res
}

The pass removes instructions whose results can be computed at compile time and propagates the corresponding constant values. This process continues until it reaches an add instruction where one of the operands is a variable.

Constant Propagation Pass for `add`

Let’s start coding! The following listing shows the run member function of our pass, which performs constant propagation for both add and fadd instructions:

PreservedAnalyses run(Function &F, FunctionAnalysisManager &) {
    errs() << "Analyzing function: " << F.getName() << "\n";
    LLVMContext &context = F.getContext();
    for (auto &BB : F) {
      for (auto it = BB.begin(); it != BB.end();) {
        if (auto *BO = dyn_cast<BinaryOperator>(&*it++)) {
          ConstantInt *lOprd = nullptr;
          ConstantInt *rOprd = nullptr;
          ConstantFP *flOprd = nullptr;
          ConstantFP *frOprd = nullptr;

          if (PatternMatch::match(BO,
                                  m_Add(PatternMatch::m_ConstantInt(lOprd),
                                        PatternMatch::m_ConstantInt(rOprd)))) {
            auto newVal = lOprd->getValue() + rOprd->getValue();

            ConstantInt *constInt = ConstantInt::get(context, newVal);
            BO->replaceAllUsesWith(constInt);
            BO->eraseFromParent();
          } else if (PatternMatch::match(
                         BO, m_FAdd(PatternMatch::m_ConstantFP(flOprd),
                                    PatternMatch::m_ConstantFP(frOprd)))) {
            auto newVal = flOprd->getValue() + frOprd->getValue();

            ConstantFP *constFP = ConstantFP::get(context, newVal);
            BO->replaceAllUsesWith(constFP);
            BO->eraseFromParent();
          } else {
            errs() << "Instruction not supported yet" << *BO << "\n";
          }
        }
      }
    }
    for (auto &BB : F) {
      errs() << "  BasicBlock:\n";
      for (auto &I : BB) {
        errs() << "    Instruction: " << I << "\n";
      }
    }
    return PreservedAnalyses::none();
  }

Let’s walk through the code step by step. The pass iterates over every instruction in the current function and tries to match them with add and fadd instructions that have constant operands. Note the use of LLVM’s pattern matching facility. Previously, we relied directly on instruction opcodes, but pattern matching is more powerful here because it extracts the operands automatically when a match is found.

When a match occurs, we perform the addition, replace all of the instruction’s uses in the rest of the code, and then remove the original instruction. Each iteration starts by filtering for binary operations to ensure only relevant instructions are processed.

Finally, we print the transformed IR to observe the effect of the constant propagation.

Used LLVM facilities

One of the goals of this post was to explore and demonstrate more of the facilities provided by LLVM for writing passes. The following table summarizes nearly all of the APIs used throughout the previous sections:

API (symbol)	Kind	What it does (in this pass/code)	Header
`PassInfoMixin<T>`	CRTP helper class / mixin	Base mixin used when writing the new-style (PM) pass type; gives pass-related helpers and boilerplate.	`#include "llvm/IR/PassManager.h"`
`PreservedAnalyses`	Class / return type	Represents which analyses are preserved after a pass; `PreservedAnalyses::none()` signals nothing preserved.	`#include "llvm/IR/PassManager.h"`
`FunctionAnalysisManager`	Class (analysis manager)	Manager for function analyses (passed into `run`); used by the new pass manager interface.	`#include "llvm/IR/PassManager.h"`
`Function`	LLVM IR type / class	Represents a function; the pass iterates over `Function &F` to visit basic blocks and instructions.	`#include "llvm/IR/Function.h"`
`BasicBlock`	LLVM IR type / class	Represents a basic block; used to iterate instructions and perform transformations per-block.	`#include "llvm/IR/BasicBlock.h"`
`Instruction`	LLVM IR class	Base class for all instructions; methods like `isTerminator()`, `use_empty()`, `eraseFromParent()` are used.	`#include "llvm/IR/Instruction.h"`
`Instruction::isTerminator()`	Method	Returns true if the instruction is a terminator (e.g., `ret`, `br`) — used to avoid removing terminators.	`llvm/IR/Instruction.h`
`Instruction::use_empty()`	Method	Checks whether the instruction has zero uses (no other IR references its result). Key to DCE.	`llvm/IR/Instruction.h`
`Instruction::eraseFromParent()`	Method	Removes (deletes) the instruction from its parent basic block and frees it.	`llvm/IR/Instruction.h`
`errs()`	Function / global `raw_ostream`	LLVM’s error/diagnostic output stream (like `std::cerr`); used to print debugging/transformed IR.	`#include "llvm/Support/raw_ostream.h"`
`Function::getName()`	Method	Returns the function name (StringRef) — used when printing which function is being analyzed.	`llvm/IR/Function.h`
`Function::getContext()`	Method	Returns the `LLVMContext &` associated with the function; needed to create `Constant*`s.	`llvm/IR/Function.h`
`LLVMContext`	Class	Context object holding uniqued types, constants, metadata, etc.; needed when creating constants.	`#include "llvm/IR/LLVMContext.h"`
`BasicBlock::begin()` / `BasicBlock::end()`	Methods / iterators	Provide iterator access to a block’s instructions (`Instruction` iterators) for manual iteration.	`llvm/IR/BasicBlock.h`
`dyn_cast<T>(Value*)`	LLVM RTTI-like cast	Safe downcast that returns `T*` if the value is of type `T` (otherwise nullptr). Used to detect `BinaryOperator`.	`#include "llvm/Support/Casting.h"`
`BinaryOperator`	LLVM IR class (subclass of `Instruction`)	Represents binary operators (`add`, `mul`, `sub`, `udiv`, `fadd`, ...); used to detect/operate on binary arithmetic instructions.	`#include "llvm/IR/Instructions.h"`
`ConstantInt`	Class	Represents integer constants; used to inspect integer operands and to create folded integer `ConstantInt`s.	`#include "llvm/IR/Constants.h"`
`ConstantFP`	Class	Represents floating-point constants; used for `fadd` constant folding.	`#include "llvm/IR/Constants.h"`
`PatternMatch::match(...)`	Pattern matching helper	High-level matcher to test instruction shapes (e.g., `m_Add(m_ConstantInt(...), m_ConstantInt(...))`) and extract operands.	`#include "llvm/IR/PatternMatch.h"`
`m_Add`, `m_FAdd` (pattern matchers)	Pattern matcher macros / functions	Match `add` and `fadd` binary operations in `PatternMatch::match`.	`llvm/IR/PatternMatch.h`
`PatternMatch::m_ConstantInt`, `PatternMatch::m_ConstantFP`	Matchers	Match constant integer / floating operands and bind them to variables on success.	`llvm/IR/PatternMatch.h`
`APInt` (implicitly via `ConstantInt::getValue()`)	Arbitrary-precision integer class	Holds the integer value extracted from `ConstantInt` (`getValue()` returns `APInt`), used for arithmetic on constants.	`#include "llvm/ADT/APInt.h"`
`APFloat` (implicitly via `ConstantFP::getValue()`)	Floating-point value class	Holds floating-point data returned by `ConstantFP::getValue()` (used for constant fp arithmetic).	`#include "llvm/IR/APFloat.h"`
`ConstantInt::get(context, APInt)`	Static factory	Creates a `ConstantInt` from an `APInt` and `LLVMContext`. Used to replace folded integer expressions.	`llvm/IR/Constants.h`
`ConstantFP::get(context, APFloat)`	Static factory	Creates a `ConstantFP` from an `APFloat` and `LLVMContext`.	`llvm/IR/Constants.h`
`ConstantInt::getValue()`	Method	Returns the `APInt` value of the `ConstantInt`.	`llvm/IR/Constants.h`
`ConstantFP::getValue()`	Method	Returns the `APFloat` value of the `ConstantFP`.	`llvm/IR/Constants.h`
`Value::replaceAllUsesWith(Value *V)`	Method (on `Value`)	Replace every use of this `Value` (instruction/result) with another `Value`. Used before erasing the original instruction.	`#include "llvm/IR/Value.h"`
`eraseFromParent()` vs `removeFromParent()` distinction	Method nuance	`eraseFromParent()` deletes the instruction and removes it from the block; `removeFromParent()` just unlinks—your code uses `eraseFromParent()`.	`llvm/IR/Instruction.h`
`StringRef` (implied by `getName()`)	Lightweight string view class	`Function::getName()` returns a `StringRef` representing the function name; used with `errs() <<`.	`#include "llvm/ADT/StringRef.h"`

I think pattern matching is one of the most useful features of LLVM, as it greatly simplifies the code.

Conclusion

We have written two transformation passes:

Dead Code Elimination (DCE)
Constant Propagation

We also saw the key points to pay attention to and how LLVM provides facilities that make writing passes easier.

Writing Your First LLVM Plugin Pass: Counting Add Instructions

cppchedy — Tue, 21 Oct 2025 11:03:02 +0000

Introduction

In my previous post, we went through the not-so-glamorous part: building LLVM from source and running a pass with opt. With that groundwork out of the way, it’s time to move on to the fun part which is writing the passes.

In this post, we’ll start with a bit of theory on LLVM passes, just enough to give you solid footing, and then jump straight into code. Our first real pass will be a simple one: counting the number of add instructions across an IR module.

Since we’ll be working directly with LLVM IR, I’m assuming you’re at least somewhat familiar with reading it. If not, I recommend taking a little time to get comfortable reading IR first because it will make your LLVM adventure much smoother.

LLVM Passes and Why They Matter

If I had to pick two pillars of LLVM, they would be the Intermediate Representation (IR) and the pass system.

IR provides a common, language-agnostic format that higher-level languages can target. This abstraction makes LLVM language-independent and enables all subsequent transformations and optimizations.
Passes break down the complex optimization process into small, focused steps. Each pass has a clear task, and LLVM executes them in a pipeline to gradually refine or analyze the IR.

Together, these two ideas form the backbone of LLVM’s flexibility and power. With passes, LLVM becomes a modular framework where analyses and optimizations can be added, removed, or combined. Without them, LLVM would be an unmanageable monolith.

LLVM organizes passes according to the granularity of the IR unit they operate on or in simpler term scope:

Module passes
Function passes
Loop passes

Let’s break down the different types of passes in more detail.

Module Pass

Scope: Operates on an entire Module (representing a program or translation unit).
Use cases:
- Interprocedural optimizations (e.g., inlining decisions).
- Whole-program analyses (e.g., building call graphs).
- Global variable optimizations.
Example: Dead Global Elimination removes unused global variables.

Function Pass

Scope: Operates on a single Function independently of others.
Use cases:
- Local optimizations inside a function.
- Instruction simplification.
- Control-flow restructuring.
Example: Instruction Combining (InstCombine) simplifies redundant instructions within a function.

Loop Pass

Scope: Operates on a single Loop (as identified by LLVM’s loop analysis).
Use cases:
- Loop-invariant code motion (LICM).
- Loop unrolling, peeling, or fusion.
- Vectorization.
Example: Loop Unroll Pass duplicates loop bodies to reduce loop overhead.

You may want to read about the use cases you find interesting elsewhere; though, maybe I will write about some of them. Next, let's take a look at the Pass Manager.

Pass Manager Machination

Passes don’t execute on their own. They must be scheduled and orchestrated. That’s the job of the Pass Manager, which arranges passes, manages dependencies, and helps preserve analysis results when possible to avoid redundant recomputation.

Before a pass can run, it has to be registered so that LLVM’s PassBuilder and PassManager know it exists and where it can fit in the compilation pipeline. Once registered, the infrastructure can include the pass in a pipeline, deciding when it runs and alongside which other passes.

Keep in mind that analysis preservation isn’t automatic. A pass must explicitly declare which analyses remain valid after it runs. Likewise, different kinds of passes (module, function, loop) are managed at different levels of granularity, and you may need adapters or careful placement when mixing them in a pipeline.

With those ideas in place, let’s move from the conceptual side to the practical one and see how all these pieces come together in code.

Getting our hands dirty

Template For Passes

In the previous sections, we discussed LLVM's pass system and briefly introduced the new Pass Manager that LLVM relies on to perform its magic.

Let’s now look at the actual structure of a pass and walk through the code you’ll need to define and register it.

#include "llvm/IR/PassManager.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"
// other includes

using namespace llvm;

// Pass logic
class MyXPass : public PassInfoMixin<MyXPass> {
public:
    PreservedAnalyses run(X &M, XAnalysisManager &AM) {
        // ...
        // Return what analyses are preserved
        return PreservedAnalyses::all();
    }
};

// Registration for `opt`
llvm::PassPluginLibraryInfo getMyXPassPluginInfo() {
    return {LLVM_PLUGIN_API_VERSION, "MyXPass", LLVM_VERSION_STRING,
            [](PassBuilder &PB) {
                PB.registerPipelineParsingCallback(
                    [](StringRef Name, XPassManager &MPM,
                       ArrayRef<PassBuilder::PipelineElement>) {
                        if (Name == "my-pass") {
                            MPM.addPass(MyXPass());
                            return true;
                        }
                        return false;
                    });
            }};
}

extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
llvmGetPassPluginInfo() {
    return getMyXPassPluginInfo();
}

What we see above is a template, usable for the type of passes we introduced in previous sections.

Note that the 'X' in the parameter type of the (member) functions denote Module, Function or Loop.

As mentioned in the previous article, we can see two distinct part of the code:

A class representing the pass logic.
A function that registers our pass with so it can be recognized by opt via --passes=.

Let's start over by opt registration part.

Registering the Pass

To run our pass with opt, we need to write the following piece of code:

llvm::PassPluginLibraryInfo getMyXPassPluginInfo() {
    return {LLVM_PLUGIN_API_VERSION, "MyXPass", LLVM_VERSION_STRING,
            [](PassBuilder &PB) {
                PB.registerPipelineParsingCallback(
                    [](StringRef Name, XPassManager &MPM,
                       ArrayRef<PassBuilder::PipelineElement>) {
                        if (Name == "my-pass") {
                            MPM.addPass(MyXPass());
                            return true;
                        }
                        return false;
                    });
            }};
}

extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
llvmGetPassPluginInfo() {
    return getMyXPassPluginInfo();
}

Let's walk through it step by step. We have a mandatory function, llvmGetPassPluginInfo, with external linkage that provides metadata of our pass/plugin to opt in order to accept or reject the loading. This function returns a PassPluginLibraryInfo object containing things like our pass name, version, LLVM version and a lambda accepting a reference to a PassBuilder object. This is the object responsible for registering our pass in the new Pass manager system(you can look up the legacy pass manager if you are interested).

Zooming on the "registration" lambda, we see that it just hooks into LLVM pipeline parsing with a callback, triggered to match the input passed to --passes= in opt. The last bit is the instantiation of our pass, executed if the input matches our pass name.

Core Function

The pass essence is written in the following listing:

class MyXPass : public PassInfoMixin<MyXPass> {
public:
    PreservedAnalyses run(X &M, XAnalysisManager &AM) {
        // ...
        // Return what analyses are preserved
        return PreservedAnalyses::all();
    }
};

What you need to pay attention to are:

Class PassInfoMixin
Run member function
PreservedAnalyses
XAnalysisManager
X

We inherit from PassInfoMixin. This class provides the boilerplate required by LLVM pass system to integrate with the new pass manger. By inheriting from this mixin, our pass gets recognized and managed by LLVM.

The run method is the core of our pass. It gets called when LLVM decides to execute our pass logic. This method has two arguments:

X &M: This represents the IR unit your pass operates on.(e.g Module.)
XAnalysisManager &AM : Gives access to analysis results and allow us to query or preserve information other passes might depend on.

The body of run member function contains the logic of our pass. At the end of this member function, we indicate to the Pass Manager that the analysis results are still valid. Specifically, if you didn't transform anything you can just return PreservedAnalyses::all(), Which tells the Pass Manager that you didn't make any change on the IR.

Getting our hands dirty for Real

That’s enough setup! Time to get our hands dirty with some real code. Let’s write a pass that counts all the add instructions in a module.

Counting `Add` instructions Pass

The following is the code of our pass:

#include "llvm/IR/PassManager.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"

using namespace llvm;

// Pass logic
class AddCounterPass : public PassInfoMixin<AddCounterPass> {
public:
  PreservedAnalyses run(Module &M, ModuleAnalysisManager &) {
    int count = 0;
    for (auto &F : M.functions()) { // iterate over functions
      errs() << "Analyzing function: " << F.getName() << "\n";
      for (auto &BB : F) {// iterate over basic blocks of a function
        for (auto &I : BB) {// iterate over instruction in a BB
          if (I.getOpcode() == Instruction::Add) {
            ++count;
          }
        }
      }
    }
    errs() << "Number of add inst in the module : " << count << "\n";
    return PreservedAnalyses::all();
  }
};

// Registration for `opt`
llvm::PassPluginLibraryInfo getAddCounterPassPluginInfo() {
  return {LLVM_PLUGIN_API_VERSION, "AddCounterPass", LLVM_VERSION_STRING,
          [](PassBuilder &PB) {
            PB.registerPipelineParsingCallback(
                [](StringRef Name, ModulePassManager &MPM,
                   ArrayRef<PassBuilder::PipelineElement>) {
                  if (Name == "add-counter-pass") {
                    MPM.addPass(AddCounterPass());
                    return true;
                  }
                  return false;
                });
          }};
}

extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
llvmGetPassPluginInfo() {
  return getAddCounterPassPluginInfo();
}

The registration bits are repetitive and I won't go over them here. Let's dig into the pass logic.

PreservedAnalyses run(Module &M, ModuleAnalysisManager &) {
    int count = 0;
    for (auto &F : M.functions()) { // iterate over functions
      errs() << "Analyzing function: " << F.getName() << "\n";
      for (auto &BB : F) {// iterate over basic blocks of a function
        for (auto &I : BB) {// iterate over instruction in a BB
          if (I.getOpcode() == Instruction::Add) {
            ++count;
          }
        }
      }
    }
    errs() << "Number of add inst in the module : " << count << "\n";
    return PreservedAnalyses::all();
  }

The logic here is very straightforward once you get past the unfamiliar LLVM APIs. We iterate over all the functions in a Module, and for each function, we iterate through its basic blocks. Then, for every instruction inside each basic block, we check whether it matches Instruction::Add. If it does, we increment our counter. Simple, right?

You can think of a Module as representing a translation unit, containing one or more IR functions. A BasicBlock is just a sequence of instructions with a single entry point and a single exit (typically a branch or terminator instruction).

Note that LLVM offers multiple ways to match instruction. we may visit them in a future post.

To compile the pass, you can adapt the previous CMakeLists.txt file or just issue the following:

clang++ -fPIC -shared -o AddCounterPass.so add_counter_pass.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core passes` -Wl,-rpath,`llvm-config --libdir`

Pay attention to which clang you are invoking. It may cause some trouble if you didn't use the one we built from source.

Testing the Pass

We will be using the following listing to test our pass:

int randomAdditionFn(int a, int b, int c) {
  return a + b+ c;
}

int anotherAddFn(int a, int b, int c, int d) {
  int interm1 = a+ c;
  int interm2 = b + d;
  return interm1 + interm2;
}

int mixedAdd(int a, int b) {
  return a * b + 13;
}

And the command to turn it to IR:

clang++ -S -emit-llvm test.cpp -o test.ll

Run the pass with:

opt -load-pass-plugin=./AddCounterPass.so -passes=add-counter-pass -disable-output < test.ll

You will get something like this, if you used the same C++ code:

Analyzing function: _Z16randomAdditionFniii
Analyzing function: _Z12anotherAddFniiii
Analyzing function: _Z8mixedAddii
Number of add inst in the module : 6

Conclusion

In this post, we covered some theory around LLVM passes and got a taste of how the pass system actually works by writing a simple one ourselves. We intentionally glossed over a few details to keep the focus on getting something running.

In future articles, we’ll circle back to those parts and dig deeper into the pieces we’ve only touched on so far.

Compiling LLVM and Running Your First Dummy Pass

cppchedy — Fri, 19 Sep 2025 22:53:53 +0000

Introduction

If you’ve ever been curious about how compilers work under the hood, LLVM is one of the best playgrounds you could ask for. It’s powerful, flexible, and used everywhere from programming languages to GPU drivers. The catch? LLVM is huge, and getting started can feel daunting.

A great place to begin is with LLVM passes. They are the building blocks of compiler optimizations and program analysis. By creating your own passes and experimenting with the IR, you’ll quickly get hands-on with the core concepts that make LLVM tick.

This post will take you from building LLVM to running a simple “hello world” pass — your first step into compiler internals.

Prerequisites and Environment Setup

Before diving in, make sure you have the required tools installed. On Ubuntu/Debian, you can get them with:

sudo apt -y install gcc g++ git cmake ccache ninja-build zlib1g-dev

We’ll also need Git for cloning LLVM’s source, and Ccache for reducing LLVM build time.

Step 1: Building LLVM from Source

Follow these steps to clone, configure, and build LLVM (based on the official LLVM guide and our experience):

git clone https://github.com/llvm/llvm-project.git
cd llvm-project/ 
git checkout -b llvm-17 llvmorg-17.0.1

mkdir build 
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
      -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE \   
      -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \   
      -DLLVM_ENABLE_PROJECTS=clang \   
      -DCMAKE_INSTALL_PREFIX=/opt/llvm \ 
      -B build -S llvm  

cmake --build build -j$(nproc) 
cmake --build build --target check-all -j$(nproc)

👉 check-all runs LLVM’s test suite. If all tests pass, you’re good to go.
👉 -DBUILD_SHARED_LIBS avoid problems when running your pass with opt.

To install LLVM to /opt/llvm, run:

cmake --install build

At this point, you have a working LLVM + Clang build.

Step 2: Writing a Hello World LLVM Pass

Now let’s write a simple dummy pass. This pass won’t change the program — it will just print a message whenever it runs.

Create a new file hellopass.cpp with the following content:

#include "llvm/IR/Function.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"
#include "llvm/Support/raw_ostream.h"

using namespace llvm;

namespace {
struct HelloWorldPass : PassInfoMixin<HelloWorldPass> {
  PreservedAnalyses run(Function &F, FunctionAnalysisManager &) {
    errs() << "Hello from: " << F.getName() << "\n";
    return PreservedAnalyses::all();
  }
};
} // namespace

// Pass registration
extern "C" ::llvm::PassPluginLibraryInfo llvmGetPassPluginInfo() {
  return {
    LLVM_PLUGIN_API_VERSION, "HelloWorldPass", LLVM_VERSION_STRING,
    [](PassBuilder &PB) {
      PB.registerPipelineParsingCallback(
        [](StringRef Name, FunctionPassManager &FPM,
           ArrayRef<PassBuilder::PipelineElement>) {
          if (Name == "hello-world") {
            FPM.addPass(HelloWorldPass());
            return true;
          }
          return false;
        });
    }
  };
}

Subsequent posts will go with an in-depth explanation on passes. Just keep in mind that, fundamentally, we have two "sections/responsibilities" in the listing above: The pass logic and its registration. The run method is where your logic goes. Here we just print the function name.

Step 3: Building the Pass

Use CMake to compile the pass as a shared library. Create a CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)

project(HelloPass LANGUAGES CXX)  

find_package(LLVM REQUIRED CONFIG) 

include_directories(${LLVM_INCLUDE_DIRS}) 
add_definitions(${LLVM_DEFINITIONS})  
add_library(HelloPass MODULE hellopass.cpp)  
# Ensure C++17 
set_target_properties(HelloPass PROPERTIES CXX_STANDARD 17 
                      CXX_STANDARD_REQUIRED YES)

Then build:

mkdir build && cd build
cmake -DLLVM_DIR=/opt/llvm/lib/cmake/llvm -DCMAKE_CXX_COMPILER='clang++' .. 
cmake --build .

This produces libHelloPass.so.

Step 4: Running the Pass

Now, compile a simple C++ file to LLVM IR:

// test.cc 
int add(int a, int b) {     
   return a + b; 
}

Compile to LLVM IR:

/opt/llvm/bin/clang++ -S -emit-llvm test.cc -o test.ll

Run the pass:

/opt/llvm/bin/opt -load-pass-plugin=./libHelloPass.so -passes=hello-world -disable-output < test.ll

You should see output like:

Hello from: add

Wrapping Up

You’ve now:

Built LLVM from source
Executed your very first LLVM pass

From here, you can extend the dummy pass to analyze control flow, optimize instructions, or even transform IR. This “Hello World” step is the gateway into real compiler development.

S'tracing WO MIC: a journey involving threading, networking, audio, and more

cppchedy — Wed, 27 Jan 2021 21:18:07 +0000

Recently, I decided to solve the issue of my broken camera without buying a new one. The idea is to develop a system capable of forwarding my phone's camera stream to my laptop and expose it to other applications.

Using a phone to extend the capability of your machine isn't new. In fact, the Playstore is rich with such apps. For example, WO MIC -an android app- enables computers to use the mic of a mobile phone by forwarding the audio input via Wi-Fi, Wi-Fi Direct, USB, or Bluetooth.

Wo MIC serves as a good case of study for my project. Understanding how it works underneath and seeing how exactly it uses the mentioned technologies will certainly provide a lot of insight when I am designing my app. Also, checking my assumptions about the system implementation will solidify my design skills.

In this article, I cover the architecture of the WO MIC system. Moreover, I go through the process of analyzing WO MIC client using strace: I document its pitfalls and explain a couple of concepts like threads from the Linux kernel's point of view.

Architecture of WO MIC

The above image shows the architecture of the WO MIC system for sharing a phone's mic with a laptop. Besides the Android app, the laptop contains a WO MIC client program responsible for receiving the streams through the router and writing it to the snd-aloop module.

Snd-aloop is a Linux kernel module that brings a loopback feature for audio I/O. When this module is loaded into the kernel, it adds a virtual device called Loopback card in which other application can read from or write to.

WO MIC's How to

After installing the android app from the Playstore, make sure that you are connected to your Wi-fi access point then open WO MIC. Select Wi-fi in the settings and click on start. An IP address is assigned and shown on the screen. This IP address will be used by the WO MIC client for connection.

Now on your laptop, Download the client. Make sure the snd-aloop module is loaded by typing the following command in your terminal:

$ sudo modprobe snd-aloop

To check if the cmd was successful, you can use the lsmod command and look for a module named snd-aloop in the output. if everything is okay with snd-aloop, we can now connect to the phone by:

$ ./micclient-x86_64.AppImage -t Wifi IP_addr_displayed_on_phone

you will see the message connected on the phone when the two machines are linked.

Stracing the WO MIC client

In the Architecture section, I presented my thoughts on how the WO MIC system works. In this section, I will be verifying what we discussed using the tool strace.

strace is a powerful command-line tool for debugging and troubleshooting programs in Linux. In my case, I will be using it to capture all the system calls executed by the WO MIC client. Therefore, with this tool's help, we will be able to understand the mechanics of the system.

Based on our own assumption, The network stack and the audio APIs are the principal components used by the WO MIC client. Accordingly, for the networking part, we will be searching for network system calls such as connect, socket, recvmsg, sendmsg, etc...

In contrast to networking, I am not sure about the audio part: I never encountered a situation where I had to look deep into the audio system calls in Linux. This leaves us with no choice but to explore all the output of strace in order to find the sound APIs. Nevertheless, This is a good opportunity for us to learn them. Further more, we could discover other interesting API while we are exploring.

Note: if you want to follow along refer to the How to section of this article or go to this page so that you know how to run WO MIC.

Let's start tracing. Connect your phone to your Wi-fi access point and start the service then switch to your laptop and type the following in your terminal:

$ strace ./micclient-x86_64.AppImage -t Wifi 192.168.102

Note: replace the IP address with the one shown on your phone's screen.

The tracing starts with calls to load the required dynamic libraries with the right protection for their memory regions.

Audio APIs

The first thing I did was to look for network system calls. Unexpectedly, I didn't encounter any. Instead, I stumbled upon some S.C that give the feeling of executing audio work.

//...
openat(AT_FDCWD, "/dev/snd/controlC0", O_RDWR|O_CLOEXEC) = 4
ioctl(4, SNDRV_CTL_IOCTL_PVERSION, 0x7ffc8994f804) = 0
ioctl(4, SNDRV_CTL_IOCTL_CARD_INFO, 0x7ffc8994f890) = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/snd/controlC1", O_RDWR|O_CLOEXEC) = 4
ioctl(4, SNDRV_CTL_IOCTL_PVERSION, 0x7ffc8994f804) = 0
ioctl(4, SNDRV_CTL_IOCTL_CARD_INFO, 0x7ffc8994f890) = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/snd/controlC1", O_RDWR|O_CLOEXEC) = 4
ioctl(4, SNDRV_CTL_IOCTL_PVERSION, 0x7ffc8994f814) = 0
ioctl(4, SNDRV_CTL_IOCTL_PCM_PREFER_SUBDEVICE, 0x7ffc8994f87c) = 0
openat(AT_FDCWD, "/dev/snd/pcmC1D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 5
ioctl(5, SNDRV_PCM_IOCTL_INFO, 0x7ffc8994f8b0) = 0
close(4)                                = 0
ioctl(5, SNDRV_PCM_IOCTL_INFO, 0x7ffc8994f720) = 0
fcntl(5, F_GETFL)                       = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
ioctl(5, SNDRV_PCM_IOCTL_PVERSION, 0x7ffc8994f700) = 0
ioctl(5, SNDRV_PCM_IOCTL_USER_PVERSION, 0x7ffc8994f710) = 0
ioctl(5, SNDRV_PCM_IOCTL_TTSTAMP, 0x7ffc8994f704) = 0
mmap(NULL, 4096, PROT_READ, MAP_SHARED, 5, 0x80000000) = 0x7efc7410b000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0x81000000) = 0x7efc740e0000
fcntl(5, F_GETFL)                       = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
fcntl(5, F_SETFL, O_RDWR|O_LARGEFILE)   = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f900) = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f500) = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f500) = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f230) = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f500) = 0
ioctl(5, SNDRV_PCM_IOCTL_HW_REFINE, 0x7ffc8994f500) = 0
//...

After a bit of googling about the requests (SNDRV_PCM_IOCTL_xx_xx) passed to ioctl, I found about lib-alsa which is an API for doing PCM playback and recording in Linux. So I was indeed correct with my guessing. We now know what the client used for manipulating audio.

Bonus: for a complete guide on which Sound API to use read this post.

Network APIs

on the contrary of what I was expecting, finding the network APIs wasn't obvious. From the output, there are clearly log messages describing the state of the connection but no API calls were visible. the following image shows the logging of connection states:

I thought that this has to do with threading. In other words, My guess is that the networking part is being handled by a different thread.

Before checking if my guess is correct, let's make a little detour and talk about threads in Linux.

Detour on threads in Linux

Certainly, anyone who took a course related to operating systems will be exposed to the concept of threads defined as lightweight processes. This is true for UNIX-like operating systems except for the Linux kernel.

In Linux, a thread is a process. Why, you ask? because the structure holding processes states(formally know as Process Control Block or PCB) is already very lightweight in comparison with other operating systems like Windows. So the design direction wasn't in favor of defining a new representation for threads. Instead, Threads are managed in groups of processes.

The Linux kernel associates processes that are created by the same parent process with each other by leveraging a couple of fields in the PCB. To repeat my point, there is no new structure for handling threads.

The name of the C structure that holds information about the processes is task_struct. You can check it out in the Linux kernel codebase, specifically the sched.h header file under include/linux.

Using This parent-child relationship to explain threads may be confusing(if you forget sharing resources) because all the processes are created from parents- until we arrive at the root, init.d- and each user process has a parent, so concretely, How does the Linux kernel see threads?

From the Linux kernel perspective a thread could be defined as a process that shares with its parent:

the address space,
filesystem resources,
file descriptors,
and signal handlers

Given the previous definition, which system call is responsible for making threads?

Let's start with the famous API fork. does this S.C spawn a new thread? The following is a quote of the description of fork from man:

fork() creates a new process by duplicating the calling process.
The new process is referred to as the child process. The calling
process is referred to as the parent process.

Clearly, this has the parent-child relationship but:

The child process and the parent process run in separate memory
spaces. At the time of fork() both memory spaces have the same
content. Memory writes, file mappings (mmap(2)), and unmappings
(munmap(2)) performed by one of the processes do not affect the
other.

As we can see, fork does make a process duplicate but it shares almost nothing. In fact, a call to exec will populate the memory space with the new instructions from the specified binary(UNIX use this approach to spawn a new process).

Further down in fork's description from man, we encounter this interesting paragraph about the difference between C library and the Linux kernel:

C library/kernel differences
Since version 2.3.3, rather than invoking the kernel's fork()
system call, the glibc fork() wrapper that is provided as part of
the NPTL threading implementation invokes clone(2) with flags
that provide the same effect as the traditional system call. (A
call to fork() is equivalent to a call to clone(2) specifying
flags as just SIGCHLD.)

This quote is saying that glibc's fork wrapper is now implemented using the clone system call rather then directly calling the fork system call of the kernel. To be exact, the call to clone is like this: clone(SIGCHLD, 0).

The system call clone is similar to fork in the sense that it spawns a new process. However, there is a crucial difference:

By contrast with fork(2), these system calls provide more precise
control over what pieces of execution context are shared between
the calling process and the child process.

In other words, clone gives you explicit control over what to share with your child process. This is exactly what we need.

clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0);

The above call specifies the sharing of the Virtual memory with CLONE_VM, the sharing of files systems CLONE_FS, the sharing of the files CLONE_FILES, and the sharing of the signal handler CLONE_SIGHAND. This is even with the previous definition of threads in Linux.

Before proceeding to the next subsection, keep in mind the definition of Linux threads.

Looking for network system calls in forks

Why did we have the discussion about threads? it's because we are suspecting that the work of the network exchanges is handled in other threads. For this reason, we need to know how exactly threads are created in order to recognize the calls in the trace.

Looking into strace output, there are two calls to clone. In the first call we have:

These flags, CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, are passed to clone. Therefore, we can say that this call is for a new process and not a new thread. if you recall the glibc's fork is implemented by passing SIGCHLD to clone. This flag is present in this call.

In the second call, we have:

According to what we discussed about threads, this call contains the same flags as the POSIX pthread_create: CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID. So this is a confirmation that WO MIC client is using threads. Moreover, I found other evidence that indicates the usage of threading such as loading the dynamic library(.so) of pthread.

Let's go back to strace now. For following forks of a process, strace provide the option -f. Executing strace with this option yields the following result:

Even though we can trace the forks of our parent process, now, the client exits with a nonzero return value. Moreover, the connection with the phone isn't established. You can compare the two output with and without -f in the following image:

In the new trace output, I spotted a couple of calls to connect and socket but they are not meant for connecting to the phone. Instead, they are doing inter-process communication(IPC) with UNIX Domain sockets:

//...
[pid 97612] socket(AF_UNIX, SOCK_STREAM, 0) = 9
[pid 97612] fcntl(9, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 97612] fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 97612] fcntl(9, F_GETFD)           = 0
[pid 97612] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid 97612] connect(9, {sa_family=AF_UNIX, sun_path="/var/lib/sss/pipes/nss"}, 110) = 0
[pid 97612] fstat(9, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
[pid 97612] poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
[pid 97612] sendto(9, "\24\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
[pid 97612] poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
[pid 97612] sendto(9, "\1\0\0\0", 4, MSG_NOSIGNAL, NULL, 0) = 4
[pid 97612] poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
[pid 97612] read(9, "\24\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
//...

Note that the second call to clone isn't executed in this instance.

Going through the traces, I understood that the process is failing to mount /dev/fuse which is vital to its inner working.

//...
[pid 97613] getgid()                    = 1000
[pid 97613] setfsgid(1000)              = 1000
[pid 97613] openat(AT_FDCWD, "/dev/fuse", O_RDWR) = 6
[pid 97613] getuid()                    = 1000
//...
[pid 97613] mount("/dev/fuse", ".", "fuse", MS_RDONLY|MS_NOSUID|MS_NODEV, "fd=6,rootmode=40000,user_id=1000"...) = -1 EPERM (Operation not permitted)
[pid 97613] write(2, "fusermount: mount failed: Operat"..., 50fusermount: mount failed: Operation not permitted
) = 50
[pid 97613] close(6)                    = 0
[pid 97613] exit_group(1)               = ?

As indicated in the message, the process don't have permission to mount fusermount.

To fix this problem, I added sudo to the cmd, so now we have:

$ sudo strace -f ./micclient-x86_64.AppImage -t Wifi 192.168.1.102

Finally, after fixing the permission problem, the network system calls are printed for pid 98629:

The above image shows a messy output. It is better to filter out any unneeded system calls and keep only the network ones.
strace provides an option for this:

$ sudo strace -tt -e trace=network -f ./micclient-x86_64.AppImage -t Wifi 192.168.1.102

We only keep the network stack calls by adding -e trace=network. The -tt option is for outputting time with each call. The result of executing this cmd is shown in the following figure:

Now that we cleared the output and localized the network system calls, it is time for us to study the content and understand what's going on.

//...
[pid 16936] 16:44:07.067208 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 4                                                                                                
[pid 16936] 16:44:07.067421 connect(4, {sa_family=AF_INET, sin_port=htons(8125), sin_addr=inet_addr("192.168.1.102")}, 16) = 0                                           
[pid 16936] 16:44:07.240644 socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 6                                                                                                 
[pid 16936] 16:44:07.241179 setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0                                                                                          
[pid 16936] 16:44:07.241418 bind(6, {sa_family=AF_INET6, sin6_port=htons(49152), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::", &sin6_addr), sin6_scope_id=0}, 28) = $
[pid 16936] 16:44:07.241612 getsockopt(6, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0 
[pid 16936] 16:44:07.241876 sendto(4, "e", 1, 0, NULL, 0) = 1                                                                                                            
[pid 16936] 16:44:07.251950 sendto(4, "\0\0\0\6", 4, 0, NULL, 0) = 4                                                                                                     
[pid 16936] 16:44:07.252200 sendto(4, "\4", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252452 sendto(4, "\4", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252702 sendto(4, "\6", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252824 sendto(4, "\2", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252940 sendto(4, "\0", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.253143 sendto(4, "\0", 1, 0, NULL, 0) = 1
[pid 16936] 16:44:07.456798 recvfrom(4, "e", 1, 0, NULL, NULL) = 1                                                                                                       
[pid 16936] 16:44:07.462912 recvfrom(4, "\0\0\0\2", 4, 0, NULL, NULL) = 4                                                                                                
[pid 16936] 16:44:07.463288 recvfrom(4, "\0", 1, 0, NULL, NULL) = 1                                                                                                      
[pid 16936] 16:44:07.463499 recvfrom(4, "\4", 1, 0, NULL, NULL) = 1                                                                                                      
[pid 16936] 16:44:07.463671 sendto(4, "f", 1, 0, NULL, 0) = 1                                                                                                            
[pid 16936] 16:44:07.463862 sendto(4, "\0\0\0\6", 4, 0, NULL, 0) = 4
//...
Connected     
strace: Process 16938 attached
strace: Process 16939 attached
[pid 16938] 16:44:07.915884 recvfrom(6, "\4\0\0\275\0\0\0\0\0\0\5\370{\346s\374\261-b\227\335\363\374\21\245\273\263\202~z\300r"..., 1024, 0, NULL, NULL) = 193
[pid 16938] 16:44:07.933227 recvfrom(6, "\4\0\0v\0\1\0\0\0\1\5\370\305<d{\"b\242\254\327t\274d\224\332\2520\02728@"..., 1024, 0, NULL, NULL) = 122
//...

First, from looking at the calls, we find that there is actually two sockets. One socket is of the type SOCK_STRAM(that means TCP protocol) while the other socket is of the type SOCK_DGRAM (means UDP). Second, the TCP socket uses IP version 4 and the UDP socket uses IP version 6. Finally, the UDP socket starts communicating with the other side only when the TCP socket sends and receives some bytes and exactly after the message Connected is printed to stdout.

//...
[pid 16936] 16:44:07.241876 sendto(4, "e", 1, 0, NULL, 0) = 1                                                                                                            
[pid 16936] 16:44:07.251950 sendto(4, "\0\0\0\6", 4, 0, NULL, 0) = 4                                                                                                     
[pid 16936] 16:44:07.252200 sendto(4, "\4", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252452 sendto(4, "\4", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252702 sendto(4, "\6", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252824 sendto(4, "\2", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.252940 sendto(4, "\0", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.253143 sendto(4, "\0", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.456798 recvfrom(4, "e", 1, 0, NULL, NULL) = 1                                                                                                       
[pid 16936] 16:44:07.462912 recvfrom(4, "\0\0\0\2", 4, 0, NULL, NULL) = 4                                                                                                
[pid 16936] 16:44:07.463288 recvfrom(4, "\0", 1, 0, NULL, NULL) = 1                                                                                                      
[pid 16936] 16:44:07.463499 recvfrom(4, "\4", 1, 0, NULL, NULL) = 1                                                                                                      
[pid 16936] 16:44:07.463671 sendto(4, "f", 1, 0, NULL, 0) = 1                                                                                                            
[pid 16936] 16:44:07.463862 sendto(4, "\0\0\0\6", 4, 0, NULL, 0) = 4                                                                                                     
[pid 16936] 16:44:07.464077 sendto(4, "\2", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.464165 sendto(4, "\2", 1, 0, NULL, 0) = 1                                                                                                           
[pid 16936] 16:44:07.464291 sendto(4, "\0\0\300\0", 4, 0, NULL, 0) = 4                                                                                                   
[pid 16936] 16:44:07.469806 recvfrom(4, "f", 1, 0, NULL, NULL) = 1                                                                                                       
[pid 16936] 16:44:07.471391 recvfrom(4, "\0\0\0\1", 4, 0, NULL, NULL) = 4                                                                                                
[pid 16936] 16:44:07.471551 recvfrom(4, "\0", 1, 0, NULL, NULL) = 1                                                                                                      
[pid 16936] 16:44:07.471675 sendto(4, "g", 1, 0, NULL, 0) = 1                                                                                                            
[pid 16936] 16:44:07.471989 sendto(4, "\0\0\0\0", 4, 0, NULL, 0) = 4                                                                                                     
[pid 16936] 16:44:07.520045 recvfrom(4, "g", 1, 0, NULL, NULL) = 1                                                                                                       
[pid 16936] 16:44:07.521841 recvfrom(4, "\0\0\0\1", 4, 0, NULL, NULL) = 4                                                                                                
[pid 16936] 16:44:07.522076 recvfrom(4, "\0", 1, 0, NULL, NULL) = 1 
//...

After this sequence of exchange, the TCP socket enters in send-only mode.

Interestingly, I wasn't quite right about the specifics of WO MIC in the sense that UDP was out of my equation. More importantly, I didn't expect the system to establish two channels and to use the second channel(UDP) for transferring the audio samples.

On one hand, The UDP socket is bound to an IPv6 address and only recv from the other end, no sending is performed. On the other hand, the TCP socket sends small chunks (and possibly the ipv6 address at the beginning though I am not sure) of data that triggers the sending of the audio samples through the UDP channel.

In a more verbose output, we find the multiplexing I/O system call select which keeps tracks of the UDP socket descriptor changes for IN events.

20952 <... select resumed>)             = 1 (in [6], left {tv_sec=2, tv_usec=920556})
20952 recvfrom(6, "\4\0\0\212\0E\0\0\5\\\5\370P\337\362\26g\5h\305\24y\233\360=\4Tk\350\f\260\324"..., 1024, 0, NULL, NULL) = 142
20952 select(7, [6], NULL, NULL, {tv_sec=3, tv_usec=0}) = 1 (in [6], left {tv_sec=2, tv_usec=999994})
20952 recvfrom(6, "\4\0\0\211\0F\0\0\5l\5\370?\276\223\363\257d\225\332\257rM:.\\\35\240El\22#"..., 1024, 0, NULL, NULL) = 141
20952 select(7, [6], NULL, NULL, {tv_sec=3, tv_usec=0}) = 1 (in [6], left {tv_sec=2, tv_usec=999994})
20952 recvfrom(6, "\4\0\0\206\0G\0\0\5\213\5\370?\341\316{\335\357\255l\353{]\351\7\335!9\262\30\317\243"..., 1024, 0, NULL, NULL) = 138
20952 select(7, [6], NULL, NULL, {tv_sec=3, tv_usec=0} <unfinished ...>
20953 <... nanosleep resumed>NULL)      = 0
20953 ioctl(5, SNDRV_PCM_IOCTL_WRITEI_FRAMES, 0x7f9c57ffe5c0) = 0
20953 nanosleep({tv_sec=0, tv_nsec=19479000}, NULL) = 0
20953 ioctl(5, SNDRV_PCM_IOCTL_WRITEI_FRAMES, 0x7f9c57ffe5c0) = 0
20953 nanosleep({tv_sec=0, tv_nsec=19481000},  <unfinished ...>
20952 <... select resumed>)             = 1 (in [6], left {tv_sec=2, tv_usec=972848})
20952 recvfrom(6, "\4\0\0\211\0H\0\0\5\232\5\370?\32$&\22[D\343n\214\356:\225J\302\231F?SS"..., 1024, 0, NULL, NULL) = 141
20952 select(7, [6], NULL, NULL, {tv_sec=3, tv_usec=0}) = 1 (in [6], left {tv_sec=2, tv_usec=991746})

This is so far what I explored with strace and what I have been able to find about WO MIC client. However, this is only the Wi-fi technology eventually I will dig into the Wi-fi Direct, Bluetooth and USB options.

Unexpected Gain

so far, we have learned about asla-lib for sound processing and we have seen how the WO MIC manages its communication to transfer data. Also, we have learned how to use strace. These were all in my expectations and my goals. However, the permission problem brought to me something that I wasn't aware of: it is fuse.

Recall that in the networking section we got a permission problem when we tried to trace all the forks and the output of the program was like this:

//...
[pid 24160] openat(AT_FDCWD, "/dev/fuse", O_RDWR) = 6                                                                                                                    
//...
[pid 24160] openat(AT_FDCWD, "/etc/fuse.conf", O_RDONLY) = 7
[pid 24160] fstat(7, {st_mode=S_IFREG|0644, st_size=38, ...}) = 0
[pid 24160] read(7, "# mount_max = 1000\n# user_allow_"..., 4096) = 38
//...
[pid 24160] openat(AT_FDCWD, "/etc/mtab", O_RDONLY|O_CLOEXEC) = 7
[pid 24160] fstat(7, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 24160] read(7, "sysfs /sys sysfs rw,seclabel,nos"..., 1024) = 1024
[pid 24160] read(7, "elatime,fd=29,pgrp=1,timeout=0,m"..., 1024) = 1024
[pid 24160] read(7, " 0\n/dev/mapper/fedora_localhost-"..., 1024) = 517
//...
[pid 24160] lstat("/tmp/.mount_miccliVllxVQ", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
[pid 24160] getuid()                    = 1000
[pid 24160] chdir("/tmp/.mount_miccliVllxVQ") = 0
 //...
[pid 24160] mount("/dev/fuse", ".", "fuse", MS_RDONLY|MS_NOSUID|MS_NODEV, "fd=6,rootmode=40000,user_id=1000"...) = -1 EPERM (Operation not permitted)
[pid 24160] write(2, "fusermount: mount failed: Operat"..., 50fusermount: mount failed: Operation not permitted
) = 50
[pid 24160] close(6)                    = 0
[pid 24160] exit_group(1)
//...
[pid 24158] wait4(24160, NULL, 0, NULL) = 24160
[pid 24158] rmdir("/tmp/.mount_miccliVllxVQ") = 0
[pid 24158] munmap(0x7eff2a679000, 262432) = 0
[pid 24158] ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid 24158] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}) = 0
[pid 24158] write(1, "\n", 1
)           = 1
[pid 24158] write(1, "Cannot mount AppImage, please ch"..., 53Cannot mount AppImage, please check your FUSE setup.
) = 53
[pid 24158] write(1, "You might still be able to extra"..., 172You might still be able to extract the contents of this AppImage 
if you run it with the --appimage-extract option. 
See https://github.com/AppImage/AppImageKit/wiki/FUSE 
) = 172
[pid 24158] write(1, "for more information\n", 21for more information
) = 21
[pid 24158] exit_group(0)
//...

Turns out that fuse means filesystem in userspace and that it's an interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a "bridge" to the actual kernel interfaces.

The above picture is a flow-chart diagram from Wikipedia that shows how FUSE works: Request from userspace to list files (ls -l /tmp/fuse) gets redirected by the Kernel through VFS to FUSE. FUSE then executes the registered handler program (./hello) and passes it the request (ls -l /tmp/fuse). The handler program returns a response back to FUSE which is then redirected to the userspace program that originally made the request.

In our case, WO MIC client setup a userspace file system at /tmp/.micclient... Though, I am not exactly sure what is its function.

Bonus: When I tweeted about fuse, Noah Watkins from Vectorized shared his implementation of a barebone, in-memory, FUSE file system leveraging the low-level FUSE interface. His Github repo is on my study list.

Conclusion

Through strace, I was able to gain a lot of insights and even to discover something I wasn't aware of. However, I only analyzed WO MIC when using Wi-fi as a sharing technology: we still have 3 other ways to look at. Moreover, I need to figure out the role of fuse in this.

Also, I have to answer the question of how to make the video stream available for my project: do I need to write a kernel module? is there a way in userspace? if there is, how much performance do I need to sacrifice?

Finally, I hope that you learned something from this: We discussed the reality of threads in Linux, the use of strace, and the audio APIs. For Fuse, maybe it will be the subject of a future article.

Switching to the NOWeb sphere projects

cppchedy — Thu, 03 Oct 2019 15:57:19 +0000

Introduction

Web have been increasingly dominating our industry for the last 20 years. In fact, the web dominates every day life and in some country if it fails the consequences are severe. But anyway, I am not writing this article for that matter: instead I am going to share my resources on how to build NOWeb(not only web) applications. I think that ,at some point of their journey, developers should try to build applications like Database, Compilers... to understand challenges involved outside of web apps.

List of resource

These are the series that I want to share with you, starting with Databases and going to other software:

Kingdb - a key value store series: This is is one of the best series I read because it covers the topic in all of it aspects. The series start with giving a general idea on key value store concept, sets guidelines for the development of the project and presents the design and architecture of two successful DBs: LevelDB and Kyoto kabinet. Moreover, it talks about the design of the exposed API and more importantly speaks in details about the main data structure used in the implementation Hashtables and how to handle SSDs.
Yet another SQLite clone: This a series of 13 parts where the author guides us to build a sqlite clone, step by step. You will learn SQLite architecture and build it's different components from scratch. Also, the series shows how to implement database abstractions like the Cursor abstraction and the core data structure B-tree.
Writing your own debugger: The author of this series did a great job explaining and implementing many features that a debugger must have such as breakpoints. It's a 10 parts series using modern C++.
MQTT Broker - implementing one from scratch: If you ever wondered how mqtt works this is for you. In this series, you get to learn about MQTT, implement the protocol and many different modules needed for the broker. I find this very interesting, especially part 2 where we get to implement an event loop using epoll.
Language VM - How to build your own: This a 33 parts series where the author shows the design and implementation of a Language VM.
Enkel - a JVM Language: If you want to build a JVM language, this 20-parts series covers that for you. You will see how to add loops, condition, scopes... and more.
MOM - a message oriented middleware: This is an ongoing series I am writing where I talk about how I built a MOM using Seastar.

Before I finish, I want to share an important resource discussing the concept of Clean Architecture introduced by Uncle bob: this is a a video from Bill Sourour of Dev mastery where he implements a comment microservice using nodejs following the Clean Architecture. I hope you find these links useful for your future projects.

If you find this resource helpful, you can follow me on twitter for more.

Rolling your own MOM, or how I did it - Messaging Model

cppchedy — Wed, 28 Aug 2019 11:31:42 +0000

Introduction

While part III of MOM series was mostly about understanding Seastar and putting the first milestone, the third part is dedicated to the Messaging Model of the System: we will present its elements and we will implement them in core module.

This article has both theory and code, so I hope you enjoy it and take it as inspiration for your models. Let's start.

Messaging Model

Some of you may wonder why we need a messaging model? I prefer to address this question before moving to the discussion on the model.

To answer this question properly, we need to consider the context of our system. If you recall, I said that we are building a software serving as a message relayer/broker for enterprise applications and depending on the situation at hand, enterprise devs need to come with a design that connects all software by using our middleware. However, I think the easier the learning curve of a system is the more it get adopted. As an example, look at c++, many complain that it's too large and difficult to learn. This is a drawback and a barrier for its adoption, even though the current number of its users doesn't reflect that.
I think that giving a small and clear mental model capturing all the details needed in the process of design and also other details of the internals of our system help to solve the problem of learning. Moreover, having a model for our system makes it easy to reason on how to solve the problem of message exchange and also is a key tool for expressing the intent of the architecture of the solution. Besides, the diagram showing the solution(using the element of the model) deployed in enterprise system serves as documentation for (new)team members.

Elements of the Messaging model

I prefer to follow an analyses approach instead of enumerating the elements with a definition. I think it's more beneficial so Let's identify user needs and extract from them concepts for the model.

Our messaging model is composed of elements that can be combined to form an architecture of message routes between its components(apps).

It's obvious that our model needs to include a concept/entity that represents enterprise application within the architecture. Thus, we introduce Client, the first element of our model.

Client lives outside the middleware and it's primarily an entity that interacts with the system. The server has a fixed and known address, so it can be accessed by any application. However, to route a message to its destination(s), the server needs the address of its client(s), therefore, the client needs to register itself in the system. This way the server gets the information needed for doing its task. Enter Context, the second element of our model that holds Client information and other related data. In contrast to Client, Context lives within the middleware and is essential for communication with the remote application.

Users of our system may need to send the same message from one to multiple application, or that some applications may be semantically related, therefore it's natural to tie them together. To enable such behavior, we add Group entity to our model. This element can be used to cluster Contexts according to user will.

Group is a central concept to our messaging model. It doesn't only relate Contexts to each other but also it defines message delivery(who exactly gets the message). We will talk more about the delivery process later on in this article.

We have been using Context for Group but let's use the right terminology: when we speak about Group, I prefer to call the contexts related to it Members.

Member can play the role of a consumer or a producer of messages. Each Group has one producer and multiple consumers. This design decision was taken to simplify the work I planned to do with the end2end ack(acknowledgment) feature (but didn't get to implement it). It's purely to respect deadlines so feel free to change this and redesign the system.

Contexts became members(with producer Xor consumer membership) of a Group when an application requests a joining(or subscribing) operation. That's when we Associate the two.

Message distribution policies

As we mentioned earlier, Groups define the process of message delivery. I planned to support 3 politics of message distribution:

Single-delivery: In this mode, the message is sent only to one consumer.
Per-match: In this policy, we are sending the message only if a certain "condition" is met. for each association with a consumer, we check if it respects the criteria, if so we send the message.
Publisher/Subscriber: You may already have guessed from the name, here, the message gets delivered to every consumer of the Group. Everyone is concerned with messages coming from the producer of the Group.

So We have three types of groups, with each type operating in one of the politics explained above.

Now that we have seen policies(and consequently defined the types of Group our system can have), we can move on to a related concept used in the Per-match mode: Annotation.

Recall that per-match groups do not deliver messages to a consumer unless they meet a criteria. In our model, we express this by Annotation placed on each Association. If the text from the message matches the Annotation, the system sends the message to the consumer.

Let's end this section with an example of a system architecture where we use the element of our model to describe the setup needed for message exchange.

This image shows a scenario where we send log messages to different destinations over the network depending on the severity level of the log(info, error, etc...). As depicted in the figure, we have a producer of logs and consumers, all of them are members of "MyGr" Group. From the picture, you see that each association is annotated with a severity level, and with it, the system can send the message to its appropriate destination.

TL;DR

Group: An entity that clusters together semantically related applications.
Client: An entity representing an enterprise application.
Context: An entity representing application inside the middleware.
Association: This entity represents the act of joining a Group.
Annotation: It's an entity representing a condition and it's attached to an Association.

I hope I was successful in explaining the messaging model of the system. Please feel free to raise questions in the comments if you find something unclear.

Implementation

Now that we discussed the messaging model and learned it's elements, we can go to talk about how we are going to implement it.

In this section, we will explore two modules: Identity module and Core module. The latter depends on the first one. We will see how.

Identity module

The notion of identity is so fundamental that you see it almost everywhere and our system is no exception. Specifically, we are using it in our model even though we didn't mention that explicitly. As we will see in the next section of this article, two elements from the model rely on the Identity class of the Identity Module.

Let's now focus on writing the module. We start with Identity class in the file src/identity/identity.hh:

#pragma once

#include <string>

// we will need these for upcomming functions
#include <boost/functional/hash.hpp>
#include <boost/uuid/string_generator.hpp>
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/uuid_io.hpp>

template <typename T>
class Identity {
  T id_;

public:
  using value_type = T;

  Identity() = default;
  Identity(const Identity &) = default;
  Identity(Identity &&) = default;
  ~Identity() = default;

  Identity &operator=(const Identity &) = default;
  Identity &operator=(Identity &&) = default;

  Identity(T id) : id_{id} {
  }

  friend bool operator==(const Identity &lhs, const Identity &rhs) {
    return lhs.id_ == rhs.id_;
  }

  friend bool operator!=(const Identity &lhs, const Identity &rhs) {
    return !(lhs == rhs);
  }

  std::string toString() const {
    return idToString(id_);
  }

  const value_type &value() const {
    return id_;
  }
};

as you have seen it's a simple value wrapper, I think it doesn't need explanation but we have to write idToString so let's add the header before class Identity and implement it in src/identity/identity.cpp:

// src/identity/identity.hh
#pragma once
//...

std::string idToString(boost::uuids::uuid uuid_);

std::string idToString(std::string str);

template <typename T>
class Identity {
//...
};

// src/identity/identity.cpp

#include "identity.hh"

std::string idToString(boost::uuids::uuid uuid_) {
  return boost::uuids::to_string(uuid_);
}

std::string idToString(std::string str) {
  return str;
}

Here we are using boost::uuids and the function that convert uuid to std:: string. Seastar depends on boost, so we are using it from there, you don't need to add anything to conan.

Spoiler, we will be using uuids as a template argument for Identity class for Clients and std::string for Groups in Core modules. we will look at it in more details.

Next we add an utility function for generating identity from chars:

//src/identity/identity.hpp
Identity<boost::uuids::uuid> makeIdentity(const char *id,
                                          boost::uuids::string_generator &gen) {
  return gen(std::string{id});
}

Don't forget to add the header of this method to identity.hh.

We need to give the user of this module to generate uuids so we are implementing two functions one that generate an Identity<bst::uuid> and another for generating many at once. We will put these into other files, src/identity/generate_identity.hh and src/identity/generate_identity.cpp:

//src/identity/generate_identity.hh
#pragma once

#include <boost/uuid/uuid.hpp>            // uuid class
#include <boost/uuid/uuid_generators.hpp> // generators
#include <boost/uuid/uuid_io.hpp>

#include "identity.hh"

Identity<boost::uuids::uuid> generateIdentity();

std::vector<Identity<boost::uuids::uuid>> generateRange(int nb);

//src/identity/generate_identity.cpp
#include "identity_generator.hh"

Identity<boost::uuids::uuid> generateIdentity() {
  static boost::uuids::random_generator generator{};
  boost::uuids::uuid uuid_ = generator();
  return Identity<boost::uuids::uuid>{uuid_};
}

std::vector<Identity<boost::uuids::uuid>> generateRange(int nb) {
  std::vector<Identity<boost::uuids::uuid>> uuids;
  uuids.reserve(nb);

  while (--nb) {
    uuids.push_back(generateIdentity());
  }

  return uuids;
}

With this cleared out, we are (almost)done with Identity module. we will now move to the Core module where we get to implement our messaging model.

Core module

Before seeing the code, I would like to share the line of thoughts on how I designed this module through a Question-driven approach(at least at the beginning).

We start with this general question: What's the intent of the module? what would it do?. In simple terms, The core module manages Entities of the model.

what do we mean exactly by manages and what are the entities we are referring to?. So manage here is perform the following operations on Clients and Groups(entities): Add, join, leave, register, unregister, delete...

So we have seen the use cases of the module, now let's think about the "API", what exactly are we going to expose and how are we going to design?

For start, we are already using Connection class in server_tcp which belongs to this module, so we know that it's going to be public.

Also, I suggest introducing a container-like class for our entities that exposes the operations discussed above. This class has the role of controlling our entities lifetime and relations. Topology is the name of this class.

To sum-up, the core module exposes two classes: Connection and Topology. Let's look at Topology declaration:

class Topology {
  std::unordered_map<Identity<boost::uuids::uuid>, Client> clients;
  std::unordered_map<Identity<std::string>, Group> groups;

public:
  Topology() = default;
  Topology(const Topology &) = default;
  Topology(Topology &) = default;
  Topology &operator=(Topology &) = default;
  ~Topology() = default;

  bool addGroup(Identity<std::string> id, const std::string &grpType);
  bool removeGroup(Identity<std::string> id);

  bool addClient(Identity<boost::uuids::uuid> id,
                 socket_address sock,
                 uint16_t portRemoteConn);

  std::string updateClient(Identity<boost::uuids::uuid> id,
                           socket_address sock,
                           uint16_t port);

  std::string registerClientAs(Identity<std::string> idGrp,
                               Identity<boost::uuids::uuid> idClt,
                               const std::string &membershipType,
                               const std::string &ann);
  std::string unregisterClientFrom(Identity<std::string> idGrp,
                                   Identity<boost::uuids::uuid> idClt,
                                   const std::string &membershipType);

  std::string pushMessageTo(Identity<std::string> idGrp,
                            Identity<boost::uuids::uuid> idClt,
                            const std::string &msg);

  size_t groupsSize() const;
  size_t clientsSize() const;

  bool clientExists(Identity<boost::uuids::uuid> id) const;
  bool groupExists(Identity<std::string> id) const;
};

I prefer to leave the definition of the methods of this class to the next part because several of them involve details of the MOZA protocol. I think it's better to show the implementation in the following article.

Let's now focus on implementing the messaging model elements that we will be controlling through Topology class. Client class is the first one we are going to discuss. This class holds the state of the external apps needed for message exchange. To be more precise, This is the Context element of our messaging model. The name I choose may appear confusing because we have a Client concept, so why are we mapping Context to Client class? I added the Client concept only for convenience, so we don't need it in the implementation. Here is the code for Client:

//src/lib/core/client.hh
#include <algorithm>
#include <iostream>
#include <map>
#include <string>

#include <boost/uuid/uuid.hpp>

#include <seastar/core/future-util.hh>
#include <seastar/core/sleep.hh>
#include <seastar/core/queue.hh>
#include <seastar/core/reactor.hh>
#include <seastar/core/temporary_buffer.hh>
#include <seastar/net/api.hh>
#include <seastar/net/inet_address.hh>

#include "connection.hh"
#include "lib/identity/identity.hh"

using namespace seastar;
using namespace net;

class Client {
  Identity<boost::uuids::uuid> m_id;
  socket_address m_remoteAddress;
  uint16_t m_remotePort;
  std::map<std::string, seastar::queue<std::string>> m_channels;
  bool m_connected;

  //...

public:
  Client() = default;
  Client(const Client &) = delete;
  Client(Client &&) = default;
  ~Client() = default;

  Client &operator=(const Client &) = delete;
  Client &operator=(Client &&) = default;

  Client(Identity<boost::uuids::uuid> id, socket_address addr, uint16_t rPort)
    : m_id{id}, m_remoteAddress{addr}, m_remotePort{rPort}, m_connected{false} {
  }

  uint16_t getClientPort() const {
    return m_remotePort;
  }

  void setClientPort(uint16_t prt) {
    m_remotePort = prt;
  }

  socket_address getClientAddress() const {
    return m_remoteAddress;
  }

  void setClientAddress(socket_address addr) {
    m_remoteAddress = addr;
  }

  const Identity<boost::uuids::uuid> &identity() const {
    return m_id;
  }

  std::string addChannel(const std::string &groupNme) {
    this->m_channels.try_emplace(groupNme, 128);
    return groupNme;
  }

  bool push(const std::string &grpNme, std::string msg) {
    auto &qu = this->m_channels.at(grpNme);
    return qu.push(std::move(msg));
  }

  void startDelivery() {
    //...
  }
};

As you can see, each Client instance is uniquely identified with a uuid stored as one of its attributes. Moreover, we store its remote address and port. This is needed when delivering messages. In addition, when a client joins a group as a consumer, it adds a dedicated channel(here a seastar::queue) and uses the name of the group as a key in the std::map.

Note that I omitted parts of the code( start delivery, etc..) because it implements one of the aspects of the protocol discussed in the next article.

After we have shown Client class, let's pass to the implementation of the Group concept from our model.

Group is a central element in our model. it is constructed from many other elements. I follow a top-down approach in the implementation of this concept. That means we will start with a top-level overview of the Group then go down to other concepts that make the group.

Let's talk about the attribute of Group. First, a group has a name that uniquely identifies it from the others. Secondly, a group has members each one with a specific role: it has a single member that produces messages and can have many consumers members. This is the state of the class representing the group. In addition to this, the main responsibility of a group is to deliver received messages to the consumers. However, as we discussed above, we have 3 policies of sending messages: single delivery, per-match, and publisher/subscriber.

In the previous paragraph, we have gathered a kind of specification for Group Concept on which we can start taking some design decision and go for the coding. One of the questions we need to ask is how are we going to implement the different policies of the delivery process? For this I suggest using polymorphism: have a base class that implements the common things and delegate differences of each policy to the other derived classes. Moreover, I prefer exposing a facade to hide the base class(a wrapper) and control instantiation via a factory method. I used Group as the name of the facade and GroupImpl as the base class. For the derived classes, we have 3: GroupSingleDelivery, GroupPerMatch, GroupPubSub. The following picture shows how each method in Group calls the one in GroupImpl and that Group holds a unique_ptr to GroupImpl.

We also show the methods and attributes of GroupImpl along with the derived classes with this picture:

Note that _distributeMessage(...) is a pure virtual member function.

Now that we exposed the design of the Group concept, we next show the implementation so here is Group code:

//src/lib/core/group.hh
#pragma once

#include <algorithm>
#include <iostream>
#include <map>
#include <memory>
#include <string>
#include <string_view>

#include <boost/uuid/uuid.hpp>

#include <seastar/core/future-util.hh>
#include <seastar/core/sleep.hh>
#include <seastar/core/queue.hh>
#include <seastar/core/reactor.hh>
#include <seastar/core/temporary_buffer.hh>
#include <seastar/net/api.hh>
#include <seastar/net/inet_address.hh>

#include "lib/identity/identity.hh"
#include "member.hh"

using namespace seastar;
using namespace net;
//...
class Group {
  std::unique_ptr<GroupImpl> m_impl;

public:
  Group(std::unique_ptr<GroupImpl> grp) : m_impl{std::move(grp)} {
  }

  Group() = default;
  Group(Group &&) = default;
  Group &operator=(Group &&) = default;
  ~Group() = default;

  bool isProducer(Identity<boost::uuids::uuid> id) {
    return m_impl->_isProducer(id);
  }
  bool isConsumer(Identity<boost::uuids::uuid> id) {
    return m_impl->_isConsumer(id);
  }

  bool setProducer(Member prod) {
    return m_impl->_setProducer(prod);
  }

  void unsetProducer() {
    m_impl->_unsetProducer();
  }

  bool addConsumer(Member cons) {
    return m_impl->_addConsumer(cons);
  }

  void removeConsumer(Identity<boost::uuids::uuid> id) {
    m_impl->_removeConsumer(id);
  }

  std::string distributeMessage(std::string msg) {
    return m_impl->_distributeMessage(msg);
  }

  bool producerIsSet() const {
    return m_impl->_producerIsSet();
  }

  Identity<std::string> identity() const {
    return m_impl->_identity();
  }
};
//...

As you can see, all methods mirror their counterparts, prefixed with _, from GroupImpl.

GroupImpl holds the state and implements all member functions except for _distributeMessage(...), so let's take a look at the code:

//src/lib/core/group.hh
#pragma
//includes...

class GroupImpl {
protected:
  Identity<std::string> m_identity;
  Member m_producer;
  std::vector<Member> m_consumers;
  bool m_prodSet = false;

public:
  GroupImpl() = default;
  GroupImpl(const GroupImpl &) = delete;
  GroupImpl(GroupImpl &&) = default;
  ~GroupImpl() = default;

  GroupImpl &operator=(const GroupImpl &) = delete;
  GroupImpl &operator=(GroupImpl &&) = default;

  GroupImpl(const std::string &str)
    : m_identity{str}, m_producer{}, m_consumers{}, m_prodSet{false} {
  }

  GroupImpl(Identity<std::string> id)
    : m_identity{id}, m_producer{}, m_consumers{}, m_prodSet{false} {
  }

  bool _isProducer(Identity<boost::uuids::uuid> id);
  bool _isConsumer(Identity<boost::uuids::uuid> id);

  bool _setProducer(Member prod);
  bool _addConsumer(Member cons);

  void _unsetProducer();
  void _removeConsumer(Identity<boost::uuids::uuid> id);

  bool _producerIsSet() const;

  Identity<std::string> _identity() const;

  void notify();

  virtual std::string _distributeMessage(std::string msg) = 0;
};

class Group {
//...
};

//...

Note that GroupImpl is before Group so make sure you respect the order. This is because we declared and defined Group in the header file so the compiler needs to know the full type of GroupImpl because we are using its methods in the definition of Group.

GroupImpl data members are marked as protected so that we can access them in derived classes. Moreover, we introduced Member struct representing the Member concept in our model. We will discuss its implementation after we deal with the derived classes from GroupImpl. Let's now look into the definition of GroupImpl member functions:

//src/lib/core/group.cpp
#include "group.hh"

bool GroupImpl::_isProducer(Identity<boost::uuids::uuid> id) {

  if (!this->_producerIsSet())
    return false;
  return m_producer.clt->identity() == id;
}
bool GroupImpl::_isConsumer(Identity<boost::uuids::uuid> id) {
  return std::any_of(
      begin(m_consumers), end(m_consumers),
      [&id](const Member &mbr) { return mbr.clt->identity() == id; });
}

bool GroupImpl::_setProducer(Member prod) {
  if (_isConsumer(prod.clt->identity()))
    return false;

  m_producer = prod;
  m_prodSet = true;
  return true;
}
bool GroupImpl::_addConsumer(Member cons) {
  if (_isProducer(cons.clt->identity()) || _isConsumer(cons.clt->identity()))
    return false;

  m_consumers.push_back(cons);
  return true;
}

void GroupImpl::_unsetProducer() {
  m_prodSet = false;
  m_producer.clt = nullptr;
}

void GroupImpl::_removeConsumer(Identity<boost::uuids::uuid> id) {
  this->m_consumers.erase(std::remove_if(begin(m_consumers), end(m_consumers),
                                         [&id](const Member &mbr) {
                                           return mbr.clt->identity() == id;
                                         }),
                          m_consumers.end());
}

bool GroupImpl::_producerIsSet() const {
  return m_prodSet;
}

Identity<std::string> GroupImpl::_identity() const {
  return m_identity;
}

I think the code presented here is clear but if not, feel free to leave a comment, I will happily explain.

With GroupImpl out of the way, let's pass to the derived classes. We start with GroupSingleDelivery but I will remove part of codes that have to do with the protocol:

//src/lib/core/group.hh
//includes
//class GroupImpl
//class group
//...
class GroupSingleDelivery : public GroupImpl {
  int nextDst = 0;

public:
  using GroupImpl::GroupImpl;
  std::string _distributeMessage(std::string msg) {
    auto consSize = m_consumers.size();
    if (consSize == 0)
      return //...;
    if (!this->m_consumers[nextDst % consSize].clt->push(m_identity.toString(),
                                                         msg))
      return //...;
    using namespace std::chrono_literals;
    seastar::sleep(1s).then([this, consSize] {this->m_consumers[nextDst % consSize].clt->startDelivery();});
    ++nextDst;
    return //...;
  }
};
//...

GroupSingleDelivery class send a received message to only one of its consumers and it keeps track of next destination using nextDst data member modulus the size of consumers in the current group.

std::string _distributeMessage(std::string msg) takes the message we want to deliver and push it to one of the consumers queue then trigger the delivery process.

Next is GroupPubSub class. Here is the code :

//src/lib/core/group.hh
//...
//class GroupSingleDe...
//...
class GroupPubSub : public GroupImpl {

public:
  using GroupImpl::GroupImpl;
  std::string _distributeMessage(std::string msg) {

    auto consSize = m_consumers.size();
    if (consSize == 0)
      return //...;
    bool queuedForAtLeastOne = false;
    for (const auto &elm : m_consumers) {
      queuedForAtLeastOne =
           elm.clt->push(m_identity.toString(), msg) || queuedForAtLeastOne ;
      elm.clt->startDelivery();
    }
    if (!queuedForAtLeastOne)
      return //...;
    return //...;
  }
};

In this version, the message gets delivered to all consumers, hence the range-based for loop iterating over the vector of consumers and pushing the message into each Client channel and start the delivery process for each consumer.

Instead of showing the code of GroupPerMatch I will jump to other concepts from the model that we are using and that we didn't define them. _distributeMessage of GroupPerMatch use a bit more of Member so I think it makes sense to go deep(Member, Association,...) and then return to finish the last derived class even though.

Member class is an abstraction that points to an actual Client member of a Group and holds information needed for the pushing and delivery process:

//src/lib/core/member.hh
#pragma once

#include <string>

#include "client.hh"
//...
struct Member {
  Client *clt;
  Association ass;

  Member() : clt{nullptr}, ass{""} {
  }
  ~Member() = default;
};

As you may have guessed, Association is the class that holds the data needed for the business logic of per-match distribution and offer to verify the criteria. This is how it's implemented:

//src/lib/core/member.hh
class Association {
  std::string m_annotation;
  std::string m_queueName;

public:
  Association(std::string qn, std::string ann)
    : m_annotation{std::move(ann)}, m_queueName{std::move(qn)} {
  }
  Association(std::string qn) : m_annotation{}, m_queueName{std::move(qn)} {
  }

  bool hasAnnotation() const {
    return m_annotation != "";
  }

  bool verify(const std::string &ann) const {
    return m_annotation == ann;
  }

  const std::string &dstChannelName() const {
    return m_queueName;
  }
};

//struct Member...

Association holds an annotation used in the per-match delivery policy and a name of the queue where to push the message. With this, we have finished implementing concept related to Group but still, we have to finish GroupPerMatch.

Disclaimer: When designing the protocol and developing the modules I forgot about something very important for _distributeMessage of GroupPerMatch. This leads to an ugly solution or rather say a hack to make it work and not something clean and sensable. So what I am presenting here is literally my mistake and not something you would take but rather avoid. I couldn't make a proper fix because I have almost no time left and I needed it to work.

What I forgot about is to pass the annotation as an argument in the protocol and I already implemented the parsing so I went to actually encode it with the data, hence, rather than getting it for free from the protocol parser I had to inject a parsing function inside the _distributeMessage method. Moreover, I didn't account for the extra annotation argument in _distributeMessage method. In the ideal case the signature would be virtual std::string _distributeMessage(std::string msg, std::string ann=0) = 0; and the argument will be extracted using the protocol parser.

What I ended up doing was encoding the annotation inside the actual data like so: *****info-Hello where Hello is the data to be delivered and info is the annotation delimited by ***** and -.

//...
class GroupPerMatch : public GroupImpl {

public:
  using GroupImpl::GroupImpl;
  std::string _distributeMessage(std::string msg) {

    auto consSize = m_consumers.size();
    if (consSize == 0)
      return //...;

    auto extractCritere =
        [](std::string &data) -> compat::optional<std::string> {
      const std::string opMark = "******";
      auto first = begin(data);
      auto last = begin(data) + 7;
      std::string_view str{data.c_str(), 7};

      if (str == opMark)
        return {};
      std::string ann;
      ann.reserve(256);

      auto _pos = std::find(begin(data) + 7, end(data), '-');
      if (_pos == end(data))
        return {};

      std::copy(begin(data) + 7, _pos, std::back_inserter(ann));

      data.erase(std::rotate(begin(data), _pos + 1, end(data)));

      return {ann};
    };

    auto critere = extractCritere(msg);

    if (!critere.has_value())
      return //...;
    auto crt = critere.value();

    auto start = begin(m_consumers);

    bool matchedOne = false;

    do {

      auto cons =
          std::find_if(start, end(m_consumers), [&crt](const Member &meb) {
            return meb.ass.verify(crt);
          });
      if (cons == end(m_consumers)) {
        if (!matchedOne)
          return //....;
        return //...;
      }
      cons->clt->push(m_identity.toString(), msg);
      cons->clt->startDelivery();
      matchedOne = true;
      start = cons + 1;
    } while (true);
  }
};
//...class GroupPubSub

So we start with our usual check for the number of consumers then we define a lambda that will parse and extract the encoded annotation if any. This lambda returns an optional. The do {} while(); loop is to check for the consumers that verifies the critiria so that we can push and start the delivery process.

As I said in the disclaimer, this is not the correct way of doing it. Just something to make work. In a future article, I will be addressing this.

Another pitfall in my design is that the core module contains pieces of code directly related to the protocol module, hence an understanding of the protocol is required before seeing them. This is why we will be leaving the definition of the Topology class methods for the next article along with the protocol module. Also, I will suggest a way to get read of this direct dependency.

Conclusion

We have covered the messaging model of our Message-oriented middleware and also have shown most of the implementation. I hope you take this model as a source of inspiration when designing your own.

Next Part

In the next article, we will be talking about the protocol MOZA, that enables communication between applications and the middleware.

Rolling your own MOM, or how I did it - Project setup

cppchedy — Thu, 18 Jul 2019 14:28:29 +0000

Introduction

If you were bored reading the previous article, sorry about that, yes it was boring but unfortunately it's required. However, from now on, We get to the concrete stuff where you could test and follow along if you want.

The main point of this part is to present the entry point of the middleware, the top level class of the code. Therefore, we will be building up to reach that goal through a Seastar introduction, the project structure and finally get to code.

Seastar

Seastar is an open source c++ framework designed to help build highly-performant server application. It's used by ScyllaDb, a high-performance NoSQL Database.

One of the key feature of Seastar is it's event-driven nature which makes it easy for it's user to write non-blocking, asynchronous code. Moreover, Seastar architecture is based on the following points:

Shared-nothing Design: Seastar uses a shared-nothing model that shards all requests onto individual cores.
Futures and Promises: An advanced new model for concurrent applications that offers C++ programmers both high performance and the ability to create comprehensible, testable high-quality code.
High-performance networking : Seastar offers a choice of network stack, including conventional Linux networking for ease of development, DPDK for fast user-space networking on Linux, and native networking on OSv
Messaging passing : A design for sharing information between CPU cores without time-consuming locking.

A little detour

Giving that I will be using Seastar through out this series, I thought that I should include a "How to use Seastar" section. I think it make sense to have this detour before going further because it will make explanations of the MOM code much simpler.

I assume that you are already familiar with the concept of Future and promise. If you would like to dig deep into this amazing framework and learn it properly then I recommend this: Tutorial, Demos and "real world" apps.

To keep things short, We will see only basic examples and present the skeleton of Seastar programs.

Program Structure

Let's start with the general structure of a Seastar program:

#include <seastar/core/app-template.hh>
//other includes...

int main(int argc, char** argv) {
    seastar::app_template app;
    /*
      if you want to add option for your application
    */
    app.add_options()
        ("foo", bpo::value<std::string>(), "foo arg")
        ;
    app.run(argc, argv, [] {
            //write your logic here
            //... could retrieve args

            return /*return a future*/;
    });
}

Generally, every Seastar program you will write, will follow this structure: you instantiate an object of type app_template and then call on run method which takes 3 args. The last one is the most relevant for us: it's a lambda containing the logic of our application.
The app_template manage configuration needed by Seastar engine: logging config, CPU config, memory configs, etc... when we start our app, we can tweak those args as needed. For example, you can control the number of threads by issuing -cNB where NB is the number of threads you want. Note that if you choose a number higher then the number of hardware threads you have on your machine, you will get a runtime exception. I like this choice, it reduces the overhead caused by threads switches.

As for run, it starts the runtime(scheduler,eventloop....) that take care of executing handlers. we will discuss this later on.

Examples

Although using Seastar brings many advantages like performance boost, Seastar asynchronous programming model comes with it's own trouble: We write code in an unconventional way and we have to pay attention to new things. Many assumptions we hold are no longer true and this is because of Seastar asynchronous, non-blocking aspect. The following examples shows some of the issues we need to be aware of and how we can address them.

The first example is for introduction. it just sleeps about two seconds and print out "this is not the right way to learn Seastar."

#include <seastar/core/app-template.hh>
#include <seastar/core/sleep.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    app.run(argc, argv, [] {
        using namespace std::chrono_literals;
        return seastar::sleep(2s).then([] {
            std::cout << "this is not the right way to learn Seastar.\n";
        });
    });
}

Note you can see the code in compiler explorer.

In this program, sleep sets up a timer and return a future that will resolve in 2s. We chain on this future with a lambda that prints "this is.." to std::cout.

The problem we have to constantly think about is object lifetime or scope. Unlike nodejs/javascript's closure, C++'s lambda doesn't spare us from dealing with object/variable accessibility. Depending on what we are doing, we can access an invalid state resulting in wrong results or we could get a runtime error. The following example demonstrates a common mistake that new users of Seastar make:

//includes ...

seastar::future<> dummy_fn(int offset) {
  int bad = 8;
  return seastar::make_ready_future<int>(offset)
      .then([&bad](int nb) { bad += nb; })
      .then([&bad] {
        bad++;
        std::cout << bad << '\n';
      });
}

int main(int argc, char **argv) {
  seastar::app_template app;
  return app.run(argc, argv, [] {
    auto ftr1 = dummy_fn(12);
    auto ftr2 = dummy_fn(46);

    return when_all(std::move(ftr1), std::move(ftr2))
        .then([](std::tuple<seastar::future<> , seastar::future<>> tf) {
            std::cout << "done computing";
            return 0;
        });
  });
}

You can check out the code in Compiler Explorer.

First let's analyze main: as usual, we have app_template and we call run to start Seastar engine and execute the task. Next the lambda: we call dummy_fn two times. This function does nothing special, it's only purpose is to show the mistake. For now, let's only keep in mind that it returns a future. The last statement start with when_all. This means that we will be waiting for futures ftr1 and ftr2 to resolve so that we can carry on with the next task(printing "computation done").

Now let's take a look at dummy_fn: We have a variable bad on which some calculation is performed. these calculation are dispatched in lambdas chained one after another. before that, we make a ready_future(one that resolves immediately). this future resolve to an int taken from the argument offset. The problem is the capture by reference of bad : remember that we are executing asynchronously and without blocking which means we are submitting tasks to Seastar scheduler and not executing them, therefore, when the lambdas will run, bad will be already destroyed because we returned from dummy_fn and we will get wrong results.

So a general rule, if you are capturing things by reference you will mostly get in trouble unless you know what you are doing. so the default here will be capture by value.

Seastar developer thought already about this problem and supplied us with the necessary utilities.

To correct the precedent example, we could use do_with function. Then dummy_fn becomes :

seastar::future<> dummy_fn(int offset) {
  return seastar::do_with(int{8}, [offset](int &bad) {
    return seastar::make_ready_future<int>(offset)
        .then([&bad](int nb) { bad += nb; })
        .then([&bad] {
          bad++;
          std::cout << bad << '\n';
          return seastar::make_ready_future();
        });
  });
}

We solve the dangling ref problem by using do_with. This utility manages for us the lifetime of object until we no longer need to use it. In short, it allocates memory and moves the values we will be using there then when the lamnda finish executing, it frees the memory. You should note that Seastar manages it's own memory which is pre-allocated from the start. Also note that you can tweak memory allocation size.

In addition to the previous solution, We can use reference counting to keep the variable alive until we finish computing. Using seastar::shared_ptr, dummy_fn becomes as follows:

seastar::future<> dummy_fn(int offset) {
  auto bad = seastar::make_shared<int>(8);
  return seastar::make_ready_future<int>(offset)
      .then([bad](int nb) { *bad += nb; })
      .then([bad] {
        (*bad)++;
        std::cout << *bad << '\n';
      });
}

Seastar offer it's own shared_ptr which is a non-thread safe version of std::shared_ptr.
In dummy_fn, we make a sstar shared_ptr by calling make_shared and capture it by copy to all other lambdas. This way, we no longer have the life time problem.

Generally, we will be using shared_ptr but when we have a variable needed within a specific scope we may use do_with.

Our final example is rather oriented to expose more of Seastar constructs then to show a specific problem. With this sample, we introduce looping. Seastar offer an elegant solution for looping: keep_doing, repeat, repeat_until_value, etc.... The following is an example of using do_until. We get a string with --message option then we print it char by char:

#include <core/app-template.hh>
#include <core/sleep.hh>
#include <iostream>
#include <string>

namespace bst = boost::program_options;

int main(int argc, char **argv) {
  seastar::app_template app;
  app.add_options()("message", bst::value<std::string>(), "Message to print");
  return app.run(argc, argv, [&app] {
    auto &&config = app.configuration();
    seastar::do_with(config["message"].as<std::string>(), int{0},
                     [](std::string &msg, int &i) {
                       return seastar::do_until(
                           [&i, &msg] { return i >= msg.size(); },
                           [&i, &msg] { 
                               std::cout << msg[i] << '\n';
                               ++i;
                               return seastar::make_ready_future();
                           });
                     });
    return seastar::make_ready_future();
  });
}

Compiler explorer link

We begin by adding message option by calling add_options. In main lambda, we start by getting the configuration and passing the value of --message option to do_with along with a counter initialized with 0.

I should point out that the lambda passed to do_with must respect the order and the type of the other parameters: for example if we want do_with to manage an int and float we write:
do_with(int{9}, float{13.01}, [](int &arg1, float& arg2) { ...});
I hope I was clear on this point.

Now, let's get to the looping part, do_until. This function repeats a task until some condition is met. It accepts two lambdas, the first one is a predicate and the second one is the body of the loop. We check each time the value of the counter with respect to the string size. The loop end when i becames greater then the size of msg.
For the body, we are printing each char and advancing the counter.

Hopefully, we can now go back to the MOM with a basic understanding of Seastar.

First steps

Setting up the project

First, we are going write cmake Files and prepare our dependency with conanfile.txt. I am assuming that you have installed cmake and conan So let's proceed to project structure:

As you can see from the picture, We have 3 main folders: one for cmake scripts, another one that contains the application it self, src, and a folder containing tests.

To integrate clang-format and clang-tidy with cmake, I found these cmake-scripts. note that there are other goodies there. all credits go to maintainers. copy the files into the cmake folder.

let's take care of conanfile.txt

[requires]
gtest/1.8.0@lasote/stable
spdlog/1.3.1@bincrafters/stable 

[generators]
cmake

we are using google's lib gtests for our tests and spdlog for logging. And we are specifying cmake as our generator.

root CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)
project(Moza)

#includes cmake scripts
include(cmake/c++-standards.cmake)
include(cmake/compiler-options.cmake)
include(cmake/sanitizers.cmake)
include(cmake/formatting.cmake)
include(cmake/tools.cmake)

# require Seastar and use it's boost dependency
find_package(Seastar REQUIRED)
include_directories(${Boost_INCLUDE_DIR})

#include conan generated file for camke 
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup(TARGETS)

#instruct seastar to use std::optional, std::variant etc, instead of it's own impl
add_definitions(-DSEASTAR_USE_STD_OPTIONAL_VARIANT_STRINGVIEW)

# clang format and clang-tidy integration
file(GLOB_RECURSE FFILES *.[chi]pp *.[chi]xx *.cc *.hh *.ii *.[CHI] )
clang_format(format ${FFILES})

clang_tidy(-format-style=file -checks=* -header-filter='${CMAKE_SOURCE_DIR}/*')

# we add only src folder
add_subdirectory(src)

Depending on the way you compiled and installed Seastar, you may need to add extras arguments for cmake so that it can find the required library. I will show how I am compiling the project later on in this article.

Next, let's look at CMakeLists.txt inside src folder:

file(GLOB SRC_CORE lib/core/*.cpp lib/core/*.hh)


add_library(MozaLib ${SRC_CORE})
target_link_libraries(MozaLib Seastar::seastar CONAN_PKG::spdlog)


file(GLOB SRC_SERV server/*.cpp)
add_executable(Moza main.cpp ${SRC_SERV})
target_link_libraries(Moza Seastar::seastar MozaLib CONAN_PKG::spdlog)

We are done with preparation now so let's now get to the source folder. I structured src in a way that separate modules from the actual logic of the MOM: We have a lib sub folders containing our modules, server containing the source code of the class server and a main.cpp file.

Main

This is the lisiting of main.cpp:

#include <seastar/core/app-template.hh>
#include <seastar/core/distributed.hh>

#include "server/server.hh"

int main(int ac, char **av) {
  using namespace seastar;
  using namespace net;
  distributed<server_tcp> tcp_server;
  app_template app;
  return app.run(ac, av, [&] {
    engine().at_exit([&] { return tcp_server.stop(); });
    // here is not there
    return tcp_server.start_single().then(
        [&tcp_server] { return tcp_server.invoke_on(0, &server_tcp::start); });
  });
}

As evoked earlier, Seastar is an event-driven framework build around the Reactor pattern. Each Seastar application has a Reactor(or an event-loop) for each logical core(or hardware thread), of course assuming default configurations(no --c flag). These event-loop are completely independent from each other(shared nothing design).

To help us shard our logic over the available core, Seastar developer introduced seastar::distributed. This utility take core of distributing the service(in our case server_tcp) passed as a template argument to all hardware threads.

we start by getting the local reactor instance of the executing core and attach a lambda that shutdown our service when we are leaving. distributed require the service to have a stop method. that's the only requirement.

Next, we call start_sinlge then invoke server_tcp::start on the shard(core) 0. start_sinlge activate only one shard.

previously I said that I wasn't using Seastar full potential. You can see that here: I am using only one shard. This is actually a mistake in my design. I didn't account for sharding from the beginning, I totlally forgot. I was stressed because deadlines are approching So even when I saw my mistake I didn't take the time to fix it even though I thought of a couple of solutions but they required changes to the protocol, something I didn't want to mess with. Hopefully, This will a topic for another post.

Server class

Let's focus now on the server, our service class. Let's get an overview before digging into the code. Note that the move and copy constructors/assignment_ops are deleted.

The following is the listing of src/server/server.hh :

#pragma once

#include <seastar/core/reactor.hh>
#include <seastar/core/temporary_buffer.hh>
#include <seastar/net/api.hh>

#include "spdlog/spdlog.h"

#include "lib/core/connection.hh"
#include "lib/core/topology.hh"
#include "lib/protocol/parser.hh"
#include "lib/identity/identity.hh"
#include "lib/identity/identity_generator.hh"

using namespace seastar;
using namespace net;


class server_tcp {
  Topology _tp;
  lw_shared_ptr<server_socket> _listener;
  std::vector<lw_shared_ptr<Connection>> connections;
  uint16_t _port = 11212;

  std::string handleRequest(Specification sp, socket_address sco);

public:
  server_tcp() =default;
  ~server_tcp() =default;

  server_tcp(const server_tcp&) =delete;
  server_tcp(server_tcp&& ) =delete;

  server_tcp& operator=(const server_tcp&) =delete;
  server_tcp& operator=(server_tcp&&) =delete;

  server_tcp(uint16_t port) : _port{port} { }

  void start();
  future<> stop();

  future<> handleConnection(lw_shared_ptr<Connection> conn);

};

Server stores a shared ptr to a socket for listening, the port it will be listening on and an object of type Topology. let's ignore the vector of connection for now.

Topology is the core of the server. it holds all the entitis of our Messaging model, we will discussed in more details in the next part of this series.

let's focus on start and handleConnection.

void server_tcp::start() {
  spdlog::info("server is staring...");
  listen_options lo;
  lo.reuse_address = true;
  _listener = engine().listen(make_ipv4_address({_port}), lo);

  spdlog::info("listening on port {}", _port);
  keep_doing([this] {
    return _listener->accept().then(
        [this](connected_socket fd, socket_address addr) mutable {
          auto conn = make_lw_shared<Connection>(std::move(fd), addr);
          handleConnection(conn);
        });
  });
}

start member function is simple. it makes a listener with the method listen of reactor class. we get the local instance of reactor through the call of engine function. Next, we start accepting connection with accept method from the listener and handle connections. We repeat this process using keep_doing utility from Seastar. Note that we are storing file descriptor(connected_socket) and the address of the client(socket_address) in a dedicated class, Connection. It enables communication with the client in both directions. let's look at the code:

//src/lib/core/connection.hh
#pragma once

#include <seastar/core/temporary_buffer.hh>
#include <seastar/net/api.hh>

using namespace seastar;
using namespace net;


class Connection {
public:
    connected_socket m_fd;
    input_stream<char> m_in;
    output_stream<char> m_out;
    socket_address m_address;
    bool m_enabled;

  public:
    Connection(connected_socket fd, socket_address addr)
      : m_fd{std::move(fd)}, m_in{m_fd.input()}, m_out{m_fd.output()}, m_address{addr}, m_enabled{true} {
      m_fd.set_keepalive(true);
      m_fd.set_nodelay(true);
    }

    Connection(Connection &&orig)
      : m_fd{std::move(orig.m_fd)}
      , m_in{std::move(orig.m_in)}
      , m_out{std::move(orig.m_out)}
      , m_address{orig.m_address}
      , m_enabled{orig.m_enabled} {
    }

    Connection &operator=(const Connection &rhs) = delete;


    Connection &operator=(Connection &&rhs) = default;

    ~Connection() {
      closeStream();
    }

    void closeStream() {
      m_enabled = false;
      m_fd.shutdown_input();
      m_fd.shutdown_output();
    }

    future<temporary_buffer<char>> readFromStream() {
      return m_in.read();
    }


    bool isAlive() {
      return m_enabled;
    }

    socket_address getRemoteAddress() const {
      return this->m_address;
    }
  };

The most important bits are output_stream and input_stream with which we can read/write from/to the client. We open the streams with input() and output() in Connection constructor.

Let's pass to handleConnection. This method reads from socket, parse the packet and execute the task(and finally report back). AS for now we are missing many modules and we didn't explain yet the messaging model nor the protocol, therefore, we will write it such that it reads some byte, log the request and send a response. Expect it to change when we advance in the series.

seastar::future<> server_tcp::handleConnection(lw_shared_ptr<Connection> conn) {
  return conn->readFromStream()
      .then([this](temporary_buffer<char> buf) {
        std::string str;
        std::copy(std::begin(buf), std::end(buf), std::back_inserter(str));
        spdlog::info("received data : {}", str);
      })
      .then([this, conn] {
        if (conn->isAlive()) {
          return conn->m_out.write("WYSIWYG")
              .then([conn] { return conn->m_out.flush(); })//lambda 1
              .then([conn] { return conn->m_out.close(); })//lambda 2
              .then([conn] {});//lambda 3
        }
        return make_ready_future();
      });
}

readFromStream returns a future of temporary_buffer passed for the next continuation in the chain when it's ready. temporary_buffer is an efficient container designed with the zero copy idiom and is move only. We copy the buffer into a std::string and log the data. Next we send a message to the client.

Note the use of flush after write. we make sure the data is send to the client and we close the connection.

Notice also that the last lambda in the chain is empty but it captures the conn by copy. This line is required to prevent an access to a destroyed object(in our case it's m_out): calling conn->m_out.close(); arranges for the connection to be closed and returns a future representing that event. Excluding lambda 3 from the chain, triggers the destruction of conn at the end of lambda 2. However, we still need m_out to be alive until we finish with closing the connection. This is why we need it, even though it's empty.

For now stop returns a ready future :

future<> server_tcp::stop() {
  return make_ready_future<>();
}

Now that we finished start and handleConnection we can pass to compiling the project.

Compiling the project

We will start by listing commands you need to type in order to compile the project then speak about some technicalties you need to be aware of.

To compile the project first you need to make a new folder in the root directory called build. This contains all results of compilation, conan, etc....

mkdir build

we cd to build then type:

conan install ..

This will install all required package described in the conanfile.txt
, specifially gtest and spdlog.

Next, we build the project with cmake:

seastar_dir=/path/to/seastar
cd ..
CC=gcc-8 CXX=g++-8 cmake -DCMAKE_PREFIX_PATH="$seastar_dir/build/;$seastar_dir/build/_cooking/installed" -DCMAKE_MODULE_PATH=$seastar_dir/cmake -H. -Bbuild

You need to point to Seastar's cmake modules required for find_package and also where you installed Seastar.

I should say that you need to comment the includes lines from the files we worked on and the Topology attribute of server_tcp before building because we didn't work on them yet.

Also, Note that I added my own profile for conan so when I run it I pass an extra command: -pr=myProf. The profile file is located in ~/.conan/profiles if you keeped the default configuration when you installed conan. let's take a look at the file:

[settings]
os=Linux
arch=x86_64
compiler=gcc
compiler.version=8.1
compiler.libcxx=libstdc++
build_type=Release
[options]
[build_requires]
[env]

I build Seastar using the cooking.sh way from their how to on building Seastar.

Conclusion

In this post we learned how to use Seastar and we wrote the first building blocks of our project. I hope you have enjoyed reading.

Next Part

Next on this series, We will discuss the Messaging model of our system, the core of MOM.

Rolling out your own MOM or how I did it - Overview

cppchedy — Thu, 18 Jul 2019 14:25:19 +0000

Introduction

This article is meant to give the reader an overview of the whole system . Specifically, we show the global architecture, it's building blocks (such as the Messaging Model), the modules we will be developing and the tools/framework we will use.

Architecture of the System

Our system is composed mainly of a two parts: a client and a "server"(or the actual middleware) as depicted by the following picture:

The client is used by enterprise applications to communicate with the middleware. The communication is done using Moza protocol which is part of the project. By that, I mean we will be designing this protocol for our specific usage. More on this later.

Middleware

The middleware is the central component of the system. Its main role is to deliver received messages to there appropriate destination. However, the delivery process isn't unique: for example, the producer of the messages may want to send it to many consumers, or it may want it to be delivered to a specific consumer or even it may only care that the message get delivered to someone ignoring who may it be. The stated scenario bring up the problem of specifying a policy of message distribution: By which mean users will express their intent and choose their desired setup?

Messaging Model

This issue is addressed by designing a coherent messaging model that captures the many aspects involved in the process of message delivery. To be more precise, using elements of the model, we can communicate to the middleware the way we are going to layout/organize message exchanges between the different applications in need for integration.

As we have seen, The messaging model plays an essential role in our system. The middleware need to be aware of the topology that the user designed for message circulation in order to do it's job. So beforehand, the user has to configure the instance of the server according to his problem. Part III speaks about this in more details.

Moza protocol

Wither to send a message or to configure the middleware, we need a way to inform the addressed system. For this project, I decided to design a protocol specifically for these tasks(the configuration and the message delivery). It specifies the format, the rules of communication and the PDUs involved in this process. We will have an in-depth look into Moza in Part IV of this series.

Modules

During the analysis phase, I identified 4 modules:

a network module : responsible for network communication as it names suggest. We will use a framework for this rather then implementing our own solution.
an identity module : it manages the notion of identity used across all part of the code base.
a core module: this is where our messaging model lives. This is central to the MOM.
Moza module : here we implement our protocol.

Each module is covered in one of the next posts.

Frameworks, libs & Tools

The Middleware is implemented using C++17. As for the compiler, I used gcc 8.1. In Addition to that, I used CMake along with conan for package management.

Relying on Seastar for the networking part was a huge boost for me: I learned a couple of new concepts and discovered an elegant code base. This is from an educational perspective. Another important point is the performance gained from using this piece of technology. Having said that, I couldn't take full advantage of it even though I had my idea on how I will do it. I was running out of time so I preferred to keep things simple, and improve things after I pass my end-year project. In the next post of the series, I will give a proper introduction to this framework.

Client

When I started working on the project, I thought I will be implementing the protocol, for the client, and distribute libraries. However, due to time constrains, I fall back to a CLI that can configure the middleware and can simulate consumer and producer. Maybe in The future I will go with implementing libraries and documents the process but for now we have what we have: A CLI implemented with nodejs.

Next part

In the next part, we will start setting up the project and get to the concrete stuff.

Rolling out your own MOM or how I did it - General Introduction

cppchedy — Thu, 18 Jul 2019 14:22:35 +0000

Introduction

This is a series of posts in which I share my experience building a very basic Message-oriented middleware for my end-year project at university. It presents the design of the system and shows it's source code.

Through this project, I want to provide an infrastructure that enables enterprises to implement integration solutions for their systems. Here, by integration I mean communication between different applications of an enterprise. More specifically, This communication follow the messaging integration style.

Warning: This is my first attempt building this kind of project, therefore, you should not take anything here as best practice or the right way to do things unless it's stated with a reference. It's best to consider this as a prototype. Moreover, it was for university so many design decision was taken with that in mind.

In Part I, we will present the architecture of the system and introduce it's main components(Messaging model, protocol, modules...) along with the frameworks/tools we will be using. This is more like a general overview of what we are going to work on.
In Part II, we give a short introduction on Seastar then we start coding our "server", the main entry point of our application. And finally we show how to compile the project.
Part III of this series is on the Messaging Model designed for the MOM. Questions like what is it, why do we need one... will be explored. Also, in this part, we show the implementation of Mess. Model.
Part IV will discuss Moza protocol(weird name I know). Here we will present the protocol, explain it and show the structure of it's PDUs. Moreover, we will implement a cli to interact with our MOM.

Note: I am not sure if this is all I want to write so I may add other parts.

I hope that this series will be helpful and informative for many of you. Specially the protocol section and the use of Seastar. You should note that I didn't use the full potential of Seastar in this first version but I plan to make changes that will improve the performance.

To conclude, I would like to thank god that without him none of this(projects, posts, my C++, name it ...) would had happened. Also, I want to give a special thank you message to: Matt Godbolt for the great discussions we had (either on this project or other cppish things), bjarne stroustrup for C++, Alexander Stepanov for STL and EoP, Sean Parent for his talks, Jon Kalb for Cppcon and the c++ committee for working on making C++ a better language.

DEV Community: cppchedy

Writing your first LLVM Transformation Passes - Dead Code Elimination and Constant Propagation

Introduction

Dead Code Elimination

Our Downsized DCE Pass

Constant Propagation Pass

Constant Propagation Pass for add

Used LLVM facilities

Conclusion

Writing Your First LLVM Plugin Pass: Counting Add Instructions

Introduction

LLVM Passes and Why They Matter

Module Pass

Function Pass

Loop Pass

Pass Manager Machination

Getting our hands dirty

Template For Passes

Registering the Pass

Core Function

Getting our hands dirty for Real

Counting Add instructions Pass

Testing the Pass

Conclusion

Compiling LLVM and Running Your First Dummy Pass

Introduction

Prerequisites and Environment Setup

Step 1: Building LLVM from Source

Step 2: Writing a Hello World LLVM Pass

Step 3: Building the Pass

Step 4: Running the Pass

Wrapping Up

S'tracing WO MIC: a journey involving threading, networking, audio, and more

Architecture of WO MIC

WO MIC's How to

Stracing the WO MIC client

Audio APIs

Network APIs

Detour on threads in Linux

Looking for network system calls in forks

Unexpected Gain

Conclusion

Switching to the NOWeb sphere projects

Introduction

List of resource

Rolling your own MOM, or how I did it - Messaging Model

Introduction

Messaging Model

Elements of the Messaging model

Message distribution policies

TL;DR

Implementation

Identity module

Core module

Conclusion

Next Part

Rolling your own MOM, or how I did it - Project setup

Introduction

Seastar

A little detour

Program Structure

Examples

First steps

Setting up the project

Main

Server class

Compiling the project

Conclusion

Next Part

Rolling out your own MOM or how I did it - Overview

Introduction

Architecture of the System

Middleware

Messaging Model

Moza protocol

Modules

Frameworks, libs & Tools

Client

Next part

Rolling out your own MOM or how I did it - General Introduction

Constant Propagation Pass for `add`

Counting `Add` instructions Pass