DEV Community: Lena

Improve the manipulation of a string at compile time with C++20

Lena — Mon, 28 Aug 2023 17:24:05 +0000

Article::Article

In my last article I showed you how to generate a string during the compilation. In my code I used a std::array to return a string, but honestly I must admit that a std::array is not as convenient as real string class.

That's why I will show you how to create a trivial compile time string class.

The layout

The only thing I need to store is an array of char with a constant size, so let's write it:

template <std::size_t ArraySize>
struct CompileTimeString
{
    char data[ArraySize];
}

I could have use a std::array but honestly in this case it wouldn't really have change anything.

Construct it easily

Writing this is not practical:

// 4 because the \0 is stored
CompileTimeString<4> str = "lol";

It's possible to write something a bit better like this:

constexpr CompileTimeString str {"lol"};
constexpr CompileTimeString str2  = {"lol"};

What if I want to write this:

constexpr CompileTimeString str = "lol";

Then I must add this constructor:

// It takes an array of chars in parameter
constexpr CompileTimeString(const char(&literal)[ArraySize]) noexcept
{
    // Then it copies the content
    std::ranges::copy(literal, data);
}

It's possible do even better thanks to user defined literals.

If you don't what is it I suggested you to read one of these articles, this one in French or this one in English.

To explain it in one sentence, I will be able to write

// hi's type is CompileTimeString<14>
constexpr auto hi = "Hello, World!"_cts;

How?

Like this:

template<CompileTimeString Str>
constexpr auto operator""_cts()
{
    return Str;
}

With C++20 it is possible to define a user defined literal with a template parameter that can be constructed from an array of char or a const char* and we did it just before. Amazing, isn't it?

Really basic iterator stuff

Now that we can construct it, it would be nice to be able to use some algorithm or use it as a range like normal thing.

So we need to implement member functions like begin() and end(). Don't worry, it will be easy because the standard library works well with built in arrays, it is possible to just to this:

constexpr auto begin() noexcept
{
    return std::begin(data); 
}

constexpr auto end() noexcept
{
    return std::end(data);
}

It can also be done for const iterators and reverse iterators.

Conversion to std::string_view

A lot of existing functions in existing code use std::string_view to handle not owning view of strings, it is simple to use and can be constructed from a std::string or a const char* for example making it really practical.

Creating a function to be able to construct a std::string_view from a CompileTimeString is really easy:

constexpr std::string_view as_string_view() const
{
    // Remove 1 because the \0 is not counted is the size in a std::stringi_view
    return std::string_view(data, ArraySize - 1);
}

There's just one issue, it's really easy to shoot yourself in the foot with this. Let me explain, std::string_view is a non-owning view, meaning that it points to something and the thing it points to must still be valid. If the view is created from a temporary, it's easy to use the view after the lifetime of the original string, creating bugs and undefined behavior.

// This one is ok, even if the std::string is temporary, it will be destroyed after the end of the full expression
std::cout << std::string_view(std::string("hi")) << std::endl;

// This one is not ok
std::string_view str = std::string("not good");
// Here the temporary std::string is already destroyed
std::cout << str << std::endl;

I don't really see a valid usecase of creating a std::string_view from a temporary CompileTimeString, so I will forbid it.

In C++, you can have different overloads of a method depending if it is a right value (a temporary for example, or a moved value) or a left value.

It's used in the standard library for std::string::substr() since C++23, if we have a lvalue (left value) it creates a new std:string, if we have a rvalue (right value) it can reuse the storage of the current string instead of allocating a new one.

With lvalue the substr is equivalent to:

return basic_string(*this, pos, count)

And with a rvalue it is the equivalent to:

return basic_string(std::move(*this), pos, count);

To do it you just need to add a & for a left value and && for a right value.

In my case, I will just write this:

constexpr std::string_view as_string_view() const &
{
    return std::string_view(data, ArraySize - 1);
}

constexpr std::string_view as_string_view() const && = delete;

Now that's done, I will write a conversion operator

constexpr operator std::string_view() const &
{
    return as_string_view();
}

constexpr operator std::string_view() const && = delete;

Yes, I know, the syntax feels a bit weird because there is no return type and the name of the function is the return type.

I could have marked it explicit and I think most of the time explicit conversion is better than the implicit one, but in this case I think that the implicit conversion of strings to string views is pretty common and expected, also it is already the behavior for std::string so at least it is coherent.

Add some basic member functions

Strings classes have a lot of utility member functions, but recoding them can be redundant, but as we can create a std::string_view from our class, let's use it in order to make it really simple to implement these methods.

constexpr bool starts_with(std::string_view sv) const noexcept
{
    return as_string_view().starts_with(sv);
}

constexpr bool starts_with(char ch) const noexcept
{
    return as_string_view().starts_with(ch);
}

constexpr bool starts_with(const char* s) const
{
    return as_string_view().starts_with(s);
}

It's not the most exciting task but at least there's little room for error.

Concatenate compile time string with operator+

Concatenating strings is something really practical that we do a lot. std::string uses the operator +, I can do the same here, it is just a simple operator overload:

template <std::size_t OtherSize>
constexpr auto operator+(const CompileTimeString<OtherSize>& other) const noexcept
{
    // Combine both size to have the size of the array of the new string
    CompileTimeString<ArraySize + OtherSize - 1> result; // -1 because one the \0 will be overwritten
    std::ranges::copy(data, result.data); // Copy the first string
    std::ranges::copy(other.data, result.data + ArraySize - 1); // Copy the second one
    return result;
}

constexpr auto concat = "Hello, "_cts + "World!"_cts;
static_assert(concat == "Hello, World!"_cts);

You can have the full code on compiler explorer here.

Article::~Article

This string class is not totally complete, I could add a lot of features in it, but it would be only grinding features and I don't plan to make a library for it, but there's a gist here if you want to use or complete the code. With this you would have a strong base, to add member functions like a substr, or have the char type templated like the the std::basic_string and std::basic_string_view.

Sources

Generate a string at compile with C++ 17

Lena — Mon, 21 Aug 2023 05:39:05 +0000

Article:: Article

Not long ago, a colleague came to me with a problem: he needed to generate a string with a simple format, something like this ?,?,?.

It bothered him to generate it during runtime when he had all the needed information during compile time (Yes, we know that having to initialize one static string at the start of the program won't be the end of the world, it's still annoying).

To recapitulate we need to:

generate a string composed of ? and ,
during the compilation
using C++17 (yeah, I know you didn't have this information so technically I'm not recapitulating)

Let's see the solutions I proposed.

Prerequisite to make code compile time friendly

If I want my function to be able to run at compile time, it must be constexpr (or consteval with C++20). It adds some constraint to what code I can write in it, but at each new standard the rules are a bit relaxed.

Another thing is that even if a function is constexpr, it's argument can't be use for a constexpr expression, meaning that this is not possible:

constexpr void foo(int a) {
    constexpr int b = a;
}

It means the only way I have to pass the number of times we want to repeat the pattern is to use a template parameter like this.

template <int A>
constexpr void foo() {
    constexpr int b = A;
}
// You can use this function like that:
foo<15>();

First idea

I already had this function in mind that I used in one of my pet projects to concatenate arrays at compile time

template <typename T, std::size_t aSize, std::size_t bSize>
constexpr auto concat_array(const std::array<T, aSize>& a, const std::array<T, bSize>& b)
{
    std::array<T, aSize + bSize> result_array;
    std::ranges::copy(a, result_array.begin());
    std::ranges::copy(b, result_array.begin() + aSize);
    return result_array;
}

It may seem a bit barbaric but it's pretty simple, there is two std::array in input. It takes their size to create a new array then copy the content of both input into the new one.

It only works with std::array in input, but with a bit more code it could work with built-in arrays, I did it here if you are curious. For the return value sadly I have to use std::array because it's not possible to return a built-in array by value.

Anyway, the goal here is just to have a proof of concept.

After that my idea was to have a function with the number of ? as a template parameter, create an array containing a ? and then recursively add ,? at each iteration. When this is over, add a \0 to terminate the string.

It gave me this code:

namespace details {

constexpr std::array<char, 2> token_to_add = { ',' , '?'};
constexpr std::array<char, 1> endstring = { '\0'};

template <std::size_t I, std::size_t current_size>
constexpr auto concat_n_question_mark_impl(const std::array<char, current_size>& current_string) {
    if constexpr (I == 0) {
        return concat_array(current_string, endstring);
    } else {
        auto new_string = concat_array(current_string, token_to_add);
        return concat_n_question_mark_impl<I - 1>(new_string);
    }
}

}

template <std::size_t I>
constexpr auto concat_n_question_mark() {
    static_assert(I != 0, "Common don't do this");
    constexpr std::array<char, 1> start = { '?'};
    return details::concat_n_question_mark_impl<I - 1>(start);
}

Then it's possible to use the code like this:

int main(int, char **)
{
    using namespace std::literals;

    constexpr auto str = concat_n_question_mark<5>();
    static_assert(str.data() == "?,?,?,?,?"sv);
    std::cout << str.data() << std::endl;
}

The complete code is available on compiler explorer

It works but there are (at least) two issues:

It feels a bit over engineered
With the recursion the compile times are probably really bad (I say probably because I'm too lazy to mesure it)

Fortunately while playing Zelda during the lunch break, I had a much simpler idea.

Final solution

The root cause of the issues in the first solution is the recursion, but I noticed that I didn't need it. I can just calculate the size of the string I want to return from the template parameter, and I can just fill the content with a simple loop (or if you really like recursive function, a non-templated recursive function).

I got this code that you can use exactly like the previous one:

template <std::size_t N>
constexpr auto concat_n_question_mark() {
    static_assert(N != 0, "Common don't do this");
    constexpr auto size = N * 2;
    constexpr auto last_char = size - 1;
    std::array<char, size> str;
    str[0] = '?';
    str[last_char] = 0;
    for (std::size_t i = 1; i < last_char; i += 2) {
        str[i] = ',';
        str[i + 1] = '?';
    }
    return str;
}

Compiler explorer link

It's smaller and there's less template. I also find it more expressive and readable.

Article::~Article

With some templates and constexpr function we can do a lot, with C++20 we can do even more, if I have the inspiration, I will probably make an article with a similar subject to see how we can do even more.

Sources

Port a project from C++ to Rust

Lena — Mon, 14 Aug 2023 08:19:09 +0000

Article::Article

As you may know, I started learning Rust recently and I like to practice on a project to learn. Fortunately, I had the perfect pet project for that: Topflight. It's a programming language I created as a joke and the goal was to be simple but with one of the worst syntax ever. I'm pretty proud of the result.

Here's what a "Hello world" can look like:

STORE hello STRING("Hello, World!")
PRINT hello

Or how to print number from 0 to 9:

# Print range 0 to 9
STORE i INTEGER(0)
STORE one INTEGER(1)
STORE nine INTEGER(9)
STORE line_return STRING("\n")

<LoopContent>
PRINT i
PRINT line_return
ADD i one i
COMPARE_LESS_OR_EQUAL i nine keep_going
CALL_IF LoopContent keep_going
</LoopContent>

CALL LoopContent

In this article, I won't explain point by point how I rewrote it, instead I will talk about the differences I noticed between the Rust version and its C++ equivalent. My remarks will be about the code itself, but also about the developer experience.

Note that I won't go into too much detail, and I will focus my explanations on the Rust side and not on the C++ concepts that I will use for comparisons.

Setup the project

It's obvious, but in Rust it is so much simpler, you just need one command, and you are ready to go. Cargo takes care of everything, no need to write a CMakeLists.txt or something similar like Meson then setup a package manager like [Conan](https://docs.conan.io/2/introduction.html or Vcpkg.

File organization

In C++ you can organize your files in an infinite number of ways. Even if you follow some templates, there are several of them.

In Rust it is stricter, but thanks to it, it is hard to make a mess and it probably one of the reason that there is less boiler plate code to setup a project.

For example, if you create an executable, your entry file will be called main.rs, if you want to write a library it will be lib.rs. Also, the way you organize your files will have a direct impact on how your modules are defined.

In C++ you can do whatever the heck you want and let's be honest, if you are not rigorous, it can easily be a mess.

Compiler errors

C++ compiler errors are not human friendly, often it is hard to see the origin of the error. If there is a lot of templates, it looks like the compiler is throwing up at you. With some experience you quickly learn to recognize the error.

Note that it is a good thing to always look at the first error to understand the problem. Another trick you can use is to use different compilers like Gcc and Clang or even Msvc if your code is cross platform.

With Rust, errors are often clear, that's nice. Maybe it is not always the case, but atleast on my little pet projects, it was.

Reflection

We can't say that C++ nor Rust have compile time reflection, but there are way in both language to do achieve the same result (at least in some case).

In Rust you can use macros. Macros are very powerful and you can have access to an AST and also modify it to add code. For example, what I did in this project, was to create a macro that for some struct it was automatically generating a function depending on the field in the struct.

With C++, there are 3 ways to have reflection:

Use a external tool that parse your code, then generate some new code
Use a lot of macro and potentially a lot of boilerplates code
Use libraries like Boost::Pfr or my own library Aggreget with a lot of templates and that only works with aggregate (to be short: default constructor, no virtual and no private or protected attributes). That's the solution I used.

The tooling

Rust tooling works well and is trivial to install. Like you don't even need to compile you project to be checked because the VSCode extension will tell you when you have errors with the compiler error message, you can start your unit tests directly in the files, auto-completion is amazing.

C++ tooling is more fragmented and because C++ syntax is context dependent it needs to parse everything to be right, if you are using templates then it is a nightmare. Tools requires to be tightly linked to your build system and often on big project it will struggle.

Static analyzer

It is similar as the tooling; in rust you have Clippy that is setup with cargo and rustc that works well. In C++ you need to install it, sometimes pay a license, it needs a lot of setup, to know a bit about how your system is built etc.

Enums/variant

Enums in rust are awesome, it can do the same as a C++ enum and a std::variant (a type safe union) but directly integrated in the language with pattern matching for example, this means no need to have templates everywhere or use std::visit.

Exhaustive pattern matching

There is no real equivalent for match in C++, but that's one of the best feature of Rust.
It's like a switch that must be exhaustive (otherwise it does not compile) that works with any conditions, works with slices and also can do destructuring and replace stuff like std::visit in C++.

Returning values

In Rust you can also use the return keyword as in C++, but that's not the idiomatic way (except in some cases like checking a precondition of a function). Instead you just don't put a semicolon ; at the end of the expression. It allows something else that is so powerful: control flows are expressions, and they can return values without exiting the function.

It means we can do this:

fn is_laughting(my_str: &str) -> bool {
    if my_str.is_empty() {
        return false;
    }

    let b = match my_str {
        "lol" => true,
        "rofl" => true,
        _ => false,
    };
    b
}

Compiler explorer link

and you can even shorten it like the by returning the result of the match:

fn is_laughting(my_str: &str) -> bool {
    if my_str.is_empty() {
        return false;
    }

    match my_str {
        "lol" => true,
        "rofl" => true,
        _ => false,
    }
}

Compiler explorer link

There are also a lot of other features associated with it, especially if you are working with slices or tuples, someone could write a whole article just about that.

Error management

Let's look at what we commonly have to handle errors in C++:

Error codes
Exception
std::expected or other similar classes (it's variant containing either the return value or an error)

In Rust there is no exception and return codes are discouraged, instead they heavily used a Result type, it's the same thing as std::excepted but with better language support. Why? The first reason is that a Result is basically an enum for either the return value or an error and enums in Rust are really well integrated. The second reason is the error propagation operator ?.

What is it? It's the thing that allow you to do this

fn read_lines(filename: &str) -> io::Result<io::Lines<io::BufReader<File>>>
{
    let file = File::open(filename)?; // Here you can see the operator "?"
    // No need to check the result of open, it if failed, the read_lines function would return the same error
    // "file" is now already unwrapped so we can use it
    Ok(io::BufReader::new(file).lines())
}

Compiler explorer link

Basically, when you call a function that return a result, if the result contains an error it will make the current function returning the same error making you cascade return if you can't handle the error in this function. Otherwise if the result is a is a success, it unwraps it.

That's one of my favorite feature in Rust. It so clean to use, yet it does not make your code verbose.

I really advise you to use a library like anyhow or thiserror to create your errors, it makes creation and using errors really easy and practical.

You can also panic for unrecoverable errors, it's the equivalent of using std::terminate.

Iterators

In C++ an iterator does not know its end that's why we almost use another iterator as a sentinel. The algorithm of the standard library all takes a pair of iterators. With C++ 20 there is a new addition that comes into the standard: ranges. To make it short and simple, it is just a very cool wrapper around pair of iterators, so even if the usage is not that different and underneath it works the same way.

std::vector<int> ints {1, 2, 3, 4, 5};

// Manually use iterator
for (auto it = ints.cbegin(); it != ints.cend(); ++it) {
    int i = *it;
    // Do stuff
}

// Range base for loop with c++11
for (int i: ints) {
    // Do stuff
}

// Count the number of even argument
auto is_even = [](int i){ return (i % 2) == 0;};
int nb_even = std::count_if(ints.cbegin(), ints.cend(), is_even);

// Iterate of even elements with ranges
for (int i: ints | std::views::filter(is_even)) {
    // Do stuff with even elements
}

Compiler explorer link

In rust, the iterator returns an optional, if it is empty, it means we are at the end and it also advance the iterator at the same time. We can also compose operations as the C++ 20 ranges, but instead of using the pipe operator | the syntax used the same syntax as a method.

    let ints = vec![1, 2, 3, 4, 5];
    for i in ints.iter() {
        print!("{}", i);
    }
    // Output is 12345

    let even_iter = ints.as_slice().iter().filter(|i| *i % 2 == 0);
    for i in even_iter {
        print!("{}", i);
    }
    // Output is 24

Compiler explorer link

Another difference is that when you implement iterators in C++, it means that you must create a class satisfying a name requirement (and if you don't the compiler is vomiting on you with thousands of line of error) or satisfy a concept (much better). In Rust you must implement a trait.

Destructive move

Destructive move mean that when a value is moved, its resources is released and it's not possible to use it anymore.

For example, in Rust:

let first_str = String::from("Reblochon");
let second_str = first_str; // Here's the move
// Now we can't use first_str anymore

Whereas in C++:

std::string first_str = "Reblochon";
std::string second_str = std::move(first_str);
// Here we can still use first_str

For the standard library, it's in the specification that all classes should still be in a valid state after the move, but everywhere else, there is no guaranty of that so be careful and read the doc or look at the code.

Shadowing

Shadowing is a bad practice in C++ as it makes unclear which variable we are really using but there is no such problem in Rust, quite the contrary. Code like this is quite common:

let input : &str = "true";
let input : bool = match input {
    "true" => true,
    "false" => false,
    _ => panic!("OMG I can't recover from this :("),
};

Compiler explorer link

In this code, the old input is destroyed (resources releases) and now it is replaced by a new variable with the same name.

Generics

Here's a simple example with templates in C++

template <typename T>
void print(const T& t)
{
    std::cout << t.size() << std::endl;
}

I just declared a template type T and then in the body of my function I hope that t has a member function size() otherwise it won't compile.

With concepts I can make sure that T has a method size returning something I can put into a stream.

template <typename T>
concept HasSize =
    requires(const T& t, std::ostream& os)
    {
        { os << t.size() };
    };

template <HasSize T>
void print(const T& t)
{
    std::cout << t.size() << std::endl;
}

Compiler explorer link success
Compiler explorer link error

And now if it fails, I will have a message saying that it cant call the print function because the type I tried to use does not satisfy the concept HasSize.

In Rust generics has the opposite logic, you can only use the property of a type that your type have. For example, if you want to use the operator+ between 2 variable using generics you need to use a where clause saying that your type can use it.

Here's an example of using generics in Rust

use std::fmt::Debug;

fn print_in_option<T>(t: &T)
where T: Debug // Where clause to assert that it implement the Debug trait
{
    println!("{:?}", t);
}

fn main() {
    let vec = vec![1, 2, 3];
    print_in_option(&vec);
}

Compiler explorer link

But Lena how can you write about rust and not talk about the borrow checker?!?

I often read about people fighting with the compiler about lifetime, but I did not have problem with the compiler and especially not about lifetime issue. There is maybe some places where I make copies that I could have avoided, but honestly on this point it did not feel different between C++ and Rust except that Rust compiler checked that I was not doing wrong stuff. I really saw the compiler as my friend and not the enemy.

Maybe it was because the project is fairly simple and I did not need to annotate the lifetime because it could deduce it, but even when I had to write an iterator for another project which needed to do it, it seemed logical to me.

My thought about it is that most dev that complains about it (I'm talking about the concept here, not the syntax) are either doing complicated stuff where lifetime is not explicit, or they never had to think about lifetime before. Feel free to comment about it if you had trouble with the borrow checker, I'm very curious about it.

Article::~Article

That's a lot of different things, some simple things like the file organization, some stuff that are just better in Rust like the build system or the enum and some others that are just different like the iterators.

One other big difference I noticed is that when I coded in rust it is way easier to have a more functional way because the language and the standard library has better built in support for it.

There is probably a lot more differences I forgot or did not see yet, because even if I have some experience in C++, I am still a newbie in Rust, so feel free to tell me about what other differences you noticed.

Sources

I used mainly Rust by Example and the online version of The Rust Programming Language

Why I changed my mind about rust

Lena — Mon, 07 Aug 2023 07:40:02 +0000

Article::Article

Five years ago, I heard about Rust, I even tried it. I thought it was cool, but I was not transcended, and did not pursue learning this language. At the time it seemed like a lot of work for only a few advantages. It just felt like a more complicated, less mature, C++ that enforced good practices and with some nice keywords (still waiting for pattern matching in C++).

Recently, I gave the language another chance (and also after my brother kept telling me how awesome it was working with it every time I saw him haha) and this time it was a very pleasing experience and I loved it! I don't even see myself use anything else on my pet projects if I have the choice.

In this article we'll see why I changed my mind.

Evolution of the language

The first and most obvious thing is that the language evolved, the borrow checker improved and there is lifetime elision meaning that it is more readable and faster to type code.

The try! macro is gone thanks to the error propagation operator ? becoming stable, and it is really wonderful to use. It combines very well these libraries like thiserror or anyhow. Error handling is now easy.

The community has grown and is now quite big, meaning that there are a lot more library about almost everything.

There are probably a lot of things that changed, like the async stuff, but I'm not there yet and this part does not aim to be exhaustive and it is more about my experience.

My evolution

Who I was

Even if the language has changed, I changed even more. ~~Now I have programming socks and a cat eared headset.~~ Back then, I did not even graduated, I had a some knowledge about C++, a little bit of Javascript and Python, that was it. My only work experience were 2 internships of 4 months and during both internships I worked on brand new projects. The dream! No legacy code!

Who I am

I now have a bit more experience, I worked part time during my last year of school, and I graduated 4 years ago. During this time, I worked mainly in C++ and with old code. I even found once code that was older than me (I'm 26).

Don't get me wrong, old code does not mean bad, but technical debt if not handled can be a mess. Also coding practice changed, smart pointer were not widely used by then, etc.

But most importantly, I learned that even the best devs can write bad code sometimes, maybe they were tired, maybe they were pressured to do it very fast. In the end, sometimes bad code end up in production, it is painful to look at and even more painful to debug. (I'm looking at you the deadlock that happens only in release mode)

I could write a lot more about my experience, but to sum up, if good practices are not imposed, the devs won't always use them and using them won't magically solve the problems from 10 years ago.

Now I can enjoy the features of rust

I'm not here to complain about the code I worked on, overall, it were all good experiences, but what I want to say is that now I understand better the challenges in the industry. I finally see that I was not matured enough to appreciate it. I think that Rust solves a lot of them and that's why my mind changed about it.

Article::~Article

The language evolved, but most importantly, I changed too. I finally really understand why a language like rust is needed.

Now I loved it, maybe that's just the honeymoon phase, I'll keep you posted in my future articles.

Sources

Creating the very long arrow operator --->

Lena — Tue, 01 Aug 2023 10:58:20 +0000

Article::Article

In my previous article, I used the long arrow -->, but it is possible to use an arrow even bigger! Your dreams are now a reality, let's make an entrance for the very long arrow --->

How? The same way that --> is a combination of -- and >, the very long arrow ---> is the combination of -- and ->.

Reminders about operator ->

The operator -> and its overload are a bit special. For the rest of the article I will assume that you know how it works. If that's not the case, I recommend you to read:

this article if you read French
otherwise the first response of this Stack Overflow thread

Implementation

Basic

The first step is to define two classes: Base and Extension. Then overload the operator -- of Base to create and return an object of type Extension. After that, overload the operator -> of Extension to return a pointer to itself. The last step is to add a member function to Extension. It is now possible to use a very long arrow --->. Here's the code:

#include <utility>
#include <iostream>

struct Base {};

struct Extension
{
    const Extension* operator->() const
    {
        return this;
    }

    void print() const
    {
        std::cout << "long arrow" << std::endl;
    }
};

Extension operator--(const Base&, int)
{
    return Extension{};
}

int main()
{
    Base base;
    base--->print();
}

Compiler explorer link

Possible usage

The previous example is totally useless, let's see a possible usage of what you can do with it: extend a class without touching it.

Let's imagine I want to add a simple method to std::string, like something to transform it from upper case to lower case. It is possible to do this:

#include <cctype>
#include <string>
#include <iostream>
#include <algorithm>

struct StringExtension
{
    StringExtension* operator->()
    {
        return this;
    }

    void to_lower()
    {
        auto lower = [](unsigned char c) { return std::tolower(c); };
        std::transform(str.begin(), str.end(), str.begin(), lower);
    }

    std::string& str;
};

StringExtension operator--(std::string& str, int)
{
    return StringExtension{ str };
}

int main()
{
    std::string str = "HELLO";
    str--->to_lower();
    std::cout << str << std::endl;
}

Compiler explorer link

The logic is simple: create a class with all the member function we want to add with a reference to the class you want to extend. Overload the operator -> of this new class to return a pointer to itself and then overload the operator -- of the class you want to extend returning an object of the extension class. That's it.

Article::~Article

That's add another string to your bow and you can be special snowflake that uses very loooooong arrows.

I hope you enjoyed this article as much as me when I wrote it and that you will find something funny, or even useful, to do with what you learned.

Sources

https://en.cppreference.com/w/cpp/language/operators

Improving pointer to member function ->* (very bad) syntax

Lena — Mon, 17 Apr 2023 07:54:31 +0000

Article::Article

Not so long ago, I was bored, so I went to browse cppreference. Yes I do that, I know it's weird but don't mock my special interest!

Anyway, I chose to go to the Function objets without any real motive, except maybe to see if std::move_only_function, added in C++23 was already documented in the website.

It was there, I also saw std::function obviously, but before I scrolled down to see std::bind, I stumbled upon one class I didn't recall, it's not even new, it was added in C++11. I'm talking about std::mem_fn.

What is it? Well, it's pretty simple, it's a function to create a wrapper around a pointer to method, instead of writing:

#include <iostream>

struct Foo
{
    void hi()
    {
        std::cout << "Hi" << std::endl;
    }
};

int main()
{
    auto ptr_hi = &Foo::hi;

    // Use .*
    Foo foo;
    (foo.*ptr_hi)();

    // Use ->*
    auto* foo_ptr = &foo;
    (foo_ptr->*ptr_hi)();
}

Compiler explorer link

You can write:

#include <iostream>
#include <functional>

struct Foo
{
    void hi()
    {
        std::cout << "Hi" << std::endl;
    }
};

int main()
{
    auto wrapped_hi = std::mem_fn(&Foo::hi);
    Foo foo;
    wrapped_hi(foo); // Work with a value
    wrapped_hi(&foo); // Also work with a pointer
}

Compiler explorer link

So, it is not very useful except in two situations:

If you plan to use it with std::bind before C++ 14 (because there is no reason to use it since C++14) or with std::bind_front since C++20.
If you want to pass a member function to a STL algorithm. There is this very good article talking about it.

I almost forgot, it also works with smart pointers without needing to call their get() member function. So you don't have to use the operator ->*.

But wait! std::unique_ptr and std::shared_ptr don't have an overload for this operator! The syntax is so bad, so few people use it that they did not implement it. Like seriously, when is the last time you used it?

Today, we are going to fix the operator ->* by making a new syntax work!

The issue with the operator ->*

We could expect to be able to write something like that:

my_ptr->*my_ptr_on_method(my_args...);

instead of that we are forced to write this:

(my_ptr->*my_ptr_on_method)(my_args...);

The reason we have to put parenthesis around my_ptr->*my_ptr_on_method is that the operator () with the precedence over the operator ->* meaning that without the parenthesis, it would try to use the operator () of my_ptr_on_method first and it won't work.

Yes this is something very minor, but it is still frustrating and I can fix it!

Idea

The idea will be very similar with what I did in this article where I simulate the unified call syntax.

The first part is to implement a wrapper around the pointer to member function. This wrapper will have the operator () overloaded and it will be able to take the same arguments as the member function it wraps, and it will return another object that I will call the proxy.

This proxy is an object that will store the argument and the wrapped member function. It will also be able, if it is provided with an object corresponding to the wrapped member function, to make the actual call.

The last part is the overload of the operator ->* that will prove the proxy with the object.

Here what it can looks like in pseudo code:


struct Proxy
{
    Proxy(member_function, args...);

    auto execute(Foo f)
    {
        (f->*member_function)(args...);
    }

    // Stored information
    args...;
    member_function;
};

struct Wrapper
{
    Wrapper(member_function);

    Proxy operator()(args...)
    {
        return Proxy(member_function, args...);
    }

    member_function;
};

auto operator->*(Foo f, Proxy p)
{
    return p.execute(f);
}

// Usage
auto wrapper = &Foo::print;
Foo foo;
foo->*wrapper("hi");

Why not the operator `.*`?

Because it can't be overloaded.

Implementation

First, we need the wrapper and it must be constructed from a member function:

template <typename MF>
class Wrapper
{
    private:
        using C = helper::ClassType<MF>;

    public:
        Wrapper(MF member_function):
            _member_function(member_function)
        {}

    private:
        MF _member_function;
};

That's nice, but could be better.

Let's add a concept to check that the template argument is indeed a member function:

template <typename MF>
concept MemberFunction = std::is_member_function_pointer_v<MF>;

and then use it by replacing the keyword typename by MemberFunction:

template <MemberFunction MF>
class Wrapper
{
    private:
        using C = helper::ClassType<MF>;

    public:
        Wrapper(MF member_function):
            _member_function(member_function)
        {}

    private:
        MF _member_function;
};

Now we need to overload the operator () to return the proxy. It must take the arguments of the member function as parameter:

template <typename... Args>
void operator()(Args&&... args)
{

}

It must also return the proxy, but we haven't defined it yet! The simplest way to implement it is to use a lambda function, and take the arguments by capturing them in its scope:

template <typename... Args>
auto operator()(Args&&... args)
{
    // [&] means it takes everything from the current scope by reference
    // It will takes 'args...' and also 'this'
    return [&]() {  };
}

If you are following, you remember that the Proxy must be able to use the member function if provided with the right object, so let's add an argument for our lambda:

// Note that I use a template argument here to be able to use a forwarding reference
[&]<typename T>(T&& t) {  };

And now call the member function:

// std::forward<Args>(args)... may seem barbaric but it just means that we forward the argument
// With the exact same type as we got them
[&]<typename T>(T&& t) { return (t.*_member_function)(std::forward<Args>(args)...); };

The last thing to do for the wrapper is to add a concept to be able to check that the arguments provided corresponds so when it fails instead of having 3 pages of unreadable errors, we get only 20 lines.

Here's the concept:

template <typename T, typename MF, typename... Args>
concept MemberFunctionArgs = 
    requires(T&& t, MF mf, Args&&... args)
    {
        (t.*mf)(std::forward<Args>(args)...);
    };

Basically, the content is the same as the lambda and it returns true if this is possible.

With this concept added, the wrapper looks like that:

template <typename MF>
concept MemberFunction = std::is_member_function_pointer_v<MF>;

template <typename T, typename MF, typename... Args>
concept MemberFunctionArgs = 
    requires(T&& t, MF mf, Args&&... args)
    {
        (t.*mf)(std::forward<Args>(args)...);
    };


namespace helper
{

template <typename C, typename R, typename... Args>
struct DeduceClassTypeFromMemberFunction
{
    using Class = C;

    DeduceClassTypeFromMemberFunction(R (C::*)(Args...)){}
};

template <MemberFunction MF>
using ClassType = typename decltype(DeduceClassTypeFromMemberFunction(std::declval<MF>()))::Class;

} // namespace helper


template <MemberFunction MF>
class Wrapper
{
    private:
        using C = helper::ClassType<MF>;

    public:
        Wrapper(MF member_function):
            _member_function(member_function)
        {}

template <typename... Args>
auto operator()(Args&&... args) requires MemberFunctionArgs<C, MF, Args...>
{
    return [&]<typename T>(T&& t) { return (t.*_member_function)(std::forward<Args>(args)...); };
}

    private:
        MF _member_function;
};

The next step is to overload the operator ->* and it is that simple:

template <typename T, typename Proxy>
constexpr auto operator->*(T&& t, Proxy&& p)
{
    return p(std::forward<T>(t));
}

But with a concept to check that p(std::forward<T>(t)) is possible, it is better:

template <typename T, std::invocable<T> Proxy>
constexpr auto operator->*(T&& t, Proxy&& p)
{
    return p(std::forward<T>(t));
}

Why object instead of pointers?

Because it is only possible to overload operators for objects, not pointers. So we would say that we are more fixing the operator .* using the ->* in this particular case and that's why the overload of the operator ->* takes a forwarding reference T&& and not a pointer (T*).

You can find the whole code source on github here.

Article::~Article

Now it is possible to write that:

using cider::operator->*;
using cider::Wrapper;

struct Foo
{
    void bar(int a, int b)
    {
        std::cout << (a + b) << std::endl;
    };
};

auto wrapped = Wrapper(&Foo::bar);
Foo foo;
foo->*wrapped(1, 1);

I don't really expect this to be used in a real-world project, because this is an edge case usage and it adds some complexity for only a little gain. My point was to show that C++ is customizable, and it is not that hard to play with C++ syntax, it just needs some key concepts and some creativity. Also, the result is not necessarily ugly even if it may still seem complicated if you are not a bit familiar with templates.

Sources

Play with the long arrow --> and overload it

Lena — Tue, 11 Apr 2023 07:50:00 +0000

Article::Article

Have you ever heard of the person who thought they found a new operator -->. If you haven't, look at this Stack Overflow thread and read the responses (except the validated and boring one), it is pretty hilarious.

Anyway, wouldn't it be cool if we could create new operators like they can do in Swift. A part of me want to learn Swift just for this feature, even if I am not a mobile dev and that I don't even have a Mac or an iPhone.

But in C++ we can't... or can we? Well, we can't create operators from scratch, but we can combine existing operators.

In this article, we'll see how to make funny thing with a long arrow -->.

The idea

When we write: x-->0, it is just the combination of the post decrement operator and then a comparison operator and both are overloadable.

It means that that if we overload the operator --, returns something, and that something we return have the operator > overloaded, even if we technically are not overloading an operator --> it feels a bit like it.

Example

If you want to have a counter, and you want to target a specific value, you could do this:

#include <iostream>

struct Counter
{
    auto operator --(int)
    {
        struct Tmp
        {
            auto operator>(int destination)
            {
                if (c.i == destination)
                {
                    return false;
                }
                else
                {
                    c.i += (c.i > destination) ? -1 : 1;
                    return true;
                }
            }

            Counter& c;
        };

        return Tmp{*this};

    }

    int i;
};

int main(int, char **)
{
    Counter c{10};
    while (c-->15)
    {
        std::cout << c.i << std::endl;
    }
}

Compiler explorer link

Mess with your colleagues

If you want to mess with your colleagues or make a good April fool's joke, just send them this code with this link on Compiler Explorer and make them believe that the operator --> was added to the standard and that it is named to (in the same way that the operator && is called and and that you can write and instead of &&):

// This code is almost the same as the snippet above, 
#include <type_traits>
#include <concepts>
#include <iostream>

struct Counter
{
    auto operator to(int destination)
    {
        if (i == destination)
        {
            return false;
        }
        else
        {
            i += (i > destination) ? -1 : 1;
            return true;
        }
    }

    int i;
};

int main(int, char **)
{
    Counter c{10};
    while (c-->15)
    {
        std::cout << c.i << std::endl;
    }
}

I promise you it will be fun, I have done it and even if almost no one felt into the trap, they still scratched their head to understand how I made this snippet.

Implement the unified call syntax with it

If you haven't read my article about unified call syntax, I advise you to read it to understand this section.

To summarize the previous article, I used the operator ->* to be able to do that:

#include <iostream>
#include <vector>
#include <algorithm>

#include <unic/unic.hpp>

int main()
{
    using unic::operator->*;
    constexpr auto find = UNIC_GENERATE_PROXY(std::ranges::find);
    std::vector ints = { 1, 2, 3, 4, 5 };

    auto it = ints->*find(3);
    if (it != ints.cend())
        std::cout << "Found : " << *it << std::endl;
    else
        std::cout << "Not found :'(" << std::endl;
}

And wrote a little library about it.

I did almost the same for the long arrow --> and wrote another little library named long_arrow (Yes, I know, I was not very inspired for the name). With it you can do that:

#include <iostream>
#include <long_arrow/long_arrow.hpp>

LONG_ARROW_GENERATE_OPERATOR_POST_DECREMENT(std::string)

constexpr std::string concat_const(const std::string& str, char c)
{
    return str + c;
}

constexpr void concat_mut(std::string& str, char c)
{
    str += c;
}

constexpr std::string concat_rvalue(std::string&& str, char c)
{
    return std::move(str += c);
}

int main()
{
    using long_arrow::operator>;

    {
        auto concat = LONG_ARROW_GENERATE_PROXY(concat_const);
        std::string str = "Hell";
        std::string res = str-->concat('o');
        std::cout << res << std::endl; // Hello
    }
    {
        auto concat = LONG_ARROW_GENERATE_PROXY(concat_mut);
        std::string str = "Hell";
        str-->concat('o');
        std::cout << str << std::endl;
    }
    {
        auto concat = LONG_ARROW_GENERATE_PROXY(concat_rvalue);
        std::string str = std::string("Hell")-->concat('o');
        std::cout << str << std::endl;
    }
}

I won't go into the detail because I could write a little article about it, I just wanted to show an example of what it is possible to do with the long arrow.

Article::~Article

I hope you had as much fun reading this article as I got writing it!

It is another article where I showed you that you can do funny stuff and play with the C++ syntax. Also, that's not because something is possible that you should do it, I don't see the "operator -->" useful except maybe if you are creating a DSL (domain specific language) in C++.

Oh and don't forget to tell me if you have found how I made the snippet to mess with the colleagues works! I'm really proud of how hard it is to see it haha.

Sources

My first attempt to simulate the unified call syntax

Lena — Mon, 03 Apr 2023 07:52:45 +0000

Article::Article

Once upon a time, ~~a young princess, in a tower protected by a dragon~~ a dev, stuck on her gamer chair, was testing her new experimental library and wrote:

void foreach_test()
{
    std::vector<std::string> results;
    // The code where I use my library[...]
    assert(results[2].starts_with("42.42"));
}

But dammit! She got a compile error; the starts_with member function for std::string was only added really recently with C++ 20 and no one implemented it yet.

error: 'std::string' {aka 'class std::__cxx11::basic_string<char>'} has no member named 'starts_with'

In rage she wrote this function, named with such distinguished taste and making a body unnecessary long when it could have been a one liner:

bool dummy_not_error_proof_start_with_because_still_not_implemented_and_that_make_me_sad(std::string_view str, std::string_view substr)
{
    if (substr.size() > str.size())
        return false;

    for (std::size_t i = 0; i < substr.size(); ++i)
        if (str[i] != substr[i])
            return false;

    return true;
}

And then used it:

void foreach_test()
{
    std::vector<std::string> results;
    // The code where I use my library[...]
    assert(dummy_not_error_proof_start_with_because_still_not_implemented_and_that_make_me_sad(results[2], "42.42"));
}

And yes, it really happened ... to me! You can see it here in the code of my library Aggreget (It is a way less mature equivalent of MagicGet/Boost::pfr but using concepts)

It would have been nice if I could just have been able to extend simply std::string to add a simple member function, without having to create a free function, my own string class or inheriting from std::string (bad idea for several reasons including the fact that std::string destructor is not virtual).

In this article, I show you one way to fix this issue! Note the solution may be worst than the initial problem, but it will be fun! Trust me!

Unified call syntax to the rescue

Unified call syntax is something that already exist in D and in Nim, the name is kinda barbaric but it is really simple. It allows to use free function with the same syntax as member function.

Let's reuse the code from the introduction:

std::string_view str = "42.420000001";
dummy_not_error_proof_start_with_because_still_not_implemented_and_that_make_me_sad(str, "42.42");

Could be written:

std::string_view str = "42.420000001";
str.dummy_not_error_proof_start_with_because_still_not_implemented_and(42.42);

Neat, isn't it?

With something like that, if you see a class of which you can't change the source (like the standard library for example) but you feel that a member function is missing, you could just write a free function, have the same developer experience when using it as if it was a member function by keeping the clarity and with the auto completion of your favorite IDE/text editor.

But sadly we don't have that in C++, and the proposed paper are not progressing at all, some people don't see the appeal, some other are afraid of the retro compatibility and how it would work during the overload resolution. Honestly, I think it's a shame, but it takes more to stop me!

Trying to implement it myself without waiting C++35

The idea

First, I need to choose an operator to overload. It needs to:

Have a semantic that I can overload as a free function because I don't want to modify the class I want to extend.
Looks like the operators . or -> because it needs to be easy to understand that we are calling a function.

That's why I chose the operator ->*, normally it is done to use pointer to member function so even its normal meaning is close.

Note that the code examples in this section are not generic at all for simplicity sakes because template don't help for making something clear.

My ideal syntax when using it would be:

// Equivalent of starts_with(str, 'c')
str->*starts_with('c')

The difficulty to do this is that the operator () has the precedence over the operator ->*, meaning that if I decompose str->*starts_with('c') it translates to:

// First step
auto tmp = starts_with('c');
// Second step
str->*tmp;

To create my ideal syntax, I need that starts_with('c') returns something usable my the operator overload of ->*.

To do that, starts_with must be an object with the operator () overloaded, and it must return another object containing the argument, a char in this case, and the function to execute.

// My function
bool starts_with_impl(const std::string& str, char c)
{
    return (!str.empty()) && (str.front() == c);
}

// A little alias for a function returning a boolean and taking a string and char as arguments
using ptrFunc = bool (*)(const std::string&, char);

// The object we will return containing the function to execute and the char argument
struct ReturnedObject
{
    ptrFunc f;
    char c;
};


struct StartsWith
{
    // The overload of the operator ()
    ReturnedObject operator()(char c)
    {
        return ReturnedObject{ .f = starts_with_impl, .c = c };
    }
};
// Instanciate an object so we can use the operator()
StartsWith starts_with{};

Then the operator ->*, will use this object to get the function to execute and the call it with the char argument.

// The overload
constexpr auto operator->*(const std::string& str, ReturnedObject ro)
{
    // Call starts_with_impl(str, 'c')
    return ro.f(str, ro.c);
}

// Example of usage
int main()
{
    std::string str = "col";
    bool b = str->*starts_with('c');
    std::cout << b << std::endl; // Print 1
}

Compiler explorer link

It works! But that's a lot of code, and I can make it shorter with the power of lambdas.

My goal here is to replace the ReturnedObject by a lambda function. This lambda function will capture the char c and then will take a const std::string& as parameter and in its body it will call starts_with_impl. I will call this lambda the proxy functor, because it is a functor (lambda are just syntaxic sugar to create functors) and it will allow us to call starts_with_impl indirectly.

struct StartsWith
{
    // The overload of the operator ()
    auto operator()(char c)
    {
        // Return the "Proxy functor"
        return
            // Which is just a lambda capturing "c" and taking a const std::string& as paramenter
            [c](const std::string& str)
            {
                // And calling starts_with_impl when using it
                return starts_with_impl(str, c);
            };
    }
};
StartsWith starts_with{};

Then in the overload of the operator ->* we just need to call the lambda:

// The overload
constexpr auto operator->*(const std::string& str, auto&& proxy_functor)
{
    // Call starts_with_impl(str, 'c')
    return proxy_functor(str);
}

Compiler explorer link

It is possible to make it shorter, but in the next part I will focus on how to make it generic and not only work for this particular use case.

Generic implementation

Let's implement it for real and for all types! The first step is to overload the ->* as a free function. As it is generic, there is 2 template arguments:

template <typename T, typename Proxy>
constexpr void operator->*(T&& t, Proxy&& p)
{}

Now let's fill the inside of this function by calling the proxy functor:

template <typename T, typename Proxy>
constexpr void operator->*(T&& t, Proxy&& p)
{
    p(std::forward<T>(t));
}

Now let's handle the return:

template <typename T, typename Proxy>
constexpr auto operator->*(T&& t, Proxy&& p)
{
    return p(std::forward<T>(t));
}

The last step is to restrict a bit this overload and make it available only if it is possible to actually call the functor with the object as first argument by using the std::invocable concept.

template <typename T, std::invocable<T> Proxy>
constexpr auto operator->*(T&& t, Proxy&& p)
{
    return p(std::forward<T>(t));
}

That's it, there is only 3 lines of actual code needed to make it work, amazing, isn't it?

I also added a macro named UNIC_GENERATE_PROXY to help create the functors to be used with the operator ->* but there is nothing special about, but if you are curious you can see the code here.

Usage

You can create the functor yourself and then use:

using unic::operator->*;

struct Compare
{
    auto operator()(std::string_view starter) const
    {
        return [starter](const auto& str) { return str.compare(starter); };
    }
};
Compare compare{};

std::string_view str = "lola";
bool b = str->*compare("Lolo");

Or wrap an existing function:

using unic::operator->*;

constexpr auto find = UNIC_GENERATE_PROXY(std::ranges::find);
constexpr std::array a = { 1, 2, 3, 4, 5 };
auto it = a->*find(3);

Or create your own function to wrap:

using unic::operator->*;

bool dummy_not_error_proof_start_with_because_still_not_implemented_and_that_make_me_sad(std::string_view str, std::string_view substr)
{
    if (substr.size() > str.size())
        return false;

    for (std::size_t i = 0; i < substr.size(); ++i)
        if (str[i] != substr[i])
            return false;

    return true;
}

constexpr auto starts_with = UNIC_GENERATE_PROXY(dummy_not_error_proof_start_with_because_still_not_implemented_and);
constexpr std::string_view str = "lola";
constexpr bool b = str->*starts_with("lol");

A way to simulate member function for enumerations;

The operator ->* is overloadable for any enumeration. If you want to know a bit more about operator overload for enumerations, I wrote an article about it.

This means that you can use this technique to use some functions almost like member functions. For example, when using magic enum, instead of writing:

Color color = Color::RED;
auto color_name = magic_enum::enum_name(color);

You could write:

Color color = Color::RED;
auto color_name = color->*to_string();

Article::~Article

You can see the whole implementation here but keep in mind that this is just a proof of concept and not a mature library. I hope that it gave you some ideas or that you learned something.

Sources

Operator overload for enumerations

Lena — Wed, 15 Mar 2023 08:54:53 +0000

Article::Article

You probably all have seen a snippet of C++ code looking like this if you ever had a course or seen a tutorial about operator overload in C++:

#include <iostream>

struct Person
{
    std::string first_name;
    std::string last_name;
};

std::ostream& operator<<(std::ostream& os, const Person& p)
{
    return (os << "First name: " << p.first_name << " Last name: " << p.last_name);
}

int main()
{
    Person p{ "Jane", "Doe" };
    std::cout << p << std::endl;
    // The output is: "First name: Jane Last name: Doe"
}

Compiler explorer link

You can note that here the overload of the operator << is a free function and not a member function.

The point?

If it is a possible to do it with free functions, maybe it could also work for other user defined types like enumerations.

Let's see what cppreference (the reference if you have to look up things about C++, yes, I know it is in the name):
When an operator appears in an expression, and at least one of its operands has a class type or an enumeration type, then overload resolution is used to determine the user-defined function to be called

My intuition was correct and some of you probably already knew it.

Some examples

The classic

You want to log your enum but you don't use a library like Magic Enum (try it, it is awesome) then you write this type of stuff:

#include <iostream>
#include <utility>

enum class ToggleStatus
{
    ON,
    OFF
};

std::ostream& operator<<(std::ostream& os, ToggleStatus status)
{
    if (status == ToggleStatus::ON)
        os << "ON";
    else
        os << "FALSE";
    return os;
}

Compiler explorer link

The chaotic evil

Are you tired of playing a horny bard in D&D? Let's play a chaotic evil developer and watch the world burn!

enum class Int
{
    ONE, // 0
    TWO, // 1
    THREE // 2
};

constexpr Int operator+(const Int& a, const Int& b) {
    if (a == Int::ONE)
        return Int::ONE;
    else if (b == Int::ONE)
        return Int::TWO; 
    else if (a == b)
        return Int::THREE; 
    else
        return Int::ONE; 
}

static_assert((Int::ONE + Int::TWO) == Int::ONE);
static_assert((Int::TWO + Int::ONE) == Int::TWO);
static_assert((Int::TWO + Int::TWO) == Int::THREE);
static_assert((Int::THREE + Int::THREE) == Int::THREE);
static_assert((Int::TWO + Int::THREE) == Int::ONE);

Compiler explorer link

The weird but funny

If you are still not using Magic Enum but you want to iterate over the values of an enumeration, here's an example of how to do it by using your enumeration like an iterator:

#include <iostream>
#include <utility>

enum class UserStoryStatus
{
    TO_REFINE, // 0
    REFINED, // 1
    TO_DO, // 2
    WORK_IN_PROGRESS, // 3
    TO_VALIDE, // 4
    DONE, // 5
    // Using out of range value as a sentinel would be UB
    // Therefore, I need to define it
    END 
};

// Pre increment operator
// Note: this function could be a one liner with 0 copy, but I detailed it for learning purpose
UserStoryStatus& operator++(UserStoryStatus& uss)
{
    // Use std::to_underlying to get the actual integer representation
    auto i = std::to_underlying(uss);
    // All values are continuous because we did not specify them
    // It means we can just increase by one the integer representation of the enum
    i += 1;
    // The end should be checked using the UserStoryStatus::END sentinel value
    // So I wont't check we are still in the right range
    // Because this should be done with the end function
    // But feel free to throw and exception if you want to do that check
    // Or keep returning UserStoryStatus::END
    uss = static_cast<UserStoryStatus>(i);
    return uss;
}

// Post increment operator
UserStoryStatus operator++(UserStoryStatus& uss, int)
{
    return ++uss;
}

// Dummy dereference operator, because iterator are supposed to be dereferenceable
UserStoryStatus operator*(UserStoryStatus uss)
{
    return uss;
}

// Dummy type to use a the range
struct UserStoryStatusRange{};

// Begin and end function so it act as a range
UserStoryStatus begin(UserStoryStatusRange status)
{
    return UserStoryStatus::TO_REFINE;
}

UserStoryStatus end(UserStoryStatusRange status)
{
    return UserStoryStatus::END;
}


int main(int, char **)
{
    UserStoryStatusRange range;
    for (auto s: range)
        std::cout << std::to_underlying(s) << std::endl;
}

Compiler explorer link

Article::~Article

I hope you enjoyed this article or even better you learned something. With a little bit of (bad)luck it created a vocation for someone to be a chaotic evil developer.

Sources

https://en.cppreference.com/w/cpp/language/operators

The specificity of the overload of the post increment and decrement operators

Lena — Fri, 10 Mar 2023 13:09:09 +0000

Article::Article

In C++ as in many language, you can use the ++ or -- operators to increment or decrement a variable representing a number. It can be placed before the variable, in the case this is the prefix operator also called postfix operators, or after, then it is a postix operator also called suffix operators.

int i = 0;
// Pre increment
++i;
// Post increment
i++
// Pre decrement
--i;
// Post decrement
i--

The difference between those operators

The first, and well known, difference, aside from the fact that one is placed before and the other is placed after, if that the prefix operators will be executed before the rest of the expression and the postfix operators will be executed after.

Here's a little example to see it better:

#include <iostream>

int main()
{
    int i = 0;
    std::cout << ++i << std::endl; // Print 1
    std::cout << i++ << std::endl; // Also print 1
    std::cout << i << std::endl; // Print 2
}

Compiler explorer link

Note: don't do this because it is undefined behavior:

// Examples taken from cppreference
// https://en.cppreference.com/w/cpp/language/eval_order
i = ++i + i++;     // undefined behavior
i = i++ + 1;       // undefined behavior
i = ++i + 1;       // undefined behavior
++ ++i;            // undefined behavior
f(++i, ++i);       // undefined behavior
f(i = -1, i = -1); // undefined behavior

f(i, i++); // undefined behavior
a[i] = i++; // undefined bevahior

The second one is that for the built-in versions of theses operators, the pre increment operator directly increment the variable and returns a reference, but for the post decrement operator, it makes a copy, increment it and then return it. Usually when overloading an operator, people tends to keep it the same way and that's why some people will argue that the pre increment operator is more optimized. But when overloading it, it is totally possible to take a reference as parameter instead of a copy, and also if the type is trivial, all compilers optimize it with only the -O1 flag. You can see it on compiler explorer: here the unoptimized version and here the optimized version.

Overload

The pre increment operators

That's the easiest one, here's how to do it:

struct Int
{
    // Overload it as a member function
    Int& operator++()
    {
        ++i;
        return *this;
    }

    int i;
};

// Overload it as free Function
Int& operator--(Int& fi)
{
    --fi.i;
    return fi;
}

Compiler Explorer link

The post increment operators

It is done the example same way, but you add an integer argument. This argument is only used to differentiate both overloads, its value will always be 0.

struct Int
{
    // Overload it as a member function
    Int operator++(int)
    {
        ++i;
        return *this;
    }

    int i;
};

// Overload it as free Function
Int operator--(Int fi, int)
{
    --fi.i;
    return fi;
}

Compiler Explorer link

The chaotic usage

When I said just above the value of the integer for the postfix operator will always be 0, that's not exactly true, if you use the operator overload like a regular function like this, it can have a value:

#include <iostream>

struct Int
{
    // Overload it as a member function
    Int operator++(int n)
    {
        i += --n;
        return *this;
    }

    int i;
};

// Overload it as free Function
Int operator--(Int fi, int n)
{
    fi.i -= ++n;
    return fi;
}

int main()
{
    Int i{5};
    auto other_i = i.operator++(5);

    std::cout << other_i.i << std::endl; // 5 + 5 -1 = 9
}

Compiler Explorer link

I hope that nobody is really doing this.

Article::~Article

The increment and decrement operators have no secrets for you anymore. You know what they do, how to use them and how to overload them.

Sources

Create a VSCode Extension for my esoteric programming language in C++ with Web Assembly

Lena — Mon, 06 Mar 2023 20:50:15 +0000

Article::Article

I cleaned up the project, I compiled it to Wasm using emscripten, that's nice; but something is missing : tools to develop in my langage. Who would like to use something without a highlighter and auto-completion? Even the brainfuck langage has some VSCode extensions to highlight the code!

But this issue is already something of the past now, because I created a VScode extension providing auto-completion, semantic highlighting.

Why semantic highlighting instead of syntax highlighting?

VScode offers two APIs to provide highlighting, syntactic and semantic. With the syntax one, I need to provide a TextMate grammars and it looks like the opposite of fun. With the semantic one, I need to provide a list of tokens with their meaning. It may seem more complicated, but I already have a complete parser, so it will be a piece of cake.

The informations VSCode needs

For the highlighting

As I said, for the semantic highlighting VSCode needs a list of tokens. Each token must have the following information:

the line where it starts
the column where it ends
the length
the type

VSCode already provides some types by default:

variable: for PainPerdu references
function: for PainPerdu labels
comment: (do it really need to say it?)
string: for PainPerdu file inclusion (because it is between two double quotes)
number: for integers (there is no decimal in PainPerdu)
operator: for symbols

For the auto-completion

VSCode will only need the list of labels and references. That's it.

The C++ implementation

Define the token

The token is defined with the following structure:

struct Token
{
    enum class Type
    {
        REFERENCE,
        LABEL,
        COMMENT,
        STRING,
        NUMBER,
        OPERATOR
    };

    bool operator==(const Token&) const = default;

    Type type;
    std::size_t line;
    std::size_t start_column;
    std::size_t length;
};

I won't linger on this code; it is a very simple structure containing the information stated in the previous section.

Get the tokens

I just need to get the tokens, therefore I won't need to build a whole parse tree, I will only use the callback system that I already used to handle the errors in this previous article to just fill a vector of token.

For the Move right operator > it would look like this:

template <>
struct ToTokenAction<operators::MoveRight>
{
    template <typename ParseInput>
    static void apply(const ParseInput& in, std::vector<Token>& tokens)
    {
        tokens.push_back(
            Token
            {
                .type = Token::Type::OPERATOR,
                .line = in.position().line,
                .start_column = in.position().column,
                .length = in.size()
            });
    }
};

But because I'm lazy and I don't want to write almost the same code twenty times, I will use the following macro:

#define DefineAction(Match, TokenType)                                                \
    template <>                                                                       \
    struct ToTokenAction<Match>                                                   \
    {                                                                                 \
        template <typename ParseInput>                                                \
        static void apply(const ParseInput& in, std::vector<Token>& tokens)           \
        {                                                                             \
            tokens.push_back(                                                         \
                Token                                                                 \
                {                                                                     \
                    .type = TokenType,                                                \
                    .line = in.position().line,                                       \
                    .start_column = in.position().column,                             \
                    .length = in.size()                                               \
                });                                                                   \
        }                                                                             \
    };
// You can then use it like this:
DefineAction(operators::MoveRight, Token::Type::OPERATOR)

There is still one issue, I need to differentiate the labels and the references, because for now they are only considered as identifiers.

I know that a label must be preceded by some specific operators, same for the references. Each time the parser see one of these operators, it knows that the next identifier will be either a reference or a label.

I can update the macro to reflect this and then handle correctly the labels and references:

// Little helper struct to centralize some information
struct ToTokenState
{
    enum NextIdentifierType
    {
        NONE, // If there is an identifier with NONE, this is a bug
        LABEL,
        REFERENCE
    };

    std::vector<Token> tokens;
    NextIdentifierType next_identifier_type;
};

// A little helper function
inline Token::Type get_token_type(ToTokenState::NextIdentifierType next_identifier_type)
{
    switch (next_identifier_type)
    {
        case ToTokenState::NextIdentifierType::LABEL :
        {
            return Token::Type::LABEL;
        }
        case ToTokenState::NextIdentifierType::REFERENCE :
        {
            return Token::Type::REFERENCE;
        }
        default:
        {
            throw std::runtime_error("Someone did a shitty code trololol");
        }
    }
}

// Empty default action
template<typename Rule>
struct ToTokenAction {};

// The macro has one more argument
#define DefineAction(Match, TokenType, IdentifierType)                                \
    template <>                                                                       \
    struct ToTokenAction<Match>                                                       \
    {                                                                                 \
        template <typename ParseInput>                                                \
        static void apply(const ParseInput& in, ToTokenState& state)                  \
        {                                                                             \
            state.tokens.push_back(                                                   \
                Token                                                                 \
                {                                                                     \
                    .type = TokenType,                                                \
                    .line = in.position().line,                                       \
                    .start_column = in.position().column,                             \
                    .length = in.size()                                               \
                });                                                                   \
            state.next_identifier_type = IdentifierType; // Update accoringly       \
        }                                                                             \
    };

// Now use the macro
DefineAction(operators::DefineLabel, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::MoveRight, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::MoveLeft, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::Increment, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::Decrement, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(ResetCase, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)
DefineAction(operators::DefineReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::UndefineReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::MoveToReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::GoToLabel, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::Rewind, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::IfCurrentValueDifferent, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::IfCursorIsAtReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::IfReferenceExists, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(GetChar, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)
DefineAction(PutChar, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)

DefineAction(Comment, Token::Type::COMMENT, ToTokenState::NextIdentifierType::NONE)
DefineAction(ReadFile, Token::Type::STRING, ToTokenState::NextIdentifierType::NONE)
DefineAction(Integer, Token::Type::NUMBER, ToTokenState::NextIdentifierType::NONE)

// Handle the identifier
template <>                                                                       
struct ToTokenAction<Identifier>
{
    template <typename ParseInput>
    static void apply(const ParseInput& in, ToTokenState& state)
    {
        if (state.next_identifier_type != ToTokenState::NextIdentifierType::NONE)
        {
            state.tokens.push_back(
                Token
                {
                    .type = get_token_type(state.next_identifier_type),
                    .line = in.position().line,
                    .start_column = in.position().column,
                    .length = in.size()
                });
            state.next_identifier_type = ToTokenState::NextIdentifierType::NONE;
        }
    }
};

// Because I'm not that dirty, I under my macro since I don't need it anymore
#undef DefineAction

And finally, I the code to use this machinery and get all the tokens:

std::vector<Token> Parser::get_tokens(std::string_view input)
{
    ToTokenState token_state;

    pegtl::memory_input mem_input(input.data(), input.size(), "");
    pegtl::parse<Grammar, ToTokenAction>(mem_input, token_state);

    return token_state.tokens;
}

Get the labels and references

Now that I am able to get the tokens, this part will be very easy.

The only way to define a label is like this: :my_label and the only way to define a reference is like this : #my_reference. I will detect these patterns to get the desired list.

There is two solutions to do it:

Make a parse tree and with a custom Selector to get only label/reference with identifiers
Use the same technique as the tokens

Both are really simple but I chose the second one because the code is really compact. Here's the code for the reference, but for the labels, the code is almost the same:

template<typename Rule>
struct GetDefinedReferencesAction {};

template<>
struct GetDefinedReferencesAction<operators::DefineReference>
{
    // This is called when a # is encountered
    template <typename ParseInput>
    static void apply(const ParseInput&, std::vector<std::string>&, bool& is_next_reference)
    {
        is_next_reference = true;
    }
};

template<>
struct GetDefinedReferencesAction<Identifier>
{
    // This is called for all identifiers
    template <typename ParseInput>
    static void apply(const ParseInput& in, std::vector<std::string>& references, bool& is_next_reference)
    {
        // If the previous caracter was a #
        if (is_next_reference)
        {
            // Add it in the list
            references.push_back(in.string());
            is_next_reference = false;
        }
    }
};

That's it!

Create the bindings

Now to create the bindings I need a new CMake target, it is the same way as the web interpreter with the difference that the nodejs shipped with VSCode does not use the flag --experimental-wasm-eh. This mean I won't use the flag -fwasm-exceptions in the CMakeLists.txt:

# Create the target
add_executable(ExtentionBindings ExtentionBindings.cpp)
# Basic options
target_compile_options(ExtentionBindings PRIVATE -Wextra -Wall -Wsign-conversion -Wfloat-equal -pedantic -Wredundant-decls -Wshadow -Wpointer-arith -O3)
# Don't forget to link with --bind
target_link_options(ExtentionBindings PRIVATE --bind)
# Obviously I need my PainPerdu library
target_link_libraries(ExtentionBindings PRIVATE PainPerdu)
# Put the output files directly in the good directory
set_target_properties(ExtentionBindings PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_CURRENT_LIST_DIR}/../../vscode_extention/painperdu-bakery/generated")

Same for the cpp files, it will be very similar as what I did in my previous article about emscripten.

Add some getter functions to the Token structure:

struct Token
{
    enum class Type
    {
        REFERENCE,
        LABEL,
        COMMENT,
        STRING,
        NUMBER,
        OPERATOR
    };
    using TypeIntType = std::underlying_type_t<Type>;

    bool operator==(const Token&) const = default;

    Type type;
    std::size_t line;
    std::size_t start_column;
    std::size_t length;

    // Getter for the bindings
    Type get_type() const { return type; }
    // Little spoiler, I added this function because VSCode will only need an index :)
    TypeIntType get_type_index() const { return static_cast<TypeIntType>(type); }
    std::size_t get_line() const { return line; }
    std::size_t get_start_column() const { return start_column; }
    std::size_t get_length() const { return length; }

};

Then I can really create the bindings

#include <vector>
#include <string>

// Do not forget this inscmude
#include <emscripten/bind.h>

#include <PainPerdu/PainPerdu.hpp>

using namespace emscripten;

// Little helper functions
std::vector<PainPerdu::parser::Token> get_tokens(const std::string& input)
{
    return PainPerdu::Parser().get_tokens(input);
}

std::vector<std::string> get_defined_labels(const std::string& input)
{
    return PainPerdu::Parser().get_defined_labels(input);
}

std::vector<std::string> get_defined_references(const std::string& input)
{
    return PainPerdu::Parser().get_defined_references(input);
}

// The module
EMSCRIPTEN_BINDINGS(PainPerduParserModule) {
    // I need to declare the vectors
    register_vector<std::string>("VectorString");

    // And the Token class with all its getters
    class_<PainPerdu::parser::Token>("PainPerduToken")
        .function("get_type_index", &PainPerdu::parser::Token::get_type_index)
        .function("get_line", &PainPerdu::parser::Token::get_line)
        .function("get_start_column", &PainPerdu::parser::Token::get_start_column)
        .function("get_length", &PainPerdu::parser::Token::get_length);

    // Then declare what a vector of Token is
    register_vector<PainPerdu::parser::Token>("VectorToken");

    // Last step, declare functions I will use in javascript
    function("get_tokens", &get_tokens);
    function("get_defined_labels", &get_defined_labels);
    function("get_defined_references", &get_defined_references);

}

The javascript implementation

The setup

I just followed the vscode tutorial. To be honest if I go any deeper I would either be paraphrasing or be less clear than the tutorial itself. It is very clear and well written.

Basically, I just did:

npm install -g yo generator-code
yo code

And it generated the structure of the extension, then I created a folder to place my wasm code with its js glue code (generated from C++). In the rest of this article, my whole code will be in a file named extension.js and at the beginning it looks like this:

// Include vscode stuff
const vscode = require('vscode');

// Use the bindings to my PainPerdu library
const bindings = require('./generated/ExtentionBindings')


/**
 * @param {vscode.ExtensionContext} context
 */
function activate(context) {

}

function deactivate() { }

module.exports = {
    activate,
    deactivate
}

The semantic highlighting

As I said earlier, we need to provide a list of tokens to VSCode and it provides utility classes to build it: SemanticTokensBuilder and SemanticTokensLegend.

The SemanticTokensLegend is just a little class describing the types of token that will be provided. There is a lot of predefined type of token like variable or function. You can see the list of predefined type of token or how to create your own in the documentation. The SemanticTokensLegend is created from a list of string and I will use this legend for now on:

const tokenTypeStrings = [
    'variable',
    'function',
    'comment',
    'string',
    'number',
    'operator'
];
const legend = new vscode.SemanticTokensLegend(tokenTypeStrings);

The SemanticTokensBuilder can then be used with the legend previously created:

const builder = new vscode.SemanticTokensBuilder(legend);
builder.push(start_line, start_column, length, index_of_the_type);
return builder.build();

Note that VSCode considers than the line and columns of a document start at 0. Also for the type it uses an index, the index of variable is 0, function is 1, etc.

Now I need a DocumentSemanticTokensProvider, when using TypeScript it is possible to implement the interface vscode.DocumentSemanticTokensProvider. It must have a method named the provideDocumentSemanticTokens that returns the list of tokens. I used it like this:

class DocumentSemanticTokensProvider {
    async provideDocumentSemanticTokens(document) {
        // Use the C++ code to create the list of tokens
        const allTokens = bindings.get_tokens(document.getText());

        // Create the builder using the lgend
        const builder = new vscode.SemanticTokensBuilder(legend);

        // Convert my list of tokens into one that VSCode understand
        for (var i = 0; i < allTokens.size(); ++i) {
            let token = allTokens.get(i);

            // In my C++ code, column and lines start at 1, but with VSCode it starts at 0
            builder.push(token.get_line() - 1, token.get_start_column() - 1, token.get_length(), token.get_type_index());
        }

        return builder.build();
    }
}

And the last step is to make VSCode use the DocumentSemanticTokensProvider:

function activate(context) {
    // I want only to use it for my language
    const selector = { language: 'painperdu' };
    // Some boiler plate code 

context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider(selector, new DocumentSemanticTokensProvider(), legend));
}

Here's what it can look like with some code from my Brainfuck interpreter written in PainPerdu:

The auto-completion

Auto-completion almost work the same, except that instead of a list of tokens I need to return a list of element with their type: in this case either a variable (for a PainPerdu reference) or a function (for a PainPerdu label).

I will create 2 GoCompletionItemProvider, one giving the list of reference, the other one the list of labels:

class GoCompletionItemProviderLabels {
    provideCompletionItems(document) {
        const labels = bindings.get_defined_labels(document.getText());

        let result = [];
        for (var i = 0; i < labels.size(); ++i) {
            let label = labels.get(i);
            let completionItem = new vscode.CompletionItem();
            completionItem.label = label;
            completionItem.kind = vscode.CompletionItemKind.Function;
            result.push(completionItem);
        }
        return result;
    }
}

class GoCompletionItemProviderReference {
    provideCompletionItems(document) {
        const refs = bindings.get_defined_references(document.getText());

        let result = [];
        for (var i = 0; i < refs.size(); ++i) {
            let ref = refs.get(i);
            let completionItem = new vscode.CompletionItem();
            completionItem.label = ref;
            completionItem.kind = vscode.CompletionItemKind.Variable;
            result.push(completionItem);
        }
        return result;
    }
}

And now I just need VSCode to use them:

    const selector = { language: 'painperdu' };
    context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider(selector, new DocumentSemanticTokensProvider(), legend));
    // These characters can only be followed by a label

    context.subscriptions.push(vscode.languages.registerCompletionItemProvider(selector, new GoCompletionItemProviderLabels(), '.', '*', '&'));
    // These characters can only be followed by a reference
    context.subscriptions.push(vscode.languages.registerCompletionItemProvider(selector, new GoCompletionItemProviderReference(), '#', '@', '$', '>', '<', '+', '-', '?', '!'));
}

And voilà! It was that simple!

Article::~Article

It is my very first VSCode extension and even if it may seem dumb, I'm proud of it. Don't hesitate to point some errors or some things I could have done better, I'm not very comfortable using Javascript and I don't know well the VSCode API.

The extension could be improved, but I doubt that anybody will really use this extension. The whole project around this language is just an excuse to try some tools and have some fun.

The extension is named PainPerdu Bakery and you can find it on the vscode marketplace.

Sources

Create Python bindings for my C++ code with PyBind11

Lena — Mon, 13 Feb 2023 11:08:57 +0000

Article::Article

Back in the day, when I was still a student, during the end of one of my internships, instead of writing my report I wrote a generator of spelling mistake for French named YololTranslator. You probably have no idea what it looks like, if you speak French, I invite you to visit the online version. Otherwise, imagine that at first it would write your instead of you're and stuff like that. (But it became way more powerfull)

It was a CLI tool written in C++, then I wrote a bot discord using it, after that, an API and finally a friend created the front for the website. The spelling mistake generator was still in C++ but everything else was in JavaScript with Nodejs. But using C++ code with Nodejs was a pain in the ass, especially when I had a different version of node between my computer and my Raspberry Pi.

Each time I improved the spelling mistake engine, deploying it was not fun at all. One day I got tired of it and I decided to re-write the Discord bot and the backend in Python. I liked Python (and still do), there are some good libraries to create a Discord bot and a backend (FastAPI is fantastic) and also using C functions in Python is trivial. At the same time I created a docker image for the bot and the website, deployment is way easier now but that's another story I may talk about one day.

This article is about how I used my C++ functions in my Python scripts.

Initial implementation with ctypes

Overview

C nowadays is the lingua franca for programming. You can, in almost any language, use C functions from a shared library easily. Python is no exception, there is a dedicated module for that: ctypes. That's why I chose it in my initial solution.

Create the C interface

To create the C interface, I needed to create functions with C linkage. It's really simple to do, just create functions with declarations containing only C code and put them inside an extern "C" statement. Then you can write whatever you want in the implementation. The limitation will only be in the argument and return type. It means no std::string, no classes, only good old const char* and struct without any method.

My higher-level C++ class looked like this:

class Translator
{
    public:
        Translator(std::string_view phonems_list_filename, std::string_view words_phonem_filename, std::string_view word_dict_filename);

        std::string operator()(std::string_view sentence_to_translate) const;
        std::string translate_word(std::string_view word_to_translate) const;

    private:
        WordTranslator _word_translator;
        WordToPhonems _word_to_phonem;
        PhonemListToString _phonem_list_to_string;
};

It's pretty straighforward, you initialize it, then you can generate error for an entire sentence or for just one word and you also have the default constructor automatically generated.

The C interface will look a lot like this:

extern "C"
{

typedef struct YololTranslationS
{
    int size;
    char* translation;
} YololTranslation;

void yolol_init(const char* phonems_to_chars, const char* word_to_phonem, const char* word_to_word);

YololTranslation yolol_translate(const char* str, int size);
void yolol_free(YololTranslation translation);

void yolol_destroy();

} // extern "C"

We have:

YololTranslation containing the result of a translation, with a char* dynamically allocated, and a size even if the string will be null terminated I think it is good practice.
yolol_init acting like the constructor, note that for simplicity's sake, you can use only one Translator at a time. It was (and still is) the only use case.
yolol_translate to generate the errors.
yolol_free because the char* in the YololTranslation is dynamically allocated.
yolol_destroy acting like the destructor

And then in the implementation I just use my Translator class:

static std::optional<Translator> translator;

void yolol_init(const char* phonems_to_chars, const char* word_to_phonem, const char* word_to_word)
{
    translator.emplace(phonems_to_chars, word_to_phonem, word_to_word);
}

YololTranslation yolol_translate(const char* str, int size)
{
    auto translation = (*translator)(std::string_view(str, static_cast<std::size_t>(size)));
    YololTranslation result;
    result.size = static_cast<int>(translation.size());
    #if defined(_WIN32)
        result.translation = _strdup(translation.data());
    #else
        result.translation = strdup(translation.data());
    #endif
    return result;
}

void yolol_free(YololTranslation translation)
{
    free(reinterpret_cast<void*>(translation.translation));
}

void yolol_destroy()
{
    translator.reset();
}

I made sure in my CMakeLists.txt that I created a shared library and not a static one:

add_library(YololTranslator SHARED ${SRCS})

The last step is to make sure the symbols of these functions are exported, with gcc and clang everything is exported by default, and with MSVC (Microsoft compiler) I added this line to make sure that it was:

set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)

I know it's ugly, but it's a little pet project, it's fast to do and it works, I did not see any harm.

Use the C interface from Python

As I said, Python has a module named ctypes to use easily, but verbosely C functions from a shared library.

To load a shared library you just need its path:

lib = ctypes.CDLL(lib_path)

To define a structure you just need to create a class inheriting from ctypes.Structure and declare the attributes:

class YololTranslation(ctypes.Structure):
    _fields_ = [
        ('size', ctypes.c_int),
        ('translation', ctypes.c_char_p)
    ]

And to declare and use a function can just do that:

# "lib" was created using ctypes.CDLL as showed previously
# ctypes.c_char_p correspond to a const char* and c_int an int
lib.yolol_translate.argtypes = [ctypes.c_char_p, ctypes.c_int]
# "YololTranslation" was the declared structure in the previous example
lib.yolol_translate.restype = YololTranslation

# A char* is just an array of bytes, so you need to encode your string to get one
buff = "Salut, je veux des fautes!".encode()
# Call the C function
translation_result = self.lib.yolol_translate(buff, len(buff))
# Use the result
str_with_spelling_mistake = translation_result.translation.decode()
# Possible content of str_with_spelling_mistake :
# "salu  je veu dè phaut!!"

You can see the file with the complete code here.

Why rewrite it?

I just wanted to try PyBind11 for something else than a "hello world" project. That's it, it wasn't hard to convince myself.

Use PyBind11

Installation

At first, I wanted to install it using Vcpkg but I ended up using CPM.cmake instead. Why? Because with Vcpkg on some platform it is messing up the config and choose a version of Python used in Vcpkg instead of the one I wanted to use. It is not that hard to fix, but why bother when I can install the library in one line with CPM.cmake

CPMAddPackage("gh:pybind/pybind11#v2.10.3")

If you want to learn more about CPM.cmake I advise you to read this article.

And if you really want to install it with Vcpkg but you have the same problem as me: no error but the module was not created (that was so much fun to understand -_-) because apparently it was choosing the wrong python, this line may help you :

find_package(Python COMPONENTS Interpreter Development)

Create a module with CMake

This part is really simple. You have this CMake function pybind11_add_module that acts like add_library or add_executable.
In my project I did this:

# You can use as many source file as your want
pybind11_add_module(yolol source_file_1)
# You can link it with a library as you would do with a regular target
target_link_libraries(yolol PUBLIC YololTranslator)

Create the bindings in the C++ code

The most basic example from the documentation is this:

#include <pybind11/pybind11.h>

int add(int i, int j) {
    return i + j;
}

PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring

    m.def("add", &add, "A function that adds two numbers");
}

It creates a module named example and it this module you have a function named add. The prototype of the function is deduced so you just to pass a pointer to this function, nothing else.

That's cool, but I don't want to export a function, I want to export a class with a constructor and a member function. That's almost as easy:

#include <Translator.hpp>

#include <pybind11/pybind11.h>

// Because I'm lazy and there is no risk of name collision
namespace py = pybind11;

PYBIND11_MODULE(yolol, m) {
    m.doc() = "Yolol bindings";

    py::class_<Translator>(m, "Translator")
        .def(py::init<std::string_view, std::string_view, std::string_view>())
        .def("translate", &Translator::operator());
}

I created a class using py::class_ with its name, then I defined the constructor, in Python it will correspond to the init method. For this constructor I need to specify the the argument.
The I added a method named translate but using the operator(), you may find it weird, but it works well because an operator overload is a regular function.

I can still improve this a little. The constructor has three arguments with the same type, it can create some confusion and I might mix up the argument. In Python there are named arguments, let's use them:

#include <Translator.hpp>

#include <pybind11/pybind11.h>

namespace py = pybind11;

PYBIND11_MODULE(yolol, m) {
    m.doc() = "Yolol bindings";

    py::class_<Translator>(m, "Translator")
        .def(py::init<std::string_view, std::string_view, std::string_view>(),
            py::arg("word_to_phonem_file"),
            py::arg("phonem_to_chars_file"),
            py::arg("word_to_word_file"))
        .def("translate", &Translator::operator());
}

I just needed to add some py::arg with the name of the argument, that's all. Easy, isn't it?

Use the bindings in the Python code

Now that the bindings are created, I want to use them. For that, I just need to import the package and then I can use my Translator class as a normal Python class:

from yolol import Translator
# If the .so/.pyd is in another directory you can write the path
# But instead of / use a .
# from dir.subdir.yolol import Translator

# Create the Translator
translator = Translator(word_to_phonem_file='a_path.txt', phonem_to_chars_file='another/path.json', word_to_word_file='foo/bar/beer.json')

# Use it
str = translator.translate("Le reblochon c'est trop bon.")
print(str) # Possible output: "l reblochon sê trrau bon"

Some notes that may help you

-fPIC missing

If you have an error like this while building:

warning: relocation against `_ZTVN8nlohmann16json_abi_v3_11_26detail10type_errorE' in read-only section `.text.unlikely'
#11 25.25 /usr/bin/ld: src/libYololTranslator.a(WordTranslator.cpp.o): relocation R_X86_64_PC32 against symbol relocation R_X86_64_PC32 against symbol `_ZZNSt8__detail18__to_chars_10_implImEEvPcjT_E8__digits' can not be used when making a shared object; recompile with -fPIC

That's because as the error says the flag -fPIC is missing and the clean way to add it in CMake is:

# For the whole project
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
# For just a target
set_target_properties(my_target PROPERTIES POSITION_INDEPENDENT_CODE TRUE)

Missing Python header

If you have an error like this:

CMake Error in CMakeLists.txt:
Imported target "pybind11::module" includes non-existent path
  "/usr/include/python3.10"
in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
* The path was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and references files it does not
provide.

It means that the Python header are not installed. On Ubuntu (and probably all the OS based on Debian) you can fixe this by installing them with apt.

RUN apt install python3-dev

Article::~Article

We have seen two ways to call my C++ code from Python: ctypes module and PyBind11. In each case the example was quite trivial and, especially for PyBind11, we did not use these tools to their maximum potential. There are a lot of other features we did not see. You can consider this article more like an introduction to the wonderful world of bindings between C++ and Python.

Note that PyBind11 is not the only library to create bindings between C++ and Python, there is also:

Boost.Python which is kinda the predecessor of PyBind11
Nanobind made by the creator of PyBind11, it has a similar interface, but it takes leverage of C++17 and it aims to have more efficient bindings in space and speed.

DEV Community: Lena

Improve the manipulation of a string at compile time with C++20

Article::Article

The layout

Construct it easily

Really basic iterator stuff

Conversion to std::string_view

Add some basic member functions

Concatenate compile time string with operator+

Article::~Article

Sources

Generate a string at compile with C++ 17

Article:: Article

Prerequisite to make code compile time friendly

First idea

Final solution

Article::~Article

Sources

Port a project from C++ to Rust

Article::Article

Setup the project

File organization

Compiler errors

Reflection

The tooling

Static analyzer

Enums/variant

Exhaustive pattern matching

Returning values

Error management

Iterators

Destructive move

Shadowing

Generics

But Lena how can you write about rust and not talk about the borrow checker?!?

Article::~Article

Sources

Why I changed my mind about rust

Article::Article

Evolution of the language

My evolution

Who I was

Who I am

Now I can enjoy the features of rust

Article::~Article

Sources

Creating the very long arrow operator --->

Article::Article

Reminders about operator ->

Implementation

Basic

Possible usage

Article::~Article

Sources

Improving pointer to member function ->* (very bad) syntax

Article::Article

The issue with the operator ->*

Idea

Why not the operator .*?

Implementation

Why object instead of pointers?

Article::~Article

Sources

Play with the long arrow --> and overload it

Article::Article

The idea

Example

Mess with your colleagues

Implement the unified call syntax with it

Article::~Article

Sources

My first attempt to simulate the unified call syntax

Article::Article

Unified call syntax to the rescue

Trying to implement it myself without waiting C++35

The idea

Generic implementation

Usage

A way to simulate member function for enumerations;

Article::~Article

Why not the operator `.*`?