DEV Community: panmanio

Initializing with std::initializer_list involves copying

panmanio — Sun, 14 Aug 2022 16:21:00 +0000

A convenient way to initialize STL containers is to use a initializer list, like this:

auto data = std::vector<std::string>{"example", "input", "data"};

There’s a caveat: the arguments of the vector constructor are first constructed and then copied. It’s a trait of std::initializer_list. It may become an issue when the parameters are not trivially copyable because they handle some resources etc. Short strings are not an issue as they should be handled with small-string optimization, but long ones (it depends on the standard library implementation which ones are small and which aren’t) will need three memory allocations: one during string creation, the second one for the vector item and the third one for the copy.

auto data = std::vector<std::string>{
    "long string that will likely be allocated on the heap"
}; // three memory allocations

Let’s write a test program that traces the number of memory allocations:

void* operator new (std::size_t count) {
    std::cout << "new " << count << " bytes\n";
    return std::malloc(count);
}

int main() {
    std::cout << "Vector with small string:\n";
    auto data = std::vector<std::string>{"small string"};
    std::cout << "\nVector with long string:\n";
    data = std::vector<std::string>{
        "long string that will likely be allocated on the heap"
    };
}

I’ve compiled it with g++ 9.4.0 with the optimizations on: g++ test.cpp --std=c++17 -O2 -o test. The output of the program execution is:

./test
Vector with small string:
new 32 bytes

Vector with long string:
new 56 bytes
new 32 bytes
new 56 bytes

We can see here that for the small string std::vector has allocated 32 bytes, which is the size of std::string. Creating the vector that handles long string involves 3 memory allocations: the first one comes from the construction of the parameter, the second one is the allocation of the memory for std::string inside the vector and the last one is the copy of our parameter.

To get rid of the copy we have to avoid std::initializer_list. Let’s write a make_vector function instead. My draft looks like this:

template <typename T, typename... U>
std::vector<std::decay_t<T>> make_vector(T&& arg, U&&... args) {
    auto result = std::vector<std::decay_t<T>>{};
    result.reserve(1 + sizeof...(args));
    result.emplace_back(std::forward<T>(arg));
    (result.emplace_back(std::forward<U>(args)), ...);
    return result;
}

Now the parameters won’t be copied because they are forwarded to the emplace_back method which constructs them in place. An important thing is to allocate the memory for all the parameters at once using reserve method, otherwise we save the copy allocations but add redundant allocations for vector items.

Let’s tweak the test program:

int main() {
    std::cout << "String size: " << sizeof(std::string) << "\n";
    std::cout << "\nVector with small string:\n";
    auto data = std::vector<std::string>{"short vector"};
    std::cout << "\nVector with a long string:\n";
    data = std::vector<std::string>{
        "a long string that will likely be allocated on the heap"};
    using namespace std::literals;
    std::cout << "\nmake_vector with a long string:\n";
    data = make_vector(
        "a long string that will likely be allocated on the heap"s);
    std::cout << "\nVector with two long strings:\n";
    data = std::vector<std::string>{
        "a long string that will likely be allocated on the heap",
        "another long string that will likely be allocated on the heap"};
    std::cout << "\nmake_vector, two long strings:\n";
    data = make_vector(
        "a long string that will likely be allocated on the heap"s,
        "another long string that will likely be allocated on the heap"s);
}

The output looks better now, I’ve added comments for the explanation:

String size: 32

Vector with small string:
new 32 bytes # vector item (sizeof std::string)

Vector with a long string:
new 56 bytes # the argument
new 32 bytes # the vector item
new 56 bytes # argument copy

make_vector with a long string:
new 56 bytes # the string argument
new 32 bytes # the vector item

Vector with two long strings:
new 56 bytes # the first argument
new 62 bytes # the second argument
new 64 bytes # memory for two vector items
new 56 bytes # copy of the 1st argument
new 62 bytes # copy of the 2nd argument

make_vector, two long strings:
new 62 bytes # the second argument
new 56 bytes # the first argument
new 64 bytes # memory for two vector items

The strings are not copied anymore when make_vector is used and using it is also convenient.

Enabling structured binding for std::complex (and custom non-POD types)

panmanio — Tue, 09 Aug 2022 22:45:00 +0000

The father of C++, Bjarne Stroustrup, mentions in A Tour of C++ that it's possible to use structured binding declaration with std::complex, which is a non-POD type (it encapsulates the data attributes, values are obtained with real() and imag() method calls). As of today it actually does not work for std::complex (checked with the Compiler Explorer), also the WG21 standard working draft does not mention that ability of std::complex. But it is possible to make it work by making std::complex a tuple-like type by incorporating these rules (see structured binding):

std::tuple_size<std::complex<T>>::value should be an integer constant expression denoting the number of identifiers that are part of the structured binding,
for each identifier an expression std::tuple_element<i, std::complex<T>>::type, where i is the constant expression identifier index, should be the identifiers types,
for each identifier, the function get<i>(c), where i is the constant expression identifier index, and c is the complex object, should provide the value of the identifiers.

We simply need to extend the std namespace and provide all ingredients.

namespace std {
    template<typename T>
    class tuple_size<complex<T>> {
    public:
        static constexpr size_t value = 2;
    };
    template<size_t I, typename T>
    auto get(const complex<T>& c) {
        if constexpr (I == 0) return c.real();
        else return c.imag();
    }
    template <size_t I, typename T>
    class tuple_element<I, complex<T>> {
    public:
        using type = decltype(get<I>(declval<complex<T>>()));
    };
}

Now this works nicely:

auto c = std::complex<int>{1,1};
auto [r, i] = c + 2;

Similarly for a custom type:

template <typename X, typename Y>
class NonPOD {
    X x;
    Y y;
public:
    NonPOD(X x, Y y): x{x}, y{y} {}
    X getX() const { return x; }
    Y getY() const { return y; }
};

namespace std {
template <size_t I, typename Arg, typename ...Args>
class type_alternatives {
public:
    using type = typename type_alternatives<I-1, Args...>::type;
};

template <typename Arg, typename ...Args>
class type_alternatives<0, Arg, Args...> {
public:
    using type = Arg;
};

template <size_t I, typename X, typename Y>
class tuple_element<I, NonPOD<X, Y>> {
public:
    using type = typename type_alternatives<I, X, Y>::type;
};
} // namespace std

template <size_t I, typename X, typename Y>
typename std::tuple_element<I, NonPOD<X, Y>>::type
get(const NonPOD<X, Y>& t) {
    if constexpr (I == 0) return t.getX();
    else return t.getY();
}

auto test() {
 auto sb = NonPOD{1, 2.};
 auto [s, b] = sb;
 return s+b;
}

In our own types we can define a member get<i>() method instead of standalone get<i>(obj).

Side note: there is an interesting property of std::complex<T>:

4
If z is an lvalue of type cv complex then:

(4.1)
the expression reinterpret_cast(z) is well-formed,

(4.2)
reinterpret_cast(z)[0] designates the real part of z, and

(4.3)
reinterpret_cast(z)[1] designates the imaginary part of z.

That means we could simply cast:

auto c = std::complex<int>{1,1};
auto [r, i] = reinterpret_cast<int(&)[2]>(c);

Better not use it with a temporary though.

Git clang-format json support issue

panmanio — Tue, 19 Oct 2021 19:46:48 +0000

With a recent update of my Slackware distro llvm package was updated to version 13.0. I've noticed that when git clang-format, which I use to keep the code formatting clean in my projects, suddenly started to complain when I have modified a json file:

Configuration file(s) do(es) not support Json: /home/user/workspace/myproject/.clang-format, /home/user/workspace/.clang-format
error: `clang-format -lines=2:3 -lines=52:52 path/to/file.json` failed

That made me unable to commit (git clang-format is a required step in my pre-commit hook. I needed to get my work done so I've went for a quick fix (without disabling the pre-commit hook) which was to filter-out the files that are touched by git clang-format. By default the tool parses all modified files (those tracked by git). The help section (git clang-format -h) says it can be changed with clangFormat.extensions setting. Used that option to ignore the jsons (global ~/.gitconfig or local .git/config in the project repository):

[clangFormat]
    extensions = "cpp,h,hpp,hxx"

Once I was done with my urgent task I've tried another way of fixing the issue. The clang-format 13.0 is able to format the json files but it has failed to do so because my .clang-format specified the settings for C++ files only. Solution was to add a section for the json documents:

---
Language: Json
BasedOnStyle: llvm

That not only removes the error message but also lets clang-format to take care of my json files formatting. It's not a perfect solution though as I would like to make it work for any supported file type, but couldn't figure out how to achieve that.

Dump tmux pane history to a file

panmanio — Sun, 04 Jul 2021 17:22:00 +0000

Consider you have executed a long lasting command that produced many lines of output; or that you were trying to reproduce a non-deterministic issue with your software that has finally reproduced, and you have hundreds of log lines to analyze. The problem is you forgot to redirect the output to a file 😒.

Your terminal can help to dump its buffer to a file. Alternatively you can select and copy all the lines manually but that is inconvenient and error prone. Another option is to take advantage of terminal multiplexer capabilities, in case you are using one.

With tmux you can save last N lines of current pane to a file with two consecutive commands:

:capture-pane -S -N
:save-buffer ~/filename

Replace N with desired number, or capture whole pane history with :capture-pane -S -.
That can be reduced to a single command when you invoke the capture-pane from the command line, like this:

tmux capture-pane -pS -10000 > ./last-10000-lines.out

Or this:

tmux capture-pane -pS - > ./pane-history

Another advantage of using command-line version is that you can store it as a function or alias in your shell, like in bash:

alias tmux-save-pane='tmux capture-pane -pS -'

Now the use is really simple:

tmux-save-pane > ~/tmux-pane-history

That command has helped me in trouble many times, I hope you also find it useful. If you are not using a terminal multiplexer, I really advise to start using one. tmux is the multiplexer I use, but as a vim addict I can't live with the default C-b prefix. First thing to do in my .tmux.conf is to change the binding to C-a 😉.

Setting up VIM for blogging

panmanio — Sat, 24 Apr 2021 01:46:06 +0000

VIM is the editor I use for programming. It is known for it's power at writing code thanks to it's built in features and numerous plugins, but how does it serve for blogging? Let's investigate on how to improve it's default behavior.

Spellcheck

Spell checking is built in feature of VIM, but it is disabled by default. I've enabled it with:

set spell spelllang=en_us

That made VIM highlight my spelling errors. I can jump between misspelled words with ]s and [s. With the cursor located on a misspelled word I can type z= to get a list of possible fixes. When VIM is wrong I can tell it to accept a word with zg (undo with zw).

Abbreviations

VIM supports expansion of user defined abbreviations. For instance, if you define "utl" as an abbreviation of "utility", VIM automatically replaces "utl" with "utility" when typing. This feature can be used for auto correction. Just define:

:iabbrev veiw view

to make VIM replace each "veiw" with "view" on the fly (in insert mode). Vim-autocorrect is one of the plugins that come with predefined useful abbreviations.

Dictionary

Next feature is auto completion. It is also supported by default but you may need to provide the dictionary and that was my case. Check if you have /usr/share/dict/words. I use Slackware and Ubuntu on daily basis; Slackware happened to have the dictionary installed, but on Ubuntu it was not there. Managed to provide the dict with sudo apt install wamerican. That tells VIM to use the dictionary file:

set dictionary=/usr/share/dict/words

Now that the dictionary is set up, it can be used with <C-x><C-k> in insert mode.

Thesaurus

Looking for synonyms? VIM supports thesaurus, however it has to be configured. I was able to configure the built in feature but it needed a hack to handle multi word synonyms and I didn't like that. I've decided to install vim-lexical plugin. Just as the built in feature, the plugin needs a synonyms file to work: grabbed one from Project Gutenberg. Tell vim-lexical where the file is and initialize the plugin:

let g:lexical#thesaurus = ['~/.vim/mthesaur.txt']
call lexical#init()

This plugin also handles dictionary and spell check features, I gave it a try. Here are some settings to set it up:

let g:lexical#spelllang = ['en_us']
let g:lexical#dictionary = ['/usr/share/dict/words']
" normal mode key mappings:
let g:lexical#spell_key = '<leader>s'
let g:lexical#thesaurus_key = '<leader>th'
let g:lexical#dictionary_key = '<leader>k'

In insert mode, we can use <C-x><C-t> for thesaurus.

Other plugins

There are more plugins that support the writers work, I present few that I have found particularly useful.

textobj-sentence

This plugin provides motion commands based on full sentence detection. You can switch around sentences with ( and ) and use it just as any other motion commands. Depends on vim-textobj-user plugin.

wordy

Wordy lets you identify phrases that are overused, misused, abused, colloquial, idiomatic etc. It's a nice lightweight tool that operates on higher level than just single words. Definitely worth checking out.

ditto

Struggling with word repetitions? Ditto is there to localize & highlight them for you.

LSP + proselint

Proselint is not a VIM plugin. It is a separate tool described as a linter of English prose. Here are some of its features:

Avoiding archaic forms
Avoiding needless backformations
Avoiding redundant currency symbols
Avoiding false plurals
Avoiding illogical forms
Avoiding the word suddenly
Avoiding oxymorons
Calling jobs by the right name
Not comparing uncomparables
Using dïacríticâl marks
Linking only to existing sites

Full list here. Since the tool is a linter, it sounds like it should work with language servers. I use CoC.nvim for LSP features. Thankfully some smart guys have figured out how to make proselint work with coc.nvim & coc-diagnostic (see here). Now it works for my blog posts just like clangd does for my C++ code.

Let's see how it works

All these solutions look promising. I've already benefited from including them in my VIM configuration while writing this blog post. Will do further testing and investigation in next days. Haven't yet tried plugins like vim-pencil or vim-abolish.

Bash anti-if

panmanio — Thu, 22 Apr 2021 23:33:37 +0000

Bash if-statements often look bloated to me, especially when the script aims to accomplish quite simple goal but needs to perform plenty of sanity checks before proceeding with the actual task. Let's take advantage of shell builtin capabilities to write simpler code.

The `test` command and logical operators

Instead of writing:

if [ $# -eq 0 ]; then
  echo Params missing;
  exit 1;
fi

we can use the test (or [) builtin command ¹:

test $# -eq 0 && { echo >&2 Params missing; exit 1; }
# or:
[ $# -eq 0 ] && { echo >&2 Params missing; exit 1; }

In the latter, ] is simply the last argument to [. Putting ] is required. I prefer test over [ as it looks less confusing for someone who sees it for first time. Also, with test we can take advantage of _ variable, which is the last argument of previous command:

test -f ~/.bash_aliases && source "$_" || { echo >&2 "$_ is missing"; exit 1; }

I often define die function so the code becomes:

die() {
  echo >&2 "$@"
  exit 1
}
test -f ~/.bash_aliases && source "$_" || die "$_ is missing"

Remember to group the commands in curly brackets when you expect they can fail or when not sure; or go with a separate function.

The commands can be chained like this:

test -x ./configure &&
  ./configure &&
  make &&
  make install ||
  die "Failed to build & install"

Many people find such concise constructs undesired as these statements may make one stop and think on the purpose of what is happening there, so they prefer to go with standard if-statements as being more expressive. This argument is perfectly fine and should be definitely considered when working in a team. But it's good to know what are the possibilities.

After all, I am not trying to fight the if-statements (pardon the click-bait 😅). Personally I use these "tricks" only for simple cases. Let's have a look at other shell capabilities we can use no matter if we go with if-statement or test.

Use default values

Let's take a look at parameter expansion and default variable values. That can also help to reduce the number of if-statements.

target=${1:-${PWD}}

If there is an argument provided, use that as the target. Otherwise choose current directory.

echo ${target:=${PWD}}

Prints target if it is set, otherwise prints ${PWD} and assigns that value to target.

echo ${target:?"Okay, Houston, we've had a problem here"}

When target is not set (or null), prints an error with the message provided and quits (when used non-interactively, like in a script).

find . -name _config.yml ${name:+-o -name $name}

Use alternate value. When name is not set or null, leave it as it is. Otherwise, replace with the alternate value. Results with:

find . -name _config.yml

or, when name is set and not null:

find . -name _config.yml -o -name $name

In case the colon is omitted in the examples above the null check is ignored (only unset is verified).

Reject unset variables by default

You can go one step further and reject any unset variable expansion by default. Just do this:

set -u

Now when you try to expand an unset variable, the shell will print an error and quit.

Traps

Traps let you register some action to be executed when the script receives a signal. That can be a standard signal (see kill -l or man 7 signal), but it can also be EXIT ²:

cleanup() {
  rv=$?
  test $rv -ne 0 && rm /tmp/mycache/* -rf
}
trap cleanup EXIT

The cleanup function will be executed when the script exits (even when the script ends it's life naturally, without explicit exit call). Once the trap is registered, you don't have to worry about the cleanup. Traps not only let you write less ifs, they are much more powerful tool (consider trap cleanup SIGABRT SIGKILL).

The null command

There is a special builtin : command that is always successful (just like true) and does nothing but expands its parameters (and performs redirections if specified). Example:

: ${target=${PWD}}

Here I've reused one of previous examples. The command above sets target to current directory when target is not set (but leaves null untouched as I have used = instead of := this time).

Final words

Bash (and other shells) is really powerful and complex language. I keep learning it and there is always much more to learn, which I find fascinating. If you have enjoyed this post I advice you to read more on the topic. I am not the first one to explore these areas, please check out Elegant bash conditionals by Tim Visée and Anybody can write good bash (with a little effort) by William Woodruff, among others.

Most often there is also a usual test/[ command on the system, compare the output of which test with type test. ↩
The bash manual specifies more signals supported by trap, like RETURN or DEBUG. ↩

Lambda expressions in unevaluated operands

panmanio — Wed, 21 Apr 2021 21:51:27 +0000

While writing some unit tests today I have run into a compile issue:

error: lambda expression in an unevaluated operand

I was astonished at first as I've just scratched a simple test case that was not using unevaluated operands. But what are unevaluated operands?

They are literally operands that are not evaluated 😄. They appear in the context of:

typeid operator (not always ¹)
sizeof operator
noexcept operator - decltype specifier
require-expressions
constraint-expressions

There is also declval function template that can only be used in unevaluated context.

Unevaluated operands are processed at compile time. What does it mean they are not evaluated? Let's come up with some examples.

#include <iostream>
#include <typeinfo>

struct S {
    S() { std::cout << "S()" << std::endl; }
    int fn() {
        std::cout << "S::fn()" << std::endl;
        return 0;
    }
};

template <typename T>
constexpr bool predicate() {
    return true;
}

template <>
constexpr bool predicate<double>() {
    return false;
}

template <typename T>
T fn() requires(predicate<T>()) {
    std::cout << "fn<T>() true branch" << std::endl;
    return T{};
}

template <typename T>
T fn() requires(!predicate<T>()) {
    std::cout << "fn<T>() false branch" << std::endl;
    return T{};
}

int check() {
    constexpr auto x = decltype(fn<int>()){};
    if (typeid(S{}.fn()) == typeid(x) &&
        sizeof(0.) == sizeof(fn<double>())) {
        return x;
    }
    return 1;
}

int main() { return check(); }

Above code prints nothing when executed: that means none of S::S(), S::fn() and fn<T>() were invoked. The predicates are also not evaluated.

Fair enough, what was the issue with my unit test? I was trying to count items in a vector using a lambda predicate:

EXPECT_EQ(std::count_if(begin(data), end(data),
                        [](const auto& d) {
                          return !d.children.empty();
                        }),
          13);

The thing is gtest's EXPECT_EQ macro performs some checks and it has put the count_if in a sizeof, but lambda-expressions are not allowed in unevaluated operands up to C++17. Starting from C++20 the limitation is no longer there. Since my project is not yet switched to C++20, I chose a simple fix:

const auto count = std::count_if(begin(data), end(data),
  [](const auto& d) {
    return !d.children.empty();
});
EXPECT_EQ(13, count);

I've also reordered the arguments because it has an impact on the error message when the expectation fails: first argument is the expected one, second is the actual one.

Out of curiosity I have checked if I can use lambda in unevaluated context in recent versions of clang (12.0.0), gcc (10.3) and msvc (v19.28):

void unevaluated_lambda() {
    decltype([]{}) x;
    (void) x;
    decltype([]{ return 0; }()) y;
    (void) y;
}

Clang complains with the same error: lambda expression in an unevaluated operand. The code compiles with gcc and msvc.

When typeid is used with an object of polymorphic type, its operand is evaluated. ↩

Inline member variables in C++

panmanio — Tue, 20 Apr 2021 21:59:40 +0000

Say we need a class member variable (non-const). Global variables should be best avoided, but let's ignore that for now and focus on the solution.

This is a bad choice most often:

struct S {
    static int i;
};
// somewhere in cpp file:
int S::i = 0;

The definition (int S::i = 0;) needs to be placed in a single translation unit in the whole project, otherwise we get an error indicating multiple definition of S::i.

In pre-C++17 world we need to find some workaround of this problem. One way is to go with a static member function:

struct S {
    static int& getStatic() {
        static int i = 0;
        return i;
    }
};

// sample use: ++S::getStatic();

Another way is to use templates:

template <typename T>
struct S
{
    static int i;
};

template <typename T>
int S<T>::i = 0;

using static_instance = S<void>;
// sample use: ++static_instance::i;

That makes a lot of boilerplate and obscure code. Our problem is solved by inline variables in C++17. That looks simple and serves well:

struct S {
    static inline int i = 0;
};

// sample use: ++S::i;

The inline variable has an external linkage if our class is not inside an unnamed namespace (i.e. when the class has external linkage too). It enables the possibility of more than one definition of the variable (hence no multiple definition errors). The variable must be declared inline in all the translation units and it always has the same memory address. All the definitions have to be identical, otherwise we break one definition rule. That may lead to hard to detect problems if we are not cautious.

// one translation unit, S has external linkage
struct S {
    static inline int i = 3;
};

// another translation unit, S has external linkage
struct S {
    static inline double i = 3.14159;
};

The code in the example above is invalid, but it compiles and can behave unexpectedly.

Static constexpr members are inline by default.

Interesting resources:

Capture structured bindings in C++17

panmanio — Tue, 20 Apr 2021 00:14:52 +0000

C++17 introduced a handy construct called structured binding:

const auto [first, second] = std::make_tuple(1,2);

Structured bindings are used to decompose arrays and structs/classes ¹ to named subobjects. Almost any object with non-static, accessible data members can be destructured this way. It works even for bit-fields:

struct BF { int x : 2; };
const auto bf (BF{1});
const auto& [y] = bf;

Resulting aliases (first, second and y in the examples above) are actually not mere variables but rather identifiers or aliases.

This may feel unintuitive but there actually is a case when we cannot use these aliases as other variables. According to this C++ Language Standard working draft we cannot use structured bindings in lambda capture list:

If a lambda-expression explicitly captures an entity that is not odr-usable or captures a structured binding (explicitly or implicitly), the program is ill-formed.

(emphasis mine). That means following code is illegal:

auto [a, b, c] = std::make_tuple(1, 3, 7);
auto d = [b] { return b; }();

It actually works on gcc (10.3) and msvc (v19.28) but fails on clang (12.0.0):

error: 'b' in capture list does not name a variable

This restriction does not apply for init-captures, which is reasonable as with init-captures there is a new variable defined that is captured and this variable is no longer the structured binding, so this compiles well in clang, gcc and msvc:

auto [a, b, c] = std::make_tuple(2, 7, 2);
auto d = [&b = b] { return b; }();

The standard got reworded along the way and in C++ 20 final working draft the restriction is no longer there. But clang still fails, even with -std=c++20, while gcc and msvc are fine with both simple- and init-capture of the structured binding. I feel way more comfortable with gcc and msvc way as there is no need to provide special construct only to capture a single value.