DEV Community: Tanker

The quest for a better hiring process

Dimitri Merejkowsky — Wed, 22 Jul 2020 15:20:36 +0000

The quest for a better hiring process

I’ve been a software engineer for about ten years, and here’s something I learned: hiring software engineers is hard. One of the main challenges is to get the most information about the candidates in the minimum amount of time. You want to know if they will be a good fit for your team, and you want to know it quickly, without wasting their time.

In this article, I’ll go through a list of the various techniques I’ve seen, describe the one we are using at Tanker, and why it works best.

A tour of existing techniques

The whiteboard

Let’s start with the infamous whiteboard. You ask the candidates to write some code from scratch on a whiteboard to solve a given problem. This can give you an idea of how they think or how they approach new challenges, but bear in mind the whiteboard can get very intimidating for some developers, so they may appear less bright than usual.

All in all, the whiteboard is a peek into the reasoning abilities of the candidate. Still, it doesn’t give any information about the ability to work in a team, understand a new codebase, or their skills with a given programming language.

The homework

Another technique is to ask the candidate to implement a project “from scratch” at home. This technique avoids some of the above pitfalls, and you can quickly gauge the candidate’s level of competence.

Note that you only get to see how they work alone and when writing code from scratch. Moreover, either it means a big-time investment for the candidate or the project is too small to correctly judge their skills.

None of these techniques match our needs: we are a team with a flat organization in which each member has the right to contribute to important decisions. Besides, we already have a pretty large codebase, so future recruits will mostly work on existing code.

For these reasons, we use a refactoring exercise, a technique I learned about during an episode of Ruby Tapas presented by Nickolas Means.

As the name implies, the candidates will have to refactor some code. They also will have to demonstrate other abilities like reasoning about new features and dealing with tests.

Here’s how it works.

The Refactoring Exercise

Before meeting the candidate

First, take a simple problem and implement it. The code does not have to be very long, but you should make sure there’s room for improvement: leave out some duplication, use functions with lots of parameters, or other anti-patterns. However, do use features you want the candidate to be familiar with.

Then write some basic tests and think about a new feature to add, but don’t implement it. This will be important later on.

Running the exercise

Remember, your goal is to gather the most valuable information about the candidates: you want to see how they work when they are at their best.

For instance, it’s crucial to tell the candidate about the refactoring exercise, so it’s not a surprise for them to make sure they bring their own development machine.

When the candidate arrives, show them the code and tell them about the problem it’s solving.

At this point, the code should be in a state that makes it hard to implement the new feature without refactoring.

Then announce the new requirements and wait. Make it clear that the code should be production-ready.

Do they ask for existing tests?
Do they try and refactor first?

Your goal is to see if they think about those “good practices” without a prompt from your end. But if they start writing the new code directly, it’s not a red flag per se.

You do want to see how they refactor and use tests, though, so if they skip all of the above, just gently nudge them in the right direction with questions like “Is there something you want to do before implementing the new feature?” or remarks like: “By the way, there are some tests if you think that’s helpful.”

Your goal is to keep them comfortable, so don’t make them think they have failed if they did not ask about tests or refactoring right away. It happens.

The rest is straightforward: watch the candidate do the refactor, implement the solution, and add tests. Note that the whole session should last about 30 minutes.

What running the exercise tells you

Let’s recap what you’ve learned during the refactoring session:

You have a pretty good idea about the candidates’ skills and knowledge in your programming language
You can see how they make decisions about adding features, adding tests, and architecture of their code
You make sure that they can convert requirements into code
You know how it would be like to pair-program with them

At Tanker, we’ve been using this technique for quite some time, and it has worked very well. It gives us a good idea of how candidates will do all day, every day — and all of that in 30 minutes — not bad!

Interested in joining us? Have a look at our job openings.

Automating version number updates: what could go wrong?

Dimitri Merejkowsky — Thu, 11 Jun 2020 09:35:17 +0000

Automating version number updates: what could go wrong?

Say you need to update (bump) your software. It’s currently at version 1.2, all the required changes have been merged, and it’s time to publish version 1.3. That’s really easy, right? Change the version in one file, commit, tag, and push. Done!

I thought that too once, but the truth is that it’s harder than it looks — let me tell you my story.

Releasing software for Softbank robotics

Our story begins in 2008, when I was release manager at Softbank Robotics.

I had two big projects to release: Choregraphe, a desktop GUI to control the NAO and Pepper robots, and qiBuild, a command-line application to ease C++ development.

For qiBuild, I had three files to patch because the code version number was hard-coded in several places (setup.py, __init__.py, and docs/conf.py).

On the other hand, for Choregraphe the version number was hard-coded in the top CMakeLists.txt file only, but there was quite a bit of code to “forward” the version number from the top CMake file down to the version.hpp and main.cpp files.

Both solutions had their pros and cons but I could not decide which one was best. Since I was pretty sure I was not the first one to have encountered this issue, I started to look around for better solutions.

Trying out bumpversion

The way bumpversion works is a nice combination of the two approaches I’ve mentioned before:

First, keep the hard-coded version number in as many files as required
Second, add a dedicated configuration file (.bumpversion.cfg) containing the current version and the aforementioned list of files.

Then, when bumping the project from version X to version Y:

Iterate over the file list
Replace all occurrences of X with Y, including in the configuration file itself.

So, I started with the qiBuild project and added the following configuration file:

# in .bumpversion.cfg
[bumpversion]
current_version = 1.0.1
commit = True
tag = True

[bumpversion:file:setup.py]

[bumpversion:file:qiBuild/__init__.py]

[bumpversion:file:docs/conf.py]

Since I had to bump qiBuild from 1.0.1 to 1.0.2, I ran:

bumpversion patch

A commit was automatically made along with a matching tag:

$ git show
commit ec50897893ce4ecfb1debaa1df266ae4c555f45b (HEAD -> master, tag: v1.0.2)
Author: Dimitri Merejkowsky <dimitri.merejkowsky@tanker.io>
Date: Wed May 20 16:58:27 2020 +0200

Bump version: 1.0.1 → 1.0.2
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
-------- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
 [bumpversion]
-current_version = 1.0.1
+current_version = 1.0.2
 commit = True
 tag = True

diff --git a/qiBuild/__init__.py b/qiBuild/__init__.py
-------- a/qiBuild/__init__.py
+++ b/qiBuild/__init__.py
-__version__ = “1.0.1”
+__version__ = “1.0.2”

diff --git a/setup.py b/setup.py
-------- a/setup.py
+++ b/setup.py
 setup(
    name="qiBuild",
-   version="1.0.1",
+   version="1.0.2",
     )

diff --git a/docs/conf.py b/docs/conf.py
-------- a/docs/conf.py
+++ b/docs/conf.py
project = "qiBuild"
- version="1.0.1",
+ version="1.0.2",

Great! Now I just had to run python setup.py sdist upload and the 1.0.2 release was published on pypi.org.

For the NAOqi project, it worked very well too.

I could delete a bunch of CMake and preprocessor code and replace it with just one line of C++ code:

// in naoqi/version.hpp
static std::string const NAOQI_VERSION = "2.3.0";

This time the .bumpversion.cfgfile looked like this:

[bumpversion]
current_version = 2.3.0
files = include/naoqi/version.hpp

Now bumping naoQI’s version could be done with a command similar to the one used to bump qiBuild.

Brilliant! So, what went wrong?

Wandering off-track

The problem can be described as being “too smart by half” and has to do with the command line syntax of bumpversion.

Indeed, bumpversion is smart enough to bump various “parts” of the version number, namely the major, minor, and patch components used in the semver spec.

Here are some examples, assuming that the current version is 1.2.3:

bumpversion patch : 1.2.3 -> 1.2.4
bumpversion minor : 1.2.3 -> 1.3.0
bumpversion major : 1.2.3 -> 2.0.0

We were using semver for qiBuild and NAOqi too at the beginning — but sometimes semver is not enough.

Let’s continue our story. When qiBuild got more usage and the software team grew, publishing qiBuild releases started becoming… scary.

New features were added, refactorings were made and qiBuild had become an essential tool for all the developers in the software team (100 of them) — I was getting nervous about what would happen if I shipped a buggy release.

So, with the help of members of my team, I decided to start making release candidates. That way, a few brave colleagues could help me catch bugs before everyone upgraded to the latest stable version.

Since Python developers had already come up with a version scheme that allowed for release candidates, I started using that. See PEP 440 for details. Basically, I could add a rcX suffix after the patch part.

And… it turned out that doing so was far from trivial. Why?

Well, bumpversion assumes you are using semver and, if you don’t, you need to specify a custom regex:

parse = (?P<major>\d+) 
 (?P<minor>\d+) 
 (\.(?P<patch>\d+))? .
 ((?P<release>rc)(?P<rel_num>\d+))?

But it does not stop there.

Now you need a way to tell bumpversion how to go from 2.0.0rc1 to 2.0.0rc2 (if the release candidate had bugs) or from 2.0.0rc2 to 2.0.1 (if it did not).

So you add even more configuration, but it’s still not enough:

serialize =
 {major}.{minor}
 {major}.{minor}.{patch}
 {major}.{minor}{release}{rel_num}

[bumpversion:part:release]
values =
 a
 b
 rc

You can find all the gory details in the GitHub issue but, in short, it was a show stopper.

I believe that bumpversion is still not used at Softbank Robotics to this day, and, as far as I know, bumping qiBuild still needs version numbers to be fixed manually in several files.

But our story does not end here — that would be a rather sad ending!

Arriving at Tanker

In March 2016, I handed in my resignation letter at Softbank and, three months later, I started my current job at Tanker.

In short, Tanker sells a product that can be used for end-to-end encryption, as well as client-side anonymization, packaged in open source SDKs (see our website for details).

Anyway, given we were releasing software and, because of my past experience, I was also in charge of release management at Tanker.

As you might expect, Tanker also had hard-coded version numbers and chunks of code whose only role was “forwarding” the version number from one file to another.

“This is exactly the same problem I had last time!”, I thought. So, I took a look at bumpversion again — but even after all this time the bug I opened was still not fixed.

That’s when I realized there were two big problems with bumpversion which would be pretty hard to fix without rewriting a lot of code.

First, it’s very hard to reliably identify “parts” of version numbers. Semantics can vary from one version scheme to the next. Even comparing version numbers is a hard task, but guessing how to bump from a release candidate to a stable one is near impossible.

Secondly, there are some hidden defaults at play which make understanding what’s going on under the hood pretty hard.

In other words, bumpversion was “too clever by half”.

So, what to do? Well, rewrite from scratch of course! (It turned out to be a good idea after all — otherwise, I would not brag about it here :P)

The birth of tbump

On December 7, 2017, I started working on a rewrite called tbump. My goal was to keep bumpversion’s good ideas and to fix its shortcomings.

Here are the main differences between tbump and bumpversion:

There are no hard-coded defaults: you must specify how the git message and the tag name will be formatted, and you also need to specify a regular expression to define the version scheme.
Instead of specifying what part of the current version you want to update, you need to pass the whole new version.

Back to our example — to go from 2.1.3 to 2.1.4 you run tbump 2.1.4 instead of bumpversion patch.

Those differences come with a price.

First, since there is no hard-coded default it’s harder to use tbump out of the box.

However, this one was easy to fix : I added an init command to generate a tbump.toml file automatically. Instead of having to read the docs, users can read the generated file and get started quickly.

Secondly, since one has to specify the new version instead of a segment one wants to bump it’s easy to make mistakes, like going from 1.0.3 to 1.0.5 instead of 1.0.4.

That’s where it gets interesting.

You see, I was pretty annoyed by some aspects of the bumpversion UX, especially when trying to tweak the configuration file.

Just watch:

$ bumpversion patch --dry-run
<nothing>
$ bumpversion patch --verbose --dry-run
current_version=1.0.2
commit=True
tag=True
new_version=1.0.3

Now look at what tbump --dry-run does:

$ tbump --dry-run 1.0.3
:: Bumping from 1.0.2 to 1.0.3
=> Would patch these files
- setup.py:3 version="1.0.2",
+ setup.py:3 version="1.0.3",
- foo.py:1 __version__ = "1.0.2"
+ foo.py:1 __version__ = "1.0.3"
- tbump.toml:2 current = "1.0.2"
+ tbump.toml:2 current = "1.0.3"
=> Would run these git commands
$ git add --update
$ git commit --message Bump to 1.0.3
$ git tag --annotate --message v1.0.3 v1.0.3
$ git push origin master
$ git push origin v1.0.3

The new version is all over the place, you can’t miss it, and at the same time, you see exactly what is going on.

The output is similar without the — dry-run option, except changes are actually applied and git commands are run.

And then I realized that every time I was bumping something, I would be following the same pattern:

Run tbump $NEW_VERSION --dry-run
Check if everything was OK by reading the output
If not, back to Step 1
Otherwise, run tbump $NEW_VERSION

This looks like an algorithm — and what do we do with algorithms? We implement them!

So here’s what the entry point of tbump looks like:

# (simplified)
def main():
    bump(dry_run=True)
    answer = input("Looking good (y/n)?")

    if answer != "y":
       sys.exit("Canceled by user")
    else:
       bump(dry_run=False)

That way you get a chance to catch any mistake in the new version just before it’s too late :)

Early adoption

In January 2018 tbump 1.0 was out (and I had been using tbump to bump itself since the 0.2 release).

I published the package on pypi.org and I started using it both for Tanker projects and my own ones.

It worked great! My fellow developers told me they liked the UX so I was pretty happy.

But there was a critical flaw in tbump’s design that would come back to bite us pretty hard in the future. Let’s see how.

yarn workspaces

Quite early in the development of our Javascript SDK, we knew we’d be using a mono-repo.

For instance, we wanted to support both Node.JS (for fast-running tests) and the browser (which is the primary use of the SDK).

So right off, we knew we would have at least three packages: @tanker/core, @tanker/client-browser, and @tanker/client-node.

It did not make sense to have different version numbers for those three packages, so here’s what we ended up with:

// In core/package.json
{
  "name": "@tanker/core",
  "version": "1.2.0",
  // ...
  "dependencies": {
    "libsodium-wrappers": "^0.5.1",
    // ...
  }
}

// In client-browser/package.json
{
  "name": "@tanker/client-browser",
  "version": "1.2.0",
   // ...
  "dependencies": {
    "@tanker/core": "1.2.0",
    // ...
   }
}

// In client-node/package.json
{
 "name": "@tanker/client-node",
 "version": "1.2.0",
  // ...
 "dependencies": {
   "@tanker/core": "1.2.0",
   // ...
  }
}

When bumping to a new version, we needed to patch the line that contains the “version” key in the top metadata:

[[file]]
src = "packages/core/package.json"
search = '"version": "{current_version}"'

[[file]]
src = "packages/client-node/package.json"
search = '"version": "{current_version}"'

[[file]]
src = "packages/client-node/package.json"
search = '"@tanker/crypto": "{current_version}"'

Note the search line: we did not want to blindly replace all occurrences of the new version in packages.json — if a line declares a third-party dependency that happens to have the same version number as the current one, it should not get patched!

Speaking of dependencies, we also needed to patch the line that specifies the version of @tanker/core used by @tanker/client-browser and @tanker/client-node:

[[file]]
src = "packages/client-browser/package.json"
search = '"@tanker/core": "{current_version}"'

[[file]]
src = "packages/client-node/package.json"
search = '"@tanker/core": "{current_version}"'

That’s a total of four blocks of configuration. Then we extracted a @tanker/crypto package from @tanker/core, and added two new blocks of configuration:

[[file]]
# Bump version of @tanker/crypto
src = "packages/crypto/package.json"
search = '"@tanker/core": "{current_version}"'

[[file]]
# Bump version for @tanker/core too — it depends on @tanker/crypto!
src = "packages/core/package.json"
search = '"@tanker/crypto": "{current_version}"'

Unbeknownst to us, that was the start of a slippery slope: every time we added a new package in the workspace, we’d have to add two blocks of configuration for this package, and one for every package that depended on it.

This is a famous anti-pattern, and before you can say “I see a polynomial complexity!”, we ended up with this kind of monstrosity: a 200 hundred lines configuration file!

Saved by my co-workers

Luckily I’m working with a team whose members never hesitate to share (constructive) criticism — and who often take matters into their own hands.

So, when faced with the task of trying to edit this gigantic configuration file just to add a new package, one of them decided to fix the root cause of the problem and submitted a pull request for tbump named “Stars and Dots”. Here are the two main changes it implemented:

Allow regular expressions in search strings: we could now use @tanker/.* instead of specifying the whole name of the dependency (@tanker/core, @tanker/crypto, and so on)
Allow using glob patterns to specify file names. Instead of having one block per file, we could now specify a whole bunch of files using packages/*/packages.json as a glob pattern.

Clever name for a clever pull request, don’t you think?

This pull request was of, course, quickly reviewed and merged by yours truly, and all the nasty blocks in the tbump.toml file were replaced with just two:

[[file]]
src = "packages/**/package.json"
search = '"version": "{current_version}"'

[[file]]
src = "packages/**/package.json"
search = '"@tanker/[^"]+": "{current_version}"'

Note that it remains more or less the same to this day, even if the sdk-js git repository now contains about 30 different packages :)

The cherry on the cake

Before I conclude, I’d like to mention a feature of tbump called hooks. It’s the ability to run arbitrary commands before patching the files or after the commit is made and pushed.

Hooks can run before the files are patched and the commit is made, or after the new commit and new tag have been pushed and are defined in the tbump.toml file.

You can find examples of this in tbump’s own configuration file:

[[before_commit]]
name = "Check Changelog"
cmd = "grep -q {new_version} Changelog.rst"

[[after_push]]
name = "Publish to pypi"
cmd = "tools/publish.sh"

Conclusion

tbump is now the deployment tool for Tanker.

We often use “before commit” hooks to perform various checks (like verifying that the version to be published is mentioned in the changelog), or to make sure that generated files are up-to-date (like yarn.lock for instance).

We also use “after push” hooks as “executable documentation” to specify how to publish new releases (like using poetry publish in case of a Python package).

So, what did we learn?

First, pay attention to algorithmic complexity, even in configuration files, take care in designing a good UX, even for command-line tools,
Second, if you can afford an extra pair of eyes to a given problem, don’t hesitate to ask.
Finally, there’s a battle-tested rewrite of bumpversion with a pretty good UX and nice features waiting for you on pypi.org. Feel free to try it out and tell us what you think.

Cheers!

PS: We’re hiring software engineers. If the way we work sounds interesting, have a look at the hack challenge available on https://www.tanker.io/rabbit/ and don’t hesitate to reach out!

Fantastic Bugs and Where to Find Them

Timothee Isnard — Mon, 20 Jan 2020 16:36:23 +0000

When developing an SDK running on all major platforms with bindings in half a dozen languages, you tend to run into interesting situations! Sometimes it's the kind of situation where the better part of a week turns into an extended debugging session. This is one of those times.

When we started developing our Golang SDK, we made the decision to reuse our native library instead of trying to rewrite everything. Ultimately it has proved to be the right choice for us, but it's not a very well-kept secret that Golang's interoperability story with C libraries is a little more complicated than for most languages.
Between the challenge of setting up cgo (Go's FFI system) to work with our build system and the various platform-specific issues we encountered, it's fair to say the development of our Go bindings had a bit of a bumpy start.

We were hopeful that things would start to settle down quietly after the initial work to get the Go SDK up and running. And if you're reading this, you won't be surprised to learn that the universe had other ideas.

First look

Monday comes around, and a Go unit test is failing. Armed with the fearless resolution of someone convinced that the hard part is already behind us, we get to work.
Strangely enough, the test seems to fail only when our native library is built in Release mode. But, before anything else, let's see what we can learn from the test suite's output.

We get some entirely normal log messages for a test that starts a Tanker session, but it looks like the go test runner can't quite tell us which line caused the failure. We quickly decide that this is no reason to be worried. After all, how hard could fixing a single unit test be?
Let's take a closer look in a debugger.

All right, so our test is actually crashing with an "Access violation at 0x00000000", the operating system's epitaph for programs in the dangerous business of dereferencing null pointers.
Already some of you may have noticed something unusual about this crash; instead of a regular Go or C++ function, Visual Studio claims our program is currently running code at address 0x00000000, as if there were a function there (there very much is not).

A closer inspection of the registers

For those curious about low-level details Visual Studio's "Registers" view is an even more interesting clue, although not strictly necessary; finding pieces of evidence is merely a guide that helps us make better debugging decisions. However, since this post is about to dig deep, let's take a moment to appreciate just how comprehensively wrong this crash is.

What this view shows is the CPU's hardware registers, and one of those in particular — called EIP — contains the address at which our program is currently trying to run code. As a matter of fact, we could already tell EIP would be 0 because of the access violation, but what's interesting is that the contents of most of the other registers are also conspicuously null!

Now that is a weird state to be in.

Accidentally handling null pointers can happen in the majority of mainstream languages, and corrupting memory is a dishearteningly common pastime for C programs, but registers are different. Even relatively low-level programming languages like C and C++ won't let you directly touch them (and for a good reason, those languages want to be portable, independent of hardware details). If all those registers ended up with zeros in them, this is either a particularly unlikely coincidence or the same malfunction that caused a jump to address 0 is responsible.
Knowing this, the hope of a simple programming mistake starts to wither away as this is no mere null pointer. Which leaves those pieces of code that do mess with registers as the prime suspects — assembly code gone rogue, or the compiler toolchain itself. Needless to say, neither of those possibilities is a particularly reassuring prospect.

Isolating the problem

To fix a crash like this reliably we must start by understanding what, exactly, is going wrong. The first step towards understanding this is isolating the problem. Once we've found that exact place where the tires leave the road, we can exculpate large parts of the codebase and try to reproduce the crash in a simplified environment. We need a Short, Self Contained, Correct, Example (also known as MCVE in some corners of the web).

Since the program crashes at the null address we don't have much surrounding context to look at, but we started with a failing Go unit test so let's go back to that in a debugger.
Thankfully, the bug looks deterministic (it fails in the same way every time), so we can follow the code until a function call crashes, repeat from a breakpoint inside the function, and quickly narrow down where things go south.

On the complexity of opening a debugger

Unfortunately for us there is a subtle difference between deterministic and predictable! A perfectly deterministic crash can disappear unless the conditions are just right (and taking a look at what makes a crash disappear can sometimes give us a hint as to what went wrong).
We found out that this bug requires a very specific situation: only the 32-bit version of our library, only on Windows, and only when compiled with MinGW in Release mode. And that means no debug symbols.

We checked that the Debug mode wouldn't crash, and against all hopes the RelWithDebInfo mode (a mix of Debug and Release) didn't crash either. But if we drill down into the compiler settings things are actually more subtle than Debug and Release. What seems to make the bug disappear when not making Release builds is when some specific optimizations — like inlining — are turned off.

Up to this point I had always thought that RelWithDebInfo meant exactly the Release mode with Debug information (the -g setting of the compiler) enabled, but surprise, it also turns off a bunch of optimizations, just enough to make a bug disappear!

Concretely, in our case, Release passes the -O3 flag to MinGW and RelWithDebInfo is only -O2. It turns out that if we take our Release build and manually add the -g flag everything crashes just fine. Awesome.

Finally, we can fire up the world renowned Visual Studio debugger and follow along, right?
Of course not! That would be too easy, and where's the fun in that?

We're using Go's FFI to call our library, and Go only really works well with libraries compiled by MinGW, not Visual Studio. But, you see, MinGW emits DWARF debug information, and Visual Studio doesn't want anything to do with that when it has its own PDB debug information format. There are a few ways to deal with this:

A linux-native debugger like GDB understands DWARF and can still be used on Windows,
We could get a really powerful cross-platform tool like IDA Pro that understands every format under the sun,
Or we could somehow try translating the DWARF debug information into a PDB for the sake of Visual Studio.

Really, it's just a matter of personal preference (and having two to ten thousand dollars lying around for an IDA license). We went with option three.

And now we're fully ready; all that's left is the tedious process of stepping through until we stumble on the program's last few actions.

The unit test we're investigating tries to perform some invalid operations on purpose, so it's not surprising that the code would end up throwing an exception at some point, and we can see in the screenshot above the location where we catch it. No crash so far, but if we continue in our debugger the next thing the code does is stash the caught exception away in memory, step out of the handler, and promptly jump to 0x00000000 through a small maze of Boost.Context functions. As you might have guessed, this exception handler is special. So special in fact that we need to take a moment and talk about its purpose.

Taking a step back: tconcurrent

I have a confession to make — I've been hiding a few things from you. Now is probably a good time to reveal that our entire C++ library's internals are implemented as async coroutines that return futures and await each other.
"But wait," you might say. "C++ won't have coroutines until the 2020 revision of the standard, and the early support in compilers is still experimental!" All very true, dear reader.
And, at the time of writing we're in fact still held back from a full switch to C++17 due to a few legacy platforms, so we couldn't actually use the standard C++ coroutines even if they were ready.

Instead we took things into our own hands and built our own async/await on top of the existing and very low-level Boost.Context engine — this is our tconcurrent library. It allows us to write asynchronous code almost as easily as synchronous code by enabling us to use stackful coroutines. (And importantly, tconcurrent is also compatible with the C++ Coroutine TS, meaning that as soon as we get stable compiler support we'll be able to toggle the tconcurrent back-end from Boost.Context to C++'s compiler-backed stackless coroutines.)

So what does all of this have to do with our crash and the exception that was caught above? The exception handler we land in right before the crash actually belongs to tconcurrent.
Coroutines work by sharing an event loop running in a thread, and the exception handler we landed in is how tconcurrent prevents coroutines in their last throes from bringing down the whole program. Instead, it catches the exception and stores it in a future as the result of the prematurely terminated coroutine, and then switches back to the event loop.

Good artists copy, great artists reproduce

We now have a pretty good idea where the code is crashing and, good news, it doesn't look like it has anything to do with the code in our unit test or in the Go bindings for our library! Hopefully, we should be able to make it crash with just our native C++ SDK and tconcurrent, but in order to be completely sure, it's time to reduce and reproduce the bug.

There are two approaches to creating a minimal reproduction of a crash. One is to take the full crashing test case and chip away at it bit by bit until only the essence of the bug is left. The other is to start with a blank slate and try to re-create the simplified circumstances of the bug, omitting anything unnecessary.
The first way is methodic and reliable, but often slow and mind-numbingly tedious. For a large codebase with many moving parts it's often worthwhile to give the second approach a shot before falling back to the tried and true.
We found a way to reproduce it very quickly, and I can finally show you what that looks like:

#include <tconcurrent/coroutine.hpp>
#include <fmt/core.h>

tc::cotask<int> bad_coroutine()
{
    throw fmt::format("Yeet.");
}

int main()
{
    try {
        tc::async_resumable(bad_coroutine).get();
    } catch (const std::string& x) {
        return 1;
    }
}

This code is actually pretty simple once you get past the cotask and async_resumable jargon! It really is the shortest possible re-creation of our failing unit test's behavior. We have a coroutine bad_coroutine that throws an exception unconditionally, and a main that starts bad_coroutine and then immediately blocks to wait for its result — a std::string exception — that it tries to catch.

However, it isn't immediately clear why we bother to throw some fmt::formatted text instead of just a string literal or a number. This is where the chaotic nature of the bug shines through. Any small change to the code like throwing a different kind of object or changing the #include files can affect whether or not it will successfully trigger the bug. Also not visible is the fact that we still need to compile with all optimizations enabled, on 32-bit Windows, and with the MinGW compiler for the crash to happen.

We arrived at this code by trying to mimic the events observed in the debugger as straightforwardly as possible, but there is always a little guesswork involved. We were lucky to stumble on a combination of factors that happened to work in just a few tries. As a result, we were able to rule out most of our codebase as unrelated to the bug.

Getting Into the Thick of Things

Having simplified the problem as much as possible, we can now focus intensely on understanding what is going on. At the start of this debugging session, I was blissfully unaware of the internals of either Boost or tconcurrent and, in fact, a substantial part of this article was new to me just a week ago! We're about to dig deeper still into some intricate details learned from carefully reading source code, documentation, and specification documents, so a humble warning — the following is correct only to the best of my knowledge (and we'll leave a dark corner partly unexplored, namely that identifying the reason why the bug was so chaotic and sometimes did not crash wasn't necessary to understanding the cause and fixing it, but I'm keen to see if exceedingly curious readers can model a more complete explanation!).

The first thing we learned is that the symptom of the crash is jumping to address 0, but we haven't looked in detail at how this happens. So, let's start back from our breakpoint in the tconcurrent exception handler.

We now know that this catch block intercepts and saves exceptions that bubble out of coroutines, which means the bad_coroutine has already finished executing by the time we get there. After a quick look at tconcurrent's source code, we learn that the next step in the case of our code above is for tconcurrent to switch back from bad_coroutine to the event loop, and to wake up the main function that's currently still waiting for a result so we can re-throw the exception we just caught.

All the interesting business seems to be happening outside our minimized repro code above and into library internals. We talked earlier about how tconcurrent defers to Boost.Context for all the low-level platform-specific coroutine switching code, but it's time to peel off the abstractions and take a look at how coroutines really work!

A little bit more context

If you're not intimately familiar with them yet (as I wasn't), the advantage a coroutine has over a regular function is that it can suspend itself instead of blocking the thread, and another coroutine that's ready to run will be resumed in its place (a form of cooperative multitasking). Eventually, whatever the first coroutine was waiting for will become available, making it possible to resume it (at the time the second coroutine is next suspended). All of those coroutines can share and wait for their turn to run on the same thread, and in our case this is the thread that hosts the tconcurrent event loop.

This may sound a little convoluted compared to plain old blocking threads, but the whole point of the system is the large efficiency gain: having many coroutines consumes little memory and switching between them is considerably faster than having many blocked threads and relying on the operating system to juggle them (and the gap has been growing, as the recent waves of CPU vulnerability mitigations continue to increase the cost of thread switching). In fact, coroutines are often called lightweight threads because of this.

So, if coroutines are similar to threads, this means tconcurrent and especially Boost.Context are going to have a job that looks a lot like a simplified operating system scheduler. Just like a thread, each stackful coroutine has its own stack that we need to keep track of, and its own set of CPU registers that must be saved when we suspend and restored as we resume it. This is exactly what Boost.Context takes care of. A context object can be seen as a suspended coroutine and contains all the state necessary to resume it!

So, if we go back to our minimized code above, the control flow starts to become a little clearer. Conceptually we start in main and we create a new coroutine, with its own context and stack, for the bad_coroutine function (that's what tc::async_resumable does). Then we block main while the tconcurrent event loop switches to bad_coroutine.

As we switch away from the event loop we suspend it, and so Boost.Context gives us a corresponding context that we can use to resume the event loop as soon as bad_coroutine finishes executing.

Speaking of bad_coroutine, the first thing it does is throw an exception and so we end up in the tconcurrent exception handler.

We save the caught exception, and now that bad_coroutine has ended we try to resume our event loop using its saved context (and, at the same time, we can unblock our main now that we have a result).

And this last resume is the point where we end up zeroing half our registers and jumping to address zero. If there was ever a time to suspect that memory used by some ~~horribly cursed~~ intricately woven assembly code is getting corrupted, this is it!

To complicate matters slightly, the exact content of a context object (that is, the serialized state of a suspended coroutine in memory) and the way it is saved is extremely platform-specific — as evidenced by the x86 registers in it — so I'm going to talk only about what is used in our particular case of i386 Windows with the MinGW compiler. This is the cryptic table shown below and extracted from a comment in Boost.Context's low-level assembly files.

  -----------------------------------------------------------------
  |   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |
  -----------------------------------------------------------------
  |   0h  |  04h  |  08h  |  0ch  |  010h |  014h |  018h |  01ch |
  -----------------------------------------------------------------
  | mxcsr | x87_cw|fi_strg|dealloc| limit |  base | fc_seh|  EDI  |
  -----------------------------------------------------------------
  -----------------------------------------------------------------
  |   8   |   9   |  10   |   11  |   12  |   13  |   14  |   15  |
  -----------------------------------------------------------------
  |  020h | 024h  | 028h  |  02ch |  030h |  034h |  038h |  03ch |
  -----------------------------------------------------------------
  |  ESI  |  EBX  |  EBP  |  EIP  |   to  |  data | EH NXT|EH HDLR|
  -----------------------------------------------------------------

What you see in each column of this table is a different piece of information that Boost.Context saves when a coroutine is suspended (or when its context is first created), but don't worry about what exactly each of those is for. What we're particularly interested in is the registers EDI through EIP, and if you remember the first time we looked at registers at the start of this article, you might be surprised that we're saving only a few of them in this table. This is a small trick of the ABI, callee-saved registers, that becomes important here.

Boost.Context uses a hand-written assembly routine to switch from one context to another, but to the compiler it looks like any other function, and like any other function on i386 Windows the compiler takes care to save some registers (on the stack) before calling the function, as mandated by the ABI. Also, like any other function, the compiler pushes the return address of the caller on the stack before doing the call, so that it knows where to return. This means that by the time we're in Boost.Context's assembly routine, a lot has already been saved onto the stack for us.

You might want to take some time to internalize this, because here comes the real magic: since each coroutine has its own stack, we know that as long as the coroutine is suspended anything we save on its stack is safe and won't be overwritten. So the assembly routine that suspends the coroutine can simply save the contents of the context right here on the stack while it's suspending, next to the return address and the callee-saved registers saved by the compiler!
And the context object that Boost.Context gives us is really just a pointer to this suspended coroutine's stack.
When it is time to resume Boost takes this pointer, restores the context's contents manually in assembly, uses it as its new stack pointer, and then "simply" returns from the magic assembly routine back to the calling code that originally suspended the coroutine — like any other function!

Okay, this is a lot to take in, so here's the takeaway — each coroutine has its own stack that no one touches while it's suspended, and saving the context on that stack is the last thing a suspended coroutine does. There's no obvious bug in the Boost code, and while the rest of the tconcurrent code does keep pointers to those stack contexts, the only times they are really used is when saving and resuming coroutines. So, if we don't have any particularly suspicious code that accesses the context, what could possibly be corrupting it and how do we find out?

To catch a corruption

Finally, we have a promising trail to follow, and a strong suspicion that the saved context we switch back to is being corrupted by something, since it somehow ends up filled with zeros when we try to resume it. The good news is that now that we have a precise target — the saved registers on the stack — there's one simple way to track whatever is causing this kerfuffle. Back to our debugger!

This time we're going to make use of a particularly awesome feature of x86 CPUs — hardware data breakpoints (these are sometimes known as watchpoints and emulated in software by single-stepping, which, unfortunately runs hundreds of times slower; the other side of that coin being that the hardware is limited to only four breakpoints at the same time).

The principle is as simple as it sounds. Instead of breaking on a line of code we're going to set a breakpoint on a location in memory and ask the debugger to break when the value in that location is modified.

Well, I didn't say this was going to be an easy bug — the data breakpoint takes us deep into the internal plumbing of the compiler runtime.

The actual corruption happens when some innocent-looking code saves a value on the stack — the completely wrong stack of our suspended event loop, overwritting our saved context! A buggy function or some invalid arguments causing the program to corrupt memory would have been perfectly normal and expected, but finding some internal runtime code merrily running on the wrong stack trampling all over our saved data is the special kind of fun one is really looking for in a debugging session.

The screenshot above is actually a little bit closer to familiar grounds, in a parent function of where the corruption usually happens (as the exact point depends on the chaotic behavior).

This is the function that'll send us to the next arcane topic everyone would really be better off never having to run into: _Unwind_SjLj_Resume.

SjLj and other four letter words

Have you ever wondered how exceptions are implemented by your compiler? No? Well, wonder no more. We're going to need a crash course in exception handling to understand what the code we landed into even is.

First of all, SjLj stands for Setjmp/Longjmp, an archaic way of implementing exception handling notorious for slowing programs in the common non-throwing path and based on an ancient standard C header that's otherwise best left alone.

There are of course better modern ways to implement exceptions, namely the Dwarf2 model which is common on Linux, or SEH-based exception handling which is a whole larger system integrated with Windows.

So why are we using SjLj, then? MinGW doesn't support SEH exceptions on i386 Windows, and as for Dwarf2 exceptions they come with their own explosive downsides since a thrown Dwarf2 exception will crash the program if it bubbles up into non-dwarf2 aware code (note though, this is actually OK in the case of our core library, since it currently exposes a C interface to bindings and doesn't leak exceptions). SjLj exceptions, apart from their age and ambient overhead, have a reputation for being a sane battle-tested default.

And what about this _Unwind_SjLj_Resume function we stopped in? This is where we run into the C++ exception handling ABI for Itanium. (Itanium is the experimental architecture that failed to become the x86_64 we use today, but did define most parts of our modern ABI).

While it makes for excellent bedside reading, I'm going to try and spare you as many of the boring details of the spec as possible. What we do need to know about this ABI is that it standardizes an interface for stack unwinding — the process of going up the stack and running exception handlers when a program throws an exception. One side of this interface is implemented by the compiler (in our case, with the SjLj method), and the other is the unwinding library described in the ABI. (And why isn't that unwinding library just an internal detail of the compiler? Because making it a separate language-independent facility enables C++ and other languages' exceptions to work together transparently, without needing every language to know about each other!)

There are ten functions provided by the unwinding library:

  _Unwind_RaiseException,
  _Unwind_Resume,
  _Unwind_DeleteException,
  _Unwind_GetGR,
  _Unwind_SetGR,
  _Unwind_GetIP,
  _Unwind_SetIP,
  _Unwind_GetRegionStart,
  _Unwind_GetLanguageSpecificData,
  _Unwind_ForcedUnwind

We don't need to understand the details of what each of these do — we can leave that to compiler writers — but we can see that _Unwind_Resume is defined here, which is, in fact, the same function as our _Unwind_SjLj_Resume. Now, according to the ABI document, this function is called after executing cleanup code in a partially unwound stack and at the end of a landing pad that performed cleanup, but did not resume normal execution. That's not very concrete, so let's look at what happens with the unwinding library when a program throws an exception.

A C++ throw statement is replaced by a call to __cxa_throw by the compiler, but the real work is delegated to the _Unwind_RaiseException function, which is called to start the stack unwinding process. This process works in two phases: the search phase walks up the stack through every stack frame to locate exception handlers, and if that first phase is successful the cleanup phase starts walking the stack frames again, but this time calling every exception handler found by the search phase on the way. (What happens if the search phase fails? That's an unhandled exception and the program terminates.)

So, to simplify, the cleanup phase is essentially responsible for running the destructor of every object that goes out of scope on the way to the nearest catch block. As far as the exception runtime knows, both destructors and catch blocks are exception handlers (what the spec calls landing pads), the main difference being that catch blocks stop the unwinding and resume normal code execution but, after a destructor is run, the frame's landing pad has to tell the runtime to continue going down the stack. And how does it do that? That's just what our _Unwind_Resume function does!

All right, so that doesn't quite tell us yet why _Unwind_Resume is running on the wrong stack, but at least we're starting to have some idea where we landed and what is happening in this function. In fact, we're getting really close! The last piece of the puzzle is going to come from the search phase.

Since the unwinding library is a generic language-independent algorithm, it knows how to walk up a stack but it doesn't actually know how to find any C++ catch blocks or destructors that need to be run. To do that, the compiler has to provide what's called a personality routine, which is going to be specific to the exception handling mechanism (and if you've ever had confusing link errors about __gxx_personality_v0, now you'll know that weird symbol is actually a personality routine, which has to match the exception-handling mechanism!).

Of course, everything I've described so far happens in the perfect abstract world of the Itanium specification, where our Sufficiently Smart Compilers provide modern, zero-cost, table-based exception handling. What makes those exception handling mechanisms so efficient is that they don't need to do anything at runtime as long as an exception isn't thrown. Instead, during the search phase the personality routine can get all the information it needs by looking up some tables embedded in the binary.

Back to SjLj, since it doesn't embed any tables the program has to re-create that same information at runtime, and compiler writers needed a way to shoehorn this old runtime system into the new world of fast inter-operable exceptions. They did that by introducing two new functions that the compiler would sprinkle throughout the code, to keep track of the entire context the search phase needs: _Unwind_SjLj_Register and _Unwind_SjLj_Unregister.

Denouement

And this is where everything starts to fall into place!

This SjLj context that needs to be maintained at runtime? That contains our stack pointer, saved by _Unwind_SjLj_Register at the start of every function that might need to run a destructor or catch an exception, and restored by the unwinder before any call to an exception handling landing-pad.

But our async code uses stackful coroutines! In other words, every coroutine has its own stack, and whenever tconcurrent switches coroutines the stack pointer is quietly updated by Boost Context's internal assembly routines — something that the compiler and the unwinder can't possibly know about.

So, we end up in a situation where _Unwind_SjLj_Register has saved a stack pointer and since then, we've switched to a completely different stack without updating SjLj's information. After a few coroutine switches, the chain of SjLj contexts that the Register/Unregister functions maintain has turned into a complete mess. (By contrast, other exception mechanisms don't need to save stack pointers or any other information at runtime, so they don't have an equivalent of the SjLj chain — nothing that needs updating.)

And finally, we can go back where we began: one of our Golang unit tests tries to exercise the error paths in our C++ library, resulting in a coroutine throwing an exception. The SjLj context chain — in complete disarray — causes destructors and other unwind code to run on the wrong stack during the cleanup phase, corrupting our saved context object for the tconcurrent event loop, which happens to fill it with zeros. Eventually tconcurrent's handler catches the exception — still on the wrong stack — and as it merrily resume normal execution, trying to switch back to the event loop, the program ends mysteriously with half its registers zeroed-out, flying off into the null address.

Still, some of you might be wondering, how did this ever work? How could the bug not trigger when the wrong kind of object is thrown, when some optimizations are disabled, or any other of a myriad of seemingly irrelevant details?

Fully explaining this is not necessary to understanding and fixing this bug, but I suspect the short answer is that this is largely due to the way the program is structured — all exceptions bubbling out of coroutines are caught in the same handler, which touches very little of the stack before switching back to the event loop's context and restoring everything saved inside. And so there is a large part of pure luck. If it just so happens that either the shuffled SjLj chain points to the right stack, or to the wrong stack but with enough buffer space to keep our saved event loop context out of harm's way, the bug will have trampled over only the stack of a terminating coroutine. If and when we manage to get back to the event loop, that coroutine's stack is never reused again, and the effect may just be invisible.

Conclusion

When faced with a bottomless rabbit-hole, jumping is rarely one's idea of a good time. In the worst of cases it can be tempting to settle for the first change that makes the problem disappear, to start reworking large chunks of code until the symptoms go away, or even to abandon entirely the feature causing all this trouble.

While it took the better part of a week to understand this bug, it's fair to say that we came away with a very simple diagnosis — the way our tconcurrent library uses Boost Context is incompatible with SjLj exceptions. And the cure is equally dull. We switched to Dwarf2 exceptions. But, having found the root cause, we can sleep soundly with the certainty that the bug is truly fixed, and having had the chance to learn a lot more than we bargained for!

No human is perfect, and no developer is either! As long as there is new code to write, new bugs will be written. But, at the end of the day, there is no substitute for understanding. So let's strive to learn from our errors, and make it our goal to ensure they only ever happen once.

Taking Mastodon's security to the next level - part 2: Exchange encrypted messages

Dimitri Merejkowsky — Tue, 19 Nov 2019 10:36:16 +0000

Introduction

This is the second article in a 2-part series of blog posts that describe our endeavor to add end-to-end encryption to Mastodon: if you haven’t already, please read Part 1: Encrypt your toots first.
In the rest of this article, we’ll refer to the Javascript code responsible for managing the UI as the client, and the Ruby on Rails code as the server.

We left on a bit of a cliffhanger - we’d managed to encrypt direct messages in the client, but hadn’t yet sent them to the server.

Actually, sending encrypted messages to the server instead of plain text messages will lead to all sorts of interesting challenges and we’ll learn even more about Mastodon’s internals than we did in the first post.

Adding an encrypted field in the database

Since we are encrypting only direct messages, it seems like a good idea to add an encrypted boolean in the database. That way, we’ll know whether statuses are encrypted or not before attempting to decrypt them.

So here’s the plan:

The client should send an encrypted boolean to the server when calling the api/v1/statuses route during the composition of direct messages
The server should store the encrypted status contents in the database, along with an encrypted boolean
The server should send the encrypted text along with the encrypted boolean back to the client.

Let’s write a new migration and migrate the db:

# db/migrate/20190913090225_add_encrypted_to_statuses.rb
class AddEncryptedToStatuses < ActiveRecord::Migration[5.2]
  def change
      add_column :statuses, :encrypted, :bool
  end
end

$ rails db:setup

Then fix the controller:

# app/controllers/api/v1/statuses_controller.rb
class Api::V1::StatusesController < Api::BaseController
  def create
    @status = PostStatusService.new.call(
                current_user.account,
                # ...
                encrypted: status_params[:encrypted])
  end

  def status_params
    params.permit(
       # ...
       :encrypted)
  end

end

Note that the controller deals only with validating the JSON request; the actual work of saving the statuses in the database is done by a service instead, so we need to patch this class as well:

# app/services/post_status_service.rb
class PostStatusService < BaseService
# ...
  def call(account, options = {})
    @encrypted = @options[:encrypted] || false
    # …
    process_status!


  end

  def process_status!
      ApplicationRecord.transaction do
      @status = @account.statuses.create!(status_attributes)
    end
  end


  def status_attributes
    # Map attributes to a list of kwargs suitable for create!
    {
       # …
       :encrypted: @encrypted
   }.compact
  end
end

Let’s write a test to make sure the PostStatus service properly persists encrypted messages:

# spec/services/post_status_service_spec.rb
it 'can create a new encrypted status' do
  account = Fabricate(:account)
  text = "test status update"
  status = subject.call(account, text: text, encrypted: true)
  expect(status).to be_persisted
  expect(status.text).to eq text
  expect(status.encrypted).to be_truthy
end

OK, it passes!

We can now use the new PostStatus API from the client code:

// app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
  let shouldEncrypt = getState().getIn(['compose', 'shouldEncrypt'], false);
  let status = getState().getIn(['compose', 'text'], '');

  if (shouldEncrypt) {
   status = await tankerService.encrypt(status);
  }

  api(getState).post('/api/v1/statuses', {
    //
    status,
    encrypted: shouldEncrypt
  });
}

We can check that this works by composing a direct message:

And then checking in the database:

rails db
# select encrypted, text from statuses order by id desc;
encrypted | text
----------+---------------------------------
 t        | A4qYtb2RBWs4vTvF8Z4fpEYy402IvfMZQqBckhOaC7DLHzw…

Looks like it’s working as expected, so it’s time to go the other way around - sending the encrypted boolean from the server to the client.

Displaying encrypted messages in the UI

This time we need to change the status serializer:

# app/serializers/rest/status_serializer.rb
class REST::StatusSerializer < ActiveModel::Serializer
  attributes :id, :created_at, :in_reply_to_id, :in_reply_to_account_id,
             # ...
             :encrypted
end

The Javascript code that fetches the status from the Rails API does not have to change.

That being said, we still want to make it clear in the UI whether the message is encrypted or not - this is useful for debugging.

So let’s update the StatusContent component to display a padlock icon next to any encrypted message:

// app/javascript/mastodon/components/status_content.js
render() {
  const encrypted = status.get('encrypted');
  let contentHtml;
  if (encrypted) {
    contentHtml = '<i class="fa fa-lock" aria-hidden="true"></i>&nbsp;' \
      + status.get('contentHtml');
  } else {
    contentHtml = status.get('contentHtml');
  }

  const content = { __html: contentHtml };
  return (
     // ...
     <div ...>
       dangerouslySetInnerHTML={content} 
     </div>
  );
}

Hooray, it works! We’re ready to call decrypt now.

Decrypt messages

First things first, let’s patch the TankerService to deal with decryption:

// app/javascript/mastodon/tanker/index.js
export default class TankerService {
  // ...

  decrypt = async (encryptedText) => {
    await this.lazyStart();

    const encryptedData = fromBase64(encryptedText);
    const clearText = await this.tanker.decrypt(encryptedData);
    return clearText;
  }
}

Now we’re faced with a choice. There are indeed several ways to decrypt statuses in the client code. For simplicity’s sake, we’ll patch the processStatus function which is called for each message returned from the server:

// app/javascript/mastodon/actions/importer/index.js
async function processStatus(status) {
  // …
  if (status.encrypted) {
    const { id, content } = status;

    // `content` as returned by the server has a <p> around it, so
    // clean that first
    const encryptedText = content.substring(3, content.length-4);
    const clearText = await tankerService.decrypt(encryptedText);
    const clearHtml = `<p>${clearText}</p>`
    dispatch(updateStatusContent(id, clearText, clearHtml));
  }

}

Note that we call an udpateStatusContent action to update the status after it has been decrypted.

I won’t go through the implementation of the updateStatusContent action and reducers as they’re pretty standard.

Anyway, we can check that our patch works by logging in as Alice, and then sending a message to ourselves:

Exchanging private messages

Being able to send encrypted messages to oneself is quite impressive, but I don’t think we should stop there :)

Let’s create a new account for Bob, and look at what happens when Alice sends a message containing @bob - this is known as a mention:

Normally, Bob should get a notification because he was sent a direct message, but this is not the case.

Clearly there is something to fix there.

After digging into the code, here's what I found out: notifications about direct messages are generated by a class named ProcessMentionsService.

Here’s the relevant part of the code:

class ProcessMentionsService < BaseService
  def call(status)
      status.text.gsub(Account::MENTION_RE) do |match|
         mentionned_account = ...
         # …
         mentions <<  \\
           mentionned_account.mentions(...).first_or_create(states)
       end

       mentions.each { create_notification(mention) }
  end
end

We can see that the server looks for @ mentions in the status text using regular expression matches and then builds a list of Mention instances.

Then something interesting happens:

# app/services/process_mentions_services.rb
class ProcessMentionsService < BaseService
   # …
   def create_notification(mention)
    mentioned_account = mention.account

    if mentioned_account.local?
      LocalNotificationWorker.perform_async(
        mentioned_account.id, 
        mention.id, 
        mention.class.name)
    elsif mentioned_account.activitypub?
       ActivityPub::DeliveryWorker.perform_async(
         activitypub_json, 
         mention.status.account_id, 
         mentioned_account.inbox_url)
     end
  end
end

So the server triggers a task from the LocalNotificationWorker if the mentioned account is local to the instance. It turns out this will later use the websocket server we discovered in Part 1 to send a notification to the client.

Side note here: if the mentioned account is not local to the instance, an Activity Pub delivery worker is involved. This is at the heart of the Mastodon mechanism: each instance can either send messages across local users, or they can use the ActivityPub protocol to send notifications across to another instance.

Back to the task at hand: it’s clear now that if the status is encrypted by the time it’s processed by the server, nothing will match and no notification will be created. That’s why Bob didn’t get any notification when we tried sending a direct message from Alice to Bob earlier.

Thus we need to process the @ mentions client-side, then send a list of mentions next to the encrypted status to the server:

//app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
// ...
  let mentionsSet = new Set();
  if (shouldEncrypt) {
    // Parse mentions from the status
    let regex = /@(\S+)/g;
    let match;
    while ((match = regex.exec(status)) !== null) {
      // We want the first group, without the leading '@'
      mentionsSet.add(match[1]);
    }

  const mentions = Array.from(mentionsSet);
  api(getState).post('/api/v1/statuses', {
    status,
    mentions,
    encrypted,
  });

}

As we did for the encrypted boolean, we have to allow the mentions key in the statuses controller and forward the mentions array to the PostStatus service:

class Api::v1::StatusesController < Api::BaseController
  def status_params
    params.permit(
      :status,
      # ...
      :encypted,
      mentions: [])
  end


  def create
    @status = PostStatusService.new.call(
                current_user.account,                                                         
                encrypted: status_param[:encrypted],
                mentions: status_params[:mentions])
end

In the PostStatus service we forward the mentions to the ProcessMentions service using a username key in an option hash:

# app/services/post_status_service.rb
class PostStatusService < BaseService
  def process_status!
    process_mentions_service.call(@status, { usernames: @mentions })
  end
end

And, finally, in the ProcessMentions service, we convert usernames into real accounts and create the appropriate mentions:

# app/services/process_mentions_service.rb
class ProcessMentionsService < BaseService
  def call(status, options = {})
    if @status.encrypted?
      usernames = options[:usernames] || []
      usernames.each do |username|
        account = Account.find_by!(username: username)
        mentions << Mention.create!(status: @status, account:account)
      end
   else
     # same code as before
   end
end

Now we can try encrypting the following status: @bob I have a secret message for you and check that Bob gets the notification.

But when Bob tries to decrypt Alice’s message, it fails with a resource ID not found error message: this is because Alice never told Tanker that Bob had access to the encrypted message.

For Bob to see a message encrypted by Alice, Alice must provide Bob’s public identity when encrypting the status. We still have some code to write, because in Part 1 we created and stored only private tanker identities. Luckily, the tanker-identity Ruby gem contains a get_public_identity function to convert private identities to public ones.

So the plan becomes:

Add a helper function to access public identities from rails
When rendering the initial-state from the server, add public identities to the serialized accounts.
In the client code, fetch public identities of the recipients of the encrypted statuses
Instead of calling encrypt with no options, call tanker.encrypt( resource, { shareWithUsers: identities }) where identities is an array of public identities

Good thing we are already parsing the @ mentions client-side :)

Sending public identities in the initial state

First we adapt our TankerIdentity class so we can convert a private identity to a public one:

# app/lib/tanker_identity.rb
def self.get_public_identity(private_identity)
  Tanker::Identity.get_public_identity(private_identity)
end

Then we add the tanker_public_identity attribute to the User class:

class User < ApplicationRecord
  def tanker_public_identity
    TankerIdentity::get_public_identity tanker_identity
  end
end

We tell the Account class to delegate the tanker_public_identity method to the inner user attribute.

# app/models/use.rb
class Account < ApplicationRecord
  delegate :email,
           :unconfirmed_email,
           :current_sign_in_ip,
           :current_sign_in_at,
           ...
           :tanker_public_identity,
           to: user,
           prefix: true
end

We adapt the account serializer:

# app/serializers/rest/account_serializer.rb
class REST::AccountSerializer < ActiveModel::Serializer 
   attributes :id, :username, 
              # ...:
              :tanker_public_identity


def tanker_public_identity
    return object.user_tanker_public_identity
end

And now the client can access the Tanker public identities of the mentioned accounts in the initial state.

Sharing encrypted messages

We can now collect the identities from the state and use them in the call to tanker.encrypt():

export function submitCompose(routerHistory) {
  // ...

  let identities = [];
  const knownAccounts = getState().getIn(['accounts']).toJS();
  for (const id in knownAccounts) {
    const account = knownAccounts[id];
    if (mentionsSet.has(account.username)) {
       identities.push(account.tanker_public_identity);
     }
   }

  // …
  const encryptedData = await tankerService.encrypt(
                                clearText, 
                                { shareWithUsers: identities });

  api(getState).post('/api/v1/statuses', {
  // ...
  });
}

Let’s see what happens after this code change. This time, when Bob clicks on the notification, he sees Alice's decrypted message:

Done!

What did we learn?

We discovered how notifications are handled in Mastodon
We found out that some server-side processing needed to be moved client-side, as is expected when client-side encryption is used.
We implemented a fully working end-to-end encryption feature for Mastodon’s direct messages, making sure direct message can be read only by the intended recipients

If you are curious, here are some statistics about the number of changes we had to write, excluding generated files:

$ git diff --stat \
   :(exclude)yarn.lock \
  :(exclude)Gemfile.lock \
  :(exclude)db/schema.rb
 41 files changed, 360 insertions(+), 40 deletions(-)

Future Work

Reminder: this is a proof of concept, and many things could be improved. Here’s a list of problems and hints about their solutions.

Improve status decryption

We are violating an implicit property of the messages in Mastodon: they are supposed to be immutable, as shown by the fact that until our patch, no action was able to change the contents of the statuses.

We probably would have to refactor the client code a bit to not violate this property, with the added benefit that the UI will no longer “flicker” when statuses go from encrypted base64 strings to clear text.

Improving the identity verification flow

We should remove the @tanker/verification-ui package and instead introduce tanker identity verification inside the existing authentication flow.

You can check out the Starting a Tanker session section of Tanker’s documentation for more details.

Provide alternative verification methods

You may have noticed that the identity verification currently works by having Tanker and Mastodon servers holding some secrets. Also, the email provider of the users can, in theory, intercept the email containing the verification code.

If this concerns you, please note that instead of using email-based verification, we could use another verification method called the verification key. You can read more about that in the Alternative verification methods section of the Tanker documentation.

Please do note that in this case, users are in charge of their verification key and will not be able to access any of their encrypted resources if they lose it.

We could implement both verification methods and let users choose between the two during onboarding.

Implement pre-registration sharing

The code assumes all users sending or receiving direct messages already have a Tanker identity registered. This can also be solved by using a Tanker feature called Pre-registration sharing.

Make encryption work across instances

Finally, our implementation works only if the sender and receiver of the direct messages are on the same instance - we need to make encryption work with the ActivityPub protocol.

I have a few ideas but fixing it seems non-trivial. Still, it would make for a pretty nice challenge :)

Conclusion

Thanks for reading this far. Writing the patch was a nice experience: Mastodon’s source code is clean and well-organized. You can browse the changes on the pull request on GitHub.

I hope this gave you an idea of the possibilities offered by Tanker. If you’d like to use Tanker in your own application, please get in touch with us.

Feel free to leave a comment below and give us your feedback!

Taking Mastodon security to the next level - part 1: Encrypt your toots

Dimitri Merejkowsky — Tue, 22 Oct 2019 14:13:38 +0000

What is this about?

My name is Dimitri Merejkowsky and I’ve been working at Tanker since June 2016. We’re a software company whose goal is to make end-to-end encryption simple. (More details on our website).

I’ve been an enthusiastic user of Mastodon since April 2017. One thing that always bugs me is that Mastodon administrators have access to everything about their users, as we'll see in a minute.

A few weeks ago, I decided to tackle this issue and try to encrypt Mastodon's direct messages with Tanker.

And that's how this series of articles was born. They’re written as something in between a tutorial and a story. You can use it to follow in my footsteps or to just enjoy the ride and have a good read: we'll discover what it actually means to implement Tanker in an existing solution and learn a few things about Mastodon’s internals. If you're curious, you can also jump to the end result on GitHub.

But first, let's go back to the problem that triggered the whole thing.

Introduction - What's wrong with Mastodon's direct messages?

Let's assume there is a Mastodon instance running with 3 users: Alice, Bob, and Eve.

First, Alice decides to send a direct message to Bob. She doesn't want her, or Bob’s, followers to see it, so she selects "Direct" visibility in the drop-down menu before sending her message:

Once the message is sent, she can see it the Direct messages column:

Bob, on the other hand, gets a notification and Alice's message appears in his column:

Finally, Eve does not get any notification, and if she tries to access the message directly using the permalink, she gets a 404 error:

At first glance, it looks as if the feature is working - only Bob can see Alice's message.

But, alas, the Mastodon admins can still read it because they have access to the database:

# select text from statuses order by id desc;
        text
-----------------
 @bob hello!

The aim of this series

In this series of articles, I would like to invite you to follow the steps I took to implement end-to-end encryption for direct messages on Mastodon. Note that I'm using Debian 10; your mileage may differ if you’re using a different distribution or another operating system.

When we're done, here's what we'll have:

Nothing will change from Alice's point of view when composing the direct message.

Bob will still see Alice's message, but this time there will be a lock to signify it’s encrypted:

And the admins will no longer be able to read all the messages.

# select encrypted, text from statuses order by id desc;
encrypted | text
----------+---------------------------------
 t        | A4qYtb2RBWs4vTvF8Z4fpEYy402IvfMZQqBckhOaC7DLHzw
 f        | @bob hello!

Sounds interesting? Let's dive in!

Getting started

We are going to make some changes in Mastodon's source code, so let's clone it and make sure we can run an instance on our development machine.

git clone git://github.com/tootsuite/mastodon
cd mastodon
# install all required libraries:
cat Aptfile | sudo apt install -y
# Install correct ruby version with rvm
rvm install ruby-2.6.1
# Install all ruby dependencies
bundle install
# Install all Javascript dependencies
yarn
# Run all processes with foreman
foreman start -f Procfile.dev

Now we can open the http://localhost:3000 URL in a browser and sign up our first user.

The "vanilla" Mastodon is running as expected, so we can start changing the code and see what happens :)

Calling encrypt() the naive way

In the API section of the Tanker documentation, we notice there’s an encrypt() function in a Tanker object. We also see a bit of code that tells us how to instantiate Tanker:

const config = { appId: 'your-app-id' };
const tanker = new Tanker(config);

We need an App ID, so let's create an application in the Tanker Dashboard and patch the front-end code directly, without thinking too much about the implications.

// In app/javascript/mastodon/actions/compose.js
export function submitCompose(routerHistory) {
  const config = { appId: 'our-app-id' };
  const tanker = new Tanker(config);
  let clearText = getState().getIn(['compose', 'text'], '');
  const encryptedData = await tanker.encrypt(clearText);
}

But then we get:

PreconditionFailed: Expected status READY but got STOPPED trying to encrypt.

After digging in the documentation, it turns out we need to start a session first.

If you’re wondering, here's why: Tanker implements an end-to-end protocol and thus encryption occurs on the users' devices. To that end, it uses an Encrypted Local Storage (containing some private keys, among other things) which can be accessed only when a Tanker session has been started.

The doc also says we need to verify users’ identities before starting a Tanker session, and that Tanker identities must be generated and stored on the application server - in our case, the Ruby on Rails code from the Mastodon project.

That means that we can’t do everything client-side in Javascript; we also need to modify the server as well as figuring out how these two communicate with each other.

Getting to know the architecture

The Mastodon development guide contains an overview of the Mastodon architecture. Here are the relevant parts:

A rails server is in charge of handling authentication (through the Devise gem and serving web pages
A Node.js WebSocket server is used for refreshing the user timeline, pushing notifications and the like
A React application using Redux to manage the state shows the main UI

To understand how the Ruby and the Javascript codes cooperate we can look at the HTML source of the page:

<!DOCTYPE html>
<html>
<head>
<!-- .. -->
<script id=”initial-state”, type=”application/json”>
{
  "meta":
  {
    "access_token": "....",
    "email": "alice@tanker.io",
    "me": "2"
    // ...
  },
}
</script>

That page is generated by Rails. The React app parses this HTML, extracts its initial state from the <script> element, and starts from there.
Note that the initial state contains a JSON object under the meta key.
The meta object contains (among other things):

An access token for the WebSocket server
The email of the current user
The ID of the current user in the database (under the me key)

So, here's the plan:

We'll generate a Tanker identity server-side
Put it inside the initial state
Fetch it from the initial state and start a Tanker session

Generating Tanker Identities

First, add the Tanker App Id and secret into the .env file:

(The Tanker app secret must not be checked in along with the rest of the source code):

TANKER_APP_ID = <the-app-id>
TANKER_APP_SECRET = <the-ap-secret>

Then we create a new file named app/lib/tanker_identity.rb containing this code:

module TankerIdentity
  def self.create(user_id)
    Tanker::Identity.create_identity(ENV["TANKER_APP_ID"], ENV["TANKER_APP_SECRET"], user_id.to_s)
  end
end

We adapt the User model:

# app/models/users.rb
class User < ApplicationRecord

  after_create :set_tanker_identity

  def set_tanker_identity
    self.tanker_identity = TankerIdentity.create_identity(self.id)
    self.update_attribute :tanker_identity, self.tanker_identity
  end

end

We write a migration and then migrate the DB:

# db/migrate/20190909112533_add_tanker_identities_to_users.rb
class AddTankerIdentitiesToUsers<ActiveRecord::Migration[5.2]
  def change
    add_column :users, :tanker_identity, :string
  end
end

$ rails db:setup

Finally, we write a new test for the AppSignUpService and run the tests:

# spec/services/app_sign_up_service_spec.rb
it 'creates a user with a Tanker identity' do
  access_token = subject.call(app, good_params)
  user = User.find_by(id: access_token.resource_owner_id)
  expect(user.tanker_identity).to_notbe_nil
end

$ rspec
...
Finished in 3 minutes 49.4 seconds (files took 8.56 seconds to load)
2417 examples, 0 failure

They pass! We now have Tanker identities generated server-side. Let's use them to start a Tanker session.

Starting a Tanker session

When starting a Tanker session you need to verify the identity of the user. This involves sending an email and entering an 8-digit code - that's how you can be sure that you’re sharing encrypted data with the correct user.

As a shortcut, Tanker provides a @tanker/verfication-ui package containing a ready-to-use UI to handle identity verification using emails.

It's used like this:

const config = { appId: "app id" };
const tanker = new Tanker(config);
const verificationUI = new VerificationUI({ tanker });
await verificationUI.start(email, identity);

We need the app ID, the Tanker identity and the email to start a Tanker session, so let's make sure they appear in the aforementioned <script> element:

# app/helpers/application_helper.rb
def render_initial_state
  state_params = {
    # ...
  }

  if user_signed_in?
    state_params[:tanker_identity] = current_account.user.tanker_identity
    # ...
  end
end

# app/presenters/initial_state_presenter.rb
class InitialStatePresenter < ActiveModelSerializers::Model
  attributes :settings, :push_subscription, :token,
             # ...
             :tanker_identity, :email, :tanker_app_id
end

# app/serializers/initial_state_serializer.rb
require_relative "../../lib/tanker"

class InitialStateSerializer < ActiveModel::Serializer
  attributes :meta, :compose, :accounts,

  # ...

  store[:tanker_identity] = object.current_account.user.tanker_identity
  store[:email]           = object.current_account.user.email
  store[:tanker_app_id]   = TANKER_APP_ID
end

Then, we fetch our values from the initial_state.js file:

// app/javascript/mastodon/initial_state.js
export const tankerIdentity = getMeta('tanker_identity');
export const email = getMeta('email');
export const tankerAppId = getMeta('tanker_app_id');

Creating a Tanker service

The challenge now becomes: how and when do we call verificationUI.start(), knowing that it will display a big pop-up and hide the rest of the UI?

After a bit of thinking, we decide to wrap calls to tanker.encrypt(), tanker.decrypt() and verificationUI.starte() in a TankerService class.

The TankerService class will be responsible for ensuring the tanker session is started right before data is encrypted or decrypted:

// app/javascript/mastodon/tanker/index.js
import { fromBase64, toBase64, Tanker } from '@tanker/client-browser';
import { VerificationUI } from '@tanker/verification-ui';


export default class TankerService {

  constructor({ email, tankerIdentity, tankerAppId }) {
    this.email = email;
    this.tankerIdentity = tankerIdentity;
    this.tanker = new Tanker({ appId: tankerAppId });
    this.verificationUI = new VerificationUI(this.tanker);
  }

  encrypt = async (clearText) => {
    await this.lazyStart();

    const encryptedData = await this.tanker.encrypt(clearText);
    const encryptedText = toBase64(encryptedData);
    return encryptedText;
  }

  decrypt = async (encryptedText) => {
    await this.lazyStart();

    const encryptedData = fromBase64(encryptedText);
    const clearText = await this.tanker.decrypt(encryptedData);
    return clearText;
  }

  stop = async() => {
    await this.tanker.stop();
  }

  lazyStart = async () => {
    if (this.tanker.status !== Tanker.statuses.STOPPED) {
      return;
    }

    if (!this.startPromise) {
      this.startPromise = this.verificationUI.start(this.email, this.tankerIdentity);
    }

    try {
      await this.startPromise;
      delete this.startPromise;
    } catch(e) {
      delete this.startPromise;
      throw e;
    }

  }

}

Next we configure Redux thunk middleware to take the TankerService as
extra argument:

// app/javascript/mastodon/store/configureStore.js
import thunkMiddleWare from 'redux-thunk';
import {
  email,
  tankerIdentity,
  tankerAppId,
} from '../initial_state';
import TankerService from '../tanker';


const tankerService = new TankerService({ email, tankerIdentity, tankerAppId });

const thunk = thunkMiddleWare.withExtraArgument({ tankerService });

export default function configureStore() {
  return createStore(appReducer, compose(applyMiddleware(
    thunk,
    // ...
  )));
}

After this change, the thunk middleware allows us to access the TankerService instance from any Redux action.

So, now we can adapt the submitCompose action properly:

// app/javascript/mastodon/actions/compose.js
export function submitCompose(routerHistory) {
  return async function (dispatch, getState, { tankerService }) {
    let visibility = getState().getIn(['compose', 'privacy']);

    const shouldEncrypt = (visibility === 'direct');

    if (shouldEncrypt) {
      const encryptedText = await tankerService.encrypt(status);
      console.log('about to send encrypted text', encryptedText);
    }

    dispatch(submitComposeRequest());

    api(getState).post('/api/v1/statuses', {
      // ...,
      visibility,
    });
  }
}

When we're done, we get those pop-ups showing us that the verification process worked:

And some logs indicating the status was indeed encrypted

Starting verification UI ...
Verification UI started
About to send  encrypted text: AxMXSEhEnboU732MUc4tqvOmJECocd+fy/lprQfpYGSggJ28

That's all for Part 1. We can now create and verify cryptographic identities of all users in our local instance, use them to start a Tanker session, and encrypt our direct messages.

But how will the server actually handle those encrypted messages?

Stay tuned for part 2!

Follow Tanker on dev.to or on twitter to be notified when the next part is published - and feel free to ask questions in the comments section below.

🚀 We've just launched FileKit on ProductHunt!

Dimitri Merejkowsky — Thu, 19 Sep 2019 12:18:52 +0000

Hello everyone,

Last week we told you about our latest product: an end-to-end encrypted file transfer toolkit for Javascript.

Today we're thrilled to announce the launch of FileKit on Product Hunt.

We wish to empower you, fellow developers, with the technology to build end-to-end encrypted applications.

FileKit comes with the following features:

Upload, download and share files up 2GB
Simple and modern API
Easy to integrate in your app
Builtin UI component for user’s identity verification

Here is an example of a file transfer application built with FileKit: https://share.tanker.io/

Go check it out and tell us what you think!

We created FileKit - The end-to-end encrypted file transfer toolkit for Javascript

Dimitri Merejkowsky — Wed, 11 Sep 2019 13:57:16 +0000

Nowadays, many applications use cloud storage providers such as Amazon S3 or Microsoft Azure. These services are very convenient; however, they have not yet cracked the data protection issue as the responsibility for data security is thrown right back at their clients. Here is a list of their security offerings and their shortcomings:

Encryption in transit: it is the “S” in https, and means that the data is encrypted while it is in transit, yet is vulnerable as soon as it arrives on the server.
Encryption at rest: this means that the cloud storage provider encrypts the data on the hard drives on their storage servers. Whether the keys used for encryption are stored on their storage service or on your servers, the threat model is largely the same. You must trust the infrastructure that runs your servers and, if it gets compromised, an attacker will be able to dump all the encrypted data and the key to decrypt them.

These two techniques protect the data: between the client and the server and between the server and the storage server, but the data is still vulnerable as it is unprotected on the server itself.

Trust is a personal choice; your users shouldn't have to trust you for your good technical choices. An ideal solution should be able to protect your app from data leaks such that if someone gets access to your server, they will not be able to decrypt your users' data. This solution would relieve you from these trust issues while keeping all of the advantages provided by cloud service providers.

End-to-end encryption is a solution where the data is encrypted on the client and the encryption keys stay on the client’s devices. With this solution, your servers don't see any clear user data. As such, they will not be able to leak that user data if they ever get compromised.

The main issue becomes that only the device that has encrypted the data is able to decrypt it. So you will need to exchange securely the encryption keys between devices to allow a user to access their data on all devices, and to allow a user to share their data with other users. This is non-trivial as we cannot expect users to handle their keys themselves.

After a few years of building solutions to help developers secure data exchanges in their apps, we at Tanker designed a new product solving an array of common use cases with cloud storage providers. This product implements end-to-end encryption and solves the key exchange problem to keep the user experience pleasant while enabling you to:

Securely share files with individuals both inside and outside your organization. Files can be read only by the intended recipient. Allow secure collaboration on files in critical domains: medical, legal, journalistic, contractual etc.
Meet regulatory and policy requirements. Enhance trust by guaranteeing your customers’ privacy.
Focus on your business logic. It is straightforward and easy to implement and you won’t have to handle the cryptography at all.

FileKit: What is it?

FileKit is a secure cloud storage service that handles all cryptography and key exchanges seamlessly, letting you focus on what's important.

FileKit comes with the following features:

Upload, download and share files up 2GB

Use end-to-end encryption
Easy to integrate in your app
Builtin UI component for user’s identity verification

Here is an example of a file transfer application built with FileKit: https://tankerhq.github.io/filekit-tuto-app/.

Get started with our file transfer tutorial, or have a look at the documentation.

PS: This article was originally written by Loïc Banet. It's been reproduced it here to give you a chance to see it in your notifications feed.

Reacting to Dropbox: another take on cross-platform C++ development

Théo DELRIEU — Wed, 04 Sep 2019 15:46:06 +0000

A few weeks ago, Dropbox announced in a blog post that they had decided to drop C++ from their codebase. This article sparked off a number of reactions, especially on the C++ subreddit.

I suggest that you read that first, but here is a TL;DR:

In 2013 Dropbox started to use C++ to share mobile code between iOS and Android, the goal being to avoid having to write it twice. 6 years later, they made a U-turn because of the overhead of:

Custom frameworks and libraries
A customized development environment
Differences between Mobile platforms
Finding and keeping experienced C++ developers

These are all valid points. At Tanker, we had to face similar challenges with our cross-platform end-to-end encryption SDK. However, we took a different approach to sharing C++ code between multiple platforms efficiently, including iOS and Android; this worked well for us.

I’ll share my thoughts about the Dropbox announcement and then present the solutions that we adopted.

Remaining obstacles

It is undeniable that having cross-platform C++ code is no easy feat, and implies several kinds of overhead.

Recruiting can be a real pain, especially when requiring expertise in C++, cross-building, and mobile development. Also, internal training is costly, and having trained people leaving a few months after they have been trained is a risk you might not want to take.

Writing and maintaining language-specific bindings is challenging, having N targeted platforms usually implies having N bindings, debugging C++ code through Java can be nightmarish, not to mention the complications of cross-building.

These issues existed when Dropbox started their C++ mobile journey in 2013, and still persist today.

An evolving environment

Fortunately, many other things have improved since. The number of active open-source Github projects has globally increased. The number of projects written in C++ has increased by a factor of 7.64 between 2013 and 2018:

I disagree with Dropbox when they claim that the easiest overhead to predict with C++ is the need to build frameworks and libraries.

I do not know how many C++ JSON libraries there were when they decided to write json11, but there are around 15 listed by isocpp at present. Any recent project should consider using one of those instead of hand-rolling one.

The C++ ecosystem has evolved greatly, and we are beginning to have a wide choice of open-source libraries to solve the usual development problems.

C++ cross-build support has also improved since 2013, when companies often had to write the tools themselves:

We needed a custom build system that created libraries containing C++ code as well as Java and Objective-C wrappers and could generate targets that were understood by both Xcodebuild and Gradle.

Nowadays, both Android NDK and Xcode ship with recent C++ compilers (especially the former). A tool like CMake can generate Xcode projects, and can be invoked directly from Gradle. There is still some boilerplate to write of course, but still less than creating a whole build system.

Cross-platform development at Tanker

As I mentioned before, Tanker builds an end-to-end encryption SDK on which our other products are based. There are two versions, the first in JavaScript, the second in C++. The latter is used in our iOS/Android SDKs, and also runs on Linux/Windows/macOS. From the start, we had the same goal as Dropbox: sharing as much C++ code between platforms as possible.

Having a small team of engineers, we cannot spend time reimplementing libraries (HTTP, cryptography, formatting, etc…) ourselves, unless we really have to. This is why we use and contribute as much as possible to open-source projects (e.g. JSON for Modern C++, fmt, sqlpp11). Relying on external projects comes with the daunting specter of dependency management, which is one of the major challenges in C++.

This is less true nowadays, thanks to the emergence of C++ package managers (Conan, Hunter and Vcpkg being the most famous). We chose Conan for various reasons, mostly for its flexibility. It has become a central element of our development environment by allowing us to make cross-building a very simple (you can watch my swampUP 2018 talk about this very subject), and providing a great way to build and package our libraries and SDK for multiple platforms. Using Conan also led us to help library authors making their projects packageable.

One key point of our architecture design is to allow building our SDK for several platforms, not only just for mobiles. This implied writing bindings in several programming languages. This is why we opted for having a thin C interface, wrapping our C++ API. C is a much simpler language that a lot of other languages can bind to natively. We keep this interface as simple as possible because writing bindings is still cumbersome. This has been described by Adi Shavit as The Salami Method

Thanks to that, we could easily write an iOS binding since Objective-C can directly call C code. Android still required some work but we used JNA to avoid writing JNI code ourselves. If there is a need for a Go, Rust, or C# binding in the future, it will be way easier to bind than if we had only C++.

In conclusion

Using various open-source libraries and managing them with Conan greatly helped us to bring our SDK to Linux, macOS, Windows, iOS and Android. Dropbox’s choices may have been justified at the time they built their product; however, nowadays, C++ seems to be a very viable way of sharing code across multiple platforms.

We plan on writing more articles to cover dependency management and cross-build processes using Conan and CI scripts at Tanker.

PS: This article was originally written by Théo Delrieu and published on Tanker’s Medium. As you might not be on Medium yourself, we've reproduced it here to give you a chance to see it in your notifications feed.

How safe are your cat pics?

Dimitri Merejkowsky — Fri, 01 Mar 2019 14:36:17 +0000

The Internet is like the wild west. Black hats are the bandits, white hats are the bounty hunters. Databases are the new banks, and data is the gold of this era. However, banks always had substantial and sturdy security around them, from vaults to guards and fences.

Internet apps are sometimes lacking in this area, as shown by the increasingly frequent data breaches we’re observing over the last few years.

Quantifying security

Let’s consider a simple application, SnapCat, which allows users to share their cat pics. SnapCat has one server, one database and a mobile application with a login form. SnapCat’s makers are obviously concerned about security and want to be sure that their users’ cat pics are safe.

How can SnapCat measure their app’s security levels?

A way to estimate a software environment’s security level is to measure its attack surface. This is usually done by a security expert, but the basic concepts are both easy enough and important to understand.

The first step in measuring the attack surface is to list all the attack vectors. An attack vector is a path an attacker could take to steal data from the environment.

SnapCat listed a few attack vectors:

Getting direct access to the database, by guessing admin credentials, or by using an SQL injection
Tricking the server into giving access to data by exploiting an API bug
Attacking the application directly by infecting the libraries it uses like what happened here
Having SnapCat’s offices infiltrated by an undercover agent working for their #1 competitor, DogPix

While obviously simplified, these attack vectors are plausible examples of what one could for a cloud-based app. A real-life attack vector analysis could list hundreds of them.

The next step is, for each attack vector, to evaluate the impact of a successful attack.

Getting access to the database (by direct access or SQL injection) has the most potential impact, as the attacker would get access to all the world’s cat pics in one go.
Exploiting an API bug has less impact, as it would require more time to extract the same amount of data. Infecting the app’s libraries would have a similar impact.
Lastly, an office infiltration from DogPix would have little impact as the data is not stored in the offices. Foolish dogs!

Measuring the attack surface

Next, we want to evaluate the difficulty (or rather the relative easiness) of each attack. This is done by taking into account the existing checks and counter-measures for each attack vector.

To do so, SnapCat’s security expert analyzed every attack vector. Here are the results:

Their database uses default credentials. This makes getting access to it very easy.
SQL injections and API bugs are more difficult to exploit, and the current development practices as SnapCat ensure a pretty good defense against these attacks.
Infecting dependencies to target a specific application is very hard.
Finally, infiltrating SnapCat’s offices would be very difficult for an undercover dog.

With this work done, we can graph SnapCat’s attack surface:

Reducing attack surface

SnapCat now has a clear view of the risks their application incurs and their respective impacts. It is time for action. The goal is to reduce the attack surface as much as possible.

There are two ways to reduce the attack surface for each attack vector: either make the attack more difficult or reduce its impact.

Most of the time, making attacks more difficult is the result of following good security practices . Reducing the impact of attacks can be done by reducing the amount of data stored, storing parts of the data in separate locations or encrypting it.

The first step for SnapCat is to change the database credentials and implement a better password policy to make guessing important credentials more difficult.

As we can see, this already reduces the “database access” attack surface quite a bit, but to drastically reduce the impact of every possible attack, SnapCat chooses to encrypt all cat pictures on their users’ devices, before they even reach the server. To do that, they start using Tanker.

This ensures the strongest data protection possible, making any database or server attack pretty much useless. Every single cat picture is independently encrypted with its own key, and only the sender and the recipient can decrypt it.

As a result, SnapCat’s updated attack surface now looks like this:

SnapCat’s team can now focus all of their energy on improving their awesome product. And maybe have a nap or two. 📦

Learn how you can integrate Tanker into your application at https://www.tanker.io.

PS: This article was originally written by Aloïs Jobard and published on Tanker’s Medium. As you might not be on Medium yourself, we've reproduced it here to give you a chance to see it in your notifications feed.

tsrc — Handling multiple git repositories without losing your HEAD

Dimitri Merejkowsky — Fri, 08 Feb 2019 11:11:46 +0000

Git has become the mainstream source code manager and is the de facto standard for many projects, both open and closed source.

However, it does not scale well with huge code bases commonly found in the enterprise world.

Some companies decide to put everything in one giant repository. By doing so, they often end up patching existing source control software as Facebook did with Mercurial.

Another way to do things is to keep using good old git: by splitting the sources across multiple repositories.

Managing complexity

In the "multi-repo" case, how do you keep track of all the repositories, and how do you synchronize changes among them?

One popular method is to elect a 'master' repository and manage the others ones through the use of git submodules. Unfortunately, it is not always possible to implement this master/slaves design. Furthermore, the ergonomics of git submodule are not always on par.

tsrc

Enter tsrc. We use it every day at Tanker to manage our many sources.

At its core lies the concept of a manifest. It's a simple YAML file that lists repositories by their relative paths in the workspace (src) and by URL:

repos:
  - src: foo
    url: git@gitlab.local:acme/foo

  - src: bar
    url: git@gitlab.local:acme/bar

That means all repositories are listed in one place.

If you have multiple teams, you may want to use groups so that each team can select which subset of the repositories they need. Each group has a name and a list of repositories:


groups:
  first_group: 
    repos: [a, b, c]
  second_group:
    repos: [d, e]

This approach is known to work with at least 300 different git repositories :)

Usage

To use tsrc, all you have to do is put the manifest itself in a git repository, and then use the following
commands to create a new workspace:

$ mkdir ~/work
$ cd work
$ tsrc init git@gitlab.local:acme/manifest.git

You can select one or several groups with the -g/--group option.

Making sure all the repositories are up to date

Once the workspace is created, you can update all the repositories at once by using tsrc sync.

The manifest itself is updated first.
If a new repository has been added to the manifest, it is cloned.
Lastly, the other repositories are updated.

Note that tsrc sync only updates the repositories if the changes are trivial. tsrc will not touch any repository if:

The repository is dirty
The local branch has diverged from upstream
There is no remote tracking branch

It is up to you to solve these issues any way you see fit. This way, there is no risk of data loss or sudden conflicts to appear. tsrc focuses on doing simple tasks right and never tries to do smart things.

Lastly, so that you know where manual intervention is required, tsrc sync also
displays a summary of errors at the end:

$ tsrc sync
...
* (2/2) spam/eggs
=> Fetching origin
=> Updating branch
fatal: Not possible to fast-forward, aborting.
Error: Failed to synchronize workspace
* spam/eggs: updating branch failed

Managing merge requests

As a bonus, tsrc is also able to automate review workflows, whether you are using GitHub or GitLab.

More info about all of this in the documentation.

Conclusion

We hope you'll give tsrc a try and that you'll find it a handy tool for your own projects.

tsrc development happens on GitHub.

Feel free to contribute, open issues, and/or give us feedback in the comment section.

PS: tsrc is made with love by the same talented people working at tanker.io. We provide an open-source privacy solution that's as user-friendly as possible. Sign up and take it for a spin here: https://tanker.io/.

10 best practices to protect your users’ data (and why they’re still not sufficient)

Dimitri Merejkowsky — Tue, 05 Feb 2019 10:03:28 +0000

Over the last ten years, data breaches have become both more damaging and frequent. Massive leaks regularly make the headlines and hackers target businesses of every size, in every field. As former FBI Director Robert Mueller said, “There are only two types of companies: those that have been hacked, and those that will be”.

Data source: https://informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/

If you own any kind of online business, you’re probably collecting user data, which is valuable to both your company and your users. In its most simple form, it can only be an email address and a password, and even such basic data can have a significant impact if leaked or stolen.

However, hackers don’t stop at passwords: credit card numbers, personal information, email addresses, business data, everything has some value, and the business impacts of such breaches can be catastrophic. To protect your users’ data, and therefore your business, here are the ten security best practices that you should follow.

Make data theft more difficult

1- Use a firewall and a VPN to protect company data

Perhaps the most obvious of security practices: protect your internal network from external access. Set up a firewall to protect access to your data and a VPN to secure remote access.

2- Have an elaborate password policy

Up to 80% of people reuse their passwords, use personal passwords for work or use very poor passwords, with “123456” and “password” still being the most used passwords in 2018¹. Having an elaborate internal password policy is critical in preventing unwanted access.

Use a password manager² and enable two-factor authentication³ wherever possible. All passwords should also periodically be changed.

3- Backup your data regularly to avoid ransomware attacks

The latest trend in cyber criminality is to get access to your business data, encrypt it and extort money from your company to get that data back. This is called a ransomware attack⁴, and it can easily be mitigated by having frequently updated backups. These backups should be encrypted and stored in protected locations.

4- Build a cybersecurity culture

With remote work and BYOD practices becoming more prevalent, more stress than ever is put on employees’ security awareness. It’s crucial to educate all your employees on security risks and issues and have well-documented security policies for them to follow.

Phishing or social engineering are common methods used by hackers to gain access to data or information. Your employees should know how to detect and react to them.

Secure your website or application

5- Use relevant development practices

Releasing a bugged website or app containing a security flaw can lead to pretty disastrous attacks: cross-site scripting, SQL injections, account theft⁵…

Using relevant development practices can reduce the risk of having such vulnerabilities. You should make sure all code pushed to production is reviewed and tested. Dependencies should be kept up to date and checked for vulnerabilities. Use tools⁶ to automatically detect potential vulnerabilities.

6- Perform third-party security audits

While it remains necessary to check and test your app’s code yourself, you should not only rely on your team to ensure your app is secure. Hire an external security company to perform security audits of your code and infrastructure on a regular basis (at least once a year).

Limit the impact of unauthorized data access

7- Don’t store plain text passwords

Storing user passwords is not an easy task. Too many companies store passwords as is, or use weak/out of date hashing algorithms. Plain text passwords are gold for any hacker, and would severely hinder your users’ trust.

Hash any password you store using a secure hash algorithm such as Argon2⁷, or use a third party authentication provider.

8- Manage employees’ permissions

While it might be tempting to grant all access to every employee, it creates a gaping hole in your security. Allowing employees to access sensitive data they don’t necessarily need increases the risk of both insider threats and external hackers.

Employees should be granted access only to information and resources that are necessary for accomplishing their jobs. This is called the principle of least privilege⁸.

9- Monitor network and actions

If someone is stealing your data, you should be able to detect it. You should monitor your network traffic and set up automatic alerts.

User action monitoring solutions are designed to record every action taken by your employees and immediately detect and investigate suspicious user activity.

10- Use at-rest encryption

Any data you collect should be stored encrypted. Any cloud storage provider should have an option to automatically encrypt all data. Check that it’s turned on. If you have your own database, use a Key Management Service⁹ to secure all your data.

This will prevent any hacker who obtained this data from exploiting it without also gaining access to the master key (which should, hopefully, be extremely difficult).

Go further

If you have implemented all of the above, congratulations! You are among the best in class on data security and privacy.

However, all these countermeasures only make breaches and leaks more difficult to pull off. Insider jobs and elaborate hacks are still possible, and an attacker gaining access to the right admin account could still siphon all user data stored in your database.

All hope is not lost though, as there is a way to guarantee your users’ security and privacy: give them back the ownership of their data. By using end-to-end encryption directly on each user’s device to secure data, you can guarantee that no one can access it except its rightful owner. This is in-app privacy.

At Tanker, we’ve spent the last two years creating an open-source privacy solution that integrates into your web or mobile app and secures user data at the source.

You can sign up for free and try it here: https://tanker.io.

Encrypting your users’ data is no longer optional

Dimitri Merejkowsky — Tue, 22 Jan 2019 10:41:22 +0000

While balancing privacy and innovation is one of today’s most critical issues in technology development, let’s identify data encryption’s opportunities.

Most of today’s digital businesses hold data on behalf of their users. These assets are valuable to both your business and your users, which means they are excellent targets for hackers. Holding onto their data has risks that can hardly be ignored. While Bill Gates said in his review of 2018 that one of the most critical areas of technology development in 2019 is the balance between privacy and innovation, it’s certainly the right time to ask yourself “how to protect my users’ data?”, especially if you work for a tech company or leading a digital project. With your users’ trust comes great responsibility, especially when more frequent data breaches have damaging consequences on companies’ brand and revenues.

The looming data breaches, their damaging effects on users’ trust and companies’ revenues.

The main risk you face while holding data is qualified by the term “data breach”. It consists of a partial or total leak of the data from your servers, or the cloud, to individuals harboring evil intentions. Over the last years these attacks have become more frequent and many companies, such as Yahoo, eBay, or Sony, have experienced critical leaks.

Still, what are the odds of being targeted by such attacks?

As Cisco’s Chief Security Officer John Stewart said, a data breach is bound to happen: “You’re eventually going to be hit, it is not worth the effort of thinking you won’t be hit.” and “The bottom line is that unfortunately, no organization is immune to a data breach in this day and age.”

Let’s keep in mind that your users’ data is a keepsake that embodies the trust relationship between you and them.

By breaking this trust, your organization sends a terrible message: you don’t consider your users’ well being, and you leave the door open for a competitor that will. Branding takes a long time to build, but a second to destroy.

Data breaches’ consequences are disastrous for any business. This can lead to situations where clients seek an alternative to your product, do not subscribe, or even leave.

Let’s make it short: losing control of customer’s data means losing customers.

And there is more and more data to prove it. On January 2018, Cisco’s Privacy Maturity Benchmark study revealed that 65% of organizations reported sales cycle delays due to clients’ data privacy concerns, with an average of 7.8 weeks delay, and a correlation between the organization’s privacy maturity level and the sales delay duration.

Cisco 2018 Privacy Maturity Benchmark Study.

Above branding and revenues issues, legislation has evolved to take privacy into account. With the GDPR’s implementation within the European Union and AB 375 in California, data breaches now have direct applicable legal consequences and financial sanctions. If your organization treats users’ personal data on the EU territory, the fine framework can be up to 20M€, or in the case of an undertaking, up to 4 % of your total global turnover of the preceding fiscal year, whichever is higher.

So now that you’re aware of Data Breaches’ damaging consequences, let’s explore the best users’ data protection options.

Actually, there are two options: securing the data access or securing the data itself. In 2019, If you want to protect your users’ data, there are two options: securing the data access or securing the data itself.

Securing the data access has its limits: unavoidable human mistakes.

Let’s review the steps necessary to secure the data access to your application together.

First, your code has to follow your organization’s security guidelines. It then has to go through a thorough code review and finally be audited by a third party. As long as there are no human mistakes, these processes guarantee you a secure code base.

Except that your software is most probably relying on a handful of external software dependencies, services, and hardware. Sadly, many of them are not built and configured with the same level of security expectations as your organization. This last statement is especially true for storage services that can easily have unsafe or confusing default configurations. Even the NSA experienced such an accident with the leak of Red Disk, over an Amazon S3 public bucket.

Moreover, even after all these steps, you have to be prepared for emergencies with a fast established process to find and deploy the fixes for security issues.

Altogether human mistakes are impossible to avoid and relying on the security of multiple components in an application means that errors are more likely to happen as each additional piece of code, dependency or service to secure adds a layer of uncertainty.

This is where encryption comes up.

Over the last decades, encryption has become one of the most popular and effective methods to secure end-users’ data privacy.

When it comes to secure the data itself, vote for end-to-end encryption.

Data encryption is defined as data’s transformation into another form, or code. When data is encrypted, only people with access to a secret key or password can read it.

In other words, encrypting the data ensures only authorized users can read it. The data is stored encrypted, and any leak from the storage only leaks the encrypted data. To get access to the clear data, one needs access to both the encrypted data and its associated encryption keys. In this regard, data encryption is a strong barrier to data breaches.

Among the multiple solutions to implement data encryption, let’s evaluate the most common ones: encryption at rest and end-to-end encryption.

In an application, many stakeholders can have access to data, primarily some or all of the employees that have access to the application server and the storage.

With encryption at rest, your application servers typically encrypt data before sending it to storage (e.g., a database server, or service in the cloud) and decrypt it after retrieving it from storage. This means any data leak occurring at storage level is not an issue anymore (e.g., an improperly secured database or a database backup made publicly available by mistake). However, all your users’ data are still accessible by the application servers themselves, some or all of your employees, and thus are vulnerable to attacks.

To send a confidential medical result to Patient Bob, Dr. Alice is using a fictive telemedicine app that adopts encryption at rest. The issue here is that Patient Bob’s data is still accessible in the app server and KMS.

In the event of an attack targeting your application servers, clear data becomes accessible at once. Popular services like AWS KMS are widely used to implement encryption in this fashion.

End-to-end encryption neutralizes servers’ breaches, but it’s not very user-friendly.

In the end-to-end encryption mode, the clear data is not accessible by the application’s employees, but only by the appropriate users. The data is transmitted as encrypted through the multiple layers of the application. The application servers and the storage only handle the encrypted data, and only the relevant user(s) can decrypt it. An attack on the application server leaks nothing, so that’s why you have to try attacking users one by one.

Here, Dr. Alice is using the same fictive telemedicine app, but here the app has embedded end-to-end encryption. Patient Bob’s medical data is only accessible at users’ device, neutralizing apps’ and servers’ attack.

As ideal as it sounds, end-to-end encryption adoption is very slow. The main hurdles are the difficulty in developing and setting up such technology, but also the impact on your application’s user workflows.

For example, common impediments on the user experience of PGP are that users have to manually save and retrieve a key to own several devices or that it is only possible to share to a group of users by encrypting again for every individual in the group.

Obviously, from a user experience standpoint, these issues are not acceptable, but we’ll develop this point in an upcoming blog post :)

The balance between privacy and user experience is an incredible challenge, and it’s the one we, at Tanker, chose to tackle. We strive to provide an open-source privacy solution as user-friendly as possible. Sign up and try it here: https://tanker.io/

PS: This article was originally written by Loïc Banet and published on Tanker’s Medium. As you might not be on Medium yourself, we've reproduced it here to give you a chance to see it in your notifications feed.