Taking Mastodon's security to the next level - part 2: Exchange encrypted messages

Introduction

This is the second article in a 2-part series of blog posts that describe our endeavor to add end-to-end encryption to Mastodon: if you haven’t already, please read Part 1: Encrypt your toots first.
In the rest of this article, we’ll refer to the Javascript code responsible for managing the UI as the client, and the Ruby on Rails code as the server.

We left on a bit of a cliffhanger - we’d managed to encrypt direct messages in the client, but hadn’t yet sent them to the server.

Actually, sending encrypted messages to the server instead of plain text messages will lead to all sorts of interesting challenges and we’ll learn even more about Mastodon’s internals than we did in the first post.

Adding an encrypted field in the database

Since we are encrypting only direct messages, it seems like a good idea to add an encrypted boolean in the database. That way, we’ll know whether statuses are encrypted or not before attempting to decrypt them.

So here’s the plan:

• The client should send an encrypted boolean to the server when calling the api/v1/statuses route during the composition of direct messages
• The server should store the encrypted status contents in the database, along with an encrypted boolean
• The server should send the encrypted text along with the encrypted boolean back to the client.

Let’s write a new migration and migrate the db:

# db/migrate/20190913090225_add_encrypted_to_statuses.rb
class AddEncryptedToStatuses < ActiveRecord::Migration[5.2]
  def change
      add_column :statuses, :encrypted, :bool
  end
end

$ rails db:setup

Then fix the controller:

# app/controllers/api/v1/statuses_controller.rb
class Api::V1::StatusesController < Api::BaseController
  def create
    @status = PostStatusService.new.call(
                current_user.account,
                # ...
                encrypted: status_params[:encrypted])
  end

  def status_params
    params.permit(
       # ...
       :encrypted)
  end

end

Note that the controller deals only with validating the JSON request; the actual work of saving the statuses in the database is done by a service instead, so we need to patch this class as well:

# app/services/post_status_service.rb
class PostStatusService < BaseService
# ...
  def call(account, options = {})
    @encrypted = @options[:encrypted] || false
    # …
    process_status!


  end

  def process_status!
      ApplicationRecord.transaction do
      @status = @account.statuses.create!(status_attributes)
    end
  end


  def status_attributes
    # Map attributes to a list of kwargs suitable for create!
    {
       # …
       :encrypted: @encrypted
   }.compact
  end
end

Let’s write a test to make sure the PostStatus service properly persists encrypted messages:

# spec/services/post_status_service_spec.rb
it 'can create a new encrypted status' do
  account = Fabricate(:account)
  text = "test status update"
  status = subject.call(account, text: text, encrypted: true)
  expect(status).to be_persisted
  expect(status.text).to eq text
  expect(status.encrypted).to be_truthy
end

OK, it passes!

We can now use the new PostStatus API from the client code:

// app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
  let shouldEncrypt = getState().getIn(['compose', 'shouldEncrypt'], false);
  let status = getState().getIn(['compose', 'text'], '');

  if (shouldEncrypt) {
   status = await tankerService.encrypt(status);
  }

  api(getState).post('/api/v1/statuses', {
    //
    status,
    encrypted: shouldEncrypt
  });
}

We can check that this works by composing a direct message:

And then checking in the database:

rails db
# select encrypted, text from statuses order by id desc;
encrypted | text
----------+---------------------------------
 t        | A4qYtb2RBWs4vTvF8Z4fpEYy402IvfMZQqBckhOaC7DLHzw…

Looks like it’s working as expected, so it’s time to go the other way around - sending the encrypted boolean from the server to the client.

Displaying encrypted messages in the UI

This time we need to change the status serializer:

# app/serializers/rest/status_serializer.rb
class REST::StatusSerializer < ActiveModel::Serializer
  attributes :id, :created_at, :in_reply_to_id, :in_reply_to_account_id,
             # ...
             :encrypted
end

The Javascript code that fetches the status from the Rails API does not have to change.

That being said, we still want to make it clear in the UI whether the message is encrypted or not - this is useful for debugging.

So let’s update the StatusContent component to display a padlock icon next to any encrypted message:

// app/javascript/mastodon/components/status_content.js
render() {
  const encrypted = status.get('encrypted');
  let contentHtml;
  if (encrypted) {
    contentHtml = '<i class="fa fa-lock" aria-hidden="true"></i>&nbsp;' \
      + status.get('contentHtml');
  } else {
    contentHtml = status.get('contentHtml');
  }

  const content = { __html: contentHtml };
  return (
     // ...
     <div ...>
       dangerouslySetInnerHTML={content} 
     </div>
  );
}

Hooray, it works! We’re ready to call decrypt now.

Decrypt messages

First things first, let’s patch the TankerService to deal with decryption:

// app/javascript/mastodon/tanker/index.js
export default class TankerService {
  // ...

  decrypt = async (encryptedText) => {
    await this.lazyStart();

    const encryptedData = fromBase64(encryptedText);
    const clearText = await this.tanker.decrypt(encryptedData);
    return clearText;
  }
}

Now we’re faced with a choice. There are indeed several ways to decrypt statuses in the client code. For simplicity’s sake, we’ll patch the processStatus function which is called for each message returned from the server:

// app/javascript/mastodon/actions/importer/index.js
async function processStatus(status) {
  // …
  if (status.encrypted) {
    const { id, content } = status;

    // `content` as returned by the server has a <p> around it, so
    // clean that first
    const encryptedText = content.substring(3, content.length-4);
    const clearText = await tankerService.decrypt(encryptedText);
    const clearHtml = `<p>${clearText}</p>`
    dispatch(updateStatusContent(id, clearText, clearHtml));
  }

}

Note that we call an udpateStatusContent action to update the status after it has been decrypted.

I won’t go through the implementation of the updateStatusContent action and reducers as they’re pretty standard.

Anyway, we can check that our patch works by logging in as Alice, and then sending a message to ourselves:

Exchanging private messages

Being able to send encrypted messages to oneself is quite impressive, but I don’t think we should stop there :)

Let’s create a new account for Bob, and look at what happens when Alice sends a message containing @bob - this is known as a mention:

Normally, Bob should get a notification because he was sent a direct message, but this is not the case.

Clearly there is something to fix there.

After digging into the code, here's what I found out: notifications about direct messages are generated by a class named ProcessMentionsService.

Here’s the relevant part of the code:

class ProcessMentionsService < BaseService
  def call(status)
      status.text.gsub(Account::MENTION_RE) do |match|
         mentionned_account = ...
         # …
         mentions <<  \\
           mentionned_account.mentions(...).first_or_create(states)
       end

       mentions.each { create_notification(mention) }
  end
end

We can see that the server looks for @ mentions in the status text using regular expression matches and then builds a list of Mention instances.

Then something interesting happens:

# app/services/process_mentions_services.rb
class ProcessMentionsService < BaseService
   # …
   def create_notification(mention)
    mentioned_account = mention.account

    if mentioned_account.local?
      LocalNotificationWorker.perform_async(
        mentioned_account.id, 
        mention.id, 
        mention.class.name)
    elsif mentioned_account.activitypub?
       ActivityPub::DeliveryWorker.perform_async(
         activitypub_json, 
         mention.status.account_id, 
         mentioned_account.inbox_url)
     end
  end
end

So the server triggers a task from the LocalNotificationWorker if the mentioned account is local to the instance. It turns out this will later use the websocket server we discovered in Part 1 to send a notification to the client.

Side note here: if the mentioned account is not local to the instance, an Activity Pub delivery worker is involved. This is at the heart of the Mastodon mechanism: each instance can either send messages across local users, or they can use the ActivityPub protocol to send notifications across to another instance.

Back to the task at hand: it’s clear now that if the status is encrypted by the time it’s processed by the server, nothing will match and no notification will be created. That’s why Bob didn’t get any notification when we tried sending a direct message from Alice to Bob earlier.

Thus we need to process the @ mentions client-side, then send a list of mentions next to the encrypted status to the server:

//app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
// ...
  let mentionsSet = new Set();
  if (shouldEncrypt) {
    // Parse mentions from the status
    let regex = /@(\S+)/g;
    let match;
    while ((match = regex.exec(status)) !== null) {
      // We want the first group, without the leading '@'
      mentionsSet.add(match[1]);
    }

  const mentions = Array.from(mentionsSet);
  api(getState).post('/api/v1/statuses', {
    status,
    mentions,
    encrypted,
  });

}

As we did for the encrypted boolean, we have to allow the mentions key in the statuses controller and forward the mentions array to the PostStatus service:

class Api::v1::StatusesController < Api::BaseController
  def status_params
    params.permit(
      :status,
      # ...
      :encypted,
      mentions: [])
  end


  def create
    @status = PostStatusService.new.call(
                current_user.account,                                                         
                encrypted: status_param[:encrypted],
                mentions: status_params[:mentions])
end

In the PostStatus service we forward the mentions to the ProcessMentions service using a username key in an option hash:

# app/services/post_status_service.rb
class PostStatusService < BaseService
  def process_status!
    process_mentions_service.call(@status, { usernames: @mentions })
  end
end

And, finally, in the ProcessMentions service, we convert usernames into real accounts and create the appropriate mentions:

# app/services/process_mentions_service.rb
class ProcessMentionsService < BaseService
  def call(status, options = {})
    if @status.encrypted?
      usernames = options[:usernames] || []
      usernames.each do |username|
        account = Account.find_by!(username: username)
        mentions << Mention.create!(status: @status, account:account)
      end
   else
     # same code as before
   end
end

Now we can try encrypting the following status: @bob I have a secret message for you and check that Bob gets the notification.

But when Bob tries to decrypt Alice’s message, it fails with a resource ID not found error message: this is because Alice never told Tanker that Bob had access to the encrypted message.

For Bob to see a message encrypted by Alice, Alice must provide Bob’s public identity when encrypting the status. We still have some code to write, because in Part 1 we created and stored only private tanker identities. Luckily, the tanker-identity Ruby gem contains a get_public_identity function to convert private identities to public ones.

So the plan becomes:

• Add a helper function to access public identities from rails
• When rendering the initial-state from the server, add public identities to the serialized accounts.
• In the client code, fetch public identities of the recipients of the encrypted statuses
• Instead of calling encrypt with no options, call tanker.encrypt( resource, { shareWithUsers: identities }) where identities is an array of public identities

Good thing we are already parsing the @ mentions client-side :)

Sending public identities in the initial state

First we adapt our TankerIdentity class so we can convert a private identity to a public one:

# app/lib/tanker_identity.rb
def self.get_public_identity(private_identity)
  Tanker::Identity.get_public_identity(private_identity)
end

Then we add the tanker_public_identity attribute to the User class:

class User < ApplicationRecord
  def tanker_public_identity
    TankerIdentity::get_public_identity tanker_identity
  end
end

We tell the Account class to delegate the tanker_public_identity method to the inner user attribute.

# app/models/use.rb
class Account < ApplicationRecord
  delegate :email,
           :unconfirmed_email,
           :current_sign_in_ip,
           :current_sign_in_at,
           ...
           :tanker_public_identity,
           to: user,
           prefix: true
end

We adapt the account serializer:

# app/serializers/rest/account_serializer.rb
class REST::AccountSerializer < ActiveModel::Serializer 
   attributes :id, :username, 
              # ...:
              :tanker_public_identity


def tanker_public_identity
    return object.user_tanker_public_identity
end

And now the client can access the Tanker public identities of the mentioned accounts in the initial state.

Sharing encrypted messages

We can now collect the identities from the state and use them in the call to tanker.encrypt():

export function submitCompose(routerHistory) {
  // ...

  let identities = [];
  const knownAccounts = getState().getIn(['accounts']).toJS();
  for (const id in knownAccounts) {
    const account = knownAccounts[id];
    if (mentionsSet.has(account.username)) {
       identities.push(account.tanker_public_identity);
     }
   }

  // …
  const encryptedData = await tankerService.encrypt(
                                clearText, 
                                { shareWithUsers: identities });

  api(getState).post('/api/v1/statuses', {
  // ...
  });
}

Let’s see what happens after this code change. This time, when Bob clicks on the notification, he sees Alice's decrypted message:

Done!

What did we learn?

• We discovered how notifications are handled in Mastodon
• We found out that some server-side processing needed to be moved client-side, as is expected when client-side encryption is used.
• We implemented a fully working end-to-end encryption feature for Mastodon’s direct messages, making sure direct message can be read only by the intended recipients

If you are curious, here are some statistics about the number of changes we had to write, excluding generated files:

$ git diff --stat \
   :(exclude)yarn.lock \
  :(exclude)Gemfile.lock \
  :(exclude)db/schema.rb
 41 files changed, 360 insertions(+), 40 deletions(-)

Future Work

Reminder: this is a proof of concept, and many things could be improved. Here’s a list of problems and hints about their solutions.

Improve status decryption

We are violating an implicit property of the messages in Mastodon: they are supposed to be immutable, as shown by the fact that until our patch, no action was able to change the contents of the statuses.

We probably would have to refactor the client code a bit to not violate this property, with the added benefit that the UI will no longer “flicker” when statuses go from encrypted base64 strings to clear text.

Improving the identity verification flow

We should remove the @tanker/verification-ui package and instead introduce tanker identity verification inside the existing authentication flow.

You can check out the Starting a Tanker session section of Tanker’s documentation for more details.

Provide alternative verification methods

You may have noticed that the identity verification currently works by having Tanker and Mastodon servers holding some secrets. Also, the email provider of the users can, in theory, intercept the email containing the verification code.

If this concerns you, please note that instead of using email-based verification, we could use another verification method called the verification key. You can read more about that in the Alternative verification methods section of the Tanker documentation.

Please do note that in this case, users are in charge of their verification key and will not be able to access any of their encrypted resources if they lose it.

We could implement both verification methods and let users choose between the two during onboarding.

Implement pre-registration sharing

The code assumes all users sending or receiving direct messages already have a Tanker identity registered. This can also be solved by using a Tanker feature called Pre-registration sharing.

Make encryption work across instances

Finally, our implementation works only if the sender and receiver of the direct messages are on the same instance - we need to make encryption work with the ActivityPub protocol.

I have a few ideas but fixing it seems non-trivial. Still, it would make for a pretty nice challenge :)

Conclusion

Thanks for reading this far. Writing the patch was a nice experience: Mastodon’s source code is clean and well-organized. You can browse the changes on the pull request on GitHub.

I hope this gave you an idea of the possibilities offered by Tanker. If you’d like to use Tanker in your own application, please get in touch with us.

Feel free to leave a comment below and give us your feedback!