Introduction
This is the second article in a 2-part series of blog posts that describe our endeavor to add end-to-end encryption to Mastodon: if you haven’t already, please read Part 1: Encrypt your toots first.
In the rest of this article, we’ll refer to the Javascript code responsible for managing the UI as the client, and the Ruby on Rails code as the server.
We left on a bit of a cliffhanger - we’d managed to encrypt direct messages in the client, but hadn’t yet sent them to the server.
Actually, sending encrypted messages to the server instead of plain text messages will lead to all sorts of interesting challenges and we’ll learn even more about Mastodon’s internals than we did in the first post.
Adding an encrypted field in the database
Since we are encrypting only direct messages, it seems like a good idea to add an encrypted
boolean in the database. That way, we’ll know whether statuses are encrypted or not before attempting to decrypt them.
So here’s the plan:
- The client should send an encrypted boolean to the server when calling the
api/v1/statuses
route during the composition of direct messages - The server should store the encrypted status contents in the database, along with an
encrypted
boolean - The server should send the encrypted text along with the
encrypted
boolean back to the client.
Let’s write a new migration and migrate the db:
# db/migrate/20190913090225_add_encrypted_to_statuses.rb
class AddEncryptedToStatuses < ActiveRecord::Migration[5.2]
def change
add_column :statuses, :encrypted, :bool
end
end
$ rails db:setup
Then fix the controller:
# app/controllers/api/v1/statuses_controller.rb
class Api::V1::StatusesController < Api::BaseController
def create
@status = PostStatusService.new.call(
current_user.account,
# ...
encrypted: status_params[:encrypted])
end
def status_params
params.permit(
# ...
:encrypted)
end
end
Note that the controller deals only with validating the JSON request; the actual work of saving the statuses in the database is done by a service instead, so we need to patch this class as well:
# app/services/post_status_service.rb
class PostStatusService < BaseService
# ...
def call(account, options = {})
@encrypted = @options[:encrypted] || false
# …
process_status!
end
def process_status!
ApplicationRecord.transaction do
@status = @account.statuses.create!(status_attributes)
end
end
def status_attributes
# Map attributes to a list of kwargs suitable for create!
{
# …
:encrypted: @encrypted
}.compact
end
end
Let’s write a test to make sure the PostStatus
service properly persists encrypted messages:
# spec/services/post_status_service_spec.rb
it 'can create a new encrypted status' do
account = Fabricate(:account)
text = "test status update"
status = subject.call(account, text: text, encrypted: true)
expect(status).to be_persisted
expect(status.text).to eq text
expect(status.encrypted).to be_truthy
end
OK, it passes!
We can now use the new PostStatus API from the client code:
// app/javascript/mastodon/actions/compose.js
export function submitCompose(routerHistory) {
let shouldEncrypt = getState().getIn(['compose', 'shouldEncrypt'], false);
let status = getState().getIn(['compose', 'text'], '');
if (shouldEncrypt) {
status = await tankerService.encrypt(status);
}
api(getState).post('/api/v1/statuses', {
//
status,
encrypted: shouldEncrypt
});
}
We can check that this works by composing a direct message:
And then checking in the database:
rails db
# select encrypted, text from statuses order by id desc;
encrypted | text
----------+---------------------------------
t | A4qYtb2RBWs4vTvF8Z4fpEYy402IvfMZQqBckhOaC7DLHzw…
Looks like it’s working as expected, so it’s time to go the other way around - sending the encrypted boolean from the server to the client.
Displaying encrypted messages in the UI
This time we need to change the status serializer:
# app/serializers/rest/status_serializer.rb
class REST::StatusSerializer < ActiveModel::Serializer
attributes :id, :created_at, :in_reply_to_id, :in_reply_to_account_id,
# ...
:encrypted
end
The Javascript code that fetches the status from the Rails API does not have to change.
That being said, we still want to make it clear in the UI whether the message is encrypted or not - this is useful for debugging.
So let’s update the StatusContent
component to display a padlock icon next to any encrypted message:
// app/javascript/mastodon/components/status_content.js
render() {
const encrypted = status.get('encrypted');
let contentHtml;
if (encrypted) {
contentHtml = '<i class="fa fa-lock" aria-hidden="true"></i> ' \
+ status.get('contentHtml');
} else {
contentHtml = status.get('contentHtml');
}
const content = { __html: contentHtml };
return (
// ...
<div ...>
dangerouslySetInnerHTML={content}
</div>
);
}
Hooray, it works! We’re ready to call decrypt
now.
Decrypt messages
First things first, let’s patch the TankerService
to deal with decryption:
// app/javascript/mastodon/tanker/index.js
export default class TankerService {
// ...
decrypt = async (encryptedText) => {
await this.lazyStart();
const encryptedData = fromBase64(encryptedText);
const clearText = await this.tanker.decrypt(encryptedData);
return clearText;
}
}
Now we’re faced with a choice. There are indeed several ways to decrypt statuses in the client code. For simplicity’s sake, we’ll patch the processStatus
function which is called for each message returned from the server:
// app/javascript/mastodon/actions/importer/index.js
async function processStatus(status) {
// …
if (status.encrypted) {
const { id, content } = status;
// `content` as returned by the server has a <p> around it, so
// clean that first
const encryptedText = content.substring(3, content.length-4);
const clearText = await tankerService.decrypt(encryptedText);
const clearHtml = `<p>${clearText}</p>`
dispatch(updateStatusContent(id, clearText, clearHtml));
}
}
Note that we call an udpateStatusContent
action to update the status after it has been decrypted.
I won’t go through the implementation of the updateStatusContent
action and reducers as they’re pretty standard.
Anyway, we can check that our patch works by logging in as Alice, and then sending a message to ourselves:
Exchanging private messages
Being able to send encrypted messages to oneself is quite impressive, but I don’t think we should stop there :)
Let’s create a new account for Bob, and look at what happens when Alice sends a message containing @bob
- this is known as a mention:
Normally, Bob should get a notification because he was sent a direct message, but this is not the case.
Clearly there is something to fix there.
After digging into the code, here's what I found out: notifications about direct messages are generated by a class named ProcessMentionsService
.
Here’s the relevant part of the code:
class ProcessMentionsService < BaseService
def call(status)
status.text.gsub(Account::MENTION_RE) do |match|
mentionned_account = ...
# …
mentions << \\
mentionned_account.mentions(...).first_or_create(states)
end
mentions.each { create_notification(mention) }
end
end
We can see that the server looks for @
mentions in the status text using regular expression matches and then builds a list of Mention instances.
Then something interesting happens:
# app/services/process_mentions_services.rb
class ProcessMentionsService < BaseService
# …
def create_notification(mention)
mentioned_account = mention.account
if mentioned_account.local?
LocalNotificationWorker.perform_async(
mentioned_account.id,
mention.id,
mention.class.name)
elsif mentioned_account.activitypub?
ActivityPub::DeliveryWorker.perform_async(
activitypub_json,
mention.status.account_id,
mentioned_account.inbox_url)
end
end
end
So the server triggers a task from the LocalNotificationWorker
if the mentioned account is local to the instance. It turns out this will later use the websocket server we discovered in Part 1 to send a notification to the client.
Side note here: if the mentioned account is not local to the instance, an Activity Pub delivery worker is involved. This is at the heart of the Mastodon mechanism: each instance can either send messages across local users, or they can use the ActivityPub protocol to send notifications across to another instance.
Back to the task at hand: it’s clear now that if the status is encrypted by the time it’s processed by the server, nothing will match and no notification will be created. That’s why Bob didn’t get any notification when we tried sending a direct message from Alice to Bob earlier.
Thus we need to process the @
mentions client-side, then send a list of mentions next to the encrypted status to the server:
//app/javascript/mastodon/actions/compose.js
export function submitCompose(routerHistory) {
// ...
let mentionsSet = new Set();
if (shouldEncrypt) {
// Parse mentions from the status
let regex = /@(\S+)/g;
let match;
while ((match = regex.exec(status)) !== null) {
// We want the first group, without the leading '@'
mentionsSet.add(match[1]);
}
const mentions = Array.from(mentionsSet);
api(getState).post('/api/v1/statuses', {
status,
mentions,
encrypted,
});
}
As we did for the encrypted
boolean, we have to allow the mentions
key in the statuses controller and forward the mentions
array to the PostStatus
service:
class Api::v1::StatusesController < Api::BaseController
def status_params
params.permit(
:status,
# ...
:encypted,
mentions: [])
end
def create
@status = PostStatusService.new.call(
current_user.account,
encrypted: status_param[:encrypted],
mentions: status_params[:mentions])
end
In the PostStatus
service we forward the mentions to the ProcessMentions
service using a username
key in an option hash:
# app/services/post_status_service.rb
class PostStatusService < BaseService
def process_status!
process_mentions_service.call(@status, { usernames: @mentions })
end
end
And, finally, in the ProcessMentions
service, we convert usernames into real accounts and create the appropriate mentions:
# app/services/process_mentions_service.rb
class ProcessMentionsService < BaseService
def call(status, options = {})
if @status.encrypted?
usernames = options[:usernames] || []
usernames.each do |username|
account = Account.find_by!(username: username)
mentions << Mention.create!(status: @status, account:account)
end
else
# same code as before
end
end
Now we can try encrypting the following status: @bob I have a secret message for you
and check that Bob gets the notification.
But when Bob tries to decrypt Alice’s message, it fails with a resource ID not found
error message: this is because Alice never told Tanker that Bob had access to the encrypted message.
For Bob to see a message encrypted by Alice, Alice must provide Bob’s public identity when encrypting the status. We still have some code to write, because in Part 1 we created and stored only private tanker identities. Luckily, the tanker-identity
Ruby gem contains a get_public_identity
function to convert private identities to public ones.
So the plan becomes:
- Add a helper function to access public identities from rails
- When rendering the initial-state from the server, add public identities to the serialized accounts.
- In the client code, fetch public identities of the recipients of the encrypted statuses
- Instead of calling
encrypt
with no options, calltanker.encrypt( resource, { shareWithUsers: identities })
whereidentities
is an array of public identities
Good thing we are already parsing the @
mentions client-side :)
Sending public identities in the initial state
First we adapt our TankerIdentity
class so we can convert a private identity to a public one:
# app/lib/tanker_identity.rb
def self.get_public_identity(private_identity)
Tanker::Identity.get_public_identity(private_identity)
end
Then we add the tanker_public_identity
attribute to the User
class:
class User < ApplicationRecord
def tanker_public_identity
TankerIdentity::get_public_identity tanker_identity
end
end
We tell the Account
class to delegate the tanker_public_identity
method to the inner user
attribute.
# app/models/use.rb
class Account < ApplicationRecord
delegate :email,
:unconfirmed_email,
:current_sign_in_ip,
:current_sign_in_at,
...
:tanker_public_identity,
to: user,
prefix: true
end
We adapt the account serializer:
# app/serializers/rest/account_serializer.rb
class REST::AccountSerializer < ActiveModel::Serializer
attributes :id, :username,
# ...:
:tanker_public_identity
def tanker_public_identity
return object.user_tanker_public_identity
end
And now the client can access the Tanker public identities of the mentioned accounts in the initial state.
Sharing encrypted messages
We can now collect the identities from the state and use them in the call to tanker.encrypt()
:
export function submitCompose(routerHistory) {
// ...
let identities = [];
const knownAccounts = getState().getIn(['accounts']).toJS();
for (const id in knownAccounts) {
const account = knownAccounts[id];
if (mentionsSet.has(account.username)) {
identities.push(account.tanker_public_identity);
}
}
// …
const encryptedData = await tankerService.encrypt(
clearText,
{ shareWithUsers: identities });
api(getState).post('/api/v1/statuses', {
// ...
});
}
Let’s see what happens after this code change. This time, when Bob clicks on the notification, he sees Alice's decrypted message:
Done!
What did we learn?
- We discovered how notifications are handled in Mastodon
- We found out that some server-side processing needed to be moved client-side, as is expected when client-side encryption is used.
- We implemented a fully working end-to-end encryption feature for Mastodon’s direct messages, making sure direct message can be read only by the intended recipients
If you are curious, here are some statistics about the number of changes we had to write, excluding generated files:
$ git diff --stat \
:(exclude)yarn.lock \
:(exclude)Gemfile.lock \
:(exclude)db/schema.rb
41 files changed, 360 insertions(+), 40 deletions(-)
Future Work
Reminder: this is a proof of concept, and many things could be improved. Here’s a list of problems and hints about their solutions.
Improve status decryption
We are violating an implicit property of the messages in Mastodon: they are supposed to be immutable, as shown by the fact that until our patch, no action was able to change the contents of the statuses.
We probably would have to refactor the client code a bit to not violate this property, with the added benefit that the UI will no longer “flicker” when statuses go from encrypted base64 strings to clear text.
Improving the identity verification flow
We should remove the @tanker/verification-ui
package and instead introduce tanker identity verification inside the existing authentication flow.
You can check out the Starting a Tanker session section of Tanker’s documentation for more details.
Provide alternative verification methods
You may have noticed that the identity verification currently works by having Tanker and Mastodon servers holding some secrets. Also, the email provider of the users can, in theory, intercept the email containing the verification code.
If this concerns you, please note that instead of using email-based verification, we could use another verification method called the verification key. You can read more about that in the Alternative verification methods section of the Tanker documentation.
Please do note that in this case, users are in charge of their verification key and will not be able to access any of their encrypted resources if they lose it.
We could implement both verification methods and let users choose between the two during onboarding.
Implement pre-registration sharing
The code assumes all users sending or receiving direct messages already have a Tanker identity registered. This can also be solved by using a Tanker feature called Pre-registration sharing.
Make encryption work across instances
Finally, our implementation works only if the sender and receiver of the direct messages are on the same instance - we need to make encryption work with the ActivityPub protocol.
I have a few ideas but fixing it seems non-trivial. Still, it would make for a pretty nice challenge :)
Conclusion
Thanks for reading this far. Writing the patch was a nice experience: Mastodon’s source code is clean and well-organized. You can browse the changes on the pull request on GitHub.
I hope this gave you an idea of the possibilities offered by Tanker. If you’d like to use Tanker in your own application, please get in touch with us.
Feel free to leave a comment below and give us your feedback!
Top comments (0)