DEV Community: Sergi Canal

Signing XRPL transactions with AWS KMS

Sergi Canal — Wed, 07 Sep 2022 12:32:45 +0000

When developing blockchain projects, how you handle the private keys is one of the most important things you need to think about. Losing a private key means losing all the control over the account, and even if the breach is solved, the account can not be used again.

While developing a project using XRPL, we researched on the best ways of handling the private keys for a server. Ideally we would be able to use some kind of hardware module that signs transactions for us, while the private key never leaves the module. While no such solution exists for the specific use case of signing blockchain transactions, KMS can handle secp256k1 elliptic curve signatures, which is exactly the curve used by XRPL (and most blockchains). KMS is one of the most popular Key management solutions in the market, and it meets enterprise-level security certifications, meaning we can be sure we are using the maximum level of security to protect our keys.

The AWS credentials used to programatically call AWS still need to be properly secured, as anybody with access to them could call AWS and generate valid signed transactions signed with your private key. The advantage however is that upon detection of the attack, the credentials can be invalidated, and after re-securing the server, the same account could still be used, or even if the credentials are eliminated before the attacker has time to sign a transaction with them, the funds are completely secure. To minimise the potential impact of a breach on the server, it is a good idea to implement a hot / cold wallet system similar to what the exchanges use. The hot wallet being the one which has credentials stored in the server. In this case only the funds in the hot wallet at the time of the breach would be compromised, while the private key and the account would remain safe for resuming operations after the problem is solved.

The issue is KMS is not designed with the purpose of blockchain in mind. It basically gives us access to two useful functions: One that returns the public key, and one that signs a given byte array with the private key and returns the result.

While this is enough to sign a transaction, the process of generating the specific data to be signed, and generating a valid transaction in the exact format Ripple expects it, is not so straight forward as it could seem.

Luckily for us, this problem has been solved for the Ethereum blockchain, which gives us very useful code to be adapted to XRPL. The way the signature is generated, however, is different in some details. While we will explain everything needed in this article, it is still interesting to read the explanations in the Ethereum article, as it provides some good explanations and it can be interesting to learn the differences between blockchain protocols. It is recommended to have a basic understanding of elliptic curve cryptography to fully understand this article and the code. Here is a video explaining the basics:

1. Set up a key on KMS

The first thing we need is to set up a key in AWS KMS. We need to use a ECC_SECG_P256K1 asymmetric key, which will allow us to generate ECDSA secp256k1 signatures.

You will then need to generate an access key id and secret with access to the KMS calls needed. These are the credentials that need to be well protected, as anyone with access to them can call KMS to sign data with your keys.

2. Getting the XRPL account address

KMS gives us access to the public key (never to the private key). From the public key it is possible to get the address, to be able to receive transactions, however it is not as straightforward as it is for Ethereum. KMS returns the public key in uncompressed form, which is what Ethereum uses, but Ripple uses the public keys in compressed form. In Elliptic curve cryptography, a public key represents a point in 2 dimensions, which satisfies the curve equation. The uncompressed form of a public key is expressed by a starting “04” byte (RFC 5480). Because of the properties of elliptic curves, at a certain x axis, only 2 values for y will satisfy the equation for the curve, since elliptic curves are symmetric on the x axis. One of them will be odd and the other will be even. The compressed form of the public key consists of a first byte “02” or “03” depending on if the y is odd or even, and the x axis, which generates a public key half as long. To get the compressed form of the public key, we need to parse the uncompressed form from the AWS response, and then convert it to the compressed form, to be turned into the address.

Finding the address then is easier, as we can make use of the xrpl library, which exports deriveAddress(publicKey).

3. Generating and signing a transaction

First, we will define a transaction that we wish to sign:

With elliptic curve cryptography, what is signed is a hash of the data we want to sign. In our case, the hash used by Ripple is sha256. We first use an xrpl function which will generate the hex payload to be signed from the transaction, and then we hash this payload which will give us the hash to be signed. Note that it is important to convert the hex payload into a Buffer before hashing, or the signature generated will not be valid. We then can send the last 32 characters of the hash to KMS to be signed.

Once we get back the signature from KMS, we are still not done, as a particular peculiarity of EC signatures makes it so you can find 2 different valid signatures by changing the sign of S. This could allow for replay attacks by sending the same transaction with the inverted signature. The solution most blockchains have implemented consists on only accepting signatures with an S value below the middle of the curve, so only one of the two signatures will be accepted by the network. KMS does not handle this for us, so we need to check and invert the value of S if needed.

We then convert the R and S values back into DER format, which is the signature format expected by Ripple.

If all the steps have been followed exactly, we should have a valid Ripple signature for our transaction. And we only need to add it to the transaction object, and encode it into a transaction payload to be broadcasted to the blockchain.

4. Broadcasting the transaction

The only step left is to broadcast the transaction to the network. If everything is correct the transaction will go through and we will have successfully broadcasted a transaction to the Ripple network that was signed on a KMS hardware module. After a successful execution of our test script, we should see something similar to this:

Note that we first need to fund at least 10XRP to the account before it can broadcast transactions, otherwise an error will be thrown.

Code

I have created an npm package with all the code needed to sign xrpl transactions on kms, and published it as xrpl-kms. The code and examples are in the following repository:

https://github.com/Peersyst/aws-kms-xrp-signing

Cryptography for programmers 4: Public key crypto and Protocols

Sergi Canal — Thu, 14 Jan 2021 15:33:17 +0000

It has been a while since I wrote the third post of the series. It turns out that the first post of this series was so successful that the dev team included me in one of their weekly must-read lists, and they sent me a sticker pack as a christmas gift for being a distinguished 2020 author! I received the gift today and it reminded me that I never got to finish the series, so forgive me for the delay if you have been waiting for this post.

I feel like this last article in the series is probably the most useful one because of the topics that it covers. However, to be able to understand it, a knowledge of the already covered topics is needed. So for anyone that is finding this article and has not read the previous ones, I would advice doing that before reading this one.

Key exchanges

So far in the series we have covered symmetric systems. Meaning that two parties need to agree on a shared secret key before starting the communication.

Chances are that you have never in your life agreed on a secret key with somebody to be able to communicate privately with them. And yet, you probably use private messaging applications every day. How is that possible?

The answer is that internet applications handle the key exchange for you. But how is it possible to share a key without first establishing a secure encrypted communication? Does it not make the encryption useless if you first share the key publicly unencrypted on the internet before starting the encrypted communication? The answer to this is yes, if the key was shared unencrypted. But it is possible to share a key between two parties without revealing the key to a middle observer. The first discovered, and most used algorithm to do this is Diffie-Hellman, and it relies on both parties having a public key that anybody can know, and a related private key, that can not be obtained from the public key. Diffie-Hellman relies on modular arithmetic maths, and it would be beyond the scope of this series to explain them, but here is a link to a video explaining how it works in an intuitive way, for those that are interested:

There is one issue with Diffie-Hellman however. If a middleman can not only read the key exchange messages, but also can modify them, then it is possible to manipulate the exchange in a way where the middleman is able to read all the encrypted messages sent after. For this reason, it is important before we start the key exchange to verify that the public key we are doing the exchange with, really belongs to who we want to communicate with. We will explore some ways of doing this when we talk about protocols.

Public key Cryptography

We should also mention Public key cryptography algorithms such as RSA, which allow sending messages without needing to share a key. They use a public-private key pair and mathematics similar to those of Diffie-Hellman.

In Asymmetric cryptography, the sender uses the receiver's public key for encrypting, and the receiver uses their private key for decrypting. In reality, this is mostly used for the purpose of exchanging keys and establishing symmetric key communications, because symmetric algorithms are much more efficient for a computer than executing asymmetric key maths.

RSA and similar algorithms have another very useful use case, which is asymmetric signing. If the sender sends a message encrypted with their private key, then a receiver could decrypt it with the public key. This is not safe encryption, since anybody can have the public key and decrypt the message. However, only someone that has the private key could generate a message that correctly decrypts with the public key, verifying the identity of the sender. This is the asymmetric alternative to symmetric signing algorithms such as HMAC, and is used very frequently in most protocols for the purpose of verifying the identity of parties during key exchanges.

In the bad practices list linked in the first post are included 3 rules related exclusively to RSA:

Don't use a short key (<2048 bits)

When generating a keypair we are given the option of selecting the size of the key. It is recommended to use keys 2048 bits or longer.

Don't use the textbook (raw) algorithm

The RSA algorithm is actually pretty simple, the hardest part being the key generation. As explained in the basic rules on the first article however, we should never program our own crypto and use libraries whenever possible.

Don't use PKCS#1 v1.5 padding

If we are using an updated library, we should use the default settings for algorithms when we are not sure of what we are doing, since they will usually be safe. This particular version of padding allows for an attack on RSA and should not be used.

Cryptographical protocols

In this section we will finally put it all together and see how cryptography is most commonly used in the real world. And how it is currently being used by your browser right now as you are reading this article.

The majority of cryptographical protocols follow the same structure.

Trust and parameter negotiation

The first step in a protocol is the most important, and is the main difference between protocols. Here the two parties in the communication need to verify their identity. In the case of a client/server scenario such as in TLS, only the server needs to verify their identity. Once the identity of the parties has been verified they will both have each other's public key which they can use in the next step.

In this initial contact there can also be parameter negotiation, where both parties agree on which algorithms and parameters they will use for each of the steps. Most protocols can use different versions and different algorithms for each of the steps. The set of algorithms chosen is referred to as a cipher suite, and it commonly contains:

a key exchange algorithm (such as Diffie-Hellman)
a symmetric block cipher for communication (such as AES-256-CBC)
a symmetric signing algorithm for message integrity and authentication (such as HMAC-sha256)

It is important to choose safe parameters and to use the latest versions of protocols. In the case of setting up a server, it is important to configure it so that it will not accept unsafe parameters from a client. Unsafe parameters meaning algorithms which we have seen in previous articles that are considered deprecated, or have important vulnerabilities.

Key exchange

Now that the identity of the parties has been verified they can use their public key to execute a key exchange with the negotiated algorithm. Common key exchange algorithms are Diffie-Hellman or RSA.

Communication

Now a secret key is shared between the two parties and they can freely communicate using a symmetric algorithm. Usually this is a session key, and is renegotiated each time a new communication session starts. The reason why block / symmetric cryptography is used instead of directly using the public keys to communicate with something like RSA is because the encryption/decryption is much more efficient with block ciphers.

Along with the encrypted message a signature is sent using the symmetric key to verify the integrity and authenticity of the message. In t he case of using a block mode such as GCM, the auth tag is included in the encryption, so no separate authentication algorithm is required.

Protocol examples

Let's see some examples of some of the most used protocols and the different trust models they have.

SSH

ssh is commonly used for connecting remotely to servers. SSH establishes an encrypted connection with a server, where we can access and execute commands remotely from our own comupter.

The way in which the client verifies their identity varies. It can be setup so that the access is granted with a simple password, or (safer) it can be set up so that it will grant access to a whitelist of public keys. In this case the client verifies their identity by signing a message with their private key.

The identity of the server should also be verified however. The reason is that someone could enter the communication and pretend to be the server, and receive all our commands pretending to be the server we are trying to connect with. If the access is set up with a password they could also capture our sent password and use it to connect to the real server.

The way in which we verify the server's identity is manually. Meaning ssh has no automatic identity verification. If you have ever connected to a server with ssh you have most likely seen this message (and chances are you ignored it):

In this message ssh is warning us that the public key of the server is not verified, and gives us a fingerprint of the ECDSA key. The fingerprint is a hash of the public key, which is easier to verify, since public keys are usually pretty long. What we are supposed to do in this situation would be to contact a trusted party that is able to confirm that this is the correct fingerprint. This could be achieved for example by calling the system administrator and checking with them. Once we confirm that the fingerprint is correct, we can continue, and it will be stored in our computer in a known_hosts file, so that this step will only need to be performed once for each server (or whenever the server changes their public key).

By typing yes without verifying the fingerprint we are placing a bet on the possibility no one is performing an attack on us. In some cases this is ok, but for important servers such as production servers, it is worth it to perform the verification. Your system administrator will be very happy that someone cares, and it will reflect positively on you 😄.

TLS

TLS, also commonly referred to as SSL (which is the previous deprecated version to TLS), is used to encrypt web traffic. It is the protocol used for protecting https webpages among other things. TLS is arguably the most important cryptographical protocol. Your device is using it right now to receive this article encrypted to your browser from dev.to.

It would not make sense for TLS verification to be manual such as in ssh, so an automatic system is set up for trust, which is based on digital certificates. TLS/SSL certificates are basically a signed public key. Anybody can sign a certificate, but for the certificate to be trusted by a browser, it needs to be signed by a reputable CA (Certificate Authority). Certificate Authorities are trusted entities which verify the identity of webpages, and if the requester passes their checks, they will receive a certificate signed by the CA's private key. The certificate contains the public key, and the domain of the webpage that requests it.

When connecting to a webpage protected by TLS, the server will send your browser their certificate, and they will sign a message with their private key to prove their identity. An attacker can not send a valid certificate since no reputable CA would have provided them with one for a domain they don't own.

The CA's public keys are included in browsers when they are downloaded. It is important when downloading a browser to do it from a trusted source. A possible attack on TLS would be to inject an attacker's public key in the list of trusted CA's in your browser, which would allow them to sign forged certificates that would be then trusted by your browser.

In TLS the browser does not need to authenticate themselves to establish a secure channel.

Conclusions

I really hope that if you got all the way through, you found the series useful and that you learned something from it. I have done my best to express the topics in the least confusing way, without entering in the details. I would encourage anybody interested by any of the topics covered to look for more information and learn by themselves.

I am looking forward to writing more posts in the future, as I enjoyed writing these. Let me know if you have any ideas about possible topics that you would like to read about regarding areas such as cryptography, blockchain, or general backend development. There are many important things that were not covered and could be additional articles in themselves. 🤖

Cryptography for programmers 3: Hashes, MACs & JWT

Sergi Canal — Tue, 13 Oct 2020 15:11:30 +0000

In the previous post I talked about symmetric key cryptography, and focused on block cryptography for sending messages with a shared key.

Sending secret messages is the use case that comes to mind when we think about cryptography. But cryptography is a wider field, and it deals with other kinds of problems. In this post I will talk about several topics, but they are all connected by a theme: Authentication.

Authentication is one of the most common uses of cryptography. It is defined as:

Authentication (from Greek: αὐθεντικός authentikos, "real, genuine", from αὐθέντης authentes, "author") is the act of proving an assertion, such as the identity of a computer system user.

For the purpose of this post we will cover two kinds of authentication, with the purpose of authenticating different kinds of assertions. One is for authenticating the integrity and source of a message, and the other will be for authentication of users. But first we need to introduce hashing, a cryptographical tool that will be used for both cases.

Hashes

A hash function in the most generic form is a function that is able to transform arbitrarily long data into a predefined amount of bits. Hash functions are used in many computer science applications, but for the purpose of cryptography, we are looking for a specific set of properties that hash functions should have to be considered secure:

it is deterministic, meaning that the same message always results in the same hash
it is quick to compute the hash value for any given message
it is infeasible to generate a message that yields a given hash value (i.e. to reverse the process that generated the given hash value)
it is infeasible to find two different messages with the same hash value
a small change to a message should change the hash value so extensively that the new hash value appears uncorrelated with the old hash value (avalanche effect).

Basically what this means is that you should not be able to reverse a hash function (you are not able to come up with an input that hashes to a given hash in reasonable time).

Hashes are often used to validate integrity. For example in the case of downloading a big file, we would receive a hash along it, we would hash the file that we received, and compare it with the received hash. If the received hash and the generated hash are different then it means that the received file changed during transit, or that the received hash is incorrect. We call this kind of hash a checksum.

Hash collisions

In the example above, we are sending a file, along with a hash. as we said a hash is always the same size, depending on the used algorithm. For example for sha256 it is 256 bits. So if we are transforming files of any size into 256 bits, it is easy to see that it is impossible to develop a hash function where two different files don't hash into the same bits. In fact, since the number of combinations of bits that we can input into the hash function is infinite, and the number of possible hashes is finite, there is an infinite amount of combinations that will produce the same hash.

The objective when developing a cryptographic hash function then, is not to make collisions impossible, but to make it impractical for current technologies to find one. As technology advances however, hash functions that were considered secure, are not anymore. For example sha1 has had many attacks, and is not considered secure.

The first rule presented in the first episode of the series is about not using such outdated hash functions.

Which hash function should I use?

For the purpose of this series, we can say that any hash function in the sha2 and sha3 family are safe, sha256 or sha512 for example would both be safe and are widely used and implemented in many libraries. There are many safe hash functions out there, but we should first check that it is considered safe nowadays before using one.

It is important also to know when a hash function is needed, and when we need something more specific. In the case of password hashing for example, sha2 is not the best tool for the job.

Password hashing: Key derivation functions

There are many articles talking about how it is important to never store passwords in plaintext. It is nowadays common knowledge that passwords should be stored as hashes. But some hashes are better than others. A cryptographic hash function is not designed for this use case in mind, and so it has some issues:

Hashes are fast: Hash functions are designed to be efficient in computation time, meaning that given a hashed password, you can try many passwords by brute force in little time. It would be better to use a slower function that would make it more costly, even for bad passwords, to be brute-forced.
Same passwords have same hashes: Given two users with the same password, we would store the same hash, and that is a leak of information. Also this property makes it easy for attackers to store rainbow tables, containing precomputed password hashes. The solution for this is adding a random salt to passwords before hashing them, and storing the salt along the hash. This is similar to the usage of IVs in block cryptography.

For this reason, there are specific functions, called key derivation functions (KDF), which are made for the specific purpose of password hashing. They use salts, and they are arbitrarily slow to calculate, usually by applying hash functions many times recursively. When dealing with passwords we should always be using KDFs. Examples of good KDFs to use would be bcrypt, scrypt, and argon2.

Message Authentication

In the previous episode of the series we talked about block cryptography. In block cryptography we share a key with the recipient, and we are able to encrypt messages and decrypt them with the same key. It turns out that a lot of the attacks on block cryptography deal with an attacker intercepting an encrypted message, modifying it and then sending it to the recipient. The recipient then receives the message thinking that it comes from the original sender and has no knowledge of the modification.

Some sort of method of authentication needs to be applied to guarantee the integrity of the message sent by the original sender.

HMAC

HMAC stands for Hash based Message Authentication Code.

Sending a hash of the message in itself to authenticate the integrity would not be useful, since a middleman attacker could also generate the hash of the modified message.

So what an HMAC will do is it will include the secret shared key somewhere in the hash. The key is unknown to the attacker, and is impossible to recover from the hash.

The naive way to implement such a construction would be to send along the encrypted message a MAC such as: H(key || message), where H is a secure hash function, and || denotes concatenation. It turns out this is actually not a good solution, since length extension attacks allow an attacker to break that particular HMAC. The standard HMAC construction will be a little more complex, but it will avoid this kinds of attacks:

Where opad and ipad are padding, and m is the message to authenticate.

The HMAC construction can use any secure hash function, and will output a hash at the end. A middleman attacker could not generate a valid HMAC without knowledge of the key. So when we send encrypted data, it is a good idea to send an HMAC along with it, so that if the message is changed, the HMAC will be invalid.

Sometimes HMACs are used as signatures. Meaning that the message is sent unencrypted along an HMAC, so that any middleman could read it, but it can not be modified without knowledge of the key without invalidating the HMAC.

Authenticated Encryption

There exist encryption algorithms that already include the message authentication. The most important of those is GCM, which is a block cipher mode of operation, that is very efficient, and includes an authentication code. GCM is the mode of operation that is most used for transmitting data, especially in client/server environments where CBC is not the best choice. For example it is very commonly used in HTTPS communication, which we will talk about in the following episode. It is very important when using GCM and other stream cipher modes that the iv is never reused. It is recommended to make sure that the iv will not repeat, by adding a counter part in the iv that increments for each message encrypted.

JWT

JWT (JSON Web Token), is a protocol for generating signed tokens containing claims. Claims is any information that you can represent in JSON-like format. For example one claim could be [name = "Sergi"], where I am claiming that Sergi is my name. If I am sharing a secret key with somebody else, then I could send them a Signed token containing this claim, and they could verify that it is correctly signed, and that the claim comes from me. It is important to note that JWT data is not encrypted. Meaning that everyone can read the data on a token, but they can not modify it without breaking the signature. The way JWT implements this for symmetric keys is with HMACs.

a JWT is organized in 3 parts, each one separated by a period: xxxx.yyyy.zzzz. Each of the parts is encoded into a string by base64url. The three parts are:

Header

The header contains information about the algorithm that is used. In the example below, it is using HMAC with the SHA-256 hash.

{
  "alg": "HS256",
  "typ": "JWT"
}

Payload

Here is where the claims are, in JSON format. There are a few special standarized claims, that some libraries treat differently, such as iss (issuer), exp (expiration time) or sub (subject).

In particular exp is interesting because even if the signature is correct, if the token contains an exp claim, the token will not be valid after the expiration.

Signature

This part contains the actual signature of the first two encoded parts, using the specified algorithm in the header, and a secret:

HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  secret)

If you want to quickly play around with JWT, at jwt.io there is a very useful tool for encoding and decoding jwt tokens.

User Authentication

User authentication is a problem that requires all of the techniques we talked about in this post, and is the most common use-case for JWT tokens. The problem in User Authentication, is not only to verify their credentials, but to provide them a way to keep an authenticated session open without having to verify their identity at every operation.

User authentication usually goes like this:

The user signs up creating a user and a password.
The auth server stores the user and a secure password hash of the password (created with a secure KDF).
The user signs in to the server, providing the user and password.
The auth server generates the hash from the provided password and compares it with the stored hash. If it matches, a JWT is provided with the sub (user) and exp (expiration) claims.
The user provides the JWT along with every operation they realize, and the server verifies the signature and expiration every time. If the JWT signature is correct, then the operation is allowed for the user in the sub claim.

In this case, the auth server has a secret key for the JWT signing, so that it is the only one able to verify and sign tokens, and the user is not capable of altering the claims provided by the server.

I provide below a simplified example of the code that an auth server would have for user authentication. Usually though, implementing authentication is not the best solution, and it might be a better choice to use an authentication system such as firebase auth or auth0, which offer many interesting features such as SSO, and social logins.

And that's it for today! I hope I was able to give a good summary of the topic of authentication in cryptography. In the next episode I will briefly talk about asymmetric public key cryptography, and we will see how all that we learned in the series comes together in cryptographic protocols such as ssh and TLS that we use every day without noticing.

Cryptography for programmers 2: Blocks and Randomness

Sergi Canal — Wed, 23 Sep 2020 16:25:09 +0000

In the previous post of the series, I made an introduction on why it is important for programmers to have a basic knowledge of cryptography, and I gave 3 basic rules to be followed by programmers to avoid the most common mistakes. The rules however, are quite vague, and need some more specific understanding to be able to be applied effectively.

In this episode we will cover two of the most important topics in cryptography: Block cryptography, and secure randomness. I have to admit that it has been quite hard to summarize such a big and exciting topic so much, and I hope that I have done a good job! 😄

Symmetric vs Asymmetric encryption

To understand what block crypto is, we need to first discuss the two main ways of encrypting and decrypting data.

In symmetric encryption both encryption and decryption are done with the same key. In the case of encrypted communication between two parties, they need to have a shared key between them that nobody else knows. This is the main limitation for symmetric cryptography, since in contexts such as the internet parties don't have shared keys between them.

In asymmetric cryptography the key used to decrypt is not the same one that is used for encryption. Asymmetric cryptography is mainly used for the purpose of communication between different parties, where the parties have no shared key or safe channel through which to send one. In asymmetric cryptography each participant has a key pair consisting of a public key and a private key. As the name implies the public key is publicly shared and anyone can have access to it. The private key should be kept secret and never shared. Then when someone wants to communicate with you, they would encrypt the message using your public key, and only those with knowledge of the private key are able to decrypt it.

As practical as asymmetric crypto sounds, as we will see in part 4 when we talk about public key and protocols, asymmetric cryptography is rarely used to transmit data, it is usually used to share a key which can then be used to continue the communication using symmetric crypto. The main reason being that symmetric cryptography, and in particular block ciphers, are much more efficient than mathematically based asymmetric ciphers.

Block Cryptography

Block cryptography is the default symmetric cryptography used nowadays, and so it will be the only symmetric cryptography that will be covered in the series. Most cryptography courses would introduce the topic with simpler symmetric ciphers such as OTP, and other XOR based methods. It is kind of interesting from the theoretical point to study them and their attacks. As a programmer however, you should not be using those, so we will skip them.

What is Block Cryptography

A block cipher is a cryptographic algorithm that is able to encrypt a fixed amount of bits in a safe way with a symmetric key. Block ciphers have a different function for decryption and encryption. There are many different block ciphers to choose from, and many of them are considered safe, but some are outdated and no longer safe.

What Block cipher should you use?

We should pay special attention when choosing a block cipher to make sure that it is still considered safe nowadays. For example DES was a very popular block cipher that was standard for a while, and maybe some old tutorials may use it and give you the impression that it is a good choice. It is not. In fact using DES itself will already be in breach of one of the rules presented in the previous episode. There is a slightly more "modern" version called triple-des or 3DES, it basically encrypts the contents 3 times with the DES algorithm, each time with a different key, effectively making the key 3 times the size. It is possible that you encounter 3DES, specially in some legacy projects. The security is not that great however, and it should not be used for a new project.

For the purpose of this post I would say as a programmer there is no reason you should be using anything other than AES. AES is the current standard, it is secure enough and will very likely continue to be for a long time, specially if you are using the variant with a 256 bit key, which I would recommend. AES is used so much in fact, that most modern processors have specific instruction sets for AES operations, so apart from being secure, it is very efficient.

Modes of operation

Now that we have chosen the Block cipher that we want to use, we are still not done. AES has a block size of 16 bytes, independently of the size of the key, so if we want to encrypt something longer than 16 bytes, we will first have to separate the message into chunks of 16 bytes, if the last chunk does not have exactly 16 bytes, we will add padding. There are various ways of combining block ciphers to be able to encrypt long messages split into blocks, with only one key. We call them modes of operation.

ECB

This is probably the solution that somebody who does not know about cryptography would give if they encountered this problem:

ECB mode consists on splitting the message into 16 byte chunks, and encrypting each of them with the same key and the same algorithm. The supposition being that if the algorithm is secure, then none of the blocks will be able to be decrypted without the key.

There is a problem however with ECB, which is that it gives up too much information about the message that was encrypted. Block ciphers are deterministic, meaning that when you encrypt the same 16 byte block with the same key and with the same algorithm, it will always return the same ciphertext. So if you were to encrypt something which contains repeated sequences, the repetition would also show after encryption. The most famous example of this is the result of encrypting a picture of a penguin, where sections of the same color, are encrypted into the same altered colors, keeping the fact that it is a picture of a penguin completely clear.

Let's imagine that say, a videoconferencing app decided to offer the security of encryption to users, and they decided to use ECB as their mode of operation. Do you think it would be a good choice?. Ehem.

ECB is also susceptible to other attacks [1] [2], and while it is not important to know them in detail, it is important to never use ECB, for any application, ever. There is always a better alternative.

CBC

In the diagram above, we see the encryption process used by the CBC mode. Where the circles with crosses refer to the bitwise XOR operation. In CBC the ciphertext of the previous block is XORd with the following block before encryption. What this does is it alters the ciphertexts depending on the previous blocks, making it so that the problem in ECB of not masking patterns does not exist in CBC. If we were to encrypt the penguin image with CBC, we would only see what would appear like random noise.

In CBC we also introduce the concept of Initialization Vector (IV). The iv is a random block that is generated for each message and is used to mask the first block from patterns. Here are a few rules from the previous episode's link related to IVs:

don't reuse IV and key pairs

When encrypting with the same IV and Key, the first block of the ciphertext is not protected from leaking information via patterns in different messages. In other modes such as OFB and CTR, reusing an IV completely destroys the security making it trivially breakable. Never reuse an IV.

Despite having to be unique however, IVs are not secret and are usually stored or sent together with the encrypted message.

don't use a badly derived IV

As said in the previous rule, IVs are not secret. They should however be unpredictable, which means they should be generated randomly. We will talk about randomness later in this article.

don't use a static IV

I assume this refers to not hardcoding an IV into the code. In that case it would mean reusing the same IV, which we already said is not a good idea.

CBC is a more secure alternative to ECB for the majority of purposes if used correctly. CBC is perfectly safe to store encrypted data in one's server for example.

There are some cases though in which you should not use CBC, specifically in cases where CBC encrypted data is transmitted in a client-server environment, where the server is susceptible to replay attacks. This has to do with my favorite crypto attack: the padding oracle attack. It is a perfect example of how very small amounts of information leaks, can lead to complete failures of encryption systems.

CBC is also not suitable for encrypting very large files or disks. Because of the way it is designed it is impossible to parallelize the execution of the encryption (decryption can be parallelized though), and in the case of encrypting a disk, it is impossible to change a block without reencrypting all the following ones.

Others

There are many modes of operation. I would recommend only ones that are recognized by standards bodies such as NIST. I think CBC is a good example of a mode of operation for study, and it is perfectly safe as long as it is used correctly and in the use cases where it works. There are specific modes of operation for specific problems, such as modes of operation made for disk encryption. We will also see one more mode of operation (GCM) when we talk about message authentication, and we will see that it is a better option for client-server situations where CBC is not a good choice.

Randomness

When we are working with cryptography we sometimes need to create secrets that no one else knows, or just data that behaves unpredictably. For example when generating a secure Block cryptography key, we should make it not only large, but random, since it does not matter how long your key is if it is just 'AAAAAA...', also IVs need to be generated randomly. So we need a way to generate random bits with a computer. The problem is that computers are the opposite of random. The processor does exactly what it is told, always in the same way, predictably. How could it be possible for the computer to generate randomness then?

PRNGs

We call random number generators in computers PRNGs (Pseudo-random Number Generators). We add Pseudo at the beginning to signify that they are not trully completely random, as in completely unpredictable. But they behave in a random way, meaning that if we were to generate many random numbers, they would follow the same kind of distribution that we would expect from a truly random process.

Generic PRNGs are included into all modern programming languages. PRNGs can be used for purposes such as generating a random distribution of numbers for statistical studies, random sampling, rolling dices ... For this purposes it is actually interesting that the random generation is repeatable. For example if we are making a study and taking random samples, we want the sample to be randomly distributed, but we want to be able to share the code so that the same sample will be generated. This PRNGs are actually complicated algorithms that generate distributions that act randomly, in a deterministic way. They take as initial input a seed, and given the same seed they will generate the same numbers.

Let's say we want to generate a secure key, and we do it with a standard PRNG. If we really wanted our key to be random, we would first need a truly random seed. Here is the problem, to generate a random seed, we need a number that is not generated with a PRNG, but it should be random. When using PRNGs a common solution is to use the current timestamp as the seed. The timestamp has the property that it changes all the time, but it does NOT act random at all. Let's say we generate our safe key using a time-seeded PRNG. And now let's say that an attacker knows that we generated the key in 2014, which is not a narrow interval. According to google, a calendar year has 3.154e+7 seconds, so roughly 31M seconds. All the attacker would need to do to learn our key would be to try generating it with the 31M possible seeds. It is not much for a computer to generate 31M random keys with a PRNG. We essentially lowered the security of our 256 bit key (115792089237316195423570985008687907853269984665640564039457584007913129639936 possible keys) into the security of a 20 bit key. And no, using millisecond (or even nanosecond) timestamps is not a good solution either.

Cryptographically secure randomness

As we have seen, we need a few more requirements from our randomness generation for them being able to be used safely for cryptographic applications. We need Random generators that do not rely on an initial input, and are not deterministic. They are called Cryptographically Secure PRNGs and they are included in most popular cryptography libraries. They use various external sources of randomness, usually physical properties, or user interaction. If we were physicists, we could probably discuss about whether true randomness even exists, or given the state of the entire universe one could determine the future. Lucky for us, we are not protecting our systems from omniscient deities, and just need them to resist clever monkeys with current technology, so we will suppose the physical properties are chaotic and noisy enough to not be able to be predicted. Things that can be used are temperature of the processor, the movement of the cursor, and even videos of lava lamps.

When programming cryptographic code it is very important that any randomness we need, is provided by cryptographically secure generators, as normal PRNGs provided by standard libraries are not suited for this use.

Code example

In the following code example I will provide a typescript implementation of an encrypt / decrypt class, using the concepts learned in this episode. It is important to note that I will be using CBC, meaning the following code should never be used in client / server communications with the possibility of a replay attack. This code would be suitable for cases such as storing encrypted data on a database, or storing a secret file on your computer.

import * as crypto from "crypto";

export interface EncryptedData {
    iv: string;
    data: string;
}

export class AES_256_CBC {
    private readonly encryptionKey: Buffer;
    private readonly ALGORITHM = 'aes-256-cbc';

    constructor(key: string) {
        // Key is a base64 encoded 256bit key (32 bytes)
        // It should be stored safely as an environment variable
        // and never uploaded into version control
        this.encryptionKey = Buffer.from(key, 'base64');
    }

    public encrypt(text: string): EncryptedData {
        // We generate the iv with a secure PRNG
        const iv = crypto.randomBytes(16);
        // We create the cipher with algorithm aes-256-cbc
        const cipher = crypto.createCipheriv(this.ALGORITHM,
            this.encryptionKey, iv)
        let enc = cipher.update(text, 'utf8', 'base64');
        enc += cipher.final('base64');

        // We return both the encrypted text and the iv in base64.
        // If we want to store it in a DB which does not support
        // JSON format, we can serialize it by appending the data
        // to the iv and converting into a single string.
        return {
            iv: iv.toString('base64'),
            data: enc,
        }
    }

    public decrypt(enc: EncryptedData): string {
        const decipher = crypto.createDecipheriv(this.ALGORITHM,
            this.encryptionKey, Buffer.from(enc.iv, 'base64'));
        let str = decipher.update(enc.data, 'base64', 'utf8');
        str += decipher.final('utf8');
        return str;
    }

}

That's it for this episode 😊. I hope you now have a better understanding of the importance of using the right parameters and modes of operation when working with block cryptography, and the importance of using cryptographically safe randomness.

In episode 3, we will talk about securing our cryptography for communication purposes with message authentication, and we will talk about the best ways of implementing a login with JWT.

Cryptography for programmers 1: Basics

Sergi Canal — Wed, 16 Sep 2020 11:33:01 +0000

The other day I read an article that talked about how many popular Android apps had basic cryptographic vulnerabilities in their code. The analysis was done with Columbia University's Crylogger, an open source dynamic analysis tool that detects cryptographic vulnerabilities. I was a bit surprised with the results (All of the analysed apps broke at least one of their chosen 26 crypto rules), but not very much. I myself have encountered more than one of this rules being broken in projects I participated. These rules do not come intuitively to people who have never learned about cryptography, and who don't understand the basics. And most programmers usually have never learned cryptography properly in a structured way. When the average programmer wants to implement some cryptography in their project, they do the same as they would with any other problem: They look for the solution on stack overflow. The problem with cryptography code, is that even if it seems to be working, it does not mean it is secure. Cryptographic code that is perfectly safe in some context, could be potentially useless in another one. Another common issue is that new projects often don't worry too much about security, and leave it for the future. But there are cases where it is too costly to refactor later on. If you were to store user passwords with an insecure hash, there is no way you can solve this without making users reset their password, or having to handle two different types of hash, so the mistake stays. Storing the passwords securely from the beginning however, is very easy and would not have taken any extra time.

The aim of this series is to teach the basics of cryptography for programmers, so that their projects will not be a part of the statistics. Sometimes cryptography can seem intimidating to get into. My aim here is not to teach the specifics of cryptography algorithms, or go into the details and nuances of academic crypto. My aim is to make it as practical as possible, while giving an idea of what are the best practices and mistakes to avoid. For those that want to dive deeper into the whys of what I will say, I will link sources into the specifics.

I will be providing code examples in typescript / node for some of the sections, but I don't recommend using the code as-is, without reading the entire series, understanding the code and thinking if it really fits your problem. I will also be providing incomplete and incorrect code, that can be improved in following sections of the series. By the end of the series, readers should be able to understand and apply the 26 crypto rules chosen by Columbia for Crylogger.

The series will be organized in 4 parts:

In part 1 (this one) I will give some basic general rules for writing secure cryptography code.

In part 2 I will talk about block cryptography, the kind of cryptography that is most used for encryption/decryption of data. And we will discuss secure ways of generating cryptographic keys and randomness.

In part 3 I will talk about Authentication, password and login management and we will put it all together to implement a login with JWT.

In part 4 I will talk about public key cryptography, and the basics of internet protocols (SSL/TLS, SSH, ...)

The Basic principles of cryptography

1. Avoid unnecessary complexity

Cryptographic systems exist inside a bigger system, and are not an isolated component. The more complex a system is, the more likely that there is a vulnerability somewhere. And your system is only as secure as the most insecure component. Let's say you are very proud because you are using state-of-the-art cryptography algorithms, but the way you are generating or storing the key is insecure. Then as good as the cryptography is, the key is easy to get by an attacker and all your data can be trivially decrypted. The more parts of your code that require security and the more complex they are, the more likely that someone, at some point, will fuck it up.

2. Kerckhoff Principle: Only the key should be a secret

In cryptographic security, only the encryption / decryption key (or the private key in asymmetric crypto) should be secret. This means, the system security should be designed so that it is secure even if an attacker knows everything about your system, except the key. The most common thing that is kept secret, in hopes that it will increase security, is the code. Some people go as far as using obscure / unusual encryption algorithms, so that an attacker will not know how the data is encrypted. The reality is that the code of a project is not a secret, since every programmer in your team has it, and so if the security of your system depends on the secrecy of the code, you are potentially giving too much power to too many people.

The right way to do it is to keep the key secret, in a way that only a very select trusted group of employees can access it (and ideally they can't read or copy it, only use it), and that is the only secret that your system security should rely on. It is not that unusual for employees of companies to be offered money by hackers in exchange for information or access.

3. Don't roll your own crypto

This is likely the most famous rule of cryptography. It basically means that you should never program the encryption algorithms yourself, or even create new encryption algorithms custom-made for your project. Inventing your own cryptography would be like building your own plane engine. Plane engines have undergone a long testing process, and they have gone through their ups and downs (hehe). The same way cryptography algorithms have survived the test of time for many years. Even algorithms created by cryptography experts need to undergo intense scrutiny and survive after years, before they are used and trusted.

This also applies to different implementations of the same algorithm. Even if you have read about RSA on the internet, and you understand the maths behind it, that does not mean that your implementation will be safe. The popular cryptography libraries (such as openSSL), have also undergone the test of time, and have evolved and updated to solve all the possible bugs that have been found, many of which you would never think about when you are implementing the algorithm yourself. Even when an algorithm is theoretically safe, the way you implement it can give out some information. These leaks of information can be used in what we call side-channel attacks, which do not deal with the theoretical security of the algorithm, but with things such as the time that it takes to perform operations, or even the sound the computer makes, and the energy it consumes.

The job of the programmer when implementing cryptographic code, is to use secure up-to-date libraries (and to keep them updated), and to provide the right parameters to the right algorithms for the job. In the next episodes we will discuss what exactly we mean by the right parameters. We will see that even using secure implementations of secure algorithms, we can still mess up if we don't know what we are doing. Most of the 26 rules that the study above is checking deal with passing insecure parameters to crypto libraries, and using out of date algorithms.

Let's learn how to not do that!