Bettina Ligero

Posted on Apr 24

RSA-OAEP Encrypt-then-Sign Messaging Tool

#security #cryptography

Machine Problem 2
Group Members: Deen, Ligero, Torres

Introduction

This machine problem involved designing and implementing a secure messaging system using public-key cryptography. Unlike the previous exercise—where the goal was to break a program's security—this one required building security in from the ground up.

The objective was to construct a command-line tool that allows users to exchange messages with two guarantees:

Confidentiality – only the intended recipient can read the message
Authenticity – the recipient can verify who sent it

To achieve both properties simultaneously, the program follows an Encrypt-then-Sign construction: the message is first encrypted using RSA-OAEP, and the resulting ciphertext is then digitally signed using RSA-PSS. This ordering ensures that the signature covers the ciphertext itself, protecting against certain classes of attacks where a valid ciphertext could be re-signed by a malicious third party.

The Cryptographic Scheme

The tool is built on two well-established RSA schemes, both using SHA-256 as the underlying hash function.

RSA-OAEP (Encryption)

RSA-OAEP (Optimal Asymmetric Encryption Padding) is used to encrypt the plaintext message. Raw RSA encryption is deterministic and vulnerable to chosen-plaintext attacks. OAEP adds randomized padding before encryption, making repeated encryptions of the same message produce different ciphertexts and significantly hardening the scheme against cryptanalysis.

The sender encrypts using the recipient's public encryption key. Only the recipient—who holds the matching private key—can decrypt it.

RSA-PSS (Signing)

RSA-PSS (Probabilistic Signature Scheme) is used to sign the encrypted message. Like OAEP, PSS incorporates randomness into the padding process, making it provably secure under standard hardness assumptions.

The sender signs using their own private signing key. Anyone with the sender's public signing key can verify the signature—but only the legitimate sender could have produced it.

Why Separate Keys?

A deliberate design choice was made to use separate key pairs for encryption and signing. While it is technically possible to reuse a single RSA key pair for both operations, this practice weakens the security boundaries between the two roles. Dedicated keys ensure that a compromise of one key does not automatically compromise the other, and it aligns with real-world standards such as X.509 certificate profiles and PGP key conventions.

Key Generation and the Directory

Each user generates two RSA key pairs (4096-bit): one for encryption and one for signing. The private keys are stored locally in a keys/ directory. The corresponding public keys are registered in a shared directory.json file, which acts as a simple public key infrastructure (PKI).

python mp2.py genkey alice
python mp2.py genkey bob

After registration, public keys can be inspected by any user:

python mp2.py list       # lists all registered identities
python mp2.py lookup alice  # displays alice's public keys

The directory.json file is a flat JSON structure mapping each identity to its public encryption and signing keys in PEM format. While functional for this exercise, this approach has a notable limitation discussed later.

Encrypting and Signing a Message

To send a message, the sender specifies their own identity, the recipient's identity, and the plaintext:

python mp2.py encrypt-sign alice bob "Hello Bob!" -o msg.json

Internally, this performs the following steps:

Retrieve the recipient's public encryption key from directory.json
Encrypt the plaintext using RSA-OAEP with the recipient's key
Retrieve the sender's private signing key from local storage
Sign the ciphertext using RSA-PSS with the sender's key
Bundle the result into a JSON file containing the ciphertext, the signature, and the sender's identity

The output msg.json contains no plaintext—only the encrypted blob, the signature over that blob, and metadata identifying the sender.

Verifying and Decrypting

The recipient processes the message with:

python mp2.py verify-decrypt bob msg.json

This performs the operations in reverse:

Look up the sender's public signing key from directory.json
Verify the signature over the ciphertext using RSA-PSS
If verification passes, decrypt the ciphertext using the recipient's private key
Output the plaintext message

Verified decryption:

If any part of the message bundle has been modified—ciphertext, signature, or sender identity—the PSS verification step will fail, and the program halts with an error before any decryption is attempted. This is the key security property of the Encrypt-then-Sign construction: tampering is caught at the verification stage, not silently tolerated.

File Structure

.
├── mp2.py
├── keys/
│   ├── <identity>_enc_priv.pem
│   ├── <identity>_enc_pub.pem
│   ├── <identity>_sign_priv.pem
│   └── <identity>_sign_pub.pem
├── directory.json
└── msg.json

Results

Testing the system end-to-end with valid keys and identities produced the expected behavior in all cases:

A message encrypted for bob could only be decrypted by bob
A message signed by alice was verified using alice's public key
Any modification to the ciphertext or signature caused verification to fail immediately
Attempting to decrypt with the wrong private key produced a decryption error

The tool correctly enforces the two core security guarantees: only the intended recipient can read the message, and the recipient can confirm who sent it.

Limitations

The implementation has several constraints worth noting:

Message size. RSA-OAEP with SHA-256 and a 4096-bit key can encrypt at most around 446 bytes. The program enforces a 140-character plaintext limit to remain within this bound. A more practical system would use hybrid encryption—generate a random AES session key, encrypt the message with AES, and then encrypt only the session key with RSA. This approach removes the message size constraint while preserving the security properties of RSA.

Directory trust. The directory.json file has no authentication mechanism. A malicious actor with write access to the directory could substitute their own public key under another user's identity—a classic key substitution attack. Real PKI systems address this with certificate authorities, web-of-trust models, or out-of-band key verification.

No forward secrecy. Because the same long-lived RSA keys are used for every message, a future compromise of a private key would expose all previously recorded ciphertexts. Systems requiring stronger security guarantees often combine asymmetric key exchange with ephemeral session keys (as in TLS or Signal's Double Ratchet protocol).

Conclusion

This machine problem demonstrated how public-key cryptography can be composed to provide both confidentiality and authenticity in a messaging system. The Encrypt-then-Sign construction—using RSA-OAEP for encryption and RSA-PSS for signing—produced a tool that correctly protects messages against eavesdropping and detects any tampering before decryption is attempted.

Separating encryption and signing into distinct key pairs was the most consequential design decision, as it follows established cryptographic hygiene and prevents cross-role key misuse. While the current implementation has practical limitations around message size and directory security, it cleanly illustrates the fundamental principles underlying modern secure messaging protocols.

Requirements

Python 3
cryptography library — install with:

pip install cryptography

Note: Key addresses and file paths may differ across environments. If transferring keys between systems, ensure the directory.json and keys/ directory are synchronized between participants.

DEV Community