MP2 RSA-OAEP Authenticated Encryption (Encrypt-then-Sign)

Starting out

When we first sat down with the spec, the sentence that stuck with us was:

"Note that RSA isn't really meant to encrypt plaintext, but you'll have to find a way to get it to do that."

That single line ended up shaping almost every decision we made. It told us the assignment wasn't really about getting RSA to swallow 140 characters — it was about recognizing why you shouldn't, and reaching for the pattern that real systems use instead.

We split the work loosely: Akhy took the key-generation and trusted-directory layout, Lars owned the encryption/signing pipeline, and Carl handled the CLI, the sample files, and hammering on edge cases. Anything involving padding parameters (OAEP, PSS, MGF1, salt lengths) we did together, because those are the kinds of details that silently compile and silently break security.

The first attempt — and why we threw it away

Our first version did the literal thing: RSA-OAEP directly on the plaintext, then sign the resulting ciphertext with the other keypair. And honestly… it worked. RSA-2048 with OAEP-SHA256 has room for about 190 bytes of plaintext, so 140 ASCII characters fits comfortably. Roundtrip passed. Tampering got detected. On paper, we were done.

But sitting with it, it felt like we had solved the letter of the spec and missed the point. The hint was practically waving at us. We also noticed that our "solution" would silently break the moment anyone tried a 141st character, because the failure would come from the RSA padding layer rather than our own input validation — an ugly coupling between message length and the choice of cipher.

So we rewrote it as hybrid encryption:

1. Generate a fresh 32-byte AES-256 session key and a 12-byte GCM nonce per message.
2. Encrypt the plaintext with AES-256-GCM.
3. Wrap only the session key with RSA-OAEP (SHA-256 + MGF1-SHA-256).
4. Bundle {encrypted_key, nonce, ciphertext}.
5. Sign the canonical JSON of that bundle with RSA-PSS (SHA-256, MAX_LENGTH salt).

That's the same shape TLS, PGP, and S/MIME all use, and crucially, the length constraint now lives in our validation code — not in the cipher.

Design decisions we argued about

Encrypt-then-sign vs. sign-then-encrypt. The spec mandated encrypt-then-sign, but we spent time making sure we actually understood why. The argument that landed for us: signing the ciphertext binds the sender's identity to the exact bytes that were transmitted, which blocks surreptitious-forwarding — a malicious recipient peeling off the signature, re-encrypting the plaintext under a third party's public key, and making it look like the original sender was talking to them all along. Sign-then-encrypt doesn't catch that.

Two keypairs, not one. The spec required it, but writing _enc_private.pem and _sig_private.pem as separate files (and loading them through separate functions) gave us a free guarantee: our code physically cannot confuse a signing key with an encryption key. That's a small architectural payoff we didn't expect going in.

What exactly gets signed. We landed on json.dumps(enc_bundle, sort_keys=True) as the canonical form. Since every value inside is base64-encoded, there's no Unicode escape ambiguity, and sort_keys=True removes any dependency on Python's dict iteration order. That made the sign/verify boundary deterministic without us having to do any byte-level gymnastics.

Why AES-GCM and not AES-CBC + HMAC. GCM gives us authenticated encryption in a single primitive. Combined with the outer RSA-PSS signature, we end up with two independent integrity checks on the ciphertext — a belt-and-suspenders property we liked.

What worked

• pyca/cryptography did all the heavy lifting — OAEP, PSS, AES-GCM, PEM serialization. No hand-rolled crypto, as the spec required.
• Tamper detection fires at the right layer. When we flipped a single byte of the ciphertext during testing, verification failed at the signature step, before we ever touched the recipient's private key. That's the whole point of encrypt-then-sign, and it was satisfying to watch it actually behave that way.
• Input validation runs before any crypto. The 140-character limit and ASCII check happen first, so bad input fails fast with a clear ValueError instead of bubbling up as a confusing padding error from deep inside the library.
• The "trusted directory" is dead simple. A local keys/ folder + a list-keys command. It's not PKI, but it's an honest model of what a trusted directory is at its core: a place where public keys live, addressed by identity.

Limitations we're aware of

We want to call these out explicitly rather than pretend the implementation is production-ready.

• No replay protection. A captured valid bundle can be resent and will still verify. A production version would need a per-recipient seen-nonce set or a timestamp window baked into the signed payload.
• No sender/recipient binding inside the signature. Right now the signature covers {encrypted_key, nonce, ciphertext} but not the sender/recipient identity fields in the outer JSON. That means an attacker who intercepted a bundle couldn't forge a new one, but they could conceivably relabel the outer envelope. We'd fix this by pulling identities into the signed bytes.
• Private keys on disk are unencrypted (NoEncryption()). Fine for a demo; a real deployment should password-encrypt them.
• No key revocation or rotation. Once a key is in the directory, it's trusted forever.
• CLI surfaces Python tracebacks on failure instead of friendly one-liners. We left it that way because it made failures easier to debug during development, but a production CLI should catch at the top level and exit cleanly.
• "Trusted directory" is trust-by-filesystem. No CA, no signatures on the public keys themselves. Good enough for this MP, not for the internet.

What we'd do next

If we kept going, the next pass would be: (1) fold sender and recipient into what gets signed, (2) add a password-protected keystore, (3) add replay protection via a per-recipient nonce registry, and (4) clean up the CLI so errors surface as single-line messages with nonzero exit codes.

What we took away

The biggest lesson was that the hard parts of applied crypto aren't the math — it's knowing which primitive to reach for, which layer binds what to what, and what your threat model actually is. The libraries are excellent; the thinking is the work. The second lesson was more practical: write the input validation before the crypto, always. It turns a whole class of obscure library errors into ordinary, debuggable application errors.

We finished the assignment with a program that's ~250 lines, uses only well-tested primitives, and fails safely in every case we could think of to test. That felt like the right outcome.