DEV Community

Aditee Niraula
Aditee Niraula

Posted on

Avoiding the 5 Mistakes Most Tutorials Make When Creating a File Encryption Tool

Why “it encrypts” doesn't equate to “it’s secure”

If you want to find a tutorial for encrypting files in code, your search results will provide dozens of tutorials. Most of these tutorials will produce code that, on the surface, performs encryption. Users can provide plaintext, receive ciphertext, and the code also performs decryption.

Unfortunately, the phrase “the output looks scrambled” is an unsecure way to test a program for security. These tutorials fail to incorporate security practices, which will result in these tools being rejected in real life security assessments.

By identifying these mistakes, we can reason about the validity of these encryption schemes. This article covers the correct way to build a file encryption tool and the mistakes that beginner encryption tools include. These mistakes will help you learn the correct way to build an encryption tool.

SecureVault (Node.js, packaged with no dependencies) is a command-line tool that is referenced throughout to help provide context to the design decisions that were made for this tool.

Prerequisite mindset: When designing secure systems, always assume that the attacker knows more than you. Do you really think that your adversary will only submit the inputs you assumed they would submit? They will submit corrupted inputs, they will submit old ciphertexts, and they will do anything you thought was impossible. You need to have a secure design. You must think "what malicious inputs can I handle here?".


The goal: three guarantees, not one

Before you even think about writing code, you need to know exactly what you mean by that something is secure. A good file encryption tool must provide three guarantees. Most of the tutorials that I have seen think only about the first one.

  1. Confidentiality - the attacker that steals the file should not be able to read the file.

  2. Integrity - If the attacker alters the encrypted file, you will know.

  3. Authenticity - The file can only be generated by a user that knows the password, and ciphertext of a wrong key cannot be generated at all.

Keep these three in mind. The mistakes that I will show you are all failures to protect one of these three guarantees.


Mistake #1: Using the password directly as the key

This is the most common mistake. The code asks for a password, and let's say that the password is password123, and the program is using the password directly as the key.

Why it's dangerous:

  • The keys to the encryption must be a certain size and must be very long and must be very high entropy . A password less than the required size is very predictable and can be easily guessed. This means that there is no protection against brute force. If verifying a certain password is cheap, the attacker can try billions of them per second.

The fix: a Key Derivation Function (KDF).

KDFs take a weak passphrase and, through intentional repetition, stretch that passphrase into a strong key. SecureVault implements PBKDF2 with 200,000 iterations of SHA-256:

function deriveKey(password, salt) {
  return crypto.pbkdf2Sync(
    password,
    salt,
    200000,        // iterations — deliberately slow
    32,            // 32 bytes = a 256-bit key
    'sha256'
  );
}
Enter fullscreen mode Exit fullscreen mode

There are two reasons why this is secure. First, the output is a full-strength 256-bit key no matter how short the password was. Second, and more cleverly, running the hash 200,000 times makes malicious password attempts much slower. You hardly notice the 200,000 iterations when you log in, but an attacker is trying to test a billion passwords, so they will suffer that cost a billion times over. This is a case where the slowness is a feature, not a bug.

Modern implementations like scrypt and Argon2 go further by also consuming a lot of memory, which defeats specialized cracking hardware. PBKDF2 is the well understood baseline; the principle — make guessing deliberately expensive — is what counts.


Mistake #2: No (or a constant) salt

Let's say two users choose the password summer2024. If you derive the key from the password, they would get the same key; an attacker would compute a huge "password → key" table and instantly crack every account. These tables are known as rainbow tables.

The solution is: a random unique salt with every file.

A salt is a random value that is appended to the password when computing the key. It is not required to be kept secret, but it must be different for each password. The same password with a different salt leads to a different key, so:

  • The attacker would need a different unique salted table for each salt.

  • Two users selecting the same password will not collide.

const salt = crypto.randomBytes(16);   // fresh, random, per file
const key = deriveKey(password, salt);
Enter fullscreen mode Exit fullscreen mode

The salt is stored in the clear alongside the ciphertext — that's fine and expected. Secrecy isn't its job; uniqueness is.


Mistake #3: Reusing the IV (or hardcoding it)

Block ciphers like AES need a second input besides the key: an IV (Initialization Vector), sometimes called a nonce. Its job is to make sure that encrypting the same plaintext twice produces different ciphertext.

Beginner code often hardcodes the IV to all zeros, or reuses one fixed value. This is catastrophic.

Why it's dangerous:

  • With a repeated IV, encrypting the same message always yields the same ciphertext. An attacker who sees two identical ciphertexts learns the two plaintexts are identical — a real information leak.
  • For some cipher modes, IV reuse is fatal — with AES-GCM specifically, reusing an IV with the same key can let an attacker recover the authentication key and forge messages. It doesn't just weaken the scheme; it collapses it.

The fix: a fresh random IV for every single encryption.

const iv = crypto.randomBytes(12);     // new every time, never reused with a key
const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);
Enter fullscreen mode Exit fullscreen mode

Like the salt, the IV isn't secret. Store it with the file. The rule is simply: never reuse an IV under the same key. The easy way to guarantee this is to generate it randomly every time.

Thus, in SecureVault's tests, encrypting the same input twice is a security test, not a triviality, if the outputs differ — it means the IV is actually doing its job.


Mistake #4: Encrypting without integrity (the big one)

This is the largest mistake on the list because the code almost looks right. A lot of tutorials will go over how to use AES in the CBC or CTR mode to encrypt the data and that is the end of the road. Congrats, you've achieved confidentiality. Now, integrity is still nonexistent.

Why it's dangerous:

While the data may be concealed, it does not prevent an attacker from changing it. Bit flipping is a perfect example of how this is performed in a mode of encryption that does not support integrity — and it is done, on many occasions, without the attacker even having access to the key. This is an actual attack that is known as the padding oracle. Your program would decrypt the tampered file and give the attacker the data, while the victim remains unaware of what has happened.

In conclusion, integrity is a must. You think the file is protected, but really, anyone in the middle can rewrite it.

The fix: authenticated encryption (AEAD)

Use encryption modes that provide both secrecy and integrity. Standard choice is AES-GCM (GCM = Galois/Counter Mode). When encrypting, GCM provides an additional output called the authentication tag: a cryptographic fingerprint of the ciphertext. It also allows the receiver to detect any modification of the ciphertext. If the tag verification fails, the ciphertext will be discarded and an empty message will be sent to the receiver.

Remember: state size of AES-GCM (Galois/Counter Mode) is 128 bits, therefore, its throughput is 128 bits. AES-GCM supports arbitrary-length messages. However, the size of the authentication field is fixed and limited to 64 bits. That offers a lot of protection, but in general, the integrity of longer messages is usually more vulnerable.

"an empty message will be sent to the receiver." should be explained in more detail.

const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);
const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()]);
const authTag = cipher.getAuthTag();   // 16-byte integrity fingerprint
Enter fullscreen mode Exit fullscreen mode

On decryption, you supply that tag back, and GCM verifies it before giving you any plaintext:

const decipher = crypto.createDecipheriv('aes-256-gcm', key, iv);
decipher.setAuthTag(authTag);
// If the password is wrong OR the file was modified, .final() throws:
return Buffer.concat([decipher.update(ciphertext), decipher.final()]);
Enter fullscreen mode Exit fullscreen mode

If even a single byte of the ciphertext were to be changed, or if the incorrect key is applied, the tag won't match and decryption will be quiet and return destructive data. That single mechanism provides integrity and authenticity for free.

If you ever need to combine a separate cipher and MAC, the right order is Encrypt-then-MAC: first, you encrypt and then you authenticate the ciphertext. But the modern advice is simpl and clearer — use AEAD and don’t invent anything.


Mistake #5: Failing "open" instead of "closed"

This last mistake is more of a behavioral than a functional problem, and it deals with the user experience of your tool.

What does your tool do in the case of an incorrect password, or if the file is corrupted, or the input malforms.

Insecure code usually fails open: the error is catched and returned, and whatever it partially produced to the output file is returned (most likely garbage). The user thinks they got their data; in reality it's nonsense or attacker influenced.

The fix: fail closed. If the verification of whatever condition fails, you do not produce output. And, in a clear and detailed manner, you state the error:

try {
  return Buffer.concat([decipher.update(ciphertext), decipher.final()]);
} catch (err) {
  throw new Error(
    'Decryption failed — wrong password, or the file has been corrupted or tampered with.'
  );
}
Enter fullscreen mode Exit fullscreen mode

A secure tool either produces absolutely correct output or no output at all. This is the same as the lock that, when picked, remains jammed shut instead of springing open.


Putting it together: the file format

The right decisions give a design that almost completes itself. Each piece of material that is non-secret and necessary for decryption is included in the file; only the password is retained in the user's head. The layout of the vault in SecureVault is as follows:

[ MAGIC | salt | iv | authTag | ciphertext ]
   4      16    12     16        N bytes
Enter fullscreen mode Exit fullscreen mode
  • MAGIC — 4 signature bytes to recognize its own format to help ignore junk early.

  • salt — changes the key with the password (Mistake #2 fixed).

  • iv — changes the key with the password (Mistake #3 fixed).

  • authTag — changes the key with the password (Mistake #4 fixed).

  • ciphertext — changes the key with the password (Mistake #1 fixed by KDF).

Taking that format and reading it back means slicing the buffer in order and KDF the key with the password and salt and letting GCM check the tag and return it.


Properties understated: knowing if you accomplished the goal

Functional tests check "does encrypt-then-decrypt return the original?" Security tests check the properties an attacker cares about. These only matter:

  • Confidentiality: the ciphertext does not contain the plaintext bytes.

  • Wrong key rejected: using the wrong key should throw an error and never return garbage.

  • Tamper detection: modifying the ciphertext means decryption will throw an error.

  • IV uniqueness: the same input should never result in the same ciphertext.

Meeting those four challenges guarantees the three goals outlined are met.

Understanding Limitations

Successful security means you get to state what you don't protect. Even a well-constructed tool has limits and we should talk about them:

  • Handling passwords. The command line won’t usually protect a password. It will show up in the shell’s command history as a visible answer. Quality tools will prompt you for a password or fetch it from a secured vault.

  • Handle large files? Memory scales to the size of the file loaded. When a file is too big to fit into memory, you have to design a tool to load a file in segments instead.

  • File size & existence. Encryption will hide the contents of files, but not the size or existence of files.

These don't justify not making a quality product, though. An honest scope statement that introduces a security tool is likely to engender some amount of trust and goodwill.


Key Takeaways

Most insecure crypto tutorials stop at “It scrambles the bytes.” The boring reality of real crypto correctly applied isn't magic, it's just the consistent application of a handful of phenomena and design principles. These include, but are not limited to, key stretching, salting, IV reuse, authentication, and ensuring the system fails to an unwanted state.

It is worth noting that at no point throughout the design process are you required to think up new and inventive ciphers (which is typically a bad idea anyway). Everything you need is already at hand and battle tested. The only difference between novelty and security is knowing which pieces go where in a system and constructing everything correctly.

The entire SecureVault source code is concise enough that you can read it all at once. Since SecureVault is a fully functioning and cryptographically correct application, reading SecureVault is a great way to learn and internalize the patterns that will help you improve your own code. Building your own version will help you learn immensely. Deliberately implementing one of the known security flaws, and then observing the security test fail will help solidify the lesson you learned.

Top comments (0)