How to work around Vercel's 4kb environment restriction

larskarbo — Fri, 24 Jun 2022 12:16:03 +0000

At Layer3 we run our whole platform in a full-stack beautifully orchestrated environment on Vercel. All server-side and client-side code is written in TypeScript and share many of the modules and types.

Everything was going well, until one day…

Vercel has a 4kb environment variable limit. It is caused by the underlying AWS Lambda infrastructure, but whereas AWS has some solutions for proper secret management, Vercel basically says you need to roll your own.

At this time, there are so many pros of being on a platform like Vercel, and we save so much time by not having to set up complex cloud infrastructure on AWS.

So we decided to fix this.

Our solution focused on two things:

Secure management and deployment of secrets in preview and production
Keep the excellent dev experience of pulling development keys to local environments

Check out the example repository here: larskarbo/next-env-encrypt-decrypt

Meet Doppler — an environment manager

Doppler is a service that specializes on environment variable management. It sounded perfect for our usecase, they even have a Vercel integration!

However, we quickly realized that even though Doppler has a Vercel integration, it doesn’t help the 4kb problem at all. It actually just — kind of contributes to it... By adding more DOPPLER_ variables.

However, the Doppler interface and API is amazing, and we figured it we could make a working solution with some hacking.

Fetching secrets from Doppler instead of Vercel

Once you add all your environment variables on Doppler instead of Vercel, you can work around the 4kb restriction pretty easily by fetching secrets from Doppler instead of Vercel.

The only environment variable you need on Vercel is a Doppler token. The easiest way to add tokens for development, preview and production is to have both Vercel CLI and Doppler CLI installed and generate three different keys from the terminal:

echo -n "$(doppler configs tokens create vercel-gitops --config dev --plain)" | vercel env add DOPPLER_TOKEN development

echo -n "$(doppler configs tokens create vercel-gitops --config stg --plain)" | vercel env add DOPPLER_TOKEN preview

echo -n "$(doppler configs tokens create vercel-gitops --config prd --plain)" | vercel env add DOPPLER_TOKEN production

We’ll then make a script fetchSecrets.ts that fetches these variables at build time and writes them to .env.

import fs from "fs/promises";
import secrets from "@larskarbo/gitops-secrets";

async function main() {
  const payload = await secrets.providers.doppler.fetch();

  let envFile = "";

  Object.entries({
    ...payload,
  }).forEach(([key, value]) => {
    envFile += `${key}=${value}\n`;
  });

  envFile += `DOPPLER_TOKEN=${process.env.DOPPLER_TOKEN}\n`;

  await fs.writeFile(".env", envFile);
}

void main();

Changes in package.json:

"scripts": {
    ...
    "build": "npm run fetch-secrets && nextjs build",
    "fetch-secrets": "ts-node fetchSecrets.ts"
}

Yes, that's all you need.

In development, you’ll simply run npm run fetch-env. This flow doesn’t add a lot of moving parts and it feels very similar to the vercel env pull workflow.

Taking it a step further with encrypted secrets

Now that we are rolling our own secrets management, why not take it a step further and improve the security?

The current environment variable setup can be a security risk. A rogue npm package could dump all the freely available process.env variables and send it to a remote server. And remember, this could be the dependency of one of your dependencies too. Most npm apps have a boatload of dependencies when you look at the dependency tree, so the surface risk area might be bigger than you think.

Our goal will be to create a system where:

Secrets are always encrypted, both in transit and at rest.
Secrets are difficult to unintentionally leak when consumed by the final application.

Many platforms have sophisticated solutions for this, like AWS KMS and Docker Secrets. The idea is that these tools hold the secret in encrypted form and provides it to the application at runtime.

We’ll solve this in a simple and custom way, with some unique considerations:

We need NEXT_PUBLIC_ variables to be available in the environment.
We want to be able to override secrets with .env.local for our local dev environments.

Building on the Doppler setup, we'll add another environment variable to Vercel, SECRETS_KEY.

gen_key () { openssl rand -base64 32 }

gen_key | vercel env add SECRETS_KEY development
gen_key | vercel env add SECRETS_KEY preview
gen_key | vercel env add SECRETS_KEY production

Now we’ll make some changes to our fetch-secrets.ts script.

It needs to:

Fetch secrets from doppler.
Write all NEXT_PUBLIC_ vars to .env
Write all other secrets to a special file .encrypted-secrets

Commit this file into git like this, and then add it to .gitignore. This allows us to run the app without depending on the generated file.

Our super-charged fetch-secrets.ts looks like this:

import Cryptr from "cryptr";
import fs from "fs/promises";

import gitopsSecrets from "@larskarbo/gitops-secrets";
import { ENCRYPTED_SECRETS_FILE } from "../src/utils";

async function main() {
  const payload = await gitopsSecrets.providers.doppler.fetch();

  if (!process.env.SECRETS_KEY) {
    throw new Error("SECRETS_KEY is not set");
  }

  const cryptr = new Cryptr(process.env.SECRETS_KEY);

  const encryptedText = cryptr.encrypt(JSON.stringify(payload));

  await fs.writeFile(ENCRYPTED_SECRETS_FILE, encryptedText);

  let envFile = "";

  Object.entries({
    ...payload,
  })
    .filter(([key]) => key.startsWith("NEXT_PUBLIC_"))
    .forEach(([key, value]) => {
      envFile += `${key}=${value}\n`;
    });

  envFile += `DOPPLER_TOKEN=${process.env.DOPPLER_TOKEN}\n`;
  envFile += `SECRETS_KEY=${process.env.SECRETS_KEY}\n`;

  await fs.writeFile(".env", envFile);
}

void main();

Then we need to decrypt the secrets in the runtime code. We'll make a helper function for this.

let decryptedSecrets: null | {
  [key: string]: string;
} = null;

import { readFileSync } from "fs";

import Cryptr from "cryptr";
import path from "path";

export const ENCRYPTED_SECRETS_FILE = ".encrypted-secrets";

export const getSecret = (key: string) => {
  // in case you have some overrides in `.env.local`
  if (process.env.NODE_ENV === "development" && process.env[key]) {
    return process.env[key];
  }

  // only decrypt secrets the first time
  if (!decryptedSecrets) {
    if (!process.env.SECRETS_KEY) {
      return undefined;
    }

    const encryptedSecrets = readFileSync(
      path.join(process.cwd(), ENCRYPTED_SECRETS_FILE),
      "utf8"
    );
    const cryptr = new Cryptr(process.env.SECRETS_KEY);
    decryptedSecrets = JSON.parse(cryptr.decrypt(encryptedSecrets));
  }

  return decryptedSecrets?.[key];
};

Voila! Now you can use the secrets anywhere in your app like this:

// back-end
const apiKey = getSecret("API_KEY")

// front-end
const somePublicKey = process.env.NEXT_PUBLIC_KEY

Check out the working demo here: (link, github repo).

Conclusion

Vercel may have a 4kb environment restriction, but with some creative engineering, you might end up with a system that is more developer-friendly and secure than before.

This approach might be right if you are an early stage startup. When you get bigger, and have stricter requirements to managing confidential stuff, you’ll probably end up with a more complex cloud orchestrated infrastructure.

At Layer3, we use Vercel and Doppler to move fast. If you enjoyed this post and love the idea of building new types of applications that take advantage of the decentralised web, you should join our team!

The three Phases of Creating Useful Stuff on the Web

larskarbo — Sat, 31 Oct 2020 10:45:36 +0000

This model will make you more successful at creating useful stuff on the web.

It is simple and easy to understand.

The three Phases

I have a lot of failed projects that never got out of phase one (at least 150), I have some things that I released, but never took it further. And I have a few successful projects.

I have a lot of failed projects that never got out of phase one

By recognizing these phases and thinking rationally about them we will become better at creating successful projects.

Phase 1: Building

This is the phase where you start coding the project.

You have this amazing idea. Maybe you think you'll be able to finish it quickly.

You make a github repo and things start to take form.

If you release it, big or small, you go on to phase two ↓.

Phase 2: Releasing

You'll stay in phase two as long as you are still releasing and improving.

Some marketing and outreach might be necessary in this phase.

When you start getting traction and people start to use your product, you move on to phase 3 ↓.

Phase 3: Traction

Congratulations! You made something that is usable and valuable for people!

From here you can leave it, continue to work on it or start monetizing it.

Common pitfall #1: Never releasing

This is the most common pitfall.

You start building something, never get to the point where you can share it.

Sometimes, it is ok. If your idea wasn't that good anyways.

Think small, and minimum viable product, to quickly move on to phase 2

Common pitfall #2: Lose motivation after first release

I love building and shipping new things. Sometimes I get extremely motivated in the beginnings.

The next step is harder. The step to take the creation and make it into something better. It requires iteration and hard work, and doesn't give you instant gratification.

The idea didn't take off, and you are unsure whether it's any point in continuing.

I think it is important to learn that it might take more work before you find traction.

Your motivation is a bad indicator of whether you should work on something or not.

A note about business.

You are probably thinking of something that is missing.

What about validation before building anything? What about monetizing and scaling?

This model is solely based on the build aspect. It doesn't mean the business parts aren't important. Because they are.

Business parts are important, but not included in this model

Conclusion

Recognize the phases and the common pitfalls and you will have an easier time building useful stuff on the web.

Make sure to share this with anyone you think would like to hear it!

Which phase do your projects usually end up in? Let me know in the comments.

DEV Community: larskarbo

How to work around Vercel's 4kb environment restriction

Meet Doppler — an environment manager

Fetching secrets from Doppler instead of Vercel

Taking it a step further with encrypted secrets

Conclusion

The three Phases of Creating Useful Stuff on the Web

The three Phases

Phase 1: Building

Phase 2: Releasing

Phase 3: Traction

Common pitfall #1: Never releasing

Common pitfall #2: Lose motivation after first release

A note about business.

Conclusion