DEV Community: Daniel Fyhr

Simple Cache for AWS Secrets Manager

Daniel Fyhr — Sun, 17 Jul 2022 15:48:50 +0000

Introduction

If you need to access some secrets from AWS Secrets Manager, it's a good idea to cache the values. This way you will fetch them less frequently and save on costs. Unfortunately, it's not built-in in aws-sdk for NodeJS. Fortunately, it's pretty straightforward to implement.

In this post we will take a look at why it's a good idea and one way to do it when using AWS Lambda functions. The provided implementation is in TypeScript.

Why you should cache the values

Every external call is a risk and there are many things that can go wrong. The network is not reliable. I once hit the rate quota for fetching values and the service was just waiting for an answer, eventually timing out.

Retrieving a cached value is faster and you can save money with just a few lines of code. Not only do you save on calls to AWS Secrets Manager but you will also have shorter duration.

Strategy

The first time an AWS Lambda function is run it creates an execution environment if there isn't one already. When the execution is done that environment will remain available for some time for subsequent executions.

We can use this as a simple caching mechanism by creating an object in the environment. When we put values in that object, they can be accessed the next time the function is invoked.

Implementation

Let's break it down into two components. First a component for caching:

class SimpleCache {
  private cache: Record<string, string> = {};
  constructor(private readonly loadValue: (key: string) => Promise<string | undefined>) {}
  async get(key: string) {
    // if we find the value in the cache, return immediately
    if (this.cache[key]) {
      return this.cache[key];
    }
    // load it with the provided function
    const res = await this.loadValue(key);
    if (res == null) {
      return res;
    }
    // put the value in the cache and return.
    // The next time we need the value, we don't have to fetch it again.
    this.cache[key] = res;
    return res;
  }
}

Then a component for fetching the value of a secret with a given key:

import SecretsManager from 'aws-sdk/clients/secretsmanager';

const secretsClient = new SecretsManager();
const client = new SimpleCache((key) =>
  secretsClient
    .getSecretValue({ SecretId: key })
    .promise()
    .then((x) => x.SecretString),
);

Putting it all together:

import SecretsManager from 'aws-sdk/clients/secretsmanager';

class SimpleCache {
  private cache: Record<string, string> = {};
  constructor(private readonly loadValue: (key: string) => Promise<string | undefined>) {}
  async get(key: string) {
    if (this.cache[key]) {
      return this.cache[key];
    }
    const res = await this.loadValue(key);
    if (res == null) {
      return res;
    }
    this.cache[key] = res;
    return res;
  }
}

// When we create these two instances outside of the handler 
// function, they are only created the first time a new 
// execution environment is created. This allows us to use it as a cache.
const secretsClient = new SecretsManager();
const client = new SimpleCache((key) =>
  secretsClient
    .getSecretValue({ SecretId: key })
    .promise()
    .then((x) => x.SecretString),
);

export const handler = async () => {
  // the client instance will be reused across execution environments
  const secretValue = await client.get('MySecret');
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: secretValue,
    }),
  };
};

Additional usage

We can use the SimpleCache implementation above for other integrations. If we fetch some other static values over the network, we can use the same caching mechanism.

For example, if we decide to use aws-sdk v3 instead we can use the same SimpleCache but change the component related to SecretsManager. V3 has a nicer syntax where we don't have to mess around with .promise().

Don't put secrets in environment variables

You can resolve the values from AWS Secrets Manager during deployment and put them in the environment variables.

Unfortunately this means your secrets are available in plain text for attackers. It's one of the first place they would look. It has happened before and will probably happen again. Here's an example of an attack like that..

Conclusion

In this post we have covered why you should cache values from AWS Secrets Manager. It will save you money and make your code more reliable and performant. There's also an implementation of how to achieve this.

If you found this post helpful, please consider following me on here as well as on Twitter.

Thanks for reading!

How do you handle conflicts at work?

Daniel Fyhr — Wed, 25 May 2022 17:05:09 +0000

If you are working in a team, conflicts are bound to happen. It can be a small disagreement about tabs or spaces or something bigger.

Dealing with conflicts is an important soft skill to learn.

Please share a disagreement or an issue you had with a colleague and how you resolved it.

5 Useful TypeScript Features In VS Code

Daniel Fyhr — Sun, 24 Apr 2022 18:02:41 +0000

When you are programming you often do small repetitive tasks. You might be inserting delimiting characters in the right place or renaming variables. Here are five features in Visual Studio Code I find useful. They help with trivial chores and lets you focus on more important things.

1. Add missing properties

You have an interface and you need to populate an object with the fields. Use this code action to help with this tedious task. It works with types too.

mac: command .
win: ctrl .

2. Convert concatenated strings to template string

Template strings are easier to read. It's easy to forget a whitespace somewhere when concatenating strings. Get rid of them with this action.

mac: command .
win: ctrl .

See there was a missing whitespace in there! 👀

3. Add braces to function

Sometimes you need to change something or add a line to a short arrow function. I always make some small mistake when doing this by hand. It's either one { too many or few but with this feature you don't have to worry about it.

mac: command .
win: ctrl .

4. Convert to named function

Can't decide between function and const? Neither can anyone else but switching between them is super simple.

mac: command .
win: ctrl .

5. Rename

Last but not least, renaming things is a simple task that you often do. A good habit is to use the built in rename feature. It will update the name in all places at once, including other files.

mac: F2
win: F2

What's missing from this list?

Did you find this helpful? Do you think I missed something that every TypeScript VS Code user should know about?

Please feel free to comment with your suggestions! Thanks for reading.

Sending emails only once with AWS Step Functions

Daniel Fyhr — Sun, 03 Apr 2022 15:36:28 +0000

Most email delivery API:s will send two emails if you send two requests. This might be a problem since there are many ways a requests can be duplicated. Probably you don't want to send more emails than necessary. For example both SQS and EventBridge guarantees at least once delivery.

Let’s take a look at how we can use Step Functions and DynamoDB to avoid sending multiple emails. We will use Condition Expressions in DynamoDB to see if we have already processed an email. The basic flow of the state machine is:

Create a hash based on the email.
Try to save the hash in DynamoDB. If the item already exists, abort and do nothing.
If the item did not exist, proceed and send the email.

Multiple executions with the same input will not send additional emails. We have idempotency.

Create the hash

We want to create a short string representation of an email and save it in a database. Let’s use a hash function that creates a unique string based on sender, receiver, subject, and content. This will be the unique key in the database, which we will use to determine whether we have processed the email before or not.

const crypto = require('crypto');

exports.handler = async (email) => {
    const combined = `${email.from}${email.to}${email.subject}${email.content}`
    const hash = crypto.createHash('sha256').update(combined).digest('base64');
    return hash;
};

Save the item

We will then try to insert the hash in our database, with a Condition Expression: The item must not already exist. If the item already exists - indicating that we have already processed it - we want to catch this error and not send the email. If it doesn’t exist we proceed to send the email.

Send the email

There are many ways to send an email. Implement according to your email delivery API. Note that this step can be replaced with pretty much anything that you want to do only once. For example payments and orders.

exports.handler = async (email) => {
    // TODO implement send email
    return "sent email";
};

If the email has not been processed before, the execution should look like this:

Email was already processed

Catch the ConditionalCheckFailedException and use the Pass state in Step Functions. It's important to end in success state even though we actually didn't send the email this time. This is a characteristic of an idempotent API. Use the Fail state for all other errors.

If the email has already been sent, the execution should look like this:

Conclusion

That's pretty much it! The definition for the state machine is available here: https://gist.github.com/danielfyhr/4144dba260cc2bce1509d12cfd998664

Standard Workflows guarantee exactly once execution of each workflow step: https://aws.amazon.com/step-functions/faqs/