DEV Community: Ben Force

Lock Down Your Data: Implement Row-Level Security Policies in Supabase SQL

Ben Force — Thu, 25 Sep 2025 10:00:00 +0000

Data breaches are everywhere these days. The numbers aren't encouraging. The chart below (based on data from Statista) shows that data compromises since 2020 have almost tripled:

I realized that if I didn't want my product ending up on lists like this, I'd need multiple security measures and multiple layers. If one gets breached, there should be another protecting user data.

Row-level security (RLS) in Supabase lets me define security policies directly in my database. These policies restrict what users can access, so even if there's a vulnerability in my API or web app that could expose data to bad actors, the database itself acts as a final line of defense.

In this guide, I'll walk through how RLS policies work, how to construct them, some advanced use cases, and how to write tests to make sure your policies are locked down before going to production.

Understanding RLS

RLS is a feature of Supabase's underlying database provider, Postgres. It allows you to define access policies for different types of queries on a table. These policies are executed every time a query is executed against that table. Normally, you would restrict access to data in an API endpoint and add the restrictions to the query directly, but adding the policy to the database ensures that the policy will always run.

RLS can provide defense in depth, a security concept where there are multiple layers of security. It's also enforced on the database, which means it can limit the blast radius of a leaked access token, restricting access to only the data belonging to the user who originally received that token.

In addition to providing access control at a lower level than the API, RLS works by essentially adding a WHERE clause to queries on a specific table. This can greatly simplify queries for multitenant apps (where companies have multiple users that can access their data) as you can write queries without thinking about which tenant is logged in. You can also allow users to write custom queries for dashboards or their own internal tools, and the RLS policies will ensure they only get data that they're supposed to see.

RLS Example Database: Recipe-Sharing Web App

This article uses a recipe-sharing web application as an example to explore creating a variety of different policies. The comments and recipes tables both reference the users table by its user_id field. This field is used in policies to make sure the correct user is given access to modify or delete recipes and comments. The simplified schema diagram below will help you follow along:

Using RLS in Supabase

When you create a table through the Supabase dashboard, RLS is enabled by default. If you aren't building your application through ClickOps, though, you'll probably be creating your tables using an SQL query. The following example creates a table named recipes and enables RLS:

-- Create the recipes table
CREATE TABLE recipes (
    ...
);

-- Enable RLS for recipes table
ALTER TABLE recipes ENABLE ROW LEVEL SECURITY;

When you've enabled RLS but haven't created the policies that it will apply, it will give authenticated users full access to your database and block all access from anonymous users. To understand this, you need to know that Supabase maps every request to one of two roles:

anon (anonymous): The user is not logged in.
authenticated: The user is logged in.

When you have a table with RLS enabled but no policies created, Supabase will block all requests made using the anon role by default. Blocking anonymous requests by default will help prevent accidental data leaks by not giving public access to your whole table.

Basic Components of SQL Policies for RLS

If you want to move beyond blocking anonymous access and allowing any authenticated user to access any data, you'll need to define some policies.

There are five parts to an RLS policy that you'll use to configure access to your database. You can get more detailed information from the Postgres Create Policy documentation.

Name
Table
Command
Role
Conditions

Name

The policy name is used to refer to the policy after it's created. It's a good idea to create a naming convention to be used by all policies and be descriptive about what the policy should do. Using a consistent naming policy will make it easier to maintain and test as your database grows.

The policy name is provided immediately after the CREATE POLICY statement. Below is an example:

CREATE POLICY "Allow anyone to view public recipes"

Table

You've told Postgres that you want to create a policy and what it's called, but it also needs to be associated with a table. Using the ON keyword assigns the policy to a specific table:

ON recipes

Command

You also need to define which operations this policy applies to. You can specify the standard CRUD operations: SELECT, INSERT, UPDATE, or DELETE.

By default, policies use ALL and apply the policy to every operation type, but writing a policy for each operation gives you more control over how users can access or modify data. For example, you might allow users to UPDATE their own profile information while restricting INSERT and DELETE operations to system-level processes only.

To specify which command your policy applies to, use the FOR keyword followed by a single command or the ALL keyword:

FOR SELECT

Role

As mentioned above, Supabase comes with two roles by default: anon and authenticated. Supabase uses Postgres as its database provider, which has its own PUBLIC role applied to all other roles. By default, new policies will be applied to the Postgres PUBLIC role, which, in turn, applies to both the anon and authenticated roles.

Most of the time, you'll want to use the authenticated role for your policies, which ensures that an authenticated user is making the request.

If you want to create a policy that applies only to authenticated users, specify it with the TO keyword:

TO authenticated

Conditions

Every part of the CREATE POLICY statement up to this point (besides giving it a name) has been to determine when to apply the policy. The next step is defining the actual policy with some conditions.

There are two different condition types that you can add to a policy. You can add a filter to restrict which row can be selected, updated, or deleted, and you add a check on data being passed to inserts and updates.

Filtering Results (USING):

Restricting which rows a command can read or update is done with the USING keyword followed by a Boolean expression. For example, to give authenticated users access to recipes they created, you can use the following expression.

USING (user_id = (SELECT auth.uid()))

You may have noticed the auth.uid() function in the snippet above. This is a helper function provided by Supabase that, as you probably guessed, returns the ID of the user making the request. Supabase also provides another helper ,auth.jwt(), that provides more details about the user.

Validating Inputs (WITH CHECK):

To restrict the data being saved to the database through INSERT and UPDATE operations, use WITH CHECK followed by a Boolean expression.

In the recipe app example, you would want to add a check to validate whether the user_id field matches the ID of the user sending the request.

WITH CHECK (user_id = (SELECT auth.uid()))

To clarify when you can use each type of condition, here's a table showing when they apply:

Command	`USING`	`WITH CHECK`
`SELECT`	Yes	No
`DELETE`	Yes	No
`INSERT`	No	Yes
`UPDATE`	Yes	Yes

Creating a `SELECT` Policy

Now that you've seen the basic components of an RLS policy, you can put it all together to create an actual policy. The example application needs to allow authenticated users to view recipes that they've created.

On a SELECT policy, there's no data to validate, so you don't need a WITH CHECK, just a filter with the USING statement:

CREATE POLICY "Allow authenticated users to view their own recipes" ON recipes
    FOR SELECT
    TO authenticated
    USING (user_id = (SELECT auth.uid()))

Using auth.uid() inside a SELECT statement like this ensures the function is only executed once and cached instead of being executed for every row.

Creating `INSERT`/`UPDATE` Policies

Your users can now see all of their own recipes, but you'll also need a policy to stop them from accessing or altering data they shouldn't—for example, creating a recipe with a different user's ID or a null user ID.

INSERT and UPDATE policies use the WITH CHECK to verify the data being sent is valid. In this case, you just need to make sure the user_id matches the ID of the currently authenticated user.

CREATE POLICY "Allow authenticated users to create recipes" ON recipes
    FOR INSERT
    TO authenticated
    WITH CHECK (user_id = (SELECT auth.uid()))

Advanced RLS Policies for Real-World Use Cases

Beyond restricting a user to viewing items that they created, you may need more advanced policies—for example, if you're using a multitenant application or require role-based access or conditional policies.

Multitenant Applications

If you have a multitenant application, you need to restrict access to the user's tenant so they can't browse your other customers' data. To prevent this, you'll create records for the tenants and tie each user to a specific tenant. To set up a multitenant schema, create a tenant table and then use Supabase's updateUserById to set tenant_id in the user's app_metadata.

Since the tenant_id is a restricted field that users can't modify, you should store it in the app_metadata. This ensures that users can't insert some other tenant_id in their user metadata and start browsing other tenants' data.

Next, similar to adding a user_id to recipe records to restrict access, you need to add a tenant_id to the tables instead. This will allow you to connect records to a tenant instead of a specific user. In the policy check, add the tenant_id in app_metadata. Ideally, you can create an auth.tenant_id() function that queries the user's JSON Web Token (JWT) claims to get the tenant_id. This function can be reused in your policies:

CREATE OR REPLACE FUNCTION auth.tenant_id()
RETURNS text
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN (SELECT auth.jwt() -> 'app_metadata' ->> 'tenant_id');
END;
$$;

A more detailed guide was published by Ryan O'Neill on his Substack.

Role-Based Access Control (RBAC)

If you have an application where a lot of users will be accessing the same data but some users will have more access than others, then you'll want to set up role-based access. While Supabase doesn't have support for this built in.

To set up RBAC with Supabase, you first create a user_roles and a role_permissions table. The role_permissions table defines roles and their permissions using an enum type, and the user_roles table ties a user_id to an app_role enum. Once these tables are created, you can define your roles and permissions and assign roles to users.

To enforce these roles, you need to ensure they're available when a user runs a query. To do that, create a custom-access token hook. This hook will be executed when Supabase generates a user access token. Overriding this hook allows you to query user_roles and add a claim to the token.

Now that you have the user's role in the JWT, you can access it when writing RLS policies:

select (auth.jwt() ->> 'user_role') into user_role

Conditional Policies

There may be times when you want to restrict data access in other ways besides simply assigning which user/role has access to which records. For example, you can make records noneditable after a certain period of time or use pg_catalog.inet_client_addr() to restrict access to your company's network. In the recipe example above, public recipes have a published_at timestamp. You can create a conditional policy to keep these recipes from being visible until after the timestamp has passed:

CREATE POLICY "Allow anonymous users to view public recipes" ON recipes
    FOR SELECT
    TO anon
    USING (is_public = true AND (published_at IS NULL OR published_at <= (SELECT NOW())))

Testing and Validating RLS Implementation

One of the trade-offs of using RLS policies is that you're adding complexity to every query on a table that isn't obvious, which can impact performance. Also, if you're writing queries using the default postgres role in Supabase's query editor, the RLS policies won't be applied. To troubleshoot these issues, use the Postgres EXPLAIN ANALYZE command to see how the RLS policies are being executed. If you want to actually see the policy being executed, you'll need to run the query as an application user.

While you're writing your RLS policies, you can use Supabase's impersonate feature in the dashboard to run queries as specific users. To do this, open the SQL Editor in the Supabase dashboard and click on the Role drop-down option next to the green Run button. A dialog will pop up that allows you to use the anon role or select any user for the authenticated role:

Now you can run any query and it'll use a JWT for the user you selected. Try it out using SELECT auth.jwt();.

Validating with Unit Tests

Once you've written your policies, you can dive into setting up tests using pgTAP and running the supabase test db command. This combination can be used to write unit tests so you can make sure your policies work as expected before deploying them to production.

The most basic test that you may want to add is to verify which policies are applied to a certain table. To verify the RLS policies that are applied to the recipes table, create a new test file using the command below:

supabase test new recipes_rls

This will create the file supabase/tests/recipes_rls_test.sql, with some boilerplate code to roll back any changes made during testing. Next, add a call to the policies_are function. It takes the schema (public), the table name (recipes), and an array of policy names that should be applied to the table. Add the following SELECT statement to recipes_rls_test.sql:

SELECT policies_are(
  'public',
  'recipes',
  ARRAY [
    'Allow anonymous users to view published public recipes',
    'Allow authenticated users to create recipes',
    'Allow authenticated users to delete their own recipes',
    'Allow authenticated users to update their own recipes',
    'Allow authenticated users to view accessible recipes',
    'Allow moderators to delete public published recipes',
    'Allow moderators to update public published recipes'
  ]
);

Now, to run your database tests, execute supabase test db and you'll see all of your tests pass successfully:

$ supabase test db
Connecting to local database...
./recipes_rls_test.sql .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.01 cusr  0.00 csys =  0.04 CPU)
Result: PASS

You can write additional tests to verify that your policies work correctly for different user roles and database operations. You can also write tests that try different CRUD operations to test your policies. For more details, see Supabase's documentation.

Troubleshooting Common Issues

If you get your policies set up and you start running into issues it can be difficult to troubleshoot. It's especially frustrating when you run a query in the Supabase console and verify that it's working, but it just returns nothing (or an error) when your app tries to run the same query. This is a typical RLS misconfiguration.

This section covers some of the more common RLS configuration issues and how to fix them.

Anonymous Data Access Issues

If you have an app that lets users view some data anonymously, like publicly shared recipes, there are a few things that may cause your anonymous queries to return empty.

The first thing you should check is to make sure you created a role TO anon. Without this, Supabase will use its default policy which will deny all access.

If you do have a policy that allows anon to access the data that you're after, make sure it's not using auth.uid(). When auth.uid() is executed for an anonymous user, it will return NULL. So, if your USING query is checking that auth.uid() equals a table column, then you won't get any results back from your query.

Server-Side Rendering

If you're using a server-side rendering framework like Next.js, you may run into an issue where you get empty results for logged in users. This is a common issue that shows up when you don't forward the user's authentication headers to Supabase.

When this happens, Supabase will see an unauthenticated request and apply the anon policies to the request.

The solution is to make sure you're making requests to Supabase on behalf of the user. This can be easily done using Supabase's SSR library.

Recursive RLS Policies

When a policy's query looks up another table, that table's policies will be applied to the USING query of the first policy. This can go several levels deep, and will cause the policy to fail if this chain of policies references a policy that's already being executed. This will result in the error ERROR: 42P17: infinite recursion detected in policy for relation "teams".

To illustrate this issue I'll use a simple database schema:

erDiagram
    users ||--o{ team_membership : "is member of"
    teams ||--o{ team_membership : "has member"

The teams table has a policy that ensures a user can only see teams they are a member of, by querying the team_membership table. Meanwhile, team_membership has a policy that checks the referenced team by querying the teams table again, creating a loop.

graph TD;
    QT(Query teams)
    QTM[Query team_membership]

    QT--Is User a Team Member?-->QTM
    QTM--Does Team Exist?-->QT

Fortunately, this will cause an error that explicitly notifies you of the infinite recursion. Once you see this error, you'll want to refactor your RLS policy to call a security definer function.

Security definer functions are special functions that run with the permissions of the user that created the function. This means you can refactor your RLS policies to call a function, and that function will avoid any future RLS policy checks.

Here's an example function that would resolve the recursive issue in the above example:

CREATE OR REPLACE FUNCTION is_member_of_team(team_id_to_check bigint)
RETURNS boolean
LANGUAGE plpgsql
SECURITY DEFINER -- Run as function creator
AS $$
BEGIN
  RETURN EXISTS (
    SELECT 1 FROM team_memberships
    WHERE team_id = team_id_to_check
    AND user_id = auth.uid()
  );
END
$$;

CREATE POLICY "Users can see teams they are members of"
ON teams FOR SELECT
TO authenticated
USING ( is_member_of_team(id) ); -- No more recursion!

For more details, see the Supabase documentation.

Conclusion

Well-written RLS policies can prevent data leaks, ensuring your customers are safe. After reading this article, you should now know how these policies are built, some advanced use cases, and how to write tests to ensure they're working as expected.

Remember that RLS policies add an extra layer of security but shouldn't be your only defense. They work best as part of a wider security strategy that includes proper authentication, input validation, and regular security audits. As your application grows and evolves, make sure to regularly review and update your policies to ensure they continue to protect your users' data effectively.

Optimizing Nest-Commander: User Inputs

Ben Force — Tue, 26 Sep 2023 14:57:47 +0000

In my previous article, you created a NestJS CLI application. It was a very basic starting point that printed a cow saying a static message in the console.

But what if your users demand the ability to display custom messages? That's what you're going to add in this article. The new and improved application will display the same message by default, or any string that you pass in with the --message argument.

Adding a Message Option

The easiest way to add an option is to add a parser function for a parameter and add the @Option decorator to it.

In the cow-say.command.ts file, add the following method to your CowSayCommand class.

@Option({
  flags: '--message <message>',
  description: 'The message we want to see the cow say',
})
parseMessage(value: string) {
  return value;
}

Decorating It

If you've used the commander library before, this should look familiar. The <message> tells commander that if the --message option is provided, the user doesn't need to provide a parameter after it. If the value was required it would be [message] instead.

The description parameter is used when the user displays help for a specific command. The description will be displayed in a column to the right of every available option.

The Parser

The @Option decorator is applied to a parser function, in this case parseMessage. The function takes in a string and needs to convert it to whatever type you'll want at runtime. If you had a --count <count> parameter, you'd need to convert a string to an integer.

Since the message should be a string, you just need to return it. Nothing fancy here.

Using the Options

To use the option, you need to accept two parameters in the run method. The first is an array of parameters that were passed in, and the second is an options object that includes all of the parsed options. Since you aren't using the parameters you can just ignore it.

async run(_passedParams: string[], options: { message?: string }): Promise<void> {
  const message = options.message ?? 'Hello World!';

  console.log(cowsay.say({ text: message }));
}

Awesome, now if you run the program with a --message "Foo Bar" parameter, the cow will be saying "Foo Bar".

Using Inquirer

Using parsers like in the previous step works great for smaller commands. It has the advantage of being simple. But what if you would like to gracefully ask the user to enter any parameters they forgot to provide? For that, you can use the InquirerService.

The first thing you need to do is define the types of inputs that will be used for each parameter. That's done with a class decorated with @QuestionSet and parsing methods decorated with @Question.

Create a cow-say.questions.ts file and add the following to it.

import { Question, QuestionSet } from 'nest-commander';

@QuestionSet({
  name: 'cowSay',
})
export class CowSayQuestions {
  @Question({
    type: 'input',
    name: 'message',
    message: 'What does the cow say?',
  })
  parseMessage(value: string) {
    return value;
  }
}

Now add this class as a provider to your module.

...
import { CowSayQuestions } from './cow-say.questions';

@Module({
  providers: [CowSayCommand, CowSayQuestions],
})
export class CowSayModule {}

Now the message parameter needs to be updated to be mandatory in the command.

@Option({
    flags: '--message [message]',
    description: 'The message we want to see the cow say',
  })

Next, inject the inquirer service into the command.

constructor(private readonly inquiererService: InquirerService) {
  super();
}

Finally, update the run method and use the inquirer service to update the options parameter.

options = await this.inquiererService.ask('cowSay', options);

console.log(cowsay.say({ text: options.message }));

Now, when you run the command if you don't pass in the --message option the application will ask you to provide it.

❯ npm run start:cli   

> my-cli-project@0.0.1 start:cli
> ts-node src/main.ts

? What does the cow say? Mooooooo
 __________
< Mooooooo >
 ----------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Summary

In this article, you expanded upon the basic NestJS CLI application created in the previous installment. You learned how to add arguments with an option parser function and used the @Option decorator. You also learned how to use the InquirerService to gracefully prompt users to enter missing parameters and integrated it into your command, enhancing user interaction.

How to Make a CLI App with NestJS: Step-by-Step

Ben Force — Fri, 15 Sep 2023 20:40:44 +0000

I recently started a role on a company's developer experience team. One of the responsibilities that comes along with this is building larger CLI applications.

As I wrote previously, I use the commander library when writing DevOps scripts for my NodeJS projects. This is great for smaller utilities that only a couple of people will use, but what about if it needs to interact with multiple services and have lots of nested commands?

As it turns out NestJS supports this using the same commander library that I used previously. In this article, I'll show you how to create a basic CLI using this setup.

Creating the Project

Creating the CLI project is the same as creating an API with NestJS. Simply follow the basic workflow on nestjs website.

npm i -g @nestjs/cli
nest new my-cli-project
cd my-cli-project

Setup `nest-commander`

The nest-commander library provides all the code that you'll need to create your commander CLI in the background. Like every other node dependency, the first step is to install the package.

npm i nest-commander

Creating a Command

Just like with normal nestjs development, we need to create a module that our command will live in.

Nest-Commander provides some tools that plug into the nest cli. I've tried using them but wasn't able to get them to work. You may have better luck, but I'll continue with the manual process.

nest g module CowSay

Normally you would create a controller or resolver in your module using the nest cli. Since we're creating a cli, we want to create a Command instead. Create a cow-say.command.ts file in the src/cow-say folder and open it.

Each command that you create will extend the CommandRunner class and use the @Command() decorator. In the cow-say.command.ts file that you just created, add the following.

import { Command, CommandRunner } from 'nest-commander';

@Command({
  name: 'cowsay',
  options: {
    isDefault: true,
  },
})
export class CowSayCommand extends CommandRunner {
  async run(): Promise<void> {
  }
}

To get this command to display something, import the cowsay library.

import * as cowsay from 'cowsay';

You'll need to install it too...

npm i cowsay

...and update the run method in cow-say.command.ts

async run(): Promise<void> {
  console.log(cowsay.say({ text: 'Hello World!' }));
}

Wiring Everything Together

Now that the command is created, you need to register it as part of the module. Open the cow-say.module.ts file and add CowSayCommand as a provider.

import { Module } from '@nestjs/common';
import { CowSayCommand } from './cow-say.command';

@Module({
  providers: [CowSayCommand],
})
export class CowSayModule {}

Tell NestJS that it's not a server

Now comes the tricky part. The nest project that you created is setup to create a REST API by default, but you don't need that. So delete the app service and controller.

rm app.service.ts
rm app.controller.ts

Next, update the AppModule to import the CowSay module.

import { Module } from '@nestjs/common';
import { CowSayModule } from './cow-say/cow-say.module';

@Module({
  imports: [CowSayModule],
})
export class AppModule {}

Finally, you need to update the main.ts file. You'll change the bootstrap function to use CommandFactory instead of NestFactory.

import { CommandFactory } from 'nest-commander';
import { AppModule } from './app.module';

async function bootstrap() {
  await CommandFactory.run(AppModule);
}
bootstrap();

Running It

I like to have the option of running my CLI tools without having to run a TypeScript build step. This helps speed up development.

To run your CLI without building it, you'll be using the ts-node package. To get started install it as a development dependency.

npm i -D ts-node

Now add a new script to package.json

"start:cli": "ts-node src/main.ts"

...and you can test your CLI by running the script

❯ npm run start:cli

> my-cli-project@0.0.1 start:cli
> ts-node src/main.ts

 ______________
< Hello World! >
 --------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Summary

This article showed how to get started building your very first NestJS CLI application. Taking this approach can help you build a larger application that is maintainable. In future articles I'll be introducing you to more advanced features of this setup.

Get All Results from DynamoDB Queries Easily!

Ben Force — Wed, 01 Mar 2023 21:32:34 +0000

A DynamoDB query can only return 1MB of results. So, how can you get all of the results of a query if it's bigger than that? There are two ways: the simple way and the cool way.

How DynamoDB Query Pagination Works

Before we start getting an insane number of records, let's back up and talk about how pagination works in DynamoDB. When a query has more records than it can return, either because you set the Limit property or there's more than 1MB of results, Dynamo will return all of the records that it can along with a property LastEvaluatedKey.

As long as the LastEvaluatedKey property is set in the query results, it means that you still have results left to get. If you want to get those results, update the original query parameters so that ExclusiveStartKey is set to LastEvaluatedKey and run the query again.

The Simple Way

The simple way is the most obvious: use a do loop. Make the condition check to see if the ExclusiveStartKey was set. If it is set, then you know there are more records, and you should run the query again.

export const getAllResults = async <T>(
    db: AWS.DynamoDB.DocumentClient,
    query: AWS.DynamoDB.DocumentClient.QueryInput
  ): Promise<Array<T>> => {

  const result = [] as Array<T>;
  console.info(`Querying Dynamo`, query.ExpressionAttributeValues);

  let queryResult: AWS.DynamoDB.DocumentClient.QueryOutput;
  do {
    queryResult = await db.query(query).promise();
    console.info(`Query Result`, queryResult);
    query.ExclusiveStartKey = queryResult?.LastEvaluatedKey;

    queryResult.Items?.forEach(item => result.push(item as T));
  } while (query.ExclusiveStartKey);

  console.info(
    `Found ${result.length} results for ${query.KeyConditionExpression}`,
    inspect(query.ExpressionAttributeValues)
  );

  return result;
};

Using an Async Generator

The problem with a do-loop is that all of the results will be stored in memory. If you have a lot of records that could be a problem. The fancier way to do this is with a node.js async generator.

An async generator is a special type of iterator that allows you to yield values as they're retrieved. The following code sample creates the DynamoDB async generator.

export async function* getAllResults<T>(
    db: AWS.DynamoDB.DocumentClient,
    query: AWS.DynamoDB.DocumentClient.QueryInput
  ) {
  console.info(`Querying Dynamo`, query.ExpressionAttributeValues);

  let queryResult: AWS.DynamoDB.DocumentClient.QueryOutput;
  do {
    queryResult = await db.query(query).promise();
    console.info(`Query Result`, queryResult);
    query.ExclusiveStartKey = queryResult?.LastEvaluatedKey;

    if(queryResult.Items) {
      yield* queryResult.Items;
    }
  } while (query.ExclusiveStartKey);
};

Calling It

Now that the generator is created, you can call it using the following awkward syntax.

for await (const record of getAllResults(db, query)) {
  console.info(record);
}

Conclusion

To conclude, if you have a DynamoDB query that returns more than 1MB of results, the simple way to handle pagination is to use a do-loop, and the more efficient way is to use an async generator.

My Node.js Toolkit for DevOps Scripting

Ben Force — Fri, 03 Feb 2023 06:00:00 +0000

Whether you need to build lambda code or hydrate a test database, most projects that you work on will involve some piece of DevOps scripting. If your application is using Node, there's no reason not to use it in your scripting too. This makes it easier for other Node developers working on the project to maintain the scripts that support it. With just a few packages you can make a great script that other developers will appreciate.

Running with Types

The first step of any scripting setup is getting your code to run. I could just write pure JavaScript and use Node to execute it directly, but I like the safety that I get with TypeScript. Also, since all of my application code is written in TypeScript, there are a lot of times that I will need to import a controller that I wrote for the application in my script.

As I described in my previous article Running TypeScript without Compiling, ts-node was my go-to solution for this for a long time. I've since started using esbuild to transpile all of my TypeScript and have discovered TypeScript Execute, which is similar to ts-node except that it uses esbuild under the hood.

Now, I globally install tsx:

npm i -g tsx

then I can just add a shebang to the top of the script

#!/usr/bin/env tsx

and finally, mark the file as executable

chmod +x ./script.ts

Parsing Parameters

Once the script is running, I'll probably need some sort of parameters. If I'm in a CI/CD environment, most of the parameters will probably come from environment variables so that I can update them without updating the pipeline definition. If I'm running the script locally, however, I'll need to be able to supply parameters manually or override them.

For this, I use the commander package because it's easy to use and, if I need to expand later, it supports adding multiple commands.

Defining Options

To get started, import the program and define some options. The nice thing is that I can specify an environment variable as the default value for an option, so I know whatever value is passed through that option is the right one.

import { program } from 'commander';

program
    .option('--someFlag')
    .option('--foo <foo>',
                        'A foo value',
                        process.env.FOO_VALUE);

Taking Action

Now that the options are defined, it's time to define what happens with those options. This is done by calling action on the builder. Usually, I just add an action to the end of my option definitions, so the above sample would become:

import { program } from 'commander';
import { inspect } from 'util';

interface CommandOptions {
    someFlag: boolean;
    foo: string;
}

program
    .option('--someFlag')
    .option('--foo <foo>',
                        'A foo value',
                        process.env.FOO_VALUE)
    .action(async (options: CommandOptions) => {
            console.info(inspect(options));

            await someCommand(options);
    });

Getting things started

Now that everything is defined, you have to kick off the actual program. The default function to use is parse(). If you're cool and return a Promise from the action function, however, you'll need to call parseAsync(). This function will grab the values in process.argv, parse them, call the appropriate action function that you defined, and wait for its result.

program.parseAsync()
    .then(() => console.info('🎉 Done!'));

Watching it Work

There are two types of scripts that I hate: ones littered with console.info() calls, and the ones that don't show any progress. If the script will finish in a short amount of time (read under a couple of seconds) then I think it's okay to just output a result at the end. For most scripts, however, there are multiple steps and it can take a little while.

To reassure me that everything is progressing, I define the steps of the script with listr. I know it hasn't been updated in 4 years as of this writing, but it just works. It displays a list of steps with a spinner and check marks when running locally, and it prints out logs when running in a CI/CD environment.

Defining Tasks

To use listr you provide it a list of steps. The task can return a Promise or a new Listr instance with sub-tasks. To define the list, create a new instance of Listr and pass in an array of task objects, each with at least a title and task property.

The task function accepts a context parameter that is passed through every step. The initial values for the context will be passed in when we execute the task list too, so for now we can assume it'll just be the CommandOptions from earlier.

const tasks = new Listr<CommandOptions>([
    {
        title: 'First Step',
        async task(ctx) {
            // TODO: Actually do something 🙄
        }
    }
]);

Skipping Tasks

What if you want to skip a task in some cases? You can provide a skip function in the task definition. The skip function gets the context parameter and returns a boolean (or a Promise<boolean>) which tells listr if the task should get skipped or not. Makes sense.

So if you wanted to skip that first step if the someFlag variable is true, you would just return it from the skip function.

const tasks = new Listr<CommandOptions>([
    {
        title: 'First Step',
        skip: (ctx) => ctx.someFlag, // Stop 🛑
        async task(ctx) {
            // TODO: Actually do something 🙄 or not? 🤷
        }
    }
]);

Parallel Processing

One of my favorite features of Listr is that I can map an array of items to create a list of sub-tasks and then run them in parallel. To run Listr tasks in parallel, pass in an options object after the list of tasks and set the concurrent property to true.

const tasks = new Listr([
    {
        title: 'Get Items',
        async task(ctx) {
            ctx.items = await getItems();
        }
    },
    {
        title: 'Process Items',
        task: (ctx) => new Listr(
            ctx.items.map(processItem),
            { concurrent: true }
        )
    }
]);

Running the Tasks

Once the task list is defined, you need to tell listr to run it and pass in the initial context value. This couldn't be easier, just call the run method on your listr instance and pass in the context. It returns a Promise with the final context, so you'll want to await the result.

// Imagine all the options calls from before... 
.action(async (options: CommandOptions) => {
    const result = await tasks.run(options);
    displayResult(result);
});

Summary

Now you have a few essentials that you can use. Next time you find yourself slogging through a manual process to build some code for the 15th time, try throwing a script together.

Getting Error Responses from Axios

Ben Force — Wed, 01 Feb 2023 14:56:17 +0000

If you search for libraries to make API calls from Node.js, chances are you'll run into the Axios package. It's easy to use, well documented, uses promises, and it works in browsers too.

When things go wrong

The main complaint from other developers is that when they make a request and an error response is returned, it's not obvious how to get information about the error. Usually, this happens when they don't catch the exception and expect the response to include the status code.

const response = await axios.get(SOME_URL);

if(response.status === 404) {
    console.error(`${SOME_URL} not found!`);
}

This approach won't work because Axios, by default, throws an exception for any status code that isn't 2xx. There is a workaround though.

Making it more responsive

You can tell Axios that you want to see every response regardless of if it's an error or not using the request options. You need to provide a validateStatus method that always returns true. This method is passed to Axios in the request options.

const response = await axios.get(SOME_URL, {
    validateStatus: () => true,
});

if(response.status === 404) {
    console.error(`${SOME_URL} not found!`);
}

Now you'll reach the if statement as expected.

There's a `.catch()`

If you look at the Axios documentation on error handling, you're supposed to catch the exception that it throws. You can use the response property on the exception to determine what status was returned. Finally, you can gracefully return some value and continue work.

const contents = await axios.get(SOME_URL)
    .then((response) => response.data)
    .catch((error) => {
        if(error.response?.status === 404) {
            console.error(`Could not find ${SOME_URL}`, response.data);
            return undefined;
        }

        // Something unexpected
        throw error;
    });

if(!contents) return;

Summary

You've seen a couple of ways to deal with error responses when making HTTP calls with Axios. The important thing to remember is that it will throw an error for non-2xx responses and the response property on the thrown exception will contain all of the data that you would normally get back (status, body, etc).

Easy Test Deployments with Terraform

Ben Force — Mon, 22 Aug 2022 13:51:22 +0000

Recently I was forced into using Terraform to define infrastructure. I'm so used to CDK and being able to adjust everything through context settings that I quickly ran into a problem: how do I deploy a copy of my project to test it?

Editing Files Directly

All I needed to update was the backend and my AWS profile. So the first way I tried was pretty obvious, edit the main terraform file where the backend and AWS provider is defined. The obvious problem is that I make frequent commits and usually just do git commit -a -m .... So I knew those changes would run in a pipeline and blow it up.

What About Variables?

My next idea was to use variables that would work for the AWS profile. The problem is the backend block needed to be set to local, and you can't pass variables into block labels.

Override Files

It turns out that HashiCorp has a solution for this: Override Files. They essentially work the same as my first method of editing the terraform file, but it's done in a special file to keep it out of the repository.

Add *override.tf to your .gitignore file, then add a file next to the one you want to update with the same name, but add _override to the end. For example, I created the file terraform_override.tf to override settings in terraform.tf.

When creating the file, you only have to define the properties you want to be updated. Continuing my example, I created the following override file:

terraform {
  backend "local" {}
}

provider "aws" {
  profile = "my-aws-profile"
}

Summary

Override files allow you to update terraform settings for local development easily. Since they're stored in files that can be easily added to .gitignore, you don't have to worry about them ending up in your repository and running on a pipeline.

CDK Integration Testing

Ben Force — Fri, 19 Nov 2021 14:15:04 +0000

Integration tests are challenging in a standard application. They're even more challenging in a decoupled, asynchronous environment. If you're running them in a pipeline, you have to figure out all of the resource ARNs and make sure you have permission to access each of them. Also, what happens if your integration tests fail? Do you send a notification so someone can manually redeploy the previous version?

Running integration tests in a CloudFormation custom resource simplifies a few of these issues. You can use references to other resources in the stack to get the required ARNs. You can use those same references to build execution rules for the integration test. Finally, if the integration tests fail, then CloudFormation will automatically roll back the deployment.

An example project can be found in the GitHub project theBenForce/cdk-integration-tests

How does it work?

The idea came from the article Testing the Async Cloud with AWS CDK. Essentially it comes down to leveraging two features of CloudFormation: CustomResources and automatic rollbacks. Instead of creating an actual resource with your custom resource handler, you're going to execute your integration tests.

CloudFormation Custom Resources

A custom resource is an escape hatch in CloudFormation that allows you to manage external resources from your IaC template. The custom resource accepts the ARN of a lambda to be called, and any properties that you want to pass in.

At deployment time, CloudFormation calls your lambda with all of the parameters supplied in the template and a few additional properties supplied by CloudFormation. The extra properties include the action to take (create, update, or delete) and a presigned S3 URL.

When your function has finished execution, it needs to upload its result to the presigned URL that CloudFormation passed in. In addition to the final status, your lambda can add any output parameters that you want to reference within your CloudFormation template.

UVU Test Runner

The UVU test runner is a good library for running your integration tests in a lambda. It's lightweight and simple to use. It doesn't currently provide a programmatic interface for running tests, but that is being actively developed.

ℹ️ Since all you need from the test runner is to know if the tests passed or failed, you won't need anything that advanced. The zora testing library looks like a good alternative if you want to try something different.

Building a CDK construct

Now that you know how the tests will work, you'll create a CDK construct that you can reuse throughout your project. Even if you aren't using the CDK, the process should be pretty easy to replicate in CloudFormation or Terraform.

Custom Resource Lambda Handler

Start by creating a new folder for your construct, call it IntegrationTests. You're going to create the actual construct inside an index file later. Before you start working on that you'll create the handler lambda.

Install the cfn-lambda package in your project. It provides some helper methods to make creating the custom resource handler easier. Once it's installed, create a file called handlerLambda.ts inside of the IntegrationTests directory. Add the following code to the file to handle basic custom resource interactions.

import CfnLambda from "cfn-lambda";

export const handler = CfnLambda({ 
    async AsyncCreate(properties) {
        return {
      FnGetAttrsDataObj: {
        TestPassed: true,
        Finished: new Date().toISOString(),
      },
    };
    },
    async AsyncUpdate(physicalId, properties) {
        return {
      FnGetAttrsDataObj: {
        TestPassed: true,
        Finished: new Date().toISOString(),
      },
    };
    },
    async AsyncDelete(physicalId, properties) {
        return {};
    }
});

Next, you need to create a method called runTests that runs the test file that is passed into the construct. You'll add the test code to the lambda by doing a lazy import of an environment variable. You'll provide the value of the environment variable at build time, which esbuild will be able to see when it's bundling the lambda's source.

import {exec, suite} from 'uvu';

...

const runTests = async (properties: unknown) => {
    const {default: initTests} = await import(process.env.TEST_FILE!);
};

The initTests method will need to accept a uvu test suite as its only parameter. You'll pass the custom resource's properties into the test suite's context so they can be accessed during runtime. You'll also need to call the test suite's run method to make sure it's added to the list of tests to be executed.

const runTests = async (properties: unknown) => {
    const {default: initTests} = await import(process.env.TEST_FILE!);

    const test = suite('Integration Test', properties);
    initTests(test);
    test.run();
};

Now you need to tell uvu to run all of the tests, then check the result. Unfortunately, uvu doesn't fully support the programmatic execution of tests, so you need to use a workaround to get the test results. Check to see if process.exitCode was set to 1, which means at least one of the tests failed. If you throw an exception, cfn-lambda will pick it up and mark your custom resource's deployment as a failure.

const runTests = async (properties: unknown) => {
        ...

    const result = await exec();

    if (process.exitCode) {
      throw new Error(`Tests Failed`);
    }
};

Now that you have the runTest method, you need to call it from the create and update handlers. You can optionally return an object with FnGetAttrsDataObj set if you want to reference some values in your CloudFormation template.

async AsyncCreate(properties) {
  await runTests(properties);

  return {
    FnGetAttrsDataObj: {
      TestPassed: true,
      Finished: new Date().toISOString(),
    },
  };
},
async AsyncUpdate(physicalId, properties) {
  await runTests(properties);
  return {
    FnGetAttrsDataObj: {
      TestPassed: true,
      Finished: new Date().toISOString(),
    },
  };
},

That's all there is to the lambda. Really it's just a uvu test runner. Now you need to create the construct that deploys it and any tests that need to be executed.

Creating the Resource

Create an index.ts file in the IntegrationTests directory. Add the following code to get started.

import * as cdk from "@aws-cdk/core";
import * as nodeLambda from "@aws-cdk/aws-lambda-nodejs";
import * as lambda from '@aws-cdk/aws-lambda';

interface IntegrationTestsProps {
  testEntry: string;
  timeout: cdk.Duration;

  /** These values will be passed into the context of your tests */
  properties?: Record<string, string>;
}

export class IntegrationTests extends cdk.Construct {
    private handler: lambda.IFunction;

    constructor(scope: cdk.Construct, id: string, props: IntegrationTestsProps) {
        super(scope, id);
    }
}

You can see there are three properties that your construct will accept. The testEntry property accepts a path to the file that will be injected into the lambda handler that you created in the previous step. timeout specifies the maximum amount of time that your lambda can spend running the integration tests. Finally, properties allows you to pass values into the custom resource that will be available when the tests execute.

Next, create the handler lambda. The only complicated part of this is injecting the testEntry property into your source code. You're going to use [esbuild](https://esbuild.github.io/) to take care of that.

The NodejsFunction construct uses esbuild to bundle lambda sources. It allows you to customize the build process using the [bundling property](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda-nodejs.NodejsFunction.html#bundling).

The bundling parameter that you're going to use is define, which tells esbuild to replace global values with a constant. This replacement will happen before bundling, so when esbuild sees your import statement it will bundle the source code found at the provided path.

this.handler = new nodeLambda.NodejsFunction(this, 'TestRunner', {
    entry: require.resolve('./handlerLambda.ts'),
    timeout: props.timeout,
    runtime: lambda.Runtime.NODEJS_14_X,
    bundling: {
        define: {
      "process.env.TEST_FILE": `"${props.testEntry}"`,
    },
    }
});

Now that the lambda is created, you can create a custom resource. Pass the lambda's ARN into the serviceToken parameter. Also, provide a property that will change each deployment to ensure the test will be executed every time.

new cdk.CustomResource(this, 'IntegrationTests', {
    serviceToken: this.handler.functionArn,
    properties: {
        ...props.properties,
        Version: new Date().toISOString(),
    }
});

Granting Permissions

Your construct is all set up to execute any test that you pass into it. More than likely most of those tests will require access to an AWS resource. To make the process of granting permissions to your lambda simpler, change your IntegrationTests construct to implement the iam.IGrantable interface.

export class IntegrationTests extends cdk.Construct implements iam.IGrantable {

The IGrantable interface only has one property: grantPrincipal. This property is used with the grant* methods of CDK constructs. For example, your handler function has the grantInvoke method that will give a principal permission to invoke it.

Since the handler lambda requires any permissions granted to your integration tests, simply return its grantPrincipal property.

get grantPrincipal(): iam.IPrincipal {
  return this.handler.grantPrincipal;
}

Running some tests

Now that you've finished building the construct, it's time to test it out. Create an empty stack and add an instance of the IntegrationTests construct to it.

import * as cdk from '@aws-cdk/core';

import { IntegrationTests } from '../constructs/integrationTests';

export class TestStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: cdk.StackProps = {}) {
    super(scope, id, props);

    new IntegrationTests(this, `IntegrationTests`, {
      testEntry: require.resolve("./tests.ts"),
      timeout: cdk.Duration.seconds(5),
      properties: {
        Foo: "Bar"
      }
    });
  }
}

Creating some tests

The testEntry property that you passed into the IntegrationTests construct points to a file called tests.ts in the same directory as TestStack. Create that file and add the following code to it.

import {Test} from 'uvu';
import * as assert from 'uvu/assert';

interface TestContext {
    Foo: string;
}

export default (test: Test<TestContext>) => {
    test('First test', async (context) => {
        assert.equal(context.Foo, 'Bar', 'Foo should be Bar');
    });
};

This code adds a single test that makes sure the Foo property passed into IntegrationTests is equal to Bar.

Testing it

Everything is set up now, you can run npx cdk deploy from your project's root directory and it should deploy successfully.

To validate that your tests will actually roll back the stack when they fail, change the Foo property that you passed into IntegrationTests from Bar to Fighters. Now deploy your stack again. This time it should fail, and if you open up the CloudFormation console you'll see a status of UPDATE_ROLLBACK_COMPLETE.

Summary

In this tutorial you've seen how to create a lambda-backed custom resource, how to inject code into a lambda using esbuild, and how to roll back deployments by throwing an error inside a custom resource handler.

Cover Photo by Scott Graham on Unsplash

Connecting Event Busses Across AWS Accounts

Ben Force — Fri, 01 Oct 2021 11:47:50 +0000

I've spent the last few weeks researching how to connect event busses in two different AWS accounts using the CDK. Most of the information that I've found has just been CloudFormation templates, which is fine, but some of the resources you have to create in CloudFormation are automatically created in the CDK.

Now that I've solved the problem, I'd like to help the next unfortunate sole that needs to implement this. Fortunately the final solution isn't that complicated.

This article leaves out some boilerplate code, like creating the stacks. For the full source go to: https://github.com/theBenForce/post-samples/tree/article/cross-account-event-bus

What's needed?

To get this to work you'll have to create a bus in each account—obviously. Beyond that, you'll need to create different resources for the sending and receiving accounts.

In the receiving account you'll need to create the rule that you want to be triggered by the message, and a policy on the target bus that allows messages from the sending account.

In the sending account you'll need to create an IAM Role that allows put events to the receiving bus. You'll also need to create a rule on the sending bus that references that role and forwards events to the receiving bus.

Setting up the target account

First, create the target bus. You'll need to set the name so that the ARN can be used in a stack in another account.

const targetBus = new events.EventBus(this, "TargetBus", {
  eventBusName: "TargetBus"
});

Since we're going to be referencing this bus from another account, we need to tell the CDK to generate a physical name during the synthesis process. This enable the CDK to generate an ARN before deployment.

targetBus._enableCrossEnvironment();

Now you need to create an event bus policy that allows messages from the source account.

new events.CfnEventBusPolicy(this, "CrossAccountPolicy", {
  action: "events:PutEvents",
  eventBusName: targetBus.eventBusName,
  principal: sourceAccountId,
  statementId: `AcceptFrom${sourceAccountId}`,
});

This wouldn't be any fun if you didn't get to see your events somewhere in the target account. To make that happen, create a CloudWatch log group that you can dump all of the received messages into.

const eventLogs = new logs.LogGroup(this, "EventLogs", {
  logGroupName: "EventLogs",
});

Finally, you need to create the rule that you want triggered. The rule will pass all events with a detailType set to TestEvent into the log group that you created.

new events.Rule(this, "TestRule", {
  eventBus: targetBus,
  eventPattern: {
    detailType: ["TestEvent"],
  },
    targets: [
    new eventsTargets.CloudWatchLogGroup(eventLogs),
  ],
});

Setting up the source account

Start by creating the bus that will be sending events.

const sourceBus = new events.EventBus(this, "SourceBus");

And create a rule that will forward TestEvent messages to the target bus. The CDK creates both the Event Bus Rule and an IAM Role that will allow your Event Bus to send messages to the target Event Bus.

new events.Rule(this, "TestRule", {
    eventBus: sourceBus,
    eventPattern: "TestEvent",
    targets: [
        new eventsTargets.EventBus(targetBus)
    ]
});

Deploy

Now that you've created both busses, go ahead and deploy them. Without any fancy setup, you'll need to deploy the stacks one at a time using the proper AWS profile for each.

npx cdk deploy --profile source SourceStack
npx cdk deploy --profile target TargetStack

Testing

Now that both stacks are deployed you can test the connection. Log into the source account's AWS console and navigate to "Event busses" in the EventBridge service. Select your bus then click Actions→Send events.

On the page that opens, make sure the "Detail type" matches the value you used when you created the event bus rule.

Now head over to your target account and open the CloudWatch log group that you created. You should see your message there.

Summary

Connecting event bridges across two AWS accounts requires several pieces to get it working, fortunately the CDK makes it pretty simple to create them. In the target account you need to create an Event Bus with an Event Bus Policy that allows messages from the source account. The source account needs an IAM Role attached to an Event Bus Rule, which the CDK takes care of behind the scenes.

Dynamic CDK Dashboards

Ben Force — Mon, 27 Sep 2021 12:22:40 +0000

One of the CDK's greatest strengths is the ability to create a library of components to be shared across your company, or shared with the world on constructs.dev. But how do you create a construct based on other constructs in the same stack without forcing the developer to call a method for each one? That's exactly what aspects allow you to do.

You can find the final source for this tutorial at https://github.com/theBenForce/dynamo-dashboard

What are Aspects?

The CDK uses a four step process to convert your code into a CloudFormation template: construct, prepare, validate, and synthesize. Since aspects are used during the second step, you need to understand how the first two steps are related.

When you run a CDK command, it starts by executing your code. Usually this is some file in the project's bin/ directory where your App is created. This will create all of the constructs that you've added to your project, keeping only the level 1, or Cfn* constructs for the next step.

The prepare step is where aspects are used. Each scope has a set of aspects registered to it during the construct phase. The CDK travels down the construct tree and if it finds a registered aspect it will pass in all of the constructs in the current scope and any child scopes.

Creating the Dashboard Construct

If you don't have the CDK installed, you can find instructions in the getting started guide.

Now that you've seen aspects work behind the scenes, let's create one of our own. We'll create a CloudWatch dashboard construct that dynamically adds widgets for every DynamoDB table in the construct's scope. Start by creating a skeleton project by running the following commands.

mkdir dynamo-dashboard
cd dynamo-dashboard
cdk init app --language typescript

Create the Construct

First, create a new construct called DynamoDashboard. Create the class in lib/dynamoDashboard.ts and make sure it implements IAspect.

import * as cdk from "@aws-cdk/core";
import * as cw from "@aws-cdk/aws-cloudwatch";
import * as cw from "@aws-cdk/aws-dynamodb";

export class DynamoDashboard extends cdk.Construct implements cdk.IAspect {
    visit(node: cdk.IConstruct): void {
    }
}

Next, you'll need to add some properties to store the dashboard's state between calls to its visit method. The first property is a reference to the CloudWatch dashboard that will be updated. The second property will store all of the widgets that you create.

private dashboard: cw.CfnDashboard;
widgets: Array<Widget> = [];

Finally, add a constructor. It should register itself as an Aspect in its scope, and create the dashboard construct.

constructor(scope: cdk.Construct, id: string) {
    super(scope, id);

    // Register this as an aspect
  cdk.Aspects.of(scope).add(this);

    this.dashboard = new cw.CfnDashboard(this, `Dashboard`, {
    dashboardBody: ""
  });
}

Implementing `visit()`

Adding a widget to the dashboard requires three steps: find a free spot, add a widget object to the widgets array, and update the dashboard body definition. Use following method to figure out where the next available spot is. You can change the width/height values, but keep in mind that the dashboard is 24 units wide.

getNextPosition(): {x: number; y: number; width: number; height: number;} {
    let x = 0;
  let y = 0;

  if (this.widgets.length > 0) {
    const lastWidget = this.widgets[this.widgets.length - 1];
    x = lastWidget.x + lastWidget.width;
    y = lastWidget.y;

    if (x >= DASHBOARD_WIDTH - 1) {
      x = 0;
      y += this.props.widgetHeight + 1;
    }
  }

    return {x, y, width: 12, height: 6};
}

Now you can implement the actual visit method. The first thing you need to do is make sure the node is a DynamoDB table. Since every construct passed into the visit method will be level 1, you will be checking if the node is a CfnTable instance.

visit(node: cdk.IConstruct): void {
    if(node instanceof dynamodb.CfnTable) {
    }
}

If the node is a table instance, you can create a new widget and update the dashboard.

visit(node: cdk.IConstruct): void {
  if (node instanceof dynamo.CfnTable) {
    const position = this.getNextPosition();

    this.widgets.push({
          ...position,
          type: "metric",
          properties: {
            metrics: [
              [
                "AWS/DynamoDB",
                "ConsumedReadCapacityUnits",
                "TableName",
                node.ref,
                { label: "Read (max)" },
              ],
              [".", "ConsumedWriteCapacityUnits", ".", ".", { label: "Write (max)" }],
            ],
            view: "timeSeries",
            stacked: false,
            region: node.stack.region,
            title: node.logicalId,
            stat: "Maximum",
            period: 300,
            liveData: true,
          },
        });

    this.dashboard.dashboardBody = JSON.stringify({ widgets: this.widgets });
  }
}

Testing It

Now comes the fun part, it's time to test your component. Go to your project's default stack and create an instance of the component. Next, add a couple DynamoDB tables. Even though your visit method is looking for CfnTable instances you can use the Table class since it creates a CfnTable instance in its constructor.

export class DynamoDashboardStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new DynamoDashboard(this, `DynamoDashboard`);

    new dynamo.Table(this, `FirstTable`, {
      partitionKey: {
        name: "pk",
        type: dynamo.AttributeType.STRING,
      },
    });

    new dynamo.Table(this, `SecondTable`, {
      partitionKey: {
        name: "pk",
        type: dynamo.AttributeType.STRING,
      },
    });
  }
}

Everything should be good to go at this point, run cdk deploy to send your stack to the cloud. Once it's done deploying, open up CloudWatch and find your dashboard. Now enjoy the fact that you don't have to manually add every table to it!

Conclusion

In this article you've seen what CDK aspects are and how to use them to dynamically update resources. I haven't seen many other examples of aspects, but hopefully you'll be inspired to create something awesome and share it!

AWS Storage Options

Ben Force — Thu, 03 Jun 2021 18:31:26 +0000

When I was nine I discovered QBASIC on my parents' DOS computer. I got addicted immediately but, in the nineties, we didn't have stack overflow where I could just copy-paste example code.

What I did have were a few books that I found at a thrift store with example programs printed in them. I would spend hours copying the code (especially since I used the hunt-and-peck typing technique).

Once I had all of that code copied, I had no questions about where to store it. On my personal, 750 kb floppy disk.

Now I'm a cloud developer, and when I want to store files it's not as simple of a decision. There are all types of storage in AWS with a different use case for each. In this article, I'm going to walk through the most commonly used storage services and describe the common use cases for them.

Types of Cloud Storage

There are three different classifications of data stores: Object, File, and Block.

Object Storage

Object storage is the most cloudy type of storage. Objects, or files, are stored using a key and the contents of the store can be spread over multiple servers. This enables high availability and durability.

File Storage

If you have a file server, or a shared folder, on your network then you're already familiar with file stores. They store files in a folder structure and you can access them through a network.

Block Storage

This type of storage is all about performance. Essentially it just acts like a hard disk connected to whatever is using it.

What are the main services?

First things first, I'll briefly describe the services before getting into use cases.

S3 (Object)

S3 is one of the first services created on AWS. It's kinda the default choice for storage in AWS. It's fast, inexpensive, and can store a LOT of data. Unlimited storage space in fact. You can also store objects (files) up to 5 Tb in size!

S3 Glacier (Object)

While S3 is fast to access, S3 Glacier is very slow to get your files out of. As the name kind of implies. Getting files (archives) out of an S3 Glacier Vault can take between a minute and twelve hours depending on how much you want to pay for it.

Instance Store (Block)

Instance store is ephemeral storage just for your EC2 instances. Anything stored in an instance store gets destroyed when the EC2 instance that it's attached to is terminated. You can also only access an instance store from a single instance.

Elastic Block Store (Block)

Elastic Block Store is another storage solution for EC2, but unlike an instance store, its data is persisted when you terminate an instance. Also, you can share an EBS between multiple instances.

Amazon EFS (File)

Elastic File Store is a service that allows you to store files in S3 as if they were actually on your local network. You attach a VM to your network and it acts as a file server that stores the files in AWS.

How can you use them?

Of course, there's a reason that all of these different storage services exist. They all have different use cases. Let's look at a couple.

Long Term Storage

How do you save data that won't be accessed regularly? Maybe something that you're required to keep for a few years in case of an audit? That's the exact use case that S3 Glacier was designed for.

EC2 Server Caching

If you're running an EC2 instance that needs to cache some data, the instance store is what you're looking for. Since its content lives and dies with the EC2 instance that it's attached to, it's ideal for caches and buffers.

Apache Web Server Content

When you have a bunch of instances that all need to have the same data, an EBS may be the way to go. You could store everything in s3 and sync it as part of your bootstrap script, but using an EBS will ensure everything stays synced between server instances.

A File Server on Your Network

This one is probably obvious if you read my descriptions of the different services. EFS is designed to give you the convenience of a local file store, with the durability of cloud storage.

Conclusion

This article provided an introduction to the main storage services in AWS. You have seen the different classifications of storage, and AWS's main services for each type. You also saw a handful of use cases and the best service to use for each.

CloudFormation Templates 101

Ben Force — Sat, 29 May 2021 15:15:05 +0000

Introduction

Have you ever tried to move resources from one AWS region to another? It can be quite painful. You have to figure out how all of the resources connect together, then plan out what order you need to recreate them. Fortunately, AWS has a simpler way of doing that. It's called CloudFormation.

CloudFormation allows you to define all of those resources (and their relationships) in a JSON or YAML file called a template. The template can take in some parameters too, which means you can define multiple environments with a single template.

In this article, I'll explain the fundamental sections of a CloudFormation template and how to use it to deploy a stack.

CloudFormation Template Structure

Cloud formation templates are YAML files with a few specific root properties that are referred to as sections. If you want to see the sections not covered in this article, checkout out the CloudFormation User Guide.

Parameters

The parameters section allows you to create parameters (duh). Using parameters allows you to create a single template that can be reused across multiple environments. Just change the parameter values and you have a new environment--or at least an updated one.

Note: you are limited to 200 parameters, but if you need that many you're probably doing something wrong!

To define a parameter, you only need to specify an ID and type. You can find the full list of parameter settings in the AWS CloudFormation User Guide. The following example shows how to define a string parameter named InputBucketName in YAML.

Parameters:
  InputBucketName:
    Type: String

Resources

Resources are the real reason you're creating this template in the first place. They define everything that you actually want to create in AWS. Similar to the parameters, each resource requires an ID and a Type.

The IDs defined in CloudFormation are only used within the template. When the template is deployed the IDs may or may not be used as the resource name. BTW: the IDs are referred to as a Logical ID.

The type names follow the pattern AWS::service::resource_type. So to create an S3 bucket you want to create a resource of type AWS::S3::Bucket.

Resources:
  BigBucket:
    Type: AWS::S3::Bucket

That's it. There's a lot of default values being used, but this will create an s3 bucket in your AWS account.

If you want to override those default values, you need to provide them in the resource's Properties. For example, if you want to set the bucket's name, use the aptly named BucketName property.

Resources:
  BigBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName:
        Ref: InputBucketName

Outputs

Now that the bucket is created, how do you use it? You could figure out the bucket's URL from the name parameter, but what if you want to create an IAM role that references it in another stack?

That's exactly what outputs are for. You define values that will be needed by another stack as an output, then you can import that value when creating the other stack.

To output the URL for our bucket, create an output section that looks like this:

Outputs:
  BigBucketUrl:
    Value:
      Fn::GetAtt: ["BigBucket", "WebsiteURL"]

Deploying your Template

Save all of the above sections into the file cf.yml and run the following command:

aws cloudformation deploy --template-file ./cf.yaml --stack-name cf-101-tutorial --parameter-overrides InputBucketName=$USER

Cleaning Up

Once you're done with your stack, you should clean it up. It's always a good idea to destroy anything that you aren't using so you don't end up with a surprise bill at the end of the month.

To delete everything we created when deploying the stack, run the delete command.

aws cloudformation delete-stack --stack-name cf-101-tutorial

Conclusion

Now you know the fundamentals of a CloudFormation template. You can take in parameters, define resources, and output details of those resources to be used in other stacks. You also saw how to deploy and delete a stack using a CloudFormation template.

Like everything else with AWS, there's a steep learning curve to CloudFormation. This article just scratched the surface of what's possible with CloudFormation. There are other sections, different methods, and a whole lot of resources that you can use.

DEV Community: Ben Force

Lock Down Your Data: Implement Row-Level Security Policies in Supabase SQL

Understanding RLS

RLS Example Database: Recipe-Sharing Web App

Using RLS in Supabase

Basic Components of SQL Policies for RLS

Name

Table

Command

Role

Conditions

Creating a SELECT Policy

Creating INSERT/UPDATE Policies

Advanced RLS Policies for Real-World Use Cases

Multitenant Applications

Role-Based Access Control (RBAC)

Conditional Policies

Testing and Validating RLS Implementation

Validating with Unit Tests

Troubleshooting Common Issues

Anonymous Data Access Issues

Server-Side Rendering

Recursive RLS Policies

Conclusion

Optimizing Nest-Commander: User Inputs

Adding a Message Option

Decorating It

The Parser

Using the Options

Using Inquirer

Summary

How to Make a CLI App with NestJS: Step-by-Step

Creating the Project

Setup nest-commander

Creating a Command

Wiring Everything Together

Tell NestJS that it's not a server

Running It

Summary

Get All Results from DynamoDB Queries Easily!

How DynamoDB Query Pagination Works

The Simple Way

Using an Async Generator

Calling It

Conclusion

My Node.js Toolkit for DevOps Scripting

Running with Types

Parsing Parameters

Defining Options

Taking Action

Getting things started

Watching it Work

Defining Tasks

Skipping Tasks

Parallel Processing

Running the Tasks

Summary

Getting Error Responses from Axios

When things go wrong

Making it more responsive

There's a .catch()

Summary

Easy Test Deployments with Terraform

Editing Files Directly

What About Variables?

Override Files

Summary

CDK Integration Testing

How does it work?

CloudFormation Custom Resources

UVU Test Runner

Building a CDK construct

Custom Resource Lambda Handler

Creating the Resource

Granting Permissions

Running some tests

Creating some tests

Testing it

Summary

Connecting Event Busses Across AWS Accounts

Creating a `SELECT` Policy

Creating `INSERT`/`UPDATE` Policies

Setup `nest-commander`

There's a `.catch()`

Implementing `visit()`