DEV Community: Roman Abdulmanov

10 Tips for Those Who Decide to Start a New Project on AWS

Roman Abdulmanov — Wed, 01 Mar 2023 08:18:37 +0000

Here I collected few tips that may be useful for those just at the beginning of their AWS journey.

1. Account separation

It is worth separating production and dev accounts from the very beginning. It costs almost nothing, but if you do not do it initially, you will have to spend precious time on rework.

This is important to do for the following reasons:

It allows you to limit people accessing the production environment
It allows you to create a loosely coupled architecture where the prod environment doesn't depend on the testing environment
You will have separate billing so that you will have the ability to see how much you pay for your production
This is necessary for passing different security and compliance checks

2. Permissions

The question may quickly arise: how to limit the permissions for developers? For example:

Protect against admins assuming roles between accounts. Developers with administrator rights could assume a role from another account by default
If you have not restricted access to SSM, decrypt methods, etc., developers can access various secrets stored in the environment (tokens, keys, etc.)
Access to the Billing Dashboard

The simple solution: grant only the necessary permissions, but this is more complex than it looks because the scope of rights changes constantly and needs to be continuously reviewed. With this approach, developers will periodically lose access when they need to develop something new that was not considered earlier.

To solve this, you can grant a more comprehensive range of rights, e.g., up to an Administrator on a test account. But set different permission boundaries.

Good post about it.

I also recommend setting up MFA for all accounts to reduce the risk of token leaks.

3. Automation

AWS is very captivating with its simplicity in setting something up manually, especially at the beginning of a project.

But behind this simplicity lies many complexities:

There is a very high probability of breaking something by accident
It is difficult to repeat the environment and, for example, create an identical staging for the developer
It isn't easy to understand how our system looks like in general because this knowledge is in our heads

To solve this, I suggest using one of the popular solutions:

4. CloudFormation

The 3 solutions above use CF. So if using CF to manage infrastructure, you should familiarize yourself with its limitations.

The most frustrating thing is the 500 resource limit, which you will run out of very quickly if you prefer a serverless approach or have a large project.

So, to avoid this problem at a crucial moment, I suggest you think about a strategy for placing resources in nested stacks (e.g. plugin for Serverless Framework).

5. Cognito

AWS Cognito is a powerful service for a customer identity and access management.

But when setting it up, you can make two mistakes that can cause serious trouble in production:

Case-sensitive user pool. If you create a User Pool programmatically, you must set the CaseSensitive parameter to false; otherwise, it will default to true. This complicates the sign-in process (e.g., User@test.co vs user@test.co). And most importantly, after creating the User Pool, you can no longer change this setting
The same applies to Mutable attributes. This means that if, for example, you mark an email as Mutable, users will not be able to change their emails, and most importantly, you cannot change the value of Mutable without recreating the User Pool

Recreating the User Pool is only possible by resetting user passwords, which would look suspicious. Therefore, if you made such mistakes when configuring your User Pool and have already launched the product, you may have unexpected problems that could have been easily avoided initially.

6. DynamoDB

DynamoDB, unlike the usual relational databases, recommends using the single table design.

For Node.js users to make this approach easier to understand and make the code incredibly simpler, I recommend looking at the OneTable library.

Here are some benefits:

Single-table design
TypeScript support
Local unit test support
Simple API
Migrations support
Indexes & conditions support
Built-in encryption support

7. Backups

AWS has a built-in backup service. This is a very powerful service that, for example, allows you to create a particular type of backup without the possibility of deleting previously created snapshots.

But this service has its limitations. We are faced with the fact that when restoring DynamoDB, we must manually restore DDB Streams. This would be problematic if, as we do, everything is deployed through CloudFormation, which does not "like" manual changes.

Therefore, to avoid surprises at the most inconvenient moment, in addition to creating backups, you need to develop, test, and document the recovery procedure and periodically perform test recovery to ensure that the process is still up to date.

8. Lambda

If you are using lambda, you probably have heard about the cold start problem.

To solve this, there are different workarounds: warm-up requests, language-specific solutions, lambda provisioned concurrency, etc.

But I want to focus on one, in my opinion, the most critical factor, in addition to the chosen language & memory: function bundle size.

Your bundle size directly affects the cold start time because AWS needs to prepare the code before it can be run. Even if you use Lambda Layers and extract dependencies, these Layers still need to be loaded.

So I advise trying to minimize the bundle size in every possible way. For example, we use Node.js, we never add AWS SDK because it is already present in the environment, we use esbuild because it showed the best results in terms of speed and size, and as a result, we don’t have a single function larger than 500KBs.

9. Alarms

The alarm system is incredibly useful for detecting suspicious activity and important alerts.

For example, we set up alarms when various errors occur in the system:

Lambda errors & throttling
Errors in logs (e.g. from docker)
WAF alarms
Billing alarms (spending more than a certain amount in $)
Custom metrics

And then, we connected SNS to send these alarms to us in a special slack channel.

10. Audit logs

You can enable activity logging via CloudTrail. It costs almost nothing but can be indispensable when analyzing suspicious activity or determining why something is not working in the system.

We had an incident with user registration in Cognito, and this logging helped to sort out the real reasons.

Production-Ready AWS CloudFront for SPA

Roman Abdulmanov — Fri, 03 Feb 2023 15:19:09 +0000

In this article, we will discuss the basic scenario for using CloudFront to serve a static SPA.

As always, we will try to make it not just work but to achieve the maximum performance available in AWS. So, I'd like to start with a basic scenario and then try to improve it.

Static files

If you work in the AWS environment, you usually upload static files to S3.

You need to configure Bucket Policy to enable public access (without authorization):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-website-bucket-static/*"
        }
    ]
}

And you need to enable Static website hosting:

And that's it! Now your site is available by the url:
http://<bucket-name>.s3-website-<region>.amazonaws.com/

HTTPS

The apparent drawback of the current solution is the lack of HTTPS. So let's set up CloudFront to resolve this.

Technically we only need to specify the correct CloudFront Origin:

When the distribution is deployed, your site will be available by CloudFront DNS name, e.g.:
https://21nnj7ewt8yqwc.cloudfront.net

The most important advantage of using CF is that you will get a full-fledged CDN, i.e., your static files will be delivered from the location closest to you, which means that the delays in the website loading will be minimal.

SPA

You will soon notice that if you have SPA, routing will not work. E.g.:
https://21nnj7ewt8yqwc.cloudfront.net/test/route
Because CloudFront will try to redirect requests to the test/route folder, which does not exist in S3.

SPA: First attempt to fix

In some posts or answers, you may find the following solution: configure Error document for S3 Static website hosting to redirect failed attempts back to index.html:

It will work, but you will have below issues:

All requests will have a 404 error code
They will be served not from the cache (CF will request content on each request from S3)
S3 will request the content twice, first at the path specified in the request, and after an error, index.html itself

SPA: Second attempt to fix

The most popular solution that you could find on the internet: configure CloudFront with Custom Error Response. In our case, it would be:

And it will fix the issue with the 404 error code:

But unfortunately, you will still have the issue with the cache and two requests. The reason for this is simple: CloudFront will request the page from S3, and when it receives a 404, it will return index.html from the cache. So you will lose all the advantages of CDN.

SPA: Solution

To fix both issues, we can implement CloudFront Function and fix routing for our SPA.

Just create a new Function:

function handler(event) {
  var request = event.request;
  var uri = request.uri;
  if (uri.endsWith('/')) {
    request.uri = '/index.html';
  } 
  else if (!uri.includes('.')) {
    request.uri = '/index.html';
  }
  return request;
}

And attach it to the CloudFront:

Now you will have the correct response code, and all requests will be served from the cache:

There are no penalties for launching these functions. They run in the exact location where the CDN runs.

S3 cache policy

You can configure some cache parameters on the S3 level. But you should do this very carefully. For example, if you take example from popular S3 sync plugin for Serverless Framework and upload index.html with below configuration:

params:
  - index.html:
    CacheControl: 'no-cache'

You will force CloudFront to request this date on each request:

Which, firstly, will work slowly, and secondly, it will load Origin with unnecessary requests.

The disadvantage of not having this setting is that you must make an invalidation request in the AWS console or via CLI / API to update the cached content.

API

If you have an API inside AWS infrastructure, please read my previous posts about Origin Shield and GraphQL.

By combining these approaches, you will achieve the best performance results and a higher level of security.

Thanks!

Improve AWS CloudFront API performance with Origin Shield

Roman Abdulmanov — Mon, 31 Oct 2022 18:01:38 +0000

This post will be helpful for those who use CloudFront with some API (API Gateway, AppSync, custom endpoint) hosted in a single region.

In this scenario, the client first interacts with CloudFront in the closest region, and then traffic through the regular network reaches the region where the API is located:

This may take time if you are far from another network.

The most reliable approach is to make a multi-regional API, i.e., host your API in different regions. But if for some reason you can't do that yet, there's a trick that can make things a little better.

Origin Shield

From AWS documentation:

CloudFront Origin Shield adds an additional layer in the CloudFront caching infrastructure that helps to minimize your origin’s load, improve its availability, and reduce its operating costs. By enabling CloudFront Origin Shield, you get the following benefits:

Better cache hit ratio – all requests from all of CloudFront’s caching layers to your origin go through Origin Shield, increasing the likelihood of a cache hit.

Reduced origin load – Origin Shield consolidates content requests for the same object to reduce the number of simultaneous requests.

Better network performance – When you enable Origin Shield in the AWS Region that has the lowest latency to your origin, you can get better network performance.

The last paragraph is what interests us. From the description, it follows that it should now work like this:

Configuration

You can enable Origin Shield through the AWS Console in your Origin settings + choose the region closest to your API:

We use the following template for Serverless Framework:

Origins:
  - Id: YourID
    OriginShield:
      Enabled: true
      OriginShieldRegion: ${aws:region}

CloudFormation documentation.

Testing

I configured two different CloudFronts to use the same API, but in one case with Origin Shield, and the other without

Without Origin Shield (1 thread * 100 requests):

With Origin Shield:

Without Origin Shield (4 thread * 50 requests):

With Origin Shield:

As we can see in this use case, the Origin Shield solution is a bit faster!

Caveats

Using Origin Shield is not free. Check pricing before usage:
Synthetic tests say nothing about how they will work in your conditions, so do not believe blindly, but test with your data first.

Thanks!

Production Ready GraphQL for AWS & Serverless

Roman Abdulmanov — Fri, 23 Sep 2022 14:06:58 +0000

What is this post about?

Many articles have been written on implementing a serverless GraphQL API using AWS. But there is almost nothing about how to make it production ready for websites.

This article will also be valid for API Gateway.

Tools

You can use different tools: AWS SAM, Serverless Framework, Pulumi, Terraform, pure AWS CDK, or something else. Ultimately, it's not all that important. The main thing is that you do it not manually. Everything must be automated for repeatability. Otherwise, you won't be able to build a reliable CI/CD process.

I like this plugin that will make your life easier. It will help you set up AppSync, manage API keys, configure WAF, etc.

Typical architecture

Most articles end up with a below architecture, service or website that directly calls the AppSync API:

The downside of this solution is that the API is susceptible to DDoS attacks, which can significantly increase your monthly bills.

Improved architecture

The obvious solution is to add a firewall with request limiting:

It will also help if you want to block GraphQL introspection:

And this could be the end, but this solution has one unpleasant flaw for websites: preflight request.

Preflight request

Since your site and API are on different domains, the browser will make 1 additional request to the server BEFORE any other requests.

This may come as an unexpected surprise, since the initial request can take almost twice as long.

Example:

Details (additional 456 milliseconds to wait for the request to complete):

As I already wrote, since your domain is hosted on mydomain.com, and the API is on https://[ID].appsync-api.[REGION].amazonaws.com, the browser will attempt to do additional validation for CORS by sending this request before the first call to the domain hosting the API.

The browser caches this request for a while, but if this is a public page and many people visit this site for the first time, then they will experience this problem.

It also starts repeating after some time when the cache expires.

Potential solution №1

The first thing that comes to mind if you are unfamiliar with preflight requests is that they can probably be disabled if the backend sets some header.

Unfortunately this is not possible.

Potential solution №2

Maybe, I can use a simple request to workaround this restriction?

This will not work if you have non-standard headers (e.g. Authorization), you will also have to set the wrong content-type. Even if it works, this does not look like a production ready solution and can be blocked by AWS / browsers anytime.

Potential solution №3

You can try to use iframe technique but it will not work for non-subdomain urls (we will talk about this later), and it will make the loading even slower because instead of a very fast request, you will have slow iframe loading.

Potential solution №4

Can I use custom domains?

Unfortunately, this is not possible too. You can't use the same domain for API (e.g. just mydomain.com). It will work only with sub-domain (e.g. api.mydomain.com). Browsers check for the same URL origin, so you can't use anything other than mydomain.com/[something].

Solution

If we are talking about AWS and Serverless, the most affordable solution is to use AWS CloudFront.

We can create two behaviors and redirect traffic to the correct Origin depending on the path pattern:

For this to work, CloudFront must have the correct certificates configured to accept requests using your DNS name.

This solution will solve the original problem, there are no more preflight requests or some kind of hacks.

The edge location of the servers would be an additional bonus. Which will reduce the connection time of clients with the endpoint.

The additional configuration of CloudFront Origin Shield will allow you to speed up traffic to AppSync or other service by using an internal faster network.

Caveat №1

We have implemented this architecture:

Since WAF works based on the IP address of the caller, it will not be the client's address but CloudFront's. Therefore, the request limit can be hit much faster and affect all users in the same location.

To solve this, we can use the X-Forwarded-For header to receive client IPs from CloudFront:

But this will make our AppSync API insecure, and if someone finds out the unique address of our endpoint, they can attack through it.

Therefore, it is more reliable to make two firewalls. Migrate the old one to handle requests in front of CloudFront. And make a new one between CloudFront and AppSync that will prohibit all requests not from CloudFront (make it directly inaccessible):

To set up New WAF, you can deny all requests by default and allow only those that contain some secret header with a value that is configured on CloudFront. WAF documentation for details.

CloudFormation:

  Rules:
   -  name: Secret
      action: Allow
      priority: 1
      statement:
        byteMatchStatement:
          searchString: [secret]
          fieldToMatch:
            singleHeader:
              name: Secret
          textTransformations:
            - priority: 0
              type: NONE
          positionalConstraint: 'EXACTLY'

Caveat №2

You should use the default сache policy (see below), even if you don't cache anything (POST requests are never cached). Because if you disable it, request compression will not work.

Caveat №3

If you have more than one endpoint, you will not be able to set up a redirect because you cannot have two identical path patterns: '/graphql'.

To resolve this, you can use some unique paths and setup CloudFront functions to redirect requests to the correct path for the Origin.

CloudFormation:

  CloudFrontGraphQLFunction:
    Type: AWS::CloudFront::Function
    Properties:
      Name: ${self:provider.stackName}-graphql-redirect
      AutoPublish: true
      FunctionCode: !Sub |
        function handler(event) {
          var request = event.request;
          request.uri = "/graphql"
          return request;
        }
      FunctionConfig:
        Runtime: cloudfront-js-1.0

Caveat №4

When developing locally, you may have problems with the frontend and CORS.

This is because the first response of the preflight request from localhost to the API does not contain all the required headers, even if you set the appropriate policy.

CloudFront will add access-control-allow-origin, but Chrome also needs (-headers and -methods).

To avoid the need to use additional extensions like this. You can configure additional headers using the CloudFront function:

  CloudFrontGraphQLCorsFunction:
    Type: AWS::CloudFront::Function
    Properties:
      Name: ${self:provider.stackName}-graphql-cors
      AutoPublish: true
      FunctionCode: !Sub |
        function handler(event) {
          var response  = event.response;
          var headers  = response.headers;
          headers['access-control-allow-origin'] = {value: "*"};
          headers['access-control-allow-headers'] = {value: "*"};
          headers['access-control-allow-methods'] = {value: "*"};
          return response;
        }
      FunctionConfig:
        Runtime: cloudfront-js-1.0

Conclusion

This approach should work not only with AppSync but also with other services, such as API Gateway.

If you have any questions, please write, and I will gladly try to answer.

Thanks