DEV Community: Leonti Bielski

AWS Budget Killswitch: Disable AWS Services When Budget Is Exceeded

Leonti Bielski — Mon, 06 Jul 2020 03:47:39 +0000

I like creating small projects and hosting them on AWS. AWS is what I use at work and using it for my personal projects makes sense because it's convenient and it's what I know.
However, my personal projects don't make me any money so a huge increase in traffic doesn't mean an increase in revenue to cover the cost of an AWS bill.
Sometimes I just want to put something out there and see if people like it. Call me paranoid, but I'm afraid of a situation when I wake up one day and discover that I have a $2700 bill in my AWS account for something that doesn't make any money.

I've always wanted to have a hard limit on AWS spending on my personal account. Something like: "Stop everything when my bill reaches $100 a month". There is no such functionality on AWS and I appreciate how hard would it be to implement it. There would not be a solution which works for everyone. Some people would only want to stop services which are the main culprits, some would like to stop everything. Would you want to have your s3 buckets emptied if you have a sudden spike in your bill? Also, there is a matter of production services. Very few companies would be ok with terminating their infrastructure when the bill goes above a certain budget.
For people who just like to tinker and are ok with their services being down in an emergency such solution might be useful. This is where "AWS Budget Killswitch" comes in.

AWS Budget Killswitch

The solution is very simple:

Create an emergency budget, a certain value which would constitute an unacceptable spending
When budget is reached send an email and disable public-facing AWS services, currently only CloudFront and ApiGateway are supported

The app is implemented as a SAM template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: ''

Parameters:
  NotificationEmail:
    Type: String
  BudgetLimit:
    Type: Number

Resources:
  KillswitchBudgetTopic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: !GetAtt DisableServicesFunction.Arn
          Protocol: "lambda"

  KillswitchBudget:
    Type: AWS::Budgets::Budget
    Properties: 
      Budget:
        BudgetLimit:
          Amount: !Ref BudgetLimit
          Unit: USD
        TimeUnit: MONTHLY
        BudgetType: COST
      NotificationsWithSubscribers:
        - Notification:
            NotificationType: ACTUAL
            ComparisonOperator: GREATER_THAN
            Threshold: 99
          Subscribers:
            - SubscriptionType: EMAIL
              Address: !Ref NotificationEmail
            - SubscriptionType: SNS
              Address: !Ref KillswitchBudgetTopic

  SnsInvokeLambdaPermission:            
    Type: AWS::Lambda::Permission
    Properties: 
      Action: lambda:InvokeFunction
      FunctionName: !Ref DisableServicesFunction
      Principal: 'sns.amazonaws.com'
      SourceArn: !Ref KillswitchBudgetTopic              

  DisableServicesFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: disable-services
      Handler: app.handler
      Runtime: nodejs12.x
      MemorySize: 256  
      Policies:
      - AWSLambdaExecute
      - Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
            - cloudfront:ListDistributions
            - cloudfront:GetDistributionConfig
            - cloudfront:UpdateDistribution
            - 'apigateway:*'
            Resource: '*'

It creates a budget with 2 subscriptions, email, and an SNS topic.

SNS topic is configured to invoke a Lambda which contains the logic to disable services. AWS doesn't have a big red "Disable" button for each service, each one is unique, so disabling them is different per service.

You can disable a CloudFront distribution, but ApiGateway stage can only be deleted. Luckily you can set throttling limits to 0, so it can also be made unreachable.

Here is how the Lambda looks like:

const AWS = require('aws-sdk')

const cloudfront = new AWS.CloudFront()
const apigateway = new AWS.APIGateway()

const disableDistribution = async(id) => {
  const config = await cloudfront.getDistributionConfig({ Id: id }).promise()
  const disabledConfig = { ...config.DistributionConfig, Enabled: false }

  await cloudfront.updateDistribution({
    Id: id,
    DistributionConfig: disabledConfig,
    IfMatch: config.ETag
  }).promise()
}

const disableCloudfront = async() => {
  const distributions = await cloudfront.listDistributions().promise()
  await Promise.all(distributions.DistributionList.Items.map(d => disableDistribution(d.Id)))
}

const setStageLimits = async(restApiId, stageName) => {
  const stage = await apigateway.getStage({
    restApiId,
    stageName,
  }).promise()

  await apigateway.updateStage({
    restApiId,
    stageName,
    patchOperations: [{
        op: 'replace',
        path: '/*/*/throttling/burstLimit',
        value: '0'
      },
      {
        op: 'replace',
        path: '/*/*/throttling/rateLimit',
        value: '0'
      },
    ]
  }).promise()

  console.log(stage)
}

const disableApiGateways = async() => {
  const apiIds = (await apigateway.getRestApis().promise()).items.map(a => a.id)

  await Promise.all(apiIds.map(async(restApiId) => {
    const stages = (await apigateway.getStages({
      restApiId
    }).promise()).item

    await Promise.all(stages.map(stage => setStageLimits(restApiId, stage.stageName)))
  }))

}

exports.handler = async(event) => {
  await disableCloudfront()
  await disableApiGateways()
};

There is nothing else to it, it's a very blunt instrument, but it will make sure that if you have a huge spike in traffic that you don't expect you'll end up with your app down instead of paying an exorbitant bill.

This is definitely something only for tinkerers like me, not for a production use.

The proper solution would be using an AWS WAF but I'm too cheap to spend $6 a month for a very unlickely event of a DDoS attack on one of my small projects.

Here is a Github repository ready to be deployed.

Testing: A Practical Approach

Leonti Bielski — Wed, 24 Jun 2020 07:44:28 +0000

Automated testing is important to build software in a sustainable way. No one wants to manually test their entire application on every change. That would be slow and a waste of time.
On the other hand, covering an application with all possible kinds of automated tests also has a significant cost. If not done right tests can make application architecture rigid and slow to change - the exact things we aim to improve with automated testing in the first place.
It's an important part of our job as software developers to make calls about what needs to be tested and to what degree.
During my career, I had changed my views on testing from just writing tests because it's a "best practice" to a more practical approach about when and which type of tests to write.
In this article, I'd like to share my thoughts on software testing with the hope it can be beneficial for someone who is on the same journey of writing better software.
This is not meant to be a definitive testing guide, it's meant to be a starting point for a more critical view on testing practices.

Context

The context in which application is built and used is very important when thinking about testing. Are you an early-stage startup or a big fintech company? Are you working on a new experimental way display content to users or are you adding a feature to an existing product which is going to be rolled out to millions of users?
Kent Beck has an excellent talk "3x Explore, Expand, Extract"

I highly recommend it. The gist of it is this - your approach to testing and software quality, in general, should depend on at what stage of product maturity you are. If you are just starting out, exploring, and experimenting then high automated testing coverage is not needed. You might even go for a while without any automated tests at all until you know what works and what needs to be solidified.
Once you know which parts of your product are going to stay you should improve test coverage and code quality in those areas.

TDD

Test-Driven Development is a bit like a religion. There are people who believe in its core values and there are fundamentalists which preach the "one true way" of doing it.
The most popular understanding of TDD seems to be: code should have 100% test coverage at any cost and that a unit is a class (in languages that have classes).
This approach leads to hundreds of useless tests which mostly test mocks just for the sake of the coverage.
It's hard to ask for the removal of such tests because no one wants to be that guy which is against automated testing ;)
When I first read DHH's post about TDD I thought the guy was crazy because TDD was meant to solve all of the problems of software quality. Now I get it.
There is a great talk by Ian Cooper titled "TDD, Where Did It All Go Wrong" which touches on the problems with the traditional understanding of TDD and how it differs from the original idea.

Here are the key takeaways:

"Unit" is not a single class, it's a module with public API.
Avoid mocks.
Test behaviour, not implementation. This approach avoids so many problems. Since we are testing a public API of a module there is no need for mocks. Imagine you test a Calculator module which can add and multiply numbers. It uses Adder and Multiplier classes underneath. But we don't care. All we care about is whether it can successfully add and multiply numbers, so we just test that. Calculator module is free to change its implementation, remove Adder and Multiplier dependencies and perform calculations using a third-party Math library. Not a single test would have to be changed and we would still be confident that Calculator works correctly. Compare this to a traditional approach:
Write a test for Calculator using Adder and Multiplier mocks. Make sure they are called with the right arguments.
Write a test for Adder.
Write a test for Multiplier. Now we have 3 brittle tests from which only 2 are testing the actual logic. If we need to switch to a third-party Math library we'd have to rewrite the Calculator test, and remove Adder and Multiplier ones. That would slow us down and more importantly when you have to change tests during refactoring how are you sure that you can rely on modified tests? It has been a joy working on projects which embrace "unit is a module" and "no mocks" approaches. Some say that if you don't test classes in isolation it leads to a tight coupling between classes. In my experience, it's not a problem at all. If developers are free to painlessly refactor and move stuff around it usually leads to a better code organisation over time. How often have you reconsidered moving a dependency because of the burden of adjusting unit tests?

Slow and fast tests

Given that everybody seems to have their own definition of what "integration test" is I prefer alternative definition where you group tests by being "fast" and "slow".
"Fast" tests are those which don't talk to slow IO (no matter how many classes are involved) and "slow" are those which do. Database and network calls are usually slow. Writing and reading small files to an SSD drive is usually pretty fast. Not doing any IO is even faster :)
The way to avoid performing IO calls during tests is to replace those parts of the code which do with stubs or "alternates". There are multiple ways to do it and functional programming particularly shines at separating IO from the rest of the code, but the main idea is to provide an interface for external interactions and 2 implementations - one for testing which responds with pre-canned values and the real one which talks to the external world (it needs to be tested separately).
Again, the context of your application is very important here and I would argue there is no universal "best practice" approach.
Imagine you have an application which takes a JSON payload and puts it into a Postgres database. What could go wrong here?

JSON codec might be misconfigured
Extracting data from the JSON payload and putting it into database might be broken.
Database schema might be different from the one that the code expects.

There is no logic to be tested here, so the test which would give you the most bang for your buck would be the one which accepts JSON payload from a string and writes it to the database.
There is nothing wrong with not having a "unit" test for this if it gives you no value. You can introduce it later when you have logic in your JSON->DB pipeline.

Conclusion

There is no single prescribed model for software testing. No application is the same and no team is the same. It is important to step back and evaluate your testing strategy.
If your current testing approach doesn't feel right maybe it's because what is considered a best practice is not right for your application. There is no point of striving for 100% unit test coverage in an application which is mostly a conduit between an HTTP request and a DB.
When you do write unit tests, avoiding mocks and testing behaviour instead of implementation goes a long way to achieve a balance between software quality and development speed.

Deploying an SPA on AWS

Leonti Bielski — Mon, 22 Jun 2020 07:47:11 +0000

Hosting a static website on AWS has many advantages but it has a few moving parts which need to fit together to get a fast and scalable static website.

While working on AvoCV I decided to host it on AWS for a few reasons.
Backend for the project is built using AWS Gateway, so hosting the frontend on AWS Gateway was a natural choice since it would reduce the burden of switching between different cloud providers.
Hosting a static website on AWS is also dirt-cheap, which was also important.

In this post I’ll describe the steps you need to take between creating your SPA frontend project and being able to access it from your own domain.

Domain name

The first thing you need to do is to get a domain name from any of the domain registrars.
After you have your domain you need to create a hosted zone in Route53, it costs $0.50 per month.
Route53 will create NS records automatically which you can use to “connect” your domain from your registrar to your hosted zone in Route53. Every registrar should have instructions on how to change NameServer records.

S3 hosting and Cloudfront

The easy way to make static files available publicly on AWS is to put them in a public s3 bucket. The url that you’ll get ain’t pretty though. It will look something like this: http://<bucket-name>.s3-website-us-east-1.amazonaws.com
It’s definitely not user-friendly and it’s not HTTPS which is a must nowadays.
This is why we need a Cloudfront distribution in front of the bucket. It will hide the ugly url and will make sure requests are HTTPS.

Public S3 bucket

The whole setup apart from Route53 is done in CloudFormation. This is how s3 bucket setup looks like:

  S3BucketLogs:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      AccessControl: LogDeliveryWrite
      BucketName: !Sub '${AWS::StackName}-logs'

  S3BucketRoot:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      AccessControl: PublicRead
      BucketName: !Sub '${AWS::StackName}-root'
      LoggingConfiguration:
        DestinationBucketName: !Ref S3BucketLogs
        LogFilePrefix: 'cdn/'
      WebsiteConfiguration:
        ErrorDocument: 'index.html'
        IndexDocument: 'index.html'

  S3BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref S3BucketRoot
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Action: 's3:GetObject'
            Principal: '*'
            Resource: !Sub '${S3BucketRoot.Arn}/*'

${AWS::StackName} will resolve to the name of your stack, so if you name your stack my-frontend the name of the bucket will be my-frontend-root.
We are telling AWS that the bucket should be publicly accessible with PublicRead access control and with a BucketPolicysaying that every object should be public.
This setup even includes access logs which are being put into a special <stack-name>-logs bucket.
Why ErrorDocument and IndexDocument are the same?
This is because in an SPA all requests go to index.html and then JavaScript decided which route to show.
When you navigate to https://avocv.com/editor AWS will try to find editor file in the bucket, but it won’t be there. Normally it would return an error page, but we are overriding it with ErrorDocument: index.html so it loads index.html anyway. Once the page is loaded JavaScript looks the url in the browser, sees /editor and knows that it needs to load Editor page.
If this behaviour is not desirable you can change this entry to something else, for example, error.html. Just make sure error.html is in the bucket.

SSL Certificate

Creating SSL in CloudFormation is easy:

  CertificateManagerCertificate:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName: !Ref DomainName
      SubjectAlternativeNames:
        - !Sub www.${DomainName}
      ValidationMethod: DNS

The only “gotcha” is that the SSL certificate will need to be verified (you’ll have to add verification entries to your domain records). You will see a pending certificate in AWS Console and since your hosted zone is in Route53 it’s as easy as pressing a button.

CloudFront SSL-only distribution

Creating a CloudFront distribution is a little bit more involved:

  CloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Aliases:
          - !Ref DomainName
        CustomErrorResponses:
          - ErrorCachingMinTTL: 60
            ErrorCode: 404
            ResponseCode: 200
            ResponsePagePath: '/index.html'
          - ErrorCachingMinTTL: 60
            ErrorCode: 403
            ResponseCode: 200
            ResponsePagePath: '/index.html'            
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
          CachedMethods:
            - GET
            - HEAD
          Compress: true
          DefaultTTL: 86400
          ForwardedValues:
            Cookies:
              Forward: none
            QueryString: true
          MaxTTL: 31536000
          SmoothStreaming: false
          TargetOriginId: !Sub 'S3-${AWS::StackName}-root'
          ViewerProtocolPolicy: 'redirect-to-https'
        DefaultRootObject: 'index.html'
        Enabled: true
        HttpVersion: http2
        IPV6Enabled: true
        Logging:
          Bucket: !GetAtt S3BucketLogs.DomainName
          IncludeCookies: false
          Prefix: 'cdn/'
        Origins:
          - CustomOriginConfig:
              HTTPPort: 80
              HTTPSPort: 443
              OriginKeepaliveTimeout: 5
              OriginProtocolPolicy: 'https-only'
              OriginReadTimeout: 30
              OriginSSLProtocols:
                - TLSv1
                - TLSv1.1
                - TLSv1.2
            DomainName: !GetAtt S3BucketRoot.DomainName
            Id: !Sub 'S3-${AWS::StackName}-root'
        PriceClass: PriceClass_All
        ViewerCertificate:
          AcmCertificateArn: !Ref CertificateManagerCertificate
          SslSupportMethod: sni-only

We are using the same trick with custom error pages so they all would return index.html.
By setting an alias we are making the public s3 bucket accessible from our domain.
SSL is configured to use the certificate created in the same stack for our custom domain.

Redirecting WWW users to a naked domain

This step is optional, but personally I prefer to have a non-www domains, so I’d like to redirect users coming to https://www.avocv.com to https://avocv.com. For this we’ll need a special s3 bucket and another CloudFront distribution:

  S3BucketWWW:
    Type: "AWS::S3::Bucket"
    Properties: 
      BucketName: !Sub '${AWS::StackName}-www-redirect'
      AccessControl: PublicRead
      WebsiteConfiguration:
        RedirectAllRequestsTo:
          HostName: !Sub ${DomainName}
          Protocol: https

  CloudFrontDistributionRedirect:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
        - DomainName: !Sub '${AWS::StackName}-www-redirect.s3-website-${AWS::Region}.amazonaws.com'
          Id: !Sub 'S3-${AWS::StackName}-www-redirect'
          CustomOriginConfig:
            OriginProtocolPolicy: http-only
        Enabled: true
        HttpVersion: http2
        IPV6Enabled: true
        Logging:
          Bucket: !GetAtt S3BucketLogs.DomainName
          IncludeCookies: false
          Prefix: 'cdn-redirects/'
        Aliases:
        - !Sub 'www.${DomainName}'
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
          CachedMethods:
            - GET
            - HEAD
          TargetOriginId: !Sub 'S3-${AWS::StackName}-www-redirect'
          Compress: True
          DefaultTTL: 604800
          ForwardedValues:
            QueryString: 'false'
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
        PriceClass: PriceClass_All
        ViewerCertificate:
          AcmCertificateArn: !Ref CertificateManagerCertificate
          SslSupportMethod: sni-only

Public S3 buckets have ability to redirect user requests. In our case we create an empty public bucket which would redirect all requests to it to https://<your-domain-name>, so now all we have to do is to make sure all requests from www.<your-domain-name> would end up in that bucket which would then redirect them to a naked domain <your-domain-name>.
We do this with another simplified CloudFront distribution which has a www alias www.${DomainName}.
What this means is that when users navigates to www.avocv.com request will be handled by a redirect CloudFront distribution which will send it to a redirect bucket, which in turn will redirect it to avocv.com and it will be picked up by the proper CloudFront distribution and the request will end up in the destination s3 bucket. That’s a lot of redirects!

Route53 records

The last thing we need to do is to tell Route53 that domain is “connected” to our CloudFront distributions:

  Route53RecordSetGroup:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      HostedZoneName: !Sub '${DomainName}.'
      RecordSets:
      - Name: !Ref DomainName
        Type: A
        AliasTarget:
          DNSName: !GetAtt CloudFrontDistribution.DomainName
          EvaluateTargetHealth: false
          HostedZoneId: Z2FDTNDATAQYW2
      - Name: !Sub 'www.${DomainName}'
        Type: A
        AliasTarget:
          DNSName: !GetAtt CloudFrontDistributionRedirect.DomainName
          EvaluateTargetHealth: false
          HostedZoneId: Z2FDTNDATAQYW2

Please note that HostedZoneId doesn’t refer to your domain’s zone id, it a special zone id (Z2FDTNDATAQYW2) for an alias record which points to a CloudFront distribution.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-aliastarget.html

Full setup

You can find the full setup in this repository: https://github.com/Leonti/aws-static-website
It also includes an example deploy script:

aws s3 sync --delete --acl "public-read" build s3://<stack-name>-root                                               
aws cloudfront create-invalidation --distribution-id <distribution-id> --paths "/index.html"

It will sync your files with s3 bucket and invalidate index.html which is the only file you need to invalidate when building a modern SPA.