DEV Community: Henry Williams

Share Text Securely with AWS Amplify Gen 2

Henry Williams — Sun, 26 May 2024 18:21:36 +0000

This is a submission for the The AWS Amplify Fullstack TypeScript Challenge

What I Built

SecureShare allows users to share text snippets securely. The user has the option to set a time for the snippet to expire, whether or not the snippet should be deleted ("burned") after being viewed, and a password for additional privacy.

When using a password, the snippet content is encrypted with the password (which isn't sent to the backend or stored anywhere), so even someone with access to the database can't decrypt it!

Sharing snippet

As a user, you can create a snippet and share it with someone else by sharing the link to it and, optionally, the password to view it. If there's no password set, then anyone with the link can view the snippet.

Managing snippets

The core functionality of the app is creating, sharing, and viewing shared snippets. While these features don't require an account, being logged allows the user to be able to manage their shared snippets.

Inspiration

Demo and Code

Demo app: https://main.d26ottewuge4ec.amplifyapp.com/
Code: https://github.com/henryjw/secure-share-app

Journey

This was an interesting project for me because I've primarily focused on backend for most of my career. I hadn't implemented any significant frontend features in over a year and hadn't used react in even longer than that. So, I'm proud that I was able to implement as many features as I did for this project.

I had an initial list of features I wanted to implement and most of them were "nice-to-have" or "optional" to keep the scope realistic given my lack of frontend development experience. The only feature I didn't get to implement was adding the ability to add a file attachment to the snippet since it would have taken more time to implement and test.

Integrations

Data: The snippets are stored in the database so they can be retrieved later
Authentication: Used for login / signup and account management
File storage: Not used
Serverless functions: Not used

Connected Components and/or Feature Full: Connected components; not feature full

Used connected components for for login / sign up and account management. I later learned that Amplify can also generate connected components based on the data schemas to further simplify development. However, I had already created most of the UI by that time so I didn't get a chance to use them. Maybe next time!

Caching PNPM Modules in Docker Builds in GitHub Actions

Henry Williams — Mon, 06 May 2024 22:01:38 +0000

TLDR;

You can use reproducible-containers/buildkit-cache-dance to reuse cached files in your Docker build and an action like tespkg/actions-cache or actions/cache to persist the cache externally. See GitHub repo for an example setup.

Jump to Implementation section to see how everything is set up.

Introduction

Problem Statement

We all know how massive the dependency tree for NPM modules can get. And while PNPM provides significant performance improvements for local development, the same can't always be said for CI pipelines. For instance, in GitHub Actions, it's not uncommon for each build to install the same NPM modules every time since the jobs start with a clean state.

This problem is compounded when we start taking into account modules like node-gyp that need to be compiled from source, which only increases the amount of time it takes to install the modules.

It's fairly straightforward to cache files in a standard CI pipeline, but it gets more complicated when the modules are installed as part of a Docker build.

Background

In my scenario, the primary problem with downloading / compiling the NPM modules every time wasn't the time wasted. The problem was that there were some native / compiled dependencies that didn't play well with ARM-based CPU, which we used since they're cheaper to run on AWS than their x64 counterparts. The result was flaky builds that failed around 30% of the time and the only workaround was to manually re-run the build until it succeeded.

Caching the modules nearly eliminated the number of build failures. The caching approach worked well since the NPM modules only changed once or twice a week for the project.

Other attempted approaches

Docker layer caching - required too much disk space and was very volatile
Base image with NPM modules - complex to implement since a new Docker image would have to be created every time the NPM modules change

Limitations

There are likely better ways to solve this problem, but given the size of the team and the urgency of the problem, we needed a solution that was relatively simple to implement, required minimal maintenance, and could be implemented sooner rather than later.

Deep Dive

As stated in the introduction, caching files in a standard CI pipeline is fairly straightforward. However, it's not as straightforward to do within Docker due to its limitations.

Docker limitations

As of writing (May 2024), Docker only supports externally caching layers, but not cache mounts; the cache mounts are only temporarily available during the build. So if, say, we install NPM modules, there's no native way to access the generated files from outside of Docker.

For instance, RUN --mount=type=cache,target=/pnpm_cache,rw will correctly cache the files in /pnpm_cache and will be able to re-use it between builds. However, any state / files generated on a worker is cleared between runs in GitHub Actions, rendering the cache useless for this scenario.

The currently proposed solution is to allow Docker to bind the cache directory in the build to a directory on the host. This way the cache could be persisted externally. However, this issue has been opened for almost 4 years (May 27, 2020) with no clear answer as to whether it'll be implemented any time soon.

This is where the reproducible-containers/buildkit-cache-dance GitHub Action comes to the rescue! This Action is able to extract the files from the Docker build so they can be persisted on an external storage like S3 and is the approach recommended on the official Docker documentation.

Solution

The solution is to use the reproducible-containers/buildkit-cache-dance GitHub Action to extract / inject the cache generated by the Docker build and then use tespkg/actions-cache to save the cache in S3.

Workflow

After running a Docker build, reproducible-containers/buildkit-cache-dance extracts the files from the mounted directory and copies them to a directory on the host machine so it can be accessed outside of the context of Docker
tespkg/actions-cache uploads cache to S3. The cached files are compressed and are much smaller (10-20%) than the extracted file. In my experience, ~3GB of cache data for PNPM is compressed to less than 300MB.

[Cache hit scenario]

tespkg/actions-cache downloads cache from S3 and extracts the contents into the provided directory
reproducible-containers/buildkit-cache-dance grabs the files from the provided directory and injects them into the Docker build

Implementation

Requirements

AWS S3 bucket
AWS IAM user with access to the created bucket
Dockerfile to build your project
GitHub Action to build the Docker image

Example setup

Below is a simple setup for caching to S3. I've also set up a GitHub repository with the full setup.

GitHub workflow

---
name: Build
on:
  push:

jobs:
  Build:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/metadata-action@v5
        id: meta
        with:
          images: Build

      - name: Cache (S3)
        uses: tespkg/actions-cache@v1
        id: cache
        with:
          bucket: ${{ vars.CACHE_BUCKET }}
          accessKey: ${{ vars.AWS_ACCESS_KEY }}
          secretKey: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          ## Fallback to GitHub cache if saving / restoring from S3 fails
          use-fallback: true
          path: |
            pnpm
          key: pnpm-cache-${{ hashFiles('pnpm-lock.yaml') }}
          restore-keys: |
            pnpm-cache-

      - name: inject cache into docker
        uses: reproducible-containers/buildkit-cache-dance@v3.1.0
        with:
          cache-map: |
            {
              "pnpm": "/pnpm"
            }
          # Skip extraction if cache was hit to avoid unnecessary I/O. This can take minutes for projects with a lot of dependencies.
          skip-extraction: ${{ steps.cache.outputs.cache-hit }}

      - name: Build
        uses: docker/build-push-action@v5
        with:
          context: .
          cache-from: type=gha
          cache-to: type=gha,mode=max
          file: Dockerfile
          push: false
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Dockerfile

Note that the target directory used when mounting the cache is the same as the the directory specified in the cache-map provided to reproducible-containers/buildkit-cache-dance in the Workflow definition since that's were the cache is injected to and extracted from.

FROM node:21-slim AS base

ENV PNPM_HOME="/pnpm"

RUN corepack enable

# Copy your application code
WORKDIR /app
COPY . .

# Log for troubleshooting. There should be files in the directory when there's a cache hit
RUN --mount=type=cache,target=${PNPM_HOME} echo "PNPM contents before install: $(ls -la ${PNPM_HOME})"

### This is where the magic happens! The cache has been mounted to `$PNPM_HOME` so it can be accessed during the build ####
RUN  --mount=type=cache,target=${PNPM_HOME} \
  pnpm config set store-dir ${PNPM_HOME} && \
  pnpm install --frozen-lockfile --prefer-offline

# Another log for troubleshooting. This should never be empty since the NPM modules were installed before running this line
RUN --mount=type=cache,target=${PNPM_HOME} echo "PNPM contents after install: $(ls -la ${PNPM_HOME})"

FROM node:alpine AS prod

WORKDIR /app

COPY --from=base /app/node_modules /app/node_modules
COPY --from=base /app .

CMD ["npm", "start"]

Cache in action

Cache miss example

Cache hit example

File saved to S3

Conclusion

Although caching NPM modules inside the Docker build worked for my use case, it might not be the best option for you. Because of the time it takes to download the cache, inject the files into the Docker image, and extract the files from the Docker image, this caching approach will likely not yield any performance improvements over just installing the modules without using a cache.

However, if you're looking to solve build failures due to something like compilation failures or NPM rate-limit issues, then caching is a viable solution.

Caveat

Like anything in software development, this approach is subject to become outdated. So look up the latest information on "preserving cache mounts in Docker" in case this has changed.

References

Intercepting HTTP Requests with NodeJS

Henry Williams — Tue, 07 Sep 2021 02:20:02 +0000

Intro

As part of a project at work, I had to develop a way to intercept and store HTTP traffic for any given backend application (microservice in this case). This would have been a fairly straightforward task, but our backend is composed of many services (and many repos). Therefore, the solution had to be as seamless as possible so that it could be easily integrated into any of the services.

TLDR;

Using @mswjs/interceptors makes it straightforward to intercept HTTP traffic on your backend app.

Intercepting HTTP Traffic

For my use case, there were two options I could think of for capturing the HTTP traffic:

Create a library that wraps an HTTP client library like Axios
Somehow intercept all HTTP traffic

In an ideal world, I would have chosen option 1 since it would be the simplest. Unfortunately, the project I work on consists of many microservices owned by different teams. Therefore, it would make it difficult for everyone to go back and refactor their code to use this new library.

Therefore, my only option was really option 2.

First attempt

My first attempt was ok, but far from perfect. After trying to intercept the traffic directly through the low-level http module, I opted for a more higher-level solution. My idea was to monkey patch Axios's request methods to inject my own logic before a request is sent and after the response is received.

function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.request = _instrumentHttpRequest(axiosInstance.request, axiosInstance);
   axiosInstance.get = _instrumentHttpRequest(axiosInstance.get, axiosInstance, "get");
   axiosInstance.post = _instrumentHttpRequest(axiosInstance.post, axiosInstance, "post");
   axiosInstance.put = _instrumentHttpRequest(axiosInstance.put, axiosInstance, "put");
   axiosInstance.patch = _instrumentHttpRequest(axiosInstance.patch, axiosInstance, "patch");
   axiosInstance.delete = _instrumentHttpRequest(axiosInstance.delete, axiosInstance, "delete");
   axiosInstance.options = _instrumentHttpRequest(axiosInstance.options, axiosInstance, "options");
}

function _instrumentHttpRequest(originalFunction: Function, thisArgument: any, method?: string) {
   return async function() {
      const {method: cleanedMethod, url, config: requestConfig, data} = _parseAxiosArguments(arguments, method);

      const requestEvent: HttpRequestEvent = {
         url,
         method: cleanedMethod,
         body: data,
         headers: requestConfig?.headers,
      };


      // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
      doSomethingWithRequest(requestEvent);

      const res = await originalFunction.apply(thisArgument, arguments);


      const responseEvent: HttpResponseEvent = {
         url,
         method: cleanedMethod,
         body: res.data,
         headers: res.headers,
         statusCode: res.status,
      };

      doSomethingWithResponse(responseEvent);

      return res;
   };
}

This method worked fine, but then I accidentally bumped into a cleaner approach while reading the Axios docs.

Second attempt

To my surprise, Axios actually offers an API for intercepting requests and responses!

import {createInterceptor, InterceptorApi, IsomorphicRequest, IsomorphicResponse} from "@mswjs/interceptors";
import {interceptXMLHttpRequest} from "@mswjs/interceptors/lib/interceptors/XMLHttpRequest";
import {interceptClientRequest} from "@mswjs/interceptors/lib/interceptors/ClientRequest";

function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.interceptors.request.use(_instrumentHttpRequest);
   axiosInstance.interceptors.response.use(_instrumentHttpResponse);
}

function _instrumentHttpRequest(requestConfig: AxiosRequestConfig): AxiosRequestConfig {
   const method = String(requestConfig.method);
   const headers = requestConfig.headers && {
      ...requestConfig.headers.common,
      ...requestConfig.headers[method],
   };

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: String(requestConfig.url),
      body: requestConfig.data,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);

   return requestConfig;
}

function _instrumentHttpResponse(response: AxiosResponse): AxiosResponse {
   const responseEvent: HttpResponseEvent = {
      url: String(response.request?.url),
      method: String(response.request?.method),
      body: response.data,
      headers: response.headers,
      statusCode: response.status,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);

   return response;
}

Ah! Much better. However, there is another complication to this approach that's also present in the first attempt: the interception has to be set for every Axios instance; this makes for a less than ideal developer experience. I initially assumed everyone used the default axios instance. However, it turns out that it's also possible to create new instances via axios.create(). So back to the drawing board 😔

Final solution

Before attempting to mess with the low-level http module, I decided to look for some existing solutions. And after digging around for some time, I stumbled into @mswjs/interceptors. This library is very well documented and is TypeScript friendly.

function _instrumentHTTPTraffic() {
    const interceptor = createInterceptor({
      resolver: () => {}, // Required even if not used
      modules: [interceptXMLHttpRequest, interceptClientRequest],
   });

   interceptor.on("request", _handleHttpRequest);

   interceptor.on("response", _handleHttpResponse);

   interceptor.apply();
}

function _handleHttpRequest(request: IsomorphicRequest): void {
   const url = request.url.toString();
   const method = String(request.method);
   const headers = request.headers.raw();

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: request.url.toString(),
      body: request.body,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);
}

function _handleHttpResponse(request: IsomorphicRequest, response: IsomorphicResponse): void {
   const url = request.url.toString();
   const headers = request.headers.raw();


   const responseEvent: HttpResponseEvent = {
      url: request.url.toString(),
      method: request.method,
      body: response.body,
      headers: response.headers.raw(),
      statusCode: response.status,
   };

   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);
}

Caviats

Although the final solution is more general and is also agnostic to the client HTTP library used, there are some downsides:

Since all HTTP traffic going thorugh the app is intercepted, there needs to be some logic in place to know which requests to ignore. E.g., instrumentation tools like NewRelic regularly send requests to capture metadata. This can add a lot of noise if not handled properly
Dependence on another library. Whether this is a big deal depends on what the interception is used for. Probably not a big deal for most projects

Populating a pre-allocated array slower than a pushing to a regular array?

Henry Williams — Sun, 19 Jul 2020 16:34:27 +0000

For no good reason, I got the urge to do a performance comparison between populating an array by pushing to it vs writing to a buffer. Then, to make things more interesting, I decided to add a static array and a pre-allocated standard array.

Let's just say the results were not what I expected.

Experiment

Populate the 4 data structures by adding 10^8 elements to each and comparing the time it took for each of them.

Candidates

Static array - populated by writing directly to the index
Pre-allocated dynamic array - array initialized to hold all elements and then populated by setting elements for each index
Buffer - populated by writing directly the offset. Should be similar to writing to an index, but there might be some internal overhead
Array - empty array populated by pushing elements to it

Expected results (from fastest to slowest)

Static array
Pre-allocated array
Buffer
Array

Actual results (from fastest to slowest)

Static array (228.039ms)
Buffer (1.135s)
Array (2.545s)
Pre-allocated array (6.062s) (Why so slow???)

What I don't understand is why the pre-allocated array performed so poorly. I would expect its performance to be on par with a static array. I definitely didn't expect it to be outperformed by an array built by pushing elements into it.

Code

const NUMBER_OF_ELEMENTS = 10**8
const ELEMENT_LEN_BYTES = 4

const array = []

console.time('array')

for (let i = 1; i <= NUMBER_OF_ELEMENTS; i++) {
    array.push(i)
}

console.timeEnd('array')

const preAllocatedArray = new Array(NUMBER_OF_ELEMENTS)

console.time('pre-allocated array')

for (let i = 1; i <= NUMBER_OF_ELEMENTS; i++) {
    preAllocatedArray[i - 1] = i
}

console.timeEnd('pre-allocated array')

const intArray = new Uint32Array(NUMBER_OF_ELEMENTS)

console.time('int array')

for (let i = 0; i < NUMBER_OF_ELEMENTS; i++) {
    intArray[i] = i + 1
}

console.timeEnd('int array')


const buffer = Buffer.alloc(NUMBER_OF_ELEMENTS * ELEMENT_LEN_BYTES, 0)

console.time('buffer')

for (let i = 1, offset = 0; i <= NUMBER_OF_ELEMENTS; i++) {
    offset = buffer.writeUInt32BE(i, offset)
}

console.timeEnd('buffer')

// Results:
// array: 2.545s
// pre-allocated array: 6.062s
// int array: 228.039ms
// buffer: 1.135s

Edit: It looks like the V8 engine's optimizations favor .push() over direct index assignment. The findings for Chrome in [this (ancient) article] are consistent with my results on Edge, Chrome, and Nodejs; all of which run on top of the v8 engine.

Thanks @alain Van Hout for sharing the link in the comments.

If anyone has any ideas how those optimizations are performed, please do share 🙂

Get more type help from VS Code

Henry Williams — Sun, 19 Jan 2020 17:21:44 +0000

Intro

This is a very short tutorial for getting more help on potential Javascript typing errors.

Type linting

VS Code offers great type inference. However, one of the most powerful feature for JS type linting is disabled by default.

Linting disabled (default)

Linting enabled

Enabling linting

Enabling this type check is surprisingly each. All you have to do is go to your User Settings and enable javascript.implicitProjectConfig.checkJs.

That's it! You've taken a step towards improving your workflow by catching potential type errors.

More on using types

When writing Javascript, I highly recommend using JSDoc annotations for most of your code for the best type inference experience. With JSDoc, VS Code will be able to provide much better type autocompletion, which makes it easier for you and others to understand and maintain your code.
Check out my other article on Javascript typings if you'd like to learn more about this.

How do you implement system testing?

Henry Williams — Tue, 17 Dec 2019 17:20:32 +0000

Just curious as to how other developers implement system testing as part of their continuous integration processes. If you don't do system testing, what's preventing you from doing it?

We're currently not implementing it in my team, but I'm looking for ideas and opinions on the subject.

My idea right now is to have a sandbox integration environment with all of the services running in it. Each service would have its own set of system tests defined. So when a build kicks off, the CI system will deploy the service into the integration environment and run all of the tests against it.

If the test pass, then it'll proceed to deploy to the target environment. Otherwise, it would roll back the service and mark the build as failed.

Better Javascript Type Autocomplete with JSdoc

Henry Williams — Sat, 20 Jul 2019 02:04:46 +0000

If you want to skip the lecture and get straight to the code, here's the repo with all of the examples.

Intro

Whether you like types or not, there's no denying that having autocomplete for types in any language is nice since it means we don't have to run the code to know what the object should look like (I'm looking at you, JavaScript).

VS Code, for example, partially solves this problem by automatically importing the type definitions for the any of the libraries you import, if available, behind-the-scenes.

So, whenever VSCode autocompletes your express methods, there's no magic! It's pulling the the DefinitelyTyped definitions behind the scenes.

However, the VSCode (and other editors) solution is limited because many times there's no way for the editor to know, for instance, the types of parameters a function. See example

Type Autocompleted

Type Not Autocompleted

Solutions

There are a few ways to solve this problem. The way you choose will depend on your project requirements, as well as in your preferences and needs.

In-line JSdoc type definition

This approach allows you to define custom definitions in-line. The downside is that the types defined this way can't be reused outside of the file they were defined in.

Imported type definition

This approach allows you to add types from existing NPM modules to your code. To implement this, you'll first have to npm install the appropriate type definitions into your project. For instance, to add express types, you'll need to npm install --save-dev @types/express.
You can find any types you need at DefinitelyTyped.

Conclusion

As I've shown, there are several ways to improve type autocomplete by using JSDoc annotations. Of course, if you need actual type checking you'll need to use a statically typed language like Typescript or a tool like flow.

There are also a few other uses for type annotations that I didn't go into detail in this article. Feel free to check them out in this repo.

References

https://ricostacruz.com/til/typescript-jsdoc
https://jsdoc.app/tags-typedef.html

Using await/async in a Simple Script

Henry Williams — Thu, 23 May 2019 00:21:25 +0000

Problem:

You want to use await/async at the top-level of your simple script like so:

async function asyncFunction() {
    return new Promise(resolve => {
        //  simulate wait
        setTimeout(resolve, 1000);
    })
}

try {
    // This won't work!
    await asyncFunction()
} catch(err) {
    console.error('Something bad')
}

Not so fast! await can only be used within an async function!

await asyncFunction()
^^^^^

SyntaxError: await is only valid in async function
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:617:28)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Function.Module.runMain (module.js:694:10)
    at startup (bootstrap_node.js:204:16)
    at bootstrap_node.js:625:3

Solution:

Enclose your code in an Immediately-Invoked Function Expression (IIFE).

async function asyncFunction() {
    return new Promise(resolve => {
        //  Simulate wait
        setTimeout(resolve, 1000);
    })
}

// This works
(async () => {
    try {
        await asyncFunction()
    } catch(err) {
        console.error('Something bad')
    }
})()

What framework do you use and why?

Henry Williams — Mon, 29 Apr 2019 20:41:41 +0000

I usually use Express.js or Restify.js for simplicity. But I'm wondering if it's better to use a more full-featured framework like Kraken.js or Sails.js, especially for larger projects.

I know it varies by use-case, but I'm curious to know why I don't see as many people in the node.js community embracing such full-featured frameworks; most devs that I've come across always recommend Express or Restify.

Array.map() much slower than for loop

Henry Williams — Sun, 28 Apr 2019 22:36:49 +0000

I was working on some coding challenges when I got the sudden urge to test which approach was faster. For an input size of 1 million numbers, Array.map() takes about 2,000ms, whereas a for loop takes about 250ms. Any ideas why?

const INPUT_SIZE = 10 ** 7
const input = []
for(let i = 1; i <= INPUT_SIZE; i++) {
    input.push(i)
}

const pow2 = value => value ** 2

console.time('map')
const mapResult = input.map(pow2)
console.timeEnd('map') // 1800-2000+ ms

console.time('loop')
const arrayResult = []
for(let i = 0; i < input.length; i++) {
    arrayResult.push(pow2(input[i]))
}
console.timeEnd('loop') // 200-300ms

I always assumed Array.map() would be faster since it's a built-in function, but it looks like I was wrong.

Test 2

So after thinking about this for a while, I decided to perform a more fair comparison: Array.forEach() vs for loop. The results were that Array.forEach() is still slower, but not by as much as .map() (550-700ms).
My guess is that .map() performs some additional logic that slows it down significantly compared to a raw for loop.

Edit: I'm aware that this isn't exactly a practical scenario as we shouldn't be processing this much data using Javascript. This scenario is for theoretical understanding and discussion.

Creating a Serverless Cron Job in AWS

Henry Williams — Thu, 28 Mar 2019 23:45:21 +0000

Setting up a cron job in AWS is easy with the Serverless framework. For those unfamiliar with cron jobs, they're essentially scheduled jobs. For instance, you might have a cron job to remove old logs every 3 days.
For this quick tutorial, I'm going to assume you're at least somewhat familiar with Serverless and AWS lambdas. If you aren't, then stay tuned for a post on those :)

Please refer to this repo for the complete example.

Serverless configuration

Your serverless.yml configuration file should look something like the one below.

service: cron-job-example

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev

  region: us-east-1
  profile: serverless

functions:
  hello-cron:
    description: Example lambda function
    handler: hello-cron.handler
    events:
      - schedule: rate(1 minute)

The magic happens under the events definition. There we define a task to run every minute. You can change this depending on your needs.

You also have the option to set up a cron expression for more complex scheduling; for instance, setting a job that runs every Tuesday at 4:03 AM. Check out this AWS documentation if you want to explore these kinds of cron expressions. Note that they're a bit different than a traditional cron expression.

Setting up your environment

The environment setup is simple. All you need is a valid set of AWS credentials in your AWS config file (~/.aws/credentials in *nix). Should look something like below

[serverless]
aws_access_key_id = <access key>
aws_secret_access_key = <secret access key>

The serverless profile (the name between the brackets) is configured in serverless.yml. This can be anything, but it must match the name specified in provider.profile in your serverless.yml config. For this example, I called it serverless. Also, the profile you select should have sufficient access on your AWS account to deploy this stack.

After configuring the profile to use for the deployment, you'll need to install the dependencies by running npm install.

Deploying the job

This is the easiest part. Once you've set everything up, simply run npm run sls:deploy and Serverless will take care of the rest.

After deploying, you should be able to go to your CloudWatch logs and see the log entries generated by the Lambda function.

If you look at the code for the Lambda function in the example repo, you'll notice that the job itself doesn't do much. It just logs "hello cron" whenever executed. I'll leave it up to you to make it do something more useful :)

module.exports.handler = () => {
    console.log('hello cron')
}

Wait, what?

There's no magic to how this all works. Although, Serverless does create a few AWS resources behind-the-scenes to support this functionality.

CloudFormation stack
Lambda function
CloudWatch event to trigger the Lambda function
CloudWatch logs

Conclusion

There you have it. With a simple Serverless setup you can have a scheduled job running in AWS in no time!

Don't trust SemVersioning in NPM Modules

Henry Williams — Sat, 16 Feb 2019 02:47:10 +0000

Problem

{
    "dependencies": {
        "some_module": "^0.3.8",
        "some_other_module": "~0.1.3",
        "dont_do_this": "*"
    }
}

Do any of those patterns look familiar to you? Yes? Then I sure hope you know whether all of your installed packages follow semantic versioning (semver)! Otherwise these could be ticking time bombs waiting to blow up in your future deployment.

Some packages follow semver better than others, but, from my experience, the ones you should be most cautious with are the sub-1.0 packages because they definitely don't follow semver; they typically use the second digit (0.x) as the major version and the third digit (0.1.x) as the patch or feature version! So the next time you run your build for a package in version 0.x you might be in for a nasty surprise.

For a more concrete example, let's use knex, which is currently in version 0.16.3. Now, if you specify in your package.json to install version 0.x, then this means that the next time you run a deployment you could be installing version 0.17.0, which might contain breaking changes that impact your app!

Solution

At this point, you might be wondering how you could solve this problem. Luckily for us, the solution is pretty simple: lock down all of the package versions! Your package.json should look more like the one below to avoid any surprises.

{
    "dependencies": {
        "some_module": "0.3.8",
        "some_other_module": "0.1.3",
    }
}

Of course, you can also relax the rules for any packages that you know follow semver. For instance, you're probably safe to trust packages like react (v16.8.2) and restify (v7.7.0) since they're both pretty popular projects with many major releases.

Edit

As mentioned in the discussion, this issue can also be mitigated by keeping your package-lock.json up to date. This works because once a version is specified in the package-lock.json, it won't be updated if it meets the required version specified in package.json. For instance, 0.24.0 fulfills the requirement for 0.x, so even if 0.25.0 is released, npm will continue to install and use 0.24.0.

References

https://docs.npmjs.com/about-semantic-versioning
https://docs.npmjs.com/files/package-locks