Caching XHR Responses in Local Storage

Dave Cozens — Tue, 12 Oct 2021 12:23:49 +0000

So, I've been working on my site recently - https://uktidetimes.com/ - and it occurred to me that since I'm serving content that only changes once a day, there's really no reason for the same user to repeatedly hit the service.

I was already using Axios for my XHR requests, so I've put together an NPM module that wraps Axios but adds additional caching capabilities using local storage in the browser.

There's two primary benefits to this:

It reduces the amount of traffic to my API (which is running on AWS Lambda) - therefore reducing costs.
It improves load time for the user if they've already views this data

https://www.npmjs.com/package/xhr-cache-with-ttl

As a coder, I obviously want to do as little typing as possible so I wanted to take the approach of wrapping a known-awesome HTTP request library (in this case Axios) because most of the work has been done already. Nobody wants to reinvent that wheel today, thank you.

function xhrCache(method, url, options = {}) {
    return new Promise(function (resolve, reject) {
        axios[method](url, options).then((res) => {
            resolve(res)
        })
    })
}

Nice and easy. Axios wrapped in a promise, passing through the method, URL and options attributes that it already uses.

This just left me to throw together a basic caching mechanism.

I decided that I'd create two keys in local storage per cached request - one for the metadata (TTL and timestamp) and one for the response. This avoids a potentially heavy deserialisation cost of expired cache items.

I'm using the URL of the request as part of the key in local storage. This helps with diagnosing any issues but there's an argument for hashing it for brevity (and differentiating more complex requests). Maybe for another day.

The flow of the function is nice and simple.

if (cache exists && cache not expired){
    return cached content
} else {
    fetch content using Axios
    cache content
    return content
}

The whole thing comes in at less than 100 lines of code, and it's open source so feel free to make off with it :)

https://github.com/davecozens/xhr-cache-with-ttl

How to use it

const result = await xhrCache("get", url, { ttl: 600 })

Triggering Github workflows from other repositories, now with locking

Dave Cozens — Fri, 06 Aug 2021 11:17:15 +0000

OK, so we live in a world of automation and GitHub Actions, because nobody wants to deploy things themselves, right?

Sometimes you have multiple repositories, because code separation is fun and we need to be able trigger downstream builds in a different repository using github actions.

Easy, you say? Comes out of the box with GitHub Actions...

Repository A:

...
    - name: Trigger my downstream workflow
        run: |
          curl -XPOST -u "${{ secrets.GIT_CREDENTIALS }}" -H "Accept:application/vnd.github" -H "Content-Type:application/json" https://api.github.com/repos/path/to/workflow.yml/dispatches --data '{"ref": "main" }'
...

Repository X:

name: Deploy Repo X
on:
  workflow_dispatch:
    inputs:
      source:
        description: 'Source of event'
        required: false
        default: 'No source specified'
...

...and off we go. Easy peasy.

But there's a downside here. Let's say we've got multiple repositories that all need to do the same thing. If any of Repos A, B, C & D change they need to trigger a build of Repo X.

Great. No problemo.

However, if all 4 repos change at once, we get 4 builds. That'll tie up the build system for an hour. Horrible log jam. Angry developers. The works. And GitHub don't provide any kind of build throttling (are you listening GitHub? ARE YOU???)

What we need is a way to prevent multiple repos from submitting a new pending job if one of its siblings had recently done the same thing.

Sounds like a job for Redis (he said with the benefit of hindsight).

SET buildlock 1 NX EX 180

enables us to set a buildlock key in Redis with an expiry of 3 minutes. The important thing here is that if the key is successfully set, Redis returns "OK", otherwise "(nil)".

So, without further ado, here's a way of only firing workflows on an upstream repo if something else hasn't recently done so:

name: Update components
on:
  push:
    branches:
      - main
jobs:
  TriggerPlatformDeploy:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      ENVIRONMENT: "dev"
    steps:
      - name: Install redis CLI
        run: |
          sudo apt-get install redis-tools -y
      - name: Check for recent push lock
        run: |
          didSet=$(redis-cli -a ${{ secrets.REDIS_TOKEN }} -h ${{ secrets.REDIS_URL }} -p$ {{ secrets.REDIS_PORT }} --no-auth-warning SET buildlock 1 NX EX 180)
          echo "didSet=$(echo $didSet)" >> $GITHUB_ENV
      - name: Trigger workflow if not locked
        if: env.didSet == 'OK'
        run: |
          curl -XPOST -u "${{ secrets.GIT_CREDENTIALS }}" -H "Accept:application/vnd.github" -H "Content-Type:application/json" https://api.github.com/repos/path/to/workflow.yml/dispatches --data '{"ref": "main" }'
      - name: Workflow is locked
        if: env.didSet != 'OK'
        run: |
          echo Suppressing build trigger because another repo has triggered within 3 minutes

DEV Community: Dave Cozens

Caching XHR Responses in Local Storage

How to use it

Triggering Github workflows from other repositories, now with locking