DEV Community: Davide Cavaliere

Running GitHub Actions Only for Changed Packages in a Monorepo

Davide Cavaliere — Thu, 19 Sep 2024 08:18:45 +0000

Running GitHub Actions Only for Changed Packages in a Monorepo

In a monorepo setup, you’re probably (like us at Microgamma) using Lerna to bump package versions, followed by some GitHub Actions to test, build, and deploy them.

One challenge we faced was: how can we run a job only if a specific package has changed?

Let’s walk through a simple GitHub workflow that shows our approach.

on:
  pull_request:
    branch: [master]

  workflow_dispatch:

jobs:
  version:
    runs-on: buildjet-2vcpu-ubuntu-2204

    outputs:
      tag: ${{ steps.released-tag.outputs.tag }}
      next: ${{ steps.next-tag.outputs.next }}
      changed: ${{ steps.changed.outputs.changed }}

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
          repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
          fetch-depth: 0

      - uses: buildjet/setup-node@v4
        with:
          node-version: '20.x'
          cache: 'npm'
          registry-url: 'https://registry.npmjs.org/'

      - name: Install dependencies
        run: npm ci

      - id: released-tag
        name: Grab Latest Release Tag
        run: |
          tag=`git describe --tags --abbrev=0`
          echo "tag=$tag" >> $GITHUB_OUTPUT

      - name: Get changed packages
        id: changed
        run: |
          CHANGED=$(npx lerna changed --json | jq 'map(.name)' -c)
          echo ${CHANGED}
          echo "changed=${CHANGED}" >> $GITHUB_OUTPUT

      - name: Lint
        run: npm run lint:ci

      - name: Version
        run: npm run version:ci

      - id: next-tag
        name: Grab Next Tag
        run: |
          tag=`git describe --tags --abbrev=0`
          echo "next=$tag" >> $GITHUB_OUTPUT

      - name: Check variable
        if: contains(env.changed, 'musicbox-web')
        run: echo ${{ env.changed }}


  should_run:
    runs-on: buildjet-2vcpu-ubuntu-2204
    needs: version
    if: contains(needs.version.outputs.changed, 'my-package-a')

    steps:
      - name: test
        run: echo ${{ needs.version.outputs.changed }}

  should_not_run:
    runs-on: buildjet-2vcpu-ubuntu-2204
    needs: version
    if: contains(needs.version.outputs.changed, 'my-package-b')

    steps:
      - name: test
        run: echo ${{ needs.version.outputs.changed }}

The key piece is this step:

      - name: Get changed packages
        id: changed
        run: |
          CHANGED=$(npx lerna changed --json | jq 'map(.name)' -c)
          echo ${CHANGED}
          echo "changed=${CHANGED}" >> $GITHUB_OUTPUT

Here, all changed package names are collected into a changed output variable, which can then be used in later jobs:

  should_run:
    runs-on: buildjet-2vcpu-ubuntu-2204
    needs: version
    if: contains(needs.version.outputs.changed, 'my-package-a')

Why we store both the previous and next release tags

You’ll notice we capture both the current release tag (before bumping) and the next release tag (after bumping). This is crucial because of how Lerna’s run command works.

By default, lerna run test executes the test script in every package that has changed since the latest tag. But after running lerna version, a new tag is created, which means the “latest tag” changes — and that breaks the logic for detecting what really changed.

To solve this:

We store the tag before bumping (so we can compare changes against it).
We store the new tag (so subsequent jobs can check it out).

For example:

  Build_UI:
    runs-on: buildjet-2vcpu-ubuntu-2204
    needs: Version

    if: needs.Version.outputs.next != needs.Version.outputs.tag

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ needs.Version.outputs.next }}
          repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
          fetch-depth: 0

This job runs only if a new tag was actually created.

From there, we can run builds and deploys only for packages changed since the previous release:

      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build:ci -- --since ${{ needs.Version.outputs.tag }}

      - name: Deploy
        run: npm run deploy:ci -- --since ${{ needs.Version.outputs.tag }}

Why not just use lerna run?

Technically, we could just use lerna run ... and it would skip unchanged packages. But to even get to that point, the workflow still needs to:

Run the job

Checkout the repo

Install dependencies

…which wastes build time if no relevant packages actually changed.

By filtering earlier in the workflow, we avoid unnecessary jobs altogether.
Hopefully this helps anyone else running into the same issue we did! 🚀

Avoid conflicts between denols and tsserver in Neovim

Davide Cavaliere — Fri, 09 Feb 2024 16:53:11 +0000

Recently I've been working on a project in a monorepo were I've got nodejs/typescript and deno projects. This is notoriously annoying because any ts file will start both tsserver and denols language services.

If you've got here you know what I'm talking about 😓

I'm setting up my lsp with lspconfig.

This is how denols is configured

lspconfig.denols.setup({
  root_dir = lspconfig.util.root_pattern("deno.json", "deno.jsonc"),
  init_options = {
    lint = true,
    unstable = true,
    suggest = {
      imports = {
        hosts = {
          ["https://deno.land"] = true,
          ["https://cdn.nest.land"] = true,
          ["https://crux.land"] = true,
        },
      },
    },
  },

  on_attach = on_attach,
})

And this is the hack to avoid tsserver to start on a file that belongs to a deno project.

spconfig.tsserver.setup({
  on_attach = function (client, bufnr)
    on_attach(client, bufnr);
    vim.keymap.set('n', '<leader>ro', function()
      vim.lsp.buf.execute_command({
        command = "_typescript.organizeImports",
        arguments = { vim.fn.expand("%:p") }
      })
    end, { buffer = bufnr,  remap = false });
  end,
  root_dir = function (filename, bufnr)
    local denoRootDir = lspconfig.util.root_pattern("deno.json", "deno.json")(filename);
    if denoRootDir then
      -- print('this seems to be a deno project; returning nil so that tsserver does not attach');
      return nil;
    -- else
      -- print('this seems to be a ts project; return root dir based on package.json')
    end

    return lspconfig.util.root_pattern("package.json")(filename);
  end,
  single_file_support = false,
})

This leverage the fact that if root_dir function returns nil then lspconfig wont's start a language service for that file. 🎉

Avoid conflicts between denols and tsserver in Neovim

Davide Cavaliere — Fri, 09 Feb 2024 16:53:10 +0000

If you've got here you know what I'm talking about 😓

I'm setting up my lsp with lspconfig.

This is how denols is configured

lspconfig.denols.setup({
  root_dir = lspconfig.util.root_pattern("deno.json", "deno.jsonc"),
  init_options = {
    lint = true,
    unstable = true,
    suggest = {
      imports = {
        hosts = {
          ["https://deno.land"] = true,
          ["https://cdn.nest.land"] = true,
          ["https://crux.land"] = true,
        },
      },
    },
  },

  on_attach = on_attach,
})

And this is the hack to avoid tsserver to start on a file that belongs to a deno project.

spconfig.tsserver.setup({
  on_attach = function (client, bufnr)
    on_attach(client, bufnr);
    vim.keymap.set('n', '<leader>ro', function()
      vim.lsp.buf.execute_command({
        command = "_typescript.organizeImports",
        arguments = { vim.fn.expand("%:p") }
      })
    end, { buffer = bufnr,  remap = false });
  end,
  root_dir = function (filename, bufnr)
    local denoRootDir = lspconfig.util.root_pattern("deno.json", "deno.json")(filename);
    if denoRootDir then
      -- print('this seems to be a deno project; returning nil so that tsserver does not attach');
      return nil;
    -- else
      -- print('this seems to be a ts project; return root dir based on package.json')
    end

    return lspconfig.util.root_pattern("package.json")(filename);
  end,
  single_file_support = false,
})

This leverage the fact that if root_dir function returns nil then lspconfig wont's start a language service for that file. 🎉

Create Your First Lambda with Serverless, Typescript and Microgamma Part 2

Davide Cavaliere — Sun, 26 Jan 2020 08:06:14 +0000

This is part 2 of my serie on Microgamma.

TLTR;
In part 1 we saw how to set up a project with typescript, microgamma and serverless and how to deploy a very simple endpoint

In this post we will:
1 - create our first REST api
2 - make it private adding an api-key
3 - secure our lambdas with an authorizer function

If you followed the steps in the first part then make sure to update:

yarn add @microgamma/apigator@latest @microgamma/serverless-apigator@latest

If you wan to start straight from here you can checkout:

git clone git@github.com:microgamma/serverless-apigator-blue-print.git

Or, if you already clone it before:

git remote update && git pull origin master

and checkout

git checkout -b part2 my-first-apigator-part-2

Before we start, let's build, deploy and check that everything is still working:

yarn build && yarn sls deploy --stage dev

yarn sls invoke local -f index

And:

yarn sls invoke -f index --stage dev

A Tickets service

Let's now make our service a bit more useful. Let's suppose that our endpoint will handle the requests for interacting with Tickets.

So we have:

GET / retrive all tickets.
GET /{id} retrieve a ticket by id.
POST / creates a new ticket (ticket is passed through request's body).
PUT / updates an existing ticket (ticket is passed through request's body).
DELETE /{id} deletes a ticket.

This is how our service will look like:

// my-first-service.ts

import { Body, Endpoint, Lambda, Path } from '@microgamma/apigator';
import { Injectable } from '@microgamma/digator';

@Endpoint({
  cors: true,
  name: 'my-first-service',
  private: false
})
@Injectable()
export class MyFirstService {

  private static tickets = [{
    id: 1,
    description: 'Buy beers'
  }, {
    id: 2,
    description: 'send email'
  }];

  private lastIndex = 2;

  @Lambda({
    method: 'GET',
    path: '/',
    description: 'get all tickets'
  })
  public index() {
    return MyFirstService.tickets;
  }

  @Lambda({
    method: 'GET',
    path: '/{id}',
    description: 'find a ticket by id'
  })
  public findById(@Path('id') ticketId) {

    const found = MyFirstService.tickets.find((ticket) => ticket.id === Number(ticketId));

    if (!found) {
      throw new Error('[404] not found');
    }

    return found;
  }

  @Lambda({
    method: 'POST',
    path: '/',
    description: 'create a new ticket'
  })
  public create(@Body() ticket) {
    ticket.id = ++this.lastIndex;
    MyFirstService.tickets.push(ticket);
    return ticket;
  }

  @Lambda({
    method: 'PUT',
    path: '/',
    description: 'updates a ticket'
  })
  public update(@Body() ticket) {
    const found = MyFirstService.tickets.find((t) => t.id === Number(ticket.id));

    if (!found) {
      throw new Error('[404] not found');
    }

    Object.assign(found, ticket);
    return found;
  }

  @Lambda({
    method: 'DELETE',
    path: '/{id}',
    description: 'delete a ticket by id'
  })
  public delete(@Path('id') ticketId) {
    const found = MyFirstService.tickets.find((t) => t.id === Number(ticketId));

    if (!found) {
      throw new Error('[404] not found');
    }

    const indexOf = MyFirstService.tickets.indexOf(found);

    MyFirstService.tickets.splice(indexOf, 1);

    return { delete: 'ok' };
  }
}

In the above snippet we can notice two new decorators: @Path('id') which retrives id from the path parameters and @Body() that retrives the request's body.

Let's build again:

yarn build

And call those new lambda locally

yarn sls invoke local -f index

[
    {
        "id": 1,
        "description": "Buy beers"
    },
    {
        "id": 2,
        "description": "send email"
    }
]

yarn sls invoke local -f findById -d '{ "path": { "id": "1" } }'

{
    "id": 1,
    "description": "Buy beers"
}

yarn sls invoke local -f create -d '{ "body": { "id": 3, "description": "my new ticket" } }'

{
    "id": 3,
    "description": "my new ticket"
}

yarn sls invoke local -f update -d '{ "body": { "id": 1, "description": "buy beers and candies" } }'

{
    "id": 1,
    "description": "buy beers and candies"
}

yarn sls invoke local -f delete -d '{ "path": { "id": "1" } }'

{
    "delete": "ok"
}

Every call should work just fine, so let's try to deploy.

yarn sls deploy --stage dev

We can call again the same commands as above without local and adding --stage dev

Or we could use postman for a quick e2e test.

Protect your Lambdas with an Api Key

In order to achieve this you just need to change the @Endpoint decorator as follows


@Endpoint({
  cors: true,
  name: 'my-first-service',
  private: true
})
@Injectable()
export class MyFirstService {

And add the following to your serverless.yml

--- a/serverless.yml
+++ b/serverless.yml
@@ -2,6 +2,8 @@ service: my-first-apigator # NOTE: update this with your service name

 provider:
   name: aws
+  apiKeys:
+    - myFirstKey
   runtime: nodejs12.x
   stage: dev
   region: eu-west-2

Build and deploy. We should see something like the following in our terminal.

api keys:
  myFirstKey: XNzWxRlNEb3mJynlLi8mDamcdL6psXmx4MWHgBCe

So now if we try to hit our endpoint we'll get a 403 Forbidden error.
Adding the above api key to our headers will let us reach our data.

Bear in mind that the header name should be: x-api-key

Add an authorizer function

Having a lambda that is called by another lambda to check whether the request comes from authorized client it a common pattern. It's most simple implementation it will check the validity of a jwt token passed through the headers of the request. So that if for example the used is authenticated it would have provided a valid jwt token.
For now we're gonna implement something way simpler but still a valid point in order to show how to do it in Microgamma.

First of all we add an authorize method to our service.

  @Authorizer()
  public authorize(@Header('Authorization') jwt) {
    return jwt === 'jolly';
  }

Decorated with @Authorizer which tells serverless that this is an authorizing function.

We also use here @Header('Authorization') which, alike @Path, extracts from the headers the field Authorization.

Then we need to set this authorizer into the lambdas we want to protect. I.e. the create lambda

  @Lambda({
    method: 'POST',
    path: '/',
    description: 'create a new ticket',
    authorizer: 'authorize'
  })
  public create(@Body() ticket) {
    ticket.id = ++this.lastIndex;
    MyFirstService.tickets.push(ticket);
    return ticket;
  }

After build and deploy. If try to hit our endpoint with 'POST' we'll get the following.

{
    "message": "Unauthorized"
}

So now we need to provide the Authorization header to our request.

That's all folks. Hope you enjoyed and thanks for reading. Any feedback is very welcome and stay tuned for more @microgamma sugar.

Create Your First Lambda with Serverless, Typescript and Microgamma

Davide Cavaliere — Thu, 16 Jan 2020 15:16:50 +0000

@microgamma is a framework that I'm developing to make lamdba developing feel like a breeze. With it we can create microservices writing Typescript classes, bind class methods to specific lambda and use the serverless framework for deploying and monitoring without bothering writing much of the configuration needed into serverless.yml. @microgamma can also be used without serverless but I think that there is where it shines.

In this first post we'll use @microgamma/apigator and @microgamma/serverless-apigator serverless plugin. It is the first of a series of posts where we'll go through the creation of a microservice REST endpoint with DynamoDB and MongoDB including unit and integration tests.

In this first post we'll focus on

create a new Typescript project
setup serverless framework and add @microgamma sugar
create a simple lambda
deploy to aws

You can checkout the code hereby discussed by cloning this repo at the tag: my-first-apigator if you clone the repo you can skip the below section or just follow the readme in there.

Create a Typescript project

We'll use typescript-starter and I have to admit that it's not optimal for us right now as we'll need to clean some stuff up, though more other things are already done so we can leave with it for now.

npx typescript-starter my-first-apigator --node --yarn --no-vscode

Clean up

First we want to enable typescript decorators support.

--- a/tsconfig.json  
+++ b/tsconfig.json  
@@ -32,8 +32,8 @@  
     "pretty": true /* Stylize errors and messages using color and context. */,  

     /* Experimental Options */  
-    // "experimentalDecorators": true /* Enables experimental support for ES7 decorators. */,  
-    // "emitDecoratorMetadata": true /* Enables experimental support for emitting type metadata for decorators. */,  
+     "experimentalDecorators": true /* Enables experimental support for ES7 decorators. */,  
+     "emitDecoratorMetadata": true /* Enables experimental support for emitting type metadata for decorators. */,  

     "lib": ["es2017"],  
     "types": ["node"],

then we need to remove src/lib, src/types and delete the content of index.ts and clean tslint.json from Functional style rules.

--- a/tslint.json  
+++ b/tslint.json  
@@ -18,17 +18,9 @@  
     "no-let": true,  
     "no-object-mutation": true,  
     "no-delete": true,  
-    "no-method-signature": true,  
+    "no-method-signature": true  

-    // Functional style rules  
-    "no-this": true,  
-    "no-class": true,  
-    "no-mixed-interface": true,  
-    "no-expression-statement": [  
-      true,  
-      { "ignore-prefix": ["console.", "process.exit"] }  
-    ],  
-    "no-if-statement": true  
     /* end tslint-immutable rules */  
+  
   }  
 }

then also

--- a/tslint.json
+++ b/tslint.json  
@@ -10,15 +10,13 @@  
     // Recommended built-in rules  
     "no-var-keyword": true,  
     "no-parameter-reassignment": true,  
-    "typedef": [true, "call-signature"],  

     // Immutability rules  
     "readonly-keyword": true,  
     "readonly-array": true,  
     "no-let": true,  
     "no-object-mutation": true,  
-    "no-delete": true,  
-    "no-method-signature": true  
+    "no-delete": true  

     /* end tslint-immutable rules */

Lastly we want to remove the build:module from package.json and remove tsconfig.module.json

--- a/package.json  
+++ b/package.json  
@@ -12,7 +12,6 @@  
     "describe": "npm-scripts-info",  
     "build": "run-s clean && run-p build:*",  
     "build:main": "tsc -p tsconfig.json",  
-    "build:module": "tsc -p tsconfig.module.json",  
     "fix": "run-s fix:*",  
     "fix:prettier": "prettier \"src/**/*.ts\" --write",  
     "fix:tslint": "tslint --fix --project .",

and that's all for now.

Setup Serverless Framework and microgamma

yarn add -D serverless @microgamma/serverless-apigator

yarn add @microgamma/apigator @microgamma/digator

Create serverless.yml file with the following content

service: my-first-apigator # NOTE: update this with your service name  

provider:  
  name: aws  
  runtime: nodejs12.x  
  stage: dev  
  region: eu-west-2  


plugins:  
  - '@microgamma/serverless-apigator'  

custom:  
  apigator:  
    entrypoint: handler  
    buildFolder: build/main

Create your first service

Create src/my-first.service.ts file with the following content

// src/my-first.service.ts
import { Endpoint } from '@microgamma/apigator'; 
import { Injectable } from '@microgamma/digator';


// Define a new endpoint. This will be able to serve all the paths under the root `/` of your endpoint
@Endpoint({  
  // enable cors for this endpoint
  cors: true, 
  // just a reference name
  name: 'my-first-service',
  // this endpoint is public  
  private: false  
})  
@Injectable()
export class MyFirstService {}

With it we are defining a new endpoint without any lambda (method) yet.

We need to export a bootstrapped version of the service in src/handler.ts file such as:

// src/handler.ts
import { bootstrap } from '@microgamma/apigator';  
import { MyFirstService } from './my-first.service';  

export = bootstrap(MyFirstService);

This will be the code that will actually execute in aws. What the bootstrap does is basically to create a new instance of the given service.

Now let's check that it builds running yarn build.
A build folder shall be in your path with the following structure :

build/  
└── main  
    ├── handler.d.ts  
    ├── handler.js  
    ├── my-first.service.d.ts  
    └── my-first.service.js

Now let's create a first, very simple lambda:

// src/my-first.service.ts
import { Endpoint, Lambda } from '@microgamma/apigator';  
import { Injectable } from '@microgamma/digator';

@Endpoint({  
  cors: true,  
  name: 'my-first-service',  
  private: false  
})  
@Injectable()
export class MyFirstService {  

  @Lambda({  
    method: 'GET',  
    path: '/'  
  })  
  public index() {  
    return `Hello world! Today is ${new Date().toISOString()}`;  
  }  
}

Deploy to aws

Build it again yarn build and deploy yarn sls deploy --stage dev

Output shall contain the following:

endpoints:  
  GET - https://xxxxxx.execute-api.eu-west-2.amazonaws.com/dev/

Which is the url of your lambda. Open your browser and hit that url or curl it ;)

You can also use serverless invoke command to invoke it from aws or locally.

Extra points

Deploy only what's strictly needed excluding everything but the build and node_modules folders.

--- a/serverless.yml  
+++ b/serverless.yml  
@@ -1,3 +1,5 @@  
 service: my-first-apigator # NOTE: update this with your service name  

 provider:  
@@ -14,3 +16,10 @@ custom:  
   apigator:  
     entrypoint: build/main/handler  
     buildFolder: /home/davide/dev/my-first-apigator/build/main  
+  
+package:  
+  exclude:  
+    - ./*  
+    - ./**  
+    - '!build/**'  
+    - '!node_modules/**

Add serverless monitoring. Check on serverless.com how to enable monitoring

--- a/serverless.yml  
+++ b/serverless.yml  
@@ -1,3 +1,5 @@  
+org: davidecavaliere  
+app: my-first-apigator  
 service: my-first-apigator # NOTE: update this with your service name  

 provider:

That's all folks. Hope you enjoyed and thanks for reading. Any feedback is very welcome and stay tuned for more @microgamma sugar.

A State Manager For Angular That Makes Sense, At Least For Me :)

Davide Cavaliere — Fri, 11 Oct 2019 09:29:34 +0000

Forewords: In the following post I give an example of how to use @microphi/store for managing state in angular apps. The project is just a PoC at the time of writing. You can find more here https://github.com/microph1/microphi/tree/master/projects/store.
The code snippets you find below are part of an example app that can be found at https://github.com/microph1/ticket-store-example

I wanted to get my head around state management in angular. So I went for ngrx. Watched some videos, read some tutorials. A big mess in my head and not really sure what I was doing. Boilerplate code? Yeah, sad days!

Then I gave a look at Reduxjs. Much better. Less boilerplate. Much less documentation to go through. But still some boilerplate and also the fact that you need to write "pure" functions in an environment where you thought yourself to use classes and OOP didn't make much sense in my head. Yeah maybe it's easier to test but still.

So I started to write a state manager myself. Happy days!

Let's suppose you've got your brand new angular app with a service to handle tickets.

// ticket.service.ts
import { Injectable } from '@angular/core';
import { Observable, of, throwError } from 'rxjs';
import { delay, map, tap } from 'rxjs/operators';
import { Ticket, User } from './ticket.interface';

function randomDelay() {
  return Math.random() * 4000;
}

@Injectable()
export class TicketService {
  storedTickets: Ticket[] = [
    {
      id: 0,
      description: 'Install a monitor arm',
      assigneeId: 111,
      completed: false
    },
    {
      id: 1,
      description: 'Move the desk to the new location',
      assigneeId: 111,
      completed: false
    },
    {
      id: 2,
      description: 'This ticket assignee has been deleted',
      assigneeId: 112,
      completed: false
    }
  ];

  storedUsers: User[] = [{ id: 111, name: 'Victor' }];

  lastId = 1;

  private findTicketById = id => this.storedTickets.find(ticket => ticket.id === +id);
  private findUserById = id => this.storedUsers.find(user => user.id === +id);

  tickets() {
    return of(this.storedTickets).pipe(delay(randomDelay()));
  }

  ticket(id: number): Observable<Ticket> {
    return of(this.findTicketById(id)).pipe(delay(randomDelay()));
  }

  users() {
    return of(this.storedUsers).pipe(delay(randomDelay()));
  }

  user(id: number) {
    const user = this.findUserById(id);
    return user ? of(user).pipe(delay(randomDelay())) : throwError('User not found');
  }

  newTicket(payload: { description: string }) {
    const newTicket: Ticket = {
      id: ++this.lastId,
      description: payload.description,
      assigneeId: null,
      completed: false
    };

    return of(newTicket).pipe(
      delay(randomDelay()),
      tap((ticket: Ticket) => this.storedTickets.push(ticket))
    );
  }

  assign(ticketId: number, userId: number) {
    const foundTicket = this.findTicketById(+ticketId);
    const user = this.findUserById(+userId);

    if (foundTicket && user) {
      return of(foundTicket).pipe(
        delay(randomDelay()),
        map((ticket: Ticket) => {
          ticket.assigneeId = +userId;
          return ticket;
        })
      );
    }

    return throwError(new Error('ticket or user not found'));
  }

  complete(ticketId: number, completed: boolean) {
    const foundTicket = this.findTicketById(+ticketId);
    if (foundTicket) {
      return of(foundTicket).pipe(
        delay(randomDelay()),
        map((ticket: Ticket) => {
          ticket.completed = completed;
          return ticket;
        })
      );
    }

    return throwError(new Error('ticket not found'));
  }
}

Now what we want to achieve is to add a state manager on top of this service. Let's start:

npm i --save @microphi/store

Create some interfaces that will be useful later:

export interface User {
  id: number;
  name: string;
}

export interface Ticket {
  id: number;
  description: string;
  assigneeId: number;
  completed: boolean;
  assignee?: User;
}

export type TicketWithState = Ticket & { isLoading?: boolean };

export interface TicketsState {
  tickets: TicketWithState[];
}

Now let's create an enum with all the actions our store will handle.

export enum TicketActions {
  FIND_ALL, // will retrive all tickets
  FIND_ONE, // will retrive one ticket
  CHANGE_STATUS // will change the status done/undone of a ticket
}

Let's create our store

// ticket.store.ts

@Store({
  name: 'ticketStore',
  initialState: { tickets: [] },
  actions: TicketActions
})
@Injectable()
export class TicketStore extends BaseStore<TicketsState> {
  public tickets$ = this.store$.pipe(
    map((state) => state.tickets)
  );

  constructor(private ticketService: TicketService) {
    super();
  }

[...]

Within the @Store decorator we define the name of the store, it's initial state and we provide the enum of our actions.

One thing to notice here is that this.store$ is an observable on which internally every and each new store state will be nexted. It's defined in the parent BaseStore class and there it's protected so that we always need to map it somehow when we want to expose it. And of course we're gonna need the TicketService.

Before we go further a bit of philosophy. Bear with me though because what follows is not about Redux/Flux neither about ngrx. It is instead my interpretation of the above to make it easier to digest and to use.

The approach is as follows:


somewhere in our app -> action -> effect -> stateupdate

Simply somewhere in our app we dispatch an action which will trigger an (optional) effect which will trigger a state change. That state change will be "visible" to everyone that subscribed to the tickets$ observable, for example.

Bear in mind that asynchronous stuff needs to be handled in an effect.

Right, now: before we start writing any effect let's set up our component so that we get the tickets$.


@Component({
  selector: 'app-tickets',
  templateUrl: './tickets.component.html'
})
export class TicketsComponent implements OnInit {

  public tickets$ = this.store.tickets$;

  constructor(private store: TicketStore) { }


  public ngOnInit(): void {
    this.store.dispatch(TicketActions.FIND_ALL);
  }
}

What we need now to create an effect to retrive the list of tickets from our backend.

// ticket.store.ts


  @Effect(TicketActions.FIND_ALL)
  private getTickets(state: Ticket[], payload) {
    let numberOfTickets = 0;

    return this.ticketService.tickets().pipe(
      switchMap((tickets) => {
        console.log('parsing tickets', tickets);
        numberOfTickets = tickets.length;
        return from(tickets);
      }),
      tap((ticket) => {
        console.log('parsing ticket', ticket);
      }),
      mergeMap((ticket: Ticket) => {

        return this.ticketService.user(ticket.assigneeId).pipe(
          map((user) => {
            ticket.assignee = user;
            return ticket;
          }),
          catchError(err => {
            console.error(err);
            // silently fail
            ticket.assignee = {
              name: `unable to find user with id ${ticket.assigneeId}`,
              id: ticket.assigneeId
            };

            return of(ticket);
          })
        );
      }),
      bufferCount(numberOfTickets)
    );
  }

Define a method decorated with @Effect(actionName). This function will be invoked whenever the given actionName is dispatched.

Methods decorated with @Effect must return an observable!

tickest() returns an array of tickets which is switchMaped into each element. Then we mergeMap it with the user associated with the assigneeId field if it exists; if not just silently fail. Each element is bufferCounted so that the observable returns an array.

In case of error we swallow it and return the original ticket with a convenience user attached to it.

Of course it can be arguable to fetch the tickets in a different way depending of what you actually want to achieve so bear with this code for demonstration purpose.

At this point we dispatched the FIND_ALL event internally the getTickets method is invoked and it's return value gets subscribed to. Once a value gets through we will next the actions$ observable again this time with a "complementary" action which will trigger the reducing method associated with FIND_ALL.

Let's write our reducer:

// ticket.store.ts

  @Reduce(TicketActions.FIND_ALL)
  private onResponse(state, payload: Ticket[]) {

// REM: 
// initial state would be { tickets: [] }
// the payload is the data coming through from the associated @Effect[ed] method

    this.state.tickets.push(...payload);

    return state;
  }

Internally the framework checks for the method associated to this action, i.e.: onResponse, invokes it and its return value is set into the _state private property through a setter which will trigger a store$.next with it.

And now you only need to use an async pipe in your view in order to see the tickets magially appear on the screen.

<pre>
  {{tickets$ | async | json}}
</pre>

Hint: if you're running the example app from the repo try to open the console and type: localStorege.debug = 'microphi:*' and you'll see some of the magic that is happening.

Cool, but what about loading state?

The store has a public property called loading$ that is is nexted with and event containing three fields type, payload and status. Obviously status will tell us whether loading is in process being true or false. type will contain information about the event being loading and the payload is the payload we passed to the dispatch method.

So for example if you want to catch any loading event going on within your effects you can just do.

// tickets.component.ts
  constructor(private ticketStore: TicketStore) {
    this.ticketStore.loading$.subscribe((status) => {

      // we can't use an async pipe for this as it would subscribe to the observable after ngOnInit hence we miss
      // the loading start event;
      this.loading = status.status;

    });

Or we may want to filter out only the loading for a specific event


  constructor(private authStore: AuthStore, private ticketStore: TicketStore) {
    this.ticketStore.loading$.pipe(
      filter((event) => {
        return event.type === this.ticketStore.getRequestFromAction(TicketActions.FIND_ALL);
      })
    ).subscribe((status) => {

      // we can't use an async pipe for this as it would subscribe 
      // to only after ngOnInit hence we miss the loading 
      // start event
      this.loadingTickets = status.status;

    });

Here getRequestFromAction is an helper function to map the FIND_ALL action to its "complementary" action.

That's all for now folk.

Let me know what you think in the comment below.

RxJS: Caching Observables with a Decorator

Davide Cavaliere — Tue, 03 Sep 2019 12:51:55 +0000

Edit: The decorator hereby discussed is now available from npm. Install with npm i @microphi/cache or yarn add @microphi/cache

I have been chasing this one from few months now.
The following should be pretty familiar to you:

@Injectable()
export class UserService {

  constructor(private _client: HttpClient) {}



  public findAll(id: number) {
    return this._client.get(`https://reqres.in/api/users?page=${id}`);
  }
}

For some reason you want to cache the response of this request.
If you look online you may find some tips on how to do it and you may end yourself doing the same thing for all the requests that you want to cache.

It happens though that I was a java developer and remember the old good days when a @Cache annotation would leverage me from a lot of repeated code.

Well in Typescript we have decorator so there must be a way to do some caching with a simple @Cache, right?

My gut feeling was: of course!

But after several attempts with no success I gave up.

Until some days ago when I found this article about caching and refreshing observable in angular by Preston Lamb which re-ignited my curiosity.

Starting from his stackbliz example I did some experiments
but again without any luck.

Until I've got an intuition: let's make a race.

@Injectable()
export class UserService {

  private cached$: ReplaySubject<any> = new ReplaySubject(1, 2500);

  constructor(private _client: HttpClient) {}

  public findAll(id): Observable<any> {

    const req = this._client.get(`https://reqres.in/api/users?page=${id}`).pipe(
      tap((data) => {
        this.cached$.next(data);
      })
    );

    return race(this.cached$, req);
  }

}

Et voila'. It worked! Just how I like it: simple and neat. So simple that I don't even need to explain it, right?

Now the thing is that if in your service you've got many of those methods that you need to cache then you'll need to repeat a lot of code. REM: decorator!

Let's move everything into a decorator

export interface CacheOptions {
  ttl: number;
}

export function Cache(options: CacheOptions) {

  return (target: any, propertyKey: string, descriptor) => {

    const originalFunction = descriptor.value;

    target[`${propertyKey}_cached`] = new ReplaySubject(1, options.ttl);

    descriptor.value = function(...args) {

      const req = originalFunction.apply(this, args).pipe(
        tap((response) => {
          this[`${propertyKey}_cached`].next(response);
        })
      );

      return race(this[`${propertyKey}_cached`], req);

    };

    return descriptor;
  };
}

What I do here is to initialize a variable named, for example, findAll_cached with a replay subject then replace the original function with a new function that will call the original function applying the same logic we saw before.

Then the service will look like the following:

@Injectable()
export class UserService {

  constructor(private _client: HttpClient) {}

  @Cache({
    ttl: 2500
  })
  public findAll(id): Observable<any> {
    return this._client.get(`https://reqres.in/api/users?page=${id}`)
  }

}

So beautiful!

Extra points

Now here it comes my friend that says: yo Davide that's cool but if you call that function with a different argument for sure you need to do the http call again. i.e.: different input different output. Right?

Oh right, that's easy:

export function Cache(options: CacheOptions) {

  let lastCallArguments: any[] = [];

  return (target: any, propertyKey: string, descriptor) => {

    const originalFunction = descriptor.value;

    target[`${propertyKey}_cached`] = new ReplaySubject(1, options.ttl);

    descriptor.value = function(...args) {

      let argsNotChanged = true;

      for (let i = 0; i < lastCallArguments.length; i++) {
        argsNotChanged = argsNotChanged && lastCallArguments[i] == args[i];
      }

      if (!argsNotChanged) { // args change
        this[`${propertyKey}_cached`] = new ReplaySubject(1, options.ttl);
      }

      lastCallArguments = args;

      const req: Observable<any> = originalFunction.apply(this, args).pipe(
        tap((response) => {
          this[`${propertyKey}_cached`].next(response);
        })
      );

      // despite what the documentation says i can't find that the complete is ever called
      return race(this[`${propertyKey}_cached`], req);

    };

    return descriptor;
  };
}

You can play with this code in the following stackbliz and find the complete source code on my github.
Please note that the code on github will probably move to another package in the future.

Caveats

If the method that we need to cache makes use of the typescript defaulting mechanism. i.e.:
```
    public findAll(id: number = 1) {
        ...
    }
```
and then it's called like service.findAll() it happens that the args variable will be [] an empty array as the defaulting takes place only when we call .apply so that in the following example no change of arguments it's detected
```
service.findAll()

service.findAll(2)
```

Let's look at the example in home.component of the forementioned stackbliz example


    setTimeout(() => {
      this.$log.d('starting subscriber');
      this.userService.findAll(1).subscribe((data) => {
        this.$log.d('starting subscribed');
        this.$log.d(data);
        this.users = data;

      })
    }, 0);

    setTimeout(() => {
      this.$log.d('first subscriber 1 sec later');
      this.userService.findAll(1).subscribe((data) => {
        this.$log.d('first subscribed');
        this.$log.d(data);

      })
    }, 1000);

    setTimeout(() => {
      this.$log.d('second subscriber 2 sec later');
      this.userService.findAll(1).subscribe((data) => {
        this.$log.d('second subscribed');
        this.$log.d(data);

      })
    }, 2000);

    setTimeout(() => {
      this.$log.d('third subscriber 3 sec later, ttl expired. shoult hit the endpoint');
      this.userService.findAll(1).subscribe((data) => {
        this.$log.d('third subscribed');

        this.$log.d(data);

      })
    }, 3000);

    setTimeout(() => {
      this.$log.d('fourth subscriber 4 sec later, argument changed. should hit the endpoint');
      this.userService.findAll(2).subscribe((data) => {

        this.$log.d(' fourth subscribed');

        this.$log.d(data);
      })
    }, 4000);

    setTimeout(() => {
      this.$log.d('fifth subscriber 5 sec later, argument changed. should hit the endpoint');
      this.userService.findAll(1).subscribe((data) => {

        this.$log.d(' fifth subscribed');

        this.$log.d(data);
      })
    }, 5000);

which outputs something like the following on console


[...]
third subscriber 3 sec later, ttl expired. shoult hit the endpoint
arguments are
[1]
argsNotChanged
true
this actually hit the endpoint
starting subscribed
{page: 1, per_page: 6, total: 12, total_pages: 2…}
first subscribed
{page: 1, per_page: 6, total: 12, total_pages: 2…}
second subscribed
{page: 1, per_page: 6, total: 12, total_pages: 2…}
third subscribed
{page: 1, per_page: 6, total: 12, total_pages: 2…}
[...]

As you can see when we subscribe again after the cache is expired all previous subscriptions are notified again.

Thanks for reading so far I hope you enjoyed and remember: if you like this article share it with your friends, if you don't keep it for yourself ;)

(2023/05/03) Edit: Update github repo link