DEV Community: kauppfbi

5 + 1 tips to reduce the noise of Renovate Bot

kauppfbi — Sat, 13 Nov 2021 18:22:27 +0000

Intro

As Software Engineers we love to automate stuff, especially if tasks are repeating and boring. Automated dependency updates are a very good example for such a task.

Luckily, we don't have to develop our own solution as there are multiple (free to use) bots at the market. The most popular ones are Greenkeeper, Dependabot (meanwhile integrated in GitHub) and Renovate Bot.

I personally like Renovate the most, because it comes with tons of features and configuration capabilites. While in a first view, those bots feel powerful and helpful, we quickly feel overwhelmed by the noise produced by them. Also our CI-system needs to deal with the heavy load.

Just have a look on the latest Onboarding PR in my current repository. Allthough it is quite young, Renovate wants to create 27 (!) PRs right away:

For me, it is clear, that we have to come up with some strategies to better control the automation. Renovate also recognized this problem and provides some useful tips on their documentation page. The following 5 + 1 tips are based on Renovates documentation and my practical experience of running Renovate in a Nx-based monorepository at my company.

1. Tip: Build Package Groups

The first tip is - for me - the most obvious, but also the most powerful configuration option of Renovate: Package Grouping.

If you get started with Renovate and you extend the config:base preset, you already make use of this feature, because the base-config itself contains the group:monorepos preset. This ruleset in turn contains the logic to group Dependency updates of multiple packages from one monorepository in just one PR. Here you can see an all supported packages. The most famous ones are for sure @angular/* or react packages.

If you like to extend this logic, you can simply create your own groups:



{
  "packageRules": [
    {
      "matchPackagePatterns": ["eslint"],
      "groupName": "eslint"
    }
  ]
}

The reason, why I think this feature is so powerful is, that you not only reduce the amount of PRs with ease, you can also apply most of the configuration options for that specific group while you follow totally different strategies in other groups.

2. Tip: Scheduling

The second tip is about scheduling. You could either use an existing scheduling preset or you configure the intended behaviour by hand.

I personally prefer the existing presets, because they already cover my needs. See the schedule nonOfficeHours as an example:



{
  "schedule": [
    "after 10pm every weekday",
    "before 5am every weekday",
    "every weekend"
  ]
}

Usually we don't work during this time. This means we can be sure, that builds caused by Renovate don't steal resources from our CI-system when we need it and we can focus on feature development.

Another strategy could be to apply different scheduling strategies for different dependency groups. Maybe it is suficient for you to update devDependencies just once a month while you want to stay up to date in your production dependencies asap.

3. Tip: Automerging

What are the best dependency updates? Those you didn't even notice, right? Automerging is a feature, which could help you there.
If you set automergeType=branch, you can achieve exactly this.

Renovate will first open a new branch with the update and waits for your CI system to pass all pipeline checks. If everything still works and all tests are passing, the branch is directly integrated without creating a PR. Only if the Pipeline fails, Renovate will take care to create a PR to notify you about the update and its state.

In general, I would not recommend you to use this for every dependency and update type. But for some non-critical ones, it can be very helpful. Also you need to ensure, that you can rely on your CI pipeline.

4. Tip: Limiting amount of PRs

Another useful tip is to limit the amount of (concurrent) PRs. Renovate knows two different types of limitation.

The first limitation is by hour. You can configure it with "prHourlyLimit": 3.

The second limitation is meant in total, so that you will never receive more than the configured amount of concurrent PRs. You can configure it with "prConcurrentLimit": 5

This setting might also save you some valuable resources in your CI system. Also, if you combine it with the Dashboard feature, you will most likely not loose the overview about open PRs. Instead, Renovate will open a GitHub Issue in your repository, where you can get an overview of updates, that are currently hold back by your rate limit.

On top of it, you can combine it with another configuration called priority. Here you can basically set a priority index for a dependency (group). The higher the number, the earlier an update is applied.

5. Tip: Excluding Dependencies and Update Types

The last tip might also seem obvious, but let me explain it to you.

Let's begin with exluding entire dependencies. Of course you can simply disable a entire dependency, but what I am acually aiming for are dependencies in subdirectories of your repository. Especially if you work in a monorepository, maybe with Lerna or Nx, you will most likely have multiple package.json files.
It's not always the case, that we want renovate considering these dependencies as well.

Therefore you can use the ignorePath configuration:



{
  "ignorePaths": ["**/examples/**"]
}

Other avoidable PRs are those, which you better update by hand, like major updates of your used framework, possibly with breaking changes. In many cases, framework authors provide some code mods or updaters, like the schematics in the area of Angular and Nx. Also typescript sometimes requires manual steps even for minor updates.

You can use the matchUpdateType configuration to set your intended strategy:



{
  "packageRules": [
    {
      "matchUpdateTypes": ["major"],
      "labels": ["UPDATE-MAJOR"]
    }
  ]
}

5 + 1 Tip: Disable Renovate

Last but not least, take into consideration, that you disable Renovate entirely. Allthough it is an automation solution, which should save you time, it still requires time from you.

For me, it only makes sense to use Dependency Bots within repositories, which are long-living and you really want to work on the project continuously. If that's not the case, maybe in the case of a sideproject or a learning project, I would not recommend to enable it. In these cases, the focus should be different and you most likely don't need the latest updates in your project and if you come back after weeks or months to continue, you don't want to be bothered by thousands of open PRs.

Bonus Tip: Renovate Config Validator

Have you heard about the renovate-config-validator?

If you now want to change your existing config, it is worth to verify the config before you integrate it into your main branch.

Therefore you can use the renovate-config-validator:



# Install renovate globally
npm install renovate -g

# Execute this command in the root of your repo
npx renovate-config-validator

That's all from my side. Do you know other helpful tips or strategies? If yes, write them in the comments below.

Happy Coding :-)

Why and how to use secondary entrypoints in your Angular Libraries in Nx

kauppfbi — Sat, 06 Nov 2021 09:05:53 +0000

Photo by Sangga Rima Roman Selia on Unsplash

Nx Devtools recently released version 13.1, which came with a very handy generator for Angular libraries, which creates secondary entrypoints for your Angular Libraries with ease. In this blog post we will cover why and how we should make use of this new function.

Why secondary entrypoints make sense?

By adding secondary entrypoints, we basically split our Angular libraries into multiple chunks, just like Angular Material does.
A Example:



// One entrypoint: 
import { MatButtonModule, MatCardModule } from '@angular/material';

// With secondary entrypoints:
import { MatButtonModule } from '@angular/material/button';
import { MatCardModule } from '@angular/material/card';

The idea behind it is not new at all, this option already exists since the beginning phase of ng-packagr. By leveraging secondary entrypoints, we aim better architecture, support of treeshaking, smaller bundle sizes and therefore faster applications.

Demo 👀

To demonstrate the impact, I prepared a little example:
Therefore I created a small greeter library and an a simple test-app, which just consumes one component out of this greeter library.

Here is the greeter library, which contains a simple greeter and a fancy greeter:



// simple-greeter.component.ts
import { ChangeDetectionStrategy, Component, Input } from '@angular/core';

@Component({
  selector: 'kauppfbi-blogs-simple-greeter',
  template: ` <p>Nice to meet you, {{ name }}!</p> `,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class SimpleGreeterComponent {
  @Input() name!: string;
}

// fancy-greeter.component.ts
...
import moment from 'moment';

@Component({
  selector: 'kauppfbi-blogs-fancy-greeter',
  template: `
    <p>Nice to meet you, {{ name }}!</p>
    <p>Today is the {{ time }}</p>
  `,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class FancyGreeterComponent implements OnInit {
  time = '';
  @Input() name!: string;

  ngOnInit(): void {
    this.time = moment().format('DD.MM.YYYY');
  }
}

Both are exported in the index.ts file (public api) of the library:



export * from './lib/simple-greeter/simple-greeter.module';
export * from './lib/simple-greeter/simple-greeter.component';

export * from './lib/fancy-greeter/fancy-greeter.module';
export * from './lib/fancy-greeter/fancy-greeter.component';

You probably noticed, that the fancy greeter imports moment... you already know, where this leads to.

Our test-app is also kept simple:



// app.module.ts
import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { SimpleGreeterModule } from '@kauppfbi-blogs/greeter';
import { AppComponent } from './app.component';

@NgModule({
  declarations: [AppComponent],
  imports: [BrowserModule, SimpleGreeterModule],
  bootstrap: [AppComponent],
})
export class AppModule {}

// app.component.ts
@Component({
  selector: 'kauppfbi-blogs-root',
  template: `<kauppfbi-blogs-simple-greeter [name]="name"></kauppfbi-blogs-simple-greeter>`,
})
export class AppComponent {
  name = 'Duke';
}

To complete the demonstration, we will use the webpack-bundle-analyzer to inspect the production bundle of the app:



# build app with production configuration and generate bundle statistics
npx nx test-app:build:production --stats-json
# inspect the bundle
npx webpack-bundle-analyzer dist/apps/test-app/stats.json

This will open up a new webpage containing the chunks of our app:

As you can see, our bundle consist largely of moment even though we did not consume the fancy greeter, but the simple greeter. 🐵

How to use the new generator?

Now as we know, why it makes sense to use secondary entrypoints in Angular Libraries, we will look into how to use them.

As it is just one command, it is pretty simple:



npx nx generate @nrwl/angular:library-secondary-entry-point \
--name=my-secondary-entrypoint \
--library=my-library

or with the Nx Console:

The command will scaffold a new folder with the name of your entrypint containing a src folder with a Readme, a package.json and a module exported by a new index.ts file. In addition, it will adjust the lintFilePattern of the entire library project and it will create the correct pathMapping in the tsconfig.base.json.

Now you can either move existing modules there or create new ones.
In our use case, it was a matter of two minutes to shift the code.

Let's analyze the result:

First of all, we can see that the bundle size was decreased significantly. Okay, this was just a demo and I wouldn't recommend to use moment nowadays at all. But, more importantly we verified that we only load necessary code and we are able to support proper treeshaking with our solution!

That's it already, happy coding! 🚀

GitHub Repository

All the code is available in a public repository on Github. You can find the greeter library here and the test-app here.

Further Links 🔗

Checkout the following blog posts to learn more about seondary entrypoints:

Improve SPA performance by splitting your Angular libraries in multiple chunks by Kevin Kreuzer
Creating Secondary Entry Points for your Angular Library by Christian Ing Sunardi

Test MSAL based SPAs with Cypress

kauppfbi — Sat, 30 Oct 2021 18:30:08 +0000

This post is about how to properly handle authentication of microsoft authentication library (msal) based single page applications in cypress e2e tests.

Why is it important?

In many cases modern single page applications are protected via authentication & authorization mechanisms like OAuth. Of course we want to test our applications properly to develop and deploy with the best possible confidence. What is better than writing some e2e tests (with Cypress)?

But wait... How do we handle authentication within the tests? Isn't there a redirect to the login page of the identity provider (IDP) triggered by the app? How do we pass user credentials? What about multi-factor-authentication?

A lot of questions need to be clarified, before we can actually start writing functional tests. Altough Cypress provides some good recipes for most common authentication protocols and providers and there are even some packages out there, which abstract the login logic, authentication against the Azure AD can be very tricky.

This is, why the post will focus on MSAL based apps, which use the @azure/msal-browser under the hood and authenticate against the Azure AD. But before you think, this is not of interest for you, you can still follow the steps and adapt the approach to your scenario, no matter if you are using a different IDP or underlying auth package.

Let's get started 🛫

How not to do it 🚨

Let's first talk about two approaches, you should not follow:

Login via UI:
Please don't! One of the most important rules in e2e testing - or testing in general - is to not test components, which are not under your control. You may be happy and your code will work for a while, but your tests will break sooner or later when Microsoft decides to change its login page.
Disable Auth in E2E Tests:
Another popular approach is to disable authentication via environment variables and mock missing parts in e2e tests. I personally really dislike the approach und would also not recommend it to anyone. The problem is that you have to touch your production code and implement some switches here and there to make it work (auth guards, route protection, http interceptor, ...).

While I think it's never worth changing your productive code implementation just for test purposes, the main problem is, that those switches increase the complexity and so also the risk for configuration errors. We are just humans and make silly mistakes, isn't it? Imagine deploying your app to production without correct settings for AuthN and AuthZ. Also, the significance of your e2e tests decreases.

How to do it then? ✅

To make it short, the approach is the following. We will

programatically request a access and id token
transform the response and save it to the localstorage
run some e2e tests

1. Acquire ID and Access Token

Usually modern SPAs are using OAuth either with Implicit or Authorization Code Flow (recommended in OAuth 2.1). If you are not yet familiar with the different flows, Auth0 comes with a great documentation.
However, both flows include some ping pong and redirect between SPA and IDP, so those are not fitting for our purpose.

Instead, we will use the Resource Owner Password Flow, because it is much simpler as you only need to make one REST call to receive the token.

Please consider the following:

[...] the Resource Owner Password Flow should only be used when redirect-based flows (like the Authorization Code Flow) cannot be used.
(https://auth0.com/docs/authorization/flows/resource-owner-password-flow)

Generate a client secret

However, the flow expects the client to be trusted. Because a SPA is usually considered a public and therefore not a trusted client, we need to pass not only the user credentials (username, password), but also a additional client secret. So let's create one in the Azure Portal (official docs):
Open Azure Portal > Azure AD > App Registrations > your app reg > Certificates & Secrets

Please note, that you can only see the secret once, when you create it. Afterwards you won't be able to reveal the entire secret again. Ideally, you directly save the secret in a Key Vault and consume it from there if you need it.

Create a technical user

Before we can continue with the login, we need to have a technical user account.
Please don't use your own user account, especially not in CI environment.
If you don't have one yet, you can simply create one in the Azure Portal (official docs):
Open Azure Portal > Azure AD > Users > New User

If this button is disabled for you, you may need to request a test user from your Azure AD Admin.

Here are some important points on test users:

MFA must be disabled for the test user
The test user must not be a guest user
You must login once via UI to grant consent for the requested scopes
Test users should not be shared among different apps or even test types
Do not check in user password and client secret into git
Pro Tip: Use the Azure Graph API to dynamically add and remove app role assignments before and after e2e test execution

Test the Login

Now we can finally try the login:



curl -X POST "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "grant_type=password" \
--data-urlencode "client_id=<client-id>" \
--data-urlencode "client_secret=<client-secret>" \
--data-urlencode "scope=openid profile email <api-scope>" \
--data-urlencode "username=<username>" \
--data-urlencode "password=<password>"

If the request is sucessfull, you will see a response like this:



{
    "token_type": "Bearer",
    "scope": "openid profile email <api-scope>",
    "expires_in": 3599,
    "ext_expires_in": 3599,
    "access_token": "<access-token>",
    "id_token": "<id-token>"
}

If you like you can verify the content of the base64-encoded JWTs with JWT.io. Just paste the encoded token there.

Request Login via cy.request()

Of course we don't use curl within the e2e test. Instead we can use cy.request() to perform the login via the REST API:



cy.request({
    url: authority + '/oauth2/v2.0/token',
    method: 'POST',
    body: {
        grant_type: 'password',
        client_id: clientId,
        client_secret: clientSecret,
        scope: ['openid profile email'].concat(apiScopes).join(' '),
        username: username,
        password: password,
    },
    form: true,
}).then(response => {
    // work with response.body
});

2. Prepare the localstorage

We already did important groundwork so that we can acquire a JWT with Cypress. In a next step, we need to transform the api response properly and set it in the localstorage.

Let's analyze the target structure first. This is a localstorage snapshot of my example app:

We can see, that there are a few entries in the localstorage. If those are present and their entries are valid (e.g. token is not expired), the SPA won't trigger a redirect to the IDP, when it is opened (by Cypress) and we can directly run some tests in a authenticated context 🎉.

To not blow up the blogpost endlessly, we will just focus on the entry for access token. If you are interested in the whole story, I can recommend you the Video of Joonas Westlin, where everything is explained in detail. Of course you can also directly jump to the end result, which I published on GitHub.

Recapture - our starting point

We already prepared the request to the API, which is now part of the login function:



// Get all necessary credentials out of Cypress environment
// Vriables can easily be replaced depending on the stage
const {
  tenantId,
  clientId,
  clientSecret,
  apiScopes,
  username,
  password,
} = Cypress.env();

const authority = `https://login.microsoftonline.com/${tenantId}`;
const environment = 'login.windows.net';

export const login = () => {
  let chainable: Cypress.Chainable = cy.visit('/');

  chainable = chainable.request({
    url: authority + '/oauth2/v2.0/token',
    method: 'POST',
    body: {
      grant_type: 'password',
      client_id: clientId,
      client_secret: clientSecret,
      scope: ['openid profile email'].concat(apiScopes).join(' '),
      username: username,
      password: password,
    },
    form: true,
  });


  chainable
    .then((response) => {
      // transforming the response and set localstorage
      injectTokens(response.body);
    })
    .reload()
    .then(() => {
      return tokenResponse;
    });

  return chainable;
};

As you can see, we are working with Cypress.Chainable. This is in preparation for later, when we wrap the login in a custom command.

Implement InjectTokens

At the end we need a function like this:



const injectTokens = (tokenResponse: any) => {
  // some logic in between
  // ...  

  localStorage.setItem(accountKey, JSON.stringify(accountEntity));
  localStorage.setItem(idTokenKey, JSON.stringify(idTokenEntity));
  localStorage.setItem(accessTokenKey, JSON.stringify(accessTokenEntity));
  ...
};

As mentioned, we will focus on the accessToken only. But to accomplish this, we already cover a lot.

Let's begin with the accessTokenKey. As we can see from the localstorage snapshot (Link), the key consists of several concatenated attributes:



const accessTokenKey = `${homeAccountId}-${environment}-accesstoken-${clientId}-${realm}-${apiScopes.join(
    ' '
  )}`;

Maybe you noticed, that we did not yet come across homeAccountId and the realm. Those attributes can be parsed from the idToken:



// npm i jsonwebtoken -D 
import { decode, JwtPayload } from 'jsonwebtoken';

  ...
  const idToken: JwtPayload = decode(tokenResponse.id_token) as JwtPayload;
  const localAccountId = idToken.oid || idToken.sid;
  const realm = idToken.tid;
  const homeAccountId = `${localAccountId}.${realm}`;
  ...

Ok, well. Now let's fill the value for the accessTokenKey:



const buildAccessTokenEntity = (
  homeAccountId: string,
  accessToken: string,
  expiresIn: number,
  extExpiresIn: number,
  realm: string,
  scopes: string[]
) => {
  const now = Math.floor(Date.now() / 1000);
  return {
    homeAccountId,
    credentialType: 'AccessToken',
    secret: accessToken,
    cachedAt: now.toString(),
    expiresOn: (now + expiresIn).toString(),
    extendedExpiresOn: (now + extExpiresIn).toString(),
    environment,
    clientId,
    realm,
    target: scopes.map((s: string) => s.toLowerCase()).join(' '),
    // Scopes _must_ be lowercase or the token won't be found
  };
};

Now put it all together:



const injectTokens = (tokenResponse: any) => {
  const idToken: JwtPayload = decode(tokenResponse.id_token) as JwtPayload;
  const localAccountId = idToken.oid || idToken.sid;
  const realm = idToken.tid;
  const homeAccountId = `${localAccountId}.${realm}`;

  const accessTokenKey = `${homeAccountId}-${environment}-accesstoken-${clientId}-${realm}-${apiScopes.join(
    ' '
  )}`;
  const accessTokenEntity = buildAccessTokenEntity(
    homeAccountId,
    tokenResponse.access_token,
    tokenResponse.expires_in,
    tokenResponse.ext_expires_in,
    realm,
    apiScopes
  );

  localStorage.setItem(accessTokenKey, JSON.stringify(accessTokenEntity));
};

This is already the whole concept. We are receiving the tokenResponse as input and we need to put all the things together as expected and save it to the localstorage.
We now showcased the concept with the accessToken. To complete the login, we need to repeat the exercise with idToken and the accountKey as well. You can find the end result here.

Wrap the login in a cypress custom command

As we need to run the login in every test, this is a perfect fit for a Cypress custom command. A custom command extends the existing API of Cypress and comes with the same capabilities as usual cy commands (cy.get(), ...) in terms of debuggability and retryability.



import { login } from './auth';

Cypress.Commands.add('login', () => {
  return login().then((tokenResponse) => {
    // ...
  });
});

We could already use the login now, but as it is, the actual login request is performed every single test, which means one API call per test. On the other site, a JWT is usually valid for 1h.
So we can add a simple caching mechanism as a last step:



// auth.ts
export const login = (cachedTokenResponse: any) => {
  let tokenResponse: any = null;
  let chainable: Cypress.Chainable = cy.visit('/');

  if (!cachedTokenResponse) {
    chainable = chainable.request({
      url: authority + '/oauth2/v2.0/token',
      method: 'POST',
      body: {
        grant_type: 'password',
        client_id: clientId,
        client_secret: clientSecret,
        scope: ['openid profile email'].concat(apiScopes).join(' '),
        username: username,
        password: password,
      },
      form: true,
    });
  } else {
    chainable = chainable.then(() => {
      return {
        body: cachedTokenResponse,
      };
    });
  }

  chainable
    .then((response) => {
      injectTokens(response.body);
      tokenResponse = response.body;
    })
    .reload()
    .then(() => {
      return tokenResponse;
    });

  return chainable;
};

// commands.ts
let cachedTokenExpiryTime = new Date().getTime();
let cachedTokenResponse: any = null;

Cypress.Commands.add('login', () => {
  // Clear our cache if tokens are expired
  if (cachedTokenExpiryTime <= new Date().getTime()) {
    cachedTokenResponse = null;
  }

  return login(cachedTokenResponse).then((tokenResponse) => {
    cachedTokenResponse = tokenResponse;
    // Set expiry time to 50 minutes from now
    cachedTokenExpiryTime = new Date().getTime() + 50 * 60 * 1000;
  });
});

With the last adaption, we added a inline caching mechanism. At the end, the token is only requested initially and is reused for 50 minutes, which should be enough for all e2e tests. If not, a new token is acquired automatically.

3. Write some tests

Finally we can write some tests and try out our new cy.login() command. In our scenario, we have an angular app, which has a profile page. This page displays some data from the /me endpoint of MS Graph API:

This is how we can test the page and its content:



describe('angular-msal-example', () => {
  beforeEach(() => {
    cy.login();
  });

  it('should display user data', () => {
    cy.visit('/#/profile')
      .fixture('user.json')
      .then((expectedUser) =>
        cy
          .get('[data-cy="firstName"]')
          .should('have.text', `First Name: ${expectedUser.firstName}`)
          .get('[data-cy="lastName"]')
          .should('have.text', `Last Name: ${expectedUser.lastName}`)
          .get('[data-cy="email"]')
          .should('have.text', `Email: ${expectedUser.email}`)
          .get('[data-cy="id"]')
          .should('have.text', `Id: ${expectedUser.id}`)
      );
  });
});

The expected user data is saved as fixture in /fixtures/users.json

Last but not least, let's see the tests turning green 🥇:

Summary

In this post, we learned several aspects of authentication handling in e2e tests with cypress:

How not to implement authentication in e2e tests
How to acquire id- and access-token via Resource Owner Password Flow
How msal works under the hood
How to fake the localstorage for msal based SPAs
How to wrap the login in a cypress custom command
How to reuse the tokenResponse for multiple tests

Next steps 🚵

As always, software is never ready and this is just the beginning.
I can imagine the following activities:

Integrate the tests in your CI-Pipeline.
Make use of the cypress-localstorage-commands package to preserve the state between tests. Use this package with caution. It is intended by design to always clear the state after a single test.
If you are working with a Nx monorepo, you can extract the custom command into a shared library and can reuse the function in different e2e-apps with ease.
Adapt the code to your needs.

Happy Coding & happy testing 😊

GitHub Repository

I added my code in this GitHub Repository. Please note, that I used a Nx workspace for it.
You can find the angular-msal-example-app here and the corresponding e2e-app / cypress tests here.

To run the e2e tests, use these commands:



npm install

npx nx e2e angular-msal-example-e2e --watch

Credits 🤝

I would like to thank my buddy Philip Riecks for proofreading my first ever blogpost. Make sure to checkout his content as well!

Here are some further links:

UI testing Azure AD protected single page applications with Cypress by Joonas Westlin
Azure AD Authentication in Cypress Tests by Tim Veletta (MSAL v1)