DEV Community: Julien Bras

How to migrate from Jest to Vitest without headaches!

Julien Bras — Sun, 17 Dec 2023 20:45:14 +0000

Introduction

Vitest is a testing library that is API-compatible with Jest. The main advantages over Jest are:

Vitest is compatible out-of-the-box with TypeScript! No need to deal with ts-jest or other complex setup.
Vitest is fast! We have seen a improvement of around 50% on your main module!

After a few months with both Vitest and Jest in our team, we have decided to give a push and migrate the remaining projects to Vitest only. This will help us to be more efficient on all the projects (it's not always easy to switch between Jest testing and Vitest testing depending on the project!).

The issue

For a majority of the projects, we have a small number of test files (between 1 and 10). So it's a quick win to migrate from one testing framework to another. In each test file a couple of modification is needed:

import the describe, it,expect keywords from vitest as it's not global by default
adjust the test if not passing

But for one project, this method seems too complex. The project is a NextJs application:

82 test files
257 tests in total

It's pretty obvious that a single Pull Request will be not a good idea, and it may take too much time to work on this task. In the team, we are not using the Trunk-Based development methodology, but we try to keep our modifications small:

not too much code modified each time
not too much time between starting creating a feature branch and the merge of the feature (about a day)

For us, it's a good way to keep the development flow and not struggle too much with merging issues. So switching 250 tests in a day doesn't seems to be a realistic objective, even for an experimented developer! Can we find a alternative path?

The baby step idea

As usual when an issue seems too complex, a good approach is to split the problem. Here the best solution we found is to split our migration in multiple smaller steps. But splitting test environment doesn't seems to be easy, right? Okay let's describe the methodology!

First step: init the Vitest setup

The first step is to install Vitest without removing Jest. So in the project let's start by running:

yarn add vitest --dev

Create a new vite.config.ts file with the following:

import { fileURLToPath, URL } from 'url';
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    setupFiles: './vitest-setup.ts',
    environment: 'jsdom',
    // TODO: remove include prop after complete Vitest migration
    include: ['tests-vitest/**/*.{test,spec}.?(c|m)[jt]s?(x)'],
    coverage: {
      reporter: ['lcov', 'text'],
    },
    outputFile: 'coverage/sonar-report.xml',
  },
  resolve: {
    alias: [{ find: '@/', replacement: fileURLToPath(new URL('./', import.meta.url))
      },
});

And a new vitest-setup.ts with:

import "@testing-library/jest-dom/vitest";

(to be able to use jest-dom !)

The idea is too keep all old Jest test in the test/ folder, and move slowly the tests in a new tests-vitest/ folder. We need to set the include prop in the configuration to run only the tests in the specific folder. We are also configuring some coverage with Sonar.

Here is the architecture of the project:

web/
    components/.       # React components
    pages/             # Still using the pages router of NextJs!
    tests/             # Jest tests
    test-vitest/       # Vitest tests

in package.json, we are adding a new command to run Vitest tests:

"scripts": {
    ...
    "test": "jest",
    "test:vitest": "vitest",
    ...
}

So we can run Jest tests with yarn test and Vitest tests with yarn test:vitest!

A final modification, we need to tweak the Jest configuration in jest.config.js to search tests only in the tests/ folder:

module.exports = {
    ...
    testMatch: ['<rootDir>/tests/**/?(*.)+(spec|test).[jt]s?(x)'],
    ...
}

And we can drop a few first tests in tests-vitest to be sure it's working! The first PR goal was about 4 tests files migrated, so it was a pretty small PR that we were able to validate in a day.

Finally don't forget to adjust your CI to run both test suite in order to still have a complete coverage.

Second step: increase the Vitest coverage

Then slowly move a couple of test files from tests/ to tests-vitest/ and adjust tests to make then pass. Again the same strategy:

add the imports for describe, it, expect, or vi
play a bit with the test
and voilà!

It took us around 11 Pull Requests to move all 82 test files and 257 tests. Only one test have been declared too complicated and we decided (for now) to disable it, we will try to re-enable it later on.

Third step: remove Jest!

Finally it's time to remove all Jest dependencies! This is the most fun part of the project because you remove a lot dependencies! 🤓 Do not forget to remove also all configuration files (jest.config.js, jest.setup.js...) and we decided to move back all Vitest tests in tests/ in this specific PR (adjust include in vitest.config.ts!).

Conclusion

The task seems at first sight impossible when you want to apply the same rules. But you are free to decide and create new way to work a specific problem! Here instead of dealing with a monster PR with maybe 100 or more modified files, it was much more easy to split the issue and have a temporary setup during the migration time. It took us around 2 weeks to complete the migration (around one PR per day during the migration process!)

🔒 Next Auth vs SST Auth

Julien Bras — Mon, 21 Nov 2022 14:39:14 +0000

In a previous article, I describe how to use SST Auth construct in order to implement an OAuth authentication workflow for your application. Next.js, the React framework for production is also providing a component named NextAuth.js that can be used to implement such authentication system. Let's compare the two solutions! 🤺

What is SST Auth?

SST is framework designed to build backend serverless applications initially. I have already written a couple of articles on this solution (here and here for example). It provides features to deploy web applications too (for example via the StaticSite construct) so it's advertised as a tool to build full-stack serverless applications.

The Auth module is a dedicated set of components built by the SST team to implement an authentication system inside your application. It works well with a web application like a React app.

What is NextAuth.js?

Next.js is probably the most famous React-based framework available today, and gain a lot of visibility in the last few years. Today with version 13, it's truly a full-stack framework solution with the support of server-side rendering options and an API layer. You can check for example the Theo Browne video: Next.js is a backend framework which is a good introduction to the backend part.

NextAuth.js is an independent library (not supported by Vercel), with the following motto: "Authentication for Next.js" It provides a built-in solution to implement an authentication system for Next.js, based on OAuth protocol.

You know the actors, it's now time to get to the comparison bullet points, let's fight 😇

Round 1: Supported Adapters/Providers!

Nowadays, authentification is not only email-password credentials. It's more social logins like Google, Facebook, or GitHub. It's more secure for the application developer (no more password to store!) and for the application end-user (no new password to remember!). Let's check what is supported by our two choices.

First, SST Auth is supporting out of the box today (November 2022) seven adapters: Google, GitHub, Twitch, Facebook, Magic Link, OAuth, and OIDC. The last two are generic adapters that can be used for any application which is supporting OAuth2 or OIDC. Finally, there is an option to build a Custom Adapter if nothing fit your needs. For example in this last article, I have built a custom adapter to support SmugMug, which is relying on the OAuth 1.0a protocol.

NextAuth.js is supporting out the box more than 20 providers: the classic ones like Google, Facebook, and GitHub are here, but there are more options compared to SST Auth. Additionally, there is also an email provider (can be compared to the Magic Link one on SST Auth) or a custom provider. Finally, the Credentials provider is an ideal solution if you need to login via username password, or other arbitrary credentials (YubiKey for example).

➡️ There is much out-of-the-box options using NextAuth.js compared to SST Auth. But both solutions provide you customization if needed.

Round 2: Easiness To Implement!

Authentication is a serious subject, and you don't want to rely on a system that you don't trust 100%. That's why it's important to understand how to implement the solution, and how easily you will be able to understand it and tweak it ultimately.

Let's get a look at SST Auth:

you first need to add the Auth construct in your infrastructure stack (generally the stacks/MyStack.ts file)

import { Auth } from "@serverless-stack/resources";

const auth = new Auth(stack, "auth", {
  authenticator: {
    handler: "functions/auth.handler",
  },
});

auth.attach(stack, {
  api: myApi,
  prefix: "/auth", // optional
});

then you can build a new serverless function in src/functions/auth.ts that will be responsible for all the authentication calls (authorize and callback endpoints for OAuth2 for example):

import { AuthHandler, GoogleAdapter } from "@serverless-stack/node/auth";

export const handler = AuthHandler({
  providers: {
    google: GoogleAdapter({
      mode: "oidc",
      clientID: "XXXX",
      onSuccess: async (tokenset) => {
        return {
          statusCode: 200,
          body: JSON.stringify(tokenset.claims()),
        };
      },
    }),
  },
});

This example case (from the Auth documentation) is creating an authentication system based on Google SignIn. It will expose 2 endpoints:

/auth/google/authorize
/auth/google/callback
Finally, it's up to your frontend application (React for example) to redirect the user to the autorize backend endpoint (this will start the OAuth2 flow).

Let's now zoom on NextAuth.js:

First, you need to install NextAuth.js in your Next.js project:

yarn add next-auth

Then you have to create a new API in your Next project, inside the designed pages/api/ folder. So create a file named pages/api/auth/[...nextauth].js with the following content:

import NextAuth from "next-auth"
import GoogleProvider from "next-auth/providers/google"
export const authOptions = {
  // Configure one or more authentication providers
  providers: [
    GoogleProvider({
      clientId: process.env.GOOGLE_ID,
      clientSecret: process.env.GOOGLE_SECRET,
    }),
  ],
}
export default NextAuth(authOptions)

This API proxy endpoint is the exact equivalent of the auth.ts created for SST. NextAuth.js will take care to create multiple endpoints for authorize and callback in order to support OAuth2.

Then it's advised to implement the SessionProvider at the top level of the application (in pages/_app.jsx):

import { SessionProvider } from "next-auth/react"
export default function App({
  Component,
  pageProps: { session, ...pageProps },
}) {
  return (
    <SessionProvider session={session}>
      <Component {...pageProps} />
    </SessionProvider>
  )
}

And finally use the dedicated methods provided by NextAuth.js to allow the end-user to sign in or sign out:

import { useSession, signIn, signOut } from "next-auth/react"
export default function Component() {
  const { data: session } = useSession()
  if (session) {
    return (
      <>
        Signed in as {session.user.email} <br />
        <button onClick={() => signOut()}>Sign out</button>
      </>
    )
  }
  return (
    <>
      Not signed in <br />
      <button onClick={() => signIn()}>Sign in</button>
    </>
  )
}

This article will not go deeper into the implementation, but each solution is providing a session system to retrieve the user data on the frontend and on the backend side. It's also possible to prevent access for not authenticated users.

➡️ To conclude on this aspect, both solutions are simple enough to be used by a frontend developer, and it does not require a big amount of code to be used. The documentation provided is also very complete. It's a draw!

Round 3: Store User-Related Information!

Once authentication is implemented, you may need to store some information relative to each user. For exemple, the profile section needs to be persisted, or any pertinent information for your precise business case. Again both solutions come with a solution for that.

For SST Auth, it's fairly natural to rely on a distinct SST construct, Table that is relying on AWS DynamoDB. By implementing the onSuccess method available in each adapter, it's possible to store the user data in DynamoDB:

export const handler = AuthHandler({
  providers: {
    smugmug: SmugMugAdapter({
      clientId: process.env.SMUGMUG_CLIENT_ID!,
      clientSecret: process.env.SMUGMUG_CLIENT_SECRET!,
      onSuccess: async (user: SmugMugUser) => {
        const ddb = new DynamoDBClient({});
        await ddb.send(
          new PutItemCommand({
            TableName: Table.users.tableName,
            Item: marshall(user),
          })
        );

        return Session.parameter({
          redirect: process.env.IS_LOCAL
            ? "http://127.0.0.1:3000"
            : ViteStaticSite.site.url,
          type: "user",
          properties: {
            userID: user.userId,
          },
        });
      },
    }),
  },
});

Example took from OAuth with Serverless using SST

With NextAuthjs, it's possible to implement an adapter (⚠️ NextAuthjs adapter is not the same as an SST Auth adapter!). There are more than 10 options, including DynamoDB, Firebase, Prisma, FaunaDB... Let's zoom in on DynamoDB adapter here:

Install the corresponding adapter:

yarn add next-auth @next-auth/dynamodb-adapter

Edit your existing API in pages/api/auth/[...nextauth].js:

import { DynamoDB } from "@aws-sdk/client-dynamodb"
import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb"
import NextAuth from "next-auth";
import Providers from "next-auth/providers";
import { DynamoDBAdapter } from "@next-auth/dynamodb-adapter"

const config: DynamoDBClientConfig = {
  credentials: {
    accessKeyId: process.env.NEXT_AUTH_AWS_ACCESS_KEY as string,
    secretAccessKey: process.env.NEXT_AUTH_AWS_SECRET_KEY as string,
  },
  region: process.env.NEXT_AUTH_AWS_REGION,
};

const client = DynamoDBDocument.from(new DynamoDB(config), {
  marshallOptions: {
    convertEmptyValues: true,
    removeUndefinedValues: true,
    convertClassInstanceToMap: true,
  },
})

export default NextAuth({
  providers: [
    Providers.Google({
      clientId: process.env.GOOGLE_ID,
      clientSecret: process.env.GOOGLE_SECRET,
    }),
  ],
  adapter: DynamoDBAdapter(
    client
  ),
});

You will have more information on the adapter here. Take note that you will need to create the DynamoDB table before using it (NextAuth.js does not handle the creation of the resource).

➡️ Both solutions implement options to store information relative to the user. SST relies on constructs and NextAuth relies on adapters. SST is a more integrated solution because it will handle the creation of the resource (infrastructure as code built-in). No clear winner here again!

Round 4: Deploy Options!

Now it's time to ship your application 🚀 ! How easy is the process for each solution? Let's find it!

SST Auth is part of SST obviously. This serverless framework is designed to be used with AWS. So you need to rely on this specific cloud provider, but it comes with a dedicated CLI command to push your application. As documented, this command will send your application to the ~~heavens~~ cloud:

yarn deploy --stage prod

It's recommended to use a specific stage for the production environment as SST is designed to use multiple instances: when you develop on the application you are using a distinct stage (like dev) with the powerful Live Lambda Development mode.

Regarding NextAuth.js, you will have to deploy a Next.js application. The short path here is to deploy your application on Vercel, the creators of Next.js. It comes with a pretty generous free tier, and it works out-of-the-box: automatic deployments with GitHub, preview deployments, etc... But a Next.js application can be also deployed in many other providers. For your own managed Node server to serverless deployment, it's up to you to choose the right one! See the complete list on Next.js documentation page.

Let's mention that it's possible to deploy a Next.js application using SST 😅.
This setup relies on the serverless-next.js project and it's hidden behind a construct named NextJsSite. It can be a very efficient solution when you want to ship your application on AWS environment (I use it for a production workload in my current company).

➡️ To conclude on this section, SST is by design more restricted on the deployment (AWS only). For Next.js, the classic way is to rely on Vercel but other options are available (including SST itself on AWS).

Conclusion

It's by design not fair to compare tools that are not in the same category! But I think it's useful sometimes to compare a specific feature (here authentication) as the implementation in both solutions is very comparable.

SST is a really good option when you need to build an application that will have to rely on multiple AWS services; it's possible to define using Infrastructure as Code and then consume a Queue, a Bucket, a Database to name a few available constructs. It's the most extensible solution here. Read my introduction to SST here: SST is The Most Underrated Serverless Framework You Need to Discover

NextAuth.js is just a library, and it's more quick to add the authentication layer in an existing Next.js application with this solution. It's the most integrated solution here.

I recommend playing with the two solutions here and giving me your feedback on Twitter (if it's still up and running when you read it 😅).

OAuth with Serverless using SST

Julien Bras — Sun, 13 Nov 2022 19:31:29 +0000

Why we Need Authentication in a Serverless Application?

Once you have built your first serverless application, you may need to quickly invest effort in a authentication layer. For example on Why and How Migrate From Firebase to Serverless Stack? we choose to use an authentication based on Google in order to save some data relative to each user of our application.

Why it's a Challenge?

It's a very classic use-case to implement an authentication system based on third-part actors, like Google or Facebook. Instead of managing locally a users and passwords system, it's generally more secure to allow a login via Google, and then Google servers is validating to our systems that the login is a success. There is multiple advantage here:

the end-user have just to use his own Google login to connect to your application, and there is no need to create and store a new password
your application doesn't have to manage and secure a password system

This classic scenario is using generally a standard named OAuth, built to allow access delegation without sharing private credentials. As an example here is an abstract flow of the OAuth2 protocol found on Wikipedia:

There is a series of steps to achieve (about 5 or 6) and it's pretty tricky to implement this flow in a serverless stack, because such functions are by definition stateless: how to store temporary the authorization grant for example? That's why we generally rely on an authentication framework!

What are the Options?

If you want to stick to serverless deployment, there is multiple options to implement an Authentication layer:

SST is providing multiple constructs (Cognito, Auth) for this precise request
Next.js, which can be considered as a full-stack platform thanks to the backend capabilities, comes with NextAuth.js which is a promising solution to implement authentication in the platform.
It's also possible to build a custom layer for the authentication, but we will not expend here on this option as it's the most time-consuming one 😅

[!NOTE] NextAuth vs SST Auth
I plan to post an article to compare the 2 options. Stay tuned!

Let's Focus on SST Auth

This article will be focused on SST Auth construct only: The Cognito construct (was named Auth before SST 1.0 ) is based on AWS Cognito User Pool and Cognito Identity Pool, and provide less flexibility to the application owner at the end of the day. Read here why the SST team decided to create the new Auth construct.

The idea behind SST Auth is to provide to the application developer a simple way to implement an authentication system based. It's shipped with adapters for common use cases, like Google, Facebook adapters but you can extend it for your own use case.

Here we will use it to develop a small application with authentication based on the SmugMug provider. It's specific because SmugMug API is supporting only OAuth 1.0a today, so we will need to build our own custom adapter.

Build Our Application (step-by-step)

[!WARNING]
This tutorial will use yarn and TypeScript as default language (as a classic convention) but feel free to explore the other option offered by SST.

You can follow the extensive documentation, but the first step is to bootstrap our application:

yarn create sst --template=examples/api-sst-auth-google smugmug-auth

This will use the examples/api-sst-auth-google starter to step up the coding time. The goal then will be to replace the Google authentication by SmugMug.

Navigate to your project:

cd smugmug-auth/

First let's add a package to handle OAuth 1.0a authentication. My best bet is today oauth which is not new but seems widely used. Unfortunately it doesn't support promises out-of-the-box.

cd services/
yarn add oauth
yarn add @types/oauth --dev
yarn add @serverless-stack/node

We can also populate the SmugMug API key and API key secret in a .env.local file (not pushed to git):

SMUGMUG_CLIENT_ID=your API key
SMUGMUG_CLIENT_SECRET=your API key secret

In order to generate such keys, you need to create a dedicated application in SmugMug Developer section.

Then let's start the SST application from the root folder. For this step it's required to be authenticated on an AWS account.

yarn start

SST is asking for a stage name. I use generally dev because I am the only one working on the application, and I deploy the production version on a dedicated prod stage later on. You can use the stage name you want.

Once the stack is up you can also start the react app:

cd web/
yarn dev

And you can try to login on Google:

It doesn't work because of the mis-configuration of the application for the deployed backend but basically it's just a configuration point. Let's set up what we need for SmugMug authentication. Here is the flow we want:

the user is accessing our application
the user click on login
the user is redirected to SmugMug in order to login and accept to share some informations (email, full name) with our application
then the user is redirected to our application and authenticated.

Backend

Let's start by tweaking the backend. The function services/functions/auth.js is the heart of the authentication system. It's actually using the GoogleAdapter inside the AuthHandler provided by SST. Unfortunately there is no SmugMugAdapter yet, neither a generic OAuth1.0aAdapter. So let's build it!

Create a new folder services/functions/smugmug and a first api.ts file inside:

import OAuth from "oauth";
const domain = "https://api.smugmug.com";
const baseUrl = `${domain}/api/v2`;

const requestUrl = `${domain}/services/oauth/1.0a/getRequestToken`;
export const authUrl = `${domain}/services/oauth/1.0a/authorize`;
const accessUrl = `${domain}/services/oauth/1.0a/getAccessToken`;
const signatureMethod = "HMAC-SHA1";

export interface OAuthToken {
  token: string;
  tokenSecret: string;
}
export const SmugMugOAuth = class {
  oauth: OAuth.OAuth;

  constructor(consumerKey: string, consumerSecret: string) {
    this.oauth = new OAuth.OAuth(
      requestUrl,
      accessUrl,
      consumerKey,
      consumerSecret,
      "1.0A",
      null,
      signatureMethod,
      undefined,
      // mandatory to get JSON result on GET and POST method (LIVE API Browser instead!)
      { Accept: "application/json" }
    );
  }

  getOAuthRequestToken = async (callback: string): Promise<OAuthToken> => {
    return new Promise((resolve, reject) => {
      this.oauth.getOAuthRequestToken(
        {
          oauth_callback: callback,
        },
        (error, oAuthToken, oAuthTokenSecret, results) => {
          if (error) {
            reject(error);
          }
          resolve({ token: oAuthToken, tokenSecret: oAuthTokenSecret });
        }
      );
    });
  };

  getOAuthAccessToken = async (
    requestToken: OAuthToken,
    oAuthVerifier: string
  ): Promise<OAuthToken> => {
    return new Promise((resolve, reject) => {
      this.oauth.getOAuthAccessToken(
        requestToken.token,
        requestToken.tokenSecret,
        oAuthVerifier,
        (error, oAuthAccessToken, oAuthAccessTokenSecret, results) => {
          if (error) {
            reject(error);
          }
          resolve({
            token: oAuthAccessToken,
            tokenSecret: oAuthAccessTokenSecret,
          });
        }
      );
    });
  };

  get = async (url: string, accessToken: OAuthToken) => {
    return new Promise((resolve, reject) => {
      this.oauth.get(
        `${domain}${url}`,
        accessToken.token,
        accessToken.tokenSecret,
        (error, responseData, result) => {
          if (error) {
            console.log(error);
            reject(error);
          }
          if (typeof responseData == "string") {
            resolve(JSON.parse(responseData).Response);
          } else {
            const err = Error("not a valid answer");
            console.log(err.message);
            reject(err);
          }
        }
      );
    });
  };
};

This file is creating a SmugMugOAuth class that is wrapping the oauth lib for us. It will then be easier to call for the various function provided by oauth, with promises support! It defines the following methods:

getOAuthRequestToken to get a Request Token (first part of the OAuth exchange)
getOAuthAccessToken to get an Access Token (second part of the OAuth exchange)
get to query the SmugMug API with a provided access token

Then in the same folder let's create an adapter.ts to build our custom SmugMugAdapter:

import {
  useCookie,
  useDomainName,
  usePath,
  useQueryParams,
} from "@serverless-stack/node/api";
import { createAdapter } from "@serverless-stack/node/auth";
import { APIGatewayProxyStructuredResultV2 } from "aws-lambda";
import { authUrl, OAuthToken, SmugMugOAuth } from "./api";

export interface SmugMugUser {
  userId: string;
  fullName: string;
  uri: string;
  webUri: string;
  accessToken: OAuthToken;
}

export interface SmugMugConfig {
  /**
   * The clientId provided by SmugMug
   */
  clientId: string;
  /**
   * The clientSecret provided by SmugMug
   */
  clientSecret: string;

  onSuccess: (user: SmugMugUser) => Promise<APIGatewayProxyStructuredResultV2>;
}

export const SmugMugAdapter = createAdapter((config: SmugMugConfig) => {
  return async function () {
    const oauth = new SmugMugOAuth(config.clientId, config.clientSecret);

    const [step] = usePath().slice(-1);

    if (step === "authorize") {
      // Step 1: Obtain a request token

      const callback =
        "https://" +
        [useDomainName(), ...usePath().slice(0, -1), "callback"].join("/");

      const requestToken = await oauth.getOAuthRequestToken(callback);

      // Step 2: Redirect the user to the authorization URL
      const expires = new Date(Date.now() + 1000 * 30).toUTCString();

      return {
        statusCode: 302,
        cookies: [
          `req-token=${JSON.stringify(
            requestToken
          )}; HttpOnly; expires=${expires}`,
        ],
        headers: {
          location: `${authUrl}?oauth_token=${requestToken.token}&Access=Full&Permissions=Modify`,
        },
      };
    }

    // Step 3: The user logs in to SmugMug (on SmugMug side)
    // The user is presented with a request to authorize your app

    // Step 4: If the user accepts, they will be redirected back to your app, with a verification code embedded in the request
    // Use the verification code to obtain an access token

    if (step === "callback") {
      const params = useQueryParams();
      const reqToken: OAuthToken = JSON.parse(useCookie("req-token"));
      const accessToken = await oauth.getOAuthAccessToken(
        reqToken,
        params.oauth_verifier!
      );
      const response: any = await oauth.get(`/api/v2!authuser`, accessToken);
      const user: any = response.User;

      return config.onSuccess({
        userId: user.NickName,
        fullName: user.Name,
        uri: user.Uri,
        webUri: user.WebUri,
        accessToken,
      });
    }

    throw new Error("Invalid auth request");
  };
});

The class SmugMugAdapter is a custom adapter created by the createAdapter method described in the Auth documentation.

It describe all the steps for the OAuth 1.0a process:

when the endpoint authorize is called, it call the method getOAuthRequestToken
then we send the user to the authorization URL on SmugMug with the request token
SmugMug will redirect the user to the application with a verification code
With the verification code we can request an access token!
Finally we are calling the /api/v2!authuser to retrieve the autenticated user information.

Then we need to adapt the services/functions/auth.ts file to use our custom adapter:

import {
  Session,
  AuthHandler,
} from "@serverless-stack/node/auth";
import { Table } from "@serverless-stack/node/table";
import { ViteStaticSite } from "@serverless-stack/node/site";
import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall } from "@aws-sdk/util-dynamodb";
import { SmugMugAdapter, SmugMugUser } from "./smugmug/adapter";

declare module "@serverless-stack/node/auth" {
  export interface SessionTypes {
    user: {
      userID: string;
    };
  }
}

export const handler = AuthHandler({
  providers: {
    smugmug: SmugMugAdapter({
      clientId: process.env.SMUGMUG_CLIENT_ID!,
      clientSecret: process.env.SMUGMUG_CLIENT_SECRET!,
      onSuccess: async (user: SmugMugUser) => {
        const ddb = new DynamoDBClient({});
        await ddb.send(
          new PutItemCommand({
            TableName: Table.users.tableName,
            Item: marshall(user),
          })
        );

        return Session.parameter({
          redirect: process.env.IS_LOCAL
            ? "http://127.0.0.1:3000"
            : ViteStaticSite.site.url,
          type: "user",
          properties: {
            userID: user.userId,
          },
        });
      },
    }),
  },
});

Basically we just remove the GoogleAdapter and put instead our SmugMugAdapter! We can also see that we are storing our user in a DynamoDB table (which was already created in the original template). It can be a nice place to store data related to our users.

Finally pass the environment variables to the auth Lambda in stacks/MyStack.ts:

  // Create Auth provider
  const auth = new Auth(stack, "auth", {
    authenticator: {
      handler: "functions/auth.handler",
      bind: [site],
      environment: {
        SMUGMUG_CLIENT_ID: process.env.SMUGMUG_CLIENT_ID!,
        SMUGMUG_CLIENT_SECRET: process.env.SMUGMUG_CLIENT_SECRET!,
      },
    },
  });

Frontend

In the frontend we just have to modify the url to redirect the user. Instead of auth/google/authorize/, it will be auth/smugmug/authorize/:

In web/src/App.jsx, line 70:

<div>
      <a
        href={`${import.meta.env.VITE_APP_API_URL}/auth/smugmug/authorize`}
        rel="noreferrer"
>   
        <button>Sign in with SmugMug</button>
      </a>
        </div>

And that's it ; you made it!

The session page is a bit buggy here because we didn't adapt the Google data to the SmugMug data. But you can check that we can retrieve user information in the developer tools:

Here is the GitHub repository of the final project:

julbrs / sst-smugmug-auth

How to use SmugMug authentication in a SST project

How to add SmugMug authentication to a serverless API

An example serverless app created with SST, with a custom adapter (SmugMug, OAuth10a support)

Getting Started

Read the tutorial

Commands

`npm run start`

Starts the Live Lambda Development environment.

`npm run build`

Build your app and synthesize your stacks.

`npm run deploy [stack]`

Deploy all your stacks to AWS. Or optionally deploy, a specific stack.

`npm run remove [stack]`

Remove all your stacks and all of their resources from AWS. Or optionally removes, a specific stack.

`npm run test`

Runs your tests using Jest. Takes all the Jest CLI options.

Documentation

Learn more about the SST.

View on GitHub

What we Learned Here?

Starting from a bootstrapped project from SST, we have easily adapted it to support the third-party authentication provider we need with the help of the easily extendable construct provided.

You can continue the journey by reading the extensive documentation about Auth. It's also possible to rely on the OAuth adapter that is supporting out of the box any OAuth2 compatible service.

Happy authentication!

Why and How Migrate From Firebase to Serverless Stack?

Julien Bras — Fri, 25 Mar 2022 03:12:04 +0000

This article is the third of a series around SST - Serverless Stack. I will try to let you discover some amazing aspects of this particular solution in the serverless world. You can find the first article here (introduction) and the second one here (some constructs presentation).

Firebase is a fantastic tool. It allows you to build mobile or web applications without having to manage a backend by yourself. But somehow, this comes with some drawbacks. In this article I will explain you why you may want to switch, and a practical guide to switch.

In a concrete example I will migrate a React application that is relying on both Firebase and a Serverless Framework backend to a single stack (with Serverless Stack)

Short Presentation of Each Solutions

Firebase is a product backed by Google. It allow you to create mobile and web applications based on a set of Firebase components. It contains an authentication layer, a database (FireStore), a storage component to save files, and a hosting solution to ship your application. It’s also possible to rely on Cloud Function to run code in backend functions.
Serverless Framework is a solution to host your backend components in a dedicated cloud provider without having to manage servers. For exemple on AWS it will allow your to manage Lambda functions easily.
Serverless Stack is a new solution that can do what Serverless Framework offer. But it offer also to handle the hosting of your web application, and provide a better developer experience in my opinion. I have already written a couple of article on the subject: here for an introduction and here for some constructs presentation.
React is a Javascript library to build user interface 😇

Why You May Want to Migrate?

I was running my system to manage Montreal library cards since a few year based on Firebase. Because I was using the free version of Firebase, I wasn’t able to use Cloud Functions. But to query Montreal library system, it was needed to run some functions somewhere. Back in the days, I have selected Serverless Framework to operate this backend API on my own AWS account. But it was not ideal, because I was dealing with too much stacks. Focusing on Firebase, here is a list of items that can limit you:

Firebase is offering a limited set of functionalities: the integrated solutions is providing a really nice set of features for common web application (authentication, storage, database...). But it’s not easily extensible. When you use directly AWS, you can use any service provided by the cloud provider. Think about Machine Learning service, Queue systems, Container workload...
Pricing model is not cheap: when you leave the no-cost plan (Spark), Firebase can be quite expensive, depending on your usage. For reference this classic article 30k bill on Firebase is a good reference! The backend-as-a-service model can lead to such issues if not well optimized. AWS is not cheap either, but you will pay only what you are using and you have more options to build your product (does the frontend is running query on the database directly or via a backend API?)
Developer experience can be limited: local development is a must for serverless application : it reduces the feedback time it take you to test each feature. Firebase offer you a local emulator suite to provide you a local environment. It will allow you to test quickly the cloud function built, without waiting them to be shipped. But it’s only an emulation, not real cloud function running on your cloud provider. On the opposite, Serverless Stack is providing you a live lambda development environment that is relying on AWS services, not emulation.

Running the Migration in 6 Steps!

Step 1: Init your Serverless Stack application

Following the quick-start:

# Create a new SST app
npx create-serverless-stack@latest my-sst-app
cd my-sst-app

Take some time to explore the organisation of the folder. stacks/ contains your infrastructure setup, src/ will contains your Lambda function code.

Step 2: Migrate from Serverless Framework to the new application

In my specific case, I was migrating functions from Serverless Framework. The guys from SST have a decent documentation for this classic case: Migrating From Serverless Framework.

Basically I have reused directly the javascript files from the old project, and place them in the src/ folder of the new project. Then inside stacks/MyStack.ts, I have created my API routes:

// Create a HTTP API
const api = new sst.Api(this, "Api", {
  defaultAuthorizationType: sst.ApiAuthorizationType.AWS_IAM,
  cors: true,
  routes: {
    "GET /cards": "src/cards.list",
    "POST /cards": "src/cards.add",
    "DELETE /cards/{id}": "src/cards.remove",
    "GET /cards/{id}/books": "src/books.list",
        ...
  },
});

The defaultAuthorizationType allow me to secure the API with an IAM authentication (see next step!).

Step 3: Replace the Firebase Authentication

Firebase is handy because it comes with an authentication layer built-in. Inside SST the best option is to use the Auth construct, that is relying behind the scene on AWS Cognito.

In stacks/MyStack.ts, I am adding:

// Create auth
const auth = new Auth(this, "Auth", {
  cognito: {
    userPoolClient: {
      supportedIdentityProviders: [UserPoolClientIdentityProvider.GOOGLE],
      oAuth: {
        callbackUrls: [
          scope.stage === "prod"
            ? `https://${prodDomainName}`
            : "http://localhost:3000",
        ],
        logoutUrls: [
          scope.stage === "prod"
            ? `https://${prodDomainName}`
            : "http://localhost:3000",
        ],
      },
    },
  },
});

if (
  !auth.cognitoUserPool ||
  !auth.cognitoUserPoolClient ||
  !process.env.GOOGLE_AUTH_CLIENT_ID ||
  !process.env.GOOGLE_AUTH_CLIENT_SECRET
) {
  throw new Error(
    "Please set GOOGLE_AUTH_CLIENT_ID and GOOGLE_AUTH_CLIENT_SECRET"
  );
}

const provider = new UserPoolIdentityProviderGoogle(this, "Google", {
  clientId: process.env.GOOGLE_AUTH_CLIENT_ID,
  clientSecret: process.env.GOOGLE_AUTH_CLIENT_SECRET,
  userPool: auth.cognitoUserPool,
  scopes: ["profile", "email", "openid"],
  attributeMapping: {
    email: ProviderAttribute.GOOGLE_EMAIL,
    givenName: ProviderAttribute.GOOGLE_GIVEN_NAME,
    familyName: ProviderAttribute.GOOGLE_FAMILY_NAME,
    phoneNumber: ProviderAttribute.GOOGLE_PHONE_NUMBERS,
  },
});

// make sure to create provider before client (https://github.com/aws/aws-cdk/issues/15692#issuecomment-884495678)
auth.cognitoUserPoolClient.node.addDependency(provider);

const domain = auth.cognitoUserPool.addDomain("AuthDomain", {
  cognitoDomain: {
    domainPrefix: `${scope.stage}-nelligan-plus`,
  },
});

// Allow authenticated users invoke API
auth.attachPermissionsForAuthUsers([api]);

This will allow me the use Google as my principal authentification system (inside Cognito User Pool). There is an alternate way to use Cognito Identity Pool with a simpler declaration:

new Auth(this, "Auth", {
  google: {
    clientId:
      "xxx.apps.googleusercontent.com",
  },
});

But it’s harder to manage in the React app so I prefer my initial version 😇.

Step 4: Replace the Firestore Database

The Firebase project rely on Firestore to store some data related to each user. On the new stack you must build a new system to store data. The equivalent structure in AWS world is a DynamoDB table, with a cost per usage. It fits well serverless deployments. There is useful Table construct available in SST:

// Table to store cards
  const table = new Table(this, "Cards", {
    fields: {
      cardId: TableFieldType.STRING,
      cardUser: TableFieldType.STRING,
      cardCode: TableFieldType.STRING,
      cardPin: TableFieldType.STRING,
    },
    primaryIndex: { partitionKey: "cardId" },
  });

Step 5: Replace the Firebase Hosting

Here there is multiple approach possible. I am suggesting the most integrated solution for an SST stack:

use the new ReactStaticSite construct
take advantage of the static-site-env to handle automatically the environment variables

First add in MyStack.ts:

// Create frontend app
const reactApp = new ReactStaticSite(this, "ReactSite", {
  path: "react-app",
  buildCommand: "yarn && yarn build",
  environment: {
    REACT_APP_REGION: this.region,
    REACT_APP_API_URL: api.url,

    REACT_APP_GA_TRACKING_ID: "UA-151729273-1",
    REACT_APP_USER_POOL_ID: auth.cognitoUserPool.userPoolId,
    REACT_APP_USER_POOL_CLIENT_ID:
      auth.cognitoUserPoolClient.userPoolClientId,
    REACT_APP_IDENTITY_POOL_ID: auth.cognitoIdentityPoolId,
    REACT_APP_USER_UI_DOMAIN: domain.domainName,
    REACT_APP_DOMAIN:
      scope.stage === "prod"
        ? `https://${prodDomainName}`
        : "http://localhost:3000",
  },
  customDomain:
    scope.stage === "prod"
      ? {
          domainName: prodDomainName,
          hostedZone: "sidoine.org",
        }
      : undefined,
});

The environment props allow to pass environment variables to the React stack. The path is the relative path that contains your React app.

Step 6: Adapt your React Application

So following step 5, in the react-app/ folder I move my existing React application and start changing it to support my new stack content. Here is a general guidance follow:

Remove any occurence of firebase library
Add aws-amplify instead (it’s a simple wrapper for using AWS ressources like auth, api, etc...)
Add @serverless-stack/static-site-env to manage environment variable from SST
Configure aws-amplify (see example here, based on environment variables)
Replace firebase calls by aws-amplify calls (that’s probably the most long task!)

For reference here is two examples of aws-amplify usage:

The SignIn component to sign in the application (rely on CognitoHostedUIIdentityProvider)
The Card component that is calling an API endpoint, using the API object from aws-amplify

Link to the Project Before and After the Migration

For reference, you can dig into the project before and after the migration:

Before the migration:

GitHub - julbrs/nelligan-plus at sls_firebase

After the migration:

GitHub - julbrs/nelligan-plus at 7fcff53b8be57a2505ccbbe1556576c46c02df98

Conclusion

The switch have been a game-changer for me. And it’s not because of the cost or features, but more for the developer experience. Before the migration, I use to first build the backend function, test it, ship it. Then use this backend function in the frontend application after shipping the backend part. Then maybe I need to go back to the backend to adapt the contract or modify the code... You get it, it was a slow back-and-forth process, not very efficient.

Today I have a single stack:

First I start SST via npx sst start
Then I start my React app locally (yarn start)

The advantages:

I am working on a development environment without link to the production system (thanks to the stages)
I can change my backend code directly in the IDE, and it’s available instantly! Thanks to Live Lambda Development!
I don’t have to manage directly the environment variables of my frontend stack (no more .env file to update!)
When it’s time to ship my project, just a single command to push both backend and frontend! npx sst deploy --stage prod

👉 Get a look at Serverless Stack (SST), or my example project (here on Github). It’s really worth the time to understand the main concepts, and then you will be more efficient building full stack serverless applications!
Continue the discussion on twitter 😎

SST: The Most Underrated Serverless Framework You Need to Discover (part 2)

Julien Bras — Sat, 13 Nov 2021 19:35:46 +0000

This article is the second of a series around SST. I will try to let you discover some amazing aspects of this particular solution in the serverless world. You can find the first article here.

So you start building using serverless principles, and you discover the Serverless framework. Great ! You will discover here another option, that I consider superior in multiple area, the Serverless Stack (SST). In this second article, I will focus on some available constructs after introducing the concept. It can help you to build faster!

Introducing some concepts

What is CDK?

The Serverless Stack (SST) is based on the AWS Cloud Development Kit (CDK). This solution has been introduced by AWS a few years ago, it allow to build infrastructure as code (IaC) by using a real programming language. If you already know Terraform, it is an equivalent product.

Terraform allow you to declare cloud resources via the HashiCorp Configuration Language (HCL):

resource "aws_instance" "iac_in_action" {
  ami               = var.ami_id
  instance_type     = var.instance_type
  availability_zone = var.availability_zone

  // dynamically retrieve SSH Key Name
  key_name = aws_key_pair.iac_in_action.key_name

  // dynamically set Security Group ID (firewall)
  vpc_security_group_ids = [aws_security_group.iac_in_action.id]

  tags = {
    Name = "Terraform-managed EC2 Instance for IaC in Action"
  }
}

On the opposite CDK allow you to declare cloud resources via TypeScript, Python, Java or .Net (maybe more now?)!

import * as ec2 from '@aws-cdk/aws-ec2';
import * as iam from '@aws-cdk/aws-iam';
import * as cdk from '@aws-cdk/core';

export class CdkStarterStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // ... rest

    // 👇 create the EC2 Instance
    const ec2Instance = new ec2.Instance(this, 'ec2-instance', {
      vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
      role: webserverRole,
      securityGroup: webserverSG,
      instanceType: ec2.InstanceType.of(
        ec2.InstanceClass.T2,
        ec2.InstanceSize.MICRO,
      ),
      machineImage: new ec2.AmazonLinuxImage({
        generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
      }),
      keyName: 'ec2-key-pair',
    });
  }
}

(this is a TypeScript example from here).

A few more differences exists between the solutions:

Terraform can deploy on any supported cloud provider but CDK is designed to work only on AWS (some tooling exist to break this limit, but it's not the goal of this article!)
CDK is relying on CloudFormation template and stacks, so the state of your infrastructure is directly stored in CloudFormation stacks (not in a local json file or a S3 bucket, classic solutions for Terraform)

This will be enough to introduce you the next important concept: constructs!

What are constructs?

From AWS Documentation:

Constructs are the basic building blocks of AWS CDK apps. A construct represents a "cloud component" and encapsulates everything AWS CloudFormation needs to create the component.

You can think about a set of basic cloud resources are managed behind a construct, and it help you to handle the complexity more easily with a few low number of lines.

For example here we can see how it's possible to use the construct ApplicationLoadBalancedFargateService in a few lines of codes:

const loadBalancedFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(stack, 'Service', {
  cluster,
  memoryLimitMiB: 1024,
  cpu: 512,
  taskImageOptions: {
    image: ecs.ContainerImage.fromRegistry("amazon/amazon-ecs-sample"),
  },
});

This single object is responsible to create the following:

a ECS task definition
a ECS service based on Fargate
an Application Load Balancer

And it can also generate for you a ECS Cluster if you provide a VPC. So in just 8 lines you create not less than 4 or 5 AWS resources!

Constructs Provided By SST

As said previously SST is based on CDK so it use intrinsically all CDK concepts, including Constructs.

One example provided in the first article was using the Api object, it's probably the most basic SST Construct available! Let's get a view on the most important constructs available in the framework and understand how it can help you.

(Go here to get the full list!)

Api

const api = new Api(this, "Api", {
  routes: {
    "GET  /notes": "src/list.main",
    "POST /notes": "src/create.main",
  },
})

The most simple one, to declare API HTTP endpoints very quickly. It mimics what can be done in the classic Serverless Framework. See my first article for a more in-depth comparison.

Auth

import { Auth } from "@serverless-stack/resources";

const auth = new Auth(this, "Auth", {
  cognito: true,
});

Auth will allow you to manage user authentication on your application, API... It's based on AWS services: Cognito User Pool and Cognito Identity Pool.

You can then manage the permissions:

auth.attachPermissionsForAuthUsers([
  api,
]);

Authenticated user of the auth object will be allowed to access the API endpoints declared by the api object (you need to secure your API for that, by default API is not secured and everyone can use it).

Table

The Table construct will handle for you a DynamoDB table:

import { Table, TableFieldType } from "@serverless-stack/resources";

const table = new Table(this, "Notes", {
  fields: {
    userId: TableFieldType.STRING,
    noteId: TableFieldType.STRING,
  },
  primaryIndex: { partitionKey: "noteId", sortKey: "userId" },
});

Then if you want to allow the Lambda functions inside your api object to be able to read / write in the table you have to specify that with the permissions prop:

new Api(this, "Api", {
  defaultFunctionProps: {
    permissions: [table],
  },
  routes: {
    "GET  /notes": "src/list.main",
    "POST /notes": "src/create.main",
  },
});

ReactStaticSite

ReactStaticSite is a very cleaver constructs that will allow you to include your frontend application inside the SST application!

Let's imagine you have a frontend/ folder inside the SST application with a classic ReactJs application. Add the following to your SST stack:

const site = new ReactStaticSite(this, "ReactSite", {
  path: "frontend/",
});

This 3-lines of code will do the following:

create for you a S3 bucket to host the frontend files
create for you a CloudFront deployment to manage a CDN for your frontend
build your ReactJs application when you are deploying the application (sst deploy)
push to S3 the build version of your ReactJs application to the S3 bucket dedicated
invalidate the CloudFront deployment (to get ride of existing cached data on the CDN)

And you can also imagine the power behind the StaticSite or the NextjsSite constructs 😅

Handle Environment Variables with ReactStaticSite

Generally you will use some REACT_APP_* variables in your ReactJs application in order to share the URL endpoint of your API, the Cognito User Pool Id, etc... A classic way to do that is to create a .env file at the root level of your ReactJs application (here for Create React App documentation)

SST is providing a very efficient way to handle that without messing with a .env file anymore! First define the variable in the ReactStaticSite construct like that:

new ReactStaticSite(this, "ReactSite", {
  path: "path/to/src",
  environment: {
    REACT_APP_API_URL: api.url,
    REACT_APP_USER_POOL_CLIENT: auth.cognitoUserPoolClient.userPoolClientId,
  },
});

Then in combination with the Live Lambda Development environment (it will be my next article!) you will be able to start your local React application by using the SST live information!

Add the following package to your ReactJs application:

npm install --save-dev @serverless-stack/static-site-env

And change the start script in package.json:

"scripts": {
  "start": "sst-env -- react-scripts start",
  "build": "react-scripts build",
  "test": "react-scripts test",
  "eject": "react-scripts eject"
},

More on this feature in the dedicated documentation.

Conclusion

This is the end of this second article about SST framework. I hope you understand better the power behind each construct provided by the framework. The team behind is very active and ship new features very often. I remember a few months ago, saying to myself: "if only some kind of static site could be embed into SST it will be just magic". Then I see v0.33.0 😱.

Just give a look at the follow resources:

documentation which is top notch
guide that contains a lot of examples, and some comparison with Serverless Framework

In the next episode of this series, I will speak about the local development feature of this framework, you can follow me to be informed when it will be out!

SST: The Most Underrated Serverless Framework You Need to Discover

Julien Bras — Sun, 26 Sep 2021 22:11:25 +0000

This article will be the first of a series around SST. I will try to let you discover some amazing aspect of this particular solution in the serverless world.

So you start building using serverless principles, and you discover the Serverless framework. Great ! You will discover here another option, that I consider superior in multiple area, the Serverless Stack (SST). I will first introduce some basic concepts for beginners, then I will expose the main difference between the solutions. Let's dig into this!

Introducing some concepts

Serverless

Serverless is "is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers" (Wikipedia). It allow you to run backend functions or applications without managing any backend server, or containers. Each major cloud provider is providing today serverless solutions:

Lambda on AWS
Microsoft Azure Functions on Azure
Cloud Functions on Google Cloud Platform

Basically you just define a function in a supported language, you push it to your cloud provider and voilà!, it's accessible and you don't have to deal with servers. It's the developer dream!

A very simple serverless function on AWS!

Framework for Serverless

Once you have build your first functions using serverless, you discover that the feedback loop to test something is generally longer that your local code:

First you develop the function
You need to deploy it on your cloud provider (manually maybe the first time)
You need to wait for the deployment, attach HTTP trigger if it's a REST API for example (API Gateway for AWS)
You test it (via Postman or using your web-application maybe)
You detected a bug
You get the log trace in the log system of the cloud provider (CloudWatch for AWS for example)
Then your go back to the start of the list

The back-and-forth is generally more costly because the code cannot be run directly on your local machine. So we try to rely on a solution that will speed up the development process, and eventually allow us to develop locally before pushing to the cloud provider (again to speed-up the process).

What is the Serverless Framework?

This particular framework was introduced back in 2015 (under the name JAWS, source: Wikipedia). The project have an astonishing 40k stars on Github! This framework offer a high number of features:

it support a large number of languages (more or less the same as supported language for serverless platform)
it is compatible with the major cloud provider here (AWS, Azure, GCP...)
it allow you to develop locally before pushing to the cloud provider
there is a million plugins that can help you to achieve what you need

The main piece in the framework is the serverless.yml file that is the configuration file of your serverless application. You can list here all the functions you want to deploy, how they are triggered (http endpoint or cron scheduling for example).

Here is an example file (from serverless/examples repo):

service: aws-node-rest-api

frameworkVersion: '2'


provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: '20201221'

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: /
          method: get

It describes in a very efficient way the provider and runtime you want to use, then the list of function you want to deploy. Each function is referenced with a handler. For nodejs runtime, it's just pointing to the file named handler.js and the exported function named hello.

Then you add events for each function if you want to trigger the function on HTTP event for example.

Under the hood the deployment is managed by CloudFormation on AWS (I do not have any experience on deployment on different cloud provider). It's possible to add custom resources in the serverless.yml file using CloudFormation template language.

My Story With This Framework

I have discover this framework back in 2018/2019, in my early day playing with Lambda on AWS. I think it is still the first thing you are testing after playing with serverless manually.

I use it to build some simple scheduling jobs on AWS, and build some full featured REST API.

What is the Serverless Stack (SST)?

This framework is much more recent. There is a couple of post here on DEV from Franck Wang, the author of this framework starting on September 2020 (here and here). The project have only 3k stars on Github.

The framework is much more limited in term of features, as it's new, but it cover the most classic use-cases:

only AWS is supported as cloud provider
only a few languages are supported for Lambda functions (no Java for example)
a local setup is available for development

The framework is relying on CDK which is an AWS tool designed to "define your cloud application resources using familiar programming languages". Behind the scene CDK is allowing you to write code that will be translated to CloudFormation template files. It's generally much more compact to write CDK that the equivalent CloudFormation template file!

A simple example of the framework:

new Api(this, "Api", {
  routes: {
    "GET  /notes": "src/list.main",
    "POST /notes": "src/create.main",
  },
})

This piece of code is using Api which is a Construct. A Construct is a CDK concept that represent a set of pre-configured resources on AWS. It's a very efficient way to define quickly an application. Same as the previous framework here I define HTTP routes. The first one GET /notes will use a Lambda function with the code available in the file src/list.js under the exported function main.

My Story With This Framework

I have discover the framework begin of 2021 and it's blowing my mind each time I am using it. I think that CDK is far superior to CloudFormation template files:

You can split your code in multiple file easily!
Your IDE is generally helping you writing CDK (TypeScript assisted for example)
You are writing less (Construct are helping to handle high order components !)

I use it actually to build a complete application (front + back) with authentication, relying a lot on Construct provided by SST team (Auth, Cron, Bucket).

Conclusion

This is the end of my first article on this framework. I hope it will help you give a try when you will have to choose a serverless Framework! In the next episode of this series, I will speak about the local development feature of this framework, you can follow me to be informed when it will be out!

How to Run a Minecraft Server on AWS For Less Than 3 US$ a Month

Julien Bras — Sat, 07 Aug 2021 21:58:49 +0000

During the first weeks of the COVID-19 pandemic, back in april 2020 my son ask me to build a Minecraft server in order to play on the same world with his school friend. After checking some available services (yeah not so expensive finally), I have chosen to build a server on a EC2 instance. This article will explain you how to optimize the cost 😜, based on the usage!

Some Tools Used in the Article

AWS

I want to rely only on AWS services as I want to increase my knowledge on this big cloud offering. There is always one service you don't know ! In this particular example I will use the following services:

EC2 (virtual servers in the cloud)
Lambda (serverless functions)
Simple Email Service (Email Sending and Receiving Service)

Minecraft

Minecraft is a popular sandbox video-game. In this case I will focus on the Minecraft Java Edition, because the server version is running well on Linux server, and my son is running a laptop on Debian.

Global Architecture of the Solution

The first month operating the server, I noticed that my son is using it a couple of hours each day, and then the server was idle. It's built on a EC2 t2.small with a 8 GB disk so I have a monthly cost of about 18 US$. Not a lot but I was thinking that there is room for improvement! The main part of the cost is the EC2 compute cost (~17 US$) and I know that it's not used 100% of the time. The global idea is to start the server only when my son is using it, but he doesn't have access to my AWS Console so I need to find a sweet solution!

Here is the various blocks used:

an EC2 instance, the Minecraft server
use SES (Simple Email Service) to receive e-mail, and trigger a Lambda function
one Lambda function to start the server
one Lambda function to stop the server

And that's it. My son is using it this way:

send an e-mail to a specific and secret e-mail address, this will start the instance
after 8h the instance is shutdown by the lambda function (I estimate that my son must not play on Minecraft more than 8h straight 😅)

Let's Build it Together

Build the EC2 Instance

This is the initial part, you must create a new EC2 instance. From the EC2 dashboard, click on Launch Instance and choose the Amazon Linux 2 AMI with the x86 option.

Next you must choose the Instance Type. I recommend you the t2.small for Minecraft. You will able to change it after the creation.

Click on Next: Configure Instance Details to continue the configuration. Keep the default settings, and the default size for the disk (8 GB) as it's enough.

For the tag screen I generally provide a Name (it's then displayed on EC2 instance list) and a costcenter (I use it for cost management later).

For the Security Group, it the equivalent of a firewall on EC2 and you must configure which port will be accessible from internet on your server. I add SSH port and the Minecraft port (25565) like you see on the following screen:

Then to start the instance you must select or create a key pair. It's mandatory and allow then to connect remotely to your EC2 instance. In my case I am using an existing key pair but if you create a new key don't forget to download on your laptop the private key file.

Yes my key is named caroline. Why not?

Then you must connect your instance via SSH, I recommend this guide if you need help. Basically you must run this kind of command:



ssh -i my_private_key.pem ec2-user@public-ipv4

The public-ipv4 is available in the instance list:

You first need java. As newer build of minecraft (since 1.17) are running only on Java 17, I recommend to use Corretto (the Amazon Java version):



sudo rpm --import https://yum.corretto.aws/corretto.key
sudo curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo
sudo yum install -y java-17-amazon-corretto-devel.x86_64
java --version

You must have something like:



openjdk 17.0.1 2021-10-19 LTS
OpenJDK Runtime Environment Corretto-17.0.1.12.1 (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.1.12.1 (build 17.0.1+12-LTS, mixed mode, sharing)

Thanks @mudhen459 for the research on this java issue ;)

And I want a dedicated user:



sudo adduser minecraft

To install Minecraft you can rely on the Minecraft server page here.

For example for the version 1.17.1 I can run the following:



sudo su
mkdir /opt/minecraft/
mkdir /opt/minecraft/server/
cd /opt/minecraft/server
wget https://launcher.mojang.com/v1/objects/a16d67e5807f57fc4e550299cf20226194497dc2/server.jar
sudo chown -R minecraft:minecraft /opt/minecraft/

⚠️ Warning regarding Java version:
It seems that starting with Minecraft 1.17, it require now a Java JRE 16 (instead of Java JRE 8).
This site is giving you links to download older Minecraft versions if needed.



Exception in thread "main" java.lang.UnsupportedClassVersionError: net/minecraft/server/Main has been compiled by a more recent version of the Java Runtime (class file version 60.0), this version of the Java Runtime only recognizes class file versions up to 52.0

I have created a little service to avoid start manually the server. I want the Minecraft process to start as soon as I start the server.

To do that I have created a file under /etc/systemd/system/minecraft.service with the following content:



[Unit]
Description=Minecraft Server
After=network.target

[Service]
User=minecraft
Nice=5
KillMode=none
SuccessExitStatus=0 1
InaccessibleDirectories=/root /sys /srv /media -/lost+found
NoNewPrivileges=true
WorkingDirectory=/opt/minecraft/server
ReadWriteDirectories=/opt/minecraft/server
ExecStart=/usr/bin/java -Xmx1024M -Xms1024M -jar server.jar nogui
ExecStop=/opt/minecraft/tools/mcrcon/mcrcon -H 127.0.0.1 -P 25575 -p strong-password stop

[Install]
WantedBy=multi-user.target

Then advise the new service by the following:



chmod 664 /etc/systemd/system/minecraft.service
systemctl daemon-reload

More information on systemd here.

Now if you restart the EC2 instance a Minecraft server must be available! You can check ✅ this first step!

I am not speaking of the fact that the IPv4 is dynamic by default. I recommend to setup an static Elastic IP for this server (here!) in order to get a static IP.

Build the Start Scenario

Let's first create our Lambda function. Go into Lambda, and click on Create function to build a new one. Name it mc_start and use a Node.js 14.x or more runtime.

Then you must have this type of screen:

Replace the content of index.js file with the following:



const AWS = require("aws-sdk");
var ec2 = new AWS.EC2();

exports.handler = async (event) => {
  try {
    var result;
    var params = {
      InstanceIds: [process.env.INSTANCE_ID],
    };
    var data = await ec2.startInstances(params).promise();
    result = "instance started"

    const response = {
      statusCode: 200,
      body: result,
    };
    return response;
  } catch (error) {
    console.error(error);
    const response = {
      statusCode: 500,
      body: "error during script",
    };
    return response;
  }
};

In Configuration, set the following:

add an environnement variable named INSTANCE_ID with the value that correspond to the Instance Id of your Minecraft server (something like i-031fdf9c3bafd7a34).
the role permissions must include the right to start our EC2 instance like this:

In Simple Email Service, it's time to create a new Rule Set in the Email Receiving section:

Click on Create rule inside default-rule-set. Take note that the Email Receiving feature is only available today in 3 regions: us-east-1, us-west-2 and eu-west-1 (source here).

If SES is receiving an email on this particular identity:

It invoke a Lambda function:

You must add the domain to the Verified identities to make this work. It's also necessary to publish an MX entry in order to declare SES as the email receiver for a specific domain or subdomain (more info here).

Build the Stop Scenario

This time we want to stop the instance after 8h. It's a simple Lambda function.

Let's first create our Lambda function. Go into Lambda, and click on Create function to build a new one. Name it mc_shutdown and use a Node.js 14.x or more runtime.

Replace the content of index.js file with the following:



const AWS = require("aws-sdk");
var ec2 = new AWS.EC2();

exports.handler = async (event) => {
  try {
    var result;
    var params = {
      InstanceIds: [process.env.INSTANCE_ID],
    };
    var data = await ec2.describeInstances(params).promise();
    var instance = data.Reservations[0].Instances[0];

    if (instance.State.Name !== "stopped") {
      var launch_time = new Date(instance.LaunchTime);
      var today = new Date();
      result = "instance running";
      if ((today - launch_time) / 3600000 > process.env.MAX_HOURS) {
        console.log("stopping the instance...");
        var stop_data = await ec2.stopInstances(params).promise();
        result = "instance stopped";
      }
    } else {
      result = "instance not running";
    }
    const response = {
      statusCode: 200,
      body: result,
    };
    return response;
  } catch (error) {
    console.error(error);
    const response = {
      statusCode: 500,
      body: "error during script",
    };
    return response;
  }
};

In Configuration, set the following:

add an environnement variable named INSTANCE_ID with the value that correspond to the Instance Id of your Minecraft server (something like i-031fdf9c3bafd7a34).
add an environnement variable named MAX_HOURS with the value that correspond to number of hours allowed after startup, like 8 for 8 hours).
the role permissions must include the right to start our EC2 instance like this:

We add a trigger to fire the task every 20 minutes:

Hurray the configuration is done !

Conclusion

This setup is working nicely here, my son is happy because he start himself the instance when he need. I am happy because it reduce a lot the cost of this service. On the last 3 months I see that the EC2 Compute cost for this server is less than 1 US$ 😅 (around 17 US$ before the optimization) so 95% less expensive !

Currently the configuration is made manually in the console, I would love to spend some time to change that one day, using for example the CDK toolkit.

It's also probably possible to manage the storage of the Minecraft world on S3 instead of the Instance EBS disk (some $$ to save here, but not a lot).

It was a very fun project to build using multiple AWS services! Do you see other usages of dynamically boot EC2 instances using Lambda functions? Let me know in the comments!

How to use React inside a Wordpress application ?

Julien Bras — Fri, 05 Mar 2021 20:49:43 +0000

The context

I was asked a few weeks ago to build a new page on a existing Wordpress site, in order to build a "shop area":

Link to page

I am not very efficient to work on Wordpress template system (not my cup of tea!), and I have now a solid background around React frontend. I want to see how it can be possible to integrate, on an existing Wordpress installation, one React application to produce this particular need.

This article will explore the Wordpress/React options, then I will show you, step by step how I have implemented a React application inside Wordpress. Finally I list you a few issues of the actual solution.

React with Wordpress?

React is a popular javascript library that is generally used to build frontend application inside the browser. There is also a huge ecosystem of solutions around React (CreateReactApp, NextJs, Gatsby...) that help to use the library in a reliable frontend application.

Wordpress is a very famous CMS (Content Management System) that is still used by a lot of website. It's very easy to use for content editor, and it comes with lots of plugins.

There is multiple ways to mix Wordpress and React, but I will show you two examples here.

Build a javascript frontend using Wordpress REST API

Wordpress comes with a nice REST API, and so it's possible to build a classic Single Page Application (using CreateReactApp for example) that consume this API. Wordpress is still used to write articles, but the website generated is driven by a different frontend application. It's the Headless CMS concept. This article is a great guide to achieve this:

How to Create a Modern Web App Using WordPress and React

Gatsby, a static site builder using React, have also a dedicated solution here:

Sourcing from WordPress

This solution is a radical one for an already existing website, as you need to work on all existing content and transfert it to your new frontend solution. It's nice but it's too big for my own project.

Integrate a React application inside Wordpress

React is only a simple javascript library. It's not needed to build an entire site, you can just load the library on a part of your existing page. From the documentation of ReactJs:

React has been designed from the start for gradual adoption, and you can use as little or as much React as you need. Perhaps you only want to add some “sprinkles of interactivity” to an existing page. React components are a great way to do that.

The majority of websites aren’t, and don’t need to be, single-page apps. With a few lines of code and no build tooling, try React in a small part of your website. You can then either gradually expand its presence, or keep it contained to a few dynamic widgets.

I have a few article disussing how to add a React application in a Wordpress site. This one show that, but for the administration panel:

Ghost Inspector - Automated Website Testing and Monitoring

I choose to go on this way because it's easier than rebuild the entire site, and it give me enough power to work like I want.

Integrate a React application in Wordpress

I want to build a page, visible by end-users, that is loading a React application showing some articles of a particular category from the Wordpress website in a grid layout. This section will guide you in the creation of this page.

The big picture

I will create a new wordpress plugin. The plugin will show the React application if I use a specific short-code in a page or an article. The React application will consume the REST API of Wordpress to show the articles.

Build a dedicated plugin

To isolate the development I choose to work in a dedicated plugin. It is also possible to work in the theme functions.php but I think it's cleaner to have a specific folder for this project.

In the plugins folder of your wordpress application, make a new folder named my-react-app. Create inside the folder a php file my-react-app.php.

Inside my-react-app let's bootstrap a new Create React App project:

npx create-react-app frontend

It will create inside the folder frontend a new React application using the class Create React App.

In the php file you can put:

<?php
/**
 * Plugin Name: my-react-app
 * Plugin URI: a url
 * Description: A react application
 * Version: 0.1
 * Text Domain: my-react-app
 * Author: Julien Bras
 * Author URI: https://sidoine.org
 */

// First register resources with init 
function my_react_app_init() {
    $path = "/frontend/static";
    if(getenv('WP_ENV')=="development") {
        $path = "/frontend/build/static";
    }
    wp_register_script("my_react_app_js", plugins_url($path."/js/main.js", __FILE__), array(), "1.0", false);
    wp_register_style("my_react_app_css", plugins_url($path."/css/main.css", __FILE__), array(), "1.0", "all");
}

add_action( 'init', 'my_react_app_init' );

// Function for the short code that call React app
function my_react_app() {
    wp_enqueue_script("my_react_app_js", '1.0', true);
    wp_enqueue_style("my_react_app_css");
    return "<div id=\"my_react_app\"></div>";
}

add_shortcode('my_react_app', 'my_react_app');

You will end with this structure:

plugins
└── my-react-app
    ├── frontend
        │     ├── README.md
        │     ├── node_modules
        │     ├── package.json
        │     ├── .gitignore
        │     ├── public
        │     └── src
    └── my-react-app.php

Good ! The basic setup is now working ! Let's test it!

Develop your React app

Go into the frontend folder. Start the development server by running:

yarn && yarn start

Replace yarn by npm if needed ! It will start a browser and show you this:

You can start by editing any of the file under frontend/src and actually develop your application.

Build your React app

In order to use your application in Wordpress you need to build it. I haven't found yet a solution to develop the application directly inside Wordpress. To build the output for Wordpress, I recommend to rely on craco, a tool that can help to generate a single js file with predictable name.

First install craco:

yarn add @craco/craco

Then create a craco.config.js in frontend folder:

// craco.config.js
module.exports = {
  webpack: {
    configure: {
      output: {
        filename: "static/js/[name].js",
      },
      optimization: {
        runtimeChunk: false,
        splitChunks: {
          chunks(chunk) {
            return false;
          },
        },
      },
    },
  },
  plugins: [
    {
      plugin: {
        overrideWebpackConfig: ({ webpackConfig }) => {
          webpackConfig.plugins[5].options.filename = "static/css/[name].css";
          return webpackConfig;
        },
      },
      options: {},
    },
  ],
};

Then edit the package.json file for the build command:

"scripts": {
    ...
    "build": "craco build",
    ...
  },

Comment the reportWebVitals(); in frontend/src/index.js: (it prevent from having a single js file, dont forget to remove the import too !)

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
// reportWebVitals();

Modify the div id used in frontend/src/index.js:

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('my_react_app')
);

Modify the div id used in frontend/public/index.html:

<body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="my_react_app"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.

      You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.

      To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>

It's important to modify the id because by default the root is too generic for something we will include on a Wordpress page.

Add also a homepage value in the package.json (this will help for images):

"version": "0.1.0",
"private": true,
"homepage": "/app/plugins/my-react-app/frontend/build/",
"dependencies": ...

Then test the build !

yarn build

It will generate a build folder inside frontend (with a single script.js file):

yarn run v1.22.4
$ craco build
Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  41.86 KB  build/static/js/main.js
  518 B     build/static/css/main.css

The project was built assuming it is hosted at /app/plugins/my-react-app/frontend/build/.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.

Find out more about deployment here:

  https://cra.link/deployment

✨  Done in 6.46s.

Test on Wordpress

Login on your Wordpress installation and activate the my-react-app plugin. Then in any page or article, use the short-code [my_react_app]like this:

If you publish the page you will see:

It's a win 🏆 !

Use REST API

Inside the React application it's very easy to consume the REST API. I am actually using a API constant that point to the correct endpoint:

export const API = process.env.REACT_APP_API || `${window.origin}/wp-json`;

So I am able to define the environment variable REACT_APP_API in the .env file if I want to not use the wordpress on the same host (development mode).

Then inside a React component, I can use a useEffect to populate a items state:

useEffect(() => {
    let category = process.env.REACT_APP_CATEGORY;
    const params = new URLSearchParams({
      categories: category,
      _fields: "id,title,meta,content,featured_media,fimg_url,tags",
      per_page: 100,
    });
    fetch(`${API}/wp/v2/posts?${params}`)
      .then((res) => res.json())
      .then(
        (result) => {
          setItems(result);
        },
        (error) => {
          setError(error);
        }
      );
  });

Extra mile with Bedrock and Trellis

On this particular application I am relying on Bedrock, a very good solution to develop on a Wordpress application with managed plugin, and on Trellis, an other very food solution to facilitate the server provisioning and solution deployment (thanks Roots !, I hope to test Sage some day !)

I have done the following to help me on this project

Using `mu-plugins` folder

Instead of deploying the plugin in plugins I am using the mu-plugins folder so I am sure the plugin is always loaded. Does not need a plugin activation.

Enhanced deploy procedure

I want to deploy only the builded version, and never the src folder. So each time I am deploying a new version I want to build my application and send only the build folder.

Inside my trellis/group_vars/SERVER/main.yml I have added:

deploy_build_before:
  - '{{ playbook_dir }}/deploy-hooks/build-before-my-react-app.yml'

This will add a script before build time.

Let's now create the build-before-my-react-app.yml file in trellis/deploy-hooks folder:

- name: Install npm dependencies
  command: yarn
  delegate_to: localhost
  args:
    chdir: "{{ project_local_path }}/web/app/mu-plugins/my-react-app/frontend"

- name: Compile assets for production
  command: yarn build
  delegate_to: localhost
  args:
    chdir: "{{ project_local_path }}/web/app/mu-plugins/my-react-app/frontend"

- name: Copy production assets
  synchronize:
    src: "{{ project_local_path }}/web/app/mu-plugins/my-react-app/frontend/build/"
    dest: "{{ deploy_helper.new_release_path }}/web/app/mu-plugins/my-react-app/frontend"
    group: no
    owner: no
    delete: yes
    rsync_opts: --chmod=Du=rwx,--chmod=Dg=rx,--chmod=Do=rx,--chmod=Fu=rw,--chmod=Fg=r,--chmod=Fo=r

Thanks for the Sage 9 build-before example 😉

Conclusion and some concerns

As it's a React application I have some concerns:

SEO: Google will probably not understand well my page...
managing correctly CSS is tricky because the Wordpress page will inject some css classes (that you will not see in development mode using yarn start)

This project have been realized because the classic plugin we were using for this kind of page (WPBakery) doesn't come out-of-the-box with filtering and ordering capabilities. Some options are available but limited in personalization. And it's fun to put some new tooling in a classic existing web application ! Get a try !

Jira: one easy solution to generate custom reporting

Julien Bras — Fri, 19 Feb 2021 03:53:43 +0000

Jira is a powerful tool

As a Product Owner, I am using a lot Jira in the company I am working for. It's my main tool and it's a very common solution to manage software development. It comes with a lot of capabilites for managing Agile team, and lots of reports:

Feel the limits

But sometimes I feel a bit limited by the output generated by the tool. The more we are using it and feeding data in it, the more information I want to be able to read.

A typical exemple: for each task each user is providing the time spent for different activities: development, review, test... We are adding tags in the comment box of the work log:

So it's really easy to put the data in Jira but there is no pre-built system to produce an analysis for each sprint and know how many time have been spent for #DEV or #TEST.

Or we are using Epics to track our most important features, but the time spent is at the Story level, there is no easy way to know how many time a specific Epic took.

Sometime a plugin can answer but I want to find a more generic solution to this issue. How can I produce the report I want to produce ?

Introducing the API and Jupyter

Jira comes with a very powerful REST API. So the data can be accessed by this way very easily.

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

So yes, it's possible to write a bit of python code that will connect to Jira Cloud (or Jira Server if you run Jupyter not so far away the Jira Server instance)

This small section will list all Epics for a specific project and a specific version, and generate a nice table (using pandas), with a footer for a sum of estimation.

# Show epics with consumed time and estimated time
project = "MYPROJECT"
version = "1.0.0"
issues = jira.search_issues('project = "{project}" and issuetype = "Epic" and fixversion = {version} ORDER BY key DESC'.format(project=project, version=version), maxResults=0,startAt=0)
formated_issues = list()
for issue in issues:
  # compute estimated time
  estimated = 0
  if(issue.fields.customfield_10663):
    estimated = issue.fields.customfield_10663

  # compute spentTime from linked issues
  spent_time = 0
  child_issues = jira.search_issues('parent = {epic}'.format(epic=issue.key))
  for child_issue in child_issues:
    try:
      spent_time += child_issue.fields.timespent
    except TypeError:
      # No spent time on this one !
      spent_time += 0

  formated_issues.append([issue.key, issue.fields.status.name, issue.fields.labels, issue.fields.summary, estimated, spent_time])
df = pd.DataFrame(formated_issues, columns=["id", "status", "labels", "summary", "estimatedTime", "spentTime"])
df["spentTime"] = round(df["spentTime"] /3600 /7, 1)
df['ratio'] = round(df['spentTime'] / df['estimatedTime'] *100, 0)
df = df.fillna(0).sort_values(["status", "ratio"])
df['id'] = df['id'].apply(lambda x: '<a href="https://mydomain.atlassian.net/browse/{0}">{0}</a>'.format(x))
df = df.append(pd.DataFrame([
                             [df.id.count(), df.estimatedTime.sum(), df.spentTime.sum(), round(df.spentTime.sum()/df.estimatedTime.sum()*100)]], 
                            index = ["Total"], 
                            columns=["id", "estimatedTime", "spentTime", "ratio"]))
HTML(df.to_html(escape=False))

Here is the generated report (and yes there is a link to go directly on each case):

I have started by using this method with Jupyter, based on this article: Using Jupyter Notebooks to Access Jira. But (there is always a but! 😅) I was thinking that it's too time-consuming to start the Jupyter via docker, and it's not very easy to share a document to a co-worker:

OK send me the Jupyter file by email and give me the procedure to launch the Jupyter image on my computer. Oh wait, I need to start Docker too ?

There is probably a more easy way to do that no ?

Introducing Colaboratory

Colaboratory, or Colab when you are in a rush, is a service operated by Google that is running Jupyter notebook for you (yeah no more server to handle !). And it's very easy to share notebooks via Google Drive or Github for example.

It's not the only one (Azure Notebooks too) but our company is relying a lot on Google Drive, so it's easier to share documents.

As of now I have written 6 or 7 notebooks that helps me dealing with Jira data:

how to plan a 3-months roadmap for a new version,
how to review my previous version and check why this Epic have not been delivered in time
how to build sprint dashboard that give me a good vision of what have been done
and so much...

Finally I can share with you some examples. I have created a dedicated repository on github, so it is easy to generate a link to a specific notebook.

julbrs / jira-notebooks

Jupyter Notebooks for Jira

Jira Notebooks

What ?

This repository is hosting a few Jupyter Notebooks that consume Jira data.

Here is the article linked to this repository: Jira: one easy solution to generate custom reporting

Who ?

I (Julien Bras) am the owner of the notebooks here.

Contribute ?

Feel free to make a PR on this repository to share some Jupyter Notebooks around Jira. Be sure to provide a document without any data.

View on GitHub

One notebook on colab : jira_roadmap_plan.ipynb

Do not hesitate to create a PR on the repository if you want to share some nice looking notebook for Jira! Thanks for reading !

Beginner guide to Maven Central publishing

Julien Bras — Tue, 05 May 2020 03:35:06 +0000

Maven Central is the central repository for Maven, the main Java repository where you can find libraries and components. Java developers are using this repository to create java application and manage efficiently dependencies.

I use to be very intimidated by pushing an artifact (i.e. a java package) to Maven Central, as it seems very complicated to me by the past. For example push to npmjs seems more easy (npm publish !).

This article is just a guide (at least for myself) on how to push easily some content to Maven Central, with just a Github account.

Before starting why pushing to Central ?

Maybe you are asking yourself why pushing to Maven Central when we have in 2020 multiple way of releasing java package ?

Maven Central is the main repository. You don't have to declare an additional repository in your pom.xml, it is just available by default in your Maven/Gradle... project.
Other main repositories are generally synced to this main one. Here I will show you how to publish to OSSRH (OSS Repository Hosting) that is the main way to publish package. This repository is synced to Maven Central.
There is not so much other solution available as pure repositories, for free (I want to publish open source content, I don't want to pay for that)
GitHub Package seems very promising but end-users must manipulate the repositories part of the pom.xml file and must be authenticated even for public packages.

First step, create account and get a groupId

create a JIRA account here
Create a new issue using this template.

The issue must describe the project you want to share. The most important information in the case is the groupId. The groupId is the main identification of you as a package provider, so it must be unique. It is generally an url in reverse order like : to.dev.

For a hobbyist developer using github or equivalent git platform I recommand the following : io.github.YOURUSERNAME.

Once the ticket is created, you will have to prove that you own the groupId provided (for GitHub it's just a matter of create a new temporary project repo)

Second step, make your project pretty

Here is the small maven project I want to share on Central:

wiiisdom / biar-manager

BIAR BI ARchive File library

It's a very simple library that is designed to read a specific file format.

There is a long list of requirement for your project to be accepted on OSSRH, you can read it here.

To resume the requirements:

you must supply javadoc (using maven-javadoc-plugin for example)
you must supply sources (using maven-source-plugin for example)
you must sign the files with GPG (using maven-gpg-plugin for example)
the pom.xml file must contains enough metadata:
- groupId (unique namespace like io.github.USERNAME)
- artifactId
- version
- name
- description
- url
- licenses part
- developers part
- scm section

For reference here his my pom.xml file that fit the requirements.

Finally deploy !

Do not forget to install gpg and create a new key if you don't have one (I found again my key generated in 2004 yes I am old !)

brew install gpg # mac example
gpg --gen-key

Using Maven it is needed to use the right distributionManagement and help yourself by using the nexus-staging-maven-plugin plugin:

<distributionManagement>
  <snapshotRepository>
    <id>ossrh</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </snapshotRepository>
</distributionManagement>
<build>
  <plugins>
    <plugin>
      <groupId>org.sonatype.plugins</groupId>
      <artifactId>nexus-staging-maven-plugin</artifactId>
      <version>1.6.7</version>
      <extensions>true</extensions>
      <configuration>
        <serverId>ossrh</serverId>
        <nexusUrl>https://oss.sonatype.org/</nexusUrl>
        <autoReleaseAfterClose>true</autoReleaseAfterClose>
      </configuration>
    </plugin>
    ...
  </plugins>
</build>

(again this is available in my pom.xml)

You must auth on the OSSRH repository by modifying your settings.xml file under ~/.m2:

<settings>
  <servers>
    <server>
      <id>ossrh</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
  </servers>
</settings>

Use the id and password you have created at the start of this article.

And then if your maven project have already the right version you can directly push to OSSRH using:

mvn clean deploy

The property autoReleaseAfterClose of nexus-staging-maven-plugin set to true will directly push the artifact from snapshot to the release repository.

Note: to change your Maven project version you can use the following:

mvn versions:set -DnewVersion=1.2.3 # set the version
mvn versions:commit # remove the pomBackup file
mvn versions:revert # back to previous version

After a few hours you will be able to see your package on Maven Search

Nice job !

Other stuff to speak of

During my exploration I have found this tool : https://jitpack.io/

It seems to be an easy way to consume an existing git repository as maven dependency. I have not yet tested it but it feels like the flexible I miss with Maven (Composer for PHP allow to consume git repo, npmjs allow to consume git repo, etc...)

References

This article rely a lot on the OSSRH guide here.

Conclusion

Well there is more steps than npmjs but it is very affordable and you can do the same. Maybe it can be also the time for me to explore Gradle or other new tooling around Java... Let me know if you have already pushed an artifact to Maven Central ? Thanks for reading !

Moving from Eclipse to VSCode by a Java Developer

Julien Bras — Sun, 03 May 2020 01:11:11 +0000

I am working for a software editor and we mainly use Java as backend language. I use to work with Eclipse since around 2010, only for Java projects. Here is my journey and a quick comparison of the tools.

First of all I need to inform that I am actually Product Owner and not anymore a full-time developer. So I am still looking at Java source code project, but with a different level of usage than before.

VSCode setup

VSCode is relying a lot on extension. Each extension add a little extra power to the tool. So the initial text editor can be compared with a fully featured IDE once the right extensions have been installed.

I don't want to present each extension, I found this article which is describing that very well.

Globally you can relly on the Java Extension Pack that install all the main Java extensions for you.

Import a project?

This is I think the most important change between Eclipse and VSCode.

Eclipse is relying on a workspace concept where you import Java projects. My main concern with this approch is around multi-modules Maven projects : once you add or remove a module, Eclipse is lost and you need to import again the missing module.

VSCode is more like other editor (Atom...), and you can simply open a folder that contain your multi-modules Maven project. If some project have been removed then you will not see it anymore.

Folder presentation in VSCode

Workspace in Eclipse

It's not a big difference but for me it is more easy to switch between projects. I do not loose anymore time to import projects, I just open the right folder. There is also a workspace system in VSCode to open multiple folder at once, it may be useful if you work on multiple projects at the same time (front and back for example).

Develop ?

The global experience is very good.

You still have auto-completion and JavaDoc is shown when it's necessary. There is also an equivalent of the Run Configuration with the Run panel to fire your project.

The Run panel rely on a launch.json file, and it can be saved on your git repository if you want to share it with team-workers.

Test ?

The right extension help to run tests. There is also some helpers to run the test directly before the test method or the test class.

Share your code ?

Git is directly available in VSCode. I have never rely on any Git addon in Eclipse, as I found some products buggy. So I was relying only on the Git command line. I still rely a lot on the command line but I am happy to see this very good integration of Git directly into the product.

And you have some indicator directly in the editor (green if new line, red if removed lines...). I have never see this kind of indication inside Eclipse. Probably I have never installed the right extension ;)

Run tooling (Maven, etc) ?

There is also a Maven extension that let you execute all Maven commands. But here I prefer using the excellent terminal that let you do what you want. It's here, just use it.

Conclusion

One month after VSCode installation and first test on Java projects, I realize that I haven't opened anymore Eclipse. Just today to make a screenshot...
I have not yet speak of the performance too. I have a decent MBP and Eclipse take always a couple of seconds to startup. VSCode start in less than a second.
Last, I am not only coding in Java but also Javascript. I have originally installed VSCode to replace Atom editor for my JS projects... So I am happy to be able to use a single tool for all my programming needs. It's faster and I am more comfortable to use a single tool for coding.
So just an advice: get a try on one of your Java project, you may be surprised to change your habits !

Add an Admin Panel to an existing Express API / React frontend

Julien Bras — Mon, 13 Apr 2020 14:15:17 +0000

Context

Here is a quick personal review of JS solution for generating an Admin Panel today (April 2020). The goal is to add an admin panel to an existing Express API backend + React frontend.

I want to have something comparable to the Django Admin as a reference.

It must manage:

some kind of authentication
some kind of file management, ideally with S3 hosting as the backend is running on Heroku
I am OK to add modules to my existing Express application, or change completely the backend.

AdminBro

Just an admin-panel to add on Express app. There is no easy way to add a file upload feature (this can help). But it is a great tool to not modify to much my existing solution.

Strapi.io

This tool is more a Content Management System than a Framework. It define itself as a headless CMS. I can find some Drupal inspiration inside the tool:

tool to manage different Content-Type (entities)
plugin system to add authentication (lots of providers supported)
support a classic database or MongoDB
obviously there is an admin panel to manage the entities
you can manage files, and it can be handled by default on various providers including S3 (good !)
it provide out of the box a REST api, and you can also activate a GraphQL api easily.

It is a very fast way to create a headless backend. But as it rely a lot on plugins, you may not find exactly the feature you need. For example there no Internationalization plugin yet to manage multiple language.

Feathersjs

This is a framework. No admin interface.

entities are service (very easy to add a new service !)
handle different backend including Mongo
no admin webapp backend but there is a react-admin plugin ! react-admin is a solution to add an admin panel directly in a react application
react-admin is not easy to implement !
nothing for file upload out of the box

Back4app

This is a hosted version of Parse server. Parse use to be a tool provided by Facebook, but it now an open-source tool here. It seems we cannot consume data as raw RESTapi but rather with a dedicated SDK.

Provide lots of stuff out of the box

API to consume the data
Authentication with multiple providers
Admin Panel (yes it is mandatory for my little selection !)
File management

React-Admin

This is a React library that can generate an admin panel. It rely on existing REST or GraphQL API , with a a DataProvider object that explain how to communicate with the API. It is needed to write the DataProvider if your API does not stick exactly with an already available DataProvider. In my case the Simple REST was not compatible directly.

Conclusion

I was pushing hard on AdminBro but I wasn't able to handle a clean and fast solution for one of my entities that is linked to a file. The file management seems not obvious for me ! AdminBro is not hard to install and setup but it require customization to fit exactly with your existing Express application. My main issue was this file management point.

Feathers is a nice idea, but it require a start from scratch regarding the backend, and the react-admin solution was not so magic.

About Back4app it is the first time I was dealing with the Parse system. I was worried to need to implement a specific Parse Client API inside my frontend to use it. The goal was not to rebuild completly the frontend application.

Finally I decided to give a try with Strapi.io and check if it fit my collaborators ! I know it is not perfect but it is very fast to setup and deploy.

Update 2020-04-22

Finally I have decided to use the react-admin project in the frontend part:

It does not need to replace my existing Express API. It save me some time.
It is more all-in-one place for all my users. Both end-users and admin-users can consume the same url, same application. It is globally easier.
It seems pretty intimidating to create a custom DataProvider for my existing API but the documentation of the project is accessible and I have managed to setup mine in a couple of hours.

Let me know in comment what is your favorite admin-panel solution, and why you need one !