DEV Community: Emmanuel Balpe

Solving the hanging tests in Micronaut application

Emmanuel Balpe — Sun, 29 Jan 2023 21:08:20 +0000

We've been facing a strange issue with our Micronaut application tests.
Specifically tests that were producing Kafka messages were successfully run bug sometimes the test engine was kept hanging and never finishing.

We are running the following versions:

Java 17
Gradle 7.6
Micronaut 3.8.3

This problem seems related with this issue described here.

After some investigation we've found out that the issue was caused by the fact that sometimes some
KafkaListener in our projects were still active and processing messages even after the test was finished.
This caused the application to keep running and the test engine to wait for the application to finish.

Our solution was to add a @PreDestroy method to a new AppApplicationListener singleton so that the listeners can terminate
their process so that the application shutdown can be successful.

@Singleton
public class AppEventListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(AppEventListener.class);

    @EventListener
    public void onStartupEvent(StartupEvent event) {
        ...
    }

    @PreDestroy
    void onShutdown() throws InterruptedException {
        LOGGER.info("PreDestroy: Shutting down after a sleep of 1000ms to allow the tests to complete.");
        LOGGER.debug("It looks like the test is stuck if we don't implement this @AfterAll method. " +
                "This is an issue with Micronaut and JUnit 5." +
                "https://github.com/micronaut-projects/micronaut-gcp/issues/638");
        Thread.sleep(1000);
    }

}

Secure your Java Servlet Application with Keycloak

Emmanuel Balpe — Wed, 27 May 2020 20:28:44 +0000

We'll see how to configure a Java Servlet based application so it can be secure with Keycloak.

Keycloak is an Open Source Identity and Access Management that can be used to delegate entirely the security of an application.

1. Keycloak configuration

The Keycloak documentation is really easy to follow. You can see for yourself here the section about the configuration of your Keycloak instance: https://www.keycloak.org/docs/latest/authorization_services/#_getting_started_hello_world_create_realm

You need to configure:

A realm
A user with role user, we'll see later how it's used
A Client. It's a representation of your Java application
- Client protocol: openid-connect
- Access Type: public
- Valid Redirect URIs: the url of your development environment or * for the time being

2. Tomcat security-constraint

We're using the Tomcat security-constraint that enable a security verification at the application level on Tomcat.
The Keycloak team developed a convenient Valve for the Tomcat Security system that handle the redirect to and from the Keycloak login page.

2.1. You need to add the following to the `context.xml` of your application:



<Context>
    <Valve className="org.keycloak.adapters.tomcat.KeycloakAuthenticatorValve"/>
</Context>

2.2. Install the Keycloak Valve libraries into the `${tomcat}/lib` directory on your Tomcat server

2.3. You need to copy the `keycloak.json` config file into `/WEB-INF/keycloak.json`

You can download the file in your Client installation tab:

2.4. Add `security-constraint` in your `web.xml`



<security-constraint>
    <web-resource-collection>
        <web-resource-name>Private area</web-resource-name>
        <url-pattern>/esp_privat/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>user</role-name>
    </auth-constraint>
</security-constraint>

<security-constraint>
    <web-resource-collection>
        <web-resource-name>Public area</web-resource-name>
        <url-pattern>/api/*</url-pattern>
    </web-resource-collection>
</security-constraint>

<login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>this is ignored currently</realm-name>
</login-config>

<security-role>
    <role-name>user</role-name>
</security-role>

Here we defined 2 URL patterns:

/esp_privat/* that require a user to be connected with a role user
/api/* that require no authentification

2.5. Results

So when you try accessing any route under /esp_privat/ in your application Keycloak valve now automatically redirect you to the login page in your Keycloak instance.
When successfuly logged in Keycloak redirects you to the asked page.

What we need to do now is to identify the user logged in thank's to the token Keycloak is adding to the cookies of the web navigator.

3. Intercept Keycloak access token to log the user into your app

3.1. Keycloak dependencies

Add the following to the pom.xml of your webapp application:



<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-core</artifactId>
    <version>9.0.2</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-adapter-core</artifactId>
    <version>9.0.2</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-adapter-spi</artifactId>
    <version>9.0.2</version>
    <scope>provided</scope>
</dependency>

Notice the scope = provided since we will be using the libraries added previously into the tomcat library folder. We don't want to override it with another version of the libraries.

3.2. Read the token

The following snippet will extract the token from the request and verify if it's lifetime is expired. It returns true in case the token is valid.




import org.keycloak.KeycloakSecurityContext;
import org.keycloak.TokenVerifier;
import org.keycloak.common.VerificationException;
import org.keycloak.representations.AccessToken;

... 

/**
* Verify if user is logged in keycloak by validating token in request
*/
public boolean isLoggedInKeycloak(HttpServletRequest request) throws VerificationException {
    KeycloakSecurityContext keycloakSecurityContextToken = getKeycloakSecurityContextToken(request);
    if (keycloakSecurityContextToken == null) {
        return false;
    }
    return !isTokenExpired(keycloakSecurityContextToken);
}

private boolean isTokenExpired(KeycloakSecurityContext keycloakSecurityContextToken) throws VerificationException {
    AccessToken token = TokenVerifier.create(keycloakSecurityContextToken.getTokenString(), AccessToken.class).getToken();
    if (token.isExpired()) {
        logger.warn("User token is expired..." + token);
        return true;
    }
    return false;
}

In our case we also needed to verify if the user is a member of the correct group so we added the following method check:



private void handleGroupMembership(@Nonnull KeycloakSecurityContext keycloakSecurityContext, String keycloakPreferredUsername) {
    Object groups = keycloakSecurityContext.getToken().getOtherClaims().getOrDefault("groups", new ArrayList<>());
    if (groups == null) {
        throw new GenericRuntimeException("Fail to read groups from the token of the user " + keycloakPreferredUsername);
    }
    ((List<String>) groups)
        .stream()
        .filter(s -> s.equalsIgnoreCase("/my-group"))
        .findFirst()
        .orElseThrow(() -> new GenericRuntimeException("User \"" + keycloakPreferredUsername + "\" is not a member of /my-group"));
}

We then called the previous method in a pre-action hook into all the call received by our servlets so that it can be catched by any servlet like so:



boolean isUserLoggedIn = request.getSession().getAttribute(USER_SESSION) != null;
if (isLoggedInKeycloak(request) && !isUserLoggedIn) {
    logger.info("User logged in Keycloak but not logged in the app. Logging in the user...");
    new KeycloakLoginService().login(request, getKeycloakSecurityContextToken(request));
}
else if (!isLoggedInKeycloak(request) && isUserLoggedIn) {
    logger.info("User not logged in Keycloak but logged in the app. Logging out the user...");
    sessionLogout.logout(request, response);
    return;
}

3.3. Logout

To logout a user from Keycloak you can use the request.logout() method. We use the following method:



public void logout(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    HttpSession session = request.getSession(false);
    if (session != null) {
        session.invalidate();
    }
    request.logout();
    request.getSession(true); // create a new session
    response.sendRedirect(request.getContextPath());
}

4. Maven profiles to compile versions with and without keycloak login

In one of our project we needed to be able to deploy a version of the app that doesn't use the Keycloak login feature but our previous login mechanism.
Of course we wanted to keep a unique codebase with the less difference as possible.
We identify that the only thing preventing us from working as before was the security-constraint section in the web.xml config file.

We will be using the Maven filtering solution with a little hack we found on SO: https://stackoverflow.com/questions/3298763/maven-customize-web-xml-of-web-app-project/8593041#8593041
It consists in adding 2 variables in your web.xml like so:



${enable.security.start}
<security-constraint>
  ...
  // all of the XML that you need, in a completely readable format
  ...
</login-config>  
${enable.security.end}

And have it replaced by comment block start <!-- and end -> in the profile where you don't want to use Keycloak.

So in our default ci profile we defined the following properties:



<enable.security.start></enable.security.start>
<enable.security.end></enable.security.end>

and in the without-keycloak profile:



<enable.security.start>&lt;!--</enable.security.start>
<enable.security.end>--&gt;</enable.security.end>

Automatiser le déploiement avec Gitlab + Docker + Ouroboros

Emmanuel Balpe — Wed, 27 May 2020 20:26:04 +0000

Le but de cet article est de vous montrer une solution simple et rapide pour mettre en place du déploiement continue.
La stack du projet est composée d’une partie API en nodejs et d’une base de donnée en mongodb. La pipeline va créer une nouvelle image de notre API en nodejs et la déployer sur notre registry privé.

1. Gitlab CI

L’objectif de notre pipeline est d’exécuter les tests unitaires et d’intégration puis de construire et déployer une image de l’API sur notre registry interne.

github gist Dockerfile

FROM node:10

ENV TZ=Europe/Paris
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN apt-get update && apt-get install -y build-essential && apt-get install -y python

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json /usr/src/app/

RUN npm install

# Bundle app source
COPY ./src/  /usr/src/app/

# Copy jest configuration so we can execute tests on the container
COPY jest.config.js /usr/src/app/jest.config.js

# Copy config .env file
COPY ./.env_docker /usr/src/app/.env

EXPOSE 3000
CMD [ "node", "server.js" ]

L'image du container Node est décrite ci dessus: on copie les sources du projet, on run un npm install, on expose le port pour Express puis on set les entry points node et server.js

github gist gitlab-ci.yml

image: docker:19.03.1

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2
  PROJECT_IMAGE: registry.domain.com/project-name
  PROJECT_RELEASE_IMAGE: $PROJECT_IMAGE:$CI_COMMIT_REF_NAME
  DOCKER_TLS_CERTDIR: ""

services:
  - docker:19.03.1-dind

before_script:
  # login sur le docker registry
  - echo "$CI_JOB_TOKEN" | docker login -u gitlab-ci-token --password-stdin registry.domain.com

stages:
  - test
  - release

test:
  stage: test
  script:
    # création d'un network pour lier mongo et nodejs
    - docker network rm network-project || true && docker network create network-project*
    # init et run du container mongo
    - docker run -d --name mongo-db --network network-project -v /builds/init-mongo.js:/docker-entrypoint-initdb.d/init-mongo.js -e MONGO_INITDB_DATABASE='project' -e MONGO_INITDB_ROOT_USERNAME='root' -e MONGO_INITDB_ROOT_PASSWORD='root' mongo:4.0
    # build de l'image node (les sources sont copiées et npm install est fait)
    - docker build --pull --network network-project -t $PROJECT_RELEASE_IMAGE node/
    # démarrage du container avec les variables d'environnement de la ci
    - docker run -d --env-file node/.env_tests --network network-project --name nodejs_test $PROJECT_RELEASE_IMAGE
    # exécution des tests
    - docker exec nodejs_test npm test
    # cleanup kill containers and network
    - docker stop nodejs_test && docker rm nodejs_test
    - docker stop mongo-db || true
    - docker network rm network-project

release:
  stage: release
  script:
    # build de l'image
    - docker build --pull -t $PROJECT_RELEASE_IMAGE node/
    # déploiement de l'image sur le repository
    - docker push $PROJECT_RELEASE_IMAGE

Nous utilisons une image docker comme base pour l’exécution de notre pipeline. Cela nous permet de pouvoir simuler notre stack complète comme sur un vrai serveur avec la base de donnée et l’API afin d’exécuter les tests d’intégration.
Vous remarquerez que nous faisons un docker login sur notre registry en before_script afin de pouvoir déployer notre image tout à l’heure.

Etapes du stage de test:

Création d’un network
Démarrage container mongo
Build et démarrage du container nodejs
Exécution des tests depuis le container nodejs
Nettoyage et rm de tous les containers et network créés

Etapes du stage release:

Build de l’image node
Déploiement sur le registry

2. Déploiement

Sur notre serveur nous utilisons le docker-compose suivant afin de déployer la stack mongo + node:

github gist docker-compose.yml

version: "3.4"

services:
  mongo:
    container_name: mongo
    image: mongo:4.0
    restart: unless-stopped
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_DATABASE: project
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: root
      TZ: Europe/Paris
    volumes:
      - /mnt/docker-volume/project/data:/data/db
      - /mnt/docker-volume/project/init-mongo.js:/docker-entrypoint-initdb.d/init-mongo.js
    networks:
      - project-network

  nodejs:
    container_name: nodejs
    restart: unless-stopped
    # on utilise l'image déployée sur notre registry 
    image: registry.domain.com/project:develop
    working_dir: /usr/src/app
    ports:
      - 3000:3000
    depends_on:
      - mongo
    networks:
      - project-network

networks:
  project-network:

Ce qu’il faut surtout retenir ici c’est que pour le service node nous utilisons l’image qui est déployée sur notre registry privée:

    # on utilise l'image déployée sur notre registry 
    image: registry.domain.com/project:develop

3. Automatisation de la mise à jour du container

Nous allons utiliser un service appelé Ouroboros: ce service va pouvoir surveiller périodiquement les containers qu’on lui précisera afin de les mettre à jour avec la version la plus récente de l’image sur le registry.

github gist docker-compose.yml

version: '3'
services:
  ouroboros:
    container_name: ouroboros
    hostname: ouroboros
    image: pyouroboros/ouroboros
    environment:
      - CLEANUP=true
      - INTERVAL=300
      - LOG_LEVEL=info
      - SELF_UPDATE=false
      - TZ=Europe/Paris
      - MONITOR=nodejs
      - DRY_RUN=false
      - NOTIFIERS=https://hooks.slack.com/services/{tokenA}/{tokenB}/{tokenC}
      - SKIP_STARTUP_NOTIFICATIONS=true
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.json:/root/.docker/config.json

Nous lui précisons de surveiller le container ‘nodejs’: MONITOR=nodejs. Afin qu’il puisse se connecter à notre registry vous pouvez faire un docker login registry.domain.com depuis votre serveur et récupérer le fichier ‘config.json’ qui a été créé dans /root/.docker/config.json

github gist config.json

{
        "auths": {
                "registry.domain.com": {
                        "auth": "tokenXXXX"
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/18.09.0 (linux)"
        }
}

Je vous encourage a aller voir la documentation d’Ouroboros afin de pouvoir tester le bon fonctionnement de votre config avant de démarrer le service sur votre serveur.
Ouroboros utilise un service de notification très complet appelé Apprise qui permet une intégration avec une grande variété de services. Vous voyez ici une implémentation avec Slack. Une notification est envoyée sur le channel configuré pour prévenir dès qu’un container à été mis à jour

Tip: Nous avons dû upgrade la version de docker de notre serveur afin de pouvoir faire fonctionner le service, en 18.x une erreur lors de l’authentification sur notre registry empêchait le bon fonctionnement.

Une fois démarrée le service avec un docker-compose up -d vous pourrez voir le système mettre à jour votre container suivant l’interval que vous aurez configurer.

J’espère que cette solution simple pour mettre en place du CI+CD avec Gitlab vous encouragera a l’essayer sur vos projets.

Jersey Injection Dependency example with HK2

Emmanuel Balpe — Wed, 27 May 2020 20:23:39 +0000

Example of a Jersey project using the dependency injection framework HK2 to inject logged user into the application via a custum annotation

Jersey is a Java Framework that is commonly used to help generate REST Api.
HK2 is a lightweight framework which allow Inversion of Control (IoC) and dependency injection (DI)

Step 1: Starting a new Jersey Project

Generate a project from Maven Archetype:

mvn archetype:generate -DarchetypeArtifactId=jersey-quickstart-grizzly2 \
-DarchetypeGroupId=org.glassfish.jersey.archetypes -DinteractiveMode=false \
-DgroupId=com.example -DartifactId=simple-service -Dpackage=com.example \
-DarchetypeVersion=2.29.1

Step 2: Business Logic

Add some business logic to make this test more interesting in com.example.business:

User
UserSvc
UserDao

Step 3: Automatically bind classes to their implementation

We want to be able to do Dependency Injection of our services. We need to register a AbstractBinder to our jersey app that will automatically match the given injected class with the implementation.

Create a class ApplicationBinder:

@Provider
public class ApplicationBinder extends AbstractBinder {
    @Override
    protected void configure() {
        bind(JustInTimeServiceResolver.class).to(JustInTimeInjectionResolver.class);
}

And register it with our application:

resourceConfig.register(new ApplicationBinder());

And an another class JustInTimeServiceResolver that will handle the automatic binding of services like Google Guice would do.

@Service
public class JustInTimeServiceResolver implements JustInTimeInjectionResolver {

    @Inject
    private ServiceLocator serviceLocator;

    @Override
    public boolean justInTimeResolution(Injectee injectee) {
        final Type requiredType = injectee.getRequiredType();

        if (injectee.getRequiredQualifiers().isEmpty() && requiredType instanceof Class) {
            final Class<?> requiredClass = (Class<?>) requiredType;

            // IMPORTANT: check the package name, so we don't accidentally preempt other framework JIT resolvers
            if (requiredClass.getName().startsWith("com.example")) {
                final List<ActiveDescriptor<?>> descriptors = ServiceLocatorUtilities.addClasses(serviceLocator, requiredClass);

                return !descriptors.isEmpty();
            }
        }
        return false;
    }
}

From there we can already make use of the dependency injection framework.
Create a new endpoint to get the list of users:

@GET
@Path("users")
@Produces(MediaType.APPLICATION_JSON)
public List<User> getUsers() {
    return userSvc.getList();
}

And in MyResource inject the UserSvc service this way:

@Inject
private UserSvc userSvc;

We can do the same in the UserSvc class with the field UserDao:

@Inject
private UserDao userDao;

Note that Injected resources need to have a no args constructor.

Wen can test ou API is responding well with the list of our users this way:

@Test
public void testGetUsers() {
    List<User> users = target.path("myresource/users").request().get(new GenericType<List<User>>() {});
    assertEquals(2, users.size());
}

Step 4: Extract the user from the JWT Token and make it available for injection with a custom annotation

Now we would like to be able to get information about the user that send requests to our API and use it where needed in our application.
Our API will have secured access managed by a JWT Token.
We will create a @PreMatching Jersey filter that will be run before any request and that:

will validate the token that should be in Authorization header
will create a user from the claims in the extracted token
will put the user in the SecurityContext so that it can be accessed by the HK2 framework

@PreMatching
@Priority(Priorities.AUTHENTICATION)
public class PreMatchingCurrentUserFilter implements ContainerRequestFilter {

    @Override
    public void filter(ContainerRequestContext requestContext) {
        try {
            Jws<Claims> jws = new AuthorizationValidator(false).validate(requestContext);
            AppSecurityContext appSecurityContext = new AppSecurityContext(
                    new HashSet<String>(),
                    new User(
                            (String) jws.getBody().get("login"),
                            (String) jws.getBody().get("compte")
                    ),
                    true
            );
            requestContext.setSecurityContext(appSecurityContext);
        }
        catch (Exception ignored) {
        }
    }
}

And register the filter with our application:

resourceConfig.register(new PreMatchingCurrentUserFilter());

Next we will create an new custom annotation that will be used to inject the user anywhere we want.
First create the new annotation, available in Field and in Constructor:

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.CONSTRUCTOR})
public @interface CurrentUser {
}

then a new InjectionResolver, where we will bind the User class with the new annotation.
We can access the SecurityContext using dependency injection and find the User we injected in the @PreMatching earlier.

public class CurrentUserInjectionResolver implements InjectionResolver<CurrentUser> {

   private javax.inject.Provider<SecurityContext> securityContextProvider;

   @Inject
   public CurrentUserInjectionResolver(
      javax.inject.Provider<SecurityContext> securityContextProvider) {
      this.securityContextProvider = securityContextProvider;
   }

   @Override
   public Object resolve(Injectee injectee, ServiceHandle<?> sh) {
      if (User.class == injectee.getRequiredType()) {
         return securityContextProvider.get().getUserPrincipal();
      }
      return null;
   }

    ...
}

And thats’it, wen can access the user using an injection like that:

@CurrentUser
private User user;

or in a constructor:

@CurrentUser
public UserSvc(User user){
    this.user = user;
}

If user was found in the JWT Token if will be accessible here.

Thank’s for reading, you can find the sample project on my Github account

Run multiple Postman Collection in parallel — Stress-Tests

Emmanuel Balpe — Wed, 27 May 2020 20:22:12 +0000

A known limitation of the Postman Collection Runner is that it can only execute collection in a consecutive way. This is just a simple implementation of the solution explained in this StackOverflow conversation.

Create your Postman Collection and corresponding tests

Here I needed to attack first /api/persons to get the list of persons ids.

And then /api/persons/:id for each person in the list.

To do that I used the postman.setNextRequest() tricks that specify the next request that will be executed in the collection run. And in each run I get the last personId and pop() it from the array in environment variables.

Export your collection and the environment variables

And save the files in a postman/ directory.

Create the new npm project

Simply run npm init -y and install the 3 dependencies: npm i async newman path

{
  "name": "newman_parallel",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "async": "^3.1.0",
    "newman": "^4.5.6",
    "path": "^0.12.7"
  }
}

The script !

It’s kind of self explanatory, update the path for your postman collection and environment, specify the number of concurrent run you want to launch with the constant PARALLEL_RUN_COUNT and execute the script with npm start

const path = require('path')
const async = require('async')
const newman = require('newman')

const PARALLEL_RUN_COUNT = 2

const parametersForTestRun = {
    collection: path.join(__dirname, 'postman/postman_collection.json'), // your collection
    environment: path.join(__dirname, 'postman/localhost.postman_environment.json'), //your env
    reporters: 'cli'
};

parallelCollectionRun = function (done) {
    newman.run(parametersForTestRun, done);
};

let commands = []
for (let index = 0; index < PARALLEL_RUN_COUNT; index++) {
    commands.push(parallelCollectionRun);
}

// Runs the Postman sample collection thrice, in parallel.
async.parallel(
    commands,
    (err, results) => {
        err && console.error(err);

        results.forEach(function (result) {
            var failures = result.run.failures;
            console.info(failures.length ? JSON.stringify(failures.failures, null, 2) :
                `${result.collection.name} ran successfully.`);
        });
    });

And here you can play around with your api and build some stress tests to see how it handles the pressure.
You can find the code in the Github repo.

Build a simple CLI tool to simplify your day-to-day tasks

Emmanuel Balpe — Wed, 27 May 2020 20:14:52 +0000

It's a common task in my current position to download and restore client schema on my local postgres to debug it. So I decided to build a small CLI tool to help me.

Fully inspired (to not say copy/pasted) from this post by Lukas White and James Hibbard:
Build a JavaScript Command Line Interface (CLI) with Node.js

How it works

I type pg-schema-restorer in my terminal and the following appears:

The tool look for the .backup files in my folders and subfolders of ~/Documents and ~/Downloads and ask me to select one.

Once chosen it asks me the name of the schema (try to deduce it from the filename)

After that I need to enter the database credentials. Only for the first time because it's saved into a local storage thank's to configstore

And from here the given schema name will be dropped and the backup restored.
I added some flavours in the script I use at my job to automatically reference the new schema into the public table with some more queries but you get the idea.

The code

I won't go into too much details because the post I referenced earlier is very detailled.
The following is just to clear the console and display the ASCII art because I can. It's using chalk to colorize the output, figlet for the art and clear

clear()
console.log(
    chalk.yellow(
        figlet.textSync('PG Schema Restorer', { horizontalLayout: 'full' }),
    ),
)

To look for the backup in a directory and all it's subdirectories I'm using a library called klawSync you can apply a filter to it so it will already find exactly what you're looking for

const findBackupsInPath = filePath => {
    return klawSync(filePath, {
        nodir: true,
        traverseAll: true,
        filter: filePath => {
            const basename = path.basename(filePath.path)
            return basename.match(/.\.backup$/);
        }
    }).map(f => f.path)
}

You can then use this list of backups to inquirer that will display a beautiful prompt with it

askBackupFile: filelist => {
    const questions = [
        {
            type: 'list',
            name: 'backupPath',
            message: 'Select the backup you want to restore:',
            choices: filelist,
        }
    ];
    return inquirer.prompt(questions);
}

I want the database credentials to be stored so that users don't have to type it anytime but in the same time they should be able to reset it if they need to restore on another database.

// retrieve databaseCredentials
const conf = new Configstore('schemaRestorer')
if (!conf.get('databaseCredentials')) {
    conf.set('databaseCredentials', await inquirer.askDatabaseCredentials())
} else {
    const db = conf.get('databaseCredentials')
    console.log(`Using these postgres credentials: ${db.host}:${db.port}/${db.database}?user=${db.user}&password=${db.password}`)
    const { confirmDbCredentials } = await inquirer.askConfirmDatabaseCredentials()
    if (confirmDbCredentials === false) {
        conf.set('databaseCredentials', await inquirer.askDatabaseCredentials())
    }
}
const databaseCredentials = conf.get('databaseCredentials')

For that I'm using configstore which simply store my configs into a json file in the npm user directory. I ask the user if he wants to continue with the saved credentials if any.

To drop the existing schema I'm using pg ORM which is really easy to manipulate.

const client = new Client(databaseCredentials)
await client.query(`DROP SCHEMA IF EXISTS ${schemaName} CASCADE`);
await client.end()

And finally to restore the schema I'm using the system pg_restore thank's to the exec command in the npm package child_process
To simplify my code I've encapsulated it into a Promise so that I can await it's result before continuing the program.

await new Promise((resolve, reject) => {
    exec(`pg_restore -d ${databaseCredentials.database} -U ${databaseCredentials.user} -C -w ${backupPath}`, async (error) => {
        if (error !== null) {
            console.error(`Error restoring the backup: ${error}`);
            reject();
            process.exit(1)
        }
        console.info(`Backup '${backupPath}' has been restored...`);
        resolve();
    })
});

If you're interested in reading the code I've made it available on github

Feel free to comment it, correct it, test it etc.. I've only been using it myself on my Ubuntu setup so for sure it needs to be somehow adapted for MacOS and Windows.

You can also follow me on Twitter if you'd like, I share mostly geeky coding stuff: https://twitter.com/m4nu56

Automatiser les tests d’intégration de votre API avec Postman et Newman

Emmanuel Balpe — Wed, 27 May 2020 20:13:19 +0000

Les tests sont une composante primordiale de n'importe quel développement et il est aujourd'hui très commun de retrouver un grand nombre de tests unitaire et de tests d'intégration avec nos projets. Mais il est moins fréquent de retrouver des tests qui s'exécutent directement sur l'application déployée. Avec les outils dont nous disposons à ce jour il est désormais très simple de mettre en place une simple batterie de tests qui permettent de reproduire un scénario d'utilisation de nos APIs et surtout de les automatiser afin qu'ils fassent partie intégrante de la CI de vos projets.

Création de la collection de requêtes avec Postman

Commencez par créer une nouvelle collection avec Postman. Vous pouvez ensuite organiser vos différentes requêtes en dossier si vous le souhaitez.

Tip: vous pouvez ajouter un token d'authentification global pour toutes vos requêtes depuis la popup d'édition de la collection (ou d'un dossier). Il suffit ensuite de préciser dans l'édition de chaque requête qu'il doit Inherit authentication from parent.

Tip: Vous pouvez créer des environnements afin de pouvoir facilement tester votre api en local et sur le serveur d'intégration ou de production par exemple:

Cliquez sur Manage environment sur l'icône en haut à droite de Postman
Créer les environnements que vous voulez
Ajouter des variables d'environnement tel que host qui pourra être http://localhost:3000 en local et https://integ.mydomain.com pour votre serveur d'intégration.
Vous pouvez utiliser ces variables n'importe où dans l'écriture de vos requêtes en utilisant la syntaxe {{host}}

L'écriture des requêtes est ensuite très simple vous trouverez plus d'information pour les écrire depuis la documentation officielle de Postman: postman documentation

L'écriture des tests avec Postman

Vous aurez remarqué un onglet Tests qui est présent dans la vue requête. Les tests que vous écrivez sont exécutés à chaque fois que vous lancez les requêtes après que la réponse ait été reçue.

L'écriture des tests se fait en javascript, vous trouverez des snippets très pratique dans un bloc à droite de l'éditeur.

La syntaxe d'écriture des tests avec Postman est très verbeuse et intuitive. Elle ressemble beaucoup aux librairies d'assertion communément utilisées en javascript. On peut facilement tester le statut de la réponse de cette manière:

On peut aussi tester les données reçues de cette manière:

Là où ça devient très intéressant c'est avec la possibilité de stocker des variables d'environnements qui sont conservées entre les requêtes. On voit qu'ici on a conservé la propriété _id reçu de la réponse dans une variable buildingId. De la même manière que pour la variable host mis en place avant on peut utiliser cette variable dans les autres requêtes {{buildingId}}.

Et on peut également écrire un scénario d'exécution en précisant la requête à exécuter ensuite à la fin des tests:

La requête qui sera exécutée en suivant est celle dont le nom est /buildings GET.

Tip: Vous pouvez rappeler en boucle une même requête en mettant le nom de la même requête et en conditionnant l'appel d'une nouvelle requête en fonction de la réponse.

Vous pouvez ensuite lancer l'exécution de tous les tests en cliquant sur la collection > "Collection runner"

Vous pouvez sélectionner l'environnement avec lequel lancer les tests et le nombre d'itération que vous voulez lancer.
Si vous cliquer sur "Keep variable values" les variables d'environnement créées pendant l'exécution des tests seront conservées. C'est très util pour débugger les tests en erreur.

Export des tests et exécution en CLI

Vous allez pouvoir lancer l'exécution de votre collection en command line grâce à un outil développé par Postman appelé (Newman)[https://www.npmjs.com/package/newman]
Vous pouvez l'installer dans votre projet avec npm s'il s'agit d'un projet qui gère ses dépendances de cette manière:

npm i newman --save-dev

Vous pouvez exporter votre collection en cliquant sur la collection > Export

Vous aurez ensuite un fichier .json qui pourra être exécuté avec Newman.
Vous aurez également besoin d'exporter vos environnements:

L'exécution des tests avec newman se fait ensuite simplement avec:

newman run MyCollection.postman_collection.json -environment localhost.postman_environment.json

Vous pouvez ensuite vous simplifier la vie en ajoutant un script au package.json de votre projet:

Afin de pouvoir lancer les tests simplement:

npm run integration-tests-local

Intégration dans la pipeline Gitlab

L'intérêt de ces tests d'intégration est de pouvoir les exécuter de manière automatique après chaque déploiement afin de pouvoir être alerté au plus vite d'une possible régression et pouvoir revenir à une version précédente ou fixer le problème.
Avec gitlab le plus simple est d'utiliser l'image officielle de Postman qui a déjà newman d'installée: postman/newman_alpine33
Exemple d'un nouveau stage d'intégration sur notre pipeline:

Tip: Nous utilisons gitlab en installation docker et la gestion des certificats est un vrai enfer du coup je n'ai jamais réussi à lancer les tests de ma collection qui attaquaient le serveur d'intégration non SSL. J'ai fini par mettre un certificat letsencrypt afin d'avoir une url en HTTPS.

Tip: Nous exécutons ces tests de manière décalée de 5 minutes car notre procédure de déploiement est asynchrone et je n'ai pas de retour dans gitlab lorsque c'est terminé. Si vous utilisez une autre solution vous pouvez les lancer directement après le déploiement.

Exécution périodique

Nous utilisons ici la fonctionnalité de Scheduled Pipeline de Gitlab et afin de ne pas relancer le build et deploy des images on peut simplement ajouter une règle d'exclusion sur les autres stages de la pipeline:

Puis dans Gitlab:

Et de cette manière seul la partie test d'intégration est exécutée lors de l'exécution de cette pipeline programmée:

J'espère que cet article vous aura plu et qu'il vous aura donné envie de vous intéresser à mettre en place ce genre de scénario de test.

Prettier to format your Java codebase

Emmanuel Balpe — Wed, 27 May 2020 20:12:58 +0000

So we've all been using Prettier in our JavaScript projects or at least have heard about it.

Prettier is an opinionated code formatter that you can plug to your IDE or in a pre-commit hook to automatically format your code.

Optionated because it comes with formatting rules by default and very few options. Although their seems to be a lot more today than in my memory..

Prettier can also format a lot of other languages than JavaScript:

JSX
Flow
TypeScript
JSON
HTML
Vue
Angular
CSS
Less
SCSS
Markdown
MDX (used in this blog btw)

And some community plugins are available for other programming languages like PHP, PostgreSQL.. and Java!

Prettier-java plugin

The community developped a plugin for java that uses node under the hood to format your java classes.
Plugin is available on github.

It's really easy to manipulate:



npm install -g prettier prettier-plugin-java
prettier --write "**/*.java"

But that's not very convenient. We don't want to ask every developer that will work on the codebase to install the plugin and run it itself.

Maven plugin for the prettier-java plugin

Thanksfully a maven plugin has been developped by HubSpot. It's also available on github.

The plugin comes bundled with node, prettier and the prettier-java plugin so developers don't have to install it themselves and it can be executed inside a maven goal.

In your pom.xml config file simply add the following plugin:



<build>
  <plugins>
    <plugin>
      <groupId>com.hubspot.maven.plugins</groupId>
      <artifactId>prettier-maven-plugin</artifactId>
      <version>0.7</version>
      <configuration>
        <printWidth>90</printWidth>
        <tabWidth>2</tabWidth>
        <useTabs>false</useTabs>
        <ignoreConfigFile>true</ignoreConfigFile>
        <ignoreEditorConfig>true</ignoreEditorConfig>
      </configuration>
      <executions>
        <execution>
          <phase>validate</phase>
          <goals>
            <goal>write</goal>
          </goals>
        </execution>
      </executions>      
    </plugin>
  </plugins>
</build>

So that each time the validate phase is run the plugin will be executed in write mode and will reformat all the project files.

You can also execute the plugin on the command line if you are in that sort of things:
mvn prettier:check or mvn prettier:write

Configuration

We've only played with the 4 following configuration options:

printWidth = 120
tabWidth = 4
useTabs = true
endOfLine = crlf

The other options available seem to not apply for Java language.

We're using a lot of enums with attributes in our projets and prettier formatting for those is terrible so I suggest you encapsulate the code you don't want prettier to format with the comments:

// formatter:off

and

// formatter:on

Like so:



public enum Field implements FieldDb {

    // @formatter:off

  ID(        "id",           Types.BIGINT,    true,  MODE_SEL_INSERT),

  ID_COMPTE( "id_compte",    Types.BIGINT,    true,  MODE_SEL_INS_UP),

  ID_CENTRE( "id_centre",    Types.BIGINT,    false, MODE_SEL_INS_UP),

    // @formatter:on

  ...

Last step

The last step would be to execute the mvn prettier:write command on a git pre-commit hook. For now we haven't configured it on our projects, when so I'll update the post with our procedure.