DEV Community: Daniane P. Gomes

Crossplane: esteróides para o Kubernetes

Daniane P. Gomes — Sun, 12 Jan 2025 10:55:59 +0000

Esses dias assisti um vídeo do sempre sensacional @badtux_ sobre Crossplane e fiquei com vontade de escrever um pouco sobre o assunto. 😁

🧵

Afinal o que é o Crossplane mesmo?

Na minha definição grosseira, é um anabolizante pro Kubernetes que permite administrar serviços na nuvem (AWS, Azure ou GCP) de dentro de um k8s cluster. Tudo isso usando só os queridos YMLs e comandos kubectl.

Mas traduzindo livremente do site oficial crossplane.io, tem uma definição mais bonita “Crossplane é um framework de código aberto de ‘control plane’ mantido pela comunidade cloud-native.”

E o que significa “control plane”?

“Control plane” e “data plane” são termos técnicos de rede, mas esses nomes também são usados no contexto de serviços na nuvem.

“Control plane” é o que trata na configuração ou modificação de recursos, enquanto “data plane” trata da entrega desses recursos.

Sendo assim, o Crossplane pode, por exemplo, provisionar um novo RDS, ou então apagar um bucket no S3, ou ainda alterar o tamanho de uma instância do Opensearch, apenas aplicando um manifesto.

Se é integrado com o ecossistema do k8s, então ele ainda precisa ser instalado em cluster que já existe, certo? Certo.

Tá, mas então qual é a vantagem disso? Porque não usar Terraform pra tudo em vez de ter uma ferramenta extra?

Com Crossplane não é necessário aprender uma nova “linguagem” como o HCL, tudo é feito em yml, então as configurações são fáceis de ler.

Integrado com as melhores práticas de GitOps, a infraestrutura provisionada se auto repara assim que detecta uma diferença entre a fonte de verdade (um repositório git onde são armazenados os yml) e os serviços que estão rodando.

Outra coisa legal do Crossplane é que dá pra criar abtrações pra estrutura dos recursos e limitar o que ou não ser feito.

Por exemplo, um time de plataforma define que instâncias de RDS só podem ser provisionadas na versão x ou y e apenas com classes “small”.

O time de aplicações vair poder criar/alterar instâncias sozinho sem o risco de usar configurações não permitidas.

Ou ainda, uma KMS pode ser automaticamente criada, simplifcando as configurações exigidas do time de aplicações.

Isso é possível através de Compositions, Composite Resource Definitions e Claims. Mas isso é assunto pra uma próxima thread. Quem sabe…

Vídeo da Linuxtips https://www.youtube.com/watch?v=Z9wXnpx9PHM

Automatize Documentation on GitLab

Daniane P. Gomes — Tue, 14 Jul 2020 21:51:34 +0000

Using Java and GitLab4J

Photo by Annie Spratt on Unsplash

GitLab is a well-known platform that allows a team to manage projects, to create documentation, pipelines, repositories and more.

It can be used to manage the project’s Sprints and to replace other tools such as Trello or Jira.

When utilized from “end-to-end” during the development cycle, it allows some automatizations, such as to generate documentation or a changelog based on requirements collected and developed during the process.

This article aims to demonstrate how to use the project’s issues or User Stories to generate a changelog, a Wiki page or “readme” file that can be delivered to costumers automatically.

Technical stack

Java 11
GitLab4J

GitLab Structure

We have a milestone that is “Sprint 1” and it has issues associated with it, as you can check here.
We have a README.md file representing a changelog file that will be packed with the application and deployed somewhere, through the pipeline. This file would be mapped to an URL inside the application and it would be available to the costumers through a link.

Milestone Sprint 1

Sprint 1 and its issues

README.md file representing the changelog file visible to the customer

Setup the integration

On your GitLab profile, go to “Settings” and then “Access Tokens”. Generate an access token with all the checkboxes checked. Click on “Create personal access token” and copy the value of the field “Your new personal access token”.

This token will be used to authenticate on GitLab through a Java application using the library GitLab4J.

Generate an access token on GitLab

Generated access token

Hands-on

Using GitLab4J we will retrieve the latest milestone (it should be the first on the list), read its issues, labels and notes and if they are closed, we will create a new commit on README.md file and a Wiki page.

Let’s inspect the code on my GitHub project that does that.

Method “generateDocumentation” reads the milestones and its issues.
Method “getChangelogMessagesFromIssues” reads the issues and its details.
Method “printIssuesLabels” just print issues’ labels. But it could be used for some filtering, for example, only issues with label “public” should be exported to the documentation.
Method “printIssuesNotes” retrieves all the comments related to the issue to add them to documentation.
Method “createCommit” commits to the repository.

Results

After running the code, a Wiki page is created, as can be observed on the image below.

Wiki page with Sprint’s and issues’ details

Also, a new commit was made and the README.md file was updated.

Automatized commit

Commit content

It is possible to read commit messages and add them to the documentation if needed.

Conclusions

When GitLab is used to control Sprints and project’s requirements, it is possible to retrieve information from it and automatize the documentation only by reading the issues’ information.

If the User Stories or issues are well organized and written, no extra effort is needed to achieve this goal.

A simple Java code and GitLab4J library can digest information and thus, a project’s documentation can be automated.

Building a continuous delivery pipeline for database migrations with GitLab and AWS

Daniane P. Gomes — Wed, 29 Apr 2020 19:06:04 +0000

Photo by JJ Ying on Unsplash

After having tested tools to automate the database migrations, it is time to integrate the chosen one with my GitLab repository and build a continuous delivery pipeline for AWS.

Project stack

You can check my previous story about a comparison between Flyway and Liquibase here, but spoiler alert this implementation has the following stack:

Flyway manage database migrations.
MySQL database.
Flyway Docker image.
GitLab CI/CD.
Amazon Web Services (AWS).
Check the repository here.

The process

GitLab CI builds a Flyway Docker image and pushes it to Amazon Elastic Container Registry (ECR).
GitLab CI triggers a lambda that runs an Amazon Elastic Container Service (ECS) task with the Flyway Docker image from ECR.
The Flyway command “migrate” is executed and the database schema is updated.

The image below illustrates the process.

The continuous delivery pipeline process

GitLab CI

A demo project has the folder db-migrations/scripts where migration scripts are placed. Every time a change is pushed to this folder on GitLab repository, the pipeline will run, build a Flyway Docker image with the scripts and push it to the Amazon Elastic Container Registry (ECR).

Additionally, GitLab CI triggers a lambda that calls an Amazon Elastic Container Service (ECS) task which will run the built image.

The image details are on the Dockerfile below.

# Get image "flyway" from Flyway's repository
FROM flyway/flyway

WORKDIR /flyway 

# Database credentials
COPY db-migrations/flyway.conf /flyway/conf

# Add the scripts I've pushed to my project folder to the Docker image
ADD db-migrations/scripts /flyway/sql

# Execute the command migrate
CMD [ "migrate" ]

The following .gitlab-ci.yml shows GitLab’s actions. Check stages “build-docker-image” and “execute-migrations”.

stages:
  - build-docker-image
  - execute-migrations

build-docker-image:
  stage: build-docker-image
  image:
    name: gcr.io/kaniko-project/executor:debug-v0.19.0
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json    
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/db-migrations/Dockerfile --destination $AWS_REPOS_FLYWAY:latest
  only:
    refs:
      - <MY_BRANCH_ON_GITLAB>
    changes:
      - db-migrations/scripts/*

execute-migrations:
  stage: execute-migrations
  image: python:3.8-alpine
  before_script:
    - apk add --no-cache python3
    - python3 -m pip install awscli
  script:
    - aws lambda invoke --function-name MyLambdaToTriggerFlyway response.json
  only:
    refs:
      - <MY_BRANCH_ON_GITLAB>
    changes:
      - db-migrations/scripts/*

Lambda

The lambda is not strictly required: the same results could be achieved using the CLI.

However, a lambda offers more flexibility to the process. It is possible to get the execution results, send emails, feed a database table with information to collect statistics and everything else your imagination allows.

Also, it keeps the infrastructure control through the code and its versions.

The lambda was written in Node.js 12.x and I have reused the code I wrote for another test. The task is triggered on line 47 with the command “ecs.runTask(params)”.

/**
 * MyLambdaToTriggerFlyway
 *
 * This lambda relies on 3 environment variables: ENV_CLUSTER, ENV_SUBNET, ENV_SECURITY_GROUP.
 * 
 */

var aws = require('aws-sdk');
var ecs = new aws.ECS();

exports.handler = async (event, context) => {

    var taskDefinition = null;

    var CLUSTER = process.env.ENV_CLUSTER;
    var SUBNET = process.env.ENV_SUBNET.split(",");
    var SECURITY_GROUP = process.env.ENV_SECURITY_GROUP;
    var LAUNCH_TYPE = "FARGATE";
    var FAMILY_PREFIX = "flyway";
    var CONTAINER_NAME = "flyway";  

    var taskParams = {
        familyPrefix: FAMILY_PREFIX
    };    

    const listTaskDefinitionsResult = await ecs.listTaskDefinitions(taskParams).promise();

    if(listTaskDefinitionsResult) {
        taskDefinition = listTaskDefinitionsResult.taskDefinitionArns[listTaskDefinitionsResult.taskDefinitionArns.length-1];
        taskDefinition = taskDefinition.split("/")[1];
    }

    var params = {
        cluster: CLUSTER,
        count: 1, 
        launchType: LAUNCH_TYPE,
        networkConfiguration: {
            "awsvpcConfiguration":  {
              "subnets": SUBNET,
              "securityGroups": [SECURITY_GROUP]
            }
        },
        taskDefinition: taskDefinition,

    };

    const runTaskResult = await ecs.runTask(params).promise();

    if (runTaskResult.failures && runTaskResult.failures.length > 0) {
        console.log("Error!");
        console.log(runTaskResult.failures);
    }

    return runTaskResult;

};

Elastic Container Service Task

The ECS task gets the Flyway Docker image and runs it. The command “migrate” will be executed, the scripts will be applied and the database schema will be updated.

In case of errors, I have decided to keep the fix execution manual for now, but it would be possible to automatize the usage of other commands such as “validate” and “repair”.

I have created an alarm on CloudWatch to notify me by email in case of any error. For future implementation, I intent to manage execution errors through the lambda.

Conclusions

It can be painful to manage database migrations manually, especially if we have multiple environments such as development, staging or production.

However, a migration tool like Flyway integrated with a continuous delivery pipeline, avoid manual execution and therefore mitigates human error. Furthermore, it relieves the burden and boredom of the activity.

This article was written in partnership with Elson Neto.

Originally posted on my Medium Stories

Database Migrations with Liquibase and Flyway

Daniane P. Gomes — Wed, 29 Apr 2020 19:02:05 +0000

A comparison between tools

Photo by Aryan Singh on Unsplash

As a software developer, I want to automatize my relational database migrations on test, staging or production environments. To achieve that, I have tested two migration tools widely know on Java universe: Liquibase and Flyway.

The stack behind this test project was:

Spring Boot & Java 11 application.
MySQL database.
Maven plugin to run both Liquibase and Flyway.

Some background

Database changes can be painful and destructive, not only on monoliths but especially on distributed systems. I have found some recommendations about the best practices, such as the article “Evolutionary Database Design” by Martin Fowler and this talk plus book by Edson Yanaga.

Roughly, the important things are:

To keep database scripts versioned with the application code.
To keep backward and forwards compatibility. For example, a table column can not be renamed right away. It is necessary to have intermediate states between this action: add a new column, write in both columns during version 1.x, change the code to read the new column during version 2.x and drop column during version 3.x.

Automate tasks

I do not want all my team to have direct access to test, staging or production databases and also I do not want to execute scripts manually. Liquibase and Flyway will play this role and all I need to do is to write the scripts.

Since I have tested the free version of both tools with MySQL database in a Spring Boot project and Maven plugin, I will summarize their functionalities and why I have chosen one instead of another.

There are plenty of articles focused on Liquibase and Flyway, so I will not extensively talk about them.

You can check these two separated stories for a brief overview:

here is Flyway’s and
here is Liquibase’s.

And the chosen one is…

In my humble opinion, both tools are great and they do the job. However, I decided to go for Flyway.

I do not believe I will ever need the all functionalities that Liquibase offers and it was a little bit more painful to configure and run it than Flyway. Also, I always found XML files kind of messy/ugly. sorry

For XML files, it is necessary to respect a pattern, worry about the mandatory tag databaseChangeLog and its version. I got some errors with the changelog file because I have started with an older version of Liquibase. When I updated it, it was necessary to change the version on XML files.

I know it is not a biggie but there is no need to do that with Flyway.

Also, the configuration (flywayConfig.properties) felt simpler and cleaner than Liquibase’s (liquibase.properties). Perhaps because there are not as many options as on Liquibase. I do not really need all the functionalities that Liquibase offers, so instead of helping they were just creating some noise.

For instance, in the beginning, I was not sure why there were two similar properties: changeLogFile and outputChangeLogFile. This concern did not exist with Flyway.

After having it “installed”, the only thing to do with is to place scripts on the configured folder respecting a sequence for the migration files’ name. No additional worries.

Flyway does the job straight away and it is much more intuitive than Liquibase. It is easier to simply write SQL in a way that everybody is already used to and thus, there is no learning curve.

Flyway is the most “lightweight” and trouble-free solution to automatize the database migrations.

Originally posted on my Medium stories.

Triggering an Amazon ECS from Amazon RDS

Daniane P. Gomes — Wed, 29 Apr 2020 18:59:43 +0000

Photo by chuttersnap on Unsplash

With a few lines of code, it’s possible to trigger an Amazon Elastic Container Service (ECS) from an Amazon Relational Database Service (RDS). This article intends to describe how to do it and the struggles I had while trying to have it working.

For the sake of understanding, I will describe an imaginary system to explain why I want to trigger an ECS task. Let’s pretend user data protection just doesn’t exist.

The company IT Paradise offers a cafeteria to its employees where they can order coffee, snacks, and drinks. Since they are riding the “healthy style” wave, they decided to follow the employee’s consumption habits.

IT Paradise implemented a system that tracks the amount of coffee and snacks an employee takes a day. So now, every time someone goes to the Cafeteria and swipe their card, the Cafeteria’s system “A” inserts the consumed items on a database table, which triggers a lambda to run an ECS task.

The ECS task starts the application “B” that will process the information about the employee intakes and feed the employee’s profile on a system “C”. When everything is done, system “B” stops itself.

Imaginary system’s flow

The database

The database is an Aurora MySQL and it will need the permissions as described on AWS documentation.

We create a trigger on the table that holds the purchases of the employee. Check line 9 for the piece of code that calls the lambda.

CREATE TRIGGER tasks_trigger
  AFTER INSERT ON employee_purchases
  FOR EACH ROW
BEGIN

    SELECT NEW.taskType INTO @taskType;

    IF @taskType = 'BUY' THEN
        CALL mysql.lambda_async('arn:aws:lambda:eu-west-1:<MY_ACCOUNT_ID>:function:triggerECSLambda', 
                         CONCAT('{"taskType" : "', @taskType, '" }')
             ); 

    END IF;

END

Database trigger to call the lambda

The Lambda Saga

Since I’m a Java enthusiast I naively wrote my lambda in Java 11. However, I faced several problems and gave up in a “NoClassDefFoundError” (line 14).

Besides that, my jar ended up with 150mb, which is a lot for what I needed, and the initialization time was also discouraging.

I leave the Java 11 non-working code below just for the records.

import com.amazonaws.services.ecs.AmazonECS;
import com.amazonaws.services.ecs.AmazonECSClientBuilder;
import com.amazonaws.services.ecs.model.AwsVpcConfiguration;
import com.amazonaws.services.ecs.model.LaunchType;
import com.amazonaws.services.ecs.model.NetworkConfiguration;
import com.amazonaws.services.ecs.model.RunTaskRequest;
import com.amazonaws.services.ecs.model.RunTaskResult;
import com.amazonaws.services.lambda.runtime.Context;

public class Application {

  public String handleRequest(final Object request, final Context context) {

    final AwsVpcConfiguration awsvpcConfiguration = new AwsVpcConfiguration()
        .withSubnets("subnet-<MY_SUBNET>")
        .withSecurityGroups("sg-<MY_SUBGROUP>");

    final NetworkConfiguration networkConfiguration = new NetworkConfiguration()
        .withAwsvpcConfiguration(awsvpcConfiguration);

    final RunTaskRequest runTaskRequest = new RunTaskRequest()
        .withLaunchType(LaunchType.FARGATE)
        .withCluster("<MY_CLUSTER>")
        .withTaskDefinition("<MY_TASK>")
        .withCount(1)
        .withNetworkConfiguration(networkConfiguration);

    final AmazonECS client = AmazonECSClientBuilder.standard().build();
    final RunTaskResult response = client.runTask(runTaskRequest);

    return String.format("Runned: %s.", response);
  }

}

When I finally accepted that it was to much trouble to a simple task, I tested a script in Node.js and it worked smoothly and painlessly. :)

It can be checked below. It has plenty of room to improve, but the important thing to note here is that I always get the last version of the task definition (line 27) and the runTask command (line 56).

var aws = require('aws-sdk');
var ecs = new aws.ECS();

exports.handler = async (event, context) => {

    var taskDefinition = null;

    var CLUSTER = process.env.ENV_CLUSTER;
    var SUBNET = process.env.ENV_SUBNET.split(",");
    var SECURITY_GROUP = process.env.ENV_SECURITY_GROUP;
    var LAUNCH_TYPE = "FARGATE";
    var FAMILY_PREFIX = "process-employee-purchases";
    var CONTAINER_NAME = "process-employee-purchases";
    var PROFILE = "";

    if(event.hasOwnProperty('taskType') && event.taskType == 'BUY') {
      PROFILE = "exportRunner";
    } else {
      PROFILE = "importRunner";
    }

    var taskParams = {
        familyPrefix: FAMILY_PREFIX
    };    

    // Get the last version of the task
    const listTaskDefinitionsResult = await ecs.listTaskDefinitions(taskParams).promise();

    if(listTaskDefinitionsResult) {
        taskDefinition = listTaskDefinitionsResult.taskDefinitionArns[listTaskDefinitionsResult.taskDefinitionArns.length-1];
        taskDefinition = taskDefinition.split("/")[1];
    }

    var params = {
        cluster: CLUSTER,
        count: 1, 
        launchType: LAUNCH_TYPE,
        networkConfiguration: {
            "awsvpcConfiguration":  {
              "subnets": SUBNET,
              "securityGroups": [SECURITY_GROUP]
            }
        },
        taskDefinition: taskDefinition,
        overrides: {
          containerOverrides: [{
            name: CONTAINER_NAME,
            environment: [{
              name: "SPRING_PROFILES_ACTIVE",
              value: PROFILE
            }]
          }]
        }
    };

    const runTaskResult = await ecs.runTask(params).promise();

    if (runTaskResult.failures && runTaskResult.failures.length > 0) {
        console.log("Error!");
        console.log(runTaskResult.failures);
    }

    return runTaskResult;

};

Lambda **working* script (Node.js 12.x)*

Lambda permissions are as below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:eu-west-1:<MY_ACCOUNT_ID>:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:eu-west-1:807210873196:log-group:/aws/lambda/triggerECSLambda:*"
      ]
    }
  ]
}

Basic permission that lambda needs to access **CloudWatch* logs.*

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:RunTask",
        "ecs:ListTaskDefinitions"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "*"
      ],
      "Condition": {
        "StringLike": {
          "iam:PassedToService": "ecs-tasks.amazonaws.com"
        }
      }
    }
  ]
}

Allows the lambda to interact with **ECS* and run tasks.*

Conclusions

As far as I’m aware, it’s not possible to trigger Amazon ECS directly from my Amazon RDS Aurora, thus the need for the lambda.

A lambda can be easily called from the Amazon RDS and then, from the lambda, it is possible to run an Amazon ECS task.

After the tests, I got a little bit disappointed with lambdas in Java on AWS, but it’s incredibly easy to work with lambdas in Node.js.

By taking the approach described, it is possible to have a detached architecture with exclusive containers running only during the necessary amount of time and avoid some “pains” such as schedulers or memory share. Also, it saves money! We don’t have the container always up and running and we will pay only for what we use.

Special thanks to my DevOps Elson Neto that guided me through the AWS permissions maze and my colleague Diego Hordi for suggesting some nice touches.

Originally posted on my Medium page.

An Experiment with User Story Mapping

Daniane P. Gomes — Wed, 29 Apr 2020 18:57:57 +0000

A while ago I attended a Scrum Developer workshop facilitated by Mario Melo where we used User Story Mapping to create backlog stories to an imaginary product.

User What?

User Story Mapping was created by Jeff Patton and it is a way to organize a product backlog that intents to allow the team to see “the big picture”.

To do so, we establish:

A “backbone” that will represent the categories or the big modules;
A narrative flow ordered from left to right that tells the story of what we want to achieve;
Small tasks representing the steps to accomplish the story.

Learn more here or check Jeff Patton’s book.

To understand the technique Mario provided us an example (also created by Jeff Patton) of narrative flow for a person getting ready to go to work. I tried to reproduce it in the image below.

We have the categories in blue representing the backbone of our story and the narrative flow in yellow.

To complete the story “Go to work” we have the MVP with the tasks in pink. Imagine that the MVP is what is done in a day when the user Dani wakes up late.

The remaining activities in purple on the backlog are the desired, but not mandatory activities. We can think of them as something that is executed only during the days when the user wakes up early (or during the next Sprints).

The advantages of it towards a flat backlog are:

The story is organized in a logical flow and can be seen as a whole.
Every member of the team can get involved.
Prioritization is visual.
Describes a journey to accomplish a goal.

I want it for my project!

The activity went so well during the workshop that I suggested the technique to my team since we have just started a brand new exciting system.

My Scrum Master delegated the responsibility to conduct the session to me and a colleague. I have never done something similar before, but I enjoy testing my anxiety limits.

After a little bit of reading about the subject and counting on Mario’s extra advises, the plan was:

Establish a common dictionary of terms with the Development Team, the Project Owner and the Test Team.
Identify possible users.
Discuss and map 3 user stories.
Every member of the team would write the activities to every user story in a “Post-It”. In the end, we would join them and discuss if they made sense or not.
We should use a timer and keep it visible to everybody.
We should have background music while people were writing. :)
Perform it in a 2-hours-meeting.

And how did it go?

In the beginning, the team seemed a little bit confused, but soon most of them started getting involved and collaborating. Additionally, we almost have won against attention leechers a.k.a. mobile phones and parallel meetings.

I struggled, however, controlling the time and the session. Since the discussions were flowing, I did not dare to interrupt. But because of that, we could not keep the quality of the discussion until the end.

I also had a hard time trying to keep things on track. I was just a software developer trying to facilitate a session and I did not feel that I had the authority to stop some discussions and simply move on. I did not want to be “bossy” or to give the idea that some member’s opinions were less valuable than others’.

After the session, I have conducted a survey to collect the team’s feedback and the answers were pretty encouraging. Between some complaints, there were positive comments and the overall opinion was that the session was essential to put every team member on the same page and to clarify what is, in fact, this new system that we are about to create.

Lessons learned

If I would like to conduct a similar session again? Yes! Nevertheless, I would try to make some things differently:

Control the time! Time flies fast when there are many opinions to be heard.
Make sure that everybody understands that some discussions might get stopped before coming to an end. The focus should not be compromised.
Split the team into groups to think about the user stories. It is easier to merge all ideas afterward and it seems to me that it would encourage collaboration rather than a debate about who is right.

It is rewarding — and fun — to try distinct ways to do the same stuff and, despite the “setbacks” and some heated discussions, it was delightful to see so many members of the team with such a great extent of engagement for that amount of time.

Originally posted on my Medium page.

Java 14 Records :: did we live to see getters and setters die?

Daniane P. Gomes — Mon, 13 Apr 2020 21:48:57 +0000

Is it the end of the line for getters, setters and Lombok project?

Photo by Mike Kenneally on Unsplash

If you try to teach Java to a millennium or to someone coming from another less verbose language, you can often face resistance regarding the writing of getters, setters, toString, equals and hashCode methods and the need of so much boilerplate code.

Vintage developers will not bother about it. We simply got used to instructing the IDE to generate them for us.

But let’s face it: we don’t particularly like them. We just got used to them. We accepted the need for these methods, but to be honest: it could be different.

Java 14 has launched records: a preview feature for a new type that dismisses the necessity of getters, setters, toString, equals and hashCode methods. It is possible to create an object with a few lines of code.

How does it work?

Since it is yet a preview feature, we should enable preview features on our compiler, IDE and/or Maven.

In summary records have the following characteristics:

They are final and immutable.
They can implement interfaces.
They can have static members.
They can define validations.
They can define default values.
They accept generics.

I have tested it and the project is available on my GitHub.

All passed! \o/

Final and Immutable

Observe that records do not have the classic getters and setters and therefore, it is impossible to modify their values once they are assigned.

Note on the lines 34 to 38 below that the reading of attributes, is done without the “get” prefix. We use the attribute’s name straight forward.

import static org.junit.Assert.assertTrue;

import java.time.LocalDateTime;
import java.time.Month;
import java.util.Objects;

import org.junit.Test;

import com.danianepg.previewfeature.data.NationalHoliday;
import com.danianepg.previewfeature.data.records.CelebrationGenericRecord;
import com.danianepg.previewfeature.data.records.SpecialDate;

/**
 * Test class to verify the records' functionalities.
 * @author Daniane P. Gomes
 *
 */
public class RecordsDemoTest {

    private String name = "My Bday";
    private Integer day = 20;
    private Month month = Month.OCTOBER;
    private LocalDateTime created = LocalDateTime.now();

    private String country = "Brazil";
    private String nationalHolidayName = "Independence Day";
    private Integer nationalHolidayDay = 7;
    private Month nationalHolidayMonth = Month.SEPTEMBER;

    @Test
    public void specialDateRecord_ok() {

        SpecialDate myBday = new SpecialDate(name, day, month, created);
        boolean isNameEquals = myBday.name().equals(name);
        boolean isDayEquals = myBday.day() == day;
        boolean isMonthEquals = myBday.month() == month;
        boolean isCreatedEquals = myBday.created().equals(created);

        assertTrue(isNameEquals && isDayEquals && isMonthEquals && isCreatedEquals);
    }

    @Test
    public void specialDateRecord_equals_ok() {
        SpecialDate myBday = new SpecialDate(name, day, month, created);
        SpecialDate myBdayCopy = new SpecialDate(name, day, month, created);
        assertTrue(myBday.equals(myBdayCopy));
    }

    @Test
    public void specialDateRecord_alternativeConstructor() {
        SpecialDate myBday = new SpecialDate(name, day, month);
        boolean isCreatedNotNull = !Objects.isNull(myBday.created());
        assertTrue(isCreatedNotNull);
    }


    @Test(expected = IllegalArgumentException.class)
    public void specialDateRecord_exceptionWhenDayOutOfTheRange() {
        try {
            new SpecialDate(name, 32, month);
        } catch (IllegalArgumentException e) {
            System.out.println("Message exception: "+e.getMessage());
            throw e;
        }
    }

    @Test
    public void celebrationGenericRecord_ok() {

        CelebrationGenericRecord<NationalHoliday> dateGenericClassic = new CelebrationGenericRecord<>(
                new NationalHoliday(country), nationalHolidayName, nationalHolidayDay, nationalHolidayMonth);

        boolean isInstanceOfNationalHoliday = dateGenericClassic.contents() instanceof NationalHoliday;
        boolean isCountryEquals = false;

        if (isInstanceOfNationalHoliday) {
            NationalHoliday dateClassic = (NationalHoliday) dateGenericClassic.contents();
            isCountryEquals = dateClassic.getCountry().equals(country);
        }

        assertTrue(isInstanceOfNationalHoliday && isCountryEquals);

    }

}

Interfaces, Static Members, Validations, Default values

Records are compatible with our classic interfaces. Check the record “SpecialDate” that implements the interface “CelebrationInterface” and therefore overwrites the method “totalDates()”. Observe lines 13 and 48.

For an extra static member check line 18.

For a validation check line 28.

For a default value, check the alternative constructor on line 41, which assigns the current date and time for the created object, even if it is not informed during the record’s object creation.

package com.danianepg.previewfeature.data.records;

import java.time.LocalDateTime;
import java.time.Month;

import com.danianepg.previewfeature.interfaces.CelebrationInterface;

/**
 * An example of a record implementing an interface, validating data, passing default values and using extra static attributes and methods.
 * @author Daniane P. Gomes
 *
 */
public record SpecialDate(String name, Integer day, Month month, LocalDateTime created) implements CelebrationInterface {

    /**
     * Additional static members
     */
    private static int totalDates;

    /**
     * Define validations for the attributtes
     * @param name
     * @param day
     * @param month
     * @param created
     */
    public SpecialDate {
        if (day < 1 || day > 31) {
            throw new IllegalArgumentException("Day must be on the interval 1-31.");
        }

        totalDates++;
    }

    /**
     * Additional constructor, to assign a default value to attribute "created"
     * @param name
     * @param day
     * @param month
     */
    public SpecialDate(String name, Integer day, Month month) {
        this(name, day, month, LocalDateTime.now());
    }

    /**
     * Additional public method to retrieve 
     */
    public int totalDates() {
        return totalDates;
    }

}

Generics

Records are flexible and accept generics! The definition can be found below and the and example on how to use it is available on the test “celebrationGenericRecord_ok()” of class “RecordsDemoTest”.

package com.danianepg.previewfeature.data.records;

import java.time.Month;

/**
 * Generic record
 * @author Daniane P. Gomes
 *
 * @param <T>
 */
public record CelebrationGenericRecord<T>(T contents, String name, Integer day, Month month) {
}

Why is this useful?

Short answer: to carrier data.

Long answer: now we can declare a simple data transfer object with a few lines of code and without the need to satisfy all the ceremonies that we are used to having in Java, meaning CLEAN code!

Since records are compatible with other Java’s important features, I am really excited to start to use. It gives me much satisfaction to remove extra code on my projects.

Conclusion

Well then… is it the end of the line for our beloved-hateful pet methods getters and setters? Hmmm… Apparently not quite yet.

Since records are immutable they cannot simply replace our classic classes. Also, another concern to be addressed is how compatible are they with frameworks such as Hibernate and Spring?

They will, however, help us to make code cleaner and smaller and in some cases, they could even eliminate the need of external libraries such as Lombok, as stated Ben Evans for Java Magazine:

“This will help many applications make domain classes clearer and smaller. It will also help teams eliminate many hand-coded implementations of the underlying pattern and reduce or remove the need for libraries like Lombok.”

It is important to keep in mind though, that this is a preview feature and as the documentation reminds:

“Preview features may be removed in a future release, or upgraded to permanent features of the Java language.”

I hope they keep it. Fingers crossed.

References

JEP 359: Records (Preview)

Record (Java SE 14 & JDK 14)

Records Come to Java

Originally posted on my Medium page.

Clean Validation in Java with Predicates

Daniane P. Gomes — Mon, 13 Apr 2020 08:19:26 +0000

Imagine that you have to consume an API to retrieve data from people of your company. Now imagine that all these data don’t follow any pattern and the API can return not only people, but robots, “phantom” accounts and all the source of irrelevant information. There are no rules: no flag to identify if the data belongs to a person or to some other creature and from time to time you can discover another variation that would classify the data as invalid.

Well, that happened. The validation could be achieved with “regex”, but it would be hard coded and the customer would always depend on the change in the code and new deploys.

Aha!

The most efficient and clean way found to do that it in Java was to create a table to save the rules that would configure a record as invalid, read and convert them to Predicates and dynamically validate each part of the API’s return to classify an object as valid or invalid.

Show me the code

This behaviour was reproduced on the project available on my GitHub, using Java 11 and Spring Boot.

Object representation

The external API’s data is represented by the class PersonDTO.
The rules that define a PersonDTO as invalid are represented and persisted through entity ExclusionRule where:

fieldName is the attribute on PersonDTO that will be checked.
operator is an operator AND or OR.
comparator is a comparator EQUALS or CONTAINS.
ruleValues are the values separated by comma that would make the fieldName invalid.

Interpret rules

The resource data.sql will initialize some rules for the purpose of this test:

INSERT INTO exclusion_rule(field_name, comparator, operator, rule_values) VALUES('name', 'CONTAINS', 'OR', '1,2,3,4,5,6,7,8,9,0');
INSERT INTO exclusion_rule(field_name, comparator, operator, rule_values) VALUES('email', 'CONTAINS', 'OR','@exclude.me,1');
INSERT INTO exclusion_rule(field_name, comparator, operator, rule_values) VALUES('internalCode', 'CONTAINS', 'AND','a,b');
INSERT INTO exclusion_rule(field_name, comparator, operator, rule_values) VALUES('location', 'EQUALS', 'OR','jupiter,mars');

The rules above can be interpreted as:

If the attribute name on PersonDTO object contains 1, 2, 3, 4, 5, 6, 7, 8, 9 or 0, the object is invalid.
If the attribute email on PersonDTO object contains “@exclude.me” or “1”, the object is invalid.
If the attribute internalCode on PersonDTO object contains “a” and “b”, the object is invalid.
If the attribute location on PersonDTO object is equals to “jupiter” or “mars”, the object is invalid.

Using Predicates

For each possible combination of operators and comparators a validation class was created (RuleContainsAnd, RuleContainsOr and RuleEqualsOr). By implementing the interface Predicate<T> those classes can be used to validate an object through the simple and elegant call of test(myFieldValue) . It is only necessary to overwrite test method and define a custom rule.

public class RuleEqualsOr implements Predicate<String> {

  private List<String> exclusionRulesLst;

  public RuleEqualsOr(final List<String> exclusionRulesLst) {
    this.exclusionRulesLst = exclusionRulesLst;
  }

  @Override
  public boolean test(final String fieldValue) {
    return this.exclusionRulesLst.stream().anyMatch(fieldValue::equals);
  }

}

Class ExclusionRuleService is the responsible to retrieve saved rules, transform them to its corresponding Predicate and keep them in a list.

/**
   * Retrieve all rules from the database and process it.
   *
   * @return
   */
  private Map<String, Predicate<String>> decodeAllRules() {
    // @formatter:off
    return this.validationRuleRepository.findAll()
        .stream()
        .map(this::deconeOneRule)
        .collect(Collectors.toMap(PairDTO::getRule, PairDTO::getPredicate));
    // @formatter:on

  }

  /**
   * According to the rule configuration, create a Predicate.
   *
   * @param validationRule
   * @return
   */
  private PairDTO deconeOneRule(final ExclusionRule validationRule) {

    PairDTO pairDTO = null;
    List<String> values = new ArrayList<>();

    if (validationRule.getRuleValues().contains(",")) {
      values = Arrays.asList(validationRule.getRuleValues().split(","));
    } else {
      values.add(validationRule.getRuleValues());
    }

    if (validationRule.getComparator() == ComparatorEnum.EQUALS && validationRule.getOperator() == OperatorEnum.OR) {
      pairDTO = new PairDTO(validationRule.getFieldName(), new RuleEqualsOr(values));

    } else {

      if (validationRule.getOperator() == OperatorEnum.OR) {
        pairDTO = new PairDTO(validationRule.getFieldName(), new RuleContainsOr(values));
      } else {
        pairDTO = new PairDTO(validationRule.getFieldName(), new RuleContainsAnd(values));
      }

    }

    return pairDTO;

  }

Where the magic lives

Now that all the validation “bed” is done, it is possible to use methods filterAllValid and isInvalid to receive an object or a list and pass them to isInvalidTestPredicate. On this last method we get the field of the class PersonDTO that matches the defined on ExclusionRule and its value using Reflections.

It is important to be aware that the heavy use of Reflections can cause performance issues, but on this particular situation I’ve considered that some performance could be sacrificed to achieve the flexibility of the validation.

The magic happens when the method testis called. No additional test is required.

/**
   * Retrieve the person's object fields by reflection and test its validity.
   *
   * @param person
   * @param entry
   * @return
   */
  private Boolean isInvalidTestPredicate(final PersonDTO person, final Entry<String, Predicate<String>> entry) {

    final Field field = this.reflectionService.getFieldByName(person, entry.getKey());
    final String fieldValue = String.valueOf(this.reflectionService.getFieldValue(person, field));

    return entry.getValue().test(fieldValue);

  }

  /**
   * Verify if a person is invalid if it fails on any determined rule.
   *
   * @param person
   * @return
   */
  public Boolean isInvalid(final PersonDTO person) {
    return exclusionRulesLst.entrySet().stream().anyMatch(e -> this.isInvalidTestPredicate(person, e));
  }

  /**
   * Get only valid objects from a list
   *
   * @param personDTOLst
   * @return
   */
  public List<PersonDTO> filterAllValid(final List<PersonDTO> personDTOLst) {
    // @formatter:off
    return personDTOLst.stream()
              .filter(person -> !this.isInvalid(person))
              .collect(Collectors.toList());
    // @formatter:on
  }

Test me

On class ExclusionRulesServiceTests we can check if the rules are being properly applied to the fields of a PersonDTO object.

@Test
  public void filterAllValidPersonLstNameContainsOr_ok() {

    final PersonDTO person = new PersonDTO();
    person.setName("Daniane P. Gomes");
    person.setEmail("danianepg@gmail.com");
    person.setInternalCode("DPG001");
    person.setCompany("ACME");
    person.setLocation("BR");

    final PersonDTO person2 = new PersonDTO();
    person2.setName("Dobberius Louis The Free Elf");
    person2.setEmail("dobby@free.com");
    person2.setInternalCode("DLTFE");
    person2.setCompany("Self Employed");
    person2.setLocation("HG");

    final List<PersonDTO> personLst = new ArrayList<>();
    personLst.add(person);
    personLst.add(person2);

    final List<PersonDTO> personValidLst = this.exclusionRuleService.filterAllValid(personLst);

    assertEquals(personValidLst.size(), 2);

  }

Conclusion

While consuming an external API we can receive data that is not properly structured. To check its relevance in a clean way we can:

Create a repository of rules and represent them as Predicate<T>
Convert the API response data to a PersonDTO object
Check if each attribute of PersonDTO is valid only by calling the method test

Originally posted on my Medium page.