DEV Community: Wojtek Krzywiec

Java Series: Flatmap

Wojtek Krzywiec — Sun, 01 Jan 2023 17:38:44 +0000

Java 8 was a great step forward toward modern programing language. One of the key features added in this release was Java streams. It provides lots of convenient operations for data processing. One of them is a flatMap() used very widely to unwrap and merge multiple collections into one.

This article is part of the "Java Series", which covers useful Java functions from standard and popular libraries. More posts on that can be found here on Dev.to or on my home page.

Problem statement

Many times when we work with Java code we end up with a following Plain Old Java Object (POJO):

public record Parent(List<Child> childs) {}

They can represent database entities or data transfer objects (DTOs). In general, they're used to structure data. Let's say that we got the list of Parent objects, but we want to operate on a list of all Child that are part of the Parent. How we could extract Child objects from all Parent objects and combine them into a single list? The naive approach would be to use a loop:

List<Child> children = new ArrayList<>();

for (Parent parent: parents) {
    List<Child> bs = parent.childs();
    children.addAll(bs);
}

But it doesn't look nice and clean. Instead, we could use the Java stream:

List<Child> children = new ArrayList<>();

parents.stream()
    .map(parent -> parent.Childs())
    .forEach(list -> children.addAll(list));

But it has a drawback too. Let's say that once we get a list of all Child objects we would like to modify them, aggregate, or do calculations on them. Ideally it would be good to perform those action in the same stream. Unfortunately, this is not the case, the forEach() method in the above example is ending the stream processing, which makes it impossible to process Child records in the same stream.

Solution

Luckily Java creators foresaw this problem and introduced a flatMap() function that is part of a java.util.stream.Stream class.

The idea is pretty straightforward. It does two things with every element of a stream:

maps - transforms one element from a stream into a new stream, so as a result we would get a stream of streams,
flattens - results of a previous operation are merged into one stream.

To visualize it consider the following situation:

Let's say that we have a stream of Author objects, that has a method called books() which returns a list of Book objects. And now let's say that we would like to have access to all books written by all authors to make further operations on them. If we would use the map() function within which we would call the books() method we would get a stream of lists of Book objects. This is not what we want to have.

What we would like to have is a stream of Book objects, not a stream of their lists. How we can overcome it? Using flatmap() instead:

As previously we need to invoke the books() method of the Author class to get a list of Books. The only difference is that input needs to be converted into multiple values represented by a Java stream. This is the requirement of the flatMap() method. Whatever operations we do within it needs to return a Stream<T> object.

The same situation can be reflected with a code:

List<List<Book>> listsOfListOfBooks = authors.stream()
    .map(Author::books)
    .toList();

List<Book> listOfBooks = authors.stream()
    .flatMap(author -> author.books().stream())
    .toList();

To translate a list of Book objects the standard Collection.stream() method was used.

When to use it?

Unwrap & operate

The most common case when the flatMap() method might be handy is when one of the stream operations produces a collection of objects and we would like to make further actions on each one of them.

To visualize it let's go back to the previous example with Book and Author records. Here are their definitions:

public record Book(String title) {}
public record Author(String name, List<Book> books) {}

Now let's say that we want to create a method that takes a list of Author objects as input and produces the list of all book titles that these authors wrote:

List<String> getAllBookTitles(List<Author> authors) {
return authors.stream()
    .flatMap(author -> author.books().stream())
    .map(Book::title)
    .toList();
}

After converting a list of Author objects into a stream the flatMap() operation is used first. The books() method is invoked to get their list and then it's changed into the stream. The flatMap() is then merging all resulting streams into one so next the title() can be called to get a String representation of a book title. Finally, the results of each element in a stream are collected into the list.

The above method can be written even better. We can split invoking books() and stream() methods into two operations - map() and flatMap() respectively - to get a nice looking code:

List<String> getAllBookTitles(List<Author> authors) {
return authors.stream()
    .map(Author::books)
    .flatMap(List::stream)
    .map(Book::title)
    .toList();
}

Merge lists

Another case when flatMap() can be very useful is when we would like to combine two or more lists (or any other java.util.Collection).

List<String> mergeLists(List<String> left, List<String> right) {
return Stream.of(left, right)
    .flatMap(List::stream)
    .toList();
}

A big plus for this approach is that after flatMap() we don't need to close the stream immediately. Instead, we can apply other operations on every object, like filtering, mapping, aggregating, etc. Which is cleaner and more efficient.

Get value from nested Optional

Apart from Java streams flatMap() method can be invoked on an Optional object. It's used to unwrap an Optional that is nested inside another Optional.

Let's say that we've got the following record:

public record Address(String street, String buildingNo, Optional<String> apartmentNo) {}

Now suppose we would have an Optional<Address> and would like to extract a value of an apartmentNo. The code without flatMap() would look like this:

String extractApartmentNo(Optional<Address> address) {
    if (address.isEmpty()) {
        return "";
    }

    return address.get().apartmentNo().orElse("");
}

The first step is to unwrap the value from the Optional, which might be empty. Only after checking it, we can proceed with unwrapping (and handling empty values) an apartment address.

This approach is ok, but can be done better with flatMap():

String extractApartmentNo(Optional<Address> address) {
return address
        .flatMap(Address::apartmentNo)
        .orElse("");
}

This approach a way nicer than the previous one. Both Optionals - parent and child - are validated whether they hold a null value in a single expression.

Summary

Introducing streams into Java made Data processing easier. It brings us a lot of handy operations. flatMap() is one of them which gives us the possibility to merge multiple streams into one or flatten nested streams into one. It's a very common pattern and is used many times in real projects.

Code and tests from this post can be found in my repository: wkrzywiec/java-series | Github

References

Originally published at https://wkrzywiec.is-a.dev on December 12, 2021.

Food Delivery App

Wojtek Krzywiec — Sun, 28 Aug 2022 18:42:13 +0000

Overview of My Submission

This is a very simple food delivery app. It allows to search for food and then order and delivery it to your home.

The idea behind this was to build an event-driven distributed system. It contains 5 microservices which communicates with each other via events (most of the time).

Apart from that I wanted to try to implement a simple event sourcing system in which all business events are stored as a sequence of state-changing events.

In this project all these concepts are covered with Redis.

But this is not everything! It also uses other Redis modules, e.g. RedisJSON and Redis Search to show how easy it's to build a reliable system with Redis.

Submission Category: Microservice Mavens

Video Explainer of this project

Language Used

Java/Spring
JS/React

Link to Code

wkrzywiec / food-delivery-redis

Food Delivery app

This is a very simple food delivery distributed system. It allows to search for food and then order them with a delivery.

The entire system is made of 5 microservices (1 React, 4 Java/Spring):

ui - React application, used by customers to place orders, manage them and track deliveries, url: http://localhost:80,
bff - backend for frontend service, used to provide REST endpoint for ui, url: http://localhost:8081/swagger-ui.html,
food - service that handles adding available meals to Redis, url: http://localhost:8084/swagger-ui.html,
ordering - core service for managing orders,
delivery - core service for managing deliveries.

Overview video (Optional)

Here's a short video that explains the project and how it uses Redis:

How it works

Here is the overview of a system architecture with used Redis modules:

Most of the communication is based on commands and events. E.g. in order to place an order a proper command…

View on GitHub

Additional Resources / Info

Check out Redis OM, client libraries for working with Redis as a multi-model database.
Use RedisInsight to visualize your data in Redis.
Sign up for a free Redis database.

Classic Den - Azure Trial Hackathon

Wojtek Krzywiec — Mon, 07 Mar 2022 14:15:40 +0000

Overview of My Submission

Classic Den is a simple website that can be used to post any announcement, thought, idea or question. Its style is inspired by the look and feel of classic PC systems.

Link to a website: https://polite-dune-017443c03.1.azurestaticapps.net
Link to a repository: https://github.com/wkrzywiec/classic-den

My target was to create a low-cost website which could be used by small communities to post announcements or other information. The main assumption is such a page would have only a few write operations (adding posts), but lots of reads (viewing page with posts).

As a result Classic Den is made of three components:

frontend - a simple, framework-less application, written with plain HTML, CSS, and JS. It's served from Azure Static Web App service.
backend functions - two Java functions deployed on Azure Functions responsible for adding and removing old posts (after 14 days).
GitHub repository - acts as a source of code for both frontend and backend, but also it's used to store posts. Whenever a new post is created a Java function is committing a change to this repo which triggers a GitHub Action which updates static web app with a new

Submission Category:

Computing Captains

Link to Code on GitHub

wkrzywiec / classic-den

Simple client and server app for publishing posts

Classic Den

Project submitted to the Microsoft Azure Trial Hackathon. Submission post on the Dev.to.

Classic Den is a simple website that can be used to post any announcement, thought, idea or question. Its style is inspired by the look and feel of classic PC systems.

About

This project was created for Microsoft Azure Trial Hackathon. My main goal was to learn something about the Microsoft Azure services, like Azure Functions or Azure Static Web Apps.

My target was to create a low-cost website which could be used by small communities to post announcements or other information. The main assumption is such a page would have only a few write operations (adding posts), but lots of reads (viewing page with posts).

As a result Classic Den is made of three components:

frontend - a simple, framework-less application, written with plain HTML, CSS, and JS. It's served from Azure…

View on GitHub

Additional Resources / Info

Components

Frontend (Azure Static Web App)

The frontend part is built with vanilla HTML, CSS, and JS. No framework was used.

There are only two external dependencies used:

BOOTSTRA.386 project - it is a JS and CSS theme inspired by the 1980s DOS. To learn more about it check its official GitHub project page or the documentation with a full demo. If you, like me, are amazed at how it looks please support its creator.
jQuery - a very popular library, in this project used to make calls to the Azure Function.

Backend (Azure Functions)

Simple plain Java application with two functions:

create new post - an HTTP triggered function used to add a new entry by translating the JSON request into HTML post representation and committing it to this repository (entries.html).
remove old posts - a time-triggered function, invoked once per day, which checks for posts that were published more than 14 days from the day there were created.

It's deployed to the Azure Functions via VS Code plugin.

GitHub repository

Source code of frontend and backend services. Also, it stores all posts as HTML representation (in the entries.html file).

Whenever a new entry is added a new GitHub Action is triggered and it deploys all static resources to the Azure Static Web App service. Only after that, an entry will be visible on a website (that's why you need to wait a couple of minutes to see it on a page).

Screenshots

Automating quality checks for Kubernetes YAMLs

Wojtek Krzywiec — Thu, 02 Sep 2021 10:44:43 +0000

Cover image by Mark Tegethoff on Unsplash

If you have ever wondered how to make sure that your YAML Kubernetes objects are defined correctly and are following industry best practices, this blog post is for you. In a few paragraphs, I'll show you how to create a GitHub Actions workflow that will first analyze K8s object definitions using Datree, then deploy it on a cluster and run some tests.

It doesn't matter if you've just started your journey with Kubernetes or you're already an expert in it; writing object definitions is not an easy task. You can very easily make a mistake that could be very costly, if deployed to a production environment. And if you're just starting to learn Kubernetes, you might need help understanding which metadata and specifications should be provided, but, by design, are not mandatory.

The ideal way to navigate these challenges would be to have an experienced colleague perform a review of your code changes. But sometimes you don't have such a person around, or it's very hard to get feedback from them. And even if you do have an amazing person to help you, it still won't prevent you from making mistakes from time to time.

Instead, I would like to show you another approach: create a simple, automated quality check of your K8s object definitions using Datree.io, Google Kubernetes Engine and Github Actions.

We'll build a GitHub Actions workflow which will be triggered after each change made on the master branch. The workflow will have two stages:

Datree analysis of missing configurations, and quality check,
Deployment and testing an example application in a real cluster (GKE).

So roll up your sleeves and let's automate it!

Workflow structure

First of all you need a GitHub repository to hold our YAML files. I'm using my old project - k8s-helm-helmfile. This repository has three folders, each containing a different approach to deploy applications into Kubernetes clusters. You can read more about those approaches in my previous blog posts about vanilla K8s, Helm and helmfile deployments.

To keep this blog post short, I'll show how to create a workflow that will use Helm to deploy an application, but you can easily do this with other approaches, such as those mentioned previously.

The first step is to create a workflow definition file. In the root folder of a repository create a new directory ./github/workflows inside of which there will be a master.yaml file:

name: Quality check
on:
  push:
    branches:
      - 'master'

It contains following specifications:

name - name of our workflow
on - a condition based on which a workflow will be triggered. A workflow will only be started when changes are committed on the master branch.

With that, you can move on to the best part - defining jobs.

Datree analysis

In this part you will use a free tool called Datree, which analyzes K8s definitions and will stop the workflow if it finds any problems. It's very important to have a safety net like this, so you can feel confident that even if you make a mistake, or aren’t aware of best practices, an assistant will keep you on track.

Before defining a GitHub workflow, let's install Datree locally. To do that, go to their official website which will guide you how to install the Datree CLI. I'm using Linux (or to be precise, WSL), so only the following command is necessary:

> curl https://get.datree.io | /bin/bash

After couple of seconds Datree will be installed.

To test it out, go to the folder where Kubernetes YAML files are located and run Datree test command (in my case I'm using a vanilla K8s file from k8s-helm-helmfile project):

> datree test adminer-deployment.yaml

where the adminer-deployment.yaml file is a Kubernetes object definition.

Here is the output that I got:

As you can see, Datree has prepared a short summary of how many rules this YAML file violates. It provides a very useful explanation with a hint on how to fix them.

Starting from here, you can work on each issue to make these tests pass. It's a great way to learn and practise. But what if you deliberately chose not to comply with some of the rules? Luckily, Datree gives a possibility to prepare a custom policy, a set of rules against which YAML files will be checked.

To set up a policy, you need to go to your dashboard. Your personal link to it is located at the end of each scan, in the summary table, in the last row called See all rules in policy (I've marked it in a previous screenshot). It will take you to a login page.

Then you need to login. For convenience, I would suggest using the GitHub account. After successful authorization you will reach a Policies page.

Here you can inspect all the rules that you can turn on and off. By default, only some of them are enabled. To activate or deactivate them, use the toggle button visible next to the rule name.

If a name of a rule is too enigmatic, you can check its details by clicking on its name. It will then show more information and a sample output when this rule is violated.

Another cool feature is that you can define your own tips on how to fix a problem. The defaults should be enough, but if you prefer to extend it, view it in your own language, or add a link to a blog post, Stack Overflow discussion or any other online material, you can put it here by simply clicking an Edit button.

Finally, you can create your own set of rules, known as policies, to run for different applications, environments and stages. Simply click the Create Policy button at the top of the page.

Let's now check the second page, available on the left panel, called History. As the name might suggest, here you can see a convenient summary of all previous test runs.

Before writing this post, I’d already played around with Datree, which is why I already have several test runs listed in the history panel, but in your case you should have only one.

That would be it for a quick tour around Datree dashboard!

Now let's build a GitHub Actions workflow.

First, provide your Datree token as an environment variable in a workflow. To achieve this, click on your avatar in the top right corner of the Datree dashboard, then click Settings. It'll guide you to the page where you can find your token.

Once you’ve copied the token, go to your GitHub project/repository’s Settings page. Then select Secrets and click the New repository secret button.

In the Name field put DATREE_TOKEN and in the Value field a token copied from Datree.

Now you can move on to the workflow's definition file and define the first job:

jobs:
  datree:
    name: Validate Helm charts
    runs-on: ubuntu-latest
    container:
      image: dtzar/helm-kubectl:3.6.3

It's called datree, it runs on the latest Ubuntu inside a dtzar/helm-kubectl Docker container. I've selected this setup because I would like to run tests against Helm release (and not vanilla K8s as it was done previously). The reason to choose this Docker image is because it contains necessary dependencies (K8s and Helm), so I can skip their installation and make my workflow faster.

Let’s define the next three steps:

    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2

      - name: Install Datree 🔨
        run: |
          helm plugin install https://github.com/datreeio/helm-datree

      - name: Datree test 🔥
        env:
          DATREE_TOKEN: ${{ secrets.DATREE_TOKEN }}
        run: |
          helm datree test ./helm/app -- --values ./helm/adminer.yaml

With the first one, actions/checkout@v2, you get code from the repository. In step two, you instal the Datree Helm plugin to run Datree tests using Helm. In the third step, you run actual tests using a Helm CLI. The DATREE_TOKEN environment variable is added to it so the result will be linked with a Datree account.

Finally, in an actual run script, I provide the location of a Helm template and the location of testing values.yaml file.

After that you can commit changes and push it to GitHub. It should trigger a workflow, which will be available in the Actions tab.

In my case, several tests are failing, demonstrated by the failing status of a workflow. To find out more, click on the failing job. It will take you to the console output, where you can investigate what went wrong.

As well as checking policy check result(s) in the workflow's console, you can go back to the History page in the Datree dashboard and analyze errors there.

If you have a similar screen to mine, stop here, correct mistakes and push changes to GitHub. If you feel that some rules are obsolete for you, turn them off in the Datree dashboard, but do not turn off too many!

After fixing all the issues and re-running the workflow, the previously marked red exes [X] will become green check marks [V], indicating that the workflow has passed the validation and policy check.

And some details about a job:

The successful result will be also visible in the Datree dashboard:

Awesome! We can now move on to the next part.

Testing on GKE

After making sure that templates are okay from Datree’s point of view, move to deploying them to a real Kubernetes cluster (which is not production) and then check if everything is working there, e.g. if a website is available over the internet, etc.

First step would be to have a cluster. I've picked Google Kubernetes Engine (GKE) because of its free tier, but if you have your own cluster (on AWS or any other cloud provider) just use it.

Right now I'll follow with steps that are necessary to set up a job for GKE, so if you already have a GKE or any other Kubernetes cluster up and running, skip this part.

Before adding a new job to a workflow, you need to set up couple of things in GKE (these instructions are based on official GKE Quickstart guide):

Create a Google Cloud account. You can start from the main page,
Create or select an already existing Google Cloud project. It can be done either from Google Cloud console or from Google project selector.
Enable billing for a project. You will need to provide your credit card information, like it's described here, but don't worry, you won't be charged anything. This step is just to verify that you're a human.
Enable GKE APIs, which can be done on this page.

Until now, all configuration was carried out in a web browser, but now you need to move to either Google Cloud Shell, or install Cloud SDK and follow the instructions in a terminal:

Set up basic configuration in gcloud tool, like default project, region and zone. In my case, the project name is k8s-helm-helmfile and the region is europe-central2, but for you it may be different.

> gcloud config set project k8s-helm-helmfile

> gcloud config set compute/region europe-central2

> gcloud config set compute/zone europe-central2-a

Create an IAM Service Account, which is an account which will be used in GitHub Action workflow. I've called mine helm-github-actions-service:

> gcloud iam service-accounts create helm-github-actions-service

Get an email from the newly created Service Account. You will need it for the next step:

> gcloud iam service-accounts list

Assign roles to a Service Account, where <EMAIL> tag is taken from previous step:

> gcloud projects add-iam-policy-binding k8s-helm-helmfile \
  --member=serviceAccount:<EMAIL> \
  --role=roles/container.admin \
  --role=roles/storage.admin \
  --role=roles/container.clusterAdmin \
  --role=roles/iam.serviceAccountUser \
  --role=roles/container.developer

Export the Service Account Key:

> gcloud iam service-accounts keys create key.json --iam-account=<EMAIL>

> export GKE_SA_KEY=$(cat key.json | base64)

Add it as a GitHub Secret to a project (the same way as for DATREE_TOKEN) with GKE_SA_KEY as key. To see the value of the exported key you can use the command:

> printenv GKE_SA_KEY

Everything is set up, now go back to workflow definition where you first created a Kubernetes cluster on GKE, deploy a sample Helm release, and test it.

First create a new job called gke :

gke:
    name: Test Helm chart on GKE 
    needs: datree
    runs-on: ubuntu-latest
    env:
      PROJECT_ID: k8s-helm-helmfile
      GKE_CLUSTER: helm-test
      GKE_REGION: europe-central2

Similarly to the previous example, you have name and runs-on configurations. There is also a needs configuration, which means that in order to run this job, the datree job needs to first be completed successfully. This prevents spinning up clusters and deploying a sample application if something goes wrong during a Datree check. The last part of job configuration are environment variables (env), which will be used in workflow steps. They're my GCP project id, K8s cluster name, and my GKE region.

Moving on to the next steps:

steps:
  - name: Checkout 🛎️
    uses: actions/checkout@v2

  - name: Setup gcloud CLI ⚡
    uses: google-github-actions/setup-gcloud@master
    with:
      service_account_key: ${{ secrets.GKE_SA_KEY }}
      project_id: ${{ env.PROJECT_ID }}

The first one is for getting code from the project.
The second is to configure (login and set up project) Google Cloud CLI which will be used in following steps.

- name: Create Autopilot GKE cluster 🔨
  run: |
    gcloud container clusters create-auto ${{ env.GKE_CLUSTER }} \
    --project=${{ env.PROJECT_ID }} \
    --region=${{ env.GKE_REGION }}

  - name: Config kubectl for GKE cluster ⚡
    uses: google-github-actions/get-gke-credentials@main
    with:
      cluster_name: ${{ env.GKE_CLUSTER }}
      location: ${{ env.GKE_REGION }}
      credentials: ${{ secrets.GKE_SA_KEY }}

The above steps create a new GKE cluster and configure kubectl so it's connected with the newly created cluster. With that, you can move on to the step where an adminer Helm release will be installed:

- name: Deploy test Helm release 🚀
  uses: deliverybot/helm@v1
  with:
    release: adminer
    namespace: default
    chart: ./helm/app
    helm: helm3
    value-files: ./helm/adminer.yaml
    values: |
        app:
          service:
            type: LoadBalancer

Stop here and analyze what's going on. First of all, you’re using the deliverybot/helm GitHub Actions, which provides a convenient way to use Helm. By adding a few parameters, you can deploy an application onto a Kubernetes cluster. The entire list of available parameters can be found on the official website.

In the above example, I've used the following steps alone:

release - a release name,
namespace - specifies the K8s namespace where the app will be installed,
chart - gives information about the location of a Helm chart,
helm - indicates which version of Helm will be used,
value-files - file used to override the default values from a Helm chart, in my case it's an Adminer's values.yaml file (the Helm chart I use for testing, which deploys popular database client - Adminer),
values - this parameter works pretty the same as previous one - it's used to override the default values from a Helm chart, but instead of doing it with a file, we can directly specify values that need to be overridden; here, I'm overriding only the Service kind, as by default it's a ClusterIP, but I don't want to change it in the adminer.yaml file.

After successful installation, our workflow can proceed with tests. I've decided to run a very simple test, which only checks if a page is opening, but you can of course build a more sophisticated test chain.

- name: Test installed application 🔥
  run: |
export IP_ADDRESS=$(kubectl get services -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
    echo "$IP_ADDRESS"
    curl http://"$IP_ADDRESS":8080

The first line in a testing script is for finding out under which IP address the adminer application is exposed. The second is for debugging, and the last for actual testing.

Once the tests are done, you need to destroy a cluster:

- name: Delete cluster GKE Cluster 💥
  if: ${{ always() }}
  run: |
    gcloud container clusters delete ${{ env.GKE_CLUSTER }} --zone=${{ env.GKE_REGION }} --quiet

The if: ${{ always() }} part is very important here. It makes sure that even if any of the previous steps fail, this one will always run. Otherwise you could end up with a bill from Google at the end of a month.

After commiting changes and pushing them to GitHub, a workflow will look like this:

First a Datree step is executed, then the installation is done on GKE. To check the details of the second step, click on its name.

Conclusion

And that's it for today! I hope that this blog post encourages you to build something like this on your own, give Datree (or any other static code analysis tool) a try, and set up a cluster for automated tests, so you will feel more confident about the changes you made in your code base. It can all be set up and operational in a flash.

How to write good quality Python code with GitHub Actions

Wojtek Krzywiec — Sat, 06 Mar 2021 15:53:33 +0000

Cover image by Valdemaras D. on Unsplash

In this blog post I want to share with you how you can set up GitHub Actions workflows in your Python project to make sure that the code you write is elegant, aligned with all best practices and it’s well tested.

Lately I’ve decided to start to learn Python. Nowadays it’s one of the most popular programming language so it was no-brainer decision to try it out. Especially that I’m still bad at algorithms so I was hoping that I’ll kill two birds with one stone — learn Python and practice it with solving some algorithmic exercises.

Therefore I’ve created a Algorithms Python project on GitHub where I put solutions to problems that I found interesting.

But when I was practicing it I’ve stomp on a problem.

How to make sure that the code I write has high quality? How can I make sure that my code is aligned with official style guide PEP 8 ? And how to check if it’s well tested?

Well there is a solution! GitHub Actions!

But what exactly I want to achieve?

I’ll show you how to:

set up integration with CodeFactor — tool that founds potential bugs in your code,
set up wemake-python-styleguide GitHub Actions workflow that checks if your code is aligned with PEP 8,
run unit tests and report its’ coverage with Codecov.

In the next steps I assume that you already has a Python project on GitHub. If you don’t have it create one 😉.

CodeFactor

It’s a very simple tool that check the quality of you code. It’s free of charge for all public repositories and for 1 private.

To start with it, go to the official website — www.codefactor.io, and create a new account by logging via GitHub. The icon for doing that should be located in the top right corner of the main page.

Once you’ll be registered add a repository to your dashboard by clicking the plus ( + ) sign located at the top right corner.

From the list pick the repository that you want to analyse and click Import at the bottom of the page.

And that’s it! You should be now redirected to a dashboard with a list of all issues that has been found. Awesome!

wemake-python-styleguide

Let’s move on to the second tool which will check if a code is written accordingly to the official Python style guide.

This time we won’t create an account in any web service. We’ll set up a GitHub Actions workflow which will be triggered whenever a pull request is created and will add comments to a review where it founds potential problems.

To those of you who don’t know what GitHub Actions is. It’s a new feature on GitHub that allows to automate many tasks and it’s usually treated as CI/CD tool (Continuous Integration/Continuous Deployment) which runs tests, runs quality checks and then deploy it. But it’s not the only purpose of it.

To start with it, in the root folder of your project create a .github/workflows folder where definitions of your workflows will be located.

Then create a new file with a name workflow-pr.yaml.

name: Python Pull Request Workflow
on: [pull_request]
jobs:

  qa:
    name: Quality check
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v1
      - name: Set up Python
        uses: actions/setup-python@master
        with:
          python-version: 3.8
      - name: Run unit tests
        run: |
          pip install pytest
          pytest
      - name: Wemake Python Stylguide
        uses: wemake-services/wemake-python-styleguide@0.13.4
        continue-on-error: true
        with:
          reporter: 'github-pr-review'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

It’s a very simple workflow with the full name Python Pull Request Workflow. It’s triggered on each pull_request, so whenever we create new or update existing one following jobs will run.

Above workflow consists only one — qa — job that has 4 steps:

actions/checkout@v1 — is necessary to let GitHub Actions workflow know that it can use code located in a repository,
Set up Python that uses actions/setup-python@master configure a Python version, in our case it’s python-version: 3.8.
Run unit tests runs all unit tests located in the project. For that I’m using pytest which first needs to be installed pip install pytest so the next command pytest can be run. If any test fails on this step the next one will not run.
Wemake Python Styleguide step is the one that we’re the most interested in. It uses the wemake-services/wemake-python-styleguide@0.13.4 action which are the atomic building blocks of the workflows. You can found them on the GitHub Marketplace, like mentioned action. This one is configured (with clause) to use github-pr-review reporter which enables the inline comments in the code review. More supported reporter options could be found on the official website. Finally this workflow requires to pass your GIHUB_TOKEN and that’s why the env clause is added.

To test it running you need to create a new branch, commit some changes and push it to the GitHub. Then create a pull request which will trigger this workflow. To check it go to the Actions tab in your project and if everything went fine it should look like this one:

If you click on Run unit tests, in the console log, you’ll see the test report:

And if you go back to the Pull request you should see comments added. Like here: https://github.com/wkrzywiec/algorithms-python/pull/6

Codecov

And finally we want to have a test report with coverage. For that we again use pytest library which will generate it for us and then we upload it to the Codecov which will take care of visualizing it.

Before defining a new workflow first you need to create an account at Codecov. Therefore go to https://codecov.io at click Sign Up button located at the top right corner.

Then choose GitHub as a sign up option.

Then you should be taken to your dashboard for GitHub projects, where you need to click the Add new repository button.

A list of all your projects will appear from where you can pick the one you want to analyze.

Then a page with your token will appear. Save it, because we will use it in next step.

Now go back to the project on GitHub and click on its Settings button. There click Secrets, and then Add a new secret where you can provide the token you’ve generated on Codecov website. To finalize it click Add secret.

Ok, everything is set up, so we can move on to defining the GitHub workflow.

name: Python Master Workflow
on:
  push:
    branches:
      - 'master'
jobs:
  codecov:
    name: Codecov Workflow
    runs-on: ubuntu-18.04

    steps:
      - uses: actions/checkout@v1
      - name: Set up Python
        uses: actions/setup-python@master
        with:
          python-version: 3.8
      - name: Generate coverage report
        run: |
          pip install pytest
          pip install pytest-cov
          pytest --cov=./ --cov-report=xml
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v1
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          file: ./coverage.xml
          flags: unittests

Again we create a separate file called workflow-master.yaml, because this time we don’t want to trigger this workflow when pull request is created. This one should run only when a new commit is pushed on the master branch.

In the jobs section there is only one job called codecov which consists of 4 steps:

uses: actions/checkout@v1 — this one, again, is just to tell GitHub Actions that we want to use the files located in current repository,
uses: actions/setup-python@master — this one was also mentioned before, here we set up a Python version, which is 3.8,
then there is a new step, which is responsible for Generate coverage report; it’s a series of scripts which cover installing pytests (pip install pytest), pytest-cov (pip install pytest-cov) and running actual tests (pytest — cov=./ — cov-report=xml),
and finally the generated test coverage report can be uploaded to the Codecov (Upload coverage to Codecov). Here we uses: codecov/codecov-action@v1 GitHub Action available on the Marketplace. In it we provide 3 arguments: token: ${{ secrets.CODECOV_TOKEN }} which value is taken from the GitHub Secrets vault where we put it, file: ./coverage.xml is location of the test coverage report (generated step before) and flags: unittests is a flag that groups our unit tests.

To test it you need to push some commits to the master branch. It can be done directly from your local repository or by merging a pull request. If everything went fine, it should like this:

Now if you go back to the Codecov to your project dashboard you should see similar output:

All right, we did it! 🍾

Just before the wrap up I want to tell you that CodeFactor, Codecov or wemake-python-styleguide are not the only tools that can help you write better quality code. In fact there plenty of them, like SonarCloud, Pylint, Coveralls, DeepSource and more. Some of them could be found on GitHub Marketplace, which is best place to start if you don’t like those what I have proposed.

Conclusion

With this blog post I hope I’ve gave you an idea of how you can set up your GitHub repository to make sure that code you write has good quality. With such tools set you can find all the bugs and vulnerabilities, but keep in mind that not every bug, not every issue is worth to be taken care of. Sometimes it’s better to focus on the real work, not on tidying smelly code base 😉.

As usual, you can find mentioned project on GitHub.

Write better code with SonarQube

Wojtek Krzywiec — Sun, 28 Feb 2021 14:35:31 +0000

Cover photo by Khai Sze Ong on Unsplash

In this blog post I introduce a SonnarQube, a static code analytic tool, which can help you write more secured, less buggy and cleaner code. I show how to run it and play around with it.

Have you recently started to code in your free time? Or maybe you’re a full-time senior developer with couple years experience? No matter how long you’re in software engineering sometimes it’s hard to write nice and clean code. Especially when you’re working on a complex project.

The best solution would be to program in pairs with another developer or ask her/him to check what you’ve written in a code review. But sometime you don’t have anyone around to help you. Luckily there is solution for this problem — static code analysis tool. Of course it won’t be the same as human, but still can help you a lot with your work.

But wait, what’s static code analysis tool?

Based on special algorithms these tools analyze the code we write and look for bugs, possible security breaches, code smells and presents it in the some kind of report that helps us, developers, find issues in our code. Moreover they can check the unit test coverage, to make sure that all crucial parts of the logic are properly tested.

Such tools not only help developers find possible mistakes, but they also allows companies to become more into DevOps culture, where programmers get instant feedback about their code quality and prevent moving possible vulnerable feature into production environment.

An example of such tools (for Java) are: Findbugs, PMD and SonarQube. And I want to talk about the last one more briefly in this blog post.

SonarQube

To learn about all its features let’s install it and check on some of my project. Therefore you need to have an instance of SonarQube Community Edition up and running on your local machine.

I prefer to use Docker image for that (I’ve recently try dockerize everything), but you can go with regular installation from a zip file. The instructions are available on the official website. All mine instructions are valid also with this approach (you only need to skip the Docker part).

Prerequisites

In order to follow my instructions you will need install:

Docker — instructions for Ubuntu , Windows , Mac (if you’re on Ubuntu, you’ll need also to install Docker Compose separately),
Maven — official installation guideline.

Docker Compose

Once you’ve got installed all prerequisites we can move to setting up the SonarQube Docker container. I would like to use sonarqube:7.9-community Docker image, but there are also two additional things that I would like to configure:

by default, all the analysis are saved in H2 database, I want to change it, so all of them will be stored in the PostgreSQL,
I would like to enable SonarQube Server configuration with sonar.properties file.

As the result I’ve came up with following structure of a folder:

    develop-env/
    ├─ docker-compose.yml
    └── sonar/
       ├── Dockerfile
       └── sonar.properties

First, let’s focus on sonar folder, which contains a Dockerfile(recipe for an image):

FROM sonarqube:7.9-community
COPY sonar.properties /opt/sonarqube/conf/

and sonar.properties file:

sonar.junit.reportPaths=target/surefire-reports
sonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml

Dockerfile is pretty straightforward, it uses official sonarqube:7.9-community image and copy paste the sonar.properties file into it. The second file is responsible for telling Sonar where to look for reports related to test coverage.

Next, let’s move on to docker-compose.yml file.

version: '3'

services:
  sonarqube:
    build: ./sonar
    ports:
      - 9000:9000
    networks:
      - sonarnet
    environment:
      - sonar.jdbc.url=jdbc:postgresql://sonar-db:5432/sonar
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
    links:
      - sonar-db

  sonar-db:
    image: postgres:9.6-alpine
    networks:
      - sonarnet
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - postgresql:/var/lib/postgresql
      - postgresql_data:/var/lib/postgresql/data

networks:
  sonarnet:
    driver: bridge

volumes:
  sonarqube_conf:
  sonarqube_data:
  postgresql:
  postgresql_data:

In a nutshell it tells Docker to create two containers: sonarqube and sonar-db. First one is an instance of a SonarQube Server, and a second one is its database.

Now you can start the server, therefore open the terminal in a folder where docker-compose.yml is located and type:

> docker-compose up -d

After few minutes you should be able to enter the main page: http://localhost:9000.

If your SonarQube Server is not starting probably it’s related to ElasticSearch memory limitation. The solution to this problem can be found on the Stack Overflow.

Maven configuration

Once we have SonarQube Server up and running let’s enable our projects to be able to generate reports that than can be consumed by SonarQube Server.

Therefore we need to configure SonnarScanner, as the name may suggest, a tool for scanning projects. There are couple approaches for achieving this. All of them are described on the official website.

I‘m picking a Maven approach because I’ve used it in my recent project. First step would be to set up global configuration of Maven, so go to your’s .m2 folder (it’s usually located in user root folder, more information about it you can find here) and add (or adjust if you already have) the settings.xml file.

<settings>
    <pluginGroups>
        <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
    </pluginGroups>
    <profiles>
        <profile>
            <id>sonar</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
            </properties>
        </profile>
     </profiles>
</settings>

Project’s POM config

After setting up the global configuration of Maven you can go to your project.

For demonstration purposes I’m using my recent project - Kanban-app, which is a Java (Spring Boot) based REST application.

The only thing that I would like to add here is a JaCoCo Maven plugin that will generate a code coverage report which can be used by SonarQube (if don’t want to have such report you can skip this part, but I strongly recommend to have it).

Therefore, open pom.xml file in your project and in the section build and then plugins add following lines:

<!--- some dependencies --->
<build>
    <plugins>
        <!--- different plugins --->
        <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>0.8.4</version>
            <executions>
                <execution>
                    <id>default-prepare-agent</id>
                    <goals>
                        <goal>prepare-agent</goal>
                    </goals>
                </execution>
                <execution>
                    <id>default-report</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>report</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Running the analysis

Everything should be set up now, so let’s run our first analysis!

But before that, make sure that your SonarQube Server is running. Then go to your project and either in your IDE or in the terminal run following command:

> mvn clean verify sonar:sonar

And then enter the SonarQube — http://localhost:9000 and you probably see the number of projects that was analyzed.

To get more details click Projects icon (at the top bar):

And then click on the project name that you would like to check. In my case it’s kanban-app.

In the new window you’ll see several issues that were discovered during the analysis. They can be Bugs, Security Vulnerabilities, Code Smells, Duplications or Code Coverage.

By clicking on each one of them you should get more detailed report. For example, when I click on Code Smells issues I’ve get following report.

I can even dig deeper into it and find out in which line of code has the code smell.

These kind of reports are similar for every kind of an issue, except for Code Coverage, to which I would like to move on.

When you enter it you should get similar report:

From my report you can see that the entire test coverage for my project is really low, so let’s fix it!

First, let’s see if all classes needs to be tested. If you look closer you might see that the report includes some model and configuration classes for which writing unit test is rather pointless. Therefore we need to exclude them from the report.

To do so, open pom.xml file once again and in <properties> section add <sonar.exclusions> section in which you need to provide packages and/or classes that you want to exclude from the coverage report:

    <properties>
       <!--- Other properties --->
       <sonar.exclusions>
          **/model/**,**/config/**,**/KanbanApplication.java
       </sonar.exclusions>
    </properties>

More information about the syntax could be found on the official website.

Second thing to do is to write new unit tests that will cover untested parts of the code. To get to know which lines of code are not tested, move back to the report and pick a class that you would like to check.

On the left, there are green and red blocks that indicates if this part of the code has unit tests.

Once you know which methods are not tested yet, roll up your sleeves and start to write some test.

After you’re finished with unit tests, rerun the analysis, with the same terminal command

> mvn clean verify sonar:sonar

And refresh the SonarQube report page and you should get a new report.

As you can see my test coverage is still pretty low, but with SonarQube I’ll be able to fix it really quick!

And that’s it! If you want to see a full code of my project you can go to my GitHub repositories:

with SonarQube Docker Compose

wkrzywiec / develop-env

Development tools in Docker container
with kanban-app

wkrzywiec / kanban-board

Single-click full-stack application (Postgres, Spring Boot & Angular) using Docker Compose

References

How to fetch REST API every 2 seconds with RxJava 2

Wojtek Krzywiec — Wed, 17 Feb 2021 13:19:02 +0000

Cover image by Matthew Smith on Unsplash

During my work on one of my Android application I’ve stomped on a problem where I needed to call some REST API every 10 seconds to check whether some data have changed or not. And with this blog post I would like to share with you how I’ve managed to achieve that.

Prerequisites

First of all, I don’t want to show the Android code, because due to some code around it it may blur the solution. Instead for this demo I go with a simple command line app.

Even though my application is very small I use couple tools that helps me develop the app and take care of some aspects that I don’t want to focus right now. So if you don’t feel comfortable with some parts of the code and you don’t know what it does, don’t worry. Here is a list of tools that I used with short explanation what it does.

Gradle —it’s used to manage dependencies (external libraries). More info: https://gradle.org/
Spring Boot — I used it for dependency injection, where Spring, as a platform, take care of the lifecycle of the objects (creates and destroy them). More info: https://spring.io/projects/spring-boot
RxJava 2 & Retrofit — both libraries are used to perform API call and make it asynchronous. They are very popular in Android app development.
Lombok — it helps to skip writting boring parts of model class like, getters, setter, constructor and so on. You only need to add some annotation above the class name and it’ll work. More info: Project Lombok.

Step 1. Set up app structure

So if you’re now ok with some of libraries that I use for this example we can go to the implementation. As it’s a Spring Boot app go to https://start.spring.io/ where you can generate Spring app. To do that just pick a Gradle project (or Maven if you prefer) and customize artifacts (group, artifact). Then click Generate Project.

After downloading it, unpack it and open with you favorite IDE (Eclipse, IntelliJ) — usually importing it as a Gradle project should work.

Then go to the RandomJokesApplication class and add create Logger object and implement CommandLineRunner interface, so it should look like this.

import org.slf4j.*;
import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;

@SpringBootApplication
public class RandomJokesApplication implements CommandLineRunner {

    private static Logger LOG = LoggerFactory
            .getLogger(RandomJokesApplication.class);

    public static void main(String[] args) {
        SpringApplication.run(RandomJokesApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {

        LOG.info("\n\n\n\n Hello World! \n\n");

    }
}

From now on, every “working” part of the code we will insert into the run(…) method. Now if you run this app you should have some Spring app info printed in the console as well as the “Hello World!”.

Great! Let’s move on to next part, but just before that, we need to add some dependencies to the build.gradle file, so all necessary libraries will be downloaded automatically.

// some other default properties

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'io.reactivex.rxjava2:rxjava:2.2.7'
    implementation 'com.squareup.retrofit2:retrofit:2.5.0'
    implementation 'com.squareup.retrofit2:converter-gson:2.5.0'
    implementation 'com.squareup.retrofit2:adapter-rxjava:2.5.0'

    compileOnly 'org.projectlombok:lombok:1.18.6'
    annotationProcessor 'org.projectlombok:lombok:1.18.6'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
}

For Lombok annotation processing you would also need to enable annotation processing in the settings of your IDE. Full instruction for each IDE can be found here: https://projectlombok.org/setup/overview

Step 2. Create RxJava 2 Observable that prints something in a console each 2 seconds

Now we can create an Observable that will print us something every 2 seconds.

import io.reactivex.Observable;
import io.reactivex.schedulers.Schedulers;

// some other stuff mentioned before

    @Override
    public void run(String... args) throws Exception {

    LOG.info("\n\n\n\n\t\t\t ---------  RandomJokesApplication is up and running --------- \n\n");


    Observable.interval(2, TimeUnit.SECONDS, Schedulers.io())
        .observeOn(Schedulers.newThread())
        .subscribe(s -> LOG.info("Tick"));

    Thread.sleep(10000);

    LOG.info("\n\n\t\t\t ---------  RandomJokesApplication ends its work --------- ");

    }

With interval() method we define a new Observable that will emits value each 2 seconds. ObserveOn() tells us on which thread subscribe() method will be running. And a final method is added to take care of the output of the emitted value (in our case it only prints the same String in the console).

You may also see that a main Thread has been paused to sleep for 10 seconds. It’s because I want to run my program only for this amount of time during which the events will be triggered.

When you run above code in the specified time frames in a console should be printed the same output — “Tick”.

Step 3. Add REST API service

So as we handle a first part, let’s move on to the next one, which is mapping the API call to the Retrofit.

A first step would be to create class models (data transfer objects) that reflects the structure of the API respond. For this demo I’m using The Internet Chuck Norris Database API that have specific URL for fetching, yes you guess so, Chuck Norris jokes! The URL is https://api.icndb.com/jokes/random and the respond looks like this:

{
  "type": "success",
  "value": {
    "id": 550,
    "joke": "Chuck Norris can speak Braille.",
    "categories": []
  }
}

So to reflect above structure we need to create following classes (notice that I map only those fields that are really needed):

import lombok.Data;

@Data
public class ChuckNorrisJoke {

    private ChuckNorrisJokeValue value;
}

@Data
public class ChuckNorrisJokeValue {

    private String joke;
}

@data annotation is a Lombok annotation, and creates for us getters, setters, toString(), equals(), hashCode() and no arguments constructor.

Then we need to create an interface that will map URL resource path to the method. The method should return the Observable and looks like this:

import retrofit2.http.GET;
import rx.Observable;

public interface ChuckNorrisJokesApi {

    @GET("/jokes/random")
    Observable<ChuckNorrisJoke> randomJoke();
}

You may notice that in @GET annotation only partial URL is provided. The remaining base needs to be provided in a service class, which we will be our utility class to fetch jokes from the Chuck Norris API.

import org.springframework.stereotype.Service;
import retrofit2.Retrofit;
import retrofit2.adapter.rxjava.RxJavaCallAdapterFactory;
import retrofit2.converter.gson.GsonConverterFactory;
import rx.Observable;

@Service
public class JokesService {

    private ChuckNorrisJokesApi chuckNorrisJokesApi;

    JokesService() {

        Retrofit retrofit = new Retrofit.Builder()
                .baseUrl("http://api.icndb.com/")
                .addConverterFactory(GsonConverterFactory.create())
                .addCallAdapterFactory(RxJavaCallAdapterFactory.create())
                .build();

        chuckNorrisJokesApi = retrofit.create(ChuckNorrisJokesApi.class);
    }

    public Observable<String> getRandomChuckNorrisJoke(){

        return chuckNorrisJokesApi.randomJoke()
                .map(respond -> respond.getValue().getJoke());

    }
}

At first glance the JokesService class may not be straight forward, but basically it only sets up the ChuckNorrisApi object and uses in a single method that is responsible for retrieving a text of the joke wrapped in a Observable.

The @Service annotation comes with Spring and tells Dependency Injection Container that it should take care of this object life cycle.

Finally we can inject our service (using @Autowired annotation) into the main class — RandomJokesApplication and then use it in run(…) method. For now just to print a single joke.

// ...beginging of the class

@Autowired
private JokesService jokesService;

// ...other middle stuff...

@Override
public void run(String... args) throws Exception {

        LOG.info("\n\n\n\n\t\t\t ---------  PeriodicQuotesApplication is up and running --------- \n\n");

    jokesService.getRandomChuckNorrisJoke()
        .subscribe(result -> LOG.info("\n\n\n\t\t\t {} \n\n", result));

    // following code...

}

If you run above code you should get a single Chuck Norris joke and which means that we have only one thing left ahead — combining it with periodically triggered Observable.

Step 4. Combing all together

To do that we could move a part of the code that is printing the joke to the subscribe() method of Observable that is printing something in each 2 seconds. But I would like to try a little different approach:

@SpringBootApplication
public class RandomJokesApplication implements CommandLineRunner {

    // some dependencies...

    @Override
    public void run(String... args) throws Exception {

        LOG.info("\n\n\n\n\t\t\t ---------  RandomJokesApplication is up and running --------- \n\n");


        Observable.interval(2, TimeUnit.SECONDS, Schedulers.io())
                .observeOn(Schedulers.newThread())
                .map(tick -> jokesService.getRandomChuckNorrisJoke())
                .doOnError(error -> LOG.info(error.toString()))
                .retry()
                .subscribe(jokeObservable -> jokeObservable.subscribe(joke -> LOG.info("\n\n\n\t\t\t {} \n\n", joke)));

        Thread.sleep(10000);

            LOG.info("\n\n\t\t\t ---------  RandomJokesApplication ends its work --------- ");

    }
}

Firs of all, I’ve added a map() step in which joke text is extracted, so the subscribe() method is much more cleaner. Also doOnError() and retry() steps were added to take care of error and make sure than even some of them occurs Observable will still fire the event.

If you run the application, you should get similar output:

And that’s it! 🤓

The entire code for this app could be found here (if you dig into commits history you can also see that a code was written similar to described story) :

wkrzywiec / Random-Jokes

Testing RxJava 2 periodic capabilities

References

What was added to Java 8? Optional class

Wojtek Krzywiec — Sun, 14 Feb 2021 09:43:10 +0000

Cover image by Isaac Benhesed on Unsplash

As a Java developer you’ve probably stomped on NullPointerException many times and usually handling this kind of error takes some time. Luckily thanks to new Optional class it’s much easier now.

This is a three-part series on Java 8 new features, other blog posts can be found here:

Lambda expression
Streams
Optional class (current)

This class works as a class container (or wrapper) which may or may not hold null value. Before introducing Optional class developers to make sure that objects references aren’t null were forced to check it before calling any method (if there were a great chance for it, of course). To illustrate this, see below example.

String pizzaRecipeText = "";
if (cookBook != null){
    Reciepes reiepes = cookBook.getRecipes();
        if (reciepes != null) {
            Reciepe pizzaReciepe = reciepes.getReciepe("Pizza");
            if (pizzaReciepe != null) {
                pizzaRecipeText = pizzaReciepe.getText();
        }
    }
}

Even if it’s a simple code it requires lots of boilerplate checking of nulls, which is not either fun to write or read. Also if we forget to check it, it becomes a potential “dangerous” part of a code.

So instead calling directly a class method that may or may not return null we can wrap it in Optional class.

CookBook cookBook = new CookBook();
Optional<Reciepes> optReciepes = Optional.ofNullable(cookBook.getReciepes());

optReciepes.ifPresent(reciepes -> System.out.println(reciepes.getReciepe("Pizza")));

CookBook class doesn’t have Reciepes class unless at least one of them was added to it. Therefore, in above example cookBook.getReciepes() method will result with a null, which then would result in our favourite exception while getting any reciepe.

Instead we can wrap it with Optional class and use ifPresent method that checks it for us and if it’s available it’ll proceed. In above example optReciepes reference holds null so the 4th line of the code won’t result in printing pizza reciepe, but it also won’t result in NullPointerException.

What if we won’t to pass default value instead? We can use orElse method.

CookBook cookBook = new CookBook();
Optional<Reciepes> optReciepes = Optional.ofNullable(cookBook.getReciepes());

Reciepes reciepes = optReciepes.orElse(new Reciepes());
Reciepe pizzaReciepe = Optional.ofNullable(reciepes.getReciepe("Pizza")).orElse(new Reciepe());

As it was earlier optReciepes holds null, but by invoking orElse method we create a default instance of a class. A default instance is passed as a parameter of this method.

Another approach for same situation would be throwing different exception, instead of returning a default value. Moving forward with an example, the Reciepes class does not have any reciepe yet, so when we call reciepes.getReciepe("Pizza") it should throw ReciepeNotFoundException.

CookBook cookBook = new CookBook();
Optional<Reciepes> optReciepes = Optional.ofNullable(cookBook.getReciepes());

Reciepes reciepes = optReciepes.orElse(new Reciepes());
Reciepe pizzaReciepe = Optional.ofNullable(reciepes.getReciepe("Pizza")).orElseThrow(ReciepeNotFoundException::new);

Except for mentioned above methods to extract the data from the Optional objects we can make it more clear by using map operator. Like in below example:

CookBook cookBook = new CookBook();
Optional<Reciepes> optReciepes = Optional.ofNullable(cookBook.getReciepes());

Long reciepesCount = optReciepes.map(Reciepes::getTotalCount);

Together with map function we can use a filter that will automatically reject the output of mapping if it won’t meet certain conditions.

Optional<CookBook> optCookBook = Optional.ofNullable(kitchen.getBook("Italian"));

Optional<Reciepes> optReciepes = optCookBook.map(CookBook::getReciepes)
        .filter(reciepe -> "Pizza".equals(reciepe.getTitle()));

To sum up Optional class allows us to handle NullPointerException in more graceful way.

References

What was added to Java 8? Streams

Wojtek Krzywiec — Wed, 03 Feb 2021 14:44:16 +0000

Cover image by Samuel Sianipar on Unsplash

Many programming tasks can be described as data processing i.e. we’ve got a collection of values which we want modify, like filter, transform or group. Until Java 8 this task was really painful (required multiple loops) and was not such efficient. Luckily there is new concept — Streams.

This is a three-part series on Java 8 new features, other blog posts can be found here:

Lambda expression
Streams (current)
Optional class (soon)

Java 8 introduced new pipeline-mechanism for data processing. It’s usually compared to pipeline or assembly line, because on start as an argument we provide a data collection and then we pass it thru the operations that will modify it and get another output.

To give you a picture of what stream capabilities are see below example. It compares the same task that was made with “standard” and stream based approach.

List<Tshirt> tshirtCollection = shop.getTshirtCollection();
List<String> selectedTshirts = new ArrayList<String>();

// without Streams
for(Tshirt tshirt: tshirtCollection){
  if (tshirt.getColor().equals("RED")) {
   if (tshirt.getSize().equals("M")) {
    if (new BigDecimal(50).compareTo(tshirt.getPrice() > 0){
      selectedTshirts.add(tShirt.getName().toUpperCase());
   }
  }
}

//with Streams
tshirtCollection.stream()
        .filter(tshirt -> tshirt.getColor().equals("RED"))
        .filter(tshirt -> tshirt.getSize().equals("M"))
        .filter(tshirt -> new BigDecimal(50).compareTo(tshirt.getPrice() > 0)
        .map(s -> s.getName().toUpperCase())
        .collect(Collectors.toList());

Even if I haven’t introduce the syntax yet we can see major benefits already. First of all, the code is much simplier and cleaner. And even if we don’t know all the operators yet they’re easy to understand.

To cover all these tasks Oracle team introduced new package java.util.stream that contains Stream class. In above example method stream() was called on collection, which results in a Stream — it means that a stream of t-shirt objects has been created from their list 👕.

All methods related to Streams can be categorized into three groups:

Stream producing methods,
Stream operating methods (intermediate methods),
Stream terminal methods.

Stream producing methods

Before we can operate on the objects stream first we need to create it. With a Java update Oracle has added a new method to Collection interface with a name stream() which converts object that implements this interface (like Lists, Maps or Sets) into the Stream.

The above statement indicates that only objects, like ArrayList, or HashSet can be converted into Stream.

We could also want to generate it from an array, using static method of the Arrays class — Arrays.stream(T[] array).

String[] arrayOfExpressions = {"Hello World!", "Hi everyone!", "Good Morning!", "How u doin'?"};

Stream<String> streamfromArray = Arrays.stream(arrayOfExpressions);

If we don’t have a list of objects we can create an infinite stream of integers that will start from 2 and will be increment by 3. To do so we can use static method of the Stream class — iterate().

Stream<Integer> infiniteStream = Stream
                                  .iterate(2, i -> i + 3)
                                  .limit(10);

The limit() step is necessary to end the infinite loop and it tells that we want to obtain a stream of 10 elements.

Another way to create a stream is to use static generate() method of Stream class. Below code will result in a 10-elements stream where each element is a “text” String.

Stream<String> stream = Stream
                          .generate(() -> "text")
                          .limit(10);

Stream operating methods (intermediate methods)

Once a stream is created we can perform multiple manipulation on it’s elements. The key concept for this is that we can create several operations that are chained together so they’re executed one after the another. It’s possible, because each operation returns a new instance of Stream class.

Here is the list of some (but not all) operations that can be performed on the Stream:

filter()— works similarly to WHERE clause in the SQL, it filters elements that match condition.

tshirtCollection.stream()
        .filter(tshirt -> tshirt.getColor().equals("RED"))

map() — it’s used when we want to transform each element in a Stream. For example, when we want to extract a value from a field in an object (name from the Tshirt class) or convert one value to another (kilograms to pounds).

Stream<String> tshirtNameStream = tshirtCollection.stream()
                                    .map(Tshirt::getName);

Stream<Double> tshirtPoundWeightStream = tshirtCollection.stream()
                                    .map(Tshirt::getKilogramWeight)
                                    .map(kg -> kg/2.205);

flatMap() — this operation is very similar to map(), but it also perform “flatten” task. It’s required when a body of map() operation returns a list or an array of values (not a single value as it was previously). If we use map() operation to perform such task we would receive a Stream of the Streams, each for an each element in a result list. But if we use flatMap() all of these Streams will be combined into single Stream.

String[] arrayOfExpressions = {"Hello World!", "Hi everyone!", "Good Morning!", "How u doin'?"};

Stream<Stream<String>> streamOfStreamsOfWords = Arrays.stream(arrayOfExpressions)
                                                  .map(exp -> exp.split("\\s+"));

Stream<String> streamOfWords = Arrays.stream(arrayOfExpressions)
                                                  .flatMap(exp -> exp.split("\\s+"));

sorted() — works similar to ORDER BY clause in the SQL, it sorts elements ascending/descending. If we don’t provide any argument to this method, all records will be natural ordered ascending. But if we input Comparator.reverseOrder() as an argument it will be order descending.

Stream<Integer> infiniteStream = Stream
                                  .iterate(2, i -> i + 3)
                                  .limit(10);

Stream<Integer> ascOrderedStream = infiniteStream.sorted();

Stream<Integer> dscOrderedStream = infiniteStream.sorted(Comparator.reverseOrder());

It might do a trick, when we need to sort Strings or digits, but if we won’t to order complex objects we need to create a Comperator object that will indicate which object field should be used for sorting.

Stream<Tshirt> ascOrderedTshirts = tshirtCollection.stream()
                                    .sorted(Comparator.comparing(Tshirt::getSize));

Stream<Tshirt> dscOrderedTshirts = tshirtCollection.stream()
                                    .sorted(Comparator.comparing(Tshirt::getSize).reversed());

distinct() — returns a Stream without any duplicate elements, all elements are unique.

Stream terminal methods

After performing all these transformation operation on a Stream we want to see a result of it. Stream class is just a wrapper so to get value from it we need to perform one of terminating actions listed below.

collect() —Stream can be compared to a list of elements, so this operation is used to convert Stream to a list (or any other collection instance). By passing an argument to it (it’s Collector object) we can specify what will be the result.

For example, if we use static method Collector.toList() to get a list of objects, or using Collector.toSet() will result in Set object.

List<Tshirt> redTshirtsList = tshirtCollection.stream()
                            .filter(tshirt -> tshirt.getColor().equals("RED"))
                            .collect(Collectors.toList());

Set<Tshirt> mediumTshirtSet = tshirtCollection.stream()
                                .filter(tshirt -> tshirt.getSize().equals("M"))
                                .collect(Collectors.toSet());

Apart from build-in Collector method we can use more sophisticated approach. We can use Collector.toColleaction() method we can specify a Collection object type (e.g. HashSet).

Set<Tshirt> mediumTshirtSet = tshirtCollection.stream()
                                .filter(tshirt -> tshirt.getSize().equals("M"))
                                .collect(toCollection(HashSet::new));

But it’s not everything that we can do with this method. For example, we can group (like in SQL) results to get their count, max value by the specific field. Or we can sum all values from specific fields. These an other examples can be found in the Oracle article.

toArray() — works similar to previous one, but here it results in an array of objects, not a Collection.
forEach() — with this method we don’t return any object (return type is void), but we can perform some action on each element that is in a Stream. For example we can print names in the console or perform an action on each of element.

tshirtCollection.stream()
             .filter(tshirt -> tshirt.getColor().equals("RED"))
             .map(Tshirt::getName)
             .forEach(System.out::println);

And that’s it. With this article I’ve only touched the surface of Java Stream but it’s a good point to start with. If you want to know more check the official documentation and play around it to know all the capabilities that it can offer.

References

What was added to Java 8? Lambda expressions

Wojtek Krzywiec — Wed, 27 Jan 2021 10:04:37 +0000

Cover image by Iswanto Arif on Unsplash

With this article I start a short series that will explain what features were added to Java 8 update. Today I’ll focus on the major buzz of this release — lambda expression (a.k.a. lambdas).

These series will be divided into three parts (links will be updated once each blog post will be published):

Part 1. Lambda expression (this one)
Part 2. Streams (soon)
Part 3. Optional (soon)

Lambda expression

When you start to learn Java and you have already passed command line examples you probably wants to create a desktop application. And for this you probably use JavaFX library (at least it was my case).

In most of JavaFX applications we need to handle events that can be triggered by the users. For example, when they press a button it’ll create an Event object that need to be handled.

Therefore we need to assign an action which will be triggered once user click the button. For this task we usually declare anonymous inner class, which has only one method that performs necessary actions.

button.setOnAction(new EventHandler<ActionEvent>() {
       @Override
       public void handle(ActionEvent e) {
           System.out.println("Button clicked");
       }
});

Even if above code is pretty straightforward it requires lots of writing and when code starts to grow it’s also becomes hard to ready. Luckily, thanks to lambda expression we can write it in a more compact way:

button.setOnAction( (e) -> System.out.println("Button clicked") );

Whoa! That’s short, but what’s going on there?

Lambda expression syntax is made of three parts. First one are brackets, (e) , that contains (or not) parameters of the abstract method of an anonymous inner class. It’s really important to remember that lamdas can be used only when a single abstract method.

In our case ActionEvent object is represented by e reference. If the method doesn’t have any parameter we can use simple () instead, like for Runnable interface.

Runnable r1 = () -> System.out.println("I'm in outside main thread!");

Finally method can have more than one arguments, like Comperator interface method compare.

Comparator<User> userComperator = 
(User first, User second) ->  first.email().compareTo(second.email());

In above example we could also don’t include argument types (User), but for clarity reasons it good to be added.

Next, after the arguments there is a newly introduce right arrow -> operator.

And finally there is a body of the implemented method. Usually it’s a one line of code, but if we require more we can surround it with {} brackets.

button.setOnAction( (e) ->  {
  System.out.println("Button clicked");
  label.setText("Clicked");
});

If a method should return a value we can use return statement as it’s in regular methods

Method reference (::)

Another topic that is tightly related and was introduced together with lambda expression is referencing the method. In short, with new operator :: we can assign method to a reference, just like the object or primitive type. This approach allow us to extract the method from the object and pass it in another place without executing them.

Object objectInstance = new Object();
IntSupplier equalsMethodOnObject = objectInstance::hashCode;
System.out.println(equalsMethodOnObject.getAsInt());

Above we assign a hashCode method to a reference IntSupplier. Then it can be passed wherever we want in the code.

With method reference we can also assign static methods (without creating instance of the class) or constructor.

References

How to declaratively run Helm charts using helmfile

Wojtek Krzywiec — Thu, 21 Jan 2021 09:33:56 +0000

Cover image by Kevin Ku on Unsplash

Have you ever wonder how to achieve fully Infrastructure as a Code approach for deployment of your applications on Kubernetes cluster? No matter if you’ve already achieve it or just started to looking for the best tool, helmfile might be the answer for you. In this blog post I’ll present how you can set up your first helmfile.

This post is a third, and the last, from my series on Kubernetes. In each one of them I present how to deploy applications on a Kubernetes cluster using one of the approaches:

using kubectl — Deployment of multiple apps on Kubernetes cluster — Walkthrough,
using Helm — How to deploy application on Kubernetes with Helm,
using helmfile — this one.

If you haven’t read first two I would recommend to do that before diving into this one. But it’s not the must, I’ll briefly introduce you to an environment which will be built.

Above picture presents the target solution. Inside the cluster there will be deployed frontend (kanban-ui) and backend (kanban-app). To persist data postgres database will be used. A source code for both microservices can be found in my GitHub repository — kanban-board.

Additionally I’ve added an adminer application, which is a GUI client for getting inside the database.

The above picture shows not only applications that will be deployed on Kubernetes cluster, but also objects that are required to run them, like Deployments or Services. With classic, kubectl approach, we were forced to create definition files for each one of them. Even if in most cases, for example in case of ClusterIPs, these objects haven’t differ that much between applications.

Definitely this approach doesn’t scale in microservice world, where usually there is a need to deploy dozens or even hundreds of applications. And if for each one of them we would be require to create couple YAML files it would really quick become a nightmare. But this problem might be solved with Kubernetes templating engine — Helm. We can define a generic Helm chart (template) and reused it across multiple applications only by injecting specific values into the template.

Is it problem solved?

To some extend yes, but here is a drawback of using Helm. In order to install or update any chart in a cluster you need to run a specific imperative command. In other words, in order to change state of a cluster you need to run a command that is specific for a given deployment.

In Java world we’re using Maven or for JavaScript we use npm. Both of these tools allows to run the same command across multiple projects to perform the same action like running tests— mvn test. Helm currently doesn’t support this feature, we’re not able to install, update or rollback all applications from an entire cluster with a single command.

But there is a helmfile, which might be a remedy.

Helmfile allows to declare a definition of an entire Kubernetes cluster in a single YAML file, it bundles multiple Helm releases (installation of Helm charts) and allows to tune a specification of each release depending on a type of environment (develop, test, production) on which you might want to deploy your applications.

The best approach would be to show how it works on an example. Before that we need to set up a working environment. If you want to follow my steps, you would need to install and configure:

Docker,
locally installed Kubernetes cluster — minikube,
Kubernetes command line tool — kubectl,
Helm (v3),
helmfile.

When everything is installed you can start up the minikube cluster and enable ingress addon:

> minikube start 
😄  minikube v1.8.1 on Ubuntu 18.04
✨  Automatically selected the docker driver
🔥  Creating Kubernetes in docker container with (CPUs=2) (8 available), Memory=2200MB (7826MB available) ...
🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.2 ...
▪ kubeadm.pod-network-cidr=10.244.0.0/16
❌  Unable to load cached images: loading cached images: stat /home/wojtek/.minikube/cache/images/k8s.gcr.io/kube-proxy_v1.17.3: no such file or directory
🚀  Launching Kubernetes ...
🌟  Enabling addons: default-storageclass, storage-provisioner
⌛  Waiting for cluster to come online ...
🏄  Done! kubectl is now configured to use "minikube"$ minikube addons enable ingress
🌟  The 'ingress' addon is enabled

Then you’ll need to edit you hosts file. Its location varies on OS:

When you find it add following lines:

172.17.0.2  adminer.k8s.com
172.17.0.2  kanban.k8s.com

To make sure that this config is correct you’ll need to check if an IP address of minikube cluster on your machine is the same as it’s above. In order to do that run the command:

> minikube ip
172.17.0.2

A preparations are done, let’s do some helmfile magic. In previous blog post — How to deploy application on Kubernetes with Helm — I’ve described how to create Helm charts for Kanban Board project. Here we will reuse them. Therefore copy-paste all the files from here — k8s-helm-helmfile/helm.

All folders — app , ingress , postgres — put inside a new one called charts and all YAML files put inside a new folder called values. Then add a new helmfile.yaml file to a root directory, so the resulting structure will looks as follows:

Let me briefly explain what each of these folders means:

./charts — here are all three Helms charts that are templates for each release — app (for adminer, kanban-ui, kanban-app), postgres and ingress,
helmfile.yaml — here will be a configuration for a helmfile, currently blank,
./values — a folder with a values that are specific for each application which will be deployed.

Before moving to filling a helmfile.yaml I want to adjust one thing. In a previous, native Helm, solution there is an ingress chart, which has a dependency to nginx-ingress Helm chart available on Helm Hub. Before installing it on a cluster we needed to download it (will be saved in .charts/ingress/charts folder) using separate helm command.

If we use helmfile, the last step would not be needed, because it will figure out and download all required dependencies inside each chart. So we could keep ingress chart as it is, but I would like to have a different approach this time. I would like to treat nginx-ingress chart as a separate Helm release, apart from the Ingress Controller configuration for routing.

Therefore remove following lines from the ./charts/ingress/Chart.yaml file:

dependencies:
  - name: nginx-ingress
    version: 1.36.0
    repository: https://kubernetes-charts.storage.googleapis.com/

Also you can delete the Chart.lock file entirely.

After those changes ingress Helm chart is responsible only for defining *Ingress Controller. The required Ingress backend service will be introduced with Helm release.

Now, finally we can move to arranging a helmfile.yaml .

First we need to indicate from which Helm repositories we would like to download charts. Because we will be downloading only one — stable/nginx-ingress — chart from a single repo, we put one entry:

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

Then we can move on to next section — defining the Helm releases. It will contain a list of all applications that helmfile will need to deploy. Starting from the first one — postgres:

releases:
  - name: postgres
    chart: ./charts/postgres
    values:
      - ./values/kanban-postgres.yaml

There is not that much here, but for clarity:

name — is a name of a release, we can name is whatever we like,
chart — tells helmfile where a base Helm chart is located,
values — because we’re using a Helm chart template, we would like to inject some values into it, so here is a list of YAML files that holds them.

Next, let’s define releases that are based on app chart:

releases:
  - name: adminer
    chart: ./charts/app
    values:
      - ./values/adminer.yaml

  - name: kanban-app
    chart: ./charts/app
    values:
      - ./values/kanban-app.yaml

  - name: kanban-ui
    chart: ./charts/app
    values:
      - ./values/kanban-ui.yaml

And next, we define a Helm release for an Ingress backend, which will be downloaded from a public repository:

releases:
  - name: ingress-backend
    chart: stable/nginx-ingress
    version: 1.36.0

In a chart section we provide information of which chart we would like to download and install. The stable/ prefix is a name of a repository that was defined in a repositories few moments ago.

In version we specify which version of a chart to use.

And finally a definition for Ingress Controller Helm release:

releases:
  - name: ingress-controller
    chart: ./charts/ingress
    values:
      - ./values/ingress.yaml

And if we merge all of them, helmfile.yaml will look as follows:

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

releases:
  - name: postgres
    chart: ./charts/postgres
    values:
      - ./values/kanban-postgres.yaml

  - name: adminer
    chart: ./charts/app
    values:
      - ./values/adminer.yaml

  - name: kanban-app
    chart: ./charts/app
    values:
      - ./values/kanban-app.yaml

  - name: kanban-ui
    chart: ./charts/app
    values:
      - ./values/kanban-ui.yaml

  - name: ingress-backend
    chart: stable/nginx-ingress
    version: 1.36.0

  - name: ingress-controller
    chart: ./charts/ingress
    values:
      - ./values/ingress.yaml

And now it is a fun part, with just two commands we can deploy all applications.

First, we need to add the repository to our instance of Helm (you’ll need to do that only once):

> helmfile repos
Adding repo stable [https://kubernetes-charts.storage.googleapis.com](https://kubernetes-charts.storage.googleapis.com)
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈

Then install all charts on a cluster:

> helmfile sync
Adding repo stable [https://kubernetes-charts.storage.googleapis.com](https://kubernetes-charts.storage.googleapis.com)
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈

Building dependency release=postgres, chart=charts/postgres
Building dependency release=adminer, chart=charts/app
Building dependency release=kanban-app, chart=charts/app
Building dependency release=kanban-ui, chart=charts/app
Building dependency release=ingress-controller, chart=charts/ingress
Affected releases are:
  adminer (./charts/app) UPDATED
  ingress-backend (stable/nginx-ingress) UPDATED
  ingress-controller (./charts/ingress) UPDATED
  kanban-app (./charts/app) UPDATED
  kanban-ui (./charts/app) UPDATED
  postgres (./charts/postgres) UPDATED

Upgrading release=postgres, chart=charts/postgres
Upgrading release=kanban-app, chart=charts/app
Upgrading release=adminer, chart=charts/app
Upgrading release=kanban-ui, chart=charts/app
Upgrading release=ingress-controller, chart=charts/ingress
Upgrading release=ingress-backend, chart=stable/nginx-ingress
Release "adminer" does not exist. Installing it now.
NAME: adminer
LAST DEPLOYED: Wed Apr 15 16:16:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^adminer$
Release "kanban-ui" does not exist. Installing it now.
NAME: kanban-ui
LAST DEPLOYED: Wed Apr 15 16:16:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^kanban-ui$
adminer default   1        2020-04-15 16:16:31.990139954 +0200 CEST deployed app-0.1.0 1.16.0

Release "kanban-app" does not exist. Installing it now.
NAME: kanban-app
LAST DEPLOYED: Wed Apr 15 16:16:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^kanban-app$
kanban-ui default   1        2020-04-15 16:16:31.968291217 +0200 CEST deployed app-0.1.0 1.16.0

Release "postgres" does not exist. Installing it now.
NAME: postgres
LAST DEPLOYED: Wed Apr 15 16:16:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^postgres$
kanban-app default   1        2020-04-15 16:16:31.958860307 +0200 CEST deployed app-0.1.0 1.16.0

postgres default   1        2020-04-15 16:16:31.958951797 +0200 CEST deployed postgres-0.1.0 1.16.0

Release "ingress-controller" does not exist. Installing it now.
NAME: ingress-controller
LAST DEPLOYED: Wed Apr 15 16:16:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^ingress-controller$
ingress-controller default   1        2020-04-15 16:16:31.958844983 +0200 CEST deployed ingress-0.1.0 1.16.0

Release "ingress-backend" does not exist. Installing it now.
NAME: ingress-backend
LAST DEPLOYED: Wed Apr 15 16:16:35 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w ingress-backend-nginx-ingress-controller'

An example Ingress that makes use of the controller:

apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: [www.example.com](http://www.example.com)
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
                path: /
        # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - [www.example.com](http://www.example.com)
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Listing releases matching ^ingress-backend$
ingress-backend default   1        2020-04-15 16:16:35.070513181 +0200 CEST deployed nginx-ingress-1.36.0 0.30.0

UPDATED RELEASES:
NAME                 CHART                  VERSION
adminer              ./charts/app             0.1.0
kanban-ui            ./charts/app             0.1.0
kanban-app           ./charts/app             0.1.0
postgres             ./charts/postgres        0.1.0
ingress-controller   ./charts/ingress         0.1.0
ingress-backend      stable/nginx-ingress    1.36.0

After couple minutes you will be able to enter http://kanban.k8s.com.

Conclusion

And that’s it! Very simple and everything — external repository set up and a list of each Helm releases is in one single file, which can be stored under version control system ( git for example).

And what is more to mention theses are not the only features of helmfile. More of them you can find in the official documentation.

To sum up the entire series, I hope you’ve learn something and you find out what are the difference for deployments in native Kubernetes, Helm and helmfile. I hope it’s more clear for you how each one of them look like and what are they pros and cons. You may comment it here or in other blog post what do you like and what you think is worth to correct from my side. I would be grateful.

As usual, here is a link to a source code — to this project:

wkrzywiec / k8s-helm-helmfile

Project which compares 3 approaches to deploy apps on Kubernetes cluster (using kubectl, helm & helmfile)

And to kanban-board project:

wkrzywiec / kanban-board

Single-click full-stack application (Postgres, Spring Boot & Angular) using Docker Compose

References

How to deploy application on Kubernetes with Helm

Wojtek Krzywiec — Thu, 14 Jan 2021 10:34:46 +0000

Cover image by Andrew Neel on Unsplash

In this blog post I present step-by-step how to deploy multiple applications on Kubernetes cluster using Helm.

This is a second part of my series on Kubernetes. It compares three approaches of deploying applications:

with kubectl — Deployment of multiple apps on Kubernetes cluster — Walkthrough
with Helm — this one,
with helmfile — How to declaratively run Helm charts using helmfile.

If you haven’t read it first one, I would advise to do that and then go back to this post. But if you prefer to jump right away to this post, don’t worry, I’ll briefly introduce you to the project.

Above picture presents the target solution. Inside the cluster there will be deployed frontend (kanban-ui) and backend (kanban-app) services together with postgres database. A source code for both microservices can be found in my GitHub repository — kanban-board.

Additionally I’ve added an adminer, which is a GUI client for getting inside the database.

If we focus on one of services, e.g. on kanban-ui we can see that it needs to create two Kubernetes objects — ClusterIP & Deployment. With plain approach, using kubectl we would needed to create two files for each Kubernetes object:

apiVersion: v1
kind: Service
metadata:
  name: kanban-ui
  labels: 
    group: backend
spec:
  type: ClusterIP
  selector:
    app: kanban-ui
  ports:
    - port: 80
      targetPort: 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kanban-ui
  labels:
    app: kanban-ui
    group: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kanban-ui
  template:
    metadata:
      labels:
        app: kanban-ui
        group: frontend
    spec:
      containers:
        - name: kanban-ui
          image: wkrzywiec/kanban-ui:k8s
          ports:
            - containerPort: 80

The same story is for any other application — they need at least two definition files. Some of them requires even more, like postgres, which needs to have PersistentVolumeClaims defined in addition. As a result for even small project we can end up with lots of files which are very similar to each other:

How to achieve the DRY in this case? Is it possible?

To reduce the number of YAML files we could merge them into one, for example, for kanban-ui it will look like this:

apiVersion: v1
kind: Service
metadata:
  name: kanban-ui
  labels: 
    group: backend
spec:
  type: ClusterIP
  selector:
    app: kanban-ui
  ports:
    - port: 80
      targetPort: 80
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kanban-ui
  labels:
    app: kanban-ui
    group: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kanban-ui
  template:
    metadata:
      labels:
        app: kanban-ui
        group: frontend
    spec:
      containers:
        - name: kanban-ui
          image: wkrzywiec/kanban-ui:k8s
          ports:
            - containerPort: 80

But it still doesn’t fix the major problem — how to avoid copy-pasting entire file just to replace couple of values? It would be great if there is a way to define a blueprint for both objects and then inject values into specific fields.

Luckily there is a solution! Helm to the rescue!

Accordingly to the official website — Helm is a package manager for Kubernetes. It helps deploy complex application by bundling necessary resources into Charts, which contains all information to run application on a cluster.

There are couple approaches how to work with Helm. One of them is to download publicly available charts from the Helm Hub. They are prepared by community and are free to use.

For instance, if we would like to run Prometheus on a cluster it would just easy as it’s described on this page — https://hub.helm.sh/charts/stable/prometheus — with the single command:

> helm install stable/prometheus

It contains some default configuration, but can be easily overridden with YAML file and passed during installation. The detailed example I’ll show in a minute.

But Helm is not only providing some predefined blueprints, you can create your own charts!

It’s very easy and can be done by a single command helm create <chart-name> , which creates a folder with a basic structure:

> helm create example
Creating example

In the templates/ folder there are Helm templates that with combination of values.yaml will result in set of Kubernetes objects.

Let’s create a first chart — postgres.

But just before that, boring installations and configurations (but promise, it’ll go quick). In my demo I’m using:

Docker,
locally installed Kubernetes cluster — minikube,
Kubernetes command line tool — kubectl,
Helm (v3).

When everything is installed you can start up the minikube cluster and enable ingress addon:

> minikube start

😄  minikube v1.8.1 on Ubuntu 18.04
✨  Automatically selected the docker driver
🔥  Creating Kubernetes in docker container with (CPUs=2) (8 available), Memory=2200MB (7826MB available) ...
🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.2 ...
▪ kubeadm.pod-network-cidr=10.244.0.0/16
❌  Unable to load cached images: loading cached images: stat /home/wojtek/.minikube/cache/images/k8s.gcr.io/kube-proxy_v1.17.3: no such file or directory
🚀  Launching Kubernetes ...
🌟  Enabling addons: default-storageclass, storage-provisioner
⌛  Waiting for cluster to come online ...
🏄  Done! kubectl is now configured to use "minikube"

> minikube addons enable ingress
🌟  The 'ingress' addon is enabled

Then you’ll need to edit you hosts file. Its location varies on OS:

When you find it add following lines:

172.17.0.2  adminer.k8s.com
172.17.0.2  kanban.k8s.com

To make sure that this config is correct you’ll need to check if an IP address of minikube cluster on your machine is the same as it’s above. In order to do that run the command:

> minikube ip
172.17.0.2

Now we can create first Helm chart:

> helm create postgres
Creating postgres

We will not need generated files inside ./templates folder, so remove them and also clear all the content inside values.yaml.

Now we can roll up our sleeves and define all necessary files to create chart for postgres database. For a reminder, in order to deploy it with kubectl we had to have following files:

They contain definitions of ConfigMap, Deployment, PersistentVolumeClaim and ClusterIP. Their full definitions could be found in a repository.

First, let’s create a template for postgres Deployment object, therefore inside the ./templates create a deployment.yaml file:

At first glance you might see these strange parts between two pairs of curly brackets, like {{ .Values.postgres.name }} . They are written in Go template language and are referring to a value located in a values.yaml which is located inside the root folder of a chart. For mentioned example Helm will try to match it with a value from values.yaml:

postgres:
    name: postgres

Another example. A value for a base Docker image, defined in image: {{ .Values.postgres.container.image }} will be taken from:

postgres:
    name: postgres
    container:
        image: postgres:9.6-alpine

And so on. We can define the structure inside this file whatever we like.

This database Deployment requires a PersistentVolumeClaim to be created to reserve some storage on a disk, therefore we need to create a Helm template — pvc.yaml inside the ./templates folder:

This one is shorter, and there is nothing new here.

Next template that we need to create is for ClusterIP, which will allow other Pods inside the cluster to enter the* Pod with postgres — service.yaml:

And finally ConfigMap template needs to be created — config.yaml. It’ll hold information about created database, user and its password (yes, not very secure, but for simple example it’s enough).

Here you might see a strange {{ -range ... }} clause, which can be translated as a for each loop known in any programming language. In above example, Helm template will try to inject values from an array defined in values.yaml:

postgres:
    config:
        data:
        - key: key
          value: value

Inside a template, there are only dummy values. If you want to have different ones you would need to override them during installation of this set of objects on a Kubernetes cluster.

The entire values.yaml presents as follow:

replicaCount: 1

postgres:
    name: postgres
    group: db
    container:
        image: postgres:9.6-alpine
        port: 5432
    service:
        type: ClusterIP
        port: 5432
    volume:
        name: postgres-storage
        kind: PersistentVolumeClaim
        mountPath: /var/lib/postgresql/data
        pvc:
          name: postgres-persistent-volume-claim
          accessMode: ReadWriteOnce
          storage: 4Gi
    config:
        name: postgres-config
        data:
        - key: key
          value: value

To finalize creating first chart we need to modify some metadata inside Chart.yaml file:

apiVersion: v2
name: postgres
description: A Helm chart for PostgreSQL database
type: application
version: 0.1.0
appVersion: 1.16.0
keywords:
  - database
  - postgres
home: https://github.com/wkrzywiec/k8s-helm-helmfile/tree/master/helm
maintainers:
  - name: Wojtek Krzywiec
    url: https://github.com/wkrzywiec

Then create a new kanban-postgres.yaml file which will hold some specific values. Put it outside the postgres chart folder so the file structure can be similar to that:

Inside the file put only values that will be specific for this deployment, i.e. postgres database credentials (all other values we can keep as default):

postgres:
    config:
        data:
        - key: POSTGRES_DB
          value: kanban
        - key: POSTGRES_USER
          value: kanban
        - key: POSTGRES_PASSWORD
          value: kanban

Everything is set up, we can now deploy postgres into a cluster. In a Helm world this step is called creating a new release:

> helm install -f kanban-postgres.yaml postgres ./postgres
NAME: postgres
LAST DEPLOYED: Mon Apr 13 16:13:16 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

> helm list
NAME     NAMESPACE REVISION  STATUS    CHART          APP VERSION
postgres default   1         deployed  postgres-0.1.0 1.16.0

> kubectl get deployments
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
postgres   1/1     1            1           2m14s

Cool, the postgres is up and running! Before moving forward let’s have a closer look on helm install command, what each of an argument means:

If you plan to use a chart only to create one Helm release you might wonder what is a value behind using Helm at all.

Let me show you that on another example. Let’s create a new chart called app — it will be a template chart for three different releases — adminer, kanban-app & kanban-ui.

> helm create app
Creating app

After removing unnecessary file from ./templates folder and cleaning the values.yaml create a deployment.yaml file:

Nothing new in particular, only some placeholders for values & two range loops to inject values from *ConfigMap*s or simple values to application container.

All 3 apps require to have also the ClusterIP objects to be deployed, therefore here is its template — service.yaml:

All default values in both templates are injected from values.yaml:

app:
    name: app
    group: app
    replicaCount: 1
    container:
        image: add-image-here
        port: 8080
        config: []
        env:
        - key: key
          value: value
    service:
        type: ClusterIP
        port: 8080

And to close a task of creating app chart — we need to define some metadata in the Chart.yaml file:

apiVersion: v2
name: app
description: A Helm chart for any application
type: application
version: 0.1.0
appVersion: 1.16.0
keywords:
  - app
  - java
  - javascript
  - angular
home: https://github.com/wkrzywiec/k8s-helm-helmfile/tree/master/helm
maintainers:
  - name: Wojtek Krzywiec
    url: https://github.com/wkrzywiec

Then, similarly to previous example, outside the app folder create YAML files with values which will override ones from values.yaml inside app chart.

An adminer.yaml file looks as follow:

app:
  name: adminer
  group: db
  container:
    image: adminer:4.7.6-standalone
    port: 8080
    env:
      - key: ADMINER_DESIGN
        value: pepa-linha
      - key: ADMINER_DEFAULT_SERVER
        value: postgres

A kanban-app.yaml file:

app:
  name: kanban-app
  group: backend
  container:
    image: wkrzywiec/kanban-app:k8s
    config:
      - name: postgres-config
    env: 
      - key: DB_SERVER
        value: postgres

A kanban-ui.yaml file:

app:
  name: kanban-ui
  group: frontend
  container:
    image: wkrzywiec/kanban-ui:k8s
    port: 80

A resulting file structure:

Now, in order to create three releases for each application use commands:

> helm install -f adminer.yaml adminer ./app
NAME: adminer
LAST DEPLOYED: Mon Apr 13 16:57:17 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

> helm install -f kanban-app.yaml kanban-app ./app
NAME: kanban-app
LAST DEPLOYED: Mon Apr 13 16:57:36 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

> helm install -f kanban-ui.yaml kanban-ui ./app
NAME: kanban-ui
LAST DEPLOYED: Mon Apr 13 16:57:54 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

> helm list
NAME       NAMESPACE REVISION  STATUS    CHART           APP VERSION
adminer    default   1         deployed  app-0.1.0       1.16.0     
kanban-app default   1         deployed  app-0.1.0       1.16.0     
kanban-ui  default   1         deployed  app-0.1.0       1.16.0     
postgres   default   1         deployed  postgres-0.1.0  1.16.0

> kubectl get deployments
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
adminer      1/1     1            1           112s
kanban-app   1/1     1            1           93s
kanban-ui    1/1     1            1           75s
postgres     1/1     1            1           45m

Perfect! There is one more thing to do — create a chart with Ingress Controller — a gateway to a cluster.

Like before, create a new chart:

> helm create ingress
Creating ingress

Remove all files from templates folder and clear content of values.yaml. This time, before moving straight away to defining a templates, let’s focus on to Chart.yaml first.

apiVersion: v2
name: ingress
description: A Helm chart for Ingress Controller
type: application
version: 0.1.0
appVersion: 1.16.0
keywords:
  - ingress
  - nginx
  - api-gateway
home: https://github.com/wkrzywiec/k8s-helm-helmfile/tree/master/helm
maintainers:
  - name: Wojtek Krzywiec
    url: https://github.com/wkrzywiec
dependencies:
  - name: nginx-ingress
    version: 1.36.0
    repository: https://kubernetes-charts.storage.googleapis.com/

Here there is a new section — dependencies— which has been added. It creates default backend services which enables the Ingress Controller features.

But it’s not the only thing we need to do here. This section only defines on what this chart depends on, it won’t download it automatically during the installation. We need to take care of it ourselves. To do that run the command:

> helm dependency update ./ingress/
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading nginx-ingress from repo [https://kubernetes-charts.storage.googleapis.com/](https://kubernetes-charts.storage.googleapis.com/)
Deleting outdated charts

Inside the ingress/charts folder a new file will appear — nginx-ingress-1.36.0.tgz .

Now we can define a template for Ingress — it will be located inside ./templates folder and will be called ingress.yaml:

The most interesting part is inside specification section. There are two nested loops there which allow to define multiple hosts and multiple paths for each host.

And also, here is a default values.yaml file:

ingress:
  name: ingress-service
  replicaCount: 1
  annotations: 
    class: nginx
  hosts:
    - host: chart-example.local
      paths: 
        - path: /
          backend:
            serviceName: serviceName
            servicePort: 8080

The resulting folder structure:

Outside the ingress chart we can now create an ingress.yaml file which will hold all routing rules for our cluster.

ingress:
  hosts:
    - host: adminer.k8s.com
      paths:
        - path: /
          backend:  
            serviceName: adminer
            servicePort: 8080
    - host: kanban.k8s.com
      paths: 
        - path: /api/
          backend:
            serviceName: kanban-app
            servicePort: 8080
        - path: /
          backend:
            serviceName: kanban-ui
            servicePort: 80

And now we’re able to create a Helm release:

> helm install -f ingress.yaml ingress ./ingress
NAME: ingress
LAST DEPLOYED: Tue Apr 14 07:22:44 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

> helm list
NAME       NAMESPACE  REVISION  STATUS    CHART          APP VERSION
adminer    default    1         deployed  app-0.1.0      1.16.0     
ingress    default    1         deployed  ingress-0.1.0  1.16.0     
kanban-app default    1         deployed  app-0.1.0      1.16.0     
kanban-ui  default    1         deployed  app-0.1.0      1.16.0     
postgres   default    1         deployed  postgres-0.1.0 1.16.0

> kubectl get deployments
NAME                                    READY  UP-TO-DATE  AVAILABLE
adminer                                 1/1    1           1           
ingress-nginx-ingress-controller        1/1    1           1           
ingress-nginx-ingress-default-backend   1/1    1           1           
kanban-app                              1/1    1           1           
kanban-ui                               1/1    1           1           
postgres                                1/1    1           1

Everything is set up, you can start testing it. Go to the http://kanban.k8s.com and you should get this page:

Congrats! 🍾🤘

Conclusions

In this short entry I’ve presented how, with Helm, you can reduce of copy-pasting tasks and can bring a one single template for deploying multiple applications on a Kubernetes cluster, which might be very handy in a microservice world. I hope you’ve enjoyed it.

But there is still one thing left, which holds us from establishing a fully declarative approach for an infrastructure. In order to deploy each application we still need to run imperative commands, like helm install. But luckily there is another tool — helmfile! But this one I’ll describe in my next story.

For now, here is my GitHub repository with a source code for this project:
wkrzywiec/k8s-helm-helmfile on github.com

And here is a source code of Kanban project:
wkrzywiec/kanban-board on github.com

DEV Community: Wojtek Krzywiec

Java Series: Flatmap

Problem statement

Solution

When to use it?

Unwrap & operate

Merge lists

Get value from nested Optional

Summary

References

Food Delivery App

Overview of My Submission

Submission Category: Microservice Mavens

Video Explainer of this project

Language Used

Link to Code

wkrzywiec / food-delivery-redis

Food Delivery app

Overview video (Optional)

How it works

Additional Resources / Info

Classic Den - Azure Trial Hackathon

Overview of My Submission

Submission Category:

Link to Code on GitHub

wkrzywiec / classic-den

Simple client and server app for publishing posts

Classic Den

About

Additional Resources / Info

Components

Frontend (Azure Static Web App)

Backend (Azure Functions)

GitHub repository

Screenshots

Automating quality checks for Kubernetes YAMLs

Workflow structure

Datree analysis

Testing on GKE

Conclusion

How to write good quality Python code with GitHub Actions

CodeFactor

wemake-python-styleguide

Codecov

Conclusion

Write better code with SonarQube

SonarQube

Prerequisites

Docker Compose

Maven configuration

Project’s POM config

Running the analysis

wkrzywiec / develop-env

Development tools in Docker container

wkrzywiec / kanban-board

Single-click full-stack application (Postgres, Spring Boot & Angular) using Docker Compose

References

How to fetch REST API every 2 seconds with RxJava 2

Prerequisites

Step 1. Set up app structure

Step 2. Create RxJava 2 Observable that prints something in a console each 2 seconds

Step 3. Add REST API service

Step 4. Combing all together

wkrzywiec / Random-Jokes

Testing RxJava 2 periodic capabilities

References

What was added to Java 8? Optional class

References

What was added to Java 8? Streams

Stream producing methods

Stream operating methods (intermediate methods)

Stream terminal methods

References

What was added to Java 8? Lambda expressions

Lambda expression

Method reference (::)

References

How to declaratively run Helm charts using helmfile

Conclusion

wkrzywiec / k8s-helm-helmfile