DEV Community: Felipe Henrique Gross Windmoller

Python FastAPI: Tutorial to Test HTTP Client Requests

Felipe Henrique Gross Windmoller — Mon, 13 May 2024 12:05:48 +0000

Disclaimer: I’m learning Python, so please correct me if I’ve written something wrong!

In this tutorial, we’ll explore how to test HTTP client calls in a FastAPI Python application using two different methods:

unittest.mock: This is a built-in Python library used for mocking objects in tests. It allows you to replace parts of your system under test and make assertions about how they have been used.
httpretty: This is a third-party library that allows you to mock HTTP requests at a low level by creating a fake HTTP server.

To run the tests, just execute this on the root of the project directory:



pytest

You should see output similar to this:



$ pytest
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/wsl/github/python-study/api-client
plugins: anyio-4.2.0
collected 3 items

tests/test_main.py .                                                                                                                                                                                      [ 33%]
tests/test_main_httppretty_mock.py .                                                                                                                                                                      [ 66%]
tests/test_main_magic_mock.py .                                                                                                                                                                           [100%]

=============================================================================================== 3 passed in 0.39s ===============================================================================================

This output indicates that all three tests passed successfully. The percentage in brackets shows the progress of the test run.

Versions of my modules



$ pip list
Package                            Version
---------------------------------- -----------
fastapi                            0.111.0
httpretty                          1.1.4
pydantic                           2.5.3
pytest                             8.2.0
requests                           2.31.0

Project structure



.
├── app
│   ├── __init__.py
│   └── main.py
└── tests
    ├── __init__.py
    ├── test_main_httppretty_mock.py
    └── test_main_magic_mock.py

__init__.py are just empty files

main.py



from pydantic import BaseModel
from fastapi import FastAPI, HTTPException
from typing import Union
import requests
import os
import logging

API_SERVER_URL = os.getenv("API_SERVER_URL", default="http://localhost:8010")
API_LOG_LEVEL = os.getenv("API_LOG_LEVEL", default="INFO")

# Configure basic logging
logging.basicConfig(level=API_LOG_LEVEL)
logger = logging.getLogger()

app = FastAPI()


class Item(BaseModel):
    name: str
    price: float
    is_offer: Union[bool, None] = None


@app.get("/")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}", response_model=Item)
def read_item(item_id: int, q: Union[str, None] = None):
    try:
        response = requests.get(
            f"{API_SERVER_URL}/items/{item_id}", params={"q": q})
        # Raises HTTPError for bad responses (4XX or 5XX)
        response.raise_for_status()
        item_data = response.json()
        item = Item(**item_data)  # Deserialize the JSON into an Item object
        return item
    except requests.RequestException as e:
        logger.error(f"Request failed: {str(e)}")  # Log the error
        raise HTTPException(status_code=500, detail=str(e))

test_main_magic_mock.py



from fastapi.testclient import TestClient
import pytest
from unittest.mock import patch, MagicMock
from app.main import app

client = TestClient(app)


@pytest.fixture
def mock_requests_get():
    # Create a MagicMock object to mock requests.get
    with patch('requests.get') as mock_get:
        # Prepare a mock response object with necessary methods
        mock_response = MagicMock()
        mock_response.json.return_value = {
            "name": "Sample Item",
            "price": 100.0,
            "is_offer": None
        }
        mock_response.raise_for_status = MagicMock()
        mock_get.return_value = mock_response
        yield mock_get


# The mock_requests_get fixture is automatically used here.
def test_read_item(mock_requests_get):
    response = client.get("/items/1")
    assert response.status_code == 200
    assert response.json() == {
        "name": "Sample Item",
        "price": 100.0,
        "is_offer": None
    }
    # Assert if requests.get was called correctly
    mock_requests_get.assert_called_once_with(
        "http://localhost:8010/items/1",
        params={'q': None}
    )

test_main_httppretty_mock.py



import httpretty
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)


@httpretty.activate
def test_read_item():
    # Mocking the external API call
    httpretty.register_uri(
        httpretty.GET,
        "http://localhost:8010/items/1",
        body='{"name": "Sample Item", "price": 100.0, "is_offer": null}',
        content_type="application/json",
        status=200  # Define the expected status code
    )

    # Call the endpoint
    response = client.get("/items/1")

    # Assertions
    assert response.status_code == 200
    assert response.json() == {
        "name": "Sample Item",
        "price": 100.0,
        "is_offer": None
    }

Strategic Moves: Exploring the Chessboard of Domain-Driven Design

Felipe Henrique Gross Windmoller — Wed, 06 Sep 2023 14:25:26 +0000

Software development and a game of chess. At first, it might not be obvious to compare these two subjects and they might seem fundamentally different for you; however, these two topics are based on navigating an abundance of possibilities and choices and have more similarities than you could anticipate. In this article, I will take you through the strategic and tactical principles of Domain-Driven Design (DDD) comparing it to strategies and tactics of a chess match.

Chess Strategy

The objective of chess is very clear: we want to win the game! However, this is challenging. Let’s imagine the beginning of a game: each Pawn can move one or two squares and each Knight can also make two different moves. So, there are twenty different options to make the first chess move. To make things more complicated, after our first move it’s our opponent’s turn and then we must come up with a new decision. The number of possibilities increases exponentially as we try to calculate the movements that can be done depending on our opponent’s decision. Some great masters are reputed to predict 15 to 20 moves ahead.

So, how do we know what is a good opening movement? Chess strategy can guide us. In the chess strategy the game is divided into three phases: opening, middle and end game. In the opening, we must follow these principles: control the center of the board, develop our minor pieces — the Knights and Bishops — castling to protect our King and move our Queen to connect our Rooks. After accomplishing these principles, we finish the opening phase and enter the middle game. Studies have proved that following these principles improve our chances of winning the game.

Having the opening strategy clarified it looks easier to choose our first move. For example, we can move our Queen Pawn to d4, and it seems perfect according to strategy principles and it meets two strategy objectives. First, the Pawn now is in position to control c5 and e5 squares and so we are in the center of the board and one step closer to get control of it. Second, it gives space for our Bishop and Queen to move.

Strategy and Heuristics

As stated before, the number of possible moves in a chess game can be overwhelming and strategy helps us find faster and better ones. The mental technique of taking shortcuts to solve complex problems fast is called heuristic. It provides us with results that are good enough for our immediate goals. This is useful when we need to make a decision, but we would take too much time analyzing all the information and possibilities. Bringing this to our discussion here imagine it is your turn to make a move in the game and there are around 20 different possibilities, however now you know there are strategies you can follow and reduce this number to 5. The idea is that heuristic would help you choose within these 5 possibilities, making it possible for you to make a good move, not necessarily the best, but fast since you had less options.

Domain-Driven Design Strategy

It has been said that the objective of a chess game is to win. In comparison, what is our goal in an IT project? The simplest answer is that we must resolve a business problem with a software solution. Therefore, the most important element in a software project is to understand the problem, because no matter how good our implementation is, if we don’t understand the business requirements it will be impossible to achieve our purpose.

Furthermore, as in chess, in the IT world we are also overwhelmed with information and different possibilities. For example: how do we evaluate our options and choose the best one to build a software that can suit the business problem? Depending on the domain complexity the number of options is huge and it’s difficult to make a decision.

Domain-driven design (DDD) is a technique which was created by Eric Evans in 2003 to help us improve the communication between businesspeople and IT and give us heuristics to decide the best software implementation. In the same way which chess strategy helps us choose what piece we should move to start the game, DDD strategies also help us make choices to build a software that serves the business needs.

The strategies, simplifying, are to identify the business domain and its subdomains, build the ubiquitous language and design the bounded contexts. We should first understand these concepts and then see how they can help us build software projects.

Business Domain and Subdomains

The business domain is the market in which a company is inserted such as investments, credit card, marketing. It is divided into subdomains which are the business building blocks working together and allowing the company to exist. Let’s take a retail company for example, some subdomains could be customers, accounting, and stocking.

There are three types of subdomains: core, generic and supporting. Core subdomains are the important ones because they differentiate a company from others and it’s what makes the company special and successful. They are complex and are always changing to generate improvements.

Supporting subdomains are the simple ones, they don’t change too much and don’t make the company unique. However, we must build them ourselves because there are no pre-made solutions available, or it’s just an easier solution rather than developing the integration with a ready-made one.

Generic subdomains are complex but not unique and so they don’t make the company special. They are problems that have been solved before, even by other industries, and there are solutions out there that we can use, as software you can buy or use for free. For instance, think about encryption, unless this is the core business of the company it is smarter to use something that is already made instead of developing it from scratch.

In chess we wonder which piece we should move first. In a software project we ask what type of architecture we should use. Perhaps, layered architecture? Hexagonal architecture? Command and query responsibility segregation (CQRS)? Strategic DDD gives us some heuristics to decide. Depending on the subdomain type we can pick the best architecture. For a simple supporting subdomain, we can go with layered architecture but for a core subdomain, which is complex and changes often, hexagonal architecture could be a better option.

Ubiquitous Language and Bounded Contexts

As mentioned earlier, communication between businesspeople and IT is challenging. Often, there are two separate languages: one understood by domain experts, and another used by software programmers. This issue increases the cognitive load on teams and makes misunderstandings more likely. It’s a bit like the telephone game kids play. The requirements from businesspeople need to go from the language domain experts use to the one programmers understand, and, frequently, this translation is far from perfect.

The strategy proposed by domain-driven design to tackle this problem of communication is to adopt the same terminology used by the business specialists in everywhere in a software project. This means that the same terms used in the meetings between the domain experts and IT people will show up in domain models, tables, and programs. That’s why the name of this pattern is ubiquitous language, which is a single language used everywhere to keep things clear and consistent.

However, when people talk, some words can have different meanings depending on the situation. For instance, in a sales context, “customer” might refer to someone who purchases goods, while in a support context, it could refer to someone seeking assistance. This dubiousness isn’t a problem for human conversations, but software programs don’t admit ambiguities, everything must have one clear definition.

To overcome this problem, strategic DDD has the bounded context pattern. The idea is just like the name suggest, we delimit a context to the meaning of our terms. In our previous example of the term “customer”, we would have a bounded context named “sales” and another one called “support”. Therefore, inside de “sales” bounded context, the word “customer” is going to have exactly one meaning.

Microservices and Domain-Driven Design

In a microservices architecture, getting the size of our services right is very important. If they are too big, they become individually very complex and difficult to maintain. On the other hand, if they are too small, the global complexity of lots of tiny microservices communicating between them is going to be high. DDD helps us with some heuristics: a service shouldn’t be bigger than the bounded context, or else our language won’t be clear, and the software could get really complicated. Usually, the size of a microservice fits well with a subdomain. The application boundaries will follow how the business works, and new features from businesspeople typically get added to a single microservice, without causing a ripple effect where multiple microservices will need to be changed and deployed together.

Strategies and Tactics

Strategies serve as guiding principles that steer us towards optimal choices, offering a roadmap to attain overarching objectives. Whether aiming to triumph in a chess match or excel in a software project, strategies outline the trajectory for long-term success. In contrast, tactics are more about quick results. They focus on getting things done in a short amount of time.

Chess Tactics

Chess tactics are clever tricks that players use to catch their opponents off guard or win quickly. One famous tactic is the double attack. This happens when a single move creates two problems for the opponent, and they can’t solve both. Look at the picture below: the Knight is attacking both the King and the Rook at the same time. The opponent must move the King, and in the next turn, he will lose the Rook.

Domain-Driven Design Tactics

While strategic DDD provides us with guidance for comprehending business requirements and crafting software solutions aligned with company objectives, tactical domain-driven design operates within the practical realm of coding programs. The three most recognized tactical patterns of DDD are value objects, entities, and aggregates.

Value objects are important components in programming that help us represent specific pieces of information, like dates or money amounts, in a clear and consistent way. They’re like little containers for data and rules that make our code easier to understand and maintain. Value objects are special because once we create them, we can’t change them, which keeps our data reliable. They’re like building blocks that help us create strong and reliable software. Value objects can be identified by the composition of their values.

Entities are essential in programming as they represent the core elements in a system that have unique identities and behaviors. Just like characters in a story, entities hold important information and can interact with each other. These objects help us model real-world concepts, like customers or orders, making our software more reflective of the actual situation. Because entities have individual identities, they’re crucial for tracking and managing data accurately. In programming, entities act as the main characters that drive our applications.

Aggregates are vital in programming because they group together related entities and value objects, creating a more organized structure for our code. Think of them as containers that hold everything needed to handle a specific part of our system. Aggregates help us manage complex data and interactions by providing a clear boundary for operations. This makes our code more understandable and reduces the chances of errors. In programming, aggregates act as the guardians of consistency and integrity, ensuring that our data stays accurate and dependable.

Final Words

To wrap things up, both DDD strategy and chess strategy guide our choices. Imagine strategy as a big plan for the long run, and tactics as practical steps for right now. It’s like having a map for the journey and tools for the adventure. Despite the differences between chess and IT, they both share a common goal: a desire for success. In chess, we aim to win the match, and in IT, we strive to succeed in our projects.

Quarkus Mutual TLS demo

Felipe Henrique Gross Windmoller — Mon, 01 May 2023 15:52:02 +0000

Project quarkus-mutual-tls-demo available on GitHub

Demonstration project that shows how to configure Transport Layer Security (TLS) on Quarkus server and Quarkus client applications.

There are two applications to this demo: server and client.

First, I explain the concepts of https, tls and mutual tls.

Then, I execute a few tests showing step by step how we configure and what are the problems that occur when we don't configure the applications properly.

Index

Concepts
Certificate Generation
Tests:
- Server with certificate and client without truststore configured
- Server with certificate and client with truststore configured
- Server with mutual TLS and client don't inform identity
- Server with mutual TLS and client inform its certificate
Credits

Concepts

HTTPS

HTTPS stands for Hypertext Transfer Protocol Secure, which is a protocol used to establish a secure and encrypted connection between a web browser (or other client) and a web server.
When you connect to a website using HTTPS, your browser and the server exchange cryptographic keys to establish a secure connection. This ensures that any data transmitted between the client and server is encrypted and protected from interception or tampering.¹

To estabilish a https connection, the server must have a server certificate. When the client access the server, it receives the server certificate and check if this certificate is valid and issued by a trusted Certificate Authority (CA).

TLS

Transport Layer Security (TLS) is a cryptographic protocol designed to provide communications security over a computer network. The protocol is widely used in applications such as email, instant messaging, and voice over IP, but its use in securing HTTPS remains the most publicly visible.²

Transport Layer Security (TLS) certificates—most commonly known as SSL, or digital certificates—are the foundation of a safe and secure internet. TLS/SSL certificates secure internet connections by encrypting data sent between your browser, the website you’re visiting, and the website server. They ensure that data is transmitted privately and without modifications, loss or theft.³

Mutual TLS occurs when the client requires identification from the server and the server requires identification from the client as well.

Certificate generation

Let's generate the server and clients certificates.

Then we will create a single keystore containing all client certificates.

The files are already on the project, if you wan't to execute the keytool commands, first delete the files.

Execute the following commands on the root folder of the project.

Server Certificate



$ keytool -genkeypair -storepass server-password -keyalg RSA -keysize 2048 -dname "CN=server" -alias server -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -validity 365000 -keystore server/server-keystore.jks

Client "A" Certificate



$ keytool -genkeypair -storepass client-a-password -keyalg RSA -keysize 2048 -dname "CN=client" -alias client -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -validity 365000 -keystore client/client-a-keystore.jks

Client "B" Certificate



$ keytool -genkeypair -storepass client-b-password -keyalg RSA -keysize 2048 -dname "CN=client" -alias client-b -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -validity 365000 -keystore client/client-b-keystore.jks

Create the server truststore

Create the server truststore file containing all the client keystores.

Generating a server truststore file:



$ keytool -genkeypair -storepass authorized-clients-password -keyalg RSA -keysize 2048 -dname "CN=client" -alias authorized-clients -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -validity 365000 -keystore server/server-truststore-clients.jks

Adding keystore from client "A" to server truststore:



$ keytool -importkeystore -srckeystore client/client-a-keystore.jks -srcstorepass client-a-password -destkeystore server/server-truststore-clients.jks -deststorepass authorized-clients-password

Adding keystore from client "B" to server truststore:



$ keytool -importkeystore -srckeystore client/client-b-keystore.jks -srcstorepass client-b-password -destkeystore server/server-truststore-clients.jks -deststorepass authorized-clients-password

Create the client truststore



$ cp server/server-keystore.jks client/client-truststore.jks

Test - Server with certificate and client without truststore configured

In this cenario, the server only accepts https connections, but the server certificate isn't present in the client trustore Certificate Authorite control.

Enabling TLS on the server side:



# Disallowing http access
quarkus.http.insecure-requests=disabled

# Configure https port
quarkus.http.ssl-port=8445

# Server Certificate
quarkus.http.ssl.certificate.key-store-file=./server-keystore.jks
quarkus.http.ssl.certificate.key-store-password=server-password

Start server:

Enter the server folder
Execute $ ./mvnw quarkus:dev

Test from Quarkus Client

Start client:

Enter the client folder
Execute $ ./mvnw quarkus:dev

Execute client endpoint that consumes server endpoint:



$ curl localhost:8080/hello

Try to access this endpoint from the client without the Trust Store configuration, we receive this error:



jakarta.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
...
Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
        at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:127)
        ... 25 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
...
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
...
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Test from curl



$ curl https://localhost:8445
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Test from curl allowing insecure server connections



$ curl -k https://localhost:8445/hello

Hello from server

Test - Server with certificate and client with truststore configured

Configure client truststore:



# Rest client specific Trust Store
quarkus.rest-client.server-api.trust-store=./client-truststore.jks
quarkus.rest-client.server-api.trust-store-password=server-password

Test from Quarkus Client

Start client:

Enter the client folder
Execute $ ./mvnw quarkus:dev

Execute client endpoint that consumes server endpoint:



$ curl localhost:8080/hello

Hello from server

Test - Server with mutual TLS and client don't inform identity

Include requiring of client identification (quarkus.http.ssl.client-auth) and the truststore of the server, that are the identities that the server will trust:



# Disallowing http access
quarkus.http.insecure-requests=disabled

# Configure https port
quarkus.http.ssl-port=8445

# Server Certificate
quarkus.http.ssl.certificate.key-store-file=./server-keystore.jks
quarkus.http.ssl.certificate.key-store-password=server-password

# Require client identification
quarkus.http.ssl.client-auth=required
quarkus.http.ssl.certificate.trust-store-file=./server-truststore-clients.jks
quarkus.http.ssl.certificate.trust-store-password=authorized-clients-password

Start server:

Enter the server folder
Execute $ ./mvnw quarkus:dev

Test from Quarkus Client

Start client:

Enter the client folder
Execute $ ./mvnw quarkus:dev

Execute client endpoint that consumes server endpoint:



$ curl localhost:8080/hello

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

Test from curl



$ curl -k https://localhost:8445

curl: (56) OpenSSL SSL_read: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0

Test - Server with mutual TLS and client inform its certificate

Add the client certificate:



# Rest client specific Trust Store
quarkus.rest-client.server-api.trust-store=./client-truststore.jks
quarkus.rest-client.server-api.trust-store-password=server-password

# Rest Client specific Key Store
quarkus.rest-client.server-api.key-store=./client-a-keystore.jks
quarkus.rest-client.server-api.key-store-password=client-a-password

Start client:

Enter the client folder
Execute $ ./mvnw quarkus:dev

Execute client endpoint that consumes server endpoint:



$ curl localhost:8080/hello

Hello from server

You can add any of the client keystores that are on the truststore of the server, that are client-a-keystore.jks (client-a-password) or client-b-keystore.jks (client-b-password).

Credits

https://quarkus.io/guides/security-authentication-mechanisms-concept#mutual-tls

https://quarkus.io/blog/quarkus-mutual-tls/

https://quarkus.io/guides/http-reference

Uploading a file through Swagger in Quarkus

Felipe Henrique Gross Windmoller — Sat, 11 Jun 2022 01:56:28 +0000

In this project I show a way where you can upload a file through the SWagger-UI in Quarkus.

Maximillian Arruda showed me how to to this and I decided to create this project to help other people who might also want to use this feature.

How it works

Start the application with:



./mvnw compile quarkus:dev

Access the swagger-ui at http://localhost:8080/q/swagger-ui and add the files you want to upload (the assets folder has two files for test):

Click execute.

You'll receive one 202 Accepted.

The application sends the Buffer from the files to a Vert.X Bus Event, where they are processed and the information is displayed.

In the developer are of the browser (usually when you press F12), you can see the HTTP POST request with all data being sent to the server:

In the logs of the application, you'll see that the HTTP endpoint is provided by by one executor thread and the processing of the file is done by one worker thread:

Under the hood

This Java class does all the magic of the swagger-ui and send the events to the Vert.X event bus:



package org.acme;

import java.io.BufferedReader;
import java.io.IOException;
import java.nio.file.Files;
import java.util.List;

import javax.enterprise.context.RequestScoped;
import javax.inject.Inject;
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.eclipse.microprofile.openapi.annotations.enums.SchemaType;
import org.eclipse.microprofile.openapi.annotations.media.Schema;
import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
import org.jboss.logging.Logger;
import org.jboss.resteasy.reactive.MultipartForm;
import org.jboss.resteasy.reactive.RestForm;
import org.jboss.resteasy.reactive.multipart.FileUpload;

import io.vertx.core.eventbus.EventBus;

@RequestScoped
@Path("upload")
public class UploadResource {

    private static final Logger LOG = Logger.getLogger(UploadResource.class.getName());

    @Inject
    EventBus bus;

    @POST
    @Consumes(MediaType.MULTIPART_FORM_DATA)
    @APIResponse(responseCode = "202")
    public Response upload(
            @MultipartForm MultipartBody body) throws IOException {

        LOG.info("upload() quantity of files + " + body.files.size());

        for (FileUpload file : body.files) {

            LOG.info("filePath " + file.filePath());

            BufferedReader br = Files.newBufferedReader(file.filePath());

            bus.send("file-service", br);
        }

        LOG.info("upload() before response Accepted");

        return Response
                .accepted()
                .build();
    }

    // Class that will define the OpenAPI schema for the binary type input (upload)
    @Schema(type = SchemaType.STRING, format = "binary")
    public interface UploadItemSchema {
    }

    // Class that will be used to define the request body, and with that
    // it will allow uploading of "N" files
    public class UploadFormSchema {
        public List<UploadItemSchema> files;
    }

    // We instruct OpenAPI to use the schema provided by the 'UploadFormSchema'
    // class implementation and thus define a valid OpenAPI schema for the Swagger
    // UI
    @Schema(implementation = UploadFormSchema.class)
    public static class MultipartBody {
        @RestForm("files")
        public List<FileUpload> files;
    }

}

This class consumes the events from the event bus and displays the data:



package org.acme;

import java.io.BufferedReader;
import java.io.IOException;

import javax.enterprise.context.ApplicationScoped;

import org.jboss.logging.Logger;

import io.quarkus.vertx.ConsumeEvent;

@ApplicationScoped
public class FileService {

    private static final Logger LOG = Logger.getLogger(FileService.class.getName());

    @ConsumeEvent(blocking = true, value = "file-service")
    public void processFile(BufferedReader br) throws InterruptedException {

        LOG.info("processFile() begin");

        try (br) {
            String currentLine = null;
            while ((currentLine = br.readLine()) != null) {
                LOG.info("currentLine " + currentLine);
            }
        } catch (IOException e) {
            LOG.error("Error", e);
        }

        LOG.info("processFile() end");

    }

}

Thanks for reading!

Controlling Hibernate Commit programmatically for batch execution on Quarkus

Felipe Henrique Gross Windmoller — Thu, 31 Mar 2022 17:59:58 +0000

Problem with many commits during batch execution

If you need to process a huge amount of data on a batch execution, it's important to control the number of commits that are done by Hibernate.

Commit is one of the most expensive database operations.

Using the @Transactional annotation

With Quarkus, you can control the execution of the commit using the @Transactional annotation. When the method annotated with @Transactional finishes successfully, Hibernate sends one commit to the database.

import javax.transaction.Transactional;
//
@Transactional(rollbackOn = Exception.class)
public void processClientOrders(Client client) {
     client.updateOrders();
     client.insertHistory();
     client.updateClient();
}

In the above example, when the processClientOrders() finishes, Hibernate sends one commit to the database.

Difficulty in using @Transactional annotation to control batch execution

On a batch execution, it's a good idea to control the numbers of commits using the quantity of processed items or time between commits.

On my point of view, is a bit difficult to control this using the @Transactional annotation, specially to control the quantity of time between commits.

Executing commit programmatically

The contribution I try to give here is one idea to control the commits using the programmatic way.

Define the commit parameters on you application

This item is not obligatory, but I think is a good idea. Using this approach you can tune your values at deploy time.

Define two parameters on your application.properties:

# Minimum quantity of items processed to execute commit
app.quantity.of.items.to.commit=1000

# Minimum quantity of time (milliseconds) to execute commit
app.quantity.of.time.ms.to.commit=1000

Control the Transaction programmatically

To control the Transaction programmatically you need to inject the UserTransaction and then begin and commit the transaction manually.

@Inject
UserTransaction transaction;

@ConfigProperty(name = "app.quantity.of.items.to.commit")
int quantityItemsCommit;

@ConfigProperty(name = "app.quantity.of.time.ms.to.commit")
int quantityTimeCommit;

public void batchProcess(List<Client> listOfClients)
                 throws SystemException,NotSupportedException, 
                 SecurityException, IllegalStateException,
                 RollbackException, HeuristicMixedException,  
                 HeuristicRollbackException {
  var beforeCommit = Instant.now();
  var quantityOfClientsPendingCommit = 0;
  transaction.begin();

  for (Client client : listOfClients ) {
    processClientOrders(client);
    var now = Instant.now();
    var delay = Duration.between(beforeCommit, now).toMillis();
    if (++quantityOfClientsPendingCommit >= quantityItemsCommit  
        || delay >= quantityTimeCommit) { 
      transaction.commit();
      transaction.begin();
      beforeCommit = Instant.now();
      quantityOfClientsPendingCommit = 0;
    }
  }
  transaction.commit();
}

private void processClientOrders(Client client) {
     client.updateOrders();
     client.insertHistory();
     client.updateClient();
}

The above code executes commit when the quantity of clients processed is greater or equal the limit or the time after the last commit is greater or equal the limit.

Final words

I hope that this article can add something and if you find some mistake that I did or have one idea to improve this concept, please comment.

Credits

Stack Overflow post about calculating time in milliseconds

Red Hat documentation about Managing JTA transactions programmatically

POC to parallelize long time processes using Kafka and Quarkus

Felipe Henrique Gross Windmoller — Mon, 13 Dec 2021 21:26:48 +0000

poc-kafka-quarkus

This project is available on GitHub

POC to parallelize long time processes using Kafka and Quarkus.

Imagine you have a big amount of items to process and each one takes a long time to finish.

This POC proposes one solution to diminish the overall process time.

The idea is simple, one application, the producer, publishes messages on a Kafka topic and another application, the processor, reads these messages and process them.

If the messages don't need to be processed in an ordered way, we can configure the application, using SmallRye Reactive Message, to create a pool of threads and process multiple messages simultaneously.

Kafka also allows that we consume messages from its topic from multiple instances of our application. To achieve this, we must configure the partition number of the topic. For instance, if our topic has a partition number of three and we run four instances of our application, three instances will be able to process one specific partition and the fourth will remain idle.

In this POC I use both ideas, that is I run the processing application in multiple instances and each instance has its own thread pool that processes multiple messages simultaneously.

POC in action

Just run the application, access the Swagger-UI to inform the quantity of items to process, execute it and check the results on the console log of the producer and the processor instances.

Swagger

The Swagger-UI is on http://localhost:8080/q/swagger-ui

Producer sending messages

Processor instances receiving and processing messages

1st instance with partition "2"

2nd instance with partitions "0" and "1"

Instructions to run

Running everything with Docker Compose

Compile the two projects

Enter the processor folder and run:



$ ./mvnw package

Enter the producer folder and run:



$ ./mvnw package

Run the Docker Compose

On the root folder, run:



$ docker-compose up

If you want to run multiple instances of the processor, just pass the scale argument:



$ docker-compose up --scale processor=2

The NUM_PARTITIONS parameter of Kafka in the docker-compose file says how many partitions the topics will have.

Running in development mode

Start producer and processor

Enter the processor folder and run:



$ mvn quarkus:dev

Enter the producer folder and run:



$ mvn quarkus:dev -Ddebug=5006

In this mode, Quarkus will automatically download one Kafka image and run it for you.

Kafka concepts

Partitions

The num.partitions parameter defines how many partitions the topic will have.

Consumer groups

The consumer group is defined by one id.

One consumer group will receive all messages sent to a topic.

One consumer group can have 'n' instances of applications running.

Each instance of the consumer group will process messages from some partitons of the topic.

Consumer groups and partitions

Each instance of the consumer group will be able to read one or more partitions of the topic.

If you have more consumers than partitions, some consumers will remain idle.

Quarkus and SmallRye Reactive Message tips

Processing messages in multiple worker threads

If the messages don't need to be processed in order, you can use the following annotation:



@Incoming("extraction-requests")
@Blocking(ordered = false, value = "my-custom-pool")
public void read(Client client) {

Inform the number of threads of your my-custom-pool in your application.properties:



smallrye.messaging.worker.my-custom-pool.max-concurrency=3

Quarkus dev mode

You can inform the number of partitions the topic will have in your Kafka test container with this parameter, informing your topic name (in this case, extraction-requests) :



quarkus.kafka.devservices.topic-partitions.extraction-requests=3

Credits

https://quarkus.io/guides/kafka

https://strimzi.io/docs/operators/latest/using.html

smallrye

https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.13/index.html

https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.1/advanced/blocking.html

https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.1/emitter/emitter.html#_emitter_and_channel

Shopping List

Felipe Henrique Gross Windmoller — Tue, 12 Oct 2021 22:22:57 +0000

The Shopping List project is open source and you can find it on GitHub.

There, you are going to find instructions on how to build it and how to run it locally.

The frontend is written in Angular and the backend in Quarkus. The database is PostgreSQL. They are deployed as containers on Heroku and the HTTP communication is protected by JWT tokens.

I describe the technique used to deploy them as containers on Heroku in this article on medium.

You can access this demo version running on Heroku and try it right now. But as this is only a demo, please don't use a real password there!

The application get dormant after some time without use, so the first access can take some extra time.

Business description

Shopping List is a web application that was created especially for mobile devices.

You'll be able to organize the products you want to buy and do your shopping more quickly.

Once you create one account, you can create shopping lists and add products to them.

When you're buying products at the market, you can check the products updating your list and switch from one shopping list to other very easily.

You can, for instance, create one shopping list for fruits, other for cleaning products and another for grocery. So, in the top menu, you will be able to switch between these shopping lists while you're checking the products you already bought.

Architecture

Demo version running on Heroku

Application

API swagger

Angular

I used Angular Material and Angular Flex-Layout in this project.

Quarkus

On the Quarkus project I used Hibernate Panache to control the database and SmallRye JWT to protect the endpoints.

Perhaps you can find it interesting the Swagger-UI, created automatically by the SmallRye OpenAPI, that has the possibility of doing one Authentication to test the endpoints protected by JWT.

Final words

Thanks for reading this article and feel free to talk with me in case you got interested in this project!

Create one Docker image of Angular + Nginx with dynamic API url

Felipe Henrique Gross Windmoller — Fri, 09 Jul 2021 19:37:52 +0000

In this tutorial, I explain one workaround that I did to create one Docker image of one Angular application which has one dynamic API URL.

The article How To Build An Angular App Once And Deploy It To Multiple Environments - Solution 4: Override environment file values explains how you can do this.

But, as I'm far from a Docker expert, I spent some time to create the Dockerfile to do this :)

Just quoting the article:

The idea is to use Angular’s environment.ts (for local development) and environment.prod.ts (for all other stages) with placeholder values which are overwritten per deployment:

export const environment = {
  apiUrl: 'MY_APP_API_URL',
  stage: 'MY_APP_STAGE',
};

If our pod is started we can then run the following script, for example in a Dockerfile, that overrides these placeholder values:

#!/bin/sh
# replace placeholder value in JS bundle with environment specific values
sed -i "s#MY_APP_API_URL#$API_URL#g" /usr/share/nginx/html/main.*.js

My contribution

Here is the Dockerfile I created to run this workaround:

FROM nginx:latest

COPY dist/your-app-name /usr/share/nginx/html

CMD ["/bin/bash", "-c", \
"echo API_URL=[$API_URL], && \
sed -i s#MY_APP_API_URL#$API_URL#g /usr/share/nginx/html/main.*.js && \
nginx -g 'daemon off;'"]

Build your project:

$ ng build

Place this Dockerfile into your Angular root folder and run this to build your image:

$ docker build -t angular/your-app-name .

After this, just run your container passing the API_URL variable:

$ docker run -p 80:80 --name your-app-name -e API_URL=http://localhost:8080 angular/your-app-name

POC where Angular requests URL of image and Quarkus provides it through a raw copy and base64

Felipe Henrique Gross Windmoller — Sun, 27 Jun 2021 19:19:06 +0000

A simple POC where Angular requests the URL of an image to the Quarkus back end and then renders the image in the screen. Quarkus returns the image in two ways: (1) a simple copy from the URL of the image; (2) the image encoded in Base64 with the Media Type.

The project is available on GitHub poc-http-copy-images.

Hypothetical problem to solve

Let's imagine a hypothetical situation where only the backend have access to the URL of some images and you want to render these images on the frontend.

In this POC, I propose two ways of solving this problem:

The backend access the URL of the image and expose the same Response object;
The backend access the URL of the image, encode it into Base64 and expose it into a JSON object with the Media Type.

Running the POC

Mock Server

Optionally, you can run a Quarkus mock server which will provide three endpoints with the below images:

*** Optionally, because you can choose any image from the internet to run the test.

Enter inside the mock-server folder and type:



$ mvn quarkus:dev -Ddebug=5006

The swagger will be available at:

http://localhost:8090/q/swagger-ui

Quarkus backend

Enter inside the quarkus folder and type:



$ mvn quarkus:dev -Ddebug=5006

The swagger will be available at:

http://localhost:8080/q/swagger-ui

Angular frontend

Enter inside the angular folder and type:



$ npm install
$ ng serve

The front end will be available at:

http://localhost:4200/

Ok, in this screen, just put the URL of any image you desire (of the mock or the internet) and click on Load Image:

After clicking in the button, the image will be displayed in two columns, the left one is the raw copy from the URL and the right one is the Base64 version:

Any new image searched will be shown and the previous ones will slide down: