DEV Community: Nándor Holozsnyák

Testing with Maven - Organizing unit and integration tests

Nándor Holozsnyák — Thu, 08 Jun 2023 20:36:00 +0000

Testing with Maven - Organizing unit and integration tests

Testing with Maven - Organizing unit and integration tests
- Introduction
- How does a Maven project look like?
- What does Maven Surefire plugin do?
- Why isn't Surefire plugin good for running integration tests?
- Maven Failsafe Plugin
- How does Maven Failsafe plugin work?
- How to organize tests properly in your Maven project?
  - Mix unit and integration tests
  - Advantages
  - Disadvantages
  - Separate unit and integration tests on package level
  - Advantages
  - Disadvantages
  - Separate unit and integration tests on root folder level
  - How to set it up?
    - Maven Compiler Plugin Way
    - build-helper-maven-plugin way
  - Advantages
  - Disadvantages
  - Surprise option: New maven module
  - Advantages
  - Disadvantages
- Summary
- Follow Rodnan Sol for more
- Follow the author too

Introduction

Are you making unit and integration tests in your project? I hope so!

Tests are the fundamentals of a good working and quality software, nobody shouls be skipping them, even if their clients says it 🔥.

If you are a Java developer you probably know the different build tools out there and their support for writing and running unit and integration tests:

Ant
Maven
Gradle
You name it

In this post we will be focusing on Maven, built and maintained by ASF (Apache Software Foundation).

Maven is a beautiful project that helped millions of projects go live, it is an industry standard, but nothing is perfect even millions of projects are depending on it. In this post we are not going to find all the negativity behind the Maven build tool, only just to focus on the testing topics, and to give you advices how your projects structure could be setup to enforce a better test and production code quality.

If you are familiar with a setup of a Maven project, and you know how the different plugins like the Surefire and Failsafe handle the compiled test classes, then you can skip to the following section, the very first few ones are just introducing how Maven is handling the tests. If you want to refresh your knowledge do read the first chapters too! :)

How does a Maven project look like?

By default, if a new projects is just initialized, it should look like this:

The UserService classes are just here for the example, there is no framework behind them, there is no real logic, only for the sake of demonstration.

How can we execute the tests that are in our project? That is easy, just run the mvn test command in your terminal.



$ mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.rodnansol:simple-test-setup >-------------------
[INFO] Building simple-test-setup 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.0:resources (default-resources) @ simple-test-setup ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- compiler:3.10.1:compile (default-compile) @ simple-test-setup ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- resources:3.3.0:testResources (default-testResources) @ simple-test-setup ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO]
[INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ simple-test-setup ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- surefire:3.0.0-M8:test (default-test) @ simple-test-setup ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.rodnansol.user.UserServiceTest
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.645 s - in org.rodnansol.user.UserServiceTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.629 s
[INFO] Finished at: 2023-06-04T17:45:29+02:00
[INFO] ------------------------------------------------------------------------

What happened in this case?

Maven has multiple phases that can be executed, as Maven has the pom.xml file, it also has a so called Effective POM that contains all the phase configurations, but they are not presented in the pom.xml file because that would be too verbose, and it would be difficult to read. These configurations are inherited from the so-called Super POM.

The Effective POM contains all the plugins that will be run when a specific phase is executed, the phases are backed by plugins, like the maven-clean-plugin, maven-compiler-plugin and the maven-surefire-plugin for example that is executing any tests.

Do not here is that the maven-surefire-plugin executes its test goal in the test phase. Even if they are having the same wording it is not the same. The other thing is also crucial, we do not see the maven-failsafe-plugin here, which will be a key later.

Okay, okay, I want to see it with my eyes, so how to create the effective-pom? It is easy: mvn help:effective-pom.

As it is being more than 250 lines, let me just copy a small section of it.

We can see the configuration of the maven-clean-plugin and the maven-surefire-plugin:

What does Maven Surefire plugin do?

TL;DR: It executes tests.

Seriously, it is executing tests, but how does it recognize tests?

The plugin has configuration values, and one of them first defines the folder of the compiled test classes and the other configuration options sets the included files, it is done mostly by patterns.

Test class directory: https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testclassesdirectory
- By default: target/test-classes - and it is compiled by the maven-compiler-plugin's testCompile goal at the test-compile phase. A lots of things are going there for sure. 🔥
Included classes: https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#includes
- By default any Java file that ends with: Test*.java *Test.java, *Tests.java, *TestCase.java.

Now may come the AHA! moment, "so this is why my shiny JUnit test was not executed, because I named it UserServiceT or UserServiceTE".

Yes! But you can configure it to include new patterns or specific files can be put into the include section.

Why isn't Surefire plugin good for running integration tests?

If we check the default lifecycle, we will see a list of the available phases:

maven.apache.org

Let's check the test phase again. Before this phase we can only see the test-compile which is necessary to compile the test classes, we definitely see some other phases, but they are not that "interesting" for us now.

After that we see no "test" related phases, but the integration-test phase, which is not associated with the Surefire plugin by default.

What if we want to run integration tests, that would require different environmental setups, like running a docker-compose.yml file before running any integration test, and stopping the started dependencies when the integration tests finished?

Maven Surefire is not created for running integration tests, this is why it is not picking up test classes having the IT,ITCase postfix in their names, for that Maven Failsafe plugin should be used!

📝 One small note here: You definitely can configure the Surefire plugin to handle the test classes ending with IT,ITCase, but it is just not meant to be used with them. On the other hand, integration tests also can be named as simple Test,Tests,TestCase, but they are different, please do not mix your unit and integration tests.

Maven Failsafe Plugin

Let me quote the following from the Failsafe plugin's documentation:

"The Failsafe Plugin is designed to run integration tests while the Surefire Plugin is designed to run unit tests. The name (failsafe) was chosen both because it is a synonym of surefire and because it implies that when it fails, it does so in a safe way.

The Maven lifecycle has four phases for running integration tests:

pre-integration-test for setting up the integration test environment.
integration-test for running the integration tests.
post-integration-test for tearing down the integration test environment.
verify for checking the results of the integration tests.

If you use the Surefire Plugin for running tests, then when you have a test failure, the build will stop at the integration-test phase and your integration test environment will not have been torn down correctly.

The Failsafe Plugin is used during the integration-test and verify phases of the build lifecycle to execute the integration tests of an application. The Failsafe Plugin will not fail the build during the integration-test phase, thus enabling the post-integration-test phase to execute." - Source: https://maven.apache.org/surefire/maven-failsafe-plugin/

It can not be described better:

For unit test: Use Surefire !
For integration tests: Use Failsafe !

How does Maven Failsafe plugin work?

It is "almost" doing the same as the Surefire plugin, it executes test from a specific directory and with a specific filename pattern.

Test classes directory: https://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#testclassesdirectory
- Same as for the Surefire, so the target/test-classes, the compiled source files are also being compiled by the Maven Compiler Plugin
Included classes: https://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#includes
- This is where the difference can be seen easily, while the Surefire plugin included java classes ending with Test,Tests,TestCase postfixes, in this case the following patterns are the default: IT*.java, *IT.java, *ITCase.java. - So for example UserServiceIT.java or CreateUserITCase.java.

The main point here is that by default unit and integration tests must be put into the "same" folder structure, so to the src/tests folder. There are no src/it or src/it-test options by default, but it can changed easily, wait till the end.

How to organize tests properly in your Maven project?

We have a few options, I will show you 3 that I'm aware of, and I will try to define the advantages and the disadvantages of the different setups.

I want to make a note here, our goal is to not break the package setups, so every time we write a unit or integration tests for a class, we would like to mirror the setup of the production and test code packages like this:

So if we have a package like this: org.rodnansol.user and we have a class in it named UserService our UserServiceTest or UserServiceIT should end up in the org.rodnansol.user package too.

There are cases when our integrations tests (because we name them IT) are covering a different level in the level test pyramid, like full end 2 end tests, in those cases it is another topic how the classes are structured, but if you think about it, in those cases, you might not know about the code you are testing, in those cases it can be a blackbox, and it is up to you how you set it up.

Quick spoiler: In the second option we will break it, but you will see.

Do not forget to set up the Maven Failsafe plugin to run the tests!



<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M8</version>
    <executions>
        <execution>
            <id>default-integration-test</id>
            <phase>integration-test</phase>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Mix unit and integration tests

As its name say, mix the unit and integration tests, just by placing them next to each other.

Advantages

Tests can be found easily next to each other.
Encapsulation should not be broken, so if the UseService will only be used in the user package, it can be package private. Just because we are writing a test for a class, it does not mean that the encapsulation should be broken, it is also true for methods, fields, constants in classes. Always work for the narrowest scope, do not extend it just because a test needs something.
- Yes, if you did not know, visibility in the production and test code are not different, so tests can access package private or protected members of the production code! Use it wisely!

Disadvantages

The unit and integration tests are mixed, it can pollute the code and if the integration test is having different utility classes they may end up next to these classes, and with time and without proper maintenance the setup can be a mess.

Separate unit and integration tests on package level

I have seen it on many projects where if we had a package structure like this: org.rodnansol.user developers introduced a middle package that meant to be used by the integration tests for example: org.rodnansol.it.user or org.rodnansol.integrationtest.user.

Advantages

Unit and integration tests are separated.
Integration test related utilities can be put next to the integration test and they can be marked as package private, to make sure other tests will not use it.

Disadvantages

Encapsulation is broken! Not just a little, but way too much!
- Our UserService class from now can not be package-private.
- None of the UserService methods can be package-private that will be used in the integration-test.
- If we want to play around with some dependency mocking if they reside in the same package as package-private classes, now they must be public to be accessible for the test.

Even if it seems tempting to use this kind of setup, please think about the consequences in the production code. Visibility modifiers are created to be used wisely and properly. Just because a test needs to test a method, it can not be a reason to change its visibility modifier! I would call it as a code smell!

Please think twice before this option is going to be used.

Separate unit and integration tests on root folder level

From my point of view probably the "cleanest" solution! Separating the unit and integration tests on "root" folder level, like on the picture below.

The main problem with this setup is that it does not work without some configuration. The Maven Compiler Plugin compiles the Java classes in the following directories:

src/main/java
src/test/java

So if we want to introduce a third folder we should be making changes even to the Maven Compiler Plugin, or we have to trick the system, and we have to make sure even if the Maven Compiler Plugin's default configuration is not overwritten our Java classes and resource files are compiled and can be used by the Maven Failsafe Plugin.

How to set it up?

First of all create the following folder stucture for your project:

<project-root-level>
- src
- it
  - java
  - resources
- main
  - java
  - resources
- test
  - java
  - resources

Maven Compiler Plugin Way

Overwrite the default configuration of the maven-compiler-plugin and for the test-compile phase overwrite the compileSourceRoots configuration values and add the following entries to it:

<compileSourceRoot>src/test/java</compileSourceRoot>
<compileSourceRoot>src/it/java</compileSourceRoot>

To make sure any test related resources, that is not a Java file, is copied to the target/test-classes folder, we must configure the <testResources> tag too, to include the resources from the src/it/resources folder too. It is done by the Maven Resources Plugin.

Check out the full plugin configuration:



<build>
    <testResources>
        <testResource>
            <directory>src/it/resources</directory>
        </testResource>
        <testResource>
            <directory>src/test/resources</directory>
        </testResource>
    </testResources>
    <plugins>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.11.0</version>
            <executions>
                <execution>
                    <id>default-testCompile</id>
                    <phase>test-compile</phase>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                    <configuration>
                        <compileSourceRoots>
                            <compileSourceRoot>src/test/java</compileSourceRoot>
                            <compileSourceRoot>src/it/java</compileSourceRoot>
                        </compileSourceRoots>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>3.0.0-M8</version>
            <executions>
                <execution>
                    <id>default-integration-test</id>
                    <phase>integration-test</phase>
                    <goals>
                        <goal>integration-test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

build-helper-maven-plugin way

In this case the maven-compiler-plugin should not be overwritten, but a new plugin must be introduced to the whole project, it is called build-helper-maven-plugin.

The folder setup is the same as before, but in this case with the plugin we hook two goals to the following phases:

generate-test-sources - "generate any test source code for inclusion in compilation." - With this phase binding our classes from the src/it/java will be compiled during the test-compile phase.
generate-test-resources - "create resources for testing." - With this phase binding all the integration test related resources files are going to be copied to the target/test-classes folder.
- With the <filtering> option if any of the resource files are containing variable substitutions (${}) maven will filter those values and replace them with values from the pom.xml - this is optional.

Check out the full plugin configuration:



<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>3.4.0</version>
            <executions>
                <execution>
                    <id>add-integration-test-sources</id>
                    <phase>generate-test-sources</phase>
                    <goals>
                        <goal>add-test-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>src/it/java</source>
                        </sources>
                    </configuration>
                </execution>
                <execution>
                    <id>add-integration-test-resources</id>
                    <phase>generate-test-resources</phase>
                    <goals>
                        <goal>add-test-resource</goal>
                    </goals>
                    <configuration>
                        <resources>
                            <resource>
                                <filtering>true</filtering>
                                <directory>src/it/resources</directory>
                            </resource>
                        </resources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>3.0.0-M8</version>
            <executions>
                <execution>
                    <id>default-integration-test</id>
                    <phase>integration-test</phase>
                    <goals>
                        <goal>integration-test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

After any of the plugin setups the following command should copy and compile any test sources and resources into the target/test-classes folder: mvn clean verify.

The verify phase will trigger the Maven Failsafe Plugin's integration-test goal within the integration-test phase.

📝 Small note:

Make sure the files under the src/test and src/it are having different names because they will overwrite each other, and it will cause surprises.

Advantages

Unit and integration tests are separated.
Integration test related utilities can be put next to the integration test and they can be marked as package private, to make sure other tests will not use it.
Encapsulation will not be broken, if the packaging is correct, both the unit and integration tests will be able to access the allowed classes.

Disadvantages

The Maven Compiler Plugin must be changed OR a 3rd party Maven plugin, named build-helper-maven-plugin must be introduced to help with the necessary setups.

Surprise option: New maven module

I will not go into details, but I have already seen a few projects where integration tests were not part of the actual Maven module, but they were in a new module which only contained integration tests.

Advantages

Unit and integration tests are separated.
Integration test related utilities can be put next to the integration test and they can be marked as package private, to make sure other tests will not use it.

Disadvantages

The new maven module must depend on the one it is verifying with the integration tests, so the maven module under test can be built without even verifying its behaviour. You decide if it is an advantage or disadvantage. The whole purpose of doing tests is to make sure an artifact is not being packaged if there are problems with its behaviour.

Summary

Wow! That was a long journey, I hope you found it useful and this setup will be utilized in your next project setup 🎉

We learnt how Maven works and how it generates and uses its Effective POM to build up its lifecycles.
We learnt how Surefire and Failsafe plugins work and how they differ.
We learnt 3 ways to set up our unit and integration tests in detail.
We learnt a 4th method briefly.

I'm curious what you think about it, if you have made it so far!

How are you setting up your projects?

Do you see similarities?
What are the problems you are facing with your current setup?
Will you consider separating your unit and integration tests?

Check out the sample project

Follow Rodnan Sol for more

Follow the author too

Commit Teller - Your personal story teller for your work

Nándor Holozsnyák — Sun, 21 May 2023 12:51:12 +0000

What I built

A new tool has been built that is able to tell a story about your project based on the commit history of your branches, pull requests.

The story can extended along the way and it can be attached to the pull request/issue that is being closed.

Category Submission:

Wacky Wildcard

App Link

https://github.com/rodnansol/commit-teller-action

Screenshots

Some other example results, the tool was run on older pull requests, and because of that the author of the comments are myself, and not the github-action bot.

Smaller ones:

https://github.com/rodnansol/spring-configuration-property-documenter/pull/53#issuecomment-1557572266

https://github.com/rodnansol/spring-configuration-property-documenter/pull/60#issuecomment-1557576442

https://github.com/rodnansol/spring-configuration-property-documenter/pull/45#issuecomment-1557578982

And it definitely can do longer ones, in this case the configuration parameters were the following:

COMMIT_TELLER_OPENAI_TEMPERATURE=0.8
COMMIT_TELLER_OPENAI_MAX_TOKEN=2048

COMMIT_TELLER_STORY_TEMPLATE='Write me a story about the following commits. \n
  The story can be only 1536 characters long. \n
  The story should be based on the commit messages, and it should have the following characteristics:'
COMMIT_TELLER_STORY_CHARACTERISTICS='documentation like, funny'

https://github.com/rodnansol/spring-configuration-property-documenter/pull/53#issuecomment-1557591882

Description

The GitHub action utilizes a CLI application that is able to read the commit messages for an given pull request and based on it, it can build a story, that can be descriptive, funny, happy, you name the mood of it. The CLI application uses the GitHub API and of course, the nowadays overhyped OpenAI. It connects to OpenAI via their public API (so an API key is needed) and it will translate your commit messages into a beautiful story.

The CLI application was built with Quarkus, as a Java developer I wanted to use a framework that can help me move fast and I have confidence and fun with that. I definitely did!

Link to Source Code

Permissive License

Apache

Background (What made you decide to build this particular app? What inspired you?)

OpenAI's ChatGPT shocked the world (you decide if it happened in a positive or in a negative way for you) and developers, companies are already utilizing their features and enhancing their productivity. There are people who are creating content with that, and I thought... Wait a minute, how about an app that can integrate OpenAI's API and it can summarize me commit messages for pull request. There are million projects out there, and you might not know its story, maybe you want to dig deeper and get to know every small detail of it, so how about a tool that can help your with that?

Don't you like writing documentation? Then write detailed commit messages and use the tool for it! It will not be a full-blown documentation for your API, but if you are just playing with your skills and thoughts this one will help your to have visibility for other people if they are interested in your work.

The characteristics, the OpenAI model and many more can be configured, so you have the ability to have your own story behind your commits :) 🎉

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

As a Quarkus fan, I'm using this Java framework whenever I can, and for CLI apps this is my go to choice.

But why a CLI app?

Because with that I can use it not just in a GitHub action but on my laptop as well, but it definitely gives us wacky questions, why would you like to run it on your laptop? 🧐

First things first, I just made the new Quarkus project, from the template that I have been already setup for CLI applications in my brand new organization called Rodnan Sol on GitHub (Do not forget to follow me here 🧑‍💻!)

After the CLI app was ready I made a new repository which contains the setup/configuration for the GitHub action as well.

The setup was easy as the app requires a GitHub token, but GitHub actions always have one implicitly setup for you, but their permission can be changes, this is something I learnt during this time I was making the action setup. This is cool because the token can be narrowed down to the least required list of permissions, so if your action does not have manage repositories you can definitely turn those off, and if they get leaked during the action's lifetime, it could cause no harm to you and your repositories.

Additional Resources/Info

A series about the implementation, from the idea to the usage: https://dev.to/nandorholozsnyak/commit-teller-the-beginning-2l5
Example GitHub repository: https://github.com/rodnansol/commit-teller-example
GitHub Action itself: https://github.com/rodnansol/commit-teller-action
The CLI application: https://github.com/rodnansol/commit-teller
Rodnan Sol GitHub pages: https://rodnansol.github.io/commit-teller/latest/index.html
Twitter: https://twitter.com/TheRealHNK
Rodnan Sol Twitter: https://twitter.com/rodnansol

Commit Teller - How to use it?

Nándor Holozsnyák — Mon, 15 May 2023 19:18:17 +0000

"Well... We are waiting"

The tool is almost ready, it is time to show of what it can do.

Let's wire it into one of our repositories, start coding and let's open a pull request. When everything is ready, merge your results and let the GitHub action run 🔥

How does our pull request look before the tool is executed?

Let's merge it! 🔥

Let's check the GitHub Action itself:

And now, the result itself:

How to wire it into my repository?

Let's look at how it could be wired into your repository.

The setup is easy, because it is also "just" a GitHub Action, and if you have already setup one you will know what it looks like.

After you created your repository clone it (or you can do it online too) and create a folder named .github into the directory. Inside that directory create another one called workflows and in that create a create-pull-request-story.yaml file.

Open the yaml file and use the following as a template:

name: Create story about Pull Request
on:
  pull_request:
    types:
      - closed

jobs:
  create-story-for-pull-request:
    runs-on: ubuntu-latest
    name: Create story for pull request at merge
    if: github.event.pull_request.merged == true
    steps:
      - uses: actions/checkout@v3
      - uses: rodnansol/commit-teller-action@v0.1.0
        name: Create story
        env:
          ISSUE_NUMBER: ${{ github.event.number }}
          COMMIT_TELLER_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          COMMIT_TELLER_OPENAI_API_KEY: ${{ secrets.OPENAI_KEY }}
        with:
          version: early-access
          arguments: create-story -pr ${ISSUE_NUMBER}
           --owner ${GITHUB_REPOSITORY%/*}
           --repository ${GITHUB_REPOSITORY#*/}
           -saf=true --file-extension ADOC
      - name: Commit story changes
        shell: bash
        env:
          ISSUE_NUMBER: ${{ github.event.number }}
        run: |
          mkdir -p docs
          mv *.adoc docs
          git add *.adoc
          git config --global user.email "${{ secrets.COMMITTER_EMAIL }}"
          git config --global user.name "${{ secrets.COMMITTER_NAME }}"
          git commit -m "Creating story about pull request #${ISSUE_NUMBER}"
          git push origin main

Configure the action to run when a pull request is closed, it starts on the second line:

on:
  pull_request:
    types:
      - closed

With that if you can specify when to really run the GitHub action, as pull request can be done without merging the branch into the specified target, but if you want to run the action only the code is merged specify the following (as seen before):

if: github.event.pull_request.merged == true

Configure the action with any available environment variables:

COMMIT_TELLER_GITHUB_TOKEN=<github-token> #1
COMMIT_TELLER_OPENAI_API_KEY=<openai-token> #2

COMMIT_TELLER_OPENAI_MODEL=text-davinci-003 #3
COMMIT_TELLER_OPENAI_TEMPERATURE=0.3 #4
COMMIT_TELLER_OPENAI_MAX_TOKEN=512 #5

COMMIT_TELLER_STORY_TEMPLATE='Write me a story about the following commits. \n
  The story can be only 512 characters long. \n
  The story should be based on the commit messages, and it should have the following characteristics:' #6
COMMIT_TELLER_STORY_CHARACTERISTICS='documentation like, funny' #7

1 - Create a GitHub token here, for GitHub actions it is optional, because the action is always having a token.
2 - Create an OpenAI token here: https://platform.openai.com/account/api-keys
3 - OpenAI’s models: https://platform.openai.com/docs/models/model-endpoint-compatibility
4 - Temperature value. More about it here: https://platform.openai.com/docs/api-reference/completions/create#completions/create-temperature
5 - Number of the maximum tokens for the completion response.
6 - The message to be passed to the language processor. It will include the commit messages as a list.
7 - Define the characteristics of the story.

For the official documentation please check the following website: https://rodnansol.github.io/commit-teller/latest/index.html

Commit stories to the repository

By default a comment will be made to the associated pull request, but if you want to collect the stories at one place you generate the result to a file and commit to the repository with the final step. 🔥

Commit Teller - Problems during the development

Nándor Holozsnyák — Sat, 13 May 2023 14:05:09 +0000

🚨 Problems

After the start I knew that I'll be integrating OpenAI's API and the GitHub API into the application, and I wanted to make sure I can go native with Quarkus. Going native requires GraalVM, but it is not enough, your code must be compatible with the native generation, and at after the first dependencies I pulled in I was not checking it frequently if my code is still able to be compiled into native.

🔨 Dependencies

I only needed two important dependencies that can fulfil my requirements:

jcabi-github - The lib to access the GitHub API easily
openai-java - The lib to access the OpenAI's API easily

🚀 Why did I want to use 3rd party libs?

I wanted to move quickly, I checked the GitHub's API, it looked fine, but the request and response objects can be really big, so if there is any lib that can help me to skip writing a lot of boilerplate code, I try to utilize them.

OpenAI integration is not different, I wanted to skip writing the boilerplate and move fast.

🧐 Problems and solution with the OpenAI API

The openai-java has 3 modules that the developers can use, and I needed all of them to fulfil the requirements.

api
client
server

The usage of the lib was easy, multiple features are implemented but for me the prompt completion was the needed one. It has small request and response objects, with a client class that can invoke the API.

The problems arrived when I tried to compile it to native, and in the background the lib uses the Retrofit library, which in my opinion created the problem. I tried to dig deeper, but a few hours just passed, and I said that is enough, I know Quarkus has an extension to invoke REST APIs, I can do it super fast by just writing the request and response classes.

The client interface:

@RegisterRestClient
public interface OpenAIClient {

    @POST
    @Path("/v1/completions")
    @ClientHeaderParam(
      name = "Authorization",
      value = "Bearer {token}"
    )
    CompletionResponse postCompletion(
      CompletionRequest request,
      @NotBody String token);
}

The request class:

public record CompletionRequest(
    String prompt,
    String model,
    double temperature,
    @JsonProperty("max_tokens")
    int maxTokens
) {
}

And the response class:

public record CompletionResponse(List<Completion> choices) {
    public record Completion(String text) {
    }
}

And the configuration for the client:

quarkus.rest-client."org.rodnansol
.committeller.core.language
.openai.OpenAIClient"
.url=https://api.openai.com

That's it, and poor me was looking for libs that can help me with that :) It took less than 10 minutes to write it and wire it into the existing code.

I used records, because these classes are just data classes, I'm not going to manipulate them later, just use the results.

The point of execution:

@ApplicationScoped
public class OpenAILanguageProcessor
    implements LanguageProcessor {
    private static final Logger LOGGER = LoggerFactory.getLogger(OpenAILanguageProcessor.class);
    private final OpenAIProperties openAIProperties;
    private final OpenAIClient openAIClient;

    public OpenAILanguageProcessor(
        OpenAIProperties openAIProperties,
        @RestClient OpenAIClient openAIClient) {
        this.openAIProperties = openAIProperties;
        this.openAIClient = openAIClient;
    }

    @Override
    public ProcessResult processPrompt(
      ProcessPromptCommand command
) {
        CompletionResponse completionResponse
            = openAIClient.postCompletion(
            new CompletionRequest(
                command.prompt(),
                openAIProperties.model(),
                openAIProperties.temperature(),
                openAIProperties.maxToken()),
            openAIProperties.apiKey()
        );

        List<CompletionResponse.Completion> choices
            = completionResponse.choices();
        if (choices == null || choices.isEmpty()) {
            throw new LanguageProcessorException(
                "No choices are returned," +
                    " unable to return result"
            );
        }
        var completionChoice = choices.get(0);
        return new ProcessResult(completionChoice.text());
    }
}

That's it, native was working fine. 🔥

🙈 Problems and (partial) solution with the GitHub API

The jcabi-github dependency looked nice in the beginning but it was having issues with the event feed management, the endpoints were not properly setup in the code, but it was only a minor issue, I was able to overcome it by using two different operations to fetch the required data.

After trying to go native, the problems just came out, the lib is written in a way that the native compiler was unable to overcome and I was looking for quick fixes. In Quarkus you can setup hints for native compiler, but it was not clear at first glance, what should be setup.

The solution was to move to another solution and use a Quarkus extension that I did not discover at the first place.

Quarkus extensions are basically plugins for the framework itself, you need/should/would write an extension when you want to utilize the framework benefits like build-time optimizations or the native binary generation. I was happy that I found it because if you find an official Quarkus extension than you can "hope" that it will compile to native.

Not in this case 🚧

The extension is backed by a different 3rd party lib that can interact with the GitHub API, it is called Hub4j GitHub API, the change from the jcabi dependency was easy and everything was working as before, BUT not the native mode.

There are some problems with that dependency too, I was digging deeper but I was unable to figure out the real reason behind that.

Outcome

Even I planned to go native but I can not, this does not mean the tool can not be used, it definitely can be, but users would need Java on their machine to run it, but the GitHub action will contain a Java 17 so there should be no problem with the execution. 🎉

Commit Teller - The implementation

Nándor Holozsnyák — Thu, 04 May 2023 20:06:25 +0000

How will it be implemented?

As a Java developer I wanted to stick to Java, and I love the Quarkus project, so I thought I will create a CLI application that can do the job, a CLI app that can be the backend of the GitHub action.

Integration points

In the CLI application I need two integration points:

One to the GitHub API - It can return me the issues, pull requests and it can create me comments on any of them.
The other to the OpenAI, this is basically the brain of the stories, this will be able to parse, and interpret your commit messages and create the story about that.

Setup

The project will contain 2 modules:

One will be the core
And the other one will be the CLI module.

I'm just used to these setup, because if the core module contains the core "business" logic I can easily use it for other modules as well, like for a JBang script or for a Maven plugin easily.

With this setup I can "practice" the setup of multi-module projects with Quarkus. While it is probably super straightforward for other frameworks or pure Java projects, in case of Quarkus, if the developer would like to use the super power of the framework, the developer mode or the continuous testing then the core module must include the Jandex Maven plugin to make sure the mentioned stuff is working well.

Uber-JAR

I'm planning to bundle the CLI application as an Uber-JAR that will contain all the necessary dependencies for the application and anybody with Java on their system will be able to run it.

How about compiling it to native?

A few weeks ago I started another project with Quarkus and I was surprised that the native compilation is getting faster and faster, I do not say it is like lightning fast, but I was shocked that a CLI app with two modules and with a few dependencies was compiled in less than 2 minutes to native mode.

In 2019 when Quarkus was released with its GraalVM integration it was not that easy to setup the whole environment, but today the Quarkus plugins (Maven or Gradle) are super smart and even if there is no GraalVM (or Mandrel) installed on the machine, if it detects Docker it can compile it to native, and on the other hand Quarkus devs also recommend to use the Docker way, because if it is being built on your local machine, you might misses some crucial dependencies, and it may run on your system but not on others.

We will see if we can compile it to native and host it as a Docker image.

What is ready from the implementation?

Almost nothing, only a small PoC that can connect to the GitHub API and to the OpenAI's API.

Stay tuned for more and leave feedback about the project idea.

Commit Teller - The beginning

Nándor Holozsnyák — Thu, 27 Apr 2023 18:10:52 +0000

Another tool for my daily job? But why?

The idea for Commit Teller came after I read the official announcement about the GitHubHack23 event, and started to think about a cool stuff that can utilize GitHub actions.

I use GitHub actions almost on a daily basis for my personal projects and for my projects under the Rodnan Sol organization. It is utilized to execute CI builds and to create releases for our applications. I used to use Bitbucket and GitLab and their CI solutions, GitHub Actions are not different, but they are not the same, the concept is similar, but the syntax and the usages are different.

What is Commit Teller?

Have you ever wondered about, when you opened an issue or a pull request in an open source project on GitHub, what could be story behind this issue or pull request? If it was a feature request issue/ticket, why did the author asked for it or if it was a pull request, what was the story of the work? - How did the author started to work on it?
How many times did they refactor their initial concept?
Can we utilize a tool that can create us documentation about our GitHub issues and/or pull requests (its commits and comments)?

Commit Teller is here to answer these questions.

The concept

Commit Teller reads your issue and pull request comments and commits after a Pull Request is being merged, and it creates a story about them that is being written to the issue or pull request. You can definitely save that story and you can commit under into your repository to make sure you can create a bigger story from that. Think about it if you could relive the story of the Spring framework or the story of the famous 3rd party libs you use on a daily basis to read CSV files, make database connections.

Not all open source projects are having well written documentations, and sometimes developers have to dig deeper into the source code itself to find answers, it would be great to rely on well written commits and create documentation from them.

Core

We are living in an age of the AI tools, this app is not going to be different, it will utilize OpenAI's functionalities to make sure the stories can be built around the comments and commits for the different issues and/or pull requests.

After your pull requests gets merged a preconfigured GitHub action will be triggered, that will utilize OpenAI's capabilities to generate the story for your pull request and the tool itself will send the generated story to the pull request.

Outro

Yes, if you are thinking how the name came, I just thought about to use the name Story Teller but as developers we are making more commits on a daily basis than stories for sure.

Stay tuned for more news on the progress for Commit Teller.

Spring Configuration Property Documenter

Nándor Holozsnyák — Fri, 10 Mar 2023 10:50:00 +0000

Problems

Have you ever felt that you have no proper way to oversee a project because it is having a lot of customizable properties, but they are not documented, and you do not know where to start a new feature or bugfix?
Have you ever felt like you are having too much services, and you do not know how to customize them externally?
Have you ever wondered is there any way to document your application's properties that you are having in your project?

If for any question your answer is YES then you are at the best place.

Spring Configuration Property Documenter

This tool lets you generate informative and useful documentations about the classes annotated with the @ConfigurationProperties in your project.

But how?

It is easy, if you are working with Spring Boot, you first need a dependency on your classpath during build time, it is annotation processor that will read your classes and generates a metadata file based on the annotation configuration classes.

In case of Maven, you should add the following dependency:



<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-configuration-processor</artifactId>
    <optional>true</optional>
</dependency>

After a mvn package the following file should appear in your target folder's META-INF folder: spring-configuration-metadata.json.

It is a simple JSON file, but it contains information about your configuration properties.

Let's see an example:



@Component
@ConfigurationProperties(prefix = "this.is.my")
class MyProperties {

    /**
     * This is my variable.
     */
    private String variable;

    @Deprecated(since = "Since you are a pilot")
    private String anotherVariable = "with default value";

    /**
     * A duration.
     */
    private Duration duration = Duration.ofDays(2);

    private Instant instant = Instant.ofEpochSecond(123);

    private LocalDate date = LocalDate.of(1995, 10, 20);

    private LocalDateTime dateTime = LocalDateTime.of(1995, 10, 20, 0, 1, 2, 3);
    // Getters & Setters

    @DeprecatedConfigurationProperty(reason = "Because it is deprecated", replacement = "instant")
    public Duration getDuration() {
        return duration;
    }
}

Generated spring-configuration-metadata.json file:



{
  "groups": [
    {
      "name": "this.is.my",
      "type": "org.rodnansol.MyProperties",
      "sourceType": "org.rodnansol.MyProperties"
    }
  ],
  "properties": [
    {
      "name": "this.is.my.date",
      "type": "java.time.LocalDate",
      "sourceType": "org.rodnansol.MyProperties"
    },
    {
      "name": "this.is.my.date-time",
      "type": "java.time.LocalDateTime",
      "sourceType": "org.rodnansol.MyProperties"
    },
    {
      "name": "this.is.my.instant",
      "type": "java.time.Instant",
      "sourceType": "org.rodnansol.MyProperties"
    },
    {
      "name": "this.is.my.variable",
      "type": "java.lang.String",
      "description": "This is my variable.",
      "sourceType": "org.rodnansol.MyProperties"
    },
    {
      "name": "this.is.my.another-variable",
      "type": "java.lang.String",
      "sourceType": "org.rodnansol.MyProperties",
      "deprecated": true,
      "deprecation": {}
    },
    {
      "name": "this.is.my.duration",
      "type": "java.time.Duration",
      "description": "A duration.",
      "sourceType": "org.rodnansol.MyProperties",
      "deprecated": true,
      "deprecation": {
        "reason": "Because it is deprecated",
        "replacement": "instant"
      }
    }
  ],
  "hints": []
}

Okay, but what's next?

We built you a tool that can use this JSON file and can generate a documentation from it in the following formats:

Markdown
AsciiDoc
HTML
XML - in case of an XSLT transformation and with XSD support
You name it

You can find it here.

What can it do?

Reading one or multiple metadata files

After it reads your metadata JSON file, a document will be created in the specified format.

Multiple JSON metadata files can be specified before the process, multi-module projects can also utilize it to have only one document where all the properties are available from the different modules.

If the metadata file is in a JAR file, it is not a problem, the tool can read it from the JAR file.

Aggregating more metadata files

If you are working in a multi-module environment, you may have multiple metadata files, but you do not have to worry, single documentations and aggregated documentations also can be produced.

Default templates and customization

The tool provides you a default template that is written with Handlebars, but it is also prepared to support custom templates as well.

By default, the Handlebars templating engine is supported, but you have the ability to use your own templating engine, but you have to write an extension for it.

Check out the documentation here.

Excluding and including groups and properties

As the tool lets you read and use metadata files from JAR files as well, you are able to extract information from the official Spring Boot JARs and you can put this into your final documenatation.

Sometimes these files are big and you may want to select specific groups and/or keys to be rendered or not rendered into the final output.

Exclusion and inclusion lists are also supported.

Please check the following samples here.

How to use the tool?

The core functionalities are built with Java 11 and these functionalities are available through:

Maven usage

Just add the following plugin to your pom.xml and you are ready to go.




<plugin>
    <groupId>org.rodnansol</groupId>
    <artifactId>spring-configuration-property-documenter-maven-plugin</artifactId>
    <version>${final-version}</version>
    <executions>
        <execution>
            <id>generate-adoc</id>
            <phase>process-classes</phase>
            <goals>
                <goal>generate-property-document</goal>
            </goals>
            <configuration>
                <type>ADOC</type>
            </configuration>
        </execution>
    </executions>
</plugin>

Check the available Maven goals here.

Template customization

The templates can be customized if the default one is not fitting your requirements, please read the docs about that here.

Checkout the samples & docs

Feel free to check out the samples and documentation because it will give a wider knowledge about the how-tos.

Samples
Docs

Are you using Spring Boot with Quarkus?

If yes, do not worry, we got you covered, check out this sample.

Follow Rodnan Sol for more

Follow the author too

Local development "environments" per Git branches with Docker Compose

Nándor Holozsnyák — Tue, 08 Nov 2022 14:20:06 +0000

What are Git Hooks?

Have you ever wondered what are the Git hooks and how can they help you be more "effective & productive" ?

Git Hooks are basically "scripts" that can run when specific events are being triggered, like you are making a commit, or you are receiving commits from the remote. There are two types of hooks, client-side and server-side. In this blog post we are going to discover one of the client-side hooks named post-checkout. These hooks must be present in the .git/hooks folder or your project, but it can be changed with the git config core.hooksPath command. If you are interested in the basics please check out it this page: https://git-scm.com/docs/githooks

There are projects where the commit messages are being validated via a specific hook to make sure the messages will be having the "conventional commit" style.

Javascript developers are probably familiar with a tool called Husky it lets you create Git hooks easily installed for the developers. In this blog post we are going to use a Maven project with a Maven plugin that will automatically install hooks into your .git repository.

What is Docker Compose?

With the help of the Docker Compose we are able to create so called YAML files, to create and manage containers "quickly". We can commit these files to our repositories and other developers are able to set up their dependencies easily when they join a project. When I talk about dependencies I mean databases, messaging services, other services (like Spring based apps that are being containerized).

For this Docker and of course the Docker Compose executables must be installed.

An example with Postgres database - the file named by default must be: docker-compose.yml and in the directory the following command must be invoked to start the Postgres container: docker-compose up



version: '3.7'
services: 
  postgres-db:
    image: postgres:14
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: example
      POSTGRES_PASSWORD: example-password

After starting the container (docker-compose up) similar logs should appear:



Creating docker-env-with-git-hooks_postgres-db_1 ... done
Attaching to docker-env-with-git-hooks_postgres-db_1
postgres-db_1  | The files belonging to this database system will be owned by user "postgres".
postgres-db_1  | This user must also own the server process.
postgres-db_1  | 
postgres-db_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres-db_1  | The default database encoding has accordingly been set to "UTF8".
postgres-db_1  | The default text search configuration will be set to "english".
postgres-db_1  | 
postgres-db_1  | Data page checksums are disabled.
...

In another terminal if we execute the docker ps command we should see similar output:



docker ps
CONTAINER ID   IMAGE        COMMAND                  CREATED          STATUS           PORTS                                    NAMES
17adc0b1b905   postgres:14  "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp docker-env-with-git-hooks_postgres-db_1

For me the container ID is 17adc0b1b905 but on your machine it is probably different, most of the columns are self-explanatory but right now I would like to focus on the NAMES column, and the name for our container is: docker-env-with-git-hooks_postgres-db_1. We did not specify any names, so it needs some explanation.

Let's break down the name: docker-env-with-git-hooks_postgres-db_1:

docker-env-with-git-hooks - in my case it is the name of the folder where the docker-compose.yml is - this can be overwritten with a flag when the docker-compose is being called
postgres-db - this is the name of the service we specified in the docker-compose.yml file - this can be overwritten with a container_name attribute in the docker-compose.yml file, but it will overwrite the whole name, nothing else will alternate it.
1 - this is replica number of the container, we can scale containers and in those cases their postfix numbers are changing - if we specify the container_name attribute for the specific service the number postfix will be gone.

To stop the running container just press Ctrl+C in the terminal window where the containers are running or open another terminal window and go to the folder where the docker-compose.yml file resided and inveok the docker-compose stop command.

Let's try to play around with the docker-compose command and overwrite the folder's name in the container name to awesome-hooks.

Run the following command to get more info on the options: docker-compose --help



...
Options:
  -f, --file FILE             Specify an alternate compose file
                              (default: docker-compose.yml)
  -p, --project-name NAME     Specify an alternate project name
                              (default: directory name)
  --profile NAME              Specify a profile to enable
  -c, --context NAME          Specify a context name
  --verbose                   Show more output
  --log-level LEVEL           Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  --ansi (never|always|auto)  Control when to print ANSI control characters
  --no-ansi                   Do not print ANSI control characters (DEPRECATED)
  -v, --version               Print version and exit
  -H, --host HOST             Daemon socket to connect to

  --tls                       Use TLS; implied by --tlsverify
  --tlscacert CA_PATH         Trust certs signed only by this CA
  --tlscert CLIENT_CERT_PATH  Path to TLS certificate file
  --tlskey TLS_KEY_PATH       Path to TLS key file
  --tlsverify                 Use TLS and verify the remote
  --skip-hostname-check       Don't check the daemon's hostname against the
                              name specified in the client certificate
  --project-directory PATH    Specify an alternate working directory
                              (default: the path of the Compose file)
  --compatibility             If set, Compose will attempt to convert keys
                              in v3 files to their non-Swarm equivalent (DEPRECATED)
  --env-file PATH             Specify an alternate environment file
...

This option key is the important for us:



  -p, --project-name NAME     Specify an alternate project name
                              (default: directory name)

Knowing that we are able to change the name for the whole docker-compose project: docker-compose --project-name awesome-hooks up



Creating awesome-hooks_postgres-db_1 ... done
Attaching to awesome-hooks_postgres-db_1
postgres-db_1  | The files belonging to this database system will be owned by user "postgres".
postgres-db_1  | This user must also own the server process.

Now let's check the running containers with docker ps.



CONTAINER ID   IMAGE       COMMAND                  CREATED          STATUS        PORTS                                     NAMES
d9601ab32d6d   postgres:14 "docker-entrypoint.s…"   11 seconds ago   Up 10 seconds 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp awesome-hooks_postgres-db_1

The name of the container now is awesome-hooks_postgres-db_1, so the awesome-hooks is now used as the prefix. AWESOME.

Just to make sure to write it down as well, all Docker objects (containers, networks, volumes etc.) are having that specific prefixes, not only the containers.

Combine Docker-Compose with Git Hooks

Think about it, you are in a middle of a big feature, you have already made changes to the application's database schema, you added new tables, new columns (that are not nullable), and you have written tons of code and then BOOM.

A new message arrives: Hey, can you check this pull request and write some feedback? - Yes of course! Let me see it.

Tons of changes in the source code and in the database schema, but you are a good developer, you always check out the code, and you always run the newly created tests, IT tests whatever...

You also know that you have created test data for your new development by hand, and you do not want to lose it, but if you check out and recreate your database you can lose all your "temporary" test data.

There are a tons of good approaches to this kind of situation, like creating a snapshot of your database, or renaming the container and the attached volumes, but it takes time, and you are in a "hurry".

How about creating new containers per git branches? When a branch is being checked out, a new docker-compose project would be launched, the other containers would be stopped, and new containers would be created where the docker-compose project's name would include the git branch name.

For example, a docker-compose file with the previous Postgres service on a branch named feature/ABC-1123-User-profile-management.

Name our project Warp and in this case the container name could be: warp-feature/ABC-1123-User-profile-management_postgres-db_1 and if the earlier containers are stopped, there will be no collisions with the ports, all your previous data would be retained in the other container and your "work" will not be lost.

Demo

Let's create a folder and in that a new git repository with the git init command.
Create a docker-compose file with the following content as before, make sure to stop the previously created containers.



version: '3.7'
services:
  postgres-db:
    image: postgres:14
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: example
      POSTGRES_PASSWORD: example-password

Add this file to the staging area and commit it:



git add docker-compose.yml
git commit -m "Initial commit"



[master (root-commit) 4a99ad5] Initial commit

We would like to create a script, that will be copied to the git hook's directory, and it will be launched at every branch checkout.

The script must be made with the name post-checkout and it must be copied to the .git/hooks folder, and execute permission must be given to it, so invoke chmod +x .git/hooks/post-checkout

From the official documentation we can see that 3 parameters are going to given to that upon execution: https://git-scm.com/docs/githooks#_post_checkout

$1 - Previous HEAD
$2 - New HEAD
$3 - 1 if checking out a branch, 0 if checking out something else, such as a file (rollbacks)

So we are going to receive the SHA values of the HEAD commit, rather than the branch names we are looking forward.

Let's add some code into the shell script:



#!/usr/bin/env bash

echo "Previous head:" $1
echo "Current head:" $2
echo "Branch checkout:" $3

After that create a new branch with the checkout or switch command: git checkout -b test-branch



Switched to a new branch 'test-branch'
Previous head: 4a99ad545a28e22b5774ae2fa90040fdc86e4b7a
Current head: 4a99ad545a28e22b5774ae2fa90040fdc86e4b7a
Branch checkout: 1

We can immediately see one of the problem, if we would use those variables for the project name it would be super confusing, these SHA values are meaning nothing to the developer, and they can be the same if you start a new branch, so we have to come up with a better solution.

With the following command you can get the name of the current branch: git branch --show-current (Available in Git 2.22.0) - For earlier versions it is tricky, but this must be working: git rev-parse --abbrev-ref HEAD



test-branch

Cool we are able to determine the branch we are on, but how can we identify the branch we are coming from?

So in theory we would like to STOP the containers from the branch we are switching from and START new containers.

I would like to propose two ways:

We use the git name-rev --name-only <sha> command to retrieve it from the $1 argument
We save the current branch name (in this case the previous branch) to a file named .git/earlier-branch, and then we would use it in the process



#!/usr/bin/env bash
if [ ! -f ".git/earlier-branch" ]; then
    echo "File: .git/earlier-branch does not exist, creating with current branch"
    echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch
fi

echo "Loading: .git/earlier-branch"
source .git/earlier-branch
UPCOMING_BRANCH=$(git branch --show-current)

echo "Before switch branch: $EARLIER_BRANCH - Upcoming branch: $UPCOMING_BRANCH"
echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch

This snippet on the first launch will create the file that will have a variable named EARLIER_BRANCH having value of the $(git branch --show-current) result. It may fail on the first try, but after it will be relatively consistent.

Okay, the 3rd parameter describes if this operation is a branch change or just a file checkout, so if it's value is 1 (one) it means it is a branch "change", we have to check it.

Let's add the docker-compose commands into the script.



#!/usr/bin/env bash
echo "Post checkout starting"

#
# Args passed to this are:
# $1 - Previous HEAD
# $2 - New HEAD
# $3 - 1 if checking out a branch, 0 if checking out something else, such as a file (rollbacks)
#
if [ '1' == $3 ]
then
    if [ ! -f ".git/earlier-branch" ]; then
        echo "File: .git/earlier-branch does not exist, creating with current branch"
        echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch
    fi

    echo "Loading: .git/earlier-branch"
    source .git/earlier-branch
    UPCOMING_BRANCH=$(git branch --show-current)

    echo "Before switch branch: $EARLIER_BRANCH - Upcoming branch: $UPCOMING_BRANCH"

    docker-compose --project-name=warp-$EARLIER_BRANCH stop
    docker-compose --project-name=warp-$UPCOMING_BRANCH up -d

    echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch
fi

We are on the test-branch now, let's go back to the master: git checkout master



Switched to branch 'master'
Post checkout starting
File: .git/earlier-branch does not exist, creating with current branch
Loading: .git/earlier-branch
Before switch branch: master - Upcoming branch: master
Creating warp-master_postgres-db_1 ... done

We can see the interference in the power: "Before switch branch: master - Upcoming branch: master" - This is not true, and this is the problem with this branch resolving strategy, the very first time it will not work, because the .git/earlier-branch file does not exist, but it can be created by hand on the clone, or with some other script.

Now if we would run the docker ps we should see the following:



CONTAINER ID   IMAGE        COMMAND                  CREATED         STATUS                 PORTS                                     NAMES
674ca666fe1c   postgres:14  "docker-entrypoint.s…"   2 minutes ago   Up About a minute      0.0.0.0:5432->5432/tcp, :::5432->5432/tcp warp-master_postgres-db_1

The name of the container is warp-master_postgres-db_1, which we wanted! Yes!!!

Now let's switch back to the test-branch or create a brand new one, like we mentioned earlier above: git checkout -b feature/ABC-1123-User-profile-management



Switched to a new branch 'feature/ABC-1123-User-profile-management'
Post checkout starting
Loading: .git/earlier-branch
Before switch branch: master - Upcoming branch: feature/ABC-1123-User-profile-management
Stopping warp-master_postgres-db_1 ... done
Creating network "warp-featureabc-1123-user-profile-management_default" with the default driver
Creating warp-featureabc-1123-user-profile-management_postgres-db_1 ... done

We can see that the warp-master_postgres-db_1 container is being stopped, and after that the warp-featureabc-1123-user-profile-management_postgres-db_1 container is being created. Awesome! This is what we wanted.

Improve the experience

Me personally if I check out a project and I see a docker-compose.yml file I check it and start it from the IntelliJ IDEA after I imported the project. But there are other people who prefer to start it from the terminal with the docker-compose executable, which is totally understandable. With these methods the created git hook will not work properly so I propose to create a few helper scripts which can be used from the terminal and the hook also can depend on it.

Let's create a start-infrastructure.sh that will contain the following:



#!/usr/bin/env bash

GIT_BRANCH=${1:-$(git branch --show-current)}
NAME="warp-$GIT_BRANCH"
echo "Starting docker compose project with name: $NAME"
docker-compose --project-name=$NAME up -d

And let's create a stop-infrastructure.sh that will contain the following lines:



#!/usr/bin/env bash

GIT_BRANCH=${1:-$(git branch --show-current)}
NAME="warp-$GIT_BRANCH"
echo "Stopping docker compose project with name: $NAME"
docker-compose --project-name=$NAME stop

These scripts can be used to start and stop the containers, and they can be parametrized, but if no parameters are given to it, the current git branch will be used.

Let's rework the git hook:



#!/usr/bin/env bash
echo "Post checkout starting"

#
# Args passed to this are:
# $1 - Previous HEAD
# $2 - New HEAD
# $3 - 1 if checking out a branch, 0 if checking out something else, such as a file (rollbacks)
#
if [ '1' == $3 ]
then
    if [ ! -f ".git/earlier-branch" ]; then
        echo "File: .git/earlier-branch does not exist, creating with current branch"
        echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch
    fi

    echo "Loading: .git/earlier-branch"
    source .git/earlier-branch
    UPCOMING_BRANCH=$(git branch --show-current)

    echo "Before switch branch: $EARLIER_BRANCH - Upcoming branch: $UPCOMING_BRANCH"

    ./stop-infrastructure.sh $EARLIER_BRANCH # Changed lines
    ./start-infrastructure.sh $UPCOMING_BRANCH  # Changed lines

    echo EARLIER_BRANCH=$(git branch --show-current) > .git/earlier-branch
fi

Check all containers

If we run docker ps -a we are going to see all the containers we created, so we can easily verify that the containers are still there and if we want to just restart any we can do it:



CONTAINER ID   IMAGE        COMMAND                  CREATED        STATUS                      PORTS    NAMES
188995a19b3f   postgres:14  "docker-entrypoint.s…"   5 days ago     Exited (0) 10 minutes ago            docker-env-with-git-hooks_postgres-db_1
5d01690861ac   postgres:14  "docker-entrypoint.s…"   7 days ago     Exited (0) 10 minutes ago            warp-featureabc-1123-user-profile-management_postgres-db_1
674ca666fe1c   postgres:14  "docker-entrypoint.s…"   7 days ago     Exited (0) 10 minutes ago            warp-master_postgres-db_1
e8b534a4bcd9   postgres:14  "docker-entrypoint.s…"   7 days ago     Exited (0) 10 minutes ago            awesome-hooks_postgres-db_1

Combine it with Maven

To make sure every developer on the project is having the same boost for their workflows we can use a Maven plugin to copy this script to their .git/hooks folder.

Create a folder named hooks and put the post-checkout file into, after that configure the following Maven plugin in your project:



<build>
        <plugins>
            <plugin>
                <groupId>com.rudikershaw.gitbuildhook</groupId>
                <artifactId>git-build-hook-maven-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <installHooks>
                        <post-checkout>hooks/post-checkout</post-checkout>
                    </installHooks>
                </configuration>
                <executions>
                    <execution>
                        <id>install-hooks</id>
                        <phase>initialize</phase>
                        <goals>
                            <goal>install</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
</build>

To make sure that the created scripts are going to be executable for other developers as well, we have to add the execution permission to all via a Git command (but first add them to the staging area):

git add hooks/post-checkout start-infrastructure.sh stop-infrastructure.sh
git update-index --chmod=+x hooks/post-checkout
git update-index --chmod=+x start-infrastructure.sh
git update-index --chmod=+x stop-infrastructure.sh

Conclusion

Maybe this was a "long" journey, but I hope you liked it, I'm still experimenting with this setup, because you may not want to create a new environment for all of your new branches, but if that is the case, I think it is a good start.

One more addition could be introduced, when a local branch is being deleted maybe a cleanup script could be executed to delete the "dangling" docker-compose projects.

I'm curious about your opinion on the topic, have you tried to do the same? Do you know a better/more professional approach to the problem? If yes please let me know down in the comments.

If you want to follow me you can do it on the following places:

Creator of the cover image: https://unsplash.com/@carrier_lost

JBang + Quarkus + AWS Lambda + Terraform = <3

Nándor Holozsnyák — Sun, 02 Jan 2022 20:55:15 +0000

Have you ever wondered how can you utilize your Java skills in the serverless world?
If not then let me take you on a small trip where we will create AWS Lambda functions with Java, and yes with JBang.

Prerequisites:

JBang
Properly configured AWS CLI with Access and Secret key
Terraform - latest/newest release would work.

I would not like to go through how these tools should be installed, I assume these things are a piece of cake.

My versions

❯ jbang version
0.86.0
❯ aws --version
aws-cli/1.22.5 Python/3.9.5 Linux/5.15.8-76051508-generic botocore/1.23.5
❯ terraform -v
Terraform v1.0.0
on linux_amd64

JBang

JBang is a powerful tool which lets you create .java files with your requested dependencies, and with different commands you will be able to build and export .jar files and native binaries.

Okay okay, we have Maven and Gradle, why would I need this?

For my answer for this is the following: If you really want to code just a small app with some dependencies rather than creating and maintaining a project with a pom.xml or a gradle.build could be overkill, like in the following use case where we are going to create a Lambda function.

My motivation

I have attended projects where all the Lambda functions were written in Python, and I'm not a Python developer, of course another programming language, can be learned easily, but with deadlines on our back, if the team is more of a Java team, then writing Lambda functions in Java makes more sense.

What I really like in the Python or JavaScript based Lambda functions are their "lightness", the authors of the functions created a small .py or .js file, and they could deploy it and invoke it and of course they have the online code editor, with Java we won't have that feature, but we can utilize our Java knowledge. Of course dependency management should happen if we need external dependencies, with Python I know it is relatively easy, and of course with Java too, Maven and Gradle are beautiful tools, but I think they are overkill for smaller functions.

I really wanted to have almost the same "workflow" with Java, just one .java file per function that can have external dependencies (like Quarkus that we are also going to use because it has a really nice integration with AWS Lambda as well) listed somewhere in the .java source file as well and can be built by anybody who has the jbang binary on their workstation.

Our first JBang "script"

After JBang got installed we can start working with that, let's create our very first script with the following command:

❯ jbang init hellojbang.java
[jbang] File initialized. You can now run it with 'jbang hellojbang.java' or edit it using 'jbang edit --open=[editor] hellojbang.java' where [editor] is your editor or IDE, e.g. 'idea'

After the file is created we have a "few" options to edit it. We can use the command it outputs with our installed IDE (IDEA,VSCode): jbang edit --open=idea hellojbang.java.
At first glance it could be a bit "weird", I was talking about having no build tool involved in the flow, but we see a build.gradle file, but do not worry, this is just a small helper project that was created for it, to have IDE support, as you can see the whole project sits in the ~/.jbang/cache folder and a symbolic link was created for it.

For IntelliJ IDEA JBang has a really nice plugin, really young, few weeks old but can do the work: https://plugins.jetbrains.com/plugin/18257-jbang in this case you do not have to use the edit command, because IDEA will have a feature to download sync all dependencies and have code completion.

If we open the file we will see the following:

///usr/bin/env jbang "$0" "$@" ; exit $?


import static java.lang.System.*;

public class hellojbang {

    public static void main(String... args) {
        out.println("Hello World");
    }
}

We can run the .java file with the following commands:

jbang hellojbang.java
jbang run hellojbang.java
./hellojbang.java

The output will be the following every time:

❯ ./hellojbang.java 
[jbang] Building jar...
Hello World

On the first run JBang creates a .jar file within its cache folder and it runs it, if codes has no changes compared to earlier run then it will not build it again.

Configuring dependencies and Java version

JBang uses // based directives to configure the dependencies for the application, and other things as well.
Let's see how we can add some dependencies and set the Java version to 11, because with the AWS Lambda we will only have a Java 11 runtime environment.

We can add dependencies with the //DEPS <gav> directive and we can set the Java version to 11 with //JAVA 11 directive

///usr/bin/env jbang "$0" "$@" ; exit $?
//JAVA 11
//DEPS org.apache.commons:commons-lang3:3.12.0

import org.apache.commons.lang3.StringUtils;

import static java.lang.System.*;

public class hellojbang {

    public static void main(String... args) {
        out.println(StringUtils.abbreviate("Hello World", 4));
    }
}

Building and running the script and the output will be:

❯ jbang hellojbang.java
[jbang] Building jar...
H...

Nice, we added a dependency and set the Java version to 11. We can add unlimited amount of dependencies and we can use BOMs as well.
That was a brief introduction to JBang and now let's see the AWS stuff.

Quarkus & JBang & AWS Lambda & Terraform

Quarkus & JBang

Create a new .java file where we can write out Lambda function code.

❯ jbang init AwsLambdaFunction.java
[jbang] File initialized. You can now run it with 'jbang AwsLambdaFunction.java' or edit it using 'jbang edit --open=[editor] AwsLambdaFunction.java' where [editor] is your editor or IDE, e.g. 'idea'

Open the file within our favourite editor: jbang edit --open=idea AwsLambdaFunction.java and add the following dependencies:

//DEPS io.quarkus:quarkus-bom:2.6.0.Final@pom
//DEPS io.quarkus:quarkus-amazon-lambda

With that we state that we would like to use Quarkus at the "newest" version: 2.6.0 and we are adding a new dependency to the "project" as well: io.quarkus:quarkus-amazon-lambda. We don't have to provide the version number, JBang is smart enough to have this information from the BOM specified above it.

If we want to create a Lambda function with Quarkus we have to implement the com.amazonaws.services.lambda.runtime.RequestHandler interface by implementing the handleRequest method.

///usr/bin/env jbang "$0" "$@" ; exit $?
//JAVA 11
//DEPS io.quarkus:quarkus-bom:2.6.0.Final@pom
//DEPS io.quarkus:quarkus-amazon-lambda
//DEPS org.projectlombok:lombok:1.18.22
//JAVA_OPTIONS -Djava.util.logging.manager=org.jboss.logmanager.LogManager

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import lombok.Builder;
import lombok.Data;
import org.jboss.logging.Logger;

public class AwsLambdaFunction implements RequestHandler<LambdaInput, LambdaOutput> {

    public static final Logger LOG = Logger.getLogger("AwsLambdaFunction");

    public AwsLambdaFunction() {
    }

    @Override
    public LambdaOutput handleRequest(LambdaInput input, Context context) {
        LOG.info("Hello from Lambda: " + input);
        return LambdaOutput.builder()
                .result("Incoming text: " + input.getInput())
                .build();
    }
}

@Data
class LambdaInput {
    private String input;
}

@Data
@Builder
class LambdaOutput {
    private String result;
}

In the "final" code snippet we can some new things:

//DEPS org.projectlombok:lombok:1.18.22 - Lombok, which is here to make the POJO classes thinner in the code.
//JAVA_OPTIONS -Djava.util.logging.manager=org.jboss.logmanager.LogManager - We would like to log, in this case we need a logger configuration.
We must have a public no-args constructor.
POJOs should be conventional Beans, with no-arg constructors and with getter/setter pairs.

❯ jbang AwsLambdaFunction.java
[jbang] Resolving dependencies...
[jbang] Artifacts used for dependency management:
         io.quarkus:quarkus-bom:pom:2.6.0.Final
[jbang] io.quarkus:quarkus-amazon-lambda
         org.projectlombok:lombok:jar:1.18.22
Done
[jbang] Dependencies resolved
[jbang] Building jar...
[jbang] Post build with io.quarkus.launcher.JBangIntegration
Jan 02, 2022 9:14:53 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 3.4.2.Final
Jan 02, 2022 9:14:53 PM io.quarkus.deployment.QuarkusAugmentor run
INFO: Quarkus augmentation completed in 610ms
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2022-01-02 21:14:54,510 INFO  [io.quarkus] (main) Quarkus 2.6.0.Final on JVM started in 0.428s. 
2022-01-02 21:14:54,515 INFO  [io.quarkus] (main) Profile prod activated. 
2022-01-02 21:14:54,516 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, cdi]

It means basically our code is using Quarkus and we are "almost done". Of course it would be nice to test it, right now we are not going to write unit tests for it, we would be able to, lets cover that topic in another time, right now just utilize Quarkus's dev mode with the following command:

❯ jbang -Dquarkus.dev AwsLambdaFunction.java
[jbang] Building jar...
[jbang] Post build with io.quarkus.launcher.JBangIntegration
2022-01-02 21:17:08,318 INFO  [io.qua.ama.lam.run.MockEventServer] (build-10) Mock Lambda Event Server Started
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2022-01-02 21:17:08,647 INFO  [io.qua.ama.lam.run.AbstractLambdaPollLoop] (Lambda Thread (DEVELOPMENT)) Listening on: http://localhost:8080/_lambda_/2018-06-01/runtime/invocation/next

2022-01-02 21:17:08,654 INFO  [io.quarkus] (Quarkus Main Thread) Quarkus 2.6.0.Final on JVM started in 1.135s. 
2022-01-02 21:17:08,659 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2022-01-02 21:17:08,659 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [amazon-lambda, cdi]

--
Tests paused
Press [r] to resume testing, [o] Toggle test output, [h] for more options>

Using the dev mode, Quarkus will start a mock HTTP event server so we can use curl or other tools to invoke an HTTP endpoint where we can POST our input object, and then we can examine the result/response as well:

❯ curl -X POST --location "http://localhost:8080" \
    -H "Content-Type: application/json" \
    -d "{
          \"input\": \"Hello World\"
        }"
{"result":"Incoming text: Hello World"}%

By the way, using Quarkus's dev mode lets you change the code "on-the-fly", and on the next invocation it will rebuild automatically. You do not have to build it every time by yourself.

AWS Lambda & Terraform

Make sure the AWS CLI is configured: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Let's deploy our code to AWS with Terraform.

# Terraform basic configuration
terraform {
  required_version = ">= 1.0.0"

  required_providers {
    aws     = "~> 3.70.0"
    local   = "2.1.0"
  }
}

# Set AWS region to eu-central-1 -> Frankfurt
provider "aws" {
  region = "eu-central-1"
}

# We have to create a role for the Lambda function, it is mandatory.
resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda_function"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

# We have to somehow create the jar file that we will deploy, we are going to use the "local-exec" provision feature for it.
# First we will build the .java file, then we have to export it, export means we are copying it from the jbang cache to the current working directory
# After that we have to update the jar file, we have to move the exported "lib" folder to the jar file, we have to bundle all dependencies that we are relaying on.
resource "local_file" "jar_file" {
  filename       = "AwsLambdaFunction.jar"
  content_base64 = filebase64sha256("AwsLambdaFunction.java")
  provisioner "local-exec" {
    command = "jbang build --fresh AwsLambdaFunction.java && jbang export portable --fresh --force AwsLambdaFunction.java && jar uf AwsLambdaFunction.jar lib/"
  }
}

# Lambda function we want to create and invoke.
resource "aws_lambda_function" "function" {
  filename         = local_file.jar_file.filename
  source_code_hash = local_file.jar_file.content_base64
  function_name    = "AwsLambdaFunction"
  role             = aws_iam_role.iam_for_lambda.arn
  #Handler method must be io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest
  handler          = "io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest"

  depends_on = [local_file.jar_file]

  runtime = "java11"
  memory_size = 256

}

Let's run the following commands:

First we have to call the terraform init, it will initialize the terraform state, and after that we can call terraform plan or terraform apply. plan will just only show what it would do if apply would be called.
After calling terraform apply we have to write yes when it asks for approval.

❯ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.70.0"...
- Finding hashicorp/local versions matching "2.1.0"...
- Installing hashicorp/aws v3.70.0...
- Installed hashicorp/aws v3.70.0 (signed by HashiCorp)
- Installing hashicorp/local v2.1.0...
- Installed hashicorp/local v2.1.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

...
❯ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_role.iam_for_lambda will be created
  + resource "aws_iam_role" "iam_for_lambda" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "lambda.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "iam_for_lambda"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # aws_lambda_function.function will be created
  + resource "aws_lambda_function" "function" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + filename                       = "AwsLambdaFunction.jar"
      + function_name                  = "AwsLambdaFunction"
      + handler                        = "io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 256
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "java11"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = "zxCVmQSXmb7Zf3EPLyKKVgL5Tv61WGLArpHz8QSum2c="
      + source_code_size               = (known after apply)
      + tags_all                       = (known after apply)
      + timeout                        = 3
      + version                        = (known after apply)

      + tracing_config {
          + mode = (known after apply)
        }
    }

  # local_file.jar_file will be created
  + resource "local_file" "jar_file" {
      + content_base64       = "zxCVmQSXmb7Zf3EPLyKKVgL5Tv61WGLArpHz8QSum2c="
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "AwsLambdaFunction.jar"
      + id                   = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

After entring yes and pressing Enter we should see the following output:

local_file.jar_file: Creating...
local_file.jar_file: Provisioning with 'local-exec'...
local_file.jar_file (local-exec): Executing: ["/bin/sh" "-c" "jbang build --fresh AwsLambdaFunction.java && jbang export portable --fresh --force AwsLambdaFunction.java && jar uf AwsLambdaFunction.jar lib/"]
local_file.jar_file (local-exec): [jbang] Resolving dependencies...
local_file.jar_file (local-exec): [jbang] Artifacts used for dependency management:
local_file.jar_file (local-exec):          io.quarkus:quarkus-bom:pom:2.6.0.Final
local_file.jar_file (local-exec): [jbang] io.quarkus:quarkus-amazon-lambda
local_file.jar_file (local-exec):          org.projectlombok:lombok:jar:1.18.22
local_file.jar_file (local-exec): Done
local_file.jar_file (local-exec): [jbang] Dependencies resolved
local_file.jar_file (local-exec): [jbang] Building jar...
aws_iam_role.iam_for_lambda: Creating...
local_file.jar_file (local-exec): [jbang] Post build with io.quarkus.launcher.JBangIntegration
local_file.jar_file (local-exec): Jan 02, 2022 9:40:29 PM org.jboss.threads.Version <clinit>
local_file.jar_file (local-exec): INFO: JBoss Threads version 3.4.2.Final
local_file.jar_file (local-exec): Jan 02, 2022 9:40:30 PM io.quarkus.deployment.QuarkusAugmentor run
local_file.jar_file (local-exec): INFO: Quarkus augmentation completed in 652ms
aws_iam_role.iam_for_lambda: Creation complete after 2s [id=iam_for_lambda_function]
local_file.jar_file (local-exec): [jbang] Resolving dependencies...
local_file.jar_file (local-exec): [jbang] Artifacts used for dependency management:
local_file.jar_file (local-exec):          io.quarkus:quarkus-bom:pom:2.6.0.Final
local_file.jar_file (local-exec): [jbang] io.quarkus:quarkus-amazon-lambda
local_file.jar_file (local-exec):          org.projectlombok:lombok:jar:1.18.22
local_file.jar_file (local-exec): Done
local_file.jar_file (local-exec): [jbang] Dependencies resolved
local_file.jar_file (local-exec): [jbang] Building jar...
local_file.jar_file (local-exec): [jbang] Post build with io.quarkus.launcher.JBangIntegration
local_file.jar_file (local-exec): Jan 02, 2022 9:40:33 PM org.jboss.threads.Version <clinit>
local_file.jar_file (local-exec): INFO: JBoss Threads version 3.4.2.Final
local_file.jar_file (local-exec): Jan 02, 2022 9:40:34 PM io.quarkus.deployment.QuarkusAugmentor run
local_file.jar_file (local-exec): INFO: Quarkus augmentation completed in 705ms
local_file.jar_file (local-exec): [jbang] Updating jar manifest
local_file.jar_file (local-exec): [jbang] Exported to /media/nandi/Data/VCS/GIT/jbang-terraform-aws/devto/AwsLambdaFunction.jar
local_file.jar_file: Creation complete after 8s [id=352a94713061363fa798146c96e188a5dd35a975]
aws_lambda_function.function: Creating...
aws_lambda_function.function: Creation complete after 8s [id=AwsLambdaFunction]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

If you want to test it from the AWS console you can do it here.

If you want to use the AWS CLI invoke the following command:

echo "{\"input\": \"Hello World from AWS Lambda\"}" > payload.json
aws lambda invoke response.txt --function-name AwsLambdaFunction --log-type Tail --output text --query 'LogResult' --payload file://payload.json | base64 --decode

The output should look like this (will be different for you, date-time and IDs):

START RequestId: b58e9171-4d9c-4d92-8056-aa7e20317619 Version: $LATEST
2022-01-02 20:45:51,864 INFO  [RequestHandlerExample] (main) Hello from Lambda: LambdaInput(input=Hello World from AWS Lambda)
END RequestId: b58e9171-4d9c-4d92-8056-aa7e20317619
REPORT RequestId: b58e9171-4d9c-4d92-8056-aa7e20317619  Duration: 1.47 ms       Billed Duration: 2 ms   Memory Size: 256 MB     Max Memory Used: 118 MB

If we make any changes to our .java file and we want to deploy it to AWS, we just have to run terraform to do the heavy lifting for us.

Outro

It is a quick and brief article on how to create and deploy Java based functions to AWS Lambda using JBang and Terraform. I really like all the used technologies here.

One thing before I close the article: Quarkus is NOT a mandatory framework to use, I just used it because I really love working with that, and if the function would need database handling libs or would like to use dependency injection then we would be able to just add more and more dependencies to it and use it. We just barely touched the topic.

Follow me on Twitter@TheRealHNK for more good stuff,

If you would like to learn more please check out the following sites:

https://www.jbang.dev/documentation/guide/latest/intro.html
https://quarkus.io/guides/amazon-lambda
https://quarkus.io/guides/funqy-amazon-lambda - Funq is a Quarkus based super thing. Take a look on it.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function

Upcoming

I'm planning to make new articles about exploring AWS Lambda triggers like SQS, S3, SNS. Stay tuned!

Cover (Photo by Gábor Molnár): https://unsplash.com/photos/Y7ufx8R8PM0

JAXB Swagger Plugin - my first contribution to the Open Source world

Nándor Holozsnyák — Sun, 23 Feb 2020 18:22:04 +0000

Introduction

Hello everybody and welcome to my very first post on dev.to :)
I have been almost here for one year now (joined: Feb 24, 2019, just checked it now :D)
I have been a developer since 2016 but I always liked computer and programming, so basically after my official job I come home and I work on side-project with friends or just for fun. My main interests are Java related topics, I like both Spring and Java EE world and I think #Hibernate is my favorite third party framework (kudos to the boys/girls at JBoss).

I live and work in Debrecen, in Hungary, this is the second largest city in the country and there are a lot of opportunities for developers, the university is here for the upcoming devs, but I prefer coding related schools with more practical stuff :)

What did I bring to you?

At my last job we were preferring XSD2Java class based DTO class generation with a lot of reasons and we were building microservice architecture based applications and we were providing functionalities for clients (both internal (internal applications) and external clients).

The documentation for these classes and API endpoints were mandatory and we were experimenting with Swagger, at this time we were a using Java EE related framework called Thorntail (a.k.a Wildfly Swarm), and it came really handy to just put some annotation on the classes and API endpoints to document them.

The problem was the following:

All of our DTOs, which were part of the communication, were generated by the maven-jaxb2-plugin and we were not able to annotate our generated classes because on the next generation the whole stuff would go to the trash can (/dev/null).

Solution: JAXB2 plugin supports external dependencies to take part in the class generation and we found a few options to extend our class generation phase.
This Redlab Swagger JAXB Plugin helped us a lot but it just hepled us to annotate our fields with the proper Swagger annotations like @ApiModel and @ApiModelProperty, nothing special, if you take a closer look at these annotations you will see that there are a lot of configuration options like, the required, allowableValues, description attributes.

As I quited the company, they moved to the OpenAPI specification which is a way more rich specification, I stayed with Swagger at my side-projects and at my job.
There is no problem with Swagger, OpenAPI just got a few new ways to express schemas.

I started to look around the internet, GitHub and I found that there is a lot of fork of the mentioned JAXB Plugin, and there is one which is way more expressive than the original one.

Andreas-Maier-NTT / swagger-jaxb

JAXB XJC Plugin for automatically adding annotations from Swagger to generated classes from an XSD

swagger-jaxb

JAXB XJC Plugin for automatically adding annotations from Swagger to generated classes from an XSD

Tests run in separate project, see here for the code https://github.com/redlab/swagger-jaxb-tck

Usage

REQUIRE Java 8 or higher!
build the plugin with maven
install it in your local repo
add the plugin to your classpath and use -swaggify on your jaxb command line or configure it i your pom or
add sonatype snapshot repository to your repo manager. ( post an issue if you really want dev version in Maven Central )

use with jaxb2-maven-plugin

    <build&gt
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>jaxb2-maven-plugin</artifactId>
                <version>2.3</version>
                <dependencies>
                    <dependency>
                        <groupId>be.redlab.jaxb</groupId>
                        <artifactId>swagger-jaxb</artifactId>
                        <version>1.5-SNAPSHOT</version>
                    </dependency>
                    <dependency>
                        <groupId>javax.xml.parsers</groupId>
                        <artifactId>jaxp-api</artifactId>
                        <version>1.4.5</version>
                    </dependency>
                    <dependency>
                        <groupId>com.sun.xml.parsers</groupId>
                        <artifactId>jaxp-ri</artifactId>
                        <version>1.4.5</version>
                    </dependency>
                    <dependency>
                        <groupId>com.sun.xml.bind</groupId>
                        <artifactId>jaxb-xjc</artifactId>
                        <version>2.2.11</version>
                    </dependency>
                    <dependency>
                        <groupId>com.sun.xml.bind</groupId>
                        <artifactId>jaxb-core</artifactId>
                        <version>2.2.11</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </pluginManagement>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>jaxb2-maven-plugin</artifactId>
            <version>2.3</version>
            <executions>
                    <execution>
                        <id>internal.generate</id>
                        <goals>
                            <goal>xjc</goal>
                        </goals>
                        <configuration>
                            <arguments>-swaggerify</arguments>
                            <clearOutputDir>true</clearOutputDir>
                            <packageName>be.redlab.jaxb.swagger.generated.model</packageName>
                            <sources>
                                <source>${project.basedir}/src/main/xsd/schema</source>
                            </sources>
                        </configuration>
                    </execution>
            </executions>
        </plugin>
    </plugins>

…

View on GitHub

This one deals with a lot of my concerns and does its job but I was still not satisfied with the solution, so I forked this repository and made my changes on it.

The contribution

Here is the repository with my changes:

nandorholozsnyak / swagger-jaxb

JAXB XJC Plugin for automatically adding annotations from Swagger to generated classes from an XSD

swagger-jaxb

JAXB XJC Plugin for automatically adding annotations from Swagger to generated classes from an XSD

Tests run in separate project, see here for the code https://github.com/redlab/swagger-jaxb-tck

What does it do?

When you are using XSD to generate your DTO classes you may want to annotate them to make it as detailed as possible for the clients whose are going to use your endpoints.
The XSD to Java classes generation can be extended with this tool to make your API objects as declarative as they can be.
The generated classes and methods will be annotated with the proper Swagger annotation, classes with @ApiModel and methods with @ApiModelProperty.

Example

Example XSD object:

 <xsd:complexType name="LoginRequestType"&gt
        <xsd:annotation>
            <xsd:documentation xml:lang="en">Login object containing the e-mail and password</xsd:documentation>
        </xsd:annotation>
        <xsd:complexContent>
                <xsd:sequence>
                    <xsd:element name="email" type="xsd:string">
                        <xsd:annotation>
                            <xsd:documentation xml:lang="en">E-mail of the user</xsd:documentation>
                        </xsd:annotation>
                    </xsd:element>
                    <xsd:element name="password" type="xsd:string">
                        <xsd:annotation>
                            <xsd:documentation xml:lang="en">Password of the user with SHA512 encoding</xsd:documentation>
                        </xsd:annotation>

…

View on GitHub

A detailed README.MD and some examples are provided, most of the features are covered by unit tests, but not all, the codes by redlab and by Andreas were a bit reworked but not all.

In my projects right now I use Springfox for Swagger functionality and with this little plugin all of my DTOs which take part in the communication with the clients are detailed.

If you are interested in a extended XSD2Java class generation with Swagger then this is your time to pull this dependency into your pom.xml and start generating :)

Summary

Thank you for everybody who read my first blog post, I really appreciate it, and because I'm really new to writing I'm eager for some feedback in the comment section.

I'm curious if anybody is still using XSD schemas to generate their Java classes or maybe JSON schemas took over (btw I validate the incoming request against these schemas, so there is no surprise in the business logic ;))

Let me know what do you guys think about that and feel free to open issues or PR's.

I try to maintain the code, the code quality is not the best, I'm still wondering on a lof of refactor options but for know, it is okay to use, as I will use it from now ;)