DEV Community: Bruno Pacheco

Installing FreeBSD on a HP 250 G9

Bruno Pacheco — Mon, 28 Oct 2024 21:36:50 +0000

I’ve acquired a Hewlett-Packard 250 15.6-inch G9 laptop with an Intel 12th Gen Core i3-1215U, 8GB of RAM, and a 500GB SSD. Not impressive hardware, I admit, but I will not run a lot of things locally anyway. So, let’s go directly to the point.

I have installed FreeBSD 14.1. Actually, it is not as daunting as it may sound. The installation process is straightforward and easy to understand. The most complex thing would be partitioning, but in my case, I used the entire disk—no need for dual boot—I let the tool do its job alone.

I will not add screenshots of each step, as much more experienced people have thoroughly covered it. Please refer to this video, from RoboNuggie. Approximately 70% of everything I needed to do is explained in that video.

I will list below what was not covered in that video, like playing protected media, graphics, audio, and WiFi configuration.

Video

This laptop has the Alder Lake-UP3 GT1 [UHD Graphics], which requires drm-61-kmod.

$ cd /usr/ports/graphics/drm-61-kmod
$ sudo make install clean
$ sudo sysrc kld_list+=i915kms

Windows Manager

I am using Mate; for that, I simply followed the FreeBSD Handbook.

Chromium and CDM

I found a couple of tutorials online, but here I will share simpler steps for supporting protected media playback—Netflix, Spotify, etc.—in Chromium:

Install Chromium and Foreign CDM:

$ sudo pkg install foreign-cdm chromium
$ sudo sysrc linux_enable="YES"
$ sudo service linux start

Download ports and FreeBSD source code to compile www/linux-widevine-cdm.

$ sudo pkg update
$ sudo pkg install portsnap git
$ sudo portsnap fetch
$ sudo portsnap extract
$ sudo git clone -b releng/14.1 https://git.freebsd.org/src.git /usr/src

Install Linux Widevine CDM

$ cd /usr/ports/www/linux-widevine-cdm
$ sudo make install clean

Open chrome://flags in Chromium and enable Cdm Storage Database and Cdm Storage Database Migration:
Close Chromium and open again.
You can test if DRM is enabled using https://bitmovin.com/demos/drm.

Printer

I have an HP OfficeJet, and from past Linux experience, I knew about HPLIP. Fortunately, it was also available as a binary to be installed with pkg install hplip.

Audio

Audio configuration is also simple. I needed to load a driver and configure hot-swap from speakers to headphones and vice versa whenever a new device was (un)plugged into the audio jack. I followed this tutorial.

Wifi

This was the most difficult part of the configuration. My first attempt was loading the driver compatible with my card. Apparently, the chipset was supported, and it worked at first. But as soon as I started consuming videos, it proved unstable. I tried a WiFi micro adapter, but it was quite slow. After a lot of reading on forums and some headaches, I discovered the Wifibox project.

It’s a clever idea of running WiFi drivers inside a Linux guest VM and bridging the connection back to the host machine. The footprint is quite small, but I can use the full potential of my WiFi. I followed this tutorial.

Not tested components

I haven’t tested Bluetooth or HDMI audio output.

Tested platforms:

Zoom
Webex
GeForce Now
Netflix
Amazon Primevideo
Spotify

Well, that is it with some manual steps, I have a operating system that is fit to my needs.

See you | Bis geschwënn | Até mais | À bientôt | Ciao

Say hello to FreeBSD

Bruno Pacheco — Thu, 17 Oct 2024 19:57:14 +0000

Hello, Bestie!

I started the hobby of testing operating systems somewhere between 2006 and 2009. However, like many of you, I relied solely on social media to document my experiences. Unfortunately, much like the Library of Alexandria, Orkut is lost — though not necessarily as significant in terms of historical relevance.

Back then, I experimented tons of different Linux distros, including a Brazilian one called Kurumin Linux!

I also explored some unconventional systems, like BeOS, OpenSolaris, OpenDarwin, and others I can no longer recall. VirtualBox was a lifesaver when hardware support was lacking — unsupported hardware is always the ultimate barrier — but whenever possible, I tested systems directly on my machine.

I remember spending hours downloading images and burning them onto CDs. My desktop was decent but still slow and limited, so installations could take up to an hour, not to mention the configuration and updates. Any mistake meant starting over from scratch. Actually, my first Debian experience came from a CD I requested online, which I received a few weeks later. I can’t fully express the mix of emotions I felt with each attempt: exhaustion, joy, frustration, accomplishment, anger, happiness, and even occasional anxiety.

All of this is just to illustrate how challenging it could be, and FreeBSD was no exception. I managed to install it after a couple of attempts. Everything needed to be compiled locally, including Java! It was the only time I ever had the opportunity to recompile a kernel!

The result was an exceptionally fast and lightweight operating system. I used it for a while, but the challenges of version upgrades and limited compatibility with browser technologies of that time (e.g., Silverlight and Adobe Flash) eventually pushed me back to Linux.

In 2015, I bought my first laptop, and that hobby came to an abrupt halt. Laptops often come with hardware specifically tailored to the software they ship with, sometimes even locked down. After a while, I just accepted using Windows with the Linux Subsystem. Everything worked as expected, and I gradually forgot the thrill of testing new operating systems.

My old lad survived nearly 10 years with just a couple of upgrades — extra RAM and replacing the HDD with an SSD — but it's no longer functional, or perhaps I just didn't want to try fixing it. I immediately disassembled it to show my daughter what it looked like inside. I was time to move on.

While searching for a new laptop, all those memories came flooding back. I decided then to buy relatively older hardware (released in 2022), without an operating system or any warranty restrictions. I wanted freedom.

Now it's 2024, and the FreeBSD ecosystem has evolved for the better. Ports and compilation are still necessary for things like certain drivers and DRM-protected content (e.g., Netflix and Spotify), but they are no longer the default or the only option available. It takes just 5 minutes to install the entire OS, followed by several extra hours to configure every possible aspect — from screen brightness adjustments and automatic switching when headphones are detected, to automounting USB drives and optimizing CPU frequency.

The result is once more fantastic, a fast, reliable and secure operating system.

P.S.: I will follow this post with instructions on how to install FreeBSD 14.1 on my hardware.

See you | Bis geschwënn | Até mais | À bientôt | Ciao

My Journey to Open Data

Bruno Pacheco — Tue, 04 Jul 2023 21:02:28 +0000

Well, this is not exactly a technical article. This post describes the end of a quest, and the beginning of a new journey: my first step towards Open Data, a world of fairy, strange people willing to change the world.

My first contact with that world happened two years ago when I started learning Luxembourgish. Quickly, I discovered the lack of online content, community support, and learning tools. Then I decided to use my superpowers to build an application called LOD to Anki. This software aims to export the words from LOD.lu into Anki flash cards. Please follow the links to learn more about them.

Unfortunately, scraping the LOD's content proved to be an inefficient strategy. The HTML structure of the old website was cumbersome, making web crawling complex and slow. How would I keep my dataset updated? How would I ensure the performance of my tool? There was only one answer to those questions. I decided to ask for the data directly.

To my surprise, the Zenter fir d'Lëtzebuerger Sprooch (ZLS) answered back. They shared the link to LOD's dataset, exposed on the Luxembourgish Open Data Platform, and I marveled!

With the dataset in hand, I rapidly developed my project. It was easy to integrate into the software, giving me time to focus on the Anki deck development instead. Thanks to that, all the important words listed by Schwätzt Dir Lëtzebuergesch books series are now documented and shared on GitHub, for anyone to use and learn.

That innocent interaction with ZLS and their dataset helped me make a small but positive impact on society. A few people expressed their gratitude for the software. Moreover, the LOD.lu's audio content was size reduced, with unnoticed quality loss, because I shared László's suggestion on audio compression with ZLS, potentially reducing their network traffic.

Indeed, the planet is round, rotating, and revolving. Two years later, I seized the opportunity to join the fairy people. I am now working for the Luxembourg National Data Service (LNDS). I will be one of the many human beings working behind the curtain, integrating and providing solutions to support research and innovation through data.

I would like to express my appreciation to all the people at Pictet Technologies. These four years of learning and growth were undoubtedly decisive in my career success.

See you | Bis geschwënn | Até mais | À bientôt

The secret of long-lasting code

Bruno Pacheco — Sun, 05 Feb 2023 02:30:44 +0000

I cannot talk about every software developer in the world, but I imagine that most of us had to deal with legacy applications at some point in our careers.

But, before diving into it, we should find a definition of legacy software, and Gartner says:

An information system that may be based on outdated technologies, but is critical to day-to-day operations. Replacing legacy applications and systems with systems based on new and different technologies is one of the information systems (IS) professionals’ most significant challenges. As enterprises upgrade or change their technologies, they must ensure compatibility with old systems and data formats that are still in use.

Now, let us review that definition, to understand the assumptions and reasons for software to be considered legacy.

Software based on outdated technologies, but critical to day-to-day operations.

For me, this is somewhat of a paradox. The software should not use outdated technologies if it is critical to day-to-day operations. So, either both day-to-day operations and the application are old, so the whole thing is old and must be reviewed, or the software took too much time to be developed, so it does not fulfill the requirements anymore.

In the end, this is about how often you review the software requirements. Keep questioning if the way you work is still correct. Or if the technology is still a good fit for your problem. No one should expect that a particular solution will last forever.

Replacing legacy applications and systems with systems based on new and different technologies.

In my viewpoint, Replacing a system represents the end of the line. It means that someone made a terrible decision of not investing time and money to keep the process fine and tuned. Or someone made a wrong technical/architectural choice. Or someone poorly analyzed the requirements. Ultimately, it is too late to fix that.

As enterprises upgrade or change their technologies, they must ensure compatibility with old systems and data formats that are still in use.

Well, that is not surprising and is not a problem for legacy applications only. No matter the size of the application or the technology you use, integration is always a challenge. It is more of a coordination issue and requires regular attention. And it can break at any minute, in unexpected places.

Summing up, Bad decision-making is solely the root cause of a legacy system. The software is not legacy. The legacy (or the burden) is the mindset that kept the software as it was until it became a legacy.

Continuous Refactoring / Upgrading

From a business perspective

The requirements evolve and require regular ongoing review, often needing refactoring to align the code with these changes. That may add overhead to the business analyst, as the new feature needs to be compatible with existing ones. And refactoring may also increase the risk of introducing defects.

But on the other hand, when we choose a shorter path, to avoid needed enhancements, we may postpone fixes to existing bugs, or increase technical debts, which is also dangerous to the project.

So, to business stakeholders, refactoring is an investment. Well-written stories and well-defined test scenarios will guide the developers in the right direction, and BDD and E2E tests are great tools to detect regressions, so don't be afraid of the new.

From the development perspective

Writing code is challenging. Enhancing existing code can be double challenging, depending on how far you are from the new requirements. Touching someone's code can be even dangerous, as you may not have that person to guide you. All this will make any developer anxious.

Regular coders will stay in their comfort zone and keep things as they are. Outstanding engineers will take the task but may fail on the first try. However, with enough perseverance, they will excel.

For me, refactoring is about rethinking and learning. It keeps all senior and junior developers up to date about the technology and the business. The more refactoring you do, the simpler and faster you will develop new features.

Risks of not continually improving your code

Well, your technical debt increases. And technical debt is a concept that refers to the cost of maintaining a software system due to quick and inefficient solutions to problems. It represents the trade-off between short-term development speed and long-term technical excellence. For more details, please follow these links:

Benefits of doing so

Refactoring/upgrading is not only rewriting code/replacing technology with something else. It is all about stability. Tackling technical debt strengthens maintainability, thread safety, security, scalability, and low footprint. All these are crucial factors for a long-lasting project.

But apart from that, keeping a stable project up and running brings satisfaction, visibility, and recognition. Nothing is more satisfying than a well-done job. And you and your project will become a reference for peers and stakeholders.

What to do then

Each project is unique, but there are some steps to follow:

Evaluate your situation and prepare a plan. You want to know where you are and what to do, before starting the journey;
Break it down into small pieces. It will be easier to get accepted and validated. You can start with single components or fix duplicated code. Clean Code provides you with many insights;
Make sure you have good test coverage, it will support you on avoiding regressions. If the test coverage is not enough, TDD (Test-driven Development) is a good thing to start practicing here;
Improve your app monitoring and traceability. Not just to prepare for possible new bugs, but also to look at your current situation. For monitoring, I suggest Prometheus, For traceability on Spring Boot, Sleuth is the easiest choice. Please, read this article understand the concept of Distributed Tracing.

Apart from that, there is not much to do than doing your job as it is supposed to be. Excellent coding principles, clean code, meaningful tests, code review, pair programming, Sonar, intelligent IDEs, AI... Use every weapon you may have to ensure the code you write is well written and well covered.

Conclusion

The golden rule here is: never to stop trying new things, or you are stuck in the past. And most probably, everyone else will waste a lot of time and money doing the same thing you did over and over. At some point, your software will become a legacy system, and everyone will complain. Eventually, it will get replaced.

So, please do not be afraid of changing the code. No piece of code is generic enough to cover all possibilities, nor so well written that is the best solution for the requirements.

See you | Bis geschwënn | Até mais | À bientôt

Optimizing Decision Making with a Trie Tree-Based Rules Engine: An Experience Report

Bruno Pacheco — Tue, 24 Jan 2023 01:41:09 +0000

In Pictet Technologies, my team relies a lot on decision models. These models allow our business analysts to input Compliance business rules directly into the systems with minimal developer intervention. When I joined the company, we used to use both Drools and Camunda. However, we faced severe memory and performance issues, specifically with Camunda, prompting me to explore alternatives.

For those not familiar with these solutions, both are rules engine providers. And a rules engine is a software system designed to process and execute sets of business rules. These rules provide automated decisions based on a set of inputs. In a nutshell, a rules engine is a very sophisticated if/else interpreter. For more details, I suggest you follow this link.

The first challenge I encountered was with the decision model and notation file (or DMN). By the way, DMN is the “standard” format that both Drools and Camunda implement. While it is relatively easy to understand, it is impossible to load the same file on both vendors. Another point is that most of our cases are simple decision tables. If it were not for the number of rules, it would be simpler to code the business rules in Java directly.

But the most critical issue related to these engines was performance. Our DMN-based services struggled to handle concurrent usage. Even on single requests, the process took longer than our clients were expecting.

Starting the journey

Initially, I evaluated alternative solutions, but ultimately, Drools and Camunda were the dominant players in the field when I started these studies. And both were and still are easy to use and integrate with, free to use, and offer tools for maintaining the DMN files.

The next step was using a database, where the rules could be stored and easily accessed through queries. However, this approach quickly resulted in poor performance.

Eventually, I began exploring the possibility of developing my solution, which prompted me to conduct research. That was when I discovered the Trie Tree, a rather basic data structure that arranges words in shared prefixes. Please, follow this link for details on how Trie Tree works. There is also this visualization tool to get acquittance with this data structure.

Image reused from GeeksForGeeks

Upon closer examination, I found that treating each input of a rule as a character in a string, Trie Tree is a good solution for a rules engine. The challenge when I started developing it was the “ANY” value, which allows any input value for a given entry. To accommodate this, it was necessary to accurately propagate the children of the ANY node to all siblings, and vice versa, ensuring the correct branch may be traversed.

Another challenge of implementing the Trie Tree was the multiple rules triggering implemented by both vendors. To achieve it using Trie, the leaf node should contain a sorted collection of rule IDs - the sorting mechanism depends on the Hit Policy selected by the developer. In most of our cases, we use the FIRST hit policy, which retrieves the first element of the collection sorted by rule id.

Reviewing the solution

The overall performance was exceptional, as demonstrated later. However, memory consumption remained high. After studying other Trie Tree implementations - Radix Tree is an example - I developed an algorithm to eliminate ANY nodes without siblings, as it is always the default branch. This approach reduced some subtrees to just one node.

After further testing, I realized that the number of nodes is significantly smaller if I ordered the inputs appropriately. That is a computationally complex problem, but luckily, there is a simple approach to avoid testing all possible scenarios. One should sort the input entries based on the number of ANY values they have. In one particular service, this technique allowed us to reduce the number of nodes from 30k to 1.5k.

Preparing this article

First, I transformed this Trie Tree implementation into an open-source project, which I called Praecepta (Rules in Latin), so anyone can try it out. Then, I added three modules to it, with a decision table based on a real scenario (blurred input/output values). Each module uses a different framework: Praecepta, Camunda 7, and Kogito (the newest Kie solution for the Cloud).

I used SpringBoot 3.0.2, GraalVM 22 (JVM mode), a MacOS 2,6 GHz 6-Core Intel Core i7, running 1000 users for 5 minutes. The idea was to test how memory consumption and CPU usage evolve. Below, I compared the footprint of these three solutions. I collected the total count of requests, throughput, memory consumption, and CPU usage using VisualVM and Gatling.

Beating Kogito and Camunda

The numbers do not lie!

Solution	Requests (total count)	Throughput (mean count/s)	Memory (Peak)	CPU
Praecepta	6.614.663	21.337	~100MB	~25%
Kogito	3.615.549	11.663	~225MB	~30%
Camunda 7	1.238.667	3.995	~450MB	~35%

Conclusions

You can see the results below.

Sadly Camunda 7 did not perform well. It was 3x slower and 2x bigger than Kogito. The last proved acceptable given the numbers, but still, it had twice as much memory as Praecepta. Praecepta, on the other hand, steadily kept memory below 100MB, with low CPU usage and 2x greater throughput, compared to the second place.

The outcome is not that Kogito nor Camunda sucks. It always depends on what we plan and want in our system. If you have a much more complex scenario, Kogito or Camunda may fit better to your needs, but first, you should evaluate other solutions before buying/using one of them.

For our team, a simple data structure had an immensely positive impact. We could provide microservices with a smaller footprint and much higher availability.

For me, It was an excellent opportunity to use my computer science background and learn new things.

Results

Praecepta - TrieTree

Kogito

Camunda

See you | Bis geschwënn | Até mais | À bientôt

Staying Sharp in Software Engineering

Bruno Pacheco — Sat, 07 Jan 2023 21:15:24 +0000

When it comes to working, learning is often a key concern. That is especially true in software engineering, where there is always a way to do something faster, cheaper, or better, which often requires new technologies. In this field, it is also crucial to ensure high-quality code. That includes not only coding, but also dependency management, testing, code review, continuous integration and deployment, tracing, monitoring, and measuring.

Fortunately, most of those tasks can be automated, allowing developers to focus on the actual coding. However, this is just the beginning of the journey to becoming a developer.

Acquiring experience in software engineering is a long and tedious process that requires constant practice and learning. However, it is easy to get comfortable, relying on familiar ideas and solutions. After a year, you may know enough about how and where to do things, but after five years, all the projects may start to look the same. Finding new ways to challenge yourself and try new things is indispensable to avoid stagnation.

I was talking to friends at work and this topic came to life. I will share what I often do to continue learning. Please, join me on this journey.

Online Coding Challenges

One way to keep learning is by participating in online coding challenges. These challenges can be a fun and engaging way to learn, as they often have a game-like structure with small challenges that increase in difficulty as you progress.

One benefit is that coding challenges can help you stay up-to-date with the latest technologies and programming languages. These platforms offer also challenges in different areas, such as algorithms, data structures, and design patterns.

Another benefit is that coding challenges can help you develop problem-solving and critical-thinking skills. Many challenges require you to think creatively and bring solutions to complex problems, which can be a great way to exercise your brain and improve your overall problem-solving abilities.

Coding challenges can also be a great way to build your portfolio and demonstrate your skills. Many platforms allow you to share your progress and accomplishments, and earning badges or ranking highly on leaderboards can be a great way to showcase your abilities to others.

However, this approach can be time-consuming, as it often requires a significant investment of time and effort to finish. Additionally, the competitive nature of these platforms can be stressful for some people and may not be a good fit for everyone.

Have a coding partner

Another way to challenge yourself and learn new things is to work with a peer. Having someone alongside can help keep your motivation up and provide opportunities for feedback and review.

There are several ways you can find a peer to learn software engineering with:

Join online communities or forums related to software engineering. These communities often have members willing to mentor or collaborate with others.
Reach out to local meetup groups or attend industry events in your area. These events can be a great way to meet other software engineers and find potential peers.
Connect with people in your professional network. Ask around to see if anyone is interested in collaborating or learning together.
Consider joining a study group or starting one of your own. You can find other people interested in learning what you want to learn and meet regularly to study and learn together.

Overall, the key is to be proactive and put yourself out there. Don't be afraid to reach out to people and ask if they would be interested in learning together. That is a valuable way to deepen your understanding of a topic and gain new perspectives, but there are potential challenges.

One challenge is time constraints. Depending on your schedules, usually, it is hard to find time to work together. We need to be flexible and work around each other's schedules.

Motivation can also be a challenge when learning with a peer, especially when working on a difficult or complex topic. You must be proactive to keep each other motivated and on track.

Finally, conflicts can arise when working with a peer. You should be open to feedback and find ways to resolve any disagreement that may emerge constructively.

Code for yourself

The benefits of working on projects relevant to your interests or needs are clear goals and purpose. By that, you are more likely to stay engaged and motivated, which can help you learn more effectively.

In addition to the learning opportunities, working on real-world problems can give you valuable experience that can be helpful when applying for jobs or working on professional projects. This hands-on experience can help you stand out in a competitive job market and give you confidence in your abilities.

Finally, seeing a project through from start to finish is very rewarding. Developing a tool or application to solve a specific need can be a very fulfilling accomplishment, as you see the results of your work practically and tangibly.

On the other hand, you may encounter different challenges, like time constraints. Balancing these projects with other commitments, such as work or family, can be quite difficult. Finding time to work on your projects can require careful planning and time management.

Debugging and troubleshooting can also be more challenging when working on your own. Without the benefit of colleagues to collaborate with, you may have to spend more time figuring out problems and fixing errors.

Finally, limited exposure can be challenging when working on projects for yourself. Without a team or organization behind you, it can be harder to get your work noticed and recognized by others. That can make it harder to build a portfolio or get feedback on your work.

To cut it short

While stepping out of your comfort zone can be time-consuming and require a lot of effort, it will help you gain new perspectives and be very beneficial in the long run.

It's imperative to seek out different resources, such as books, courses, real-world projects, or friends, to ensure that you are well-rounded and have a broad range of experience.

Useful links

Brilliant - Huge variety of content, from Quantum Mechanics to Computer Science
3Blue1Brown - Youtube Channel to learn Linear Algebra, Calculus and more
HackerRank - Very good online coding platform
Kaggle - Public datasets and competitions for Machine Learning
NotRocketScience_ - Technical challenges proposed by László
Google Qwiklabs - Hands-on oriented web courses for Google Cloud Computing
LinkedIn Learning - Web courses
Coursera - More web courses
Udemy - And more web courses

See you | Bis geschwënn | Até mais | À bientôt

Covid led me to my first open-source contribution

Bruno Pacheco — Wed, 04 Jan 2023 01:00:58 +0000

It's been a while since I've shared anything here. Mostly because my interests and priorities have changed. But now I'm back, and I hope to produce more content.

Let's go back to 2020, when I stopped writing. The Covid era, baby!

There were people going crazy because they couldn't see other people. There were people baking bread and brewing beer at home. And I was becoming a first-time parent, or at least starting the journey.

I'm an engineer by love and a developer to the bone. Like everyone else, I was looking for something to focus my thoughts on. And an article about a COBOL system that was overloaded in the US caught my attention. Of course, I had heard of COBOL in university, but the most "hardcore" language I had briefly touched was C/C++.

I was bored, and I didn't want to see more of REST, more of C-like languages, or more of architectural problems. I wanted something more basic, easy to start and easy to stop. Why not COBOL? I thought it might be useful in the future.

So, like any developer would do, I started a web course. If you're interested, you can check it out on LinkedIn Learning.

What did I find out? COBOL is an elitist and segregating language. You have to have access to mainframes to code, and you have to have access to paid tools to code and debug. There is only one open-source compiler for COBOL, GnuCOBOL. And when I started my journey, there were zero open-source debuggers.

I was totally disappointed. How could that be? COBOL is a f**** old language. So I started digging around forums and I found an IBM learning platform that was oriented towards COBOL. If I remember correctly, the students got access to mainframes. But one thing caught my attention: there was another frustrated person complaining about the lack of open-source tools to code on a regular computer. They shared that they were writing a COBOL debugger for VSCode.

I tried out that tool and I thought it was a great idea. But soon I realized it wasn't mature enough to debug even one of the simplest tasks I did in the web course I mentioned.

Well, as we say in Rio, I had the cheese and the knife in my hands. I'm a developer, and I wanted to learn COBOL. So why not contribute to that tool? That little comment turned my course in a divergent direction. Soon I forgot about "how to read a file in COBOL" and started asking "how GnuCOBOL transpiles code into C", "how to debug C in the command line", and "how gdb integrates with VSCode."

In the middle of the project, I was already in touch with GnuCOBOL developers, asking them for new compiler features to help debug float and binary data types supported by that compiler. I was also adding support for JavaScript expressions in the debugger inspection view. And in my latest contributions, I was working on debugging remote processes and SSH tunneling.

All of this, all of those long nights coding, just because of a comment on a forum about COBOL.

And you know what? It was an amazing experience. I learned a lot of things, perhaps nothing directly useful for my current daily duties, but at least I contributed to an open-source project, impacting the lives of thousands of other anonymous developers.

See you | Bis geschwënn | Até mais | À bientôt

Quarkus, Kotlin and GraalVM

Bruno Pacheco — Fri, 22 Nov 2019 19:48:13 +0000

During a chat with László, I've asked him for some ideas to practice Quarkus and GraalVM, he suggested to code the classic game Snake (AKA Worms by me :P).

Motivations

Why Quarkus? I've read an article saying that Quarkus is a great and promising solution for Java in the Cloud, as it drastically reduces the application's footprint and has an insanely fast startup when running on native code.

GraalVM is a general-purpose VM for running applications written in a bunch of different languages, allowing a polyglot code. At first glance, I felt it was an amazing thing to try, and so I did. In the end, this first trial wasn't fruitful, due to some strange bug on a node app I was coding at that time, so I just gave up for that moment.

Kotlin has no reasonable arguments to be here except it is a brand-new Java dialect which I was already learning, but I have never used it on something substantial, so why not?

Starting the adventure

I've started the project just reading the guides on Quarkus.io, which are quite hands-on, but superficial. Quarkus is simple to use, as any recent framework for web development. So simple that in five minutes I had a rest service running locally... of course, an HTTP call returning a "Hello World" string doesn't deserve to be called a service, does it?

So, to start your project from scratch, they have implemented a bootstrap command. Please refer to the followin command:

mvn io.quarkus:quarkus-maven-plugin:1.0.0.CR1:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=getting-started \
    -DclassName="org.acme.quickstart.GreetingResource" \
    -Dpath="/hello"

And an easy way to add new dependencies:

mvn quarkus:add-extension -Dextensions="vertx"

Moving on, my first steps were: defining the domain, JPA, H2, a couple of CRUD endpoints and some DTOs. After having this square-shaped code, I decided to migrate it to Kotlin, and bugs started popping up as expected.

The first problem which I had to face was JPA. It requires open classes and by default it isn't true for Kotlin. To fix this, there's a flag to enable JPA support during compilation. I don't like this kind of solution, but this is somewhat my fault as I wanted to use a mature Java library instead of using something more Kotlin-like.

The second problem was the JSON payload serialization. I had some issues with JSON-B (the default option), and then I decided to try Jackson. Again, just for the sake of simplicity, as I was already testing tons of new things.

It worked nicely and the DTOs were cleaner than before. However, another plugin had to be added to the Kotlin compiler (the no-arg plugin option) and a custom annotation to help the compiler identifying the DTOs.

@Dto
data class MatchMapPlayer(
        var playerId: Long,
        var status: MatchPlayerStatus = MatchPlayerStatus.PLAYING,
        var wormLength: Int = 0,
        var direction: Direction = Direction.DOWN,
        var position: List<MapPoint> = arrayListOf()
)

The third problem was similar to the JPA issue. All managed beans have to be open. Can you guess the solution? Yes another Kotlin compiler flag.

So, in the end, my Koltin compiler configuration looked like the following:

<configuration>
  <compilerPlugins>
    <plugin>jpa</plugin>
    <plugin>no-arg</plugin>
    <plugin>all-open</plugin>
  </compilerPlugins>
  <pluginOptions>
    <option>all-open:annotation=javax.ws.rs.Path</option>
    <option>all-open:annotation=javax.enterprise.context.ApplicationScoped</option>
    <option>all-open:annotation=javax.enterprise.context.RequestScoped</option>
    <option>all-open:annotation=javax.ws.rs.ext.Provider</option>
    <option>all-open:annotation=javax.persistence.MappedSuperclass</option>
    <option>all-open:annotation=javax.persistence.Entity</option>
    <option>all-open:annotation=javax.persistence.Embeddable</option>
    <option>no-arg:annotation=com.dev.bruno.worms.dto.Dto</option>
  </pluginOptions>
</configuration>

I've also created an abstract Repository class to have a common behavior for all entity repos (copying Spring), but I had a problem with the dependency injection in the superclass.

I tried using constructor dependency injection, passing it from the child to the superclass using super, but it hasn't worked properly. The work around was injecting the EntityManager directly in the field, like this:

    @PersistenceContext
    protected lateinit var em: EntityManager

Then, I had to change all @OnToMany from List/Set to MutableList/MutableSet.

And finally, I don't remember why, but now all fields in the domain and DTO classes were var, instead of val. I have to check why again.

I've also implemented some business code, which is not important to discuss here (but if you are interested, go to
this link).

And GraalVM enters the scene...

After trespassing the first barrier, I was ready for the next boss (the final one?): compiling the project to native code.

According to Quarkus.io, this shouldn't be complicated as they aim to simplify native code compilation using their on maven plugin. Even their guides being simplistic, they are not lying. But they could have mentioned the Reflection limitation in GraalVM, which will be explained in the following lines.

So, to build the app, you have to run a similar mvn command:

mvn clean package -Pnative -Dnative-image.docker-build=true

By the way, I was building a containerized native code, that's why there's this docker-build arg. They also provide a dockerfile sample.

I thought at this stage that would be enough, but no. After running the build and running the app on Docker, I faced exceptions on Jackson. So, reflection is not supported by default on native code and it took me a while to understand why, and a bit more to find a way to fix it.

On a side note, the build consumes a lot of memory and CPU, almost killing my old Lenovo. :(

So, GraalVM does allow you to use reflection, but for that, you need to add a config file, located at META-INF/native-image/reflect-config.json (it can be in a different folder with a different name, but this is the default path).

First I decided not to manually write this file, and after googling I've found this blog presenting an agent to do this job. The idea is to observe the JVM during runtime and generate the config.

To do this, you simply have to run the app with the following command:

$JAVA_HOME/bin/java -agentlib:native-image-agent=config-output-dir=META-INF/native-image your.jar

After running the app with this agent, the generated file was hugely polluted and it didn't fix the problem anyway, as the agent didn't add all needed classes. So, I ended up typing the file manually. And surprisingly, I only had to add the DTO classes.

The file should look something like this:

[
  {
    "name": "com.dev.bruno.worms.dto.ExceptionResponse",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "fields": [
      {
        "name": "message",
        "allowWrite": true
      }
    ]
  }
  ...
]

Conclusions

I do recognize the benefits of using Kotlin such as nullability, immutability, less verbose code, tons of new coder-friendly functions, however they are not that important when you have clean code and efficient testing.

Also, I don't like polluted POM files, and it annoyed me having to add build plugins, compiler flags and configurations. I felt guilty about using JPA and Jackson though. Perhaps, if I had used something more Kotlin oriented, I wouldn't have had these issues. Let's see what happens in the next project.

Quarkus has an insanely fast startup indeed, and the memory footprint is great. I have used a micro instance on Google Cloud to test the game, and it handled properly the load, without crashing anything. Quarkus is a bit immature yet, as it required some refactoring when I decided to move to newer versions. Nonetheless, it is promising, as it has huge support for different technologies and libraries. I'll use it again.

I have loved GraalVM at first sight, as the wish of a polyglot code is in my mind for a while, and now it's becoming reality. In the end, I didn't use this feature, however, the native code generation fascinated me as well. I didn't like the way it consumes resources though. Compiling any other language code to native is not consuming 10% of what GraalVM has consumed from my machine (not judging...). I would wait for improvements on this area before using it on commercial solutions.

To sum up: I have enjoyed playing with all these new technologies all together. Due to time and availability constraints, I couldn't investigate more. Perhaps improving and refactoring the code. The final solution is not bad and it has proven to me that Java definitely can be used in the cloud, it's just a matter of investment in new solutions, like Quarkus and GraalVM.

Please, refer to the project repository to have a better picture of what I have coded. And feel free to comment out there.