The thousand dollars one line mistake - SBT + PlayFramework

Gustavo De Souza — Mon, 01 Jul 2024 11:28:26 +0000

Nowadays everyone talks about how important it is to have a good developer experience, as it will have many good side effects, such as but not limited to:

Development speed / Productivity
Code quality / Maintenance
Saving costs, etc

However, often we get ourselves working on projects that at some time in the past, a small piece of code was added to make the project faster, or even to fix something, maybe someone was trying to make the build faster, or even trying to give the engineers a better development experience. This was the case in this story.

A few years back, in a project that we worked on ( before I was even part of the company ) an issue with building SBT, Scala and play framework, was identified, where the compilation time for building the project locally was taking around 3 to 5 minutes depending the machine. An attempt to fix the issue was made. The project structure was split in 2 like below:
Before

ProjectA
  /api
 /core
 /app

After

ProjectA
 /core
 /app

ProjectApi
  /api

The following was added to the build.sbt

lazy val projectA = (project in file("."))
  .enablePlugins(...)
  .settings(commonSettings)
  .aggregate(api)
  .dependsOn(api)

lazy val api = project.settings(commonSettings)

By doing so, it did improve the compilation time, only during the build on the CI pipeline, I'm not sure if it helped during the development phase, however, it added a new and horrible bug that made developers waste thousands of hours of work.

After this line was added, developers started noticing how long it was taking just to run a simple
sbt run locally, as for every change in the codebase now, a full compilation was needed.

The journey to understand the issue

As per documented in SBT reference manual - Multiproject

Is important to note the two definitions of Aggregation and depends on

Aggregation means that running a task on the aggregate project will also run it on the aggregated projects

A project may depend on code in another project. This is done by adding a dependsOn method call. For example, if core needed util on its classpath.

After spending one day or two reading documents and many frustrated attempts to fix the issue, I ended up arriving at this Github - Spurious recompilation in multi-project build This was not the fix itself, however, gave me light by the end of the tunnel to understand the problem was indeed with the multi project setup.

And further more, I understood what was happening, and by so, now my build.sbt file was just as simple as:

lazy val projectA = (project in file("."))
  .enablePlugins(...)
  .settings(commonSettings)
  .dependsOn(api)

lazy val api = (project in file("api"))
  .settings(commonSettings)

There was a problem with how we set up projectA in SBT. We told SBT to include the project's API (which was right), but the API definition pointed to the entire project root. This meant that:

Whenever the API needed compiling, SBT would also try to compile projectA itself.
Since projectA needed the API to compile, it would trigger another API compilation.
This created an endless loop, forcing developers to kill SBT and manually clean and compile everything for every code change.
Here's what happened in simpler terms:

We told SBT to include the project's API.
The API definition pointed to the whole project.
Compiling the API triggered a full project compilation (including the API again).
This loop made SBT very slow and frustrating for developers.

The team had worked with this problem for at least 4 years...

Aftermath - fixing the issue

After I said to my teammates that I had a surprise feature already merged on master, people did not understand what was happening, however, I wanted to see the happiness on their faces, I told the whole team to pull master into any branch that they were working on, some of them did not notice anything in the first try, other started to notice that after changing any code in the codebase, it was compiling only the affected file in a matter of seconds, not minutes as before. And the best surprise was when one of the teammates noticed and said out of loud in the office.

Gust ... did you fix the compilation loop issue? I'm working in something here and I am getting instant feedback for any changes in the code.

At that time I had to admit and, share the news with all the other engineers, it made me a happier engineer as now I am happy working on this project and not waiting long period of times for a full compilation on our project.

If you feel that something is our way, whenever you are, whatever you are doing, remember you have the chance to make it better, never forget what you have started.

If you like reading this article or would like it to have more content, please let me know in the comments, I am happy to share more about this journey.

Thank you for reading.

A few stats from the project:

Project start:

Around 4 thousand Java files
Around 300 Twirl templates
Compile time before improvements 3 to 5 minutes for any change in the code
Compile time after improvements average of 1 minute and 20 seconds for full compile
Compile time after improvements average of 5 to 10 seconds for any change with instant feedback ( the most time spent is Playframework restarting the HTTP Server )

Cover image made by AI.

Building Java Swagger 3 Intellij plugin: Boosting productivity, type safe generation and documentation adoption.

Gustavo De Souza — Tue, 14 May 2024 14:26:08 +0000

Over the past couple of years, I’ve been part of a dynamic team of 12 engineers. Wrangling all our API Dtos, models, and TypeScript objects to stay in harmony has been quite the adventure! Whenever we need to tweak a DTO, it’s like embarking on a treasure hunt: “Where does this ripple effect lead? Will adding a new field here trigger a cascade of chaos elsewhere?”

Issue #1: Syncing Struggles

The other biggest issue, is that for each and every DTO on our backend, we need the same TS type for the frontend, and this was often seen as a productivity drainer, as this was leading to duplicated code within the backend and frontend, many bugs due to some changes in the backend not reflecting the types in the frontend and so on.

The pluggin

Intellij Swagger 3 Java plugin by Gustavo de Souza
For these reasons and many more. I have adopted Swagger 3 in our project, in fact the frontend team has been really happy getting all types auto generated out of the box with Swagger 3 and OpenAPI specs, however, they were not satisfied with documenting all the current endpoints that they already use and migrating them, for this reason I have build the Java Swagger 3 Generator that added support to a few actions and some “magic” tricks behind the scenes.

Swagger Generator Actions Intellij by Gustavo de Souza
If you’re diving deep into Swagger 3 territory and craving a shortcut to endpoint enlightenment, behold the plugin! Here’s a taste of its powers:

Calling All Swagger Fans!

If you feel that you heavily use Swagger 3 to your projects and are trying to find something that will boost documenting endpoints, I would suggest giving a try to the plugin.

And the most important one is SwaggerGenerate that will auto add Swagger annotations to Models, Dtos, Controllers …

This action will run a few checks in the current class and define which one it could be e.g

Controllers
Will add @Tag annotations and map all methods that have any Spring REST annotations and/or Jax-RS annotations.
Models / DTO’s
Will add @Schema annotation for any field, and mark them as required or not based on type, validations, and other constraints, fully supporting javax.validation.* or jakarta.validation.*

The fact that just by creating something simple, that can help you, your team, and even make the engineers a bit more happy, is satisfying and has a nice feeling. Sometimes we tend to focus on solving only our daily tickets, or building things that were planned in the previous spring, however, many times we can step out a bit, and look things from a different perspective, and in fact create something that will benefit everyone around you.

Sometimes, being a better engineer is as simple as lending a hand to your fellow engineers. Forget the daily grind; let’s sprinkle a little happiness dust on our projects! By thinking outside the code, we can craft solutions that make everyone’s day a little brighter.

Source code on Github :)

— — — — —

So, if you dug this article and crave more tech tales, dive deeper into the world of Swagger and watch the magic unfold!