DEV Community: Julien Lengrand-Lambert

KotlinConf 2025 is a real bowl of fresh air for backend Devs

Julien Lengrand-Lambert — Thu, 22 May 2025 14:34:02 +0000

TL;DR : After years of focus on the compiler and KMP, The JetBrains folks are coming it out with tons of announcements for server folks too, and that feels great

The past few years

You probably know me by now, I've been liking Kotlin a lot for a long time. Back in 2019 already we were the first at ING moving some of of our production code from Java to Kotlin. Over the past years though, I've also been vocal online about my worries that all of the love was going towards KMP. And Google deciding to sunset the (non Android) Kotlin category has just been another pointer in the same direction. I've even mentioned this exact thing already right after visiting KotlinConf 2023.

[

My KotlinConf experience

My experience and impressions attending KotlinConf a couple weeks back. And some thoughts on the future of Kotlin.

Julien's DevRel cornerJulien

](https://dev.to/jlengrand/my-kotlinconf-experience-45mf)

Even with Java releasing more often, and getting more and more new features every iteration I'm still much happier using Kotlin. But for pure backend developers is the developer experience still that differentiating that it makes it worth switching? After all, Java got pretty good records those days. A much better functional programming experience. And virtual threads. The list goes on....

Note : This article is my personal impression based on my usage of the language. Depending what you're working on, your opinion may vary 😊. I've also included many screenshots from the Keynote itself, please watch it for the complete announcements!

Note 2 : There's also been lots of great AI related announcements, but I'll intentionally skip them in this article. I'm generally happy JetBrains picks up the AI wave and innovates while staying open. I'm a mostly happy user of their product myself.

New language features

As a backend engineer, I think the main improvements I've personally seen in Kotlin that made a difference to me the past years are the time API (back in 2021!) and the K2 compiler. Of course, the K2 compiler is a huge (and needed) improvement that came with Kotlin 2.0. But it also didn't bring a huge amount of new language features.

Now, today's keynote was a VERY welcome change to this. Others will do a complete breakdown better than me but here some actual new language features I'm excited about!

Name-based destructuring to ensure I am grabbing the right properties. An actual improvements compared to the current version.

Screenshot from Day 1 Keynote: https://www.youtube.com/live/PYAPymKRKVA?si=lKlYZoRzImgKMptb

Rich errors : an improvement to error handling, sitting on top of the already great null safety features of the language. Not only do we now have complete return types but they also get propagated so we can neatly handle all error cases in a single location.

Screenshot from Day 1 Keynote: https://www.youtube.com/live/PYAPymKRKVA?si=lKlYZoRzImgKMptb

Even Kotlin compiler plugins are getting more powerful, with Power Assert offering an expressive and clear error message. Look at this, we're almost topping the already amazing Elm error handling capabilities.

Screenshot from Day 1 Keynote: https://www.youtube.com/live/PYAPymKRKVA?si=lKlYZoRzImgKMptb

This is developer experience like I haven't seen in a long time from the teams at JetBrains, and it genuinely makes me happy.

Tooling ecosystem, and strategic partnerships

I've already mentioned lately that I was a big fan of Kotlin notebooks. I find them generally amazing, because they combine the unique experience of Python Notebooks with the gigantic JVM ecosystem (and without the Python horrible dependency management issues).

[

Julien Lengrand-Lambert 🥑👋 (@jlengrand.bsky.social)

Been doing some data analysis lately and #Kotlin notebooks, together with dataframe, kandy and sqlite are a really freaking powerful combination! Can’t wait to show you the results :)

Bluesky Social

](https://bsky.app/profile/jlengrand.bsky.social/post/3lgg7aa5yxe2l?ref=lengrand.fr)

I was legit 🤯 this morning when I saw that the folks at JetBrains combine this already amazing experience together with the ecosystem that pretty much every JVM dev uses on the planet : Spring!

The screenshots aren't doing justice to the announcement, so please just go and have a look at the Keynote yourself. The latest version of notebooks can attach itself to a running Spring kernel and have access to all of its context. And that even for applications that aren't using Kotlin yet.

Kotlin notebook, straight from a running Spring kernel

Again, this is next level for me in terms of developer experience : combining the flexibility of notebooks for experimentation together with the sturdiness of a production like application. With the kandy and dataframe data manipulation capabilities, I can definitely see this speed up my day to day development speed.

And of course, JetBrains also announcing a strategic partnership with Spring on stage shows that they clearly intend to give us "corporate" backend developers some serious love.

Growing Kotlin Foundation

For a long time, Kotlin was completely in the hands of JetBrains. This has massively changed over the past years with the creation of the Kotlin Foundation.

As a professional having to pick a technology, it is crucial to look at how promising its future looks. Seeing that Kotlin is jointly supported by several companies, and especially companies that aren't making a living of the language makes it a much safer business case to adopt inside a company.

Seeing two more companies (and none other than Meta) join the foundation gives a great idea of how bright of a future Kotlin has 😊.

Kotlin foundation members

The Kotlin built Java library ecosystem

A few weeks back, I was mentioning that for the first time I found a library that was completely built in Kotlin, but had a clear Java compatibility too. If you're interested, I'm talking about the KBsky.

[

Julien Lengrand-Lambert 🥑👋 (@jlengrand.bsky.social)

Interestingly, this is the first time ever that I find a more mature ecosystem of libraries for my need in #kotlin than in #java quite interesting isn’t it. I was looking for ATProto libraries, it seems the main java was got deprecated in favor of a kotlin version? https://github.com/uakihir0/bsky4j

Bluesky Social

](https://bsky.app/profile/jlengrand.bsky.social/post/3lmk4xy6s2o2q?ref=lengrand.fr)
A Java library, built in Kotlin

Today, I learnt that the two most successful JVM AI libraries (the OpenAI Java SDK and the Anthropic Java SDK) both are actually written in Kotlin.

See Keynote for complete info : https://www.youtube.com/watch?v=PYAPymKRKVA

I find this a great sign for the language. Of course, the amount of Java developers out there is much larger than people writing Kotlin and it makes sense for library builders to want to maximize their reach. But them choosing Kotlin to write the library speaks volume about the quality of the language, but is also a great sign for its future.

The official KOTLIN LANGUAGE SERVER

This is, by far, the announcement I am most excited about and I genuinely didn't see it coming. JetBrains announced the official release of their Kotlin Language Server protocol today.

For those unaware, the language server protocol "defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc.". Having an official language server protocol implementation basically means that all IDEs (from vim to IntelliJ, or SublimeText) can support the language equally well.

The fact that JetBrains is behind Kotlin always made it rather logical for them to not Open-Source a language server. After all, they're selling their IDEs and that'd pretty much mean they're competing against themselves. And it's always been one of my biggest issues with the language.

[

Julien Lengrand-Lambert 🥑👋 (@jlengrand.bsky.social)

Agreed. Absolutely love IntelliJ and the company behind it, but I don’t like have to start the powerhouse all the time for some simple scripting. Let me use IntelliJ for feature full, multiplatform stuff and give me a powerful language server for the simple backend / scripting thing please

Bluesky Social

](https://bsky.app/profile/jlengrand.bsky.social/post/3lfadcojxvk2a?ref=lengrand.fr)

Yes, IntelliJ is a great IDE. I use it every single day, have paid for my license for years and I love it. But in simply just don't believe in the success of gated products. More and more people use VSCode every day. People now use Cursor. Or SublimeText. Some still use vim. Forcing folks to use a specific IDE just feels wrong and tames the excitement using a new language should bring.

Yes, there's been community driven language server implementations for Kotlin, but they've always been struggling with support. And literally being a company building IDEs for a living, JetBrains always felt like the best party to handle this properly.

The main kotlin language server implementation struggling with support

I'm over the moon that JetBrains has the courage to open up and even lead the language server efforts in the future. It shows a strong trust in the future of the language and that can only help drive adoption across the industry.

ING as use case

As I mentioned already, I was part of the first team to start using Kotlin at ING back in 2019. Five years later, adoption has grown a lot internally and I was super happy to see JetBrains offer us to be one of the leading industry use cases for their server side Keynote.

Kotlin powers some of our critical services today, and it's a great symbol of how good and powerful the language is. I can't wait to see where we'll be in 5 years.

Simone, one of my colleagues, talking about Kotlin adoption at ING. Well done Simone!

I've always been a fan of Kotlin, and it's been an honour to be able to contribute to the content of the program, no matter how small my contribution was.

Recording the video, behind the scenes

Conclusion

There's much more I'd like to talk about (amper, klibs, kmp, ...) but I'll keep it to the essential : It feels to me with their keynote that JetBrains intentionally started putting backend developers in the spotlight again. They made big, impactful announcements, both at an ecosystem and language levels and I can't wait to try out all of the new goodies.

Well done JetBrains, well done!

Visualizing your AAARRP priorities as a way to manage up in your DevRel team

Julien Lengrand-Lambert — Tue, 12 Dec 2023 09:46:10 +0000

TL;DR : we have been using the AAARRRP framework as a visual and easy way to align with stakeholders which types of activities are the most relevant for our customers.

Some context

Even though I have been doing Developer Relations for a few years unofficially or semi-officially, it was only two years ago that I got my first official role as a Developer Advocate. And after just a few months, I was offered to lead the team. We're a small team of three (me included) and our team falls under engineering.

It was never really an issue for us to create a strategy on what to achieve, or how to achieve it. We focussed on our 3 pillars: Code, Content and Community and selected the most relevant activities.

For example, being a B2B company we knew that running the conference circuit wasn't the most efficient way to create value, for the simple reason that people wouldn't be able to sign up for our services. We quickly decided instead to embed ourselves close to documentation and provide many samples for the companies who were using us, bringing a lot of internal feedback at the same time.

But even though we had no doubt about our priorities, it wasn't always as clear for higher management, nor other stakeholders. After all, Developer Relations was a relatively new thing inside the company and only a few people had direct experience with the domain in the past. And we all know that even for companies seasoned with DevRel, measuring impact and making sure activities align are a difficult topic.

We received a lot of challenging questions over time regarding our priorities :

We see that the insert competitor DevRel team is very present on Youtube. How is it that your are not doing the same?
How is it that your Twitter account gets very few likes compared to insert other company.
Why do we not see many community contributions on your GitHub account, compared to insert large company?
Would you please join us at the booths for insert conference, we want to increase hiring.

All those questions are very fair, and they deserve clear answers. Thing is, they were quite obvious for us given the context we had and it was a struggle for me to communicate those answers at scale and in a strategic way (understand : Find a way that people understand those answers, and hopefully give enough context that they come to the same decisions by themselves).

It's after reading about the AAARRRP framework that a potential solution came to mind 😊.

Quick sideline about priorities / activities

My point with this method is to focus on how to communicate in a scalable way the type of activity we plan on focussing on, not the exact activity performed.

For example, we might want people to understand that writing code samples is the most impactful activity we can do.

However, we will not use it to communicate which code sample to write. For this, we might instead use a combination of data sources like documentation statistics, or support tickets for example. I may write more about this later in another article.

A quick intro about the AAARRRP framework

I'm not going to expand myself much about this, because I would be ripping off the original article. I recommend you to go read it instead if you have never heard of it before.

Here is the very minimal version. As Developer Relations team, we can operate on several key moments of the customer journey :

The AAARRRP framework funnel

Depending on the type of company we're working on and the strategic priorities of that company, some of those parts will be more crucial than others.

Early on, many SAAS products want to focus on awareness and acquisition to drive as many new customers in as possible, and grow fast (The famous MAD) .

As your product grows in maturity, you may want to increase the "stickiness" of your product and increase retention , or work on activation of additional features for your existing customer base.

Typical DevRel activities that we carry have impact of different parts of that funnel (usually several at the same time).

Activities and funnel mapping. Original from https://www.leggetter.co.uk/aaarrrp/

Note: The original post also adds weight to the equation in order to select activities. We will not here, for simplicity's sake. The whole idea stays valid here.

From AAARRRP to strategic communication

Note: All those charts have been altered and do not contain internal information 😊

So far, nothing new under the sun. That's where we'll go just one step further and use the content above for strategic communication.

Let's say that based on the current context, we decide to focus on activation, retention and product.

We can draw this as a chart to help communication with stakeholders.
We can also draw (rough, given we do not have internal knowledge) equivalent charts for other companies in the space for comparison.
We can even draw our own chart year on year, to show how our activities vary as priorities shift, or impact decreased (law of diminishing returns).
Now for the key part : We use these charts to validate assumptions with stakeholders, who will be able to derive the corresponding activities themselves.

For example, given the decision above we can draw this:

Possible DevRel priorities focussing on activation, retention and product

With those priorities agreed, the common understanding is that we should focus on activities like Code Samples, Guides, Tutorials, and answering Stack Overflow questions.

Again, we can more easily explain why we have a lesser focus on social media and the conference circuit that others by comparing main focuses between entities.

possible DevRel team profile from another company focussing more on MAD

Once the shape of our strategy is agreed to, the type of activities expected become clearer for everyone and we can focus on the impact of those activities. And if the priorities change, it's always time to come back to the drawing board 🎉.

Let's imagine now that our company's internal focus heavily shifts towards "closing deals" because we need to extend our runway. It might be time for the DevRel team to start acting on the Revenue part of the funnel, reduce Product efforts and do Pre-Sales activities. We can update the plan, propose a new shape and double check with stakeholders that the change fits the new landscape :

DevRel team proposing to shift priorities following a company strategic change

Of course, whether the type of activities to be carried on fits DevRel well, is sustainable long term and more is another discussion altogether. The main point here is to make your priorities clear and communicate them accordingly.

About the scale of priorities

One question you might wonder is : How do you calculate the scale of each priority in your bar chart?

Well, so far our method isn't very scientific 😊. We mostly want to show relative priorities of activities and as such we're using a set number of points and sharing them across the different axes of the chart. For example, pick 70 points (10 per angle) and subdivide. The chart then helps us make decisions quarter to quarter or day to day. Of course, when drawing other team's chart, all numbers are best effort and based on impressions.

As we use these graphs more, we're experimenting with other ways to measure that could be more relevant. For example incorporating alignment and relevant weights that the original AAARRRP article mentions.

A word of conclusion

Overall, the only thing we've been doing is taking the AAARRRP framework as an inspiration to clearly visualise what our team focusses on and bring some clarity internally. We've been using it on all of our presentations and internal pages focussing on our team's strategy.

It has helped to focus discussions with stakeholders more on how much impact we can bring, rather than what activities to carry and why and it's really been useful for us.

Having joined DevRelCon this year I know that internal communication is always a strong topic in DevRel teams. Maybe this can help bring some clarity on the topic!

Cheers, and talk soon.

Julien

Creating an OpenAPI generator from scratch : From YAML to JetBrains HTTP Client

Julien Lengrand-Lambert — Sat, 04 Nov 2023 17:40:33 +0000

This is the online version of the article with the same name I wrote for the Dutch Java Magazine 😊.

Creating your own code generator for the @jetbrains http client from an #openapi specification? It’s possible and you can read how in the latest edition of the Dutch #java magazine!

And yes, it’s Jetbrains, not yet brains 😂 pic.twitter.com/zLGkpibLtN

— Julien Lengrand-Lambert 🥑👋 (@jlengrand) November 3, 2023

In the previous edition of the magazine, we discussed how the JetBrains HTTP Client could be used to run HTTP Queries, automate them and even use them in your CI/CD pipelines. Just like Postman, but text based and can be part of your source code. Pretty cool.

For reference, it could look like this.

### Github API - Traffic per day

GET https://api.github.com/repos/{{owner}}/{{repo}}/traffic/views?per=day
Accept: application/vnd.github+json
X-GitHub-Api-Version: 2022-11-28
Authorization: Bearer {{github_key}}

With an environment file that looks like this :

{
  "dev": {
    "github_key": "not_that_easy",
    "owner": "jlengrand",
    "repo": "elm-firebase"
  }}

Now, that is very nice, but it requires a lot of manual work. Wouldn’t it be nice to be able to automate this? Fortunately, most of us developing APIs also generate OpenAPI Specifications for them. When I looked however, there was no OpenAPI generator yet available for the Jetbrains HTTP Client. This is the story of how I've implemented it from scratch, and how you could too if you find yourself in the same situation! We'll use the JetBrains HTTP Client as a practical example, but the knowledge is transferable 🙂.

The OpenAPI generator project contains a core engine, as well as many packages with each a specific generator (Java, Ada, …). The Jetbrains HTTP Client generator is actually published, and you can find the merge request as well as the documentation on GitHub. If you have installed the last available OpenAPI generator release, you can actually try it out in a terminal as such :

$ openapi-generator generate -i https://api.opendota.com/api -g jetbrains-http-client -o dotaClient

At its core, the idea of the OpenAPI generator is quite simple : It takes a specification file (JSON or YAML), transforms it into a set of objects in memory, and uses those objects to generate code / files using mustache template files.

You can actually find most of that logic in the DefaultGenerator source file of the library. There, you can see that actions are separated into 3 groups :

models (basically data types)
operations (actual operations)
supporting files (environments, READMEs, …).

Each of those is illustrated by a method, and takes separate objects as inputs :

 …
  void generateModels(List<File> files, List<ModelMap> allModels, List<String> unusedModels) {…}
…
    void generateApis(List<File> files, List<OperationsMap> allOperations, List<ModelMap> allModels) {...}
…
    private void generateSupportingFiles(List<File> files, Map<String, Object> bundle) {...}
…

You can find the actual source file on GitHub. The objects for each of those methods are large Map classes that contain the necessary data in a semi-structured format. Here is an example of how allModels looks like:

a debug view of the allModels object

As you can see, the object is essentially a lot of key/value pairs that are quite recognisable and directly come from the OpenAPI specification file.

To create our own client, we will take advantage of this nice work. Let's dive into it. We first clone the repository

$ git@github.com:OpenAPITools/openapi-generator.git; cd openapi-generator

We can then use the /new.sh script to generate a few placeholder files for us. We'll be generating a client, and since we're not creating any bugs we won't be generating test files.

$ ./new.sh -n java-magazine-client -c

Creating modules/openapi-generator/src/main/java/org/openapitools/codegen/languages/JavaMagazineClientClientCodegen.java
Creating modules/openapi-generator/src/main/resources/java-magazine-client/README.mustache
Creating modules/openapi-generator/src/main/resources/java-magazine-client/model.mustache
Creating modules/openapi-generator/src/main/resources/java-magazine-client/api.mustache
Creating bin/configs/java-magazine-client-petstore-new.yaml
Finished.

The library nicely generates a client generator for us, as well as some template files and even a config so we can test it easily! The config uses the well known petstore by default.

This is how the config file looks like :

generatorName: java-magazine-client
outputDir: samples/client/petstore/java/magazine/client
inputSpec: modules/openapi-generator/src/test/resources/3_0/petstore.yaml
templateDir: modules/openapi-generator/src/main/resources/java-magazine-client
additionalProperties:
  hideGenerationTimestamp: "true"

It nicely mentions to the OpenAPI generator library which generator to use, which sample OpenAPI file to use as input, where the mustache template files are located and where to store the output

Let's run it!

$ ./mvnw clean package # package once to have the generator inside the generated jar
$ ./bin/generate-samples.sh bin/configs/java-magazine-client-petstore-new.yaml

Let's see what the generated output looks like :

a tree view of the generated client

We haven't done any work yet, and our generator is already spitting out things! Unfortunately, as we can see, all those files are empty. That's because our mustache template files also are empty. Let's fix that now!

We'll start by customizing the JavaMagazineClientClientCodegen to fit our needs. We want a very minimal implementation that fits in this article, so we'll actually decide to NOT implement any supporting files (the README), nor Models and instead focus solely on the API. The way to do this in a custom generator is to extend the postProcessOperationsWithModels from the CodeGenConfig interface. We change the .zz extension into .http files that will be recognised by IntelliJ. And because in this specific (simplistic) case, we will not need any alterations to the OperationsMap object we can actually only call the super method. Our final class looks like this :

package org.openapitools.codegen.languages;
import org.openapitools.codegen.*;
import java.io.File;
import java.util.*;
import org.openapitools.codegen.model.ModelMap;
import org.openapitools.codegen.model.OperationsMap;

public class JavaMagazineClientClientCodegen extends DefaultCodegen implements CodegenConfig {

    public CodegenType getTag() { return CodegenType.CLIENT; }

    public String getName() { return "java-magazine-client"; }

    public String getHelp() { return "Generates a java-magazine-client client."; }

    public JavaMagazineClientClientCodegen() {
        super();

        outputFolder = "generated-code" + File.separator + "java-magazine-client";
        apiTemplateFiles.put("api.mustache", ".http");
        embeddedTemplateDir = templateDir = "java-magazine-client";
        apiPackage = "Apis";
    }

    @Override
    public OperationsMap postProcessOperationsWithModels(OperationsMap objs, List<ModelMap> allModels) {
        return super.postProcessOperationsWithModels(objs, allModels);
    }
}

Note: At first glance, the method and variable names may look a bit like magic. It is because most of the logic comes from DefaultGenerator, and CodeGenConfig. If you feel lost, those two classes are where it's at.

Now that we have our baseline, what we want to do is work on our mustache files. Those files are basically templates that will be fed into the processing pipeline to generate our .http files.

We know we want one file per main API endpoint, with some documentation. We also want the @name unique identifier from the Jetbrains HTTP Client to be able to reference our code. Finally, we want to add the supported content type for the calls.

If we look at the data object available for operations, we end up with this, where each {{item}} notation is the value of the item key inside the object.

## {{classname}}
{{#operations}}
{{#operation}}

### {{#summary}}{{summary}}{{/summary}}
# @name {{operationId}}
{{httpMethod}} {{basePath}}{{path}}
{{#consumes}}Content-Type: {{{mediaType}}}
{{/consumes}}
{{/operation}}
{{/operations}}

We can see it clearly if we look at the object during processing

Note: Unfortunately, to my knowledge the best way to dive into the data model is still to go pause at runtime, I haven't yet found a complete data model documentation online. If you do, let me know!

Let's rerun the generation and see what we get now :

$ ./mvnw package 
$ ./bin/generate-samples.sh bin/configs/java-magazine-client-petstore-new.yaml

the result of the generation of our client

Great! Only API files, and one per API, as wanted. Let's see what they contain!

## PetApi

### Add a new pet to the store
# @name addPet
POST http://petstore.swagger.io/v2/pet
Content-Type: application/json
Content-Type: application/xml

### Deletes a pet
# @name deletePet
DELETE http://petstore.swagger.io/v2/pet/{petId}

Looks great to me! Let's try to run one of the calls

Running one of the calls that's just been generated

It works just fine, and we get a 200 response as well. Success!

Now, there's only one little issue. The variables! In the Jetbrains HTTP Client format, variables are written as {{variable}}. OpenAPI implementations only have single braces. We need to fix that!

What we'll be doing here is implement a custom mustache lambda that doubles up the braces when it finds them. The lambda is essentially a string replacement

public static class DoubleMustacheLambda implements Mustache.Lambda {
        @Override
        public void execute(Template.Fragment fragment, Writer writer) throws IOException {
            String text = fragment.execute();
            writer.write(text
                    .replaceAll("\\{", "{{")
                    .replaceAll("}", "}}")
            );
        }
    }

In order to make it available in our Generator, the openapi generator library offers the same mechanism as for the rest : We have to override a ready-made method.

    @Override
    protected ImmutableMap.Builder<String, Mustache.Lambda> addMustacheLambdas() {

        return super.addMustacheLambdas()
                .put("doubleMustache", new JavaMagazineClientClientCodegen.DoubleMustacheLambda());
    }

The code above is added to our JavaMagazineClientClientCodegen class.

Next, we also need to modify our mustache template to add that lambda at the right location (around the path parameter). If that path is a variable, the braces will then be doubled

## {{classname}}
{{#operations}}
{{#operation}}

### {{#summary}}{{summary}}{{/summary}}
# @name {{operationId}}
{{httpMethod}} {{basePath}}{{#lambda.doubleMustache}}{{path}}{{/lambda.doubleMustache}}
{{#consumes}}Content-Type: {{{mediaType}}}
{{/consumes}}
{{/operation}}
{{/operations}}

Et voilà! Running the sample again, we can now use variables as they are meant to be inside IntelliJ!

In the sample below, I'm using the following local environment file :

{
  "dev": {
    "petId": 3
  }
}

There is still a lot more to do with this generator. READMEs, payload, auth, headers, … But now it's a matter of updating the mustache files as we want.

I'd love to have a more fleshed out generator, because it'd be an amazing and cheap way together with the client CLI to have a great automated integration tests pipeline and get people running in second with your API.

I hope this article made you feel like trying to create your own OpenAPI generator. The only limit is your imagination! And as you can see, the merging process is actually relatively pleasant, because the volunteers of the project LOVE to see people bringing out new ideas to life.

Happy to hear your thoughts, as always!

[Unit] Testing Supabase in Kotlin using Test Containers - PART 2

Julien Lengrand-Lambert — Fri, 20 Oct 2023 21:16:46 +0000

TL;DR : You can run a full Supabase instance inside Test Containers quite easily. See this repository.

In my last article, I was listing a few attempts I had done at running tests against my Kotlin Supabase application. The way the Supabase-Kt library is built makes it hard to mock, and I ended up building a minimal Docker Compose setup that was mimicking a Supabase instance.

In this second part, we're gonna push the madness further and actually run a FULL SUPABASE instance locally, still using Test Containers.

Right after I finished pushing my repository last week, I realised that Supabase actually offered a Docker Compose file to self-host their platform. So I decided to push the madness further and see how easy it was to use that file inside TestContainers. In short : Relatively easy.

The setup

The setup isn't actually much different from my homecrafted Docker Compose. version. Here it is in its entirety.

A few notable things:

I'm relying on a local clone of Supabase, and point a ComposeContainer to the src/test/resources/supabase/docker/docker-compose.yml file.
The setup uses an .env file, so I use a dotenv implementation to grab the parameters there and make the code slightly dynamic.
I have to run a database statement to populate and flush my database in between tests. The Docker Compose setup from Supabase comes with persistent volumes, which needs to be accounted for.
I don't have test for those here, but all services (auth, functions, storage), ... should actually be supported, given that we're running a full local instance.

import io.github.cdimascio.dotenv.dotenv
import io.github.jan.supabase.SupabaseClient
import io.github.jan.supabase.createSupabaseClient
import io.github.jan.supabase.postgrest.Postgrest
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterAll
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.BeforeAll
import org.junit.jupiter.api.Test
import org.testcontainers.containers.ComposeContainer
import org.testcontainers.junit.jupiter.Container
import org.testcontainers.junit.jupiter.Testcontainers
import java.io.File
import java.sql.DriverManager

@Testcontainers
class MainKtTest {

    @Test
    fun testEmptyPersonTable(){
        runBlocking {
            val result = getPerson(supabaseClient)
            assertEquals(0, result.size)
        }
    }

    @Test
    fun testSavePersonAndRetrieve(){
        val randomPersons = listOf(Person("Jan", 30), Person("Jane", 42))

        runBlocking {
            val result = savePerson(randomPersons, supabaseClient)
            assertEquals(2, result.size)
            assertEquals(randomPersons, result.map { it.toPerson() })

            val fetchResult = getPerson(supabaseClient)
            assertEquals(2, fetchResult.size)
            assertEquals(randomPersons, fetchResult.map { it.toPerson() })
        }
    }

    companion object {

        private const val DOCKER_COMPOSE_FILE = "src/test/resources/supabase/docker/docker-compose.yml"
        private const val ENV_LOCATION = "src/test/resources/supabase/docker/.env" // We grab the JWT token from here

        val dotenv = dotenv{
            directory = File(ENV_LOCATION).toString()
        }

        private val jwtToken = dotenv["SERVICE_ROLE_KEY"]
        private val dbPassword = dotenv["POSTGRES_PASSWORD"]
        private val db = dotenv["POSTGRES_DB"]

        private lateinit var supabaseClient: SupabaseClient

        @Container
        var container: ComposeContainer = ComposeContainer(File(DOCKER_COMPOSE_FILE))
            .withExposedService("kong", 8000)
            .withExposedService("db", 5432) // Handy but not required

        @JvmStatic
        @AfterAll
        fun tearDown() {
            val dbUrl = container.getServiceHost("db", 5432) + ":" + container.getServicePort("db", 5432)

            val jdbcUrl = "jdbc:postgresql://$dbUrl/$db"
            val connection = DriverManager.getConnection(jdbcUrl, "postgres", dbPassword)

            try {
                val query = connection.prepareStatement(
                    """
            drop table public.person;
        """
                )

                query.executeQuery()
            } catch (ex: Exception) {
                println(ex)
            }
        }

        @JvmStatic
        @BeforeAll
        fun setUp() {
            val supabaseUrl = container.getServiceHost("kong", 8000) + ":" + container.getServicePort("kong", 8000)
            val dbUrl = container.getServiceHost("db", 5432) + ":" + container.getServicePort("db", 5432)

            supabaseClient = createSupabaseClient(
                supabaseUrl = "http://$supabaseUrl",
                supabaseKey = jwtToken
            ) {
                install(Postgrest)
            }

            val jdbcUrl = "jdbc:postgresql://$dbUrl/$db"
            val connection = DriverManager.getConnection(jdbcUrl, "postgres", dbPassword)

            try {
                val query = connection.prepareStatement(
                    """
                create table
                    public.person (
                                    id bigint generated by default as identity not null,
                                    timestamp timestamp with time zone null default now(),
                                    name character varying null,
                                    age bigint null
                ) tablespace pg_default;
                """
                )

                query.executeQuery()
            } catch (ex: Exception) {
                println("Error is fine here. This should actually run only once")
                println(ex) // Might be fine, this should actually run only once
            }
        }
    }
}

To achieve those results, a few manual steps are required. The Docker Compose file provided by Supabase uses container_name parameters, which aren't supported by Test Containers.

I needed to :

Clone the Supabase repository locally
Copy the env file
Run some magic to remove container_name
Once in a while, the Supabase repository will have to be pulled

The results

The results are as outrageous as I expected them to be, if not more : All tests are running fine, though it takes almost 1 minute to run them. The ComposeContainer is starting no less than 12 containers (!!!) so it is to be expected.

Obviously, that setup is not to be used for unit testing. That being said, I find it absolutely freaking cool to be able to recreate your complete environment locally that easily, and I'd definitely consider that an option for bigger integration tests. The confidence I didn't have with my home brewed Docker Compose file is much higher now, given that it's directly provided by Supabase. No network needed to run my tests, pretty cool.

Gradle running those massive tests just fine

What more

My original complete intent was to build a small layer on top of the Docker Compose file, kinda like AtomoicJar does it with its modules. It would have been cool to have a simple interface for a Supabase instance to start, while providing a locally for starting scripts, user roles, maybe a new set of credentials, ...

Here is how they describe it for NGinx for example. I would have loved to have something similar :

@Rule
public NginxContainer<?> nginx = new NginxContainer<>(NGINX_IMAGE)
    .withCopyFileToContainer(MountableFile.forHostPath(tmpDirectory), "/usr/share/nginx/html")
    .waitingFor(new HttpWaitStrategy());

All of the implementation I've seen extend from GenericContainer though, not ComposeContainer so I've decided to hold that off and keep it simple for now.

Could maybe be something for the future, who knows.

In conclusion

That was a fun experiment, in which I've learnt more about TestContainers 😊. I'm as happy as usual with the way Supabase shows love for their users. Providing a seemless Docker Compose like this allows for a great experience. And I'm also impressed with TestContainers and how they can run such complex flaws without breaking a sweat!

If anything, I'd like them to at least ignore the container_name parameter if possible. I've seen many folks being blocked by it, and I can imagine many cases, like this one where people are not in control of their compose file. I don't necessarily ask for support, but an option to ignore without throwing an exception would be great.

That's it folks, till next time!

[Unit] Testing Supabase in Kotlin using Test Containers

Julien Lengrand-Lambert — Wed, 11 Oct 2023 22:01:10 +0000

TL;DR : The easiest way I found to test my database service is to mimick Supabase using Docker Compose and Test Containers. Here's the code

In case you don't know it, I'm a big fan of Supabase. I love that they're a viable alternative to Firebase. I love that they're built on top of Open-Source pieces. I love how innovative they are, and how much they give back to the community. And as you already know it, I love Kotlin as well.

Lately, I've been building a side-project which consists of a Ktor webapp, and uses Supabase-Kt to communicate with the database. I've been looking into ways to test my Kotlin component that interacts with the database, and it's been harder than expected.

In this article, I'll dive into several methods I've been looking into and why I finally decided to go for a Docker Compose / Test Containers solution. You can check the example repository here AND THE FINAL CODE HERE.

What I want to achieve

Let's imagine a minimal code example that contains a Person data class, and wants to save/fetch persons via a SupabaseClient. It can look like this:

@Serializable
data class Person (val name: String, val age: Int)

@Serializable
data class ResultPerson (
    val id: Int,
    val name: String,
    val age: Int,
    val timestamp: String
)

fun main() {
    val supabaseClient = createSupabaseClient(
        supabaseUrl = "",
        supabaseKey = ""
    ) {install(Postgrest)}

    runBlocking {
        savePerson(listOf(Person("Jan", 30), Person("Jane", 42)), supabaseClient)
    }
}

suspend fun getPerson(client: SupabaseClient): List<ResultPerson> {
    return client
        .postgrest["person"]
        .select().decodeList<ResultPerson>()
        .filter { it.age > 18 }
}

suspend fun savePerson(persons: List<Person>, client: SupabaseClient): List<ResultPerson> {
    val adults = persons.filter { it.age > 18 }

    return client
        .postgrest["person"]
        .insert(adults)
        .decodeList<ResultPerson>()
}

The SQL definition of our table looks like this

create table
    public.person (
                    id bigint generated by default as identity not null,
                    timestamp timestamp with time zone null default now(),
                    name character varying null,
                    age bigint null
) tablespace pg_default;

We want to be able to test that our functions behave properly. For the sake of this minimal example, I've decided to filter all non adults, but you can imagine any other use case where the functions contain some business logic.

First attempt : Mock Supabase

When unit testing code using third parties that I don't have control over, my first reflex is to try and mock it.

It stopped being fun really quickly. The Supabase-Kt library is making use of a lot of inline function and I ended up having to mock more and more parts of the library and never managed to get a functional tests.

The short version is that because they are as the name indicates, inlined, inline functions cannot be mocked in Kotlin. So that was the end of that experiment

The MainKtTestMock file of my example repository reflects that attempt.

class MainKtTestMock {

    private lateinit var supabaseClient : SupabaseClient

    @BeforeTest
    fun setUp() {

        supabaseClient = mockk<SupabaseClient>()
        val postgrest = mockk<Postgrest>()
        val postgrestBuilder = mockk<PostgrestBuilder>()
        val postgrestResult = PostgrestResult(body = null, headers = Headers.Empty)

        every { supabaseClient.postgrest } returns postgrest
        every { postgrest["path"] } returns postgrestBuilder
        coEvery { postgrestBuilder.insert(values = any<List<Path>>()) } returns postgrestResult
    }

    @Test
    fun testSavePerson(){
        val randomPersons = listOf(Person("Jan", 30), Person("Jane", 42))

        runBlocking {
            val result = savePerson(randomPersons, supabaseClient)
            assertEquals(2, result.size)
            assertEquals(randomPersons, result.map { it.toPerson() })
        }
    }
}

Here's the final error I encountered :

java.lang.IllegalStateException: Plugin rest not installed or not of type Postgrest. Consider installing Postgrest within your supabase client builder
    at io.github.jan.supabase.postgrest.PostgrestKt.getPostgrest(Postgrest.kt:172)
    at MainKtTestMock$setUp$1.invoke(MainKtTestMock.kt:34)
    at MainKtTestMock$setUp$1.invoke(MainKtTestMock.kt:34)

Second attempt: Encapsulate the Supabase Client

My second attempt was to get around the problem by encapsulating the problematic client inside a class of mine that I can then control.

It can be as simple as this :

class DatabaseClient(private val client: SupabaseClient){
    suspend fun savePerson(persons: List<Person>): List<ResultPerson> {
        val adults = persons.filter { it.age > 18 }

        return client
            .postgrest["person"]
            .insert(adults)
            .decodeList<ResultPerson>()
    }
}

And my test can then look like this (see MainKtTestSubclass):

class MainKtTestSubclass {

    private lateinit var client : DatabaseClient

    @BeforeTest
    fun setUp() {
        client = mockk<DatabaseClient>()
        coEvery { client.savePerson(any<List<Person>>()) } returns listOf(ResultPerson(2, "name_2", 2, "timestamp_2"))
    }

    @Test
    fun testSavePerson(){
        val fakePersons = listOf(Person("name_1", 1), Person("name_2", 2))

        runBlocking {
            val result = client.savePerson(fakePersons)
            assertEquals(2, result.size)
        }
    }
}

My main issue now is that because I have to indicate every single time what my output should be. It also just displaces the problem, because I don't really have any nice and clean way to check that my business logic works as intended, since I'm mocking it.

Third attempt : Ktor mock

The main contributor of the project gave another possible workaround in the GitHub issue I created : mock the internal Ktor engine of the Supabase client.

See the MainKtTestMockEngine :

class MainKtTestMockEngine {

    private val supabaseClient : SupabaseClient = createSupabaseClient("", "",) {
        httpEngine = MockEngine { _ ->
            respond(Json.encodeToString(Person.serializer(), Person("name_1", 16)))
        }
    }

    @Test
    fun testSavePerson(){
        val randomPersons = listOf(Person("Jan", 30), Person("Jane", 42))

        runBlocking {
            val result = savePerson(randomPersons, supabaseClient)
            assertEquals(2, result.size)
            assertEquals(randomPersons, result.map { it.toPerson() })
        }
    }
}

This is actually not a bad idea, it's light and it gets the job done is a clear and readable way. Those tests are also fast to run.

My main issue with this method would be that to test my business logic I'd have to dive into the received requests of the mock engine every time, which is a little cumbersome and prone to lots of maintenance.

I do want to investigate it further though.

Proposed solution : Test Supabase db

Now, one semi obvious solution would be to fire up a test database in supabase itself, and test there!

That'd work. I even do it to test my release deployments!

It has some obvious downsides though:

We couldn't be further away from unit tests, since we're testing on the cloud
Tests run slower, require internet, and also require a setup database. Cleanup can also be a mess
I'd be terrified to run that against the wrong database
It uses my bandwidth and projects, that either are limited, or I have to pay for!

Final solution : Docker Compose and Test Containers

I had one last idea, and that's the one I've decided to stick with for now. It leverages the fact that at its core, Supabase is built on a lot of Open-Source. And when we're using the Supabase client, we're essentially interacting with a glorified PostgreSQL / postgrest combo!

I've decided to create a Docker Compose setup that would mimick the actual Supabase production setup and connect to this instead.

A few things had to be taken into account for this to work :

I had to redirect all my postgrest calls to /rest/v1, which is the path that Supabase expects. So a GET on /persons should actually be on /rest/v1/persons.
postgrest uses JSON Web Tokens for authentication, so we have to set that up as part of the test class.

One last thing to note is that I would have to do MORE work in case I start using any of the other services of Supabase (say auth for example).

The Docker Compose setup looks like this :

version: '3'

services:
  ################
  # postgrest-db #
  ################
  postgrest-db:
    image: postgres:16-alpine
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
      - DB_SCHEMA=${DB_SCHEMA}
    volumes:
      - "./initdb:/docker-entrypoint-initdb.d"
    networks:
      - postgrest-backend
    restart: always

  #############
  # postgrest #
  #############
  postgrest:
    image: postgrest/postgrest:latest
    ports:
      - "3000:3000"
    environment:
      - PGRST_DB_URI=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgrest-db:5432/${POSTGRES_DB}
      - PGRST_DB_SCHEMA=${DB_SCHEMA}
      - PGRST_DB_ANON_ROLE=${DB_ANON_ROLE}
      - PGRST_JWT_SECRET=${PGRST_JWT_SECRET}
    networks:
      - postgrest-backend
    restart: always

  #############
  # Nginx #
  #############
  nginx:
    image: nginx:alpine
    restart: always
    tty: true
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf
    ports:
      - "80:80"
      - "443:443"
    networks:
      - postgrest-backend

networks:
  postgrest-backend:
    driver: bridge

Note that a few auxiliary files are needed for this to work. You can find everything in the test/resources folder of the example GitHub repository.

There is :

A short nginx.conf file.
An SQL file to setup the database (note that in my actual repo, this guy already exists since I need it to setup production :)).
a .env file to list all my environment variables

Once that is done, it is already possible to run $ docker-compose up -d and to run your application against localhost like if you were interacting with the real Supabase. (don't forget to call $docker-compose down --remove-orphans -v to kill and delete all containers once you're done).

To make the magic complete, we're gonna use the power of TestContainers to run this as unit/integration tests. My final MainKtTestTestContainers test class looks like this :

import io.github.jan.supabase.SupabaseClient
import io.github.jan.supabase.createSupabaseClient
import io.github.jan.supabase.postgrest.Postgrest
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.Test
import org.testcontainers.containers.ComposeContainer
import org.testcontainers.junit.jupiter.Container
import org.testcontainers.junit.jupiter.Testcontainers
import java.io.File

@Testcontainers
class MainKtTestTestContainers {

    // The jwt token is calculated manually (https://jwt.io/) based on the private key in the docker-compose.yml file, and a payload of {"role":"postgres"} to match the user in the database
    private val jwtToken = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoicG9zdGdyZXMifQ.88jCdmcEuy2McbdwKPmuazNRD-dyD65WYeKIONDXlxg"

    private lateinit var supabaseClient: SupabaseClient

    @Container
    var environment: ComposeContainer =
        ComposeContainer(File("src/test/resources/docker-compose.yml"))
            .withExposedService("postgrest-db", 5432)
            .withExposedService("postgrest", 3000)
            .withExposedService("nginx", 80)

    @BeforeEach
    fun setUp() {
        val fakeSupabaseUrl = environment.getServiceHost("nginx", 80) +
                ":" + environment.getServicePort("nginx", 80)

        supabaseClient = createSupabaseClient(
            supabaseUrl = "http://$fakeSupabaseUrl",
            supabaseKey = jwtToken
        ) {
            install(Postgrest)
        }
    }

    @Test
    fun testEmptyPersonTable(){
        runBlocking {
            val result = getPerson(supabaseClient)
            assertEquals(0, result.size)
        }
    }

    @Test
    fun testSavePersonAndRetrieve(){
        val randomPersons = listOf(Person("Jan", 30), Person("Jane", 42))

        runBlocking {
            val result = savePerson(randomPersons, supabaseClient)
            assertEquals(2, result.size)
            assertEquals(randomPersons, result.map { it.toPerson() })

            val fetchResult = getPerson(supabaseClient)
            assertEquals(2, fetchResult.size)
            assertEquals(randomPersons, fetchResult.map { it.toPerson() })
        }
    }
}

All of the magic happens at the beginning, to setup a fake Supabase URL and connect to it. Once that is done, we can write our tests as easily as ever, since we're actually interacting with an actual light Supabase clone! (For reference, the tests take about 2 seconds to run on my machine)

A word of conclusion

It took me a little while to get all these tests running, but I'm very happy about the final result. It might not be the best solution for production grade apps, but the trade off of running test containers for my side project definitely makes up for the fact that I literally have no boilerplate to run and can avoid using mocks.

I'll check in the future how much I can extend the docker compose image as I get to use more Supabase services. Maybe it would be nice of Supabase to offer that image themselves so we can test easily and avoid using the cloud where not necessary :).

Building a feature platform guided by your ethos

Julien Lengrand-Lambert — Wed, 04 Oct 2023 08:06:16 +0000

In this blog, we discuss our decision process to build a data platform that is powerful but yet stays guided by our ethos.

My wife and I are quite different in our shopping habits. She likes analysing the market trends and she’s good at deciding what she likes. She’s even better at deciding whether to buy or not — disclaimer: usually she does buy it, and then our entrance at home looks like a package pick-up point (not true, but also not entirely not-true).

I, however, have a different poison. When I am looking for something, say a new set of headphones, I analyse and analyse again, and at some point I conclude that a certain product is probably the best fit for what I am looking for. Then, I check, yet again, the distance for the second choice and after that I might end up going back to the fundamental question of whether I actually needed it in the first place.

I am quite frugal, so if I am not very enthusiastic about the top choice I will probably end up discarding it (and bloating those non-conversion metrics for the A/B test behind the ecommerce site that leaves the team scratching their head about what is wrong with their website).

In my defence I will say that if I am sure I need something and there is a clear market winner, I buy immediately and I do not think about it anymore.

The point is, choosing what to buy and when to buy it, is quite transcendental. Choosing your tech stack is a somehow similar problem. You need to understand and reflect very well on a number of things:

Whether you actually need it, or is a fun exciting but limited-value exercise (hello ChatGPT demos 🙊)
How do you sweep and track the market and assess which tool or framework to adopt?
Should you build or buy?
How fast do you need it and whether you are sacrificing something instead?

Without thorough consideration of the above, a leadership team (often proud of their choice and/or invested in another way) may spiral the team downwards in terms of productivity and motivation. That’s why it is important to give it the right amount of love, and eventually make a well-informed choice together with the technical experts.

It is remarkably difficult if you end up in a situation where there’s a clear need for something better than what you have now but there is no industry standard or a clear choice. That’s where my shopper-persona would collapse, and a Feature Store, or should I say Platform (will come to that later), has been a primary example of this sort of conundrum. At Adyen we have gone through this exercise a number of times and we have learned, sometimes the hard way, how to go about these choices. It really boils down to two things that we embrace in our ethos: iterating and control.

Our first attempt to build a Feature Store

Let me use an example of a use case for a feature store at Adyen. Every payment that goes through Adyen — and we do a lot of those (860 billions USD in 2022) — undertakes a journey where a number of decisions are made through an inference service fueled by a machine learning model that we have trained and deployed. We have a few instances of those services with different purposes: our risk system (is the transaction fraud or legit), our authentication service (should we authenticate the user, and if so in which way), our routing algorithm, our transaction optimiser, our retry logic and others.

We are talking about a service that can take several thousands of requests per second per model and respond in less than 100 ms. The final goal of all these models is to land as many good transactions as possible in the most efficient way, without ending up in a chargeback or a retry or higher costs.

Now, these models need features, of course. They need features both at training time and at scoring time.

Let’s zoom in on the risk system. The service was initially built through rules across three different data sources:

Block/allow look-up-tables, powered by PostgreSQL.
Velocity database, powered by PostgreSQL, and able to provide information such as how many times has this card been used in this last minute.
Our “shopper” database, also powered by PostgreSQL.

The “shopper” database deserves its own paragraph. We have used, and maybe even abused, PostgreSQL in a very beautiful way: to identify shoppers in real time. This system in the end provides an elegant and simple graph algorithm by identifying communities of attributes (cards, emails, …) that relate to the same person. It does that very efficiently and very fastly, but it has its own complications around flexibility and scalability. My colleague Burak wrote a great article about it.

When introducing a new approach based on a supervised classifier, we thought: hey, let’s include features about the merchant, that’ll boost AUCPR. Small note, a merchant in the fintech jargon is a seller or a company, take for example, Uber or Spotify. In this sense then, we’d include, as a feature, datapoints like the size of the merchant, the country of the merchant, for how long have we been processing for this merchant, the authorisation rate of that merchant across different sliding windows and so on.

However, many of these example features are slow-moving, medium cardinality, high data volume, features. And PostgreSQL wasn’t gonna cut it in terms of crunching all that volume. So we added a new database called “Feature Store” (read that as Dr. Evil with his famous quote hands).

We took advantage of the fact that we were sending all our transaction events to our Big Data Platform via Kafka. We collect all this info in the form of Hive tables and then we use Spark to crunch all this information, all beautifully orchestrated through Airflow. We had all this information and tooling available in our Big Data Platform because that’s where we train our models. We just need to use an abstraction layer to define the features (we chose Feast) and then deploy on another posgresql instance in the real-time flow (we love postgresql, in case you haven’t figured this out yet). The ML artifacts are deployed to our real-time platform through our wrapper around MLflow, called Alfred, that allows us to stage their rollout into ghost, test, canary live settings and default live modes.

The final picture to our first crack at a “Feature Store” looks like this.

And happy we were ever after, thinking that we had a “Feature Store”. And by “ever after” I mean a couple of months.

The build-vs-buy tradeoff

Hidden in all this there’s something that might have passed unnoticed, the redux of a very important lesson that we have learned through the years: our build-vs-buy trade-off.

There are great vendors that promise and deliver a seamless turn-key experience: it’s fast and it just “works”. It’s a very amenable choice, and I can only show my honest respect for these startups: they are like rain in the middle of a drought for a lot of companies that want to instantly get in the gig of Feature Stores, MLOps, Experimentation, Data Governance and any other sweet problem to solve. And they do a good job at it.

At Adyen, we like to stay in control and understand what happens under the hood. We also have a very solid principle: we won’t use a vendor for anything touching our core business. In this case, processing a payment is indeed core business and we don’t want to introduce a dependency on a third-party. That has two important implications that are worth calling out:

First, we do use vendors, but we are critical about which part of the system they impact. On the one hand we buy our laptops from a vendor — it wouldn’t be optimal if we had a team building laptops. On the other hand, because we are set in building for the long term, we believe in controlling all our supply-chain (take SpaceX, procuring their own screws even).

Second. If we don’t use vendors, do we then build in-house? We have made that choice in the past, and now in perspective, it was a mistake. Picture a top-performing engineer, machete in their teeth, mumbling a classical “hold my beer” and then proceeding to build from scratch something that already exists because “it will only take me a week to do it better”.

We have been there and after the first month I can guarantee the fun is over. Looking ahead at the feature parity roadmap sinks you in despair, the operational debt and preventable bugs itch you more than usual, and you end up going to bed every night thinking “why did we do this”.

So what’s the answer?

Well, open source.

We use open source as much as possible to build our infra and rails, and then we build on top of it our core business. We also contribute back by merging PRs and adding new features that we found useful. At the end of the day, the internet runs on open source.

Iterating on the Feature Store

I already gave away one bit of our ethos: strong control of our dependencies, fueled by long-term thinking. A second big trait of our way of thinking is the iteration culture.

At some point when building software, and even hardware systems, someone figured out that working in waterfall contracts doesn’t really help and that instead working in an agile way gets you further and faster, and it is also more fun. The point of agile is not to adopt scrum. The point of agile is to embrace that the MVP is minimum (and therefore rusty and barely presentable) and also viable (it works, it’s not a WIP commit), and from there onwards, you have to iterate and quickly.

We took a cold look at what we had built that we proudly called “feature store”, tried to remove any emotional attachments to it, and ended up concluding that it wasn’t actually great. We also conclude that we might want to do some soul-searching and write a requirement list about what we want the whole thing to do.

We ended up with a letter to Santa with everything we wanted. At least from there we could make a conscious choice about what we will not get given the cost and possibilities.

Feature Parity: the features and values on the training and scoring flows must be identical.
Retrieval latency: we need the inference service to work under 100 ms, so there’s that.
Recency: the features should not be old, we should be able to refresh them very fast.
Cardinality: we want to be able to store billions of features.
Distributed: we need instances of the feature store around the world, because we process globally.
Storage / Scalability: our transaction volume grows quite a lot every year, and we build for 20x.
Availability / Uptime: we need the system to be there 99.99999% of the time.
Self-service: ideally we want data scientists to go about themselves when prototyping and deploying new features
Complex calculation: some of these features can be complex to compute, that should be alright.
Feature diversity: it’d be great if we didn’t have to maintain three different databases and we just had one endpoint with all sorts of data inside.

After seeing that list we thought “wow, it’s a long list” but we also figured that there was an underlying difference between a pure storage place and a place where things are computed. And that’s also where we read Chip Huygen’s fantastic article on Feature Platforms, and it was one of those click/a-ha moments, when you confidently say out loud “we were building a feature platform, not a feature store” and you can hear the non-existent triumphant music behind you.

The main difference lies in facilitating the computing of features, apart from the storing and serving which is indeed captured under the definition of the feature store.

The new blueprint

Based on this we also saw the need to have a system that spans across two different platforms, our real-time platform (where payments, KYC, payouts, refunds and financial interactions with the world happen) and our big data platform (where we crunch the data). That’s not a surprise given that you have a need in two very different flows: your inference flow in real-time and your training flow off-line.

We needed an abstraction layer that would glue both systems and we chose to keep on using Feast, the open source package to allow data scientists and engineers to define features uniquely and ensure consistency across the two environments. We also evaluated LinkedIn’s Feather, but we deemed it too opinionated and opted for the openness of Feast.

The general idea behind the synching happening on the two environments is that some features will be computed on the real time flow and stored on hot storage and sync back to the cold storage (big data platform) while the slow-moving features will be computed on the big data platform, stored there (cold storage) and synched to the hot storage for inference.

While the batch computation engine and storage were already there, Spark and Hive/Spark/Delta, we still had a few choices to make regarding the on-line flow. For stream computing we are inclined to use Apache Flink — but we can define it later. For the storage layer we hit a dilemma across a few contenders: Redis, Cassandra, Cockroach and sweet old posgresql.

Here is where I will circle back to where I started: you need to make good decisions and that probably means that you need to involve technical experts. Even there, you also want to be able to tap into the wider organisation to make sure that you are not too biassed or forgetting anything. That’s why we have a TechRadar procedure where engineers can share ideas for technology contenders, spar, benchmark and eventually decide on which to adopt in a “makes sense” fashion.

We decided for Cassandra. Redis’ in-memory storage makes it quite expensive at the cardinality we are looking for, Cockroach is really keen on read-write consistency at the expense of speed and posgresql, well, didn’t cut it for our needs.

We still are evaluating choices for on-line computing engines (as said, Flink looks good) and feature monitoring, where it might be that we just use the monitoring stack available, largely consisting of Prometheus, Elastic and Grafana.

That’s an honest look at where we are today. We are making an informed choice and not shooting from the hip. We have determined what we need, we have also determined what is important to us and what we are willing to pay for it. We have analysed the market and open-source offering and are back to our beloved execution mode.

Even if there’s no clear and obvious choice, my shopper persona is still happily going through this procurement journey and enjoying the benefits of learning, discovering the possibilities and deciding. Because if we don’t get it right at first, we will build, fail, learn and iterate.

Thoughts on the Climate crisis and being a Developer (Advocate)

Julien Lengrand-Lambert — Tue, 05 Sep 2023 17:10:08 +0000

This blog is not meant to judge anyone, nor to make anyone feel guilty in any way. Apologies if it comes out the wrong way.

What I mean to share here is a personal journey we've embarked on as a family a couple years back. I do realise how privileged we are to be able to have such a position. Oh and by the way I'm always looking for ways to do better, so please do voice your opinion if you have good ideas.

It's quite clear by now that we humans have a strong impact on the Earth's climate. And it's also quite clear things aren't changing as fast as they need to to make a meaningful difference globally.

As a new Developer Advocate (I've only been in the field for two years officially), my impact on the world around me is something I keep in the back of my head pretty much every day. I want to go with you through some of these thoughts today.

Advocating through individual actions?

As an individual, there are a few directions you can go to reduce your impact on climate:

Individual actions : That's the easiest thing to do, but also what has the least amount of impact. Reducing your footprint might feel good, but it doesn't really help in the grand scheme of things.
Group actions : Voting, supporting political groups, protesting, boycotting, ... Basically helping change the system any way you can as a part of society.

Thing is, part of my job as a Developer Advocate is to be public. Of course, I'm not a Kylian Mbappé, or a Rihanna, ... I'm pretty much a nobody. But I still have a voice in certain niches. Through my job I am spending some of my time in online communities. I like to believe that my individual actions (and those of my family) can ripple to others because I have a few means of promoting them around me.

In this blog, I will focus on the individual actions that I do to reduce my impact on the climate. I'll also limit myself to the ones that have a relationship with my job as Developer Advocate. I won't go into investing or producing food for example. I have enough other blogs on these topics 😊.

Traveling, conferences and the airport's life

Most of the Developer Advocates I meet are public folks. They travel A LOT. If you want to maximize your presence as a Developer Advocate, you want to participate in as many events as possible; which typically means flying. The funny thing is, often the public figures are the same from one conference to the next; meaning that as a speaker 60% of the folks you meet are the same in Germany as they were in UK the week before.

I can't help but think how destructive we are as a group living this completely unsustainable way of living. "it's part of the job". When I see " ATL ✈️ CDG ✈️ TYO✈️ WLG ✈️ AMS ✈️ ATL" on my favourite social media, the first thing I think is that this flight path alone would place me in the top 1% of flyers worldwide.

I've managed to not board a plane since 2019. Not for holidays, not for work. If I go to a conference, I'll take the train to go there. If I can't, well then I'll either send someone who lives closer, or simply not go. And it's not even a sacrifice, actually. I love the train. I can work in it, read books, no security gates to go through, or rough passport control, not dumb dumb 2 hours before rule, no shoes removal involved. The only thing I may miss is the mini bar, but that's a me thing 🍾.

I'm not saying I'll never fly. That'd be hypocritical. For example I dream of visiting Korea. But I want to reduce it as much as I can, for as long as I can because it's by far the easiest way to cut my emissions. Flying accounted for 40% of my yearly footprint until 2019. And I wasn't a Developer Advocate yet back then. This is something I made clear when joining my current employer, and they agreed. If we have to go to many conferences overseas, we'll hire a developer advocate over there to do the work.

And to be fair, it's not like our company suffers from this. Instead of travelling, we're releasing more samples, more content, more experiments, and I sincerely think we're simply serving our customers better. Most of them don't go to conferences. They want stuff that works well, with a good developer experience, that is well documented, and feel welcome with the product. Heck, it's even more inclusive in a way : Not everyone can afford to take days off and travel to go and see you. Online content is available for everyone to see, and it's more scalable by definition. Of course, this is personal experience and your mileage may vary.

One of the things that infuriates me though, is that even to this day flying is the cheapest option anywhere. Taking the train from Amsterdam to Barcelona takes me 2 days and costs 400 euros, while flying there is 30 euros and only 2 hours. Our governments have to do better...

One last thing : Until about 100 years ago, we've never been used to travel that much. People would do one trip to Italy in their life, and be amazed. Travelling has become a complete commodity. But we've also never been as connected as today! Phones, internet, apps, MOOCs, videos, ... There are so many ways to stay in contact with each other. There has to be better options than flying somewhere to do my advocacy!

The picture is not all dark though, I've seen several people doing so much better than me, like Liz Rice biking to go to a conference in April 🤯. There are many other folks as well taking the train to travel.

Is anyone thinking about cycling from London to Amsterdam for #kubecon in April? 🤔🚴‍♀️

— Liz Rice 🐝 (@lizrice) January 10, 2023

About conference planning

Conference planning is a hard problem. Most people go to only a few conferences a year. And they're going to the most appealing ones. If you can get huge names to come to your conference, it's typically a nice way to ensure your tickets are sold. I hear that problem very much, especially post COVID where many conference organisers have piled up huge costs due to unexpected cancellations.

At the same time, I have seen some things at conferences that sometimes disturbed me. Without having to be political, conferences are definitely a place where organisers can send a strong message. The huge majority of the talks I have given that were climate related either happened in the very first slot of the morning before most people wake up, or in the last slot while most of the attendees were already enjoying some beer.

In one instance, I gave a climate related talk at 8h in the morning. That talk was followed by 3 cryptos talks. In another case, I've seen someone give a "reduce your CO2 emissions in the cloud" talk. That person flew all the way from South America to Europe to give a talk to 40 people. As much as I appreciated the chat, I found it hard to believe that one of the main cloud companies in the world didn't have someone in Europe at the ready to give that talk. That hampered the message of the speaker by a lot to me.

Again, I'm not complaining at all about my spot in the conferences here. There are gazillions of speakers better than me 😊. My main point is that the programmation of talks is definitely sending a message to a huge amount of humans at the same time. Conferences aren't typically very clean events in terms of emissions or amount of waste produced. Maybe we can make them matter even more?

The past years, I've been part of redaction commissions for magazines, and in the review committee of a few conferences. Without being completely biased, that's one of the places where I can help sending a message. Having more local speakers, inviting speakers with strong topics on stage, .... (And this goes for much more than climate by the way, diversity, inclusion, ...).

Again, the picture is not completely dark. I've seen great examples of conferences who really outdid themselves to make sure that all the messaging was aligned from one end to the other. There are many great examples, but the best I have seen is DevFestLille. Vegetarian options by default, with reusable cutlery, an adapted programmation, speaker dinner in a local restaurant using locally produced food, sustainable gifts and more... They go much deeper than this, putting accessibility and inclusivity at the core of their values. I admire that conference. Check it out.

Nous clôturons le #DevfestLille avec @fs0c131y qui nous parle d'empreinte numérique ! pic.twitter.com/d4gFq7X0Sj

— Devfest Lille (@DevfestLille) May 26, 2023

Who to work for?

Here's another one of my main struggles: As a Developer Advocate, you're by definition at the forefront of your company and how it presents itself to developers. As such, it is crucial that you're in line with the values they represent.

I am fortunate with the fact that this is something that I really find at Adyen today. Of course, Adyen is here to make profit, but that's obviously OK and I'm definitely here to help. What's important (for me) is that it does it in an ethical, sustainable way. And in this sense it does much more than the large majority of the companies I've been working for. I don't want to blatantly sound like I'm promoting Adyen here, but "building an ethical business and driving sustainable growth for our merchants" is one of the core principles of Adyen since its inception. And it also adds words to action. For example Adyen dedicates 1% of its net revenue to programs that are meant to positively impact ESGs in line with the UN SDG commitment. And it combines global action like this, with local actions on the ground. It also accomodates with my travel choices, as long as it doesn't impact my work. Of course, it's always possible to do more. But it's more than I've ever seen in my career so far.

There's a whole lot of other jobs in DevRel in the industry and I can't say that I associate with all of them. There's been a large amount of advocacy positions open in Web 3, Crypto and NFT related tech lately for example. Historically, those technologies are not particularly known for their clean or efficient computing. Finding positions that line up with my values, is something that I personally struggle with at times,

I do see some VERY interesting positions as well though. For example, I admire Asim Hussain's work and the Green Software Foundation he founded. Most cloud providers are heavily invested in reducing their and their customer's footprints. That's partly why they love your serverless workloads so much 😅.

Another example who brightened my day a while back is Jamund, finding his way into sustainability at AWS :

Super excited to announce that *today* I started with AWS here in London as a Sr. Frontend Engineer with a focus on supply chain and sustainability 🚚⛓📊🌎

— Jamund Ferguson (@xjamundx) September 5, 2022

I have been looking for opportunities in the sustainability domain, but fact of the matter is there doesn't seem to be much yet for Developer Advocates (and DevRel in general).

We're in a great position to challenge the status quo

It may not be true for everyone, but turns out most of us are compensated very well. Of course, I'm not saying we're rich. There's still rent to pay, tuition for the kids, supporting folks around you, ... Especially given how shaky the industry has been the past year. I'm not saying any of us has it easy. That being said, looking at the world inequality database humbled me. I am in a favored position to make a difference on the market, simply because I have enough wealth to have all my major needs fulfilled and I have a platform available to me.

I was impressed to see the impact that many Developer Advocates did supporting Ukraine the past year for example. People like Alina are inspiring. As I was saying above, Asim is also a great example for substainability. As shown in the past, we are in a position to help make a difference.

Educating yourself, and listening to others

One of the nice things of being a Developer Advocate is that whether physically or virtually, you get to meet a lot of people. And your job is pretty much to learn new stuff and share it. I like that as part of my profile I get to share some of the things I've learnt about climate and sustainability. I've given talks on the topic, I've raised awareness internally and externally. I've been sharing our journey online.

This is getting completely out of hands, but there isn't a single book in that pile that I'd remove 😊. Looking forward reading them all cover to cover! #books pic.twitter.com/tuofRLwaLq

— Julien Lengrand-Lambert 🥑👋 (@jlengrand) August 16, 2023

Of course, many are doing it much better than me, and I am very well aware that posting photos of potatoes won't fix the climate crisis. But I've met people along the way who gave me energy to do more, try new things, learn new stuff. We animated a panel last years with 3 other folks on sustainability in tech. I've helped run an online conference dedicated to Simpler and Cleaner Tech.

In France we say "les petits ruisseaux font les grandes rivières" ("They say that small streams make big rivers."). My only goal is to be surrounded by many more of those little streams 😊. Not to feel good about myself and not feel guilty. But more as a way to do ever better every day.

But why do this job at all then?

Some of you might ask "but after all those issues, why be a Developer Advocate at all then"? And that'd be a great question. I ask it to myself every day.

And the answer is relatively short as well : because I absolutely love my job to bits, and it took me years to find something that I liked, that someone would pay me for, and that I'd be good at. I do think that we're also something the world needs. Human seek connection, and we have a role to play there. Given all of the positive examples I've given above from all those inspiring folks, I do see a way to do my job in a way that lines up with my internal voice. You all inspire me to do better every day.

A photo of Ikigai (https://www.japan.go.jp/kizuna/_src/7994686/ikigai_japanese_secret_to_a_joyful_life_pic.png?v=1693289036040)

That's it for now. I really hope this blog doesn't come as moralizing; that really wasn't the point, quite the opposite. I'll be at DevRelCon next week, maybe we can chat there (or in the train on the way 😛). Or any other time, you can hit me on Twitter, Mastodon, or by subscribing :).

And if you've recently been laid off, and are still searching for a new gig; I'm so sorry. The industry hasn't been great for us the past year. If that helps I know that Adyen is searching for a Team Lead Internal Developer Advocacy and a friend of mine at DHL is searching for an internal Developer Advocate in the Netherlands to kickstart their advocacy program. Hit me up.

Cheers,

Julien

My KotlinConf experience

Julien Lengrand-Lambert — Mon, 01 May 2023 09:42:07 +0000

The past week, KotlinConf took place in Amsterdam. It was my very first KotlinConf ever, and to be fair I had been excitingly waiting for it for a long time 😊.I'll be trying to share some of my excitement as well as my impressions here. This post is obviously biased, for official news I recommend you directly go to the source!

The community

The first and main reason for me to go to the conference was to meet all the community again, as always. I've had a glimpse of it a couple months back at the GDE summit, but even more people were there this time 😊.

Many known faces of course, from Louis to James, Duncan, Trisha or Amanda, who was also speaker at the very first conference I spoke at (in Belarus, how times change...) but also so many people I know online for a long time and who I finally got to meet. Isa, a content creator from Spain, Sebastian, which content I follow not only because of the Kotlin magic but also to learn more about the craft of content creation itself. And I finally got to meet Margaryta, who introduced me to the GDE program and came with the whole family (seriously though, more kids at conferences!)

Meeting up with Isa, we've been talking to each other on Twitter for a long time ^^

Somehow, I still didn't manage to Josh and Svetlana ^^. That'll be for another time.

And of course, there's all the new folks. Most notably Salih, we nerded about developer advocacy and leadership.

The new stuff

The Keynote announcements were quite exciting to be honest. From the announcement of Kotlin Notebooks (been wishing for this for a long time, try them out!), to obviously the K2 compiler and much of the goodies that come with it. The notebooks announcement (as well as some of the ML/AI related talks at the conference) also hint at Kotlin becoming a potential alternative for Python for ML engineers out there in the future. Why not?

I care a lot about the language features but even more about the ecosystem, and seeing more companies joining the Kotlin Foundation was the best news of the day to me. It is a great step for a long living healthy language and I'm sure it'll help drive adoption further.

The venue

I really liked all of the sponsors that were present at the event. The venue was also dope for me, with a large Jetbrains booth in the middle stuffed with Product and Engineering folks. After all, Kotlin is a creation of Jetbrains and having the product smack in the middle of everyone with demos running at all times was super cool. Of course, you know me as a big Jetbrains fan so it was really a nerd fest for me.

The massive Jetbrains booth at the centre of the venue

Sponsors also made a lot of sense, and I liked that they were from various industries and domains. Hardware, Multiplatform, server, small and big companies, they were all represented and that was really cool.

Meeting the guys from KodeinKoders, with whom I've interacted online for a while

Quite a few of the companies who also have a large impact on the community via their contributions made it a strong part of their booth and I liked it a lot. From Xebia (47deg) with Arrow, Kodein or even http4K, it was nice to have strong players present as well and see industry support for the ecosystem. It is obviously a big PR and recruitment move for them, but hey I find it smart and it's a strong win/win situation in my opinion.

The Adyen booth at KotlinConf. Adyen uses Kotlin a lot for the payment terminals

The sessions

I spent most of my time roaming around talking to people, so I won't describe all of the sessions I joined but instead I want to give some items which I really liked or who caught my attention particularly.

First, I wanna start with the one session which was an absolute blast for me : Video Game Hacking using Kotlin/Native by Ignat Beresnev. It was fun, entertaining, I've learnt stuff, delivery was amazing and the topic was great as well. Surprisingly, it packed a whole lot of punch for a lightning talk. Honestly though, what's more fun that learning about memory hacking and seeing cars appear straight in GTA, all of that in Kotlin. The presentation is full of well timed jokes, and honestly it grew me as a speaker. I can't recommend it enough.

Then, there was the crash course on compilers by my colleague Amanda. Damn, it's impressive to see deep dive talks from people who are really passionate about their craft. Imposter syndrome hit hard there, and the room was absolutely packed! I have to admit that the topic was quite far from my usual dabblings and I'm still unsure what I could build a compiler plugin for ^^, but I've definitely learnt a lot about how Kotlin works under the hood!

Crash course on compilers, in a dope room, fully packed

Compose, compose and compose

You probably know it, I'm a backend guy. I make do with frontend if I have to, mostly using Web Components. I tried Compose back in the past, mostly for web but I really never got excited about it. I mean, I really do like the components idea and the language is a blast but maybe it was too early. Things weren't documented as much as I wanted to, reloading was slow, tooling wasn't quite there yet, ... I really wasn't reaching the experience that I wanted.

I have to say however that they did manage to hype me up for it again during the conference.

First, Sebastian Aigner adnd Nikita Lipsky announcing Compose Multiplatform on iOS with a banging demo and a flawless (and fun) distribution.

Then John O'Reilly and Martin Bonnin building a Kotlin Multiplatform conference app in 40min live on stage. I mean, of course those speakers rock, they're seasoned and it's clear they're having a blast on stage. But it's also hard not to get seduced seeing so much code reuse between the front and the backend of an app and seeing it run both on android and iOS in minutes.

I also went to see Victor Kropp talk about how they used compose to build the Jetbrains toolbox (use it if you don't yet, it's great!). The toolbox is one application that I quite admire, it's lean and mean while still getting the job done. It's always there for you and damn, it's pretty. So when I learnt it was built using Jetpack Compose for Desktop, my interest for compose obviously grew even more.

Finally, adding the Compose for web demos (using WASM) constantly running on the Jetbrains booth, it's hard not to get a little excited about the future for Kotlin.

I've been into the industry for a little while and one learns to be pessimistic and cautious over time, but has the time finally coming where I can use one language (that I love, sorry Javascript) across all my stacks?

How much compose is too much though?

One thing however bothered me a little bit during the conference, is the amount of Kotlin Multiplatform / Android relative to the server content that was present there. In terms of content, sessions, but also sponsors and visitors. I might be biased really, because that's something that I seem to see in the ecosystem already. Let's have a quick look at it.

The numbers I'm about to give are not precise, because the frontier between Multiplatform and backend is not always crystal clear, but if we look at the ~80 talks present at the conference, I counted about 20 compose related talked, and 30 backend related ones. The rest is either compiler, platform, community, ... related.

It's honestly better than I was expecting, much likely because I used the conference as an opportunity to learn more about Compose.

Out of those 30 backend talks though, more than 12 of those are coroutines related one way or another. That's massive and there's clearly something to see here ^^.

If we look at the Kotlin Foundation partners, the Kotlin partners are Jetbrains, Google, and the newly announced ones are Gradle, Shopify and Touchlab.

Of course, all of those are invested in backend as well with Kotlin being the default for backend at Google those days for example. But even keeping Jetbrains apart (even though it obviously is MASSIVELY invested in KMM), touchlabs and shopify are heavily weighting in the KMM ecosystem. And Google is obviously behind most of the Jetpack Compose and Android technologies. That leaves us with Gradle, which arguably is mostly involved in tooling.

It might be too black and white statement, but as far as I see today most of the partners are more influential in the KMM part of the ecosystem.

Obviously, as I mentioned this is a bet that I'd love to see succeed. The more successful KMM is, the better the whole ecosystem will become. At the same time, I can't stop thinking that there is SO MUCH backend Java going on out there, it feels a little like a missed opportunity. Java is also has been getting a fresh look lately with the faster release cadence and the accrued amount of DevRel going on around it. It looks to me like backend folks aren't being seduced by the language as much as they could be.

In any case, KotlinConf was an amazing experience for me, it was really great to be around so many folks in the ecosystem and seeing how vibrant it becomes over time. And a big thanks to Jetbrains for all of the hard work. Definitely count me in for next time :).

See ya!

Julien

Replacing Postman with the Jetbrains HTTP Client

Julien Lengrand-Lambert — Thu, 02 Feb 2023 10:17:10 +0000

TL;DR : I've created the first version of an openapi-generator for the jetbrains HTTP Client, and together with the CLI runner it allows you to play against APIs without ever going out of your terminal and it can even run in your CI/CD pipeline. See repository here

I work a lot with APIs, whether for an app I develop myself, or to interact with others. Most people I know tend to use Postman for this. But I don't quite like the product any more, and it forces me to move out of my IntelliJ environment which really kills my productivity.

With a combination of the Jetbrains HTTP Client and OpenAPI generators we can do much better, in a semi automated way and even reuse our code on our CI/CD. Let's dive into how!

A quick introduction to the Jetbrains HTTP Client

A lot of people don't know about the Jetbrains HTTP Client, which is a Jetbrains proprietary text file format that allows you to run API requests easily.

One thing I love about it is that it's integrated into the IDE, and it can generate those requests for you. For example, looking at this Spring Boot Kotlin Sample, next to each Mapping there is a "Open in HTTP Client" option in the gutter.

The "Open in HTTP Client" option in the gutter of our IDE

When clicking it, it will generate a scratch file for this request :

The generated request

You can do many things with those requests, like setting Content-Type, sending body payload and more butI'll direct you to the documentation for more.

The one thing that I want to show you here is that it defines variables for you that you can fill in using a separate configuration file. Pick an environment, and fill in a value for the variables you need. You can either create a public file, or a private one for, say, API Keys.

Me creating a Public Environment file

Running the request gets me the response, as expected, and it's even saved in a log file for me.

Running the request with the slug set to ispum returns a valid response

The best thing about all this is that everything is a text file, so I can put my request file with its config in a folder in my repository, pushes it to my repository together with my source files and I suddenly have an easy way to interact with my API.

I hope that by this time you're convinced of the value of the Client 😊. But wait, we can do much more!

Generating full API Clients

Many of you are probably aware of the OpenAPI specification (formerly swagger files). It is basically a standard to describe your API. If you write a spec file for it, you basically write a "contract" for how your API works. The nice thing about it is that you can then use this contract to automagically generate a lot of things for your API! That can be clients, in the language of your choice, mock servers, or even documentation. There is a great project for this, conveniently called OpenAPI Generator.

During the Christmas holidays, I set myself on a quest to create a simple generator for the Jetbrains HTTP Client, which didn't exist yet. As of yesterday, it's released! Let's look through it with an example!

I'm a big DOTA 2 player, and let's imagine I want to start playing with the Open DOTA API. Conveniently, it provides us with an exhaustive OpenAPI specification file. Let's have a quick look

The beginning of the OpenAPI Spec file

As we can see, it describes the location of the API, as well as what kind of HTTP Calls you can make with the associated parameters.

Let's generate a full HTTP Client for it. First, we install the OpenAPI Generator

$ brew install openapi-generator

Then, we generate the client in a new folder

$ openapi-generator generate -i https://api.opendota.com/api -g jetbrains-http-client -o dotaClient

✨That's it!✨

When we open the folder in IntelliJ, we're presented with a few very nice things (Here is the GitHub repo if you want to see it for yourself):

First, a complete documentation with all available endpoints and how they work in the README. Each call is clickable and link to locations in our source code.

A screenshot of the Dota Client README

Then, an Apis folder with all available calls and their related documentation. Here is the generated code for the Heroes endpoints :

## HeroesApi

### GET /heroes
# @name heroesGet
GET http://api.opendota.com/api/heroes

### GET /heroes/{hero_id}/durations
# @name heroesHeroIdDurationsGet
GET http://api.opendota.com/api/heroes/{{hero_id}}/durations

### GET /heroes/{hero_id}/itemPopularity
# @name heroesHeroIdItemPopularityGet
GET http://api.opendota.com/api/heroes/{{hero_id}}/itemPopularity

### GET /heroes/{hero_id}/matches
# @name heroesHeroIdMatchesGet
GET http://api.opendota.com/api/heroes/{{hero_id}}/matches

### GET /heroes/{hero_id}/matchups
# @name heroesHeroIdMatchupsGet
GET http://api.opendota.com/api/heroes/{{hero_id}}/matchups

### GET /heroes/{hero_id}/players
# @name heroesHeroIdPlayersGet
GET http://api.opendota.com/api/heroes/{{hero_id}}/players

Now let's create en environment file and run all those calls for a given hero. Let's use the id of my favourite hero : Crystal Maiden.

{
  "dev": {
    "hero_id": "5"
  }
}

Just like this, I can start running queries against the API, and for example see which players have the most wins with Crytal Maiden :

A screenshot of the results of running the Heroes to Player endpoint

I can also decide to run the Health endpoint and see whether the DOTA or Steam servers are down 😊

Results of running a query against the Health endpoint

Objective reached! I can play against external APIs locally, in source code, without having to use any tool like Postman and without going out of IntelliJ.

Adding tests

Running endpoints is nice, but we also want to be able to make sure expectations are fulfilled! Thankfully, there is a way to do this the the HTTP Request client :

### GET /heroes
# @name heroesGet
GET http://api.opendota.com/api/heroes

> {%
    client.test("Request executed successfully", function() {
        // client.assert(response.status === 200, "Response status is not 200");
        client.assert(response.body.length == 126, "DOTA 2 currently has 123 heroes");
    });
%}

Creating a purposefully failing test

This should fail, as DOTA 2 currently has 123 heroes :

Bingo!

Going even further

I've always been a bit frustrated at Jetbrains for their HTTP Client, because it was completely bound to Intellij. As amazing as it is, I can't spend a lot of time crafting nice custom API calls if I still have to create tests for my CI anyways. At the end of the day, it's double the work for me.

But last month, they announced something that filled me with joy : A CI runner for the HTTP Request files! All of a sudden, all the work we did above became 1000x more useful!

Let's have a look into it :

We download the (preview, for now), tool and unzip it:

$ curl -f -L -o ijhttp.zip "https://jb.gg/ijhttp/latest
$ unzip ijhttp.zip

Finally, we run it against the files we have generated above, with our environment file:

$ ./ijhttp/ijhttp Apis/HeroesApi.http --env-file Apis/http-client.env.json --env dev

Et voilà!

$./ijhttp/ijhttp Apis/HeroesApi.http --env-file Apis/http-client.env.json --env dev  ✔ ╱ 6s  ╱ 10:23:37 
┌─────────────────────────────────────────────────────────────────────────────┐
│ Running IntelliJ HTTP Client with │
├──────────────────────┬──────────────────────────────────────────────────────┤
│ Files │ HeroesApi.http │
├──────────────────────┼──────────────────────────────────────────────────────┤
│ Public Environment │ hero_id = 5 │
├──────────────────────┼──────────────────────────────────────────────────────┤
│ Private Environment │ │
└──────────────────────┴──────────────────────────────────────────────────────┘
Request 'heroesGet' GET http://api.opendota.com/api/heroes
Failed HeroesApi.http#0 Request executed successfully
Request 'heroesHeroIdDurationsGet' GET http://api.opendota.com/api/heroes/5/durations
Request 'heroesHeroIdItemPopularityGet' GET http://api.opendota.com/api/heroes/5/itemPopularity
Request 'heroesHeroIdMatchesGet' GET http://api.opendota.com/api/heroes/5/matches
Request 'heroesHeroIdMatchupsGet' GET http://api.opendota.com/api/heroes/5/matchups
Request 'heroesHeroIdPlayersGet' GET http://api.opendota.com/api/heroes/5/players

6 requests completed, 1 have failed tests
RUN FAILED

We still have our failing test, which will fail our CI. Task failed successfully!

The last step to have a complete setup is to setup a CI action. Let's use GitHub actions for this, fix our failing test, and see what happens:

name: Run API tests based on Jetbrains HTTP Client

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Downloads and runs the Jetbrains HTTP Client CLI
        run: |
          curl -f -L -o ijhttp.zip "https://jb.gg/ijhttp/latest"
          unzip ijhttp.zip
          ./ijhttp/ijhttp Apis/HeroesApi.http --env-file Apis/http-client.env.json --env dev

We checkout, set the Java version to 17 and run the ijhttp package like we did locally.

Success! We now have reliable CI tests using Jetbrains HTTP Client requests!

Going further

There is so much more I could talk about, like the fact that the HTTP Client supports other protocols like GraphQL, handles redirections and more, but I'll keep it to this for now.

Do keep in mind that the Jetbrains HTTP Client generator is still in a very early phase at the moment. For example, I have no support for authentication or most headers at the moment. It won't break anything, but you might have to add things manually to the generated files.

But hey, I do welcome requests and contributions!

Hope you found the read useful. Feel free to ping me on Mastodon or Twitter!

2022 retro, and what's up

Julien Lengrand-Lambert — Wed, 11 Jan 2023 10:07:52 +0000

Just like for many other people, the end of the year is a good time to stop for a second and look back at what one has accomplished. I'm notoriously bad at celebrating my successes, always instead looking at the next thing. So please bear with me for this one time humblebrag post.

So let's dive in, here's a dive into what's happened for me last year, professionally and at a more personal level (which, honestly I'm even more proud of :)).

At work

My debuts into DevRel

2022 was my first professional year in the realm of Developer Relations! I can't believe it took so long for me to find what many would call Ikigai. Something I love, I'm good at, and I can be paid for!

An illustration of Ikigai (https://www.calmsage.com/10-rules-of-ikigai/)

I absolutely love my job, and even though it's clear I can do much much better, the world around me seems to agree with the fact that I'm doing OK at it. I think it's genuinely the first time in my professional career where I feel like I exactly belong in the environment I evolve in, and I make a direct impact.

So there's that!

From entity, to team

Last year ended in a rocky way, with my manager (and mentor) living the company and Adyen deciding to change it's strategy. It took me (and us) a while to figure out what we were going to be doing. After all, Adyen being a closed product, what value can Developer Relations bring ? Turns out, a lot!

A year later, we now officially are our own team, with a clear strategy. I have the chance and honor to lead 3 motivated folks with a different set of strenghts, and it really feels like we can move mountains together. And it's so much fun as well, I'm super proud of them, and what we've done together.

Moving the needle

Talking about this, our team helped create a lot of brand awareness for Adyen. Collectively, we talked to over 6 thousand (!!) people, gave over 30 conference talks and in total reached over 150k folks. It's mad to think about it, especially for a team team where everyone is basically starting up in DevRel 😊.

But even more important, we also helped a lot bring awareness about Developer Experience inside the company, so much so that it's becoming one of the priorities for 2023. It's a great achievement, because it'll improve even further our focus for our merchants, and bring many internal teams together. We're definitely looking forward 2023.

Apoorva and I speaking about Developer Experience at the internal Adyen conference

Meetups, and Conferences

I've been quite present on the conference scene this year, even though it's definitely not one of my main objectives. It brings me crazy amounts of energy to meet folks, share knowledge and to get people excited. I'm proud of the fact that I've started speaking of things that are much closer to my heart as well (sustainability). It's also the first year that people actually started contacting me to speak at conferences, and that honestly feels amazing. This year is also the very first time I was invited on a podcast, to talk sustainability and DevRel, and it felt great!

A photo of me speaking about the Climate Emergency during the DevFest Lille. The Q/A turned into a discussion for almost 30 minutes.

This year, I've also helped reboot the Amsterdam DevRel salon and learnt a lot from the folks there. It's definitely different to organize a Meetup for people who's job it is to speak 😊. Gaining a lot of value from it, and there's demand for it. Our first 2023 session expect almost 50 people!

Oh, and thanks to Floor for helping organize!

A photo of Floor and me at the Halloween edition of the DevRel Salon. We respectively wear an egg, and a lion costume.

Industry recognition

I love spending time create awareness for tools I love. And if you know me, you definitely are aware I'm in love with Kotlin and Gitpod.

Even though it's definitely not a main goal for me, it feels seeing recognition for my efforts. In 2022, I became a Kotlin Google Developer Expert as well as a Gitpod Hero. There's some nice perks to it, but being closer to the community gives me so much energy every day, I'm super grateful about it.

Blogging and Writing

Even though I definitely didn't spend as much time as I wanted to writing this year, I still managed to write a few pieces on this blog and in written magazines. Again, the pleasure I take writing is nothing compared to the joy of hearing that it helped someone or getting feedback.

My face in a magazine, talking about things that are close to my heart (Gitpod)

From speaker, to reviewer

It's been a couple years already that I help the Dutch Java community by being one of the editors of the Dutch Java Magazine. I also absolutely love JFall above all other conferences, and try to always have a speaking slot there.

This year, I had the honor to be part of the selection committee as well for the talks. It was great having the opportunity helping shape this conference that I love so much. That, and helping first time speakers prepare for their talk as well.

And at home

As I was mentioning, even though I'm quite happy with how my work life turned out last year, I'm even prouder of everything we achieved at home.

No compromises

It's been a few years already that we're conscious about out footprint on the planet, and we've decided to drasctically reduce our impact anywhere we could. And one of the major items there is flying. As of 2018, plane travel totalled almost 40% of my CO2 footprint. I decided to stop flying, unless really necessary.

Combining this with being a Developer Advocate is not always simple. After all, a big part of our job is to be at conferences, close to our audience. Well I managed to be present at over 15 conferences this year, without ever boarding a plane. I've also done absolutely no compromise on the topics I wanted to present.

Turns out, you can go far with the train. I've even managed to go all the way to Barcelona 😊. I'm happy to see it's possible, and glad Adyen was nice enough to give me the extra bandwidth necessary for this.

Producing a good chunk of our food

It's been a couple years we try to buy less, and produce some of what we consume at home. It started with many home products (washing powder, dishwasher tablets, detergent, ...) and last year we started the gardening experiment.

The family together on the garden, enjoying our very first red cabbage

We went bigger this year, with a 200m2 garden! It was quite the task, actually much more than we expected. Especially with close to 0 previous experience ^^. Almost 60% of everything we planted died 😅.

But even then, we managed to produce over 160 kg of food this year! This is by far my proudest achievement with the family. We are, and have been quite largely feeding ourselves with our own produce. We made cans, froze a lot, and it's such a joy to create meals that come 100% from stuff you grew yourself! We're definitely planning on doubling down on this next year.

That's what 30Kg of potatoes look like!

Gardening also made us so much more aware of the climate emergency. When you're in the field, you realize how f*ed and unreliable the weather is becoming. I'm glad about it, because it's something we can't ignore anymore and we can bring our kids on the journey. Acting is definitely better than plain despair.

This is what the garden looked like back in July when it stopped raining for almost 90 days in a row. We live in the Netherlands!

So what's with 2023?

It's hard to reflect back without ending with a glimpse at what's coming. This year will be slightly different than last year, but even more exciting.

I'll most likely be less present at conferences, as we will be focusing a lot more on content creation with the rest of the team. This means that I'll get my hands much more dirty and that's great! I actually already managed to do a minor Open-Source contribution to OpenAPI early this year so that's a great start.

So, more content. I also will dive much deeper into Developer Experience, and especially how to measure it. Developer Advocacy definitely ties into Developer Experience, being on the forefront of the product and I want to learn better to convince stakeholders by leveraging numbers. Finally, I will learn the business side of software. This is one of the parts I have been missing in the past years, and it's a gap I want to fill. I want to be generating revenue myself this year, by learning to recognize gaps in product offerings.

So let's get to it! So what are your plans for 2023?! I'm mostly available on Mastodon and Linkedin those days 😊, though you can still find me on Twitter...

A tale of fixing a tiny OpenAPI bug

Julien Lengrand-Lambert — Sat, 31 Dec 2022 15:35:19 +0000

TL;DR : I found and fixed a tiny bug in OpenAPI, learnt about the library, jackson and git in the process.

Yesterday, I was sharing some tips on how to get started with OpenAPI generators. Turns out, within minutes of me starting to use the library, I found a bug that was causing heap space exceptions. This blog is about what I learnt while fixing it 😊.

Facing the bug

The first thing I started doing while testing the application, before even creating my own generator was running a few existing generators for languages I know. I started with Kotlin, and added the debug flags to it to see what kind of objects were generated by the OpenAPI parser.

Within minutes, I was running the Kotlin (of course) generator, with the debugSupportingFiles global property activated.

IntelliJ debug configuration with the debugSupportingFiles global property

After a few seconds, the process crashed with a Java heap space exception:

[main] INFO o.o.codegen.TemplateManager - writing file /Users/julienlengrand-lambert/Developer/openapi-generator/samples/client/petstore/kotlin/http/client/docs/UserApi.md
[main] INFO o.o.codegen.DefaultGenerator - ############ Supporting file info ############
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at com.fasterxml.jackson.core.util.TextBuffer.carr(TextBuffer.java:967)
    at com.fasterxml.jackson.core.util.TextBuffer.expand(TextBuffer.java:928)
    at com.fasterxml.jackson.core.util.TextBuffer.append(TextBuffer.java:682)
    at com.fasterxml.jackson.core.io.SegmentedStringWriter.write(SegmentedStringWriter.java:58)
    at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.writeRaw(WriterBasedJsonGenerator.java:605)
    at com.fasterxml.jackson.core.util.DefaultIndenter.writeIndentation(DefaultIndenter.java:94)
    ...

Not knowing at all what I was doing, I asked for directions on the Slack channel of the project and got suggestions within minutes with ideas of things to try out.

For me, the main question was : Is it only happening on my machine 😬?

Justin was nice enough to file a bug for me, and I started diving into the problem.

Finding the culprit

Turns out, the issue only happens in debug mode, with that specific flag set. It was also happening for all of the generators I tried, and all the specification files. Extending the heap size to humongous values wasn't helping. It had to come from the code somehow.

After a little while, I could track down the line that was creating the crash. It was all happening in the DefaultGenerator (which all generators extend from), and only if the debug flag was set. Makes sense.

if (GlobalSettings.getProperty("debugSupportingFiles") != null) {
    LOGGER.info("############ Supporting file info ############");
    Json.prettyPrint(bundle); // BIG BOUM, BIG BADABOUM 
}

Jason, again, very helpful on Slack, mentioned that the objects generated by the generators can contain infinite loops. Those loops, when trying to transform them into a JSON file would crash the JVM. That's also what was showing in the stack trace of the exception.

It was time to track down when the bug was introduced!

Git bisect to the rescue!

I won't go deep into how git bisect works. You can read about it yourself. The basic idea is that you provide git with a valid commit (before the bug was there) and a faulty commit (with the bug). You'll also need a quick way to see if the bug is present (in my case, running the run configuration and seeing the heap space crash).

Git will iteratively select commits and ask you to tell him whether the bug is there, until you know precisely the ONE COMMIT which introduced the problem.

It basically looked like this :

$ git checkout master # clean state
$ git bisect start
$ git bisect bad HEAD # bug in there
$ git bisect good v5.0.0 # no bug
$ git bisect good # no crash
$ git bisect bad # crash
$ git bisect good
$ git bisect bad
$ git bisect good
$ git bisect bad
$ git bisect good
$ git bisect end
$ git bisect reset # back to work, let's check the bug now

Honestly, I never had to use it in the past in my professional life and I feel blessed to have an extra tool in my box now 😊. The whole thing was sorted in less than 5 minutes 😅.

Understanding the bug

Once we know the commit which introduced the issue, it's easy to track down where exactly the problem is. Look, it's right there :

    /**
     * @return contentTypeToOperation
     * returns a map where the key is the request body content type and the value is the current CodegenOperation
     * this is needed by templates when a different signature is needed for each request body content type
     */
    public Map<String, CodegenOperation> getContentTypeToOperation() {
        LinkedHashMap<String, CodegenOperation> contentTypeToOperation = new LinkedHashMap<>();
        if (bodyParam == null) {
            return null;
        }
        LinkedHashMap<String, CodegenMediaType> content = bodyParam.getContent();
        for (String contentType: content.keySet()) {
            contentTypeToOperation.put(contentType, this);
        }
        return contentTypeToOperation;
    }

The method which introduced the bug creating those heap space exceptions

Once you understand what the method does, it's quite clear why the heap explodes. This method is basically adding to an object, the object itself, but with a key of a certain type of format. No wonder it generates infinite loops when the JSON parser runs through it 😊.

A look inside the object in debug mode at runtime

Now, the actual problem is not that method itself. The real issue is that it's not called anywhere 😅. It shouldn't be run at all. Actually, there's literally 0 direct usages of the method in the whole library (even in the toString method)

IntelliJ indicating the method isn't used anywhere

Obviously, the method is used. But only in the Python template, where it is actually working as expected.

class BaseApi(api_client.Api):
{{#if bodyParam}}
    {{#each getContentTypeToOperation}} // Our method here
    {{> endpoint_args_baseapi_wrapper contentType=@key this=this}}

    {{/each}}
    {{> endpoint_args_baseapi_wrapper contentType="null" this=this}}

{{else}}
    @typing.overload
    def _{{operationId}}_oapg(
    {{> endpoint_args isOverload=true skipDeserialization="False" contentType="null"}}
{{/if}}

Still, placing a breakpoint makes it clear enough that Jackson DOES enter the method when converting the object to JSON.

I tried upgrading Jackson and other shenanigans, until one of my colleagues made it all obvious : "Would your method start with get or set by any chance?".

Finally, it all made sense : Jackson uses reflection to convert the object into JSON representation. Somehow, it takes the class and goes through all of the getters and setters iteratively. The method was assumed to be a getter, hence creating the crash.

Fixing the bug

As usualy with software, once we know what the problem is it's actually quite easy to fix it. And even then, I went through 2 iterations.

My first thought was to add the @JsonIgnore annotation to the method, to tell Jackson to ignore it. It worked.


    @JsonIgnore // The magic line
    public Map<String, CodegenOperation> getContentTypeToOperation() {
        LinkedHashMap<String, CodegenOperation> contentTypeToOperation = new LinkedHashMap<>();
        if (bodyParam == null) {
            return null;
        }
        LinkedHashMap<String, CodegenMediaType> content = bodyParam.getContent();
        for (String contentType: content.keySet()) {
            contentTypeToOperation.put(contentType, this);
        }
        return contentTypeToOperation;
    }

I didn't quite like it though. It put some direct dependency to Jackson in a place there wasn't before, and there was no clear explanation either. The me from the future would have hated me for doing this.

The second attempt was, I think, better and that's the one which got merged : Rename the method to remove the get 😅.

public Map<String, CodegenOperation> contentTypeToOperation() {
        LinkedHashMap<String, CodegenOperation> contentTypeToOperation = new LinkedHashMap<>();
        if (bodyParam == null) {
            return null;
        }
        LinkedHashMap<String, CodegenMediaType> content = bodyParam.getContent();
        for (String contentType: content.keySet()) {
            contentTypeToOperation.put(contentType, this);
        }
        return contentTypeToOperation;
    }

_Method renamed! _

William was nice enough to add a test to test for future regression. I wasn't sure how to do this (how to you test for "no crash of the JVM"). Well, you don't, because the crash isn't there. Here is his commit :

# test debugSupportingFiles
          ./bin/generate-samples.sh ./bin/configs/python.yaml -- --global-property debugSupportingFiles

A word of conclusion

Most of what I've learnt about OpenAPI happened through this bug. The folks on Slack are super helpful and they've been gentle giving directions while expecting me to do the work.

I write a lot of open code, but I don't work much in other people's libraries and that felt super nice. the bugshould be closed in the coming weeks I expect.

Thanks everyone, and happy new year already ! 🎉

Don't hesitate to reach out if you have any questions! I'm mostly available on Mastodon and Linkedin those days 😊, though you can still find me on Twitter...

Getting started with OpenAPI Generators : tips and tricks

Julien Lengrand-Lambert — Fri, 30 Dec 2022 23:57:08 +0000

TL;DR : In this article, I'm sharing some tips and tricks on how to get productive with creating OpenAPI generators

Lately, I've started dabbling more and more with OpenAPI. The OpenAPI specification is a super useful way to describe the API that you're exposed to your users, both internally and externally.

Some of you may also have heard about those under the name of "Swagger files". They're both kinda the same in the common speak, though OpenAPI is an open standard (lead with the OpenAPI Initiative) and Swagger is a set of tools built by a company (SmartBear). Some of those tools are Open-Source, while others have a pro license. I won't go into the details of the specification here, there are plenty of great getting started resources, typically for different technologies. Here's one for Java and Spring.

Generators are the parts who take the OpenAPI model that's being created from a definition file, and generates server / client code and / or documentation. Or anything you want, really. There's many generators available out of the box, so typically what you want is use or adapt one of them. It's not always the case though. For example, there is currently no generator available for the Jetbrains HTTP Client.

That's where my story begins 😊, but that's for another day!

Using the debugging flags

Typically, if you're playing with a generator, you either want to generate a model file (your data structure), an operation file (your logic), or a supporting file (basically anything else, READMEs, docs, ...).

At its core, the idea of OpenAPI is quite simple : It takes a specification file, transforms it into a set of objects in memory, and uses those objects to generates code / files using mustache template files. You can read more about it here.

That's why it's crucial to have a good look into those objects, so that you can find what know where the data you need is located inside your mustache template. Here, debugging flags become vital. There's 3 of them, whether you want to see the data to generate models, operations or supporting files.

You can use respectively debugModels, debugOpenAPI and /or debugSupportingFiles. and you set them by running the desired generate command with the correct flag. For example

$ java -cp modules/openapi-generator-cli/target/openapi-generator-cli.jar org.openapitools.codegen.OpenAPIGenerator generate -g java -o out -i petstore.yaml --global-property debugModels=true

I won't print the entire output here because it's huge, but as part of the output it will basically spit a giant json representation of all the models available inside that specified yaml file. Here's a tiny part of the beginning :

[ {
  "importPath" : "org.openapitools.client.model.Category",
  "model" : {
    "anyOf" : [],
    "oneOf" : [],
    "allOf" : [],
    "name" : "Category",
    "classname" : "Category",
    "title" : "Pet category",
    "description" : "A category for a pet",
    "classVarName" : "category",
    "modelJson" : "{\n \"title\" : \"Pet category\",\n \"type\" : \"object\",\n \"properties\" : {\n \"id\" : {\n \"type\" : \"integer\",\n \"format\" : \"int64\"\n },\n \"name\" : {\n \"type\" : \"string\"\n }\n },\n \"description\" : \"A category for a pet\",\n \"xml\" : {\n \"name\" : \"Category\"\n }\n}",
    "dataType" : "Object",
    "xmlName" : "Category",
    "classFilename" : "Category",
    "unescapedDescription" : "A category for a pet",
    "isAlias" : false,
    "isString" : false,
    "isInteger" : false,
    "isLong" : false,
    "isNumber" : false,
    "isNumeric" : false,
    "isFloat" : false,
    "isDouble" : false,
    "isDate" : false,
    "isDateTime" : false,
    "isDecimal" : false,
    "isShort" : false,
    "isUnboundedInteger" : false,
    "isPrimitiveType" : false,
    "isBoolean" : false,
    "additionalPropertiesIsAnyType" : false,
    .........

Running and Debugging a generator

Whether you want to play with an existing generator or create a new one, the existing documentation tends to offer you to compile and run the generators using the generic ./mvnw clean package followed by ./bin/generate-samples.sh bin/configs/spring-boot.yaml (replace with the file you want to use) command. The first one will compile the source, while the second one will use the created openapi-generator-cli.jar and actually run the code.

For example, the config file spring-boot.yaml :

generatorName: spring
outputDir: samples/server/petstore/springboot
inputSpec: modules/openapi-generator/src/test/resources/2_0/petstore.yaml
templateDir: modules/openapi-generator/src/main/resources/JavaSpring
additionalProperties:
  artifactId: springboot

will run the command

$./modules/openapi-generator-cli/target/openapi-generator-cli.jar org.openapitools.codegen.OpenAPIGenerator generate \
-g spring \
-i modules/openapi-generator/src/test/resources/2_0/petstore.yaml \
-o samples/server/petstore/springboot \
-t modules/openapi-generator/src/main/resources/JavaSpring \
--additional-properties=artifactId=springboot

Useful, but quite cumbersome. On top of this, you typically want to be able to run / debug code as you go directly in your IDE.

Using IntelliJ, you can do it by creating a run configuration as such :

A run configuration in IntelliJ used to run a specific generator

Set the OpenAPIGenerator class to run from the openapi-generator-cli jar file. Pick your generator name, input yaml file and output folder (the same as in the config file described above and set the working directory to be the root of the openapi-generator git project. You're done!

Now you can go in the generator's CodeGen file, (for example JMeterClientCodegen if your generator name is jmeter ) and set breakpoints where you want 😊.

Setting breakpoints in the right locations

As described above already OpenAPI generators take a specification file, transform it into a set of objects in memory, and use those objects to generates code / files using mustache template files.

Typically, generators will take those existing objects and add or modify some of their content to fit the output to be generated. Most generators will extend from DefaultCodeGen. That's where you will find the most useful locations to set breakpoints to see what to change and where.

The most interesting part of this class is located in the generate method. You will also be able to find the lines that are being used to print the debugging flags by searching for the Json.prettyPrint calls. Here's an example for the models (as of now, line 566 of DefaultCodeGen :

if (GlobalSettings.getProperty("debugModels") != null) {
    LOGGER.info("############ Model info ############");
    Json.prettyPrint(allModels);
}

A screenshot of a debug of the model, showing the content of the "allModels" variable

Once you're in there, you can dive into the data and find out what you want to do with it.

Extending the default generator

(Thanks Beppe for the tip!)

Most generators typically add extra bits of necessary information inside their objects, for example inside the bundle for supporting files : bundle.put("distinctPathParameters", distinctPathParameters); . The trick is to find where to do this.

Luckily for us, the smart developers of OpenAPI have created locations to do just that! You'll want to search for the methods called postProcess* in DefaultCodeGen, those are placeholders that are set at the end of the various generatemethods that you can override at your convenience in your custom generator.

Here is what a completely useless generator could do :

@Override
public Map<String, Object> postProcessSupportingFileData(Map<String, Object> bundle) {
    bundle.put("bloggingAt1AM", "isFun");

    return bundle;
}

Which you could then use inside a README.mustache template, for example :

# Important announcement

{{bloggingAt1AM}} // Will print "isFun" once ran

Generating custom lambdas

One of the super powers of mustache is its lambdas, which you can declare in the templates. There is a list available on the mustache website, but the OpenAPI templates define a few more. You can search the code for any class implementing the Mustache.Lambda interface 😊.

A list of many custom lambdas in the OpenAPI source code

You can easily create new lambdas as well by implementing that interface yourself.

Here is a concrete example : Typically, parameters are surrounded with braces when using a generator. For example, for a GET request, with a petId parameter it will come out like this : DELETE http://petstore.swagger.io/v2/pet/{petId}.

That being said, The Jetbrains HTTP Client defines parameter with double braces (!). I need this to have a valid call : DELETE http://petstore.swagger.io/v2/pet/{{petId}}.

Instead of having to play around a lot with the templating, I decided to create a custom lambda for this in my generator. This is how it looks like :

public class JetbrainsHttpClientClientCodegen extends DefaultCodegen implements CodegenConfig {

    @Override
    protected ImmutableMap.Builder<String, Mustache.Lambda> addMustacheLambdas() {
       return super.addMustacheLambdas()
                .put("doubleMustache", new DoubleMustacheLambda());
    }

    public static class DoubleMustacheLambda implements Mustache.Lambda {
        @Override
        public void execute(Template.Fragment fragment, Writer writer) throws IOException {
            String text = fragment.execute();
            writer.write(text
                    .replaceAll("\\{", "{{")
                    .replaceAll("}", "}}")
            );
        }
    }
}

Two things happen here :

I create a DoubleMustacheLambda class that implements Mustache.Lambda that itself implements the execute method. The execute method does nothing else than rewriting some of the generated text.
I override the addMustacheLambdas method, and use it (just like for my data in the previous tip) to insert my custom lambda.

Once that is done, I can use it inside my api.mustache template! For example : {{#lambda.doubleMustache}}{{path}}{{/lambda.doubleMustache}}

A word of conclusion

I hope those tips will help you hit the ground running with OpenAPI generators. Most of the tips directly come from the open Slack channel, the folks there are super useful. The official documentation is nice, but I honestly found it quite sparse, and once you dive in, a lot of the advice comes down to "have a look at the other generators for inspiration". I find it logical, because in 99% of the cases, the generator that you need already exists! If it doesn't though, you're in for some fun.

Hopefully I'll save you a couple hours this way!

Don't hesitate to reach out if you have any questions! I'm mostly available on Mastodon and Linkedin those days 😊, though you can still find me on Twitter..