AsyncAPI Codegen, a code generator from AsyncAPI spec v2 and v3.

This post is about an open-source tool that I’m currently writing here.

AsyncAPI, an initiative to rule them all

What is AsyncAPI?

As a response to the lack of tools to define contracts between asynchronous services, Fran Méndez started the project AsyncAPI as a set of tools heavily inspired by OpenAPI initiative to define communication inside an Event-Driven Architecture (EDA).

Like OpenAPI, you have a specification, in YAML or JSON, that can be used to define the interfaces of asynchronous APIs. It can use any type of protocols and brokers (NATS, Kafka, RabbitMQ, gRPC, etc) but also any format (JSON, Protobuf, etc).

After years of development, it grew, became an important project and joined the Linux Foundation Projects with the goal to be industry standard for defining asynchronous APIs.

An AsyncAPI specification example

Here is what it can look like, from the official documentation:

asyncapi: 3.0.0
info:
  title: Hello world application
  version: '0.1.0'
channels:
  hello:
    address: 'hello'
    messages:
      sayHello:
        payload:
          type: string
          pattern: '^hello .+$'
operations:
  receiveHello:
    action: 'receive'
    channel:
      $ref: '#/channels/hello'

Some elements may look familiar to you, like the components defining schemas or the information part, as they are directly inspired from OpenAPI specifications.

Some are quite different: the first level channels defines the channels used by the application. You can also find a messages definition in the channels that describes the messages that can be sent and/or received on the channel. There is also an operations part that describes all actions that the application is doing.

You can then read this document as an application that is listening on hello channel and is expecting a sayHelloMessage on it. This message should contain a payload with

Messages payload can be displayed in JSON format:

"hello world"

If you are curious of the numerous possiblities, you can read the official reference.

The code, the spec and the maintainability

Of course, It can be really cumbersome to maintain the source code associated to the specification. That’s why there is two ways of doing that:

• Generate the specification from the code
• Generate the code from the specification

You can already see where we are going, as this post is about code generation!

An official tool for code generation… but with Javascript and NPM

For that, the project provides a Javascript tool to generate source code from specification in numerous languages: Python, Java, Markdown, PHP, … and even Go! All you have to do is install the corresponding NPM packages and launch the right command tool.

However, despite the great advantages of the Javascript ecosystem, a lot of developers (including myself) are a bit reluctant to install NPM packages with the really heavy stack that it implies. Especially for Go project where a specific command exist to run Go programs (and other tools) directly from the official Go command: go generate.

To use it, we only have to add some preprocessing instructions at the top of some files:

//go:generate <insert here the shell command>

package mypkg

We could use directly the official tool with go generate but it would require to have the Javascript tools installed and to download the NPM packages.

Ideally, we would need something in Go to run the go run command inside the go generate preprocessing command. It would be the most portable way to generate the code, as you would just have to have the Go stack installed. One language, one stack!

The inspiration: a really popular Go tool, but for OpenAPI

During daytime, and especially work time, I used a great tool to generate code from OpenAPI specification: deepmap/oapi-codegen.

This tool takes an openapi specification and generate all the code needed to make any major Go HTTP server (Gin, Echo, Chi, etc) with the given specification.

On top of that, you can use the //go:generate preprocessing annotation to automatically generate the source code.

//go:generate go run github.com/deepmap/oapi-codegen/cmd/oapi-codegen@latest -package mypkg openapi.yaml > openapi.gen.go
package mypkg

At the time, I craved for this tool on AsyncAPI, and as it was not existant, I decided to wrote it. And a few month later (and few PR also), asyncapi-codegen was available!

AsyncAPI-Codegen, hands on user signup

Code architecture

This schema describe how the code generated by AsyncAPI Codegen will interact between existing application, user and broker:

Here are the different part of the code that will be autogenerated:

• Provided (or custom) code for broker [orange]: you can use one of the provided broker (NATS, Kafka), or provide you own by satisfying BrokerController interface.
• Generated code [yellow]: code generated by asyncapi-codegen.
• Calling the generated code [red]: you need to write a few lines of code to call the generated code from your own code.
• App/User code [blue]: this is the code of your existing application and/or user. As it is conform to AsyncAPI spec, you can use it on only one of the two side against the app/user on another language.

Installing AsyncAPI-Codegen

Well let’s take the example we had in the first part of this post, let’s save it in an asyncapi.yaml and generate the corresponding code.

First, we should install the tool:

# Install the tool
go install github.com/lerenn/asyncapi-codegen/cmd/asyncapi-codegen@latest

Then create the new golang project with the following lines:

# Create a new directory for the project and enter
mkdir $GOPATH/src/asyncapi-example
cd $GOPATH/src/asyncapi-example

# Create the go module files
go mod init

# Create the asyncapi yaml into the repository
touch asyncapi.yaml

# Then edit it to add the YAML content provided earlier

Then we can generate the code:

# Generate the code from the asyncapi file
# One for the application, one for the user and one for the common types
asyncapi-codegen -i ./asyncapi.yaml -p main -g application -o ./app.gen.go
asyncapi-codegen -i ./asyncapi.yaml -p main -g types -o ./types.gen.go
asyncapi-codegen -i ./asyncapi.yaml -p main -g user -o ./user.gen.go
# Note: you can generate all in one file by removing the '-g' argument

# Install dependencies needed by the generated code
go get -u github.com/lerenn/asyncapi-codegen/pkg/extensions

Deep dive into the generated code

Application generated code

Let’s analyse what is present in app.gen.go:

// AppController is the structure that provides publishing capabilities to the
// developer and and connect the broker with the App.
type AppController struct

// NewAppController links the App to the broker.
func NewAppController(/* ... */) (*AppController, error)

// SubscribeToReceiveHelloOperation waits for 'SayHello' messages from 'hello' channel.
func (c *AppController) SubscribeToReceiveHelloOperation(ctx context.Context, fn func(msg SayHelloMessage)) error

// UnsubscribeFromReceiveHelloOperation stops subscription on 'SayHello' messages from 'hello' channel.
func (ac *AppController) UnsubscribeFromReceiveHelloOperation(ctx context.Context)

// Close will clean up any existing resources on the controller.
func (c *AppController) Close(/* ... */)

We can see that we can create a new AppController based on a broker controller, that will allow the application to receive SayHello messages on hello channel.

User generated code

Let’s analyse what is present in user.gen.go:

// UserController is the structure that provides publishing capabilities
// to the developer and and connect the broker with the User.
type UserController struct

// NewUserController links the User to the broker.
func NewUserController(/* ... */) (*UserController, error)

// SendToReceiveHelloOperation will publish a hello world message on the "hello" channel.
func (c *UserController) SendToReceiveHelloOperation(ctx context.Context, msg SayHelloMessage) error

// Close will clean up any existing resources on the controller.
func (c *UserController) Close(ctx context.Context)

Like the application controller, we can see that we can create a new UserController based on a broker controller. It will allow us to send sayHello messages to the application.

Use the generated code

Let’s use the generated code to simulate a real system !

So we will need to have:

• A broker (for the sake of this example, we will use a NATS broker)
• An application emitting user signup events
• A user receiving user signup events

The broker

We will add some packages to use NATS directly in our code and launch a docker container to have a running NATS:

# Get missing code for NATS
go get github.com/lerenn/asyncapi-codegen/pkg/extensions/brokers/nats

# Launch NATS with docker
docker run -d --name nats -p 4222:4222 nats

You can then create a file named main.go with the following code as a placeholder for the application and user code:

package main

import (
 "github.com/lerenn/asyncapi-codegen/pkg/extensions"
 "github.com/lerenn/asyncapi-codegen/pkg/extensions/brokers/nats"
)

func app(brokerController extensions.BrokerController) {
  // Will be filled later
}

func user(brokerController extensions.BrokerController) {
  // Will be filled later
}

func main() {
  // Create a broker controller
  brokerController, err := nats.NewController("nats://localhost:4222")
  if err != nil {
    panic(err)
  }

  // Launch application listening
  app(brokerController)

  // Launch user that will periodically send 'hello world'
  user(brokerController)
}

We can see here that we create a broker controller based on NATS. It could have been any other available broker controller, as they fulfill the extensions.BrokerController that app() and user() need.

At the time these lines are written, there is only NATS Core/Jetstream and Kafka, but you can take inspiration from these ones to implement your own. In fact, the Kafka one is a PR from a contributor that wanted to use this tool for it. Feel free to explore!

Then, there is the launch of the user and the start of the application. We will see now what is needed in each function.

The user

Here is the code you will need in the user:

func user(brokerController extensions.BrokerController) {
  // Create the user controller
  userController, err := NewUserController(brokerController)
  if err != nil {
    panic(err)
  }

  // Publish users signing up events randomly
  for i := 0; ; i++ {
    // Wait a second between sends
    time.Sleep(time.Second)

    // Send message
    userController.SendToReceiveHelloOperation(
      context.Background(),
      SayHelloMessage{
        Payload: fmt.Sprint("hello ", i),
      },
    )
  } 
}

It repeatedly generates incremental sayHello messages with incrementing payload. It waits one second and publishes each event using the userController, created at the beginning with the brokerController.

The application

And now, for the application part:

func app(brokerController extensions.BrokerController) {
  // Create the user controller
  appController, err := NewAppController(brokerController)
  if err != nil {
    panic(err)
   }

  // Create a callback that will listen
  callback := func(_ context.Context, msg SayHelloMessage) {
    log.Println("Received message:", msg.Payload)
  }

  appController.SubscribeToReceiveHelloOperation(context.Background(), callback)
}

Here is what the code do:

1. It creates an app controller.
2. Defines a callback function (fn) to handle sayHello messages, logging the payload.
3. Subscribes to sayHello messages on hello channel, using the app controller.

Execute the code

Now that we have all the code, we can execute it:

# Run the code
go run .

Here is the output you should see:

2023/10/15 14:08:20 Hello 0
2023/10/15 14:08:21 Hello 1
2023/10/15 14:08:22 Hello 2
2023/10/15 14:08:23 Hello 3
2023/10/15 14:08:24 Hello 4

If you use the NATS cli tool, you can also see the messages being sent:

nats sub ">"
[#0] Received on "hello"
{"hello 0"}

[#1] Received on "hello"
{"hello 1"}

[#2] Received on "hello"
{"hello 2"}

[#3] Received on "hello"
{"hello 3"}

[#4] Received on "hello"
{"hello 4"}

You can see that the AsyncAPI specification is respected, as the messages are sent and received with the same format.

More on AsyncAPI Codegen

Extensions

Sending and receiving messages is not the only thing you can do with the generated code. You can also use the extensions to add more features to the generated code.

Here is the list of what can be done, linked to the README.md (and later a link to specific blog posts):

• Request/Response: Send a message on a channel and wait for the answer on another.
• Multiples Brokers: Kafka, NATS, or Custom broker
• Middlewares: manage the messages before sending or after receiving them.
• Context: use context keys set by middlewares or by the user to get info on messages (publishing/reception, channel name, etc).
• Logging: log messages sent and received, but also internal generated code logs.
• Versioning: manage different versions of an AsyncAPI specification and plan migrations.
• Annotations: add annotations to your specification for more features to generated code.

What’s next?

Well there is a lot of things that can be done to improve the tool, and I’m currently working on some of them:

• Add more brokers: RabbitMQ, gRPC, etc
• Add more middlewares: openetelemetry logging/tracing/metrics, etc
• Add more formats: Protobuf, etc

But I’m also open to any contribution, so feel free to open an issue or a PR if you want to add something to the tool!