Create C# Classes from Avro IDL

I was recently involved in a project where we built a .NET service which consumed messages from various Kafka topics, the messages themselves were serialised using Apache Avro.

This is the first time I had come across Avro, and there aren't many tools in the .NET ecosystem around this. In this series, I'll cover how to generate C# classes from Avro schemata, automate the steps, and finally how to integrate it with your project's build.

Avro Schema

Avro schemata are defined as JSON in file with the extension .avsc. An example schema of a message which contains a single string would be:

{
  "type" : "record",
  "name" : "ExampleMessage",
  "namespace" : "example.avro",
  "fields" : [ {
    "name" : "body",
    "type" : "string"
  } ]
}

You can generate C# classes from .avsc files using avrogen, which comes as part of https://www.nuget.org/packages/Apache.Avro.Tools/.

Avro IDL

In our project, the messages were already defined and all we had to do was create the corresponding C# classes to hydrate into.

There was an extra catch for us: the schemata themselves were defined using the higher-level Avro IDL language and there currently isn't a tool to generate C# classes from Avro IDL.

The Avro IDL version of above schema would be:

@namespace("example.avro")
protocol ExampleMessage {
  record ExampleMessage {
    string body;
  }
}

Avro IDL to C# Class

Avro IDL files have the extension .avdl. To turn these files into C# classes we had to:

• Turn AVDL into AVSC using the Java tool avro-tools - this requires Java
• Turn AVSC into a C# class using the avrogen tool mentioned earlier

Steps

Take the example Avro IDL above and save it to a file called ExampleMessage.avdl.

Turn it into an .avsc file with:

java -jar /path/to/avro-tools-1.10.0.jar idl2schemata ExampleMessage.avdl .

This produces ExampleMessage.avsc, which we then turn into a C# class with:

avrogen -s ExampleMessage.avsc .

The final file created is ./example/avro/ExampleMessage.cs:

namespace example.avro
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using Avro;
    using Avro.Specific;

    public partial class ExampleMessage : ISpecificRecord
    {
        public static Schema _SCHEMA = Avro.Schema.Parse("{\"type\":\"record\",\"name\":\"ExampleMessage\",\"namespace\":\"example.avro\",\"fields\":[{\"n" +
                "ame\":\"body\",\"type\":\"string\"}]}");
        private string _body;
        public virtual Schema Schema
        {
            get
            {
                return ExampleMessage._SCHEMA;
            }
        }
        public string body
        {
            get
            {
                return this._body;
            }
            set
            {
                this._body = value;
            }
        }
        public virtual object Get(int fieldPos)
        {
            switch (fieldPos)
            {
            case 0: return this.body;
            default: throw new AvroRuntimeException("Bad index " + fieldPos + " in Get()");
            };
        }
        public virtual void Put(int fieldPos, object fieldValue)
        {
            switch (fieldPos)
            {
            case 0: this.body = (System.String)fieldValue; break;
            default: throw new AvroRuntimeException("Bad index " + fieldPos + " in Put()");
            };
        }
    }
}

Summary

While this works for a single file the experience doesn't really scale when we start dealing with multiple files.

For Java developers, there are plugins which handle this so all developers need to do is add AVDL files to their projects and the tooling generates the Java classes.

Ideally, we should have some equivalent in .NET, which is what I'll be covering in the rest of this series!