DEV Community

Cain Voong
Cain Voong

Posted on

1

Automate AVDL to C# Generation

In the previous post, you saw how to take an Avro IDL file and manually turn that into a C# class.

For a small number of files, this is fine but as soon as you start dealing with a lot of them it'll be something you'll want to automate. I've written a PowerShell script to do this will cover how to use it + what it does.

Prerequisites

Folder Structure

Create a folder called Avro and within that:

  • Add a file called Generate.ps1
  • Add a folder called AVDL and place any .avdl files in there

It should look something like:

.
└── Avro
    ├── AVDL
    │   └── ExampleMessage.avdl
    └── Generate.ps1        
Enter fullscreen mode Exit fullscreen mode

The Generate.ps1 Script

The PowerShell script is what does all the work. The contents can be copied from this gist:

$ErrorActionPreference = "Stop"
$idlPath = "./AVDL"
$rootPath = "./_generated"
$outputRoot = "$rootPath/avro"
$schemaRoot = "$outputRoot/schema"
$classesRoot = "$outputRoot/classes"
# Store tools outside the repo (saves repeated downloads)
$avroToolsUri = "https://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.10.0/avro-tools-1.10.0.jar"
$avroToolsRoot = "$HOME/.avro-tools"
$avroToolsPath = "$avroToolsRoot/avro-tools-1.10.0.jar"
# Where avrogen is installed to (doing it this way works within a docker container)
$avrogenPath = "$HOME/.dotnet/tools/avrogen"
$runBuild = $false
# Do we need to run this?
if (Test-Path -Path $outputRoot)
{
$newestClass = Get-ChildItem -Path $outputRoot -Recurse | Sort-Object LastWriteTime | Select -Last 1
$newestIdl = Get-ChildItem -Path $idlPath -Recurse -Filter "*.avdl" | Sort-Object LastWriteTime | Select -Last 1
if ($newestIdl.LastWriteTime -gt $newestClass.LastWriteTime)
{
Write-Output "AVDL change detected, regenerating..."
Remove-Item -Recurse $outputRoot
$runBuild = $true
}
else
{
Write-Output "No AVDL changes detected"
}
}
else
{
Write-Output "Generating Avro classes..."
$runBuild = $true
}
if ($runBuild)
{
# Create the folder if it doesn't exist
if (-Not (Test-Path -Path $rootPath))
{
New-Item -Path $rootPath -ItemType Directory | Out-Null
}
# Download avro-tools if it can't find it
if (-Not (Test-Path -Path $avroToolsPath))
{
New-Item -Path $avroToolsRoot -ItemType Directory -Force | Out-Null
Write-Output "avro-tools not found, downloading from $avroToolsUri to $avroToolsPath"
Invoke-WebRequest -Uri $avroToolsUri -OutFile $avroToolsPath
}
$inputIdls = Get-ChildItem -Path $idlPath -Recurse -Filter "*.avdl" | Resolve-Path -Relative
foreach ($inputIdl in $inputIdls)
{
$schemaOutputPath = $schemaRoot + (Split-Path -Path $inputIdl).Replace($idlPath, "")
$command = "java -jar $avroToolsPath idl2schemata $inputIdl $schemaOutputPath"
Write-Output "Running: $command"
Invoke-Expression -Command $command
}
$inputSchemas = Get-ChildItem -Path $schemaRoot -Recurse -Filter "*.avsc" | Resolve-Path -Relative
foreach ($inputSchema in $inputSchemas)
{
$command = "$avrogenPath -s $inputSchema $classesRoot"
Write-Output "Running: $command"
Invoke-Expression -Command $command
}
}
view raw Generate.ps1 hosted with ❤ by GitHub

What it does:

  • Downloads avro-tools to $HOME/.avro-tools if it's not there already
  • Scan the contents of the AVDL folder
  • For each .avdl file found, generate the .avsc and from that, the C# class
  • All generated files are written to the Avro/_generated folder

Running

To run the script just do:

pwsh Generate.ps1
Enter fullscreen mode Exit fullscreen mode

The results should end up like:

.
└── Avro
    ├── AVDL
    │   └── ExampleMessage.avdl
    ├── Generate.ps1
    └── _generated
        └── avro
            ├── classes
            │   └── example
            │       └── avro
            │           └── ExampleMessage.cs
            └── schema
                └── ExampleMessage.avsc
Enter fullscreen mode Exit fullscreen mode

The script also does some basic change detection so it'll only generate the files if something has changed within the AVDL folder. It's just a simple check of last modified dates, so it's not perfect (i.e. doesn't work with deletions) but it works for most cases.

Summary

We took the manual steps from the previous post and automated it using PowerShell. In the final post, we'll look at adding this to a project so it gets executed as part of the build process.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)