DEV Community: Cain Voong

Add AVDL Support to a .NET Project

Cain Voong — Sun, 24 Jan 2021 19:56:28 +0000

By now, we have a good grasp on:

Taking Avro IDL files and generating C# classes
Automating the above so we can do it on lots of files with ease

In this final step, I will create a C# project and configure it in a way so developers only need to add/modify avdl files and build.

Create the Project

We'll start from scratch, by creating the project and then adding an Avro folder:

dotnet new classlib --name MyProject
cd MyProject
mkdir Avro

Add the package Confluent.SchemaRegistry.Serdes.Avro:

dotnet add package Confluent.SchemaRegistry.Serdes.Avro --version 1.5.3

Then copy the AVDL folder and Generate.ps1 script from the previous post.

The project should end up looking like:

MyProject
├── Avro
│   ├── AVDL
│   │   └── ExampleMessage.avdl
│   └── Generate.ps1
├── Class1.cs
└── MyProject.csproj

Modify .csproj

Before building the .NET project, we want MSBuild to generate our Avro classes. This can be done as a PreBuild step which is configured within MyProject.csproj.

Inform MSBuild

Open the file then add the following:

<Target Name="PreBuild" BeforeTargets="PreBuildEvent">
  <Exec WorkingDirectory="$(ProjectDir)Avro" Command="pwsh -ExecutionPolicy RemoteSigned -File Generate.ps1" />
  <ItemGroup>
    <Compile Include="$(ProjectDir)Avro/_generated/**/*$(DefaultLanguageSourceExtension)" Exclude="@(Compile)" />
  </ItemGroup>
</Target>

This does a couple of things:

Execute the PowerShell script to generate Avro classes first
Inform MSBuild of the new files, but also ensure not to include them twice if they already exist (see here)

Tweak for Visual Studio

Also, add:

<ItemGroup>
  <EmbeddedResource Include="Avro\AVDL\**\*.avdl" />
</ItemGroup>

This is a little hack to trick Visual Studio into marking the build as out of date if you make any changes in the AVDL folder.

Build

Now, we're ready to build:

dotnet build

Which results in:

Microsoft (R) Build Engine version 16.7.0+7fb82e5b2 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /home/cainux/code/MyProject/MyProject.csproj (in 313 ms).
  Generating Avro classes...
  avro-tools not found, downloading from https://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.10.0/avro-tools-1.10.0.jar to /home/cainux/.avro-tools/avro-tools-1.10.0.jar
  Running: /home/cainux/.dotnet/tools/avrogen -s ./_generated/avro/schema/ExampleMessage.avsc ./_generated/avro/classes
  MyProject -> /home/cainux/code/MyProject/bin/Debug/netstandard2.0/MyProject.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Conclusion

We went from manually generating C# classes from Avro IDL files to making it happen automagically withing a .NET project. Developers only need to put their AVDL files in the right place and run build to generate the classes. Now you can focus on building the rest of your application with one less thing to think about.

If you're managing your code with source control (if not, why on earth not?!) such as Git - remember to ignore the _generated folder and its contents as they are build assets and shouldn't be checked in to source control.

Automate AVDL to C# Generation

Cain Voong — Sun, 24 Jan 2021 19:56:05 +0000

In the previous post, you saw how to take an Avro IDL file and manually turn that into a C# class.

For a small number of files, this is fine but as soon as you start dealing with a lot of them it'll be something you'll want to automate. I've written a PowerShell script to do this will cover how to use it + what it does.

Prerequisites

Folder Structure

Create a folder called Avro and within that:

Add a file called Generate.ps1
Add a folder called AVDL and place any .avdl files in there

It should look something like:

.
└── Avro
    ├── AVDL
    │   └── ExampleMessage.avdl
    └── Generate.ps1

The Generate.ps1 Script

The PowerShell script is what does all the work. The contents can be copied from this gist:

What it does:

Downloads avro-tools to $HOME/.avro-tools if it's not there already
Scan the contents of the AVDL folder
For each .avdl file found, generate the .avsc and from that, the C# class
All generated files are written to the Avro/_generated folder

Running

To run the script just do:

pwsh Generate.ps1

The results should end up like:

.
└── Avro
    ├── AVDL
    │   └── ExampleMessage.avdl
    ├── Generate.ps1
    └── _generated
        └── avro
            ├── classes
            │   └── example
            │       └── avro
            │           └── ExampleMessage.cs
            └── schema
                └── ExampleMessage.avsc

The script also does some basic change detection so it'll only generate the files if something has changed within the AVDL folder. It's just a simple check of last modified dates, so it's not perfect (i.e. doesn't work with deletions) but it works for most cases.

Summary

We took the manual steps from the previous post and automated it using PowerShell. In the final post, we'll look at adding this to a project so it gets executed as part of the build process.

Create C# Classes from Avro IDL

Cain Voong — Sun, 24 Jan 2021 19:53:52 +0000

I was recently involved in a project where we built a .NET service which consumed messages from various Kafka topics, the messages themselves were serialised using Apache Avro.

This is the first time I had come across Avro, and there aren't many tools in the .NET ecosystem around this. In this series, I'll cover how to generate C# classes from Avro schemata, automate the steps, and finally how to integrate it with your project's build.

Avro Schema

Avro schemata are defined as JSON in file with the extension .avsc. An example schema of a message which contains a single string would be:

{
  "type" : "record",
  "name" : "ExampleMessage",
  "namespace" : "example.avro",
  "fields" : [ {
    "name" : "body",
    "type" : "string"
  } ]
}

You can generate C# classes from .avsc files using avrogen, which comes as part of https://www.nuget.org/packages/Apache.Avro.Tools/.

Avro IDL

In our project, the messages were already defined and all we had to do was create the corresponding C# classes to hydrate into.

There was an extra catch for us: the schemata themselves were defined using the higher-level Avro IDL language and there currently isn't a tool to generate C# classes from Avro IDL.

The Avro IDL version of above schema would be:

@namespace("example.avro")
protocol ExampleMessage {
  record ExampleMessage {
    string body;
  }
}

Avro IDL to C# Class

Avro IDL files have the extension .avdl. To turn these files into C# classes we had to:

Turn AVDL into AVSC using the Java tool avro-tools - this requires Java
Turn AVSC into a C# class using the avrogen tool mentioned earlier

Steps

Take the example Avro IDL above and save it to a file called ExampleMessage.avdl.

Turn it into an .avsc file with:

java -jar /path/to/avro-tools-1.10.0.jar idl2schemata ExampleMessage.avdl .

This produces ExampleMessage.avsc, which we then turn into a C# class with:

avrogen -s ExampleMessage.avsc .

The final file created is ./example/avro/ExampleMessage.cs:

namespace example.avro
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using Avro;
    using Avro.Specific;

    public partial class ExampleMessage : ISpecificRecord
    {
        public static Schema _SCHEMA = Avro.Schema.Parse("{\"type\":\"record\",\"name\":\"ExampleMessage\",\"namespace\":\"example.avro\",\"fields\":[{\"n" +
                "ame\":\"body\",\"type\":\"string\"}]}");
        private string _body;
        public virtual Schema Schema
        {
            get
            {
                return ExampleMessage._SCHEMA;
            }
        }
        public string body
        {
            get
            {
                return this._body;
            }
            set
            {
                this._body = value;
            }
        }
        public virtual object Get(int fieldPos)
        {
            switch (fieldPos)
            {
            case 0: return this.body;
            default: throw new AvroRuntimeException("Bad index " + fieldPos + " in Get()");
            };
        }
        public virtual void Put(int fieldPos, object fieldValue)
        {
            switch (fieldPos)
            {
            case 0: this.body = (System.String)fieldValue; break;
            default: throw new AvroRuntimeException("Bad index " + fieldPos + " in Put()");
            };
        }
    }
}

Summary

While this works for a single file the experience doesn't really scale when we start dealing with multiple files.

For Java developers, there are plugins which handle this so all developers need to do is add AVDL files to their projects and the tooling generates the Java classes.

Ideally, we should have some equivalent in .NET, which is what I'll be covering in the rest of this series!

Dockerising .NET Core & Svelte

Cain Voong — Sat, 23 Jan 2021 19:09:55 +0000

The previous post covered how to set up a web application with a .NET Core backend and Svelte frontend.

In this post we'll package it up into a single Docker image ready to be deployed somewhere.

Prerequisites

Docker

Approach

We'll be taking advantage of Docker's multi-stage builds to:

build the frontend using Node
then the backend using .NET Core SDK
copy the results to an image containing only the ASP.NET Core runtime

Dockerfile

Create a file named Dockerfile. I usually do this with:

touch Dockerfile

Frontend Build Stage

We'll start the file off with the below snippet. I've added comments to describe each step:

# Use Node.js 14, name this stage 'frontend'
FROM node:14 AS frontend

# Our working directory within the image
WORKDIR /build

# Copy package and lock files then install dependencies
COPY package.json .             
COPY package-lock.json .
RUN npm install

# Copy the rest of the files for the frontend
COPY rollup.config.js .
COPY svelte-app ./svelte-app

# Build - this'll output files to /build/wwwroot
RUN npm run build

If you're wondering "why copy the package and lock files first?": it's so Docker creates and caches an intermediate layer containing your dependencies (in node_modules). This gets reused in subsequent builds if those files remain unchanged and saves having to redownload dependencies on each build.

Backend Build Stage

Now add:

# Use .NET Core SDK, name this stage 'backend'
FROM mcr.microsoft.com/dotnet/sdk:5.0-alpine AS backend
WORKDIR /build

# Copy the csproj file then install dependencies
COPY dotnet-svelte.csproj .
RUN dotnet restore dotnet-svelte.csproj

# Copy everything else
COPY . .

# Publish, and output the results to /publish
RUN dotnet publish -c Release -o /publish

Similar to the frontend stage, we copy the csproj file first and restore dependencies to create an intermediate layer.

Combine the Results

We named our stages in the earlier snippets with AS frontend and AS backend. In our final stage we can reference the earlier stages and copy files from them. Add to the file:

# ASP.NET Core Runtime
FROM mcr.microsoft.com/dotnet/aspnet:5.0-alpine

# Where our application will live
WORKDIR /app

# Copy the build results of the frontend stage
COPY --from=frontend /build/wwwroot ./wwwroot

# Copy the build results of the backend stage
COPY --from=backend /publish .

# Run our web application on startup (will listen on port 80 by default)
ENTRYPOINT /app/dotnet-svelte

Final Dockerfile

Should look like this (comments omitted):

FROM node:14 AS frontend
WORKDIR /build
COPY package.json .             
COPY package-lock.json .
RUN npm install
COPY rollup.config.js .
COPY svelte-app ./svelte-app
RUN npm run build

FROM mcr.microsoft.com/dotnet/sdk:5.0-alpine AS backend
WORKDIR /build
COPY dotnet-svelte.csproj .
RUN dotnet restore dotnet-svelte.csproj
COPY . .
RUN dotnet publish -c Release -o /publish

FROM mcr.microsoft.com/dotnet/aspnet:5.0-alpine
WORKDIR /app
COPY --from=frontend /build/wwwroot ./wwwroot
COPY --from=backend /publish .
ENTRYPOINT /app/dotnet-svelte

Building & Running

Builds an image from the Dockerfile, and name it dotnet-svelte:

docker build -t dotnet-svelte .

Then run with:

docker run --rm -p 5000:80 dotnet-svelte

--rm tells docker to automatically remove the container once stopped
-p 5000:80 maps your local port 5000 to the container's port 80

Finally, browse to http://localhost:5000 and once again you will see:

Congratulations! You now have an image ready to be deployed somewhere.

Summary

This post was really more about Docker and multi-stage builds than anything else. We:

Used a Node image to build the Svelte application
Used a .NET Core SDK image to build the ASP.NET application
Copied the results of both to an ASP.NET runtime image

The resulting image contains only the parts we need: static files for the frontend, binary files for the backend and the runtime required to run it.

.NET Core and Svelte

Cain Voong — Thu, 03 Dec 2020 09:03:45 +0000

In this post I will show you how to set up a .NET Core application with a Svelte frontend for development.

Svelte

You've probably heard of React, Vue and Angular by now. Svelte is an alternative to those with some important differences which make it stand out:

Traditional frameworks allow you to write declarative state-driven code, but there's a penalty: the browser must do extra work to convert those declarative structures into DOM operations, using techniques like virtual DOM diffing that eat into your frame budget and tax the garbage collector.

Instead, Svelte runs at build time, converting your components into highly efficient imperative code that surgically updates the DOM. As a result, you're able to write ambitious applications with excellent performance characteristics.

https://svelte.dev/blog/svelte-3-rethinking-reactivity#What_is_Svelte

Personally, I like Svelte because I just find it so easy - jQuery easy - to build applications with. If you already have a good grasp of JavaScript, HTML and CSS then there isn't much more to learn in order to get started.

Before you begin

Have the following installed:

At the time of writing, I'm running .NET Core 5.0.100 and Node.JS 14.15.1

Creating



npx degit sveltejs/template dotnet-svelte

This creates a dotnet-svelte directory then places the Svelte starter template in there. Next thing to do is spin up a .NET Core Web Application within that folder:



cd dotnet-svelte
dotnet new web

Configuration

Rename Directories

This makes things a bit more in line with .NET conventions.



# Rename 'src' to 'svelte-app'
mv src svelte-app

# Rename 'public' to 'wwwroot'
mv public wwwroot

`rollup.config.js`:

The Svelte starter template uses sirv during development to serve static files. We can switch to using our .NET application by applying the modifications below.

In function serve():



server = require('child_process').spawn('npm', ['run', 'start', '--', '--dev']



server = require('child_process').spawn('dotnet', ['watch', 'run']

In export default:



input: 'svelte-app/main.js', // <- we renamed src to svelte-app
output: {
    sourcemap: true,
    format: 'iife',
    name: 'app',
    file: 'wwwroot/build/bundle.js' // <- renamed public to wwwroot
}

A little further down, also update the directory livereload watches:



!production && livereload('wwwroot')

`Startup.cs`

On the .NET side we can tell it to serve the static content which Svelte generates.

In the Configure method:



// Add these before app.UseRouting();
app.UseDefaultFiles();
app.UseStaticFiles();

Running

Now we're ready!



npm install
npm run dev

Open your browser at http://localhost:5000 and you should be greeted with:

Development

If you modify any of the frontend files they'll get automatically rebuilt and refreshed in the browser. Backend changes are also automatically rebuilt but you'll have to manually refresh.

Summary

We took the Svelte starter template, tweaked it a little to fit with some .NET Core defaults then quickly set up a .NET Core service to serve the static files built by Svelte.

From here you can start building out your application.

At some point, you will need to deploy this somewhere which will require building both applications separately then combining the results. I'll cover that in a separate post.

DEV Community: Cain Voong

Add AVDL Support to a .NET Project

Create the Project

Modify .csproj

Inform MSBuild

Tweak for Visual Studio

Build

Conclusion

Automate AVDL to C# Generation

Prerequisites

Folder Structure

The Generate.ps1 Script

What it does:

Running

Summary

Create C# Classes from Avro IDL

Avro Schema

Avro IDL

Avro IDL to C# Class

Steps

Summary

Dockerising .NET Core & Svelte

Prerequisites

Approach

Dockerfile

Frontend Build Stage

Backend Build Stage

Combine the Results

Final Dockerfile

Building & Running

Summary

.NET Core and Svelte

Svelte

Before you begin

Creating

Configuration

Rename Directories

rollup.config.js:

Startup.cs

Running

Development

Summary

`rollup.config.js`:

`Startup.cs`