gRPC on dotnet core 3.1 on RaspberryPi 3

gRPC on dotnet core 3.1 on RaspberryPi 3 (originally written for preview3, the nuget packages have been updated for the latest version)

tl;dr

It's not officially supported, but by compiling the native libgrpc_csharp_ext library for ARM, it is possible. I've made it easier with a set of scripts to help you compile it on your own. I've also created a nuget package too, so you don't have to compile it and a template dotnet new grpcpi to connect all the dots and supercharge your grpc pi life. Also, if your goal is to put your code on github to share (please do!), try gitstub, a dotnet cli global tool that encourages github-first development.

• To compile libgrpc_csharp_ext on your own
  • https://github.com/erikest/libgrpc_csharp_ext
• To get an already compiled libgrpc_csharp_ext.x86.so (compiled for arm) that you can add to your project right now
  • https://www.nuget.org/packages/libgrpc_csharp_ext.arm7
• To start a fresh grpc project ready to publish to linux-arm for the pi.
  • https://github.com/erikest/grpcpi
  • https://www.nuget.org/packages/grpcpi/
  • dotnet new -i grpcpi
  • note: as of .net core 3.1, there is an issue with the publish command selecting the wrong version of libgrpc_csharp_ext.x86.so when publishing for linux-arm.
• To get gitstub, so you can get all your projects up to GitHub with minimal fuss, see my other post about it or just install and discover on your own
  • dotnet tool install --global gitstub

Intro

It's a great day when you sit down to get something done with a new magic bauble, the excitement visible in your eyes, only to discover the coolness is not yet supported for 'your architecture/platform/environment'. Thus was the case when I decided to start a new client/server project with RaspberryPis. Being eager to use dotnet core, but wanting something a little more terse than json over the wire, I discovered the efficient client/server protocol gRPC, was now available for my favorite cross-platform framework - dotnet core. The tooling is new and shiny, with a canonical 'hello world' template ready to go in Visual Studio 2019 and dotnet core 3 preview 6 from the command line (dotnet new grpc). Unfortunately, the officially supported processor architectures are x64 and x86 - arm has been left out for now. However, we'll see that with a little elbow grease and hackery, we can get the new gRPC bits working with the RaspberryPi, then get the demo running and let out a satisfied chortle when the pi talks back.

Overview

We're going to look at a couple of aspects of getting this working:

1. First we'll discuss the set up - devices, development environments, tooling, etc.
2. Then we'll walk through the process of compiling the native libgrpc_csharp_ext for the arm7 processor on the RaspberryPi. We'll look at the automated build/test script and an experimental script in the repository that is used to compile the library for android as our inspiration. Don't worry, it turns out to not be too complicated and I've got you covered with scripts to make it straight forward if you want to follow along.
3. With a native library in hand, we'll examine the canned gRPC template and what we need to do to get it working on the pi
4. After crawling through that, we'll bring in a nuget package that will save us some tedium
5. Finally, we'll look at the new grpcpi template, install it and generate a new project that ties it all together. We'll work through this example to send a message from the server to the pi via gRPC, which the pi will then use to generate a series of blinks on an led. At the end we'll have all the pieces we need to start developing more meaningful applications on this platform.
6. As a bonus, we'll install gitstub, a dotnet cli global tool that connects up a new or existing project to a github repository to jump start your dev cycle and get your source open in record time.

Let's Get Started!

Tools, devices, environments, oh my!

We'll need some tools. First let's talk about our main development machine. In my case an old x64 based dell laptop. On this box, be it physical or virtual, we'll need the dotnet preview6 sdk and runtime. Those are available here. Since we're going to examine the new templates in visual studio, we'll need Visual Studio 2019

Next, for the RaspberryPi, we'll be using raspbian as the linux distro, though others will likely work fine. We'll also need a few other bits to compile the library. You can step through these individually below or you can use the following commands to download a script that should work to automate this and compile the library all in one step.

As always, don't trust me and my script until you read it and verify that it isn't a nefarious evil doer masquerading as an innocent attempt to get you compiling this obscure library.

On the pi:

wget https://github.com/erikest/libgrpc_csharp_ext/setupAndCompile.sh
chmod +x ./setupAndCompile.sh
./setupAndCompile.sh

To compile this library, we'll need the build tools, the source code and as we'll discuss later, we'll need to determine how to build just the library we're interested in, as the grpc source code contains lots of other bits that we don't need to recompile for this task.

1. Get git - if you're following along on raspbian, then you've already got git, good.
2. Install the build tools (script).
   • In the README.md for the csharp submodule, we're told to look at the BUILD.md.
   • BUILD.md tells us for linux compilation, the tools we'll need are:

$ [sudo] apt-get install build-essential autoconf libtool pkg-config
$ [sudo] apt-get install libgflags-dev libgtest-dev
$ [sudo] apt-get install clang libc++-dev

• What they don't tell you, probably because it's not supported, is that we'll also need cmake, which is listed under Windows as a build dependency and we'll see why when we go over the build process.

$ [sudo] apt-get install cmake

1. Git the source (script)

 $ mkdir ~/libgrpc_csharp_ext/
 $ cd ~/libgrpc_cshar_ext/
 $ git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
 $ cd grpc
 $ git submodule update --init

• This is also from BUILD.md
• If you want to deploy your app to the pi as a framework-dependent deployment or a framework-dependent executable, you'll need the dotnet runtime. Otherwise, publishing for a particular runtime will create a self-contained deployment), meaning all of the necessary libraries are published with the executable. You can read more about the three styles here. For this walkthrough, we'll being doing a framework-dependent executable, so we'll need the runtime.
  • As of this writing, the latest version, 3.1, can be had thus (script):

#Dependencies
sudo apt-get install curl libunwind8 gettext

#SDK needed to compile
wget https://download.visualstudio.microsoft.com/download/pr/d52fa156-1555-41d5-a5eb-234305fbd470/173cddb039d613c8f007c9f74371f8bb/dotnet-sdk-3.1.101-linux-arm.tar.gz

#Runtime needed to run apps
wget https://download.visualstudio.microsoft.com/download/pr/da60c9fc-c329-42d6-afaf-b8ef2bbadcf3/14655b5928319349e78da3327874592a/aspnetcore-runtime-3.1.1-linux-arm.tar.gz

#Make a home and extract the tars
mkdir -p $HOME/dotnet
tar zxf dotnet-sdk-3.1.101-linux-arm.tar.gz -C $HOME/dotnet
tar zxf aspnetcore-runtime-3.1.1-linux-arm.tar.gz -C $HOME/dotnet

#Set environment variables, including the path so the dotnet cli can be found
export DOTNET_ROOT=$HOME/dotnet
export PATH=$PATH:$HOME/dotnet

#push the path to the profile so it 'sticks' after restarts
sudo echo "export DOTNET_ROOT=$HOME/dotnet" >> /etc/profile
sudo echo "export PATH=$PATH:/$HOME/dotnet" >> /etc/profile

OK! We've got tools, we've got source, now we just need to put them to work

Compiling libgrpc_csharp_ext

If we again inspect the csharp submodule README, we'll see the recommended way to get compiling is:

# from the gRPC repository root
$ python tools/run_tests/run_tests.py -l csharp -c dbg --build_only

If you tried to run that now, you'd get a fair number of errors. The problem is that this run_tests.py is trying to compile all the stuff in the csharp submodule - we only want the native library. What we need to do is examine run_tests.py to see what it is doing and if we can peel away the other build steps and just get the native library out.

In run_tests.py, each language has a class which is switched using the -l flag, as in the command above -l csharp. When we do a find for csharp, we see some goodies. Specifically, let's note:

• a method called pre_build_steps. Hmm, that sounds interesting and it's pointing to another script
• a method called make_targets. Since make is typically orchestrating gcc/g++ builds and we're trying to compile a native c++ library, this is also intriguing. And it references a target called 'grpc_csharp_ext' - that sounds like just what we need!

As a bit of confirmation, let's look at the csharp/experimental folder, where there are scripts for compiling the native library for android, ios and unity. In the android script we see, after a call to cmake to set up the build environment: make -j4 grpc_csharp_ext AHA! the -j4 switch tells the compiler how many threads to spawn and then we have our grpc_csharp_ext target. Okay, we're feeling good now that we're on the right track. We don't actually have a makefile yet though, so we need to get that sorted.

Before we can make, we have to cmake a makefile, so let's make way back to that pre_build_script and take a look under the hood.

set -ex

# cd to repository root
cd "$(dirname "$0")/../../.."

mkdir -p cmake/build
cd cmake/build

cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

cd ../../src/csharp

dotnet restore Grpc.sln

Looks promising, we've got a little directory shuffling, then a call to cmake (which should now make clear why we needed that tool installed as well), and finally a restore on the grpc solution, to get ready for the dotnet part of the build. Well, we don't want the dotnet part though, we just want the cmake call to set up the build environment and then to build our native target.

By combining what we learned from the make target (grpc_csharp_ext), with the prebuild step here, we can arrive at the commands to perform the compilation. These are executed relative to the root of our work, which if you're following along should be at ~/libgrpc_csharp_ext/.

here's our script, also located here

set -ex

#intended to be run from the repository root
cd grpc/

#create location for build files
mkdir -p cmake/build
cd cmake/build

#setup the build files
cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

#compile libgrpc_csharp_ext.so
make -j4 grpc_csharp_ext

cp libgrpc_csharp_ext.so ../../../libgrpc_csharp_ext.x86.so

Everything there looks solid, but, why are we copying libgrpc_csharp_ext.so to libgrpc_csharp_ext.x86.so? Well, that's a perfect segue into the next topic!

gRPC template

On our dev box, we can create a new grpc project using the included template

mkdir grpcDemo
cd grpcDemo
dotnet new grpc

If we then publish the client for the linux-arm runtime identifier (RID)

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

If we don't include the --self-contained false, then we'll get a whole bunch of framework libraries as well. This could be good for portability, since you can just push the whole publish folder to any pi and get going, but for iterative development, doing the one time set up of the framework to avoid the extra file copy seems like the way to go.

we'll find in the /bin/debug/netcoreapp3.0/linux-arm/publish folder two copies of the libgrpc_csharp_ext, one .x86.so and one .x64.so

If we deploy this to an existing directory on the pi, for example by using scp

scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

as noted here, the scp command no longer supports the handy /. which would copy the files without including the directory they are in. So we get a 'publish' folder on the pi when we don't want one. So it goes.

and run it from the pi

cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

we'll get an error like this person did on SO

Which basically says it can't load the libgrpc_csharp_ext.x86.so library. Here's where our problem and the beginning of all this work started - this library isn't compiled for the arm processor, but it is still being included in the publish output and is being referenced by the NativeExtension class. When we dig into the repository and look at the source, specifically here we notice that the file name it looks for depends on the OS, linux in this case which needs a static library, .so, and the architecture. Despite the pi having a 64 bit processor, the reality is that the OS is ARM32, thus the architecture string gets set as x86. All that is to say, on a raspberrypi running raspbian, the file name it tries to load is libgrpc_csharp_ext.x86.so - so, let us give it what it wants!

On the pi

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
./grpcDemo.Client

This time it should pause for about 15-20 seconds before we see

Unhandled Exception: Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="Connect Failed")
   at gRPCpi.Program.Main(String[] args) in C:\Users\Geocoder\Source\Repos\gRPC Template\content\gRPCpi\src\gRPCpi.Clien
t\Program.cs:line 26
   at gRPCpi.Program.<Main>(String[] args)
Aborted

But, hey, that means it is running! It just couldn't connect to the server. We're getting closer to shaving our yak! Well, it turns out it's trying to connect to the server on the localhost by default. So we need to make a couple of modifications, since our server will be running over a network on our dev machine.

• In the Client's program.cs, we need to aim the grpc client at the server's (dev box) ip address. If you don't know yours, ipconfig on windows or ifconfig on linux will get you there.

 //CHANGE THIS TO YOUR SERVER ADDRESS
var serverIpAddress = "YOUR IP";

var channel = new Channel($"{serverIpAddress}:" + port, ChannelCredentials.Insecure);

• We also need the server to be listening on that ip address and port and by default it is not, it listens only on localhost. To open this up to our network IP, we need to make a change to the kestrel server configuration:

Host.CreateDefaultBuilder(args)
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
        webBuilder.UseUrls("http://*:50051");
    });

Here we've said listen on all (*) IP addresses bound to the network interface.

Now, let's republish the client to the pi. So on the dev box again, at the project root

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm
scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

Before we run the client again, let's get the server started and ready. In Visual Studio you can just run a new debug instance, or from the command line

dotnet run --project grpcDemo/grpcDemo.Server/grpcDemo.Server.csproj

Then back on the pi, let's run the client, after we overwrite the libgrpc_csharp_ext library again

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

If all has gone to plan, we should get this

Greeting: Hello GreeterClient

Chortle! woo! we did it - time to hang up our hats, jobs done... Well not exactly, while this shows that we can 'get it working', what we really want is to have the correct library come across with all the other published artifacts, so we can continue development without so many steps. Sure, we could write more scripts to make this less painful, but isn't there a better way?

Libgrpc_csharp_ext.arm7 Nuget Package

So far, we've manually overwritten the 'wrong' native library with our arm compiled version. What we really want is for the correct library to be brought in during publish - so whichever runtime we publish for, it will have the correct version. If you publish for windows or linux or osx, you'll notice the publish command correctly copies over the version needed for each runtime.

What's going on here?

In the Grpc.Core nuget package, which is referenced in the template, are copies of that native library for each of win, linux, and osx runtimes. You can verify this by examining the package cache at YourUserName.nuget\packages\grpc.tools<version>\runtimes folder. The publish command uses this convention to find additional runtime specific components to copy to the build and publish output. This same mechanism can be used to provide the library for the linux-arm runtime. Let's go ahead and add a reference to this package

dotnet add grpcDemo\grpcDemo.Client package libgrpc_csharp_ext.arm7

Then publish again

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

We can verify (by file size) that our version of the library has been copied over and is ready for deployment. Hey, now this project has legs and quick iterations are at hand!

Next up

In the next post, we'll finish all this work up by making it even easier to get started by adding a new project template which puts it all together. Then we'll use gitstub to jumpstart a new open source client/server blinking led project that everybody will surely want!

Comments

[Wayne Taylor]

Thank You So Much For This ! It allowed me to work around a show stopped for running my twitch bot on a pi - but it calls out the Anki Vector Robot (to allow chat to make him talk), Vector uses gRPC under the hood and it wouldn't run on the pi, until I found your post and compiled and it worked :D Thank You !!!!!

[erikest]

Woo hoo! So stoked you were unstucked! When you get it all working, you could do a sweet post about how to make this robot work as a twitch bot on a pi :)

[Wayne Taylor]

Could I be a little cheeky and ask if you could update the Nuget Package to the latest version (I'm not sure how to do)? - Thanks

[erikest]

I've been meaning to update it to the latest version, compiled against the latest version of .net core 3... Should be able to turn my attention to it in the next day or so :)

[erikest]

An updated version of the package compiled against the v1.26 branch of the grpc repo is live. Be warned, while testing against the latest sdk using the dotnet publish command, it incorrectly chose the .so file from the grpc core package, ignoring the runtime identifier - this wasn't the case before, so I've filed an issue on github: github.com/dotnet/sdk/issues/4195

In the meantime, after publish, you may need to manually copy the correct version, or set it up as a post build action.

[Wayne Venables]

Hello Erik,

This is an awesome write up -- I came across your blog while trying to solve this very problem for a different platform and thinking that the issue would affect Raspberry pi's as well and someone, somewhere, might have solved it.

I have a request! Is it possible for you to build an ARM64 version of your package? I'm trying to do dotnet coding on an Android tablet with an Ubuntu userland (oh yeah) and surprisingly everything works but gRPC! Before I found your blog, I tried using the ARM64 Android .so and copying it over the x64.so but unfortunately that didn't work. I'm hoping an ARM64 build would be an quick task for you and would solve my problem.

Thanks!

[Wayne Venables]

I compiled it myself using the scripts and steps in your github project and it worked perfectly! In fact, I compiled it on my Android tablet.

I think perhaps the Android version would have worked but I didn't realize that dotnet overwrites the native libraries in your bin/Debug directory from the .nuget package store on every dotnet run.

[erikest]

Excellent! That's exactly what I was going to suggest.

And regarding the overwriting issue - that could be related to a bug, which I've filed a github issue for, which has to do with the tooling's selection process for the native library: github.com/dotnet/sdk/issues/4195.

I might have to check out these vector robots as you're the author of the .net sdk and another Wayne who commented earlier, was trying to control said Vector robot from a Pi, presumably using your sdk :)

[J Allen]

I was so excited to deploy my first gRPC .NET 5 project to my Raspberry Pi 4...only to see errors in my GitHub Actions when it tried to build my container. Searching the interwebs led me to this post which seems so tantalizingly close but IIUC your nuget package works for linux-arm (rpi3), not linux-arm64v8 (rpi4). I'm running the 64-bit version of Raspberry Pi OS and I'm stymied. :-( I'm trying to leave the box itself as untouched as possible and put all my dependencies in my Docker images so I'm not sure about building my own version of libgrpc_csharp_ext. Do you know of a resource or a way I could find a way to do gRPC with .NET 5 on arm64v8?

[evil3dm]

Hi,

Thank you very much for your research and nuget package.
I saw that you posted about regression issue that your nuget package does not override the file. But want to clarify is it related to issue that I have.
Basically I see the same bahavior as in this ticket github.com/grpc/grpc/issues/23336

I'm trying to build in the docker on raspberry pi 4 and getting:
/root/.nuget/packages/grpc.tools/2.29.0/build/_protobuf/Google.Protobuf.Tools.targets(264,5): error MSB6003: The specified task executable "/root/.nuget/packages/grpc.tools/2.29.0/tools/linux_x86/protoc" could not be run. System.ComponentModel.Win32Exception (8): Exec format error

As I understant it's because linux_x86/protoc is not built for arm. And libgrpc_csharp_ext.x86.so would not help much ? I'm a bit confused, sorry. I assume that grpc team have changed their approach in building stubs ?

[erikest]

Ya, so my quick take here is that you're trying to do the compilation on the docker/pi4. I believe that grpc.tools and specifically that protoc is a compile time dependency.

In my post, I'm compiling for a specific runtime, but I'm doing it from a dev machine that is x64. Then I'm pushing those binaries over to the pi. My guess is that the compile time tools for x64 are being used to produce the protoc files at that point, so no proc arch mismatch. If you follow that workflow, you'll probably be fine. I'm not sure if you have a special use case for compiling on the pi directly, but it's probably faster to develop on a beefier machine and then just have script to push the binaries, which could be setup as a post build script, so the pi is always in sync. I have some (probably terrible) example scripts on the github project to do this.

Otherwise, you're right, linux_x86 native libs aren't going to do you any good on an arm processor - runtime or compile time. What would be needed there is essentially this same post, but pivoted to compile the protoc for arm32 and arm64. However, I would carefully consider the value of compilation on the pi - it's slow and you likely have a better dev machine already set up. That issue you linked to pretty clearly states the gRPC teams position on supporting arm, in short paraphrase, "we can't maintain the tooling and test environments, so you're on your own".

There is one 'meta' issue, which is that it's selecting a binary named ...x86.. even though it's on an arm processor. Someday, I'll post a PR to the grpc team that at least fixes this so that if community creates and maintains the arm versions of these libs, the code that selects them will at least be looking for the correctly named file in the correct location, instead of the hack we have now where I just named the arm file for x86.

[Przemek]

Hello all, I am trying to run gRPC on raspberry pi 4 with NET5.0 but I am receiving an error from GetArchitectureString method: "Unsupported architecture" . RuntimeInformation.ProcessArchitecture returns "ARM" what seems to be not supported. Do you have any idea whether it is possible to run it on rpi4 ?

I am using raspberry pi 4 with 32OS. When I used the same on raspberry pi3 then this problem was not existing.

[Nikolai Orekhov]

Note also there are RPi with ARMv6. You won't be able to compile it with a stock compiler so more work needed here. Also I believe you can also allow linux-arm64 RID because armv7 is compatible with v8.

[pawelpetruch]

Hi @erikest ,

I am trying to run grpc core in a container with a built native .so library but I am getting this error:
dotnet: symbol lookup error: /app/libgrpc_csharp_ext.x86.so: undefined symbol: __atomic_fetch_add_8

Have you ever encountered this issue?

Thanks,
Pawel

[erikest]

I have not encountered that one specifically, no. Not sure what guidance to give you at this point as I would need a lot more context to understand.

[Kirill Pinzari]

Very helpful post! I can't describe how much I've struggled with libgrpc_csharp_ext.x86, thank you for all you effort! github repo with steps for building the library saved my day :)

[erikest]

It brings me such joy to hear from someone who benefited from this :) May you find great success in your gRPC pi project!

[Nucu Labs]

That's amazing! Thanks for making this!

[erikest]

I really hoped while getting through this post that someone would find the work and get to carry on blissfully with their awesome open source .net core pi project and poking into your github repo and seeing the package reference there made me very happy :) Good luck with NucuCar (github.com/dnutiu/NucuCar), looks like a fun project!