DEV Community: erikest

I'm a new dev blogger - I can has your tools?

erikest — Mon, 15 Apr 2019 17:19:49 +0000

As a new dev blogger, my tool box is a little, er, sparse. I've been learning markdown, which is nice for formatting, but then I see other people's beautiful looking posts and I think - I want their tools, their methods, their flare. As part of my ascent into blogging nirvana, I thought I'd spark a discussion and see what others think.

Screenshots - I've seen images that blur all but the important stuff, others that highlight/number the relevant clicks and inputs, etc. In my StackOverflow posts, I've used... Paint... Probably time to upgrade, what would you suggest?
Code examples - There's the gist thing from github which I haven't explored, things like codepen and jsfiddle for web stuff. Of course I can \`<language> in markdown - what else? Any pro tips?
Productivity boosters - are you composing your markdown in the dev.to editor or is there a better way? Live previews? Templates?
Other 'nice to haves' - Any other wizzbang I should consider to make my posts POP?

GitStub - a GitHub-first development boostrapper for dotnet core

erikest — Mon, 15 Apr 2019 16:50:09 +0000

tl;dr

My first foray into my own open source project is a tool to lower the barrier to getting code up onto GitHub. I created it because I wanted to start a different open source project and in the process of learning what to do, realized that there are a number of repetitive steps. At first, I sought to create myself a set of scripts that would ease the process by interacting with the GitHub rest api and the dotnet core cli. Soon, I realized that taking these steps and creating a cross platform dotnet core global tool would both be better in terms of ease of use and also allow me to share this with the widest audience - hopefully saving someone else some time and tedium.

You can install it and try it yourself with
dotnet tool install --global gitstub

For issues, requests and ideas to make it better
GitStub on GitHub

Problem

You want to create a new dotnet core project so you

mkdir  MyKillerWebApp
cd MyKillerWebApp
dotnet new webapp

Actually, it needs a solution and a little rearranging you decide

mkdir -p src/MyKillerWebApp
mv * src/MyKillerWebApp
dotnet sln -n MyKillerWebApp

Then you want to put it into a git repository, but first you need to find a .gitignore to add - maybe you have one you used before in another project

cp /users/myuser/repos/someotherdotnetproject/.gitignore ./

Now you can set up your first commit

git init
git add -A
git commit -m "First Commit! The only way is up from here!"

At some point you realize, it should actually be a GitHub project, because you want to share this killer webapp with the world. From here, you go to your browser and GitHub.com, you log in, create a new project, filling out the details.

A few minutes later, the GitHub project is there, but it and your local git are not connected so you still have to

git remote add origin https://user:pass@github.com/user/projectName.git
git push --set-upstream origin master

Finally, you've got everything on GitHub and you can iterate from there.

Or maybe you stop somewhere before that, before even creating the local git repo. You tell yourself, "once it is 'worth it'" you'll take those steps and make a GitHub project. Then maybe you do, maybe you don't. Perhaps you get second thoughts - what if my code isn't 'good enough' yet. Sometimes though, code that is good enough or would be good enough just stays on your laptop, where it dies. Or another time, away from your machine, you are with another dev and you think - "Oh! I started working on something like this - but, ack, it's on my machine, so I'll have to go create a GitHub project and push it up for you to see it... maybe when I get back to my computer." Or maybe it's not that big of a barrier and you do go through those steps every time - but wouldn't it be nice if it was just a little faster, a little easier?

Solution - GitHub-first development with GitStub

The goals of this project are two fold.

Save time - simply put, these are repetitive tasks, they are begging for automation.
Promote a development paradigm where you start projects, public or private, on GitHub and commit right away - GitHub-first.
- This not only makes the code immediately available everywhere, it also facilitates your progress in open source by having the initiation of the project start the cycle of contributions. You get to overcome your fear and perfectionism and replace it with continual public improvement as a feature of your dev life.
- By combining that with dotnet new project templates, you get a bootstrapped, scaffolded project up and ready to be iterated. The first commit is the template, outta the box. Your second commit will contain all the deltas between the base template and what value you've added thus far - and away ye go!
- Hopefully, it takes one small barrier out of the way that might stop you, on the margin, from trying something and putting it on GitHub because you "can't be bothered" to set up the project and connect everything.

How To Use

First, install the tool using

dotnet tool install --global gitstub

Now, suppose I want to start a new console application in a folder called MyKillerApp.
From the command line in the that folder, I can

gitstub console -gsu myusername -gsp mypassword

By default, gitstub will use the current folder as the name for a public project on GitHub. The two parameters after console are my GitHub username (-gsu) and GitHub password (-gsp). This will do the stuff, including adding a .gitignore file that ignores most of the common things for dotnet projects. A few moments later, voila, I can now go to https://github.com/myusername/MyKillerApp and see everything there. If I open up the project in Visual Studio, I can proceed to use the Team Explorer window to process further commits.

Perhaps I want a little more structure with a solution/project laid out like

/
- MyKillerApp.sln
- src/
  - MyKillerApp/
    - MyKillerApp.csproj
    - <other files>

Then I can say

gitstub console --sln -gsu myusername -gsp mypassword

And gitstub will create a solution and a project and arrange them thus.

If I had a different name I wanted to use for the project and solution, I can specify that as well

gitstub console -gsr "FirstKillerApp" -gss "KillerApps" -gsu myusername -gsp mypassword

If it needs to be a private repository, then I can add --private.

gitstub console -gsr "FirstKillerApp" -gss "KillerApps" -gsu myusername -gsp mypassword --private

I can give the GitHub project a description and I can change the first commit message as well

gitstub console --sln -gsu myusername -gsp mypassword -gsd "The best app yet" -gsc "Initialized with the console template"

If my template has additional parameters, I can pass those along - any parameter not reserved for gitstub will be passed to the dotnet new command in original order, so

gitstub webapp -au Individual --sln -gsu myusername -gsp mypassword -gsd "Authed web app example" -gsc "Initialized with webapp and individual authentication"

will run dotnet new webapp -au Individual at the project creation step.

If I have an existing project, I can still use gitstub to get things moving by including the --existing switch. With this, gitstub will skip the project/solution creation step and just commit what's already in the folder. It still adds the .gitignore, so it won't pick up obj/bin/.vs and other stuff.

gitstub -gsu myusername -gsp mypassword --existing

What's Next

You can follow along, post issues, feature requests, tell me how to be better at this open source thing, etc. at the GitHub repo.

In my sights for future features are

getting rid of having to pass username and password each time by implementing GitHub OAuth and caching a token for the session.
a devops switch that would create an Azure Devops project and connect it to the GitHub project
a nuget switch that would search the template output for a .nuspec file and modify it with the appropriate project repository and project name

I'm very open to feedback and suggestions, so let me know what you think!

gRPC on dotnet core 3.1 on RaspberryPi 3

erikest — Tue, 09 Apr 2019 03:35:29 +0000

gRPC on dotnet core 3.1 on RaspberryPi 3 (originally written for preview3, the nuget packages have been updated for the latest version)

tl;dr

It's not officially supported, but by compiling the native libgrpc_csharp_ext library for ARM, it is possible. I've made it easier with a set of scripts to help you compile it on your own. I've also created a nuget package too, so you don't have to compile it and a template dotnet new grpcpi to connect all the dots and supercharge your grpc pi life. Also, if your goal is to put your code on github to share (please do!), try gitstub, a dotnet cli global tool that encourages github-first development.

To compile libgrpc_csharp_ext on your own
- https://github.com/erikest/libgrpc_csharp_ext
To get an already compiled libgrpc_csharp_ext.x86.so (compiled for arm) that you can add to your project right now
- https://www.nuget.org/packages/libgrpc_csharp_ext.arm7
To start a fresh grpc project ready to publish to linux-arm for the pi.
- https://github.com/erikest/grpcpi
- https://www.nuget.org/packages/grpcpi/
- dotnet new -i grpcpi
- note: as of .net core 3.1, there is an issue with the publish command selecting the wrong version of libgrpc_csharp_ext.x86.so when publishing for linux-arm.
To get gitstub, so you can get all your projects up to GitHub with minimal fuss, see my other post about it or just install and discover on your own
- dotnet tool install --global gitstub

Intro

It's a great day when you sit down to get something done with a new magic bauble, the excitement visible in your eyes, only to discover the coolness is not yet supported for 'your architecture/platform/environment'. Thus was the case when I decided to start a new client/server project with RaspberryPis. Being eager to use dotnet core, but wanting something a little more terse than json over the wire, I discovered the efficient client/server protocol gRPC, was now available for my favorite cross-platform framework - dotnet core. The tooling is new and shiny, with a canonical 'hello world' template ready to go in Visual Studio 2019 and dotnet core 3 preview 6 from the command line (dotnet new grpc). Unfortunately, the officially supported processor architectures are x64 and x86 - arm has been left out for now. However, we'll see that with a little elbow grease and hackery, we can get the new gRPC bits working with the RaspberryPi, then get the demo running and let out a satisfied chortle when the pi talks back.

Overview

We're going to look at a couple of aspects of getting this working:

First we'll discuss the set up - devices, development environments, tooling, etc.
Then we'll walk through the process of compiling the native libgrpc_csharp_ext for the arm7 processor on the RaspberryPi. We'll look at the automated build/test script and an experimental script in the repository that is used to compile the library for android as our inspiration. Don't worry, it turns out to not be too complicated and I've got you covered with scripts to make it straight forward if you want to follow along.
With a native library in hand, we'll examine the canned gRPC template and what we need to do to get it working on the pi
After crawling through that, we'll bring in a nuget package that will save us some tedium
Finally, we'll look at the new grpcpi template, install it and generate a new project that ties it all together. We'll work through this example to send a message from the server to the pi via gRPC, which the pi will then use to generate a series of blinks on an led. At the end we'll have all the pieces we need to start developing more meaningful applications on this platform.
As a bonus, we'll install gitstub, a dotnet cli global tool that connects up a new or existing project to a github repository to jump start your dev cycle and get your source open in record time.

Let's Get Started!

Tools, devices, environments, oh my!

We'll need some tools. First let's talk about our main development machine. In my case an old x64 based dell laptop. On this box, be it physical or virtual, we'll need the dotnet preview6 sdk and runtime. Those are available here. Since we're going to examine the new templates in visual studio, we'll need Visual Studio 2019

Next, for the RaspberryPi, we'll be using raspbian as the linux distro, though others will likely work fine. We'll also need a few other bits to compile the library. You can step through these individually below or you can use the following commands to download a script that should work to automate this and compile the library all in one step.

As always, don't trust me and my script until you read it and verify that it isn't a nefarious evil doer masquerading as an innocent attempt to get you compiling this obscure library.

On the pi:

wget https://github.com/erikest/libgrpc_csharp_ext/setupAndCompile.sh
chmod +x ./setupAndCompile.sh
./setupAndCompile.sh

To compile this library, we'll need the build tools, the source code and as we'll discuss later, we'll need to determine how to build just the library we're interested in, as the grpc source code contains lots of other bits that we don't need to recompile for this task.

Get git - if you're following along on raspbian, then you've already got git, good.
Install the build tools (script).
- In the README.md for the csharp submodule, we're told to look at the BUILD.md.
- BUILD.md tells us for linux compilation, the tools we'll need are:

$ [sudo] apt-get install build-essential autoconf libtool pkg-config
$ [sudo] apt-get install libgflags-dev libgtest-dev
$ [sudo] apt-get install clang libc++-dev

What they don't tell you, probably because it's not supported, is that we'll also need cmake, which is listed under Windows as a build dependency and we'll see why when we go over the build process.

$ [sudo] apt-get install cmake

Git the source (script)

 $ mkdir ~/libgrpc_csharp_ext/
 $ cd ~/libgrpc_cshar_ext/
 $ git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
 $ cd grpc
 $ git submodule update --init

This is also from BUILD.md
If you want to deploy your app to the pi as a framework-dependent deployment or a framework-dependent executable, you'll need the dotnet runtime. Otherwise, publishing for a particular runtime will create a self-contained deployment), meaning all of the necessary libraries are published with the executable. You can read more about the three styles here. For this walkthrough, we'll being doing a framework-dependent executable, so we'll need the runtime.
- As of this writing, the latest version, 3.1, can be had thus (script):

#Dependencies
sudo apt-get install curl libunwind8 gettext

#SDK needed to compile
wget https://download.visualstudio.microsoft.com/download/pr/d52fa156-1555-41d5-a5eb-234305fbd470/173cddb039d613c8f007c9f74371f8bb/dotnet-sdk-3.1.101-linux-arm.tar.gz

#Runtime needed to run apps
wget https://download.visualstudio.microsoft.com/download/pr/da60c9fc-c329-42d6-afaf-b8ef2bbadcf3/14655b5928319349e78da3327874592a/aspnetcore-runtime-3.1.1-linux-arm.tar.gz

#Make a home and extract the tars
mkdir -p $HOME/dotnet
tar zxf dotnet-sdk-3.1.101-linux-arm.tar.gz -C $HOME/dotnet
tar zxf aspnetcore-runtime-3.1.1-linux-arm.tar.gz -C $HOME/dotnet

#Set environment variables, including the path so the dotnet cli can be found
export DOTNET_ROOT=$HOME/dotnet
export PATH=$PATH:$HOME/dotnet

#push the path to the profile so it 'sticks' after restarts
sudo echo "export DOTNET_ROOT=$HOME/dotnet" >> /etc/profile
sudo echo "export PATH=$PATH:/$HOME/dotnet" >> /etc/profile

OK! We've got tools, we've got source, now we just need to put them to work

Compiling libgrpc_csharp_ext

If we again inspect the csharp submodule README, we'll see the recommended way to get compiling is:

# from the gRPC repository root
$ python tools/run_tests/run_tests.py -l csharp -c dbg --build_only

If you tried to run that now, you'd get a fair number of errors. The problem is that this run_tests.py is trying to compile all the stuff in the csharp submodule - we only want the native library. What we need to do is examine run_tests.py to see what it is doing and if we can peel away the other build steps and just get the native library out.

In run_tests.py, each language has a class which is switched using the -l flag, as in the command above -l csharp. When we do a find for csharp, we see some goodies. Specifically, let's note:

a method called pre_build_steps. Hmm, that sounds interesting and it's pointing to another script
a method called make_targets. Since make is typically orchestrating gcc/g++ builds and we're trying to compile a native c++ library, this is also intriguing. And it references a target called 'grpc_csharp_ext' - that sounds like just what we need!

As a bit of confirmation, let's look at the csharp/experimental folder, where there are scripts for compiling the native library for android, ios and unity. In the android script we see, after a call to cmake to set up the build environment: make -j4 grpc_csharp_ext AHA! the -j4 switch tells the compiler how many threads to spawn and then we have our grpc_csharp_ext target. Okay, we're feeling good now that we're on the right track. We don't actually have a makefile yet though, so we need to get that sorted.

Before we can make, we have to cmake a makefile, so let's make way back to that pre_build_script and take a look under the hood.

set -ex

# cd to repository root
cd "$(dirname "$0")/../../.."

mkdir -p cmake/build
cd cmake/build

cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

cd ../../src/csharp

dotnet restore Grpc.sln

Looks promising, we've got a little directory shuffling, then a call to cmake (which should now make clear why we needed that tool installed as well), and finally a restore on the grpc solution, to get ready for the dotnet part of the build. Well, we don't want the dotnet part though, we just want the cmake call to set up the build environment and then to build our native target.

By combining what we learned from the make target (grpc_csharp_ext), with the prebuild step here, we can arrive at the commands to perform the compilation. These are executed relative to the root of our work, which if you're following along should be at ~/libgrpc_csharp_ext/.

here's our script, also located here

set -ex

#intended to be run from the repository root
cd grpc/

#create location for build files
mkdir -p cmake/build
cd cmake/build

#setup the build files
cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

#compile libgrpc_csharp_ext.so
make -j4 grpc_csharp_ext

cp libgrpc_csharp_ext.so ../../../libgrpc_csharp_ext.x86.so

Everything there looks solid, but, why are we copying libgrpc_csharp_ext.so to libgrpc_csharp_ext.x86.so? Well, that's a perfect segue into the next topic!

gRPC template

On our dev box, we can create a new grpc project using the included template

mkdir grpcDemo
cd grpcDemo
dotnet new grpc

If we then publish the client for the linux-arm runtime identifier (RID)

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

If we don't include the --self-contained false, then we'll get a whole bunch of framework libraries as well. This could be good for portability, since you can just push the whole publish folder to any pi and get going, but for iterative development, doing the one time set up of the framework to avoid the extra file copy seems like the way to go.

we'll find in the /bin/debug/netcoreapp3.0/linux-arm/publish folder two copies of the libgrpc_csharp_ext, one .x86.so and one .x64.so

If we deploy this to an existing directory on the pi, for example by using scp

scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

as noted here, the scp command no longer supports the handy /. which would copy the files without including the directory they are in. So we get a 'publish' folder on the pi when we don't want one. So it goes.

and run it from the pi

cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

we'll get an error like this person did on SO

Which basically says it can't load the libgrpc_csharp_ext.x86.so library. Here's where our problem and the beginning of all this work started - this library isn't compiled for the arm processor, but it is still being included in the publish output and is being referenced by the NativeExtension class. When we dig into the repository and look at the source, specifically here we notice that the file name it looks for depends on the OS, linux in this case which needs a static library, .so, and the architecture. Despite the pi having a 64 bit processor, the reality is that the OS is ARM32, thus the architecture string gets set as x86. All that is to say, on a raspberrypi running raspbian, the file name it tries to load is libgrpc_csharp_ext.x86.so - so, let us give it what it wants!

On the pi

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
./grpcDemo.Client

This time it should pause for about 15-20 seconds before we see

Unhandled Exception: Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="Connect Failed")
   at gRPCpi.Program.Main(String[] args) in C:\Users\Geocoder\Source\Repos\gRPC Template\content\gRPCpi\src\gRPCpi.Clien
t\Program.cs:line 26
   at gRPCpi.Program.<Main>(String[] args)
Aborted

But, hey, that means it is running! It just couldn't connect to the server. We're getting closer to shaving our yak! Well, it turns out it's trying to connect to the server on the localhost by default. So we need to make a couple of modifications, since our server will be running over a network on our dev machine.

In the Client's program.cs, we need to aim the grpc client at the server's (dev box) ip address. If you don't know yours, ipconfig on windows or ifconfig on linux will get you there.

 //CHANGE THIS TO YOUR SERVER ADDRESS
var serverIpAddress = "YOUR IP";

var channel = new Channel($"{serverIpAddress}:" + port, ChannelCredentials.Insecure);

We also need the server to be listening on that ip address and port and by default it is not, it listens only on localhost. To open this up to our network IP, we need to make a change to the kestrel server configuration:

Host.CreateDefaultBuilder(args)
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
        webBuilder.UseUrls("http://*:50051");
    });

Here we've said listen on all (*) IP addresses bound to the network interface.

Now, let's republish the client to the pi. So on the dev box again, at the project root

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm
scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

Before we run the client again, let's get the server started and ready. In Visual Studio you can just run a new debug instance, or from the command line

dotnet run --project grpcDemo/grpcDemo.Server/grpcDemo.Server.csproj

Then back on the pi, let's run the client, after we overwrite the libgrpc_csharp_ext library again

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

If all has gone to plan, we should get this

Greeting: Hello GreeterClient

Chortle! woo! we did it - time to hang up our hats, jobs done... Well not exactly, while this shows that we can 'get it working', what we really want is to have the correct library come across with all the other published artifacts, so we can continue development without so many steps. Sure, we could write more scripts to make this less painful, but isn't there a better way?

Libgrpc_csharp_ext.arm7 Nuget Package

So far, we've manually overwritten the 'wrong' native library with our arm compiled version. What we really want is for the correct library to be brought in during publish - so whichever runtime we publish for, it will have the correct version. If you publish for windows or linux or osx, you'll notice the publish command correctly copies over the version needed for each runtime.

What's going on here?

In the Grpc.Core nuget package, which is referenced in the template, are copies of that native library for each of win, linux, and osx runtimes. You can verify this by examining the package cache at YourUserName.nuget\packages\grpc.tools<version>\runtimes folder. The publish command uses this convention to find additional runtime specific components to copy to the build and publish output. This same mechanism can be used to provide the library for the linux-arm runtime. Let's go ahead and add a reference to this package

dotnet add grpcDemo\grpcDemo.Client package libgrpc_csharp_ext.arm7

Then publish again

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

We can verify (by file size) that our version of the library has been copied over and is ready for deployment. Hey, now this project has legs and quick iterations are at hand!

Next up

In the next post, we'll finish all this work up by making it even easier to get started by adding a new project template which puts it all together. Then we'll use gitstub to jumpstart a new open source client/server blinking led project that everybody will surely want!