DEV Community: Magnus Stråle

The Ray Tracer Challenge in Rust

Magnus Stråle — Wed, 18 Sep 2019 10:57:35 +0000

The time has come..

This summer I started to learn Rust by using The Ray Tracer Challenge book and implementing it in Rust. It's been a side project but extremely fun and challenging. I also promised to publish my code on GitHub over two months ago, so this is long overdue. The code is not complete, there are several chapters that I still haven't worked thru so expect to see some more changes over time.

Anyway - now I will expose my stumbling steps in Rust to the world. After some consideration I decided to actually publish the entire Git repo with history intact rather than just start from the current position. It is a bit embarrassing to look at some of the earlier things I did, but I fully expect the current state of the code to be embarrassing for my future self, so what the heck...

https://github.com/magnusstrale/raytracer

Some reflections on the code

One thing I found interesting when going over the Git history was my ambivalence in deciding how to deal with matrices (a fundamental part of ray tracing), more specifically how the data should be stored in the Matrix struct. The biggest hurdle was the fact that an array in Rust has to have a fixed size, since the size of the array is part of its type. You either have to use a Vec (vector, resembling a variable size array) or provide different functions to call depending on the array size.

The solution I have right now (maybe I'll change my mind again in the future) is to internally always use a 4x4 array to store the data, even when working with 2x2 or 3x3 matrices. The main reason for this decision was that I wanted to have the Copy trait implemented for Matrix (which is not possible when using a Vec). This allows for the expressions with matrix arithmetic look much cleaner since I don't need to put in a lot of &'s to borrow the data. See here for an example of "clean" matrix multiplication.

There are earlier states of the code base where I have used a Vec and it's possible that I'll switch back to Vec again in the future.

Next steps

Finishing the book. There are several chapters in the RTC book remaining and I intend to continue until the end.
Performance optimizations. Right now I've just tried to keep the code as clean as possible, but I look forward to performance tuning. I will play around with Vec vs Array, the possible performance implications of Copy etc.
Target WASM / browsers for ray tracing. I want to get more experience with Web Assembly and the Rust tool chain for producing WASM.
Build a Ray Tracing web site. Having something fairly simple where it will be possible to interactively create a world and then get a nice ray-traced image from the WASM package running in your browser. Of course I will use use Rust for any server-side coding.

Closing remarks

Feel free to browse the code here, download and play around with it. I will be very happy for any comments on how to improve the code, especially if there is anything in Rust that can be leveraged to reduce the amount of code or make it more idiomatic.

Rust Trait objects in a vector - non-trivial...

Magnus Stråle — Wed, 11 Sep 2019 07:59:07 +0000

One (of many) challenges with learning Rust is to un-learn a lot of object-oriented thinking. My recent struggle with a refactoring to put trait objects into a Vec made this painfully obvious.

If you don't want to read about the road to enlightenment but skip straight to the answer, scroll down to the final code snippet now.

Basically I had a Vec<Sphere> and needed to generalize the Vec to take a Trait Shape instead. I e something like Vec<Shape>. My first OO-based mistake is to assume that Traits behave like interfaces. This mistake immediately makes the compiler puke all over the screen.

// Original code
struct Sphere {
    transformation: Matrix, // FYI - implements Copy trait
    material: Material
}

fn do_stuff(objects: Vec<Sphere>) {
}

// New code - won't compile
trait Shape {
    fn material(&self) -> &Material;
    fn transformation(&self) -> Matrix;
}

struct Sphere {
    transformation: Matrix,
    material: Material
}

impl Shape for Sphere {
    fn material(&self) -> &Material {
        &self.material
    }
    fn transformation(&self) -> Matrix {
        self.transformation
    }
}

fn do_stuff(objects: Vec<dyn Shape>) {
}

The compiler is quite helpful here

error[E0277]: the size for values of type `(dyn Shape + 'static)` cannot be known at compilation time
  --> src\lib.rs:45:1
   |
45 | / fn do_stuff(objects: Vec<dyn Shape>) {
46 | | }
   | |_^ doesn't have a size known at compile-time
   |
   = help: the trait `std::marker::Sized` is not implemented for `(dyn Shape + 'static)`
   = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
   = note: required by `std::vec::Vec`

So far quite obvious - Shape is a trait that can be implemented by any number of types with vastly differing memory footprints and this is not ok for Rust. The solution is to Box your Trait objects, which puts your Trait object on the heap and lets you work with Box like a regular, sized type. (I will experiment a bit with the Sized trait - probably subject of a future blog post, but let me walk down this path first) I e something like this:

fn do_stuff(objects: Vec<Box<dyn Shape>>) {
}

Yup - it compiles. All good then? Not really since here comes the interesting parts. I want to check if one Shape is equal to another Shape, sounds trivial right? We have PartialEq and all should be good. Let's give it a first try

// Will not compile
fn do_stuff(objects: Vec<Box<dyn Shape>>) {
    let obj1 = &objects[0];
    let obj2 = &objects[1];
    if obj1 == obj2 { println!("Equal"); }
}

The helpful compiler error says

error[E0369]: binary operation `==` cannot be applied to type `&std::boxed::Box<dyn Shape>`
  --> src\lib.rs:48:13
   |
48 |     if obj1 == obj2 { println!("Equal"); }
   |        ---- ^^ ---- &std::boxed::Box<dyn Shape>
   |        |
   |        &std::boxed::Box<dyn Shape>
   |
   = note: an implementation of `std::cmp::PartialEq` might be missing for `&std::boxed::Box<dyn Shape>`

So we follow the directions and try to implement PartialEq for Box

impl PartialEq for Box<dyn Shape> {
    fn eq(&self, other: &Box<dyn Shape>) -> bool {
        // Double deref to get to the Shape, ref to avoid E0277 - 
        // "...size for values...unknown at compile time"
        let s = &**self; 
        let o = &**other;
        s == o
    }
}

// Also tweaked the trait definition to ensure that we have PartialEq trait on Shape
trait Shape: PartialEq {
    fn material(&self) -> &Material;
    fn transformation(&self) -> Matrix;
}

Compiler says no

error[E0038]: the trait `Shape` cannot be made into an object
  --> src\lib.rs:47:30
   |
47 |     fn eq(&self, other: &Box<dyn Shape>) -> bool {
   |                              ^^^^^^^^^ the trait `Shape` cannot be made into an object
   |
   = note: the trait cannot use `Self` as a type parameter in the supertraits or where-clauses

Time to think... I cannot get to the real implementation of PartialEq unless I work with the actual type for both self and other. I can delegate the work in the Box...eq function to the trait. This means that when the call is made to the trait method we know what the actual type is for &self (see box_eq below). The final step is figuring out the actual type of "other" and if it is the same as &self, convert other to that type and call Eq.

This took quite a lot of experimenting, reading of Rust docs and Googling. This StackOverflow question was what finally got me on the right track. Putting all the pieces together I ended up with this

#[derive(PartialEq)]
struct Sphere {
    transformation: Matrix,
    material: Material
}

trait Shape: Any {
    fn box_eq(&self, other: &dyn Any) -> bool;
    fn as_any(&self) -> &dyn Any;    
    fn material(&self) -> &Material;
    fn transformation(&self) -> Matrix;
}

impl Shape for Sphere {
    fn as_any(&self) -> &dyn Any {
        self
    }
    fn box_eq(&self, other: &dyn Any) -> bool {
        other.downcast_ref::<Self>().map_or(false, |a| self == a)
    }
    fn material(&self) -> &Material {
        &self.material
    }
    fn transformation(&self) -> Matrix {
        self.transformation
    }
}

impl PartialEq for Box<dyn Shape> {
    fn eq(&self, other: &Box<dyn Shape>) -> bool {
        self.box_eq(other.as_any())
    }
}

fn do_stuff(objects: Vec<Box<dyn Shape>>) {
    let obj1 = &objects[0];
    let obj2 = &objects[1];
    if obj1 == obj2 { println!("Equal"); }
}

The important bits are:

box_eq - delegates to the Shape-implementing type which gives us the actual type of self.
Any - it is the closest thing to reflection there is in Rust. It allows us to do runtime type-casting with downcast_ref. This gives us the actual type and allows call to Eq.
as_any() - gives us "other" as Any in box_eq.

Note that the implementations for as_any and box_eq will be textually identical for any Shape implementation. Could make sense to turn it into a macro.

I hope this text might save you some time if you're faced with a similar problem.

.NET Framework to .NET Core to Docker

Magnus Stråle — Fri, 26 Jul 2019 10:38:47 +0000

This is a "lessons learned" from taking an existing .NET Framework-based REST-ful service and getting it to run under .NET Core in a Docker container.

I was fortunate enough to start out with a service built TDD style, meaning that there was good test coverage and an overall nicely designed application with appropriate abstractions. This means I felt fairly confident that if I could just have all tests green, then I had a working solution.

The service is based on Nancy ("Lightweight, low-ceremony, framework for building HTTP based services on .Net") and using TopShelf ("The simplest ways to get started with winservice development") to run the code as a Windows service. Both are ageing projects and TopShelf is simply not applicable on Linux. The decision was made to continue with Nancy and host it under Kestrel. Even if there has been no official Nancy release that supports .NET Core there is a pre-release that supports .NET Standard 2.0.

.NET Standard vs .NET Core vs "old" .NET Framework is something that can easily confuse you. I won't try to explain what is what here, but refer you to this article instead https://www.infoq.com/news/2017/10/dotnet-core-standard-difference which does a nice job of explaining these concepts.

If you want to get your code running on Linux / Docker the basic principle is for your projects to target .NET Standard whenever possible and .NET Core for those cases where .NET Standard is not sufficient. More on this later on.

Getting started

Since the project files has been significantly changed for .NET Core, primarily describing the exceptions from conventions rather than detailing every little thing that should be included in your project, it made sense to simply throw away all existing csproj files and start from scratch with this template:

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>netstandard2.0</TargetFramework>
    </PropertyGroup>
</Project>

Then you keep adding NuGet and project references until you run out of "[CS0246] The type or namespace name 'Xxx' could not be found (are you missing a using directive or an assembly reference?)". Most packages that I needed were available in .NET Standard versions so it was a fairly painless process (at least compared to what I expected). The two major issues were embedded resources and app settings. Neither of these are really well implemented in old-fashioned .NET Framework and have been significantly reworked (in the case of configuration settings) or simply left unsupported/removed (in the case of Resx files).

Anyway with the csproj files converted and some commented code the whole thing compiled. The really nice part was that the csproj files had significantly shrunk in size. The biggest file of 3756 lines had been reduced to a tiny 38 lines. The main reason for this is conventions and proper wildcard support - this is what killed off 3000+ lines:

<ItemGroup>
  <None Update="TC\TC*\**">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
  <None Update="TCData\**">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
</ItemGroup>

This also makes the csproj files possible to work with as plain text / XML files in the editor rather than just letting them be implicitly updated as a result of interactions in the IDE.

Resources and Resx files

Two of the projects contained embedded resources. This is simply a way to put things that are not code into the assemblies produced by the compiler, such as files, localized text strings, icons etc. In .NET Framework this is handled by the IDE which creates a .resx file with information and auto-generates a Resources class which gives you fully typed access to your resources. This process is mostly magic.

When converting to .NET Core, the Resx file is still there and JetBrains Rider (which I use instead on Visual Studio for the simple reason that it is included in the IntelliJ license that I have) almost makes it work. However this process pulls in old assemblies for reading and parsing the Resx file to figure out how to parse the other embedded resources. After spending a few hours trying to make the service get access to the embedded files (got it working) and still be able to have a nice editing experience from the IDE (got it working, but broke the code) I gave up on Resx and decided to do it by directly accessing the ResourceManager, the class responsible for working with embedded resources. Turns out that it was dead easy, partly because the only type of resource that I needed was files. This code is what I needed instead of trying to struggle with backwards compatibility:

public static class Resources 
{
    private static readonly string _prefix = typeof(Resources).FullName + ".";
    private static readonly Assembly _assembly = typeof(Resources).Assembly;
    public static Stream GetStream(string name)
    {
        return _assembly.GetManifestResourceStream(_prefix + name);
    }
}

Configuration settings

In classic .NET Framework you usually have your app.config or web.config with all the configuration settings for your application. Any custom settings can be put under an section, or you can write your own fully typed config section handler. However under .NET Core the "official" way of doing it is a lot more customizable, you can choose where to pull the configuration settings from (config file, environment variables, command line parameters etc). However I just wanted to get the thing to work with as little work as possible. Looking at the NuGet feeds theer is actually a System.Configuration package that supports .NET Standard 2.0, which wasnt the case originally.

There were some issues with how Owin / Kestrel was configured - it looked like using the new configuration system made for a cleaner interface to Kestrel. Also considering my recent experience with resources so I decidede to switch to the new configuration system. It was almost as easy as the Resource bit.

public static IWebHost BuildWebHost(string[] args)
    {
        var builder = new ConfigurationBuilder()
            .AddJsonFile("appsettings.json")
            .AddCommandLine(args)
            .AddEnvironmentVariables();
        var configuration = builder.Build();
        return WebHost.CreateDefaultBuilder(args)
            .UseConfiguration(configuration)
            .UseUrls($"http://*:{configuration["app:serverPort"]}")
            .UseStartup<Startup>()
            .Build();
    }

Above is actually a sneak peek at the Kestrel configuration in addition to the configuration handling. The code above goes together with this appsettings.json file:

{
  "app": {
    "ApiUrl": "http://127.0.0.1:10010",
    "serverPort": "8096",
    "ProgressEventBusHost": "192.168.32.128",
    "ProgressEventExchangeName": "dev.global.exchange",
    "MaxParallellism": "1",
    "QueueCapacity": "1000",
    "HangSafeGuardTimeSeconds": "300"
  }
}

Hosting in Kestrel and OWIN

Well - you've already seen parts of it in the previous code snippet. As previously mentioned there is a pre-release version of nancy that supports .NET Standard. It has the ominous name Nancy 2.0.0-clinteastwood Most of the old configuration code for Nancy (as per the CustomBootstrapper referenced below) worked untouched. The one thing that was really needed was the Startup class that we referred to in the previous BuildWebHost method:

    class Startup
    {
        private readonly IConfiguration _configuration;

        public Startup(IConfiguration configuration)
        {
            _configuration = configuration;
        }

       public void Configure(IApplicationBuilder app, IHostingEnvironment env)
       {
           Console.WriteLine(env.ContentRootPath);
           Console.WriteLine(Directory.GetCurrentDirectory());

            var appSettings = new AppSettingsProvider(_configuration);
            app.UseOwin(x => x.UseNancy(b =>
            {
               b.Bootstrapper = new CustomBootstrapper(appSettings);
            }));
        }
    }

Dotnet from the command line

Before putting stuff on Docker I made sure that I had the builds and tests working from Windows command line with the dotnet command. Up until now I have been running everything from within Rider. Putting this little script together I had it all compiling and tests green:

dotnet build src/App.sln -c Release
forfiles /p .\src /m *Tests /c "cmd /c dotnet test --no-build -c Release @path"
dotnet publish -c Release -o out -f netcoreapp2.0 src/AppHost/AppHost.csproj

Now for the ultimate test - start the service and see that it responds as expected. Execute the following from the out folder that the solution has been published to:

dotnet AppHost.dll

No luck - "A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in '…AppHost\out\'.
Failed to run as a self-contained app. If this should be a framework-dependent app, add the …AppHost\out\AppHost.runtimeconfig.json file specifying the appropriate framework." At least it's a very good error message, one AppHost.runtimeconfig.json coming up!

{
  "runtimeOptions": {
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "2.0.0"
    }
  }
}

Yes! It runs! Pointing a browser to http://localhost:9922 does indeed produce the expected response.

Putting it in Docker

Since I had done all the hard work I expected this to be a walk in the park. Well not quite… It started out quite well, using my superior Google skills I found something that looked like a good starting point https://docs.docker.com/engine/examples/dotnetcore/
I started out with the Dockerfile in the article and tweaked it slightly to make the service build and run the tests in the container:

FROM microsoft/dotnet:sdk AS build-env
WORKDIR /app/src
COPY . .
RUN dotnet restore
RUN dotnet test -c Release DomainModel.Tests/DomainModel.Tests.csproj
RUN dotnet test -c Release App.Services.Tests/App.Services.Tests.csproj
RUN dotnet test -c Release App.Tests/App.Tests.csproj

The domain model worked like a charm, all green, but the other two projects started throwing massive lists of test failures at me. Turns out that (almost) all problems had a common denominator...

Slash vs backslash

I started to write about PC-DOS and the historical background just to show off my age and experience but decided not to. Read this little piece instead if you're interested in the background https://www.howtogeek.com/181774/why-windows-uses-backslashes-and-everything-else-uses-forward-slashes/

The service makes fairly extensive use of files, persisting data as files serialized with protobuf. This means that the use of backslash as folder separators is present in many parts of the code. As I've already hinted at, this immediately broke almost all tests related to files when running under Docker. Fixing this was easy since Windows does in fact accept the forward slash as folder separator.

When writing .NET code for .NET Standard / .NET Core, always use '/' as folder separator. Even better - use Path.Combine to add folder names together since Path.Combine will use the correct separator depending on your OS. The final part is to not use absolute paths (like C:\temp, even written as C:/temp it makes no sense on Linux).

Time zones and Docker containers

Once I got rid of the slash vs backslash failures I had one more red test to tackle. It basically sent a query via Nancy Browser and expected the returned data to exactly match a JSON string. Unfortunately it did not, it showed this:

Failed   ThenTheJsonBodyIsSerializedAsBefore
Error Message:
   String lengths are both 742. Strings differ at index 178.
  Expected: "..."0001-01-01T00:00:00.0000000+01:00","endTime":"0001-01-01T..."
  But was:  "..."0001-01-01T00:00:00.0000000+00:00","endTime":"0001-01-01T..."
  --------------------------------------------^

The test was written by someone in time zone UTC+1 and since I am in Stockholm (UTC+1) it "worked on my machine". However it seems like most Docker containers available for production workloads are set up for UTC and in many (most?) cases simply don't have time zones configurable since /usr/share/zoneinfo is missing. IMO this makes perfect sense - a docker container should behave the same regardless of where it is deployed and therefore time zone specific settings are simply not relevant.

I do think that the test is a bit shady - these types of "replay" tests are fragile and I would prefer to have verification of the actual data rather than the serialized representation. The one exception to this rule-of-thumb would be if you need to ensure backwards compatibility of persisted data.

Rewriting this test is on my todo-list, but for now I simply tweaked the test verification to ignore time zone differences. Hey, I'm not perfect…

Final dockerization pieces

Getting all the necessary files into a single place for final deployment is done with the "dotnet publish" command, putting it in the out folder. The -f flag to define which framework version to use is required in this case, but the docs seems to imply that this may be optional. I suspect it has to do with the fact that I'm mixing targets for different projects.

The really interesting aspect is that the docker file makes use of multi-stage build. This simply means that I can slap anothe FROM statement in my dockerfile and it will create a new image with the possibility to use files from previous stages in the dockerfile. Note that the first FROM references the microsoft/dotnet:sdk image while the second FROM uses microsoft/dotnet:aspnetcore-runtime, which is a significatly smaller image. Finally we copy the output from the publish command onto our new image and set an entry point. I e the runtime image will only contain the bare minimum to run our app, no other tools or non-runtime build artifacts.

For more info on multi-stage builds see http://recurse.se/2017/09/docker-multistage-builds/

FROM microsoft/dotnet:sdk AS build-env
WORKDIR /app/src
# Copy and restore as distinct layers
COPY . .
RUN dotnet restore
RUN dotnet test -c Release DomainModel.Tests/DomainModel.Tests.csproj
RUN dotnet test -c Release App.Services.Tests/App.Services.Tests.csproj
RUN dotnet test -c Release App.Tests/App.Tests.csproj
RUN dotnet publish -c Release -o out -f netcoreapp2.0 AppHost/AppHost.csproj
# Build runtime image
FROM microsoft/dotnet:aspnetcore-runtime
WORKDIR /app
COPY --from=build-env /app/src/AppHost/out .
ENTRYPOINT ["dotnet", "AppHost.dll"]

Lessons learned

Don't pull in compatibility stuff to save you the work of rewriting a class or two. The end result will be cleaner and more functional (looking at you Resx files and System.Configuration).
Target .NET Standard whenever possible. It allows your code to be used both in old .NET Framework and .NET Core.
Use multistage builds to create small Docker containers.
Don't hard-code paths with '\' - Windows is perfectly happy with '/' in most cases. Even better - use Path.Combine which will use the correct separator character depending on your platform.
Stand on the shoulders of giants. Not everything that I have written here was solved by yours truly, I was fortunate enough to be able to look at another project that had undergone similar treatment by my colleague Martin Hellspong. Check out his blog here http://recurse.se/

Rust - first impressions

Magnus Stråle — Mon, 22 Jul 2019 11:00:41 +0000

I have started playing with Rust and here are some observations from a newbie point-of-view. My reasons for looking into Rust is that it's a modern language and does away with manual memory management without resorting to garbage collection or ref counting. This was a concept that I found very intriguing. How the heck did they solve that?

First of all I need a goal whenever I start to pick up a new language, otherwise I tend to just skip over the difficult parts. In this case I picked up the book "The Ray Tracer Challenge" which starts from zero and shows you how to build a ray tracer (bonus points for doing it TDD-style!). Since I've never done any ray tracing work before it turned out to be quite educational in that aspect as well.

Keep in mind that this is written by a Rust newbie, so please excuse any mistakes/errors in this post (and let me know about them!).

So what about memory management in Rust?

The idea is to have strict rules about memory ownership and lifetime management. Memory is automatically allocated for your data structures and released when your structure goes out of scope. For local variables this is really straight-forward but for data that gets passed around it does get a bit more complicated. You can either let a method borrow your data (ownership still remains within the calling code) or you can move your data (ownership is passed to the called method, meaning that you cannot reference that data in the calling code after the method call).

There is also the concept of copying the data, which allows you to pass a separate copy of the data and still keeping the original intact. This is done by implementing a special Copy Trait (similar to interfaces in other languages).

It does take quite a bit of trial-and-error in the beginning to get the borrow / move semantics correct, randomly adding '&' and 'mut' here and there. The good news is that the compiler does give quite helpful messages and there is a lot of helpful documentation online, as well as a very helpful community.

Anything besides memory management?

Speed! Both the tooling as well as your compiled code runs really fast. I haven't done any serious benchmarking but the code/compile/test cycle is at least as quick as anything else I've worked with.
Documentation. I did mention helpful online documentation and I must say that Rust does shine in this department. Yes - the language is still under active development so there are changes happening that might trip you up, but the docs have saved me every time. Not only language and API documentation but also a very nice book (aka 'The Book') teaching you the basics on Rust.
Immutability by default. If you want to have mutable data you have to explicitly state so, otherwise your data is safe and immutable.

Is it all good in Rust-land?

There are of course things that annoys me...

No support for overloading. I like to have the possibility to overload a method that takes somewhat different set of parameters. Not possible in Rust - you have to use another name. This is particularly annoying for the new method (the name is a convention for constructor-like methods) where I really want to have a couple of versions for convenience.
Strict separation between data and code. You use a struct to define the data for your objects and a separate impl section where you put your methods. Not saying that I completely dislike this, but it's different to what I'm used to.
Operator overloading. There is a lot of repetition (types!) for defining operator overloads. Can we at least get rid of the "type Output = ..."?

What are my conclusions so far?

The systems programming concern clearly shines thru in a number of design decisions for Rust which makes it feel very "close to the metal"/C-like when coding. But surprisingly enough this is coupled with a number of high-level constructs (lambdas, functional programming support etc) which enables you to write very expressive and concise code. The safety aspects are also very nice, it even looks like Microsoft is considering using Rust for this very reason.

Rust is definitely a language that I will invest more time in, especially since I will dig into WebAssembly in the future (which Rust fully supports). Will it be my go-to language for quick hacks/side projects? Not sure - C# is still what I'm most comfortable with.