Forward and Backward Compatibility in .NET Core

I only recently discovered load contexts in .NET Core having been using app domains up to now. The load context, I learned, is designed to solve the problem of isolating code using different versions of the same assembly. This was exciting news to me! If you have ever worked on large applications that load a lot of 3rd party code you invariably run into this problem. Most of the time this problem can be solved by binding to the highest version of the assembly used. Most of the time however is not "All of the time". The moment a higher version of an assembly contains breaking changes then this strategy goes out the window. The larger the application you are working on, the more likely you will be outside the window looking in. I'm old enough to remember "DLL Hell". If you are of a certain age you remember that assemblies were supposed to solve DLL Hell. What we have all learned since is that Assembly Hell is ten times hotter than DLL Hell.

So I decided to write some code to check out this capability. Alas! I was disappointed to discover that load contexts solve backward compatibility but not forward compatibility. Let me explain.

Let's say we have an interface…

namespace AssemblyAInterface
{
    public interface IClassAv1
    {
        public string OriginalMethod();
    }
}

With this implementation

namespace NamespaceA
{
    public class ClassA : IClassAv1
    {
        public string OriginalMethod() { return "this is the original method in v1"; }
    }
}

We've shipped this code some time back and now want to make some improvements. Best practice is to not change the interface but create a new one.

namespace AssemblyAInterface
{
    public interface IClassAv2
    {
        public string OriginalMethod();
        public string NewMethod();
    }
}

We update our implementation to support both the old and the new interface…

namespace NamespaceA
{
    public class ClassA : IClassAv1, IClassAv2
    {
        public string OriginalMethod() { return "this is the original method in v2"; }
        public string NewMethod() { return "this is a new method in v2"; }
    }
}

So now old code can use the v1 interface while new code can access the v2 interface. Voila!

Well…

Let's write some code to load both the new and the old implementations. This was possible using app domains but very difficult and as far as I know no one ever used app domains to solve this specific versioning problem. However with load contexts, the solution is straightforward…

namespace LoadContextBench
{
    class Program
    {
        static void Main(string[] args)
        {
            // Our versioned assemblies are in separate directories

            string apathv1 = Path.Combine(Directory.GetCurrentDirectory(), @"..\v1\net5.0\AssemblyA.dll");
            string apathv2 = Path.Combine(Directory.GetCurrentDirectory(), @"..\v2\net5.0\AssemblyA.dll");

            // Separate load contexts for the different versions.

            AssemblyLoadContext alcv1 = new AssemblyLoadContext("v1");
            AssemblyLoadContext alcv2 = new AssemblyLoadContext("v2");

            // Load v1 and v2 of the same assembly

            Assembly av1 = alcv1.LoadFromAssemblyPath(apathv1);
            Assembly av2 = alcv2.LoadFromAssemblyPath(apathv2);

            // Look for our types in the various contexts.  Doesn't exist in default context.

            Type t0 = Type.GetType("NamespaceA.ClassA"); // null in default context.
            Type tv1 = av1.GetType("NamespaceA.ClassA");
            Type tv2 = av2.GetType("NamespaceA.ClassA");

            // Create v1 and v2 objects.  Reference interfaces.  Call methods.

            try
            {
                var dv1 = Activator.CreateInstance(tv1);
                var dv2 = Activator.CreateInstance(tv2);
                IClassAv1 cv1 = (IClassAv1)dv1;
                IClassAv1 cv1a = (IClassAv1)dv2;
                IClassAv2 cv2 = (IClassAv2)dv2;
                Console.WriteLine(cv1.OriginalMethod());
                Console.WriteLine(cv1a.OriginalMethod());
                Console.WriteLine(cv2.NewMethod());
            }
            catch(Exception ex)
            {
                Console.WriteLine("Exception: " + ex.Message);
            }
        }
    }
}

This results in the following output…

this is the original method in v1
this is the original method in v2
this is a new method in v2

This demonstrates backward compatibility: new code calling old code. It also demonstrates executing both old and new versions of the same code simultaneously which is very special.

Note that I had to reference the new interface code in order to reference the v2 interface. Now let's examine forward compatibility. To demonstrate forward compatibility we need to show old code calling new code. So in the main program we reference the v1 interface instead of the v2 interface. Some code changes need to be made since the v2 interface does not exist in the old code.

                var dv1 = Activator.CreateInstance(tv1);
                var dv2 = Activator.CreateInstance(tv2);
                IClassAv1 cv1 = (IClassAv1)dv1;
                IClassAv1 cv1a = (IClassAv1)dv2;
//                IClassAv2 cv2 = (IClassAv2)dv2; // v2 only
                Console.WriteLine(cv1.OriginalMethod());
                Console.WriteLine(cv1a.OriginalMethod());
//                Console.WriteLine(cv2.NewMethod()); // v2 only 

This doesn't work. This statement returns null…

Type tv2 = av2.GetType("NamespaceA.ClassA");

This surprised me greatly I expected the type to be resolved. After all the v2 interface assembly is in the v2 directory along with the v2 implementation of the interface. Looking closer it seems that the interface type must be resolved against the interface in the default load context and while v1 resolves, v2 does not. Forward compatibility is simply not supported.

I could not accept this result. Surely there was a way to call v2 from v1. I decided to implement a custom load context, the idea being to explicitly load the dependencies.

namespace LoadContextBench
{
    public class MyLoadContext : AssemblyLoadContext
    {
        private AssemblyDependencyResolver _resolver;
        public MyLoadContext(string folder)
        {
            _resolver = new AssemblyDependencyResolver(folder);
        }
        protected override Assembly Load(AssemblyName assemblyName)
        {
            string assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName);
            if (assemblyPath != null)
            {
                return LoadFromAssemblyPath(assemblyPath);
            }

            return null;
        }
    }
}

The custom loader forces the loaded assembly to load dependencies from the folder in which the assembly resides

This custom loader is referenced like so…

            // AssemblyLoadContext alcv1 = new AssemblyLoadContext("v1");
            // AssemblyLoadContext alcv2 = new AssemblyLoadContext("v2");
            AssemblyLoadContext alcv1 = new MyLoadContext(apathv1);
            AssemblyLoadContext alcv2 = new MyLoadContext(apathv2);

So here was another surprise. The forward compatibility code got a lot further. The types were found and instantiated. But both casts to the v1 interface failed.

                IClassAv1 cv1 = (IClassAv1)dv1;
                IClassAv1 cv1a = (IClassAv1)dv2;

It seems that now, not only can old code not call new code, but old code cannot even call old code anymore! This was truly shocking. The v1 interface assembly loaded in both the default and the v1 context are identical…identical meaning they are exactly the same assembly. One is loaded into the default context and one is loaded into the v1 context. Yet the interface cannot be cast from the v1 to the default contexts. This seems very wrong. The only way it seems to be able to cast interfaces is when the exact same assembly is shared between load contexts.

Switching the old code back to the new code by referencing the v2 interface in the main program and still using the custom load context yields a similar problem. This time the new code can call neither the old code nor the new code.

Summary and Discussion

What I have illustrated here with examples is how load contexts support backward compatibility but fail completely to support forward compatibility. I spent quite a bit of time trying to get forward compatibility to work….after all the overarching promise of managed code was an end to the brittleness inherent in unmanaged code. What I have discovered to my great disappointment is that brittleness is still with us in managed code. Assembly hell is with us forevermore.

My search for a solution to forward compatibility in .NET Core came to end when I discovered this line in https://docs.microsoft.com/en-us/dotnet/core/compatibility/categories:

Maintaining forward compatibility is not a goal of .NET Core.

Really? Why not?

Why Forward Compatibility is Necessary

But isn't backward compatibility all we need? We have a long experience with backward compatibility and expect it in every bit of software we use. Our typical experience with compatibility is not from code to code, but from code to data.

So for example whenever we update a desktop app to a new version we expect all the files created by the old version to be read by the new version. This is backward compatibility and no application would survive long without supporting it.

Forward combability is never an issue as long as we are an island: that is we read only files we have created and we update apps only we use. This was good enough until about 1995. Once we entered a networked world where we needed to share files among a large set of colleagues running different versions of the same software. Like a convoy that moves only as fast as its slowest boat, no one who needed to share documents could upgrade their applications until the last colleague (who was invariably both the most backward and the most likely to be able to cause trouble for everyone) updated theirs. As a result everyone's software was 5 years out of date.

So of course you are thinking that this is the problem that SaaS solves. Everyone is forced to use the latest software and so forward compatibility is no longer an issue. Indeed this solves the problem for the end user, but the general need for forward compatibility is not eliminated…it is simply moved to a different constituency. Which explains why your SaaS software stack is a pit of forward and backward compatibility horrors. From this observation we might derive a general principle:

No problem in software is ever solved…it simply becomes the responsibility of people who are paid more and care less.

Why Forward Compatibility is Hard

The reason that forward compatibility is rarely supported is that it is hard. For backward compatibility to work all you need to do is make sure that all the old code and data you created in the past is still working. Old code and data is something you know completely.

Forward compatibility means that all the code and data you write in the future will work with code you are writing today. The code and data of the future is solving problems you haven't encountered yet and have no idea how to solve today. The solutions you put in place today to solve forward compatibility might be totally inadequate when it comes time to address the problems which need to be solved in the future. Indeed this happens quite often.

Signoff

I hope this excursion into forward and backward compatibility and the associated rants have been useful and entertaining. All the code used in this post can be found at

https://github.com/bobrundle/LoadContextBench

I invite comments and critiques. Perhaps here is something I am missing here. Perhaps there is a magic .deps.json file that will solve the forward compatibility problem in the same way there was a way to do forward compatibility in the .NET Framework using an app.config file with assembly redirects that essentially defeated the versioning system entirely. But you have to convince me that this is really better.

Sure I can have forward compatibility by instantiating the class in the future code base and using GetMethod() and Invoke() to find and call methods. Is this really what we want? Is this what you would call elegant?

These are the enduring questions.