Exploring the Microsoft Developer Control Plane at the heart of the new .NET Aspire

As I write this, less than a week after the end of .NET Conf 2023 and the release of .NET 8, .NET Aspire is in preview version (8.0.0-preview.1.23557.2). Therefore, it's possible that some aspects may have changed by the time you read this article.

During the .NET Conf 2023, Microsoft announced .NET Aspire, a new .NET workload designed to ease the development of applications and microservices in a cloud-native context. Having personally experienced difficulties with developing and orchestrating multiple microservices in a local environment, I was pleasantly surprised by this announcement.

If you haven't yet seen the deep-dive video by Glenn Condron and David Fowler about .NET Aspire, I invite you to immediately stop reading this article and watch it. It will better equip you to understand the rest of this discussion.

This isn't just another high-level introductory article on .NET Aspire. I'm sure many others have already done that, and done it better than I could. What I want to delve into here concerns the inner workings of .NET Aspire, beyond its open-source code.

Being very familiar with the source code of the Tye project — the experiment that inspired Microsoft's development teams to create .NET Aspire — one of my first reactions was to try to understand the internals of .NET Aspire. Specifically, I was interested in how it orchestrates the resources developers declare in their .NET Aspire host. How does .NET Aspire compile and launch other projects? How does it manage the lifecycle of arbitrary executables? How does it interact with the Docker engine to start containers? How does service discovery work?

In the next few minutes, you will discover that .NET Aspire, as it was presented, is just the tip of the iceberg. Indeed, .NET Aspire is built on top of an undocumented orchestrator, also developed by Microsoft. This is the Microsoft Developer Control Plane, otherwise referred to by the acronym DCP. In short, DCP is a sort of miniature Kubernetes, which can be controlled with tools such as kubectl or the official C# client for Kubernetes.

A different approach from project Tye

In the Tye project, the execution of services, containers, and other executables declared in the tye.yaml YAML configuration was orchestrated by C# code, as can be seen in the source code of the ProcessRunner class and the DockerRunner.

Tye's application model, being aware of all the resources to orchestrate, thus knows all the URLs, ports, and connection strings of these resources. It can then inject them via environment variables, for example.

The application model of .NET Aspire is similar. Instead of using YAML code, the developer declares their resources with C# code.

var builder = DistributedApplication.CreateBuilder(args);

var backend = builder.AddProject<Projects.MyBackend>("backend");
var frontend = builder.AddProject<Projects.MyFrontend>("frontend")
    .WithReference(backend);

builder.Build().Run();

The developer then has two ways to execute their host application:

With a simple dotnet run, .NET Aspire will:

• Build and consolidate its application model based on the resources declared by the developer.
• Start an instance of the Microsoft Developer Control Plane (DCP).
• Ask DCP to start the resources declared in the application model.
• Listen to DCP events and receive logs and traces from the started resources.
• Start and open the .NET Aspire dashboard.

This main mode of operation is what developers will use to start their local microservices environment. This is particularly what interests us in this article.

The second mode of operation allows .NET Aspire to generate a JSON manifest representing all the resources declared in the developer's C# code.

dotnet run -- --publisher manifest --output-path manifest.json

This manifest can then be consumed by other tools such as azd to provision the Azure infrastructure necessary for running the application in a cloud environment.

You could also write your own tool that consumes this manifest and outputs Terraform code according to your needs.

The Microsoft Developer Control Plane, an unknown and undocumented component

Start a .NET Aspire host project and list the processes running on your machine. You should see the following processes (here, on Windows):

• dcp.exe
• dcpctrl.exe
• dcpd.exe

These processes are part of the .NET Aspire workload. They can be found in the folder <dotnet-sdk-dir/packs/Aspire.Hosting.Orchestration.<RID>/8.0.0-preview.1.23557.2/tools/. This directory also includes an EULA file outlining the terms of use for "Microsoft Developer Control Plane". I have searched extensively online, particularly on GitHub, for information regarding DCP, but I haven't found anything that clarifies its purpose and operation. The DCP's code does not seem to be open-source, at least not currently. According to this GitHub issue, the URL for the DCP Git repository is https://github.com/microsoft/usvc-apiserver, but it is private.

Let's engage in some debugging within the publicly available code of .NET Aspire to figure out how it engages with DCP.

Starting DCP through .NET Aspire

At the beginning of the .NET Aspire host startup, the DcpHostService class launches an instance of the dcp.exe process.

ProcessSpec dcpProcessSpec = new ProcessSpec(dcpExePath)
{
    WorkingDirectory = Directory.GetCurrentDirectory(),
    Arguments = $"start-apiserver --monitor {Environment.ProcessId} --detach --kubeconfig \"{locations.DcpKubeconfigPath}\"",
    OnOutputData = Console.Out.Write,
    OnErrorData = Console.Error.Write,
};

There is a mention of kubeconfig. Initially, I thought that Kubernetes was being used for resource orchestration. With a breakpoint in the decompiled code, here is what the content of this kubeconfig file looks like:

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://[::1]:60803
  name: apiserver_cluster
contexts:
- context:
    cluster: apiserver_cluster
    user: apiserver_user
  name: apiserver
current-context: apiserver
kind: Config
preferences: {}
users:
- name: apiserver_user
  user:
    token: owamluvvhtdf

My first reflex was to try to list all the pods with kubectl pointing to this kubeconfig file:

kubectl --kubeconfig "C:\Users\simmo\AppData\Local\Temp\aspire.jbnsxb1r.4u2\kubeconfig" get pods

But it did not work. Indeed, the server does not seem to recognize the resource type pods:

error: the server doesn't have a resource type "pods"

I then tried to list the supported resource types:

kubectl --kubeconfig "C:\Users\simmo\AppData\Local\Temp\aspire.jbnsxb1r.4u2\kubeconfig" api-resources

The result was much more interesting:

NAME                    SHORTNAMES   APIVERSION                            NAMESPACED   KIND
containers              ctr          usvc-dev.developer.microsoft.com/v1   false        Container
containervolumes        ctrvol       usvc-dev.developer.microsoft.com/v1   false        ContainerVolume
endpoints               end          usvc-dev.developer.microsoft.com/v1   false        Endpoint
executablereplicasets   exers        usvc-dev.developer.microsoft.com/v1   false        ExecutableReplicaSet
executables             exe          usvc-dev.developer.microsoft.com/v1   false        Executable
services                svc          usvc-dev.developer.microsoft.com/v1   false        Service

It appears that this DCP server, which resembles Kubernetes, supports resource types specific to .NET Aspire, especially Executable and Container.

Having started my .NET Aspire host with the starter template, I slightly modified my setup to have a frontend server, two backend server replicas, and a Redis container:

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedisContainer("cache");

var apiservice = builder.AddProject<Projects.Aspire01_ApiService>("apiservice")
    .WithReplicas(2);

builder.AddProject<Projects.Aspire01_Web>("webfrontend")
    .WithReference(cache)
    .WithReference(apiservice);

builder.Build().Run();

I then tried to describe the Container resource associated with my Redis container:

> kubectl --kubeconfig "C:\Users\simmo\AppData\Local\Temp\aspire.jbnsxb1r.4u2\kubeconfig" get containers

NAME    CREATED AT
cache   2023-11-19T04:38:57Z

> kubectl --kubeconfig "C:\Users\simmo\AppData\Local\Temp\aspire.jbnsxb1r.4u2\kubeconfig" describe containers cache

Name:         cache
Namespace:
Labels:       <none>
Annotations:  service-producer: [{"serviceName":"cache","address":null,"port":6379}]
API Version:  usvc-dev.developer.microsoft.com/v1
Kind:         Container
Metadata:
  Creation Timestamp:  2023-11-19T04:38:57Z
  Finalizers:
    usvc-dev.developer.microsoft.com/container-reconciler
  Managed Fields:
    API Version:  usvc-dev.developer.microsoft.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"usvc-dev.developer.microsoft.com/container-reconciler":
    Manager:      dcpctrl.exe
    Operation:    Update
    Time:         2023-11-19T04:38:57Z
    API Version:  usvc-dev.developer.microsoft.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:service-producer:
      f:spec:
        f:image:
        f:ports:
        f:restartPolicy:
    Manager:         unknown
    Operation:       Update
    Time:            2023-11-19T04:38:57Z
  Resource Version:  35
  UID:               936080b7-e85c-4b1b-9366-3ec489a5895b
Spec:
  Image:  redis:latest
  Ports:
    Container Port:  6379
    Protocol:        TCP
  Restart Policy:    no
Status:
  Container Id:       40f382e164839b6bb5cbc3e7acf25ff182141316eeb64cc1ee70cbf448581c12
  Finish Timestamp:   <nil>
  Startup Timestamp:  2023-11-19T04:38:57Z
  State:              Running
Events:               <none>

My Redis cache is indeed orchestrated by DCP. I can also see the Docker instance that was started in Docker Desktop:

DCP, a resource orchestrator controlled by .NET Aspire

Subsequently, I went through almost the entire codebase of .NET Aspire to better understand the nature of the project. I was still struggling to differentiate between the open-source .NET Aspire project and the responsibilities of the Microsoft Developer Control Plane.

With the help of Rider, I was able to debug the .NET Aspire code and see how it interacts with DCP. Once DCP is started and the application model (the list of resources and their dependencies) is consolidated, .NET Aspire uses the .NET Kubernetes client to execute "Kubernetes-ish" commands on DCP. The C# class in .NET Aspire responsible for interacting with DCP is ApplicationExecutor.

Here is the content of one of the HTTP requests sent by the .NET Aspire host to DCP to start my Redis container:

{
  "spec": {
    "image": "redis:latest",
    "ports": [
      {
        "containerPort": 6379,
        "protocol": "TCP"
      }
    ],
    "env": [],
    "restartPolicy": "no"
  },
  "metadata": {
    "annotations": {
      "service-producer": "[{\"serviceName\":\"cache\",\"address\":null,\"port\":6379}]"
    },
    "name": "cache",
    "namespace": ""
  },
  "apiVersion": "usvc-dev.developer.microsoft.com/v1",
  "kind": "Container"
}

All these clues lead to the belief that DCP is the central element of .NET Aspire for local development. It is DCP that knows how to:

• Interact with Docker Desktop and start containers.
• Interact with the operating system and start arbitrary processes.
• Expose the ports of the started resources to the host machine.
• Manage the lifecycle of the started resources.
• Handle multiple replicas of the same resource.

What a pity that at this stage there is no documentation on DCP. Running dcp.exe --help does not yield anything very useful:

DCP is a developer tool for running multi-service applications.

        It integrates your code, emulators and containers to give you a development environment
        with minimum remote dependencies and maximum ease of use.

Usage:
  dcp [command]

Available Commands:
  generate-file   Generate file from a template.
  help            Help about any command
  up              Runs an application
  version         Prints version information

Flags:
  -h, --help              help for dcp
  -v, --verbosity level   Logging verbosity level (e.g. -v=debug). Can be one of 'debug', 'info', or 'error', or any positive integer corresponding to increasing levels of debug verbosity. Levels more than 6 are rarely used in practice.

Use "dcp [command] --help" for more information about a command.

The command start-apiserver, which is used by .NET Aspire to start DCP, is not found there. One can try to invoke it manually with .\dcp.exe start-apiserver --help:

Starts the API server and controllers, but does not attempt to run any application

Usage:
  dcp start-apiserver [flags]

Flags:
      --detach                   If present, instructs DCP to fork itself as a detached process.
  -h, --help                     help for start-apiserver
      --kubeconfig string        Paths to a kubeconfig. Only required if out-of-cluster.
  -m, --monitor int              If present, tells DCP to monitor a given process ID (PID) and gracefully shutdown if the monitored process exits for any reason. (default -1)
  -i, --monitor-interval uint8   If present, specifies the time in seconds between checks for the monitor PID.
      --port int32               Use a specific port when scaffolding the Kubeconfig file. If not specified, a random port will be used.
  -r, --root-dir string          If present, tells DCP to use specific directory as the application root directory. Defaults to current working directory.

Global Flags:
  -v, --verbosity level   Logging verbosity level (e.g. -v=debug). Can be one of 'debug', 'info', or 'error', or any positive integer corresponding to increasing levels of debug verbosity. Levels more than 6 are rarely used in practice.

This provides us with a bit more information. At this stage, we could imagine being able to interact directly with DCP. For that, we would need to know the specification of the resources supported by DCP. Fortunately, the open-source code of .NET Aspire already contains part of this information. In the folder src/Aspire.Hosting/Dcp/Model/, one can find the specifications of all the resources supported by DCP.

It seems there is still a long way to go for .NET Aspire and the Microsoft Developer Control Plane. For instance, the specification of a container describes that one could declare the command to execute within the container as well as the arguments. However, the public C# API of .NET Aspire does not yet allow these properties to be defined. Thus, it is impossible to start an instance of Redis or any other container with custom arguments.

Conclusion

.NET Aspire is a project which, in my opinion, has the potential to become an indispensable tool for facilitating the development of cloud-native applications, even if those applications are not developed with .NET. The Aspire dashboard and the presence of observability and instrumentation as first-class citizens will undoubtedly allow many developers to become aware of what is happening in their applications.

However, the public C# API of .NET Aspire does not yet allow today to exploit the full potential of the Microsoft Developer Control Plane. It is also not possible to stop or restart applications once the .NET Aspire host has started. If one of the resources crashes, the entire application must be restarted. The number of issues on GitHub has noticeably increased since the announcement of .NET Aspire at .NET Conf 2023. The Microsoft development teams will have a lot of work in the coming weeks to decide and prioritize the next features to implement.

For me, the big surprise was discovering the existence of the Microsoft Developer Control Plane. This program, probably coded in Go, seems much more complex than the C# code of .NET Aspire. I hope it will become open-source one day.