Docker : Build complex Solution with multi-contexts and Bake

In the .NET world, Solutions serve as a structural mechanism to organize multiple projects, such as applications and libraries, under a single umbrella. Typically, a Solution includes all the projects that make up a full application, streamlining the process of sharing DLLs. Instead of publishing them to a NuGet feed, they are directly referenced in the .csproj files within the Solution.

Enabling Docker support for a project prompts VisualStudio to generate a Dockerfile in that project. However, the image must be built from the Solution directory, not the individual project directory. This approach ensures that all necessary code, including that which resides outside the project directory within the Solution, is accessible during the build.

docker build -t repo/image:1.0.0 -f Project/Dockerfile .
# or, build from the project folder
# docker build -t repo/image:1.0.0 ..

It's fine if you have only 2 projects on the solution (The domain, and the API for example).

What if you have a big solution and you want to build one project that depends only on one of these projects ?

Imagine building an IoT application. You need to develop two modules to retrieve data from different data sources, with one module dedicated to each source. These modules will convert the data into a format that the rest of your application can understand and then transmit it to a queue system. On the receiving end of the queue, there are two modules: one for aggregating and storing the data, and another for generating alerts and dispatching them to a notification channel. Additionally, there is a fifth module that exposes the aggregated data through an API.

For the data ingestion modules, you establish an external library that encompasses the domain of each data source. To facilitate data exchange between your ingestion and processing modules, you create another library that defines the object structures. Lastly, you develop a separate library for your domain, which is utilized by your API.

We have 5 applications, and 4 libraries

flowchart TB
  IngestionA(IngestionA)
  IngestionA.Domain{{IngestionA.Domain}}
  IngestionB(IngestionB)
  IngestionB.Domain{{IngestionB.Domain}}
  Exchange{{Exchange}}
  Data(Data)
  Alerts(Alerts)
  API(API)
  Domain{{Domain}}

  IngestionA --> IngestionA.Domain
  IngestionA --> Exchange
  IngestionB --> IngestionB.Domain
  IngestionB --> Exchange
  Data --> Exchange
  Alerts --> Exchange
  API --> Domain

Let's build our 5 applications

docker build -t repo/ingestionA:1.0.0 -f IngestionA/Dockerfile .
docker build -t repo/ingestionB:1.0.0 -f IngestionB/Dockerfile .
docker build -t repo/data:1.0.0 -f Data/Dockerfile .
docker build -t repo/alerts:1.0.0 -f Alerts/Dockerfile .
docker build -t repo/api:1.0.0 -f API/Dockerfile .

Docker build context

When executing the docker build command, Docker uses the specified folder as the build context and transfers it to the Docker daemon. It omits files and folders specified in the .dockerignore file but includes all other files (the complete codebase).

For building our API, Docker transfers the entire solution to the build context, as denoted by the . in our command, despite the fact that we might only require the API and the domain code.

✏️ Docker, with BuildKit is smart enough to lazy send files to context with only required files. The purpose of this still apply, but not for context optimizations. It's apply for codebase that is not in the same folder (context) as your default one

Docker build mutli-contexts

To avoid sending the entire codebase, we can limit the context to only the essential parts of the code. However, it's crucial to define the context for the dependent code.

This can be achieved with the multi-contexts build feature, which necessitates an update to the Dockerfile to accommodate this new approach.

This is our full Dockerfile base for the IngestionA module

# file: Dockerfile
FROM mcr.microsoft.com/dotnet/runtime:8.0 AS base
USER app
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY IngestionA.Domain/IngestionA.Domain.csproj ./IngestionA.Domain/
COPY Exchange/Exchange.csproj ./Exchange/
COPY IngestionA/IngestionA.csproj ./IngestionA/
RUN dotnet restore ./IngestionA/IngestionA.csproj
COPY . .
WORKDIR /src/IngestionA
RUN dotnet build ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/build --no-restore

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false --no-restore

FROM base AS final
EXPOSE 5000
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "IngestionA.dll"]

Let's focus on the build stage to use multi-contexts

# file: Dockerfile
...

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY --from=domain IngestionA.Domain.csproj ./IngestionA.Domain/
COPY --from=exchange Exchange.csproj ./Exchange/
COPY IngestionA.csproj ./IngestionA/
RUN dotnet restore ./IngestionA/IngestionA.csproj
COPY --from=domain . ./IngestionA.Domain/
COPY --from=exchange . ./Exchange/
COPY . ./IngestionA/
WORKDIR /src/IngestionA
RUN dotnet build ./IngestionA.csproj -c $BUILD_CONFIGURATION -o /app/build --no-restore

...

To build our image, we need to define each context

# from the IngestionA project folder
docker build -t repo/ingestionA:1.0.0 \
  --build-context domain=../IngestionA.Domain/ \
  --build-context exchange=../Exchange/ \
  -f Dockerfile .

⚠️ Each context must define it's own .dockerignore

Now, to build all 5 applications, we must write quite long commands

docker build -t repo/api:1.0.0 \
  --build-context domain=./Domain
  API/
docker build -t repo/ingestionA:1.0.0 \
  --build-context domain=./IngestionA.Domain/ \
  --build-context exchange=./Exchange/ \
  IngestionA/
docker build -t repo/ingestionB:1.0.0 \
  --build-context domain=./IngestionB.Domain/ \
  --build-context exchange=./Exchange/ \
  IngestionB/
docker build -t repo/data:1.0.0 \
  --build-context exchange=./Exchange/ \
  Data/
docker build -t repo/alerts:1.0.0 \
  --build-context exchange=./Exchange/ \
  Alerts/

Improve developer experience with Bake

Docker buildx bake is a new way to build our images while drawing parallelization and build's orchestration.

We need to add one configuration file named docker-bake.hcl and to change the way we build images.

// file: docker-bake.hcl

variable "TAG" {
  default = "latest"
}

target "api" {
  context = ./API/
  contexts = {
    domain = ./Domain/
  }
  tags = [
    "repo/api:${TAG}"
  ]
}

target "ingestionA" {
  context = ./IngestionA/
  contexts = {
    domain = ./IngestionA.Domain/
    exchange = ./Exchange
  }
  tags = [
    "repo/ingestionA:${TAG}"
  ]
}

target "ingestionB" {
  context = ./IngestionB/
  contexts = {
    domain = ./IngestionB.Domain/
    exchange = ./Exchange
  }
  tags = [
    "repo/ingestionB:${TAG}"
  ]
}

target "data" {
  context = ./Data/
  contexts = {
    exchange = ./Exchange
  }
  tags = [
    "repo/data:${TAG}"
  ]
}

target "alerts" {
  context = ./Alerts/
  contexts = {
    exchange = ./Exchange
  }
  tags = [
    "repo/alerts:${TAG}"
  ]
}

group "default" {
  targets = [
    "data",
    "alerts",
    "api",
    "ingestionA",
    "ingestionB"
  ]
}

group "ingestions" {
  targets = [
    "ingestionA",
    "ingestionB"
  ]
}

group "data-pipeline" {
  targets = [
    "ingestionA",
    "ingestionB",
    "data",
    "alerts"
  ]
}

To build our 5 applications, we have to run a single command

docker buildx bake

By default, Bake will search for a group default (or target if group isn't defined). In this case, the group "default" contains all our targets

If we want to build only the ingestion and processing's modules, we must target data-pipeline group

docker buildx bake data-pipeline

To build only the API, we must call bake with the target api

docker buildx bake api

All the images we build will be tagged with latest, we can change this behavior in different ways.

1. Override the TAG value using ENV

TAG=1.0.0 docker buildx bake api

1. Override the tag's image using --set argument

docker buildx bake api --set api.tags=api:1.0.0

1. Define a variable for each target and override them in a parameter file

// file: docker-bake.hcl

variable "API_TAG" = {

  default = "latest"

}

target "api" { 

 context = ./API/ 

 contexts = {

   domain = ./Domain/

 }

 tags = [ 

   "repo/api:${API_TAG}" 

 ] 

}

...

// file: bake-parameters.hcl

API_TAG = "1.0.0"

docker buildx bake api -f docker-bake.hcl -f bake-parameters.hcl

Conclusion

Docker's multi-context builds are particularly beneficial for large solutions with numerous projects, as they help to reduce build context's size.
While they introduce complexity by requiring the definition of each context, this is mitigated when combined with Bake.

Bake streamlines the building process of the entire solution, allowing for the simultaneous construction of all images with a single command.