The project so far makes it possible to scan photos in a directory, run them through various vision models and store the details in a database. Using Aspire instead of Docker Compose for local development as I would usually do has been fun so far.
Aspire 9.0 has handy new features, that are interesting for the next stage of the project. This post will summarise them with some examples.
As the next stage in the project is evaluating the performance of the models I am using, at this point I needed to make Jupyter Notebooks easily accessible inside my codebase and development environment. With .Net Aspire 9.0 this becomes a convenient process.
Getting Ready for Next Steps
Given we have several options from open source when it comes to computer vision models go generate photo summaries, we need to be able to evaluate the results from these models to be able to choose one that suits our domain.
One workflow to do this effectively is using Jupyter Notebooks where we can retrieve our results from the database and then compare with results obtained from commercial models.
Introducing Jupyter Notebooks that runs on a remote host to our Project means the following would be important:
- Containers are running on remote host so we need to be able to include the notebooks in version control
- However docker volumes are on a remote host, no easy way to copy them
- Also we need container to container connection Jupyter Server to MongoDB both of which are running on a remote server and Jupyter Server needs to be able to speak to MongoDb.
As we will see below, .NET Aspire 9.0 takes care of these.
Here is the list of features in .Net Aspire 9.0 that are relevant to this project and will be covered in this post
- Tooling
- No longer relying on workloads: Now we can set up .Net Aspire using packages and project templates.
- Templates can also be installed as following:
dotnet new install Aspire.ProjectTemplates::9.0.0
- Dashboard and UX
- Managing Resource Lifecycles: Start, Stop, Restart from the dashboard.
- Browser Telemetry Support.
- App Host (Orchestration)
- Waiting for dependencies.
- Resource health Checks
- Persistent Containers
- Resource commands
- Container networking
- One from .Net 9.0
- Enabling DI registration of metrics using IMeterFactory
Browser Telemetry Support
Earlier on, I was curious how to integrate traces from the front end and see the distributed traces. .Net Aspire 9.0 brings an out of the box way for this as below.
It is important to remember that Open Telemetry client instrumentation on the browser is experimental.
- Define
DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URLenvironment variable for Apphost launch settings. '"DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URL": "http://localhost:16175"' - I still needed to inject
DOTNET_DASHBOARD_OTLP_ENDPOINT_URLto the .Net applications. - Front end required the HTTP endpoint as well as the following environment variable:
"OTEL_EXPORTER_OTLP_PROTOCOL","http/protobuf"
With these in place, I was able to follow Microsoft examples to make it work on a Stencil Js application.
...
//https://www.honeycomb.io/blog/opentelemetry-browser-instrumentation
//https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-xml-http-request
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-user-interaction
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-long-task
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-long-task
export default async () => {
const otlpOptions = { omitted };
const attributes = { omitted }
provider.addSpanProcessor(new SimpleSpanProcessor(new OTLPTraceExporter(otlpOptions)));
provider.register({contextManager: new StackContextManager()});
registerInstrumentations({
instrumentations: [
getWebAutoInstrumentations({
'@opentelemetry/instrumentation-xml-http-request': {
clearTimingResources: true,
}
}),
new LongTaskInstrumentation({
observerCallback: (span, longtaskEvent) => {
span.setAttribute('location.pathname', window.location.pathname)
}
}),
new FetchInstrumentation({
propagateTraceHeaderCorsUrls: [new RegExp('\\/api\\/*')],
ignoreUrls: [new RegExp('\\/tile\\/*')],
})],
});
...
};
With the changes above and the existing setup in our backend, we can see the end to end traces below. The use case below is for selecting a model and then seeing a request to extract summaries from all 50 images in the db. We can see the durations of db calls, calls to inference endpoints as well as transport and our backend components as they all been triggered by the ui.
Resource Health Checks
If we would like to specify resource dependencies to control startup-process, it is important to be able to define what health check meant to various components.
- If no health checks defined, a resource is considered healthy if it is in running state
- If the Resource exposes an http health check, we can register with one call
.WithHttpHealthCheck("/health") - Sometimes external resources do not provide health checks suitable for us, we can also define, register and use our health checks. This is the method that will be discussed here.
Checking if Nominatim Resource is healthy:
public class NominatimHealthCheck : IHealthCheck
{
... ignored
public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
CancellationToken cancellationToken = new CancellationToken())
{
var ready = await IsServerReady(cancellationToken);
return ready
? HealthCheckResult.Healthy()
: HealthCheckResult.Unhealthy();
}
private async Task<bool> IsServerReady(CancellationToken cancellationToken = default)
{
const string searchUrl = "/search.php?q=avenue%20pasteur";
// check if we have success result or not. Code omitted.
}
}
// we also need to register this as below:
builder.Services.AddHealthChecks()
.AddTypeActivatedCheck<NominatimHealthCheck>("nominatim-healthcheck",..);
var nominatimResourceBuilder = builder.AddResource(nominatimResource)
.WithHealthCheck("nominatim-healthcheck")
....);
And once all registered, we can also see the health in the dashboard:
Defining Resource Dependencies
Once we have defined our health checks, we can declare our dependencies so that our applications will not start before services they might depend on at start up.
Prior to .NET Aspire 9.0, it was possible to achieve this following an example provided by David Fowler in Issue #921 in Aspire repository as linked below.
So now, once all the additional code deleted, all we need is to use .WaitFor(resource) from the framework as below:
var apiService = builder.AddProject<Projects.PhotoSearch_API>("apiservice")
.WithReference(ollamaContainer)
.WithReference(mongodb)
.WithReference(messaging)
.WithReference(openai)
.WaitFor(ollamaContainer)
.WaitFor(mongodb)
.WaitFor(messaging);
This ensures all dependencies spin up first and become healthy and then the application will start. This also helps in cases where the containers need to download data on first run, which might take several minutes.
Persistent Containers
In this project we have some containers that take time to lad and get ready such as OSM Map Tile Server, Nominatim container for reverse geocoding and Ollama is starting the first time as it needs to download the model.
So if we make code changes, containers would stop and then we would need to wait for all dependencies again.
This is another area where Aspire 9.0 comes to the rescue with a single method call as below:
var nominatimResourceBuilder = builder.AddResource(nominatimResource)
.WithLifetime(ContainerLifetime.Persistent)
... other calls omitted;
With this in place, we can start debugging stop and start again lightning fast without having to wait for the containers.
Resource Commands
Resource commands allow the developers register commands that can be accessed from the Aspire Dashboard against a resource.
As I have added Jupyter Notebooks container to the project this weekend, command have helped sole one problem when running the Jupyter Server on a remote Docker host: How do we manage the notebook files? Ideally we manage them in the same repository.
With a download and upload command, we can upload the notebook from our local drive where our git repository is and then we can download the model when modified inside the Jupyter Notebook container.
Downloading the notebook in regular intervals will be one of the next steps.
public static IResourceBuilder<ContainerResource> WithUploadNoteBookCommand(
this IResourceBuilder<ContainerResource> builder, string jupyterToken, string jupyterUrl)
{
builder.WithCommand(
name: "upload-notebook",
displayName: "Upload Notebook",
executeCommand: context => OnUploadNotebookCommandAsync(builder, context, jupyterToken, jupyterUrl),
updateState: OnUpdateResourceState,
iconName: "ArrowUpload",
iconVariant: IconVariant.Filled);
return builder;
}
private static async Task<ExecuteCommandResult> OnUploadNotebookCommandAsync(... params omitted)
{
var notebookData = read from disk
// setup omitted
var result = await httpclient.SendAsync(uploadNoteBookHttpRequestMessage);
// handle the response
}
// And command is registered as below:
var jupyter = builder.AddContainer(name, image)
.WithUploadNoteBookCommand(token, "http://localhost:8888")
.WithDownloadNoteBookCommand(token, "http://localhost:8888")
... other calls omitted;
From a high level the process looks as below:
Although the above illustrates the case when we are using a remote Docker daemon, the experience is the same if we were using a local Docker daemon.
Container Networking
This is another feature that made life easy. With workloads running on other machines on local network, so far container to container access have not been that important.
With the Jupyter Notebooks container, as our data source is a MongoDb container on the same docker host, it was necessary to be able to access the database.
This is another change where improvements to Aspire have been transparent to the developer and gained the benefits without extra work as below:
There is not much to do besides ensuring the connection string are injected as below:
builder.AddJupyter("jupyter", !string.IsNullOrWhiteSpace(dockerHost), "secret",portMappings["JupyterPort"].PublicPort)
.WithReference(mongodb);
Once that is done, we can access the connection string in python as following: connection_string = os.environ.get('ConnectionStrings__photo-search')
Custom Metrics via Dependency Injection
This is not aspire but as a colleague pointed last week on elf the features in .Net 9.0 was out of the box ability to use Dependency Injection (DI) for registering consuming metrics.
We can achieve this by injecting IMeterFactory to a utility class where we manage our meters.
Here is an example from this project for measuring number of models summarised as well as durations for each image:
using System.Diagnostics.Metrics;
namespace PhotoSearch.ServiceDefaults;
public class ConsoleMetrics
{
private readonly Counter<int> _photosSummariesCounter;
private readonly Histogram<double> _photosSummaryHistogram;
public ConsoleMetrics(IMeterFactory meterFactory)
{
var meter = meterFactory.Create("PhotoSummary.Worker");
_photosSummariesCounter = meter.CreateCounter<int>("photosummary.summary.generated");
_photosSummaryHistogram = meter.CreateHistogram<double>("photosummary.summary.durationseconds");
}
public void PhotoSummarised(string model, int quantity)
{
_photosSummariesCounter.Add(quantity,
new KeyValuePair<string, object?>("photosummary.summary.model", model));
}
public void PhotoSummaryTiming(string model,string photo, double durationSeconds)
{
_photosSummaryHistogram.Record(durationSeconds,
new KeyValuePair<string, object?>("photosummary.summary.model", model),
new KeyValuePair<string, object?>("photosummary.summary.photo", photo));
}
}
// Register:
builder.Services.AddSingleton<ConsoleMetrics>();
// Ensure our meter is added when configuring OpenTelemetry:
builder.Services.AddOpenTelemetry()
.WithMetrics(metrics =>
{
metrics.AddMeter(InstrumentationOptions.MeterName);
// other calls
})
// Inject and use:
builder.Services.AddSingleton<ConsoleMetrics>();
As we see below, total 70 images summarised using two models so far.
Finally in the following screen capture, we can see the timings of each photo summary request. While generally they are 5 - 10 seconds, there are some outliers taking about 5 minutes. We can dig into the metrics, find out which photo / model combination causes the spikes at that stage using the traces and then understand if it is a random GPU issue or a consistent delay under some circumstances.
If we investigate the traces, the results are interesting.
To summarise a photo, currently there are 3 calls to Ollama container:
- Get the overall summary
- Using the context, get a list of objects
- Again using the context so far, get a list of possible categories
And the traces show us, we spent 4 minute 27 seconds waiting for categories to be generated.
This is worth investigation and as we have a notebook, it is also easier to experiment with the same prompt / image combinations.
Conclusion
.Net Aspire 9.0 changes make Aspire a great alternative to using Docker Compose. By adopting standard container technologies, managing a local development environment using Aspire is worth having a go.
It has also been great witnessing how Aspire 9.0 features have been discussed in Github issues and ended up ready to consume for the masses with the new release. The transparency and the speed of improvement makes it a great choice for development.
Now that upgrade is out of the way, next step will be generating the summaries using Open API models and then comparing / ranking each summary generated locally against it and evaluating results. I have also come across a paper that proposes a more systemic approach to evaluation using state of the art models and will be experimenting with that too.
Top comments (0)