DEV Community: Tommaso Stocchi

Aspire Multilanguage with .NET Aspire Community Toolkit

Tommaso Stocchi — Mon, 25 Nov 2024 12:10:26 +0000

Not long ago, I posted about how to use .NET Aspire even when our distributed application involves different languages. In that article, I used the ability of .NET Aspire to support different languages natively or via containers to build a distributed calculator. In this article, I will show how this has improved thanks to the .NET Aspire Community Toolkit.

What is the .NET Aspire Community Toolkit?

It's an open-source repository where third parties can contribute their own integrations and extensions to the .NET Aspire ecosystem. Writing such integrations is easy, as demonstrated by Aaron Powell during dotnetconf.

The .NET Aspire Community Toolkit already offers all the integrations we need to move the distributed calculator away from local containers and run executables locally.

How to use the .NET Aspire Community Toolkit

To use the Golang integration, simply add the appropriate NuGet package to your project:

dotnet add package CommunityToolkit.Aspire.Hosting.Golang

Next, use the following code to configure the Golang app:

var add = builder.AddGolangApp("addapp", "../../go-adder")
    .WithHttpEndpoint(env: "APP_PORT")
    .PublishAsDockerFile();

This is a significant improvement over our previous setup:

var add = (builder.ExecutionContext.IsPublishMode
    ? builder.AddContainer("addapp", "acrt6xtihl2b3uxe.azurecr.io/addapp")
    : builder.AddContainer("addapp", "addapp"))
        .WithHttpEndpoint(targetPort: 6000, env: "APP_PORT", name: "http")
        .WithOtlpExporter()
        .WithEnvironment("OTEL_SERVICE_NAME", "addapp")
        .PublishAsContainer();

Additionally, we can leverage the NodeJs Extensions to automatically install the necessary node packages for our NodeJs divider app and frontend:

var divide = builder.AddNodeApp(name: "divideapp", scriptPath: "app.js", workingDirectory: "../../node-divider")
    .WithNpmPackageInstallation()
    // other config

//...

builder.AddViteApp(name: "calculator-front-end", workingDirectory: "../../react-calculator", packageManager: "yarn")
    .WithYarnPackageInstallation()
    // other config

The WithNpmPackageInstallation and WithYarnPackageInstallation methods will automatically install the necessary packages for the NodeJs and React apps, respectively.

More than just other languages

The .NET Aspire Community Toolkit is not just about languages. It also offers integrations with other services, such as Data API Builder. In this blog, I demonstrated how to use Data API Builder to create a chat-with-your-database application. Integrating DAB with .NET Aspire resulted in this:

var dabService = builder.AddContainer("dab", "mcr.microsoft.com/azure-databases/data-api-builder")
    .WithHttpEndpoint(targetPort: 5000, name: "http")
    .WithBindMount(@"D:\src\distributed-dab-sk\aw-data-api-builder\dab-config.json", "/App/dab-bm0/dab-config.json", true)
    .WithArgs("--ConfigFileName", "./dab-bm0/dab-config.json")
    .WithReference(sql)
    .WithOtlpExporter()
    .PublishAsContainer();

Using the Data API Builder integration, we can simplify it to:

var dabService = builder.AddDataAPIBuilder("dab", @"..\aw-data-api-builder\dab-config.json");

Useful Links

.NET Aspire and Dapr. Overlap or complementary?

Tommaso Stocchi — Sun, 29 Sep 2024 10:53:23 +0000

Although .NET Aspire and Dapr are two different projects, they have some overlap in their goals. Both projects aim to simplify the development of distributed applications by providing a set of building blocks that developers can use. However, there are some key differences between the two projects that make them complementary rather than competing.

Let's take a closer look at the goals of each project and how they can work together to help developers build better distributed applications.

.NET Aspire

.NET Aspire aims to simplify the development of distributed applications, including architectural components such as cache, databases, and queues. These components - now called integrations - are designed to be easy to use and configure, so developers can focus on building their applications rather than worrying about the underlying infrastructure.

On top of that, Aspire offers the ability to inject the connection strings of both the integrations and the different services where needed. Let's take a look at the basic .NET Aspire template.
Let's run the following command:

dotnet new aspire-starter --use-redis-cache

In the <Project_Name.AppHost> project that has been created for us, we can find the following code:

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedis("cache");

var apiService = builder.AddProject<Projects.DistributedDAB_ApiService>("apiservice");

builder.AddProject<Projects.DistributedDAB_Web>("webfrontend")
    .WithExternalHttpEndpoints()
    .WithReference(cache)
    .WithReference(apiService);

builder.Build().Run();

As you can see, we are adding a Redis cache and a project called ApiService. We are also adding a project called WebFrontend that has external HTTP endpoints and references the cache and the ApiService. .NET Aspire is actually injecting the connection strings of the cache and the ApiService. In .NET, we can use the connection string like this:

builder.Services.AddHttpClient<WeatherApiClient>(client =>
    {
        client.BaseAddress = new("https+http://apiservice");
    });

In other coding languages we would have read the connection string as an environment variable:

services__apiservice__http__0

On top of that, Aspire offers observability via the Aspire Dashboard leveraging OpenTelemetry. Using the dashboard, developers can monitor the performance of their applications, read logs and traces, and troubleshoot issues in real-time. As a cherry on top, we can leverage the Azure Developer CLI to easily deploy our applications to Azure.

.NET Aspire actually follows the deployment of distributed applications from the IDE to the cloud.

Dapr

Dapr is a portable, event-driven runtime that makes it easy for developers to build resilient, microservices-based applications. Dapr provides a set of building blocks that developers can use to build their applications, including state management, pub/sub messaging, and service invocation. These building blocks are designed to be language-agnostic, so developers can use them with any programming language.

Dapr uses the sidecar pattern. Basically, our microservice doesn't talk to the Redis cache or the API service directly. Instead, it talks to the Dapr sidecar, which in turn talks to the cache or the API service. This allows Dapr to provide a consistent interface for all the building blocks, regardless of the underlying implementation.

What's awesome about Dapr is the ability to define a building block — say a pub/sub message — in a YAML file and then use it in any programming language. This is possible because Dapr is a sidecar that runs alongside the application, providing the building blocks as a service. These YAML files can be swapped out for different environments, making it easy to switch from a Redis Queue to an Azure Service Bus without changing any code!!!!.

Take a look at this example of an API that subscribes to a topic, and guess which pub/sub component I'm using:

builder.Services.AddDaprClient();

...

app.UseCloudEvents();
app.UseEndpoints(endpoints => endpoints.MapSubscribeHandler());

app.MapPost("/", [Topic("queue", "topic")] async (DaprClient daprClient) => 
{
    ...

The answer is: it depends. Locally, it is using a Redis pub/sub running in a container, but when it's deployed to the cloud, it's using Azure Service Bus. The only thing that changes is the YAML file that defines the pub/sub component.

Moreover, Dapr offers the ability to implement service-to-service invocation via HTTP. If we wanted to call the previous WeatherApiClient from the ApiService, we would do it like this:

// Using generic http client
var client = new HttpClient();
var response = await client.GetAsync("http://localhost:<DAPR_SIDECAR_PORT>/v1.0/invoke/apiservice/method/weatherforecast");

// using Dapr client (SDK specific)
var response = await _daprClient.InvokeMethodAsync<WeatherForecast[]>("apiservice", "weatherforecast");

Overlap or complementary?

Dapr can be integrated into .NET Aspire. We can define a Dapr building block in a YAML file and then use it in our .NET Aspire application.

Here's an example of how to add Dapr components to a .NET Aspire application, but for the full picture please refer to the official documentation.

var secretstore = builder.AddDaprComponent("secretstore", "secretstores.azure.keyvault", new DaprComponentOptions { LocalPath = "..\\.dapr\\components\\secretstore.yml"});
var pubsub = builder.AddDaprPubSub("pubsub", new DaprComponentOptions { LocalPath = ".\\.dapr\\components\\pubsub.yml"});

Cloud component -> complementary

Aspire can define resources, inject their connection strings where needed, and also create those resources when deploying to the cloud via the Azure Developer CLI. Dapr can reference resources already created, defining endpoints and configurations in YAML files that can easily be swapped.

In situations where we are testing our application locally and don't want to leverage cloud resources, we could easily reference the local Redis cache container as a pub/sub handler and then switch to Azure Service Bus when deploying to the cloud. When we do this, we can still leverage the Aspire integration with the Azure Developer CLI to create all of our resources during deployment.

Service to service invocation -> overlap (?)

As I mentioned, .NET Aspire will provide us with the correct connection string for whichever service we need to invoke. Dapr can do the same thing, but it will do it via the sidecar. This is something we need to keep in mind when making our decision.

Here's a schema of how Dapr implements service-to-service invocation:

Service A makes an HTTP or gRPC call targeting Service B. The call goes to the local Dapr sidecar.
Dapr discovers Service B’s location using the name resolution component, which is running on the given hosting platform.
Dapr forwards the message to Service B’s Dapr sidecar. Note: All calls between Dapr sidecars go over gRPC for performance. Only calls between services and Dapr sidecars can be either HTTP or gRPC.
Service B’s Dapr sidecar forwards the request to the specified endpoint (or method) on Service B. Service B then runs its business logic code.
Service B sends a response to Service A. The response goes to Service B’s sidecar.
Dapr forwards the response to Service A’s Dapr sidecar.
Service A receives the response.

These extra steps can take a toll on performance, but it is platform agnostic. It doesn't matter which platform we are running our distributed application on: service-to-service invocation won't need to take that into consideration. Also, all these HTTP/gRPC requests are performed on localhost.

On the other hand, if we leverage the .NET Aspire and Azure Developer CLI integration, we can deploy our application to Azure Container Apps and always have the correct connection string injected.
When running the <Project_Name.AppHost> locally, all these endpoints will be http://localhost:<PORT>. When deploying to Azure Container Apps, all these endpoints will be those of the correct Container App instances. Our code won't change because we are referencing an injected environment variable and we don't need to know the actual value of the endpoints.

In conclusion, .NET Aspire and Dapr are two different projects that can be used together to build better distributed applications. .NET Aspire streamlines the process of going from local development to the cloud, providing all the tools needed for building modern and resilient distributed applications. Dapr can help make our applications flexible and as cloud-agnostic as possible. By using both projects together, developers can take advantage of the strengths of each project to build better distributed applications.

.NET Aspire for Data Chatting

Tommaso Stocchi — Thu, 19 Sep 2024 13:26:53 +0000

.NET Aspire is a great way to orchestrate our distributed applications. It helps from the local development process right to the cloud deployment. And it works with different coding languages as well! Let's use it to build something a little bit more complex than a distributed application as we've seen the last time. Let's build a chat application! Specifically, I want to be able to chat with my database.

The idea

With the growth of AI, we can see more and more chatbots around. They are used in many different scenarios, from customer service to personal assistants. But what if we could chat with our database? We could ask it to retrieve some data, update some records, or even create new ones. This is the idea behind this project. We will build a chat application that allows us to interact with our database using natural language.

In order to achieve this, we will use the following technologies:

I've already discussed .NET Aspire in a previous article, so I won't go into details here. Semantic Kernel is a lightweight, open-source development kit that lets you easily build AI agents and integrate the latest AI models into your C#, Python, or Java codebase.

Data API Builder is a tool that allows you to create RESTful and GraphQL APIs for your database without writing any code.

As I mentioned, I want to be able to talk to my database using natural language. To do this, we will use Semantic Kernel to process the text input and generate a query that can be executed by DAB. The response from DAB will be sent back to the user, completing the loop. Of course, we will need to leverage Semantic Kernel's capability to use plugins and the GraphQL API provided by DAB.

GraphQL is a query language for your API that allows you to request only the data you need, therefor Semantic Kernel will be able to generate the perfect query based on the user input.

Data API Builder

Let's start with the core of our application - Data API Builder. DAB is cross-platform, open-source, and independent of language, technology, and frameworks. It's free, with no premium tier, and can run in any cloud. Best of all, it doesn't require any coding! We can either use the dab-cli or write the json file ourselves to have the API up and running in no time.

For this project, I have created a database from the AdventureWorks template. Using the dab-cli, I can simply run few commands to initialize the configuration file

dab init --database-type "mssql" --connection-string "@env('ConnectionStrings__sql')"

dab add "Customer" --source "SalesLT.Customer" --permissions "anonymous:*"
dab add "ProductModel" --source "SalesLT.ProductModel" --permissions "anonymous:*"
...

and some others to define the relationships between the tables

dab update Address --relationship customerAddresses --target.entity CustomerAddress --cardinality many --relationship.fields "AddressID:AddressID"
dab update Customer --relationship customerAddresses --target.entity CustomerAddress --cardinality many --relationship.fields "CustomerID:CustomerID"
...

This will generate a json file that looks something like this


  "$schema": "https://github.com/Azure/data-api-builder/releases/download/v1.1.7/dab.draft.schema.json",
  "data-source": {
    "database-type": "mssql",
    "connection-string": "@env('ConnectionStrings__sqldb')",
    "options": {
      "set-session-context": false
    }
  },
  "runtime": {
    "graphql": {
      "enabled": true,
      "path": "/graphql",
      "allow-introspection": true
    },
  },
  "entities": {
    "Customer": {
      "source": {
        "object": "SalesLT.Customer",
        "type": "table"
      },
      "graphql": {
        "enabled": true,
        "type": {
          "singular": "Customer",
          "plural": "Customers"
        }
      },
      "permissions": [
        {
          "role": "anonymous",
          "actions": [
            {
              "action": "*"
            }
          ]
        }
      ],
      "relationships": {
        "customerAddresses": {
          "cardinality": "many",
          "target.entity": "CustomerAddress",
          "source.fields": [
            "CustomerID"
          ],
          "target.fields": [
            "CustomerID"
          ]
        },
        ...

Using DAB we can define everything - authentication, authorization, data access roles, row level security, and more.

Semantic Kernel

Because we want to be able to chat with our database, we need to process the text input and generate a query that can be executed by DAB. This is where Semantic Kernel comes in. We can add an OpenAPI plugin to Semantic Kernel so that it can invoke an API. Earlier, we defined a GraphQL API using DAB. We can describe the GraphQL API in an OpenAPI file in yaml, so that it can be consumed by Semantic Kernel.

openapi: 3.0.1
info:
  title: Data API builder - Graphql Endpoint
  version: 0.12.0
servers:
  - url: http://localhost:5000
paths:
  /graphql:
    post:
      description: a graphql endpoint.
      operationId: graphql_post
      ...

Now the only thing left is importing the plugin into Semantic Kernel

await kernel.ImportPluginFromOpenApiAsync(
    pluginName: "GQLPlugin",
    filePath: Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"Plugins/DAB_GraphQL/openapi.yml"),
    executionParameters: new OpenApiFunctionExecutionParameters()
    {
        ServerUrlOverride = new Uri(Environment.GetEnvironmentVariable("services__dab__http__0")!),
        EnableDynamicPayload = true,
        IgnoreNonCompliantErrors = true,
        EnablePayloadNamespacing = true,
    });

and of course a good system prompt to make sure it will use the plugin correctly.

The most important thing is that the Kernel doesn't try to guess the GraphQL schema, rather uses introspection to understand it:

...
### on the graphql_post tool:
- the grapghql_post tool provides access to data useful to answer user question.
- the graphql_post tool is a graphql endpoint.
- **Do not** guess schema type name, relationships or data types. 
- You **must** use introspection to understand the whole schema before planning the query you will execute.
- **Remember** names, including field names and type names, are case sensitive in grapql.   
- when composing the graphql request you can use **only** the following parameters: filter, orderBy, first and after .
- when composing the graphql request you can use **only** the following operators: eq,neq,gt,gte,lt,lte,isnull,contains,notcontains,startsWith,endsWith,and,or .
- mutation operations are not allowed.
...

Here's the result of a simple request:

As you can see, I let the agent outline its reasoning and give a somewhat formatted output. Of course, for business scenarios, we will need to cut out the reasoning and show the results in a more user-friendly way.

Putting it all together

I've started mentioning that this whole project rely on .NET Aspire. It's the orchestrator that will make sure the correct services are running and the correct endpoints are being injected were needed.

Of course, Data API Builder can be ran in a container, so our App Host becomes pretty simple:

var builder = DistributedApplication.CreateBuilder(args);

var sql = builder.AddConnectionString("sqldb");

var dabService = builder.AddContainer("dab", "mcr.microsoft.com/azure-databases/data-api-builder")
    .WithHttpEndpoint(targetPort: 5000, name: "http")
    .WithBindMount(@"D:\src\distributed-dab-sk\aw-data-api-builder\dab-config.json", "/App/dab-bm0/dab-config.json", true)
    .WithArgs("--ConfigFileName", "./dab-bm0/dab-config.json")
    .WithReference(sql)
    .WithOtlpExporter()
    .PublishAsContainer();
var dabServiceEndpoint = dabService.GetEndpoint("http");

builder.AddProject<Projects.DataAPIBuilder_AI_Web>("webfrontend")
    .WithExternalHttpEndpoints()
    .WithReference(dabServiceEndpoint);

builder.Build().Run();

Notice how the connection string for the database is being injected into the DAB container and the connection string for the DAB container is being injected into the web frontend. This variable will override the one in the above mentioned OpenAPI plugin file.

Having this, we can leverage the .NET Aspire manifest to deploy the whole application to the cloud, hosting it on Azure Container Apps.
We now have a chatbot for our database, that requires no coding when new tables are added and that runs in a serverless and scalable environment!

You can find the full code along with some more tecnical details in the GitHub repository.

.NET Aspire Multilanguage

Tommaso Stocchi — Sun, 08 Sep 2024 12:40:54 +0000

In this article I want to explore how we can leverage a tool such as .NET Aspire to improve the way we build and deploy distributed applications, even when we work with languages and frameworks other than .NET.

What is .NET Aspire

.NET Aspire is an opinionated, cloud ready stack designed to improve the experience of building observable, production ready, distributed applications. It is delivered through a collection of NuGet packages that handle specific cloud-native concerns.

.NET Aspire is designed to help you with:

Orchestration: .NET Aspire provides features for running and connecting multi-project applications and their dependencies for local development environments.
Integrations: .NET Aspire integrations are NuGet packages for commonly used services, such as Redis or Postgres, with standardized interfaces ensuring they connect consistently and seamlessly with your app.
Tooling: .NET Aspire comes with project templates and tooling experiences for Visual Studio, Visual Studio Code, and the dotnet CLI to help you create and interact with .NET Aspire projects.

The core of .NET Aspire can be found in the AppHost project that's created when you start working with Aspire from the .NET template. The AppHost project defines the components of our distributed application, from services - such as backend or fronted - to resources - such as databases or caches.
This project can generate a manifest describing from an infrastructure perspective our application. It can therefor be interpreted by the Azure Developer CLI to deploy the application to Azure, without the need to write any Infrastructure as Code.

Work with different languages

Of course when we work with distributed applications and microservices we might incur in a scenario in which different teams like to write code in different languages. And of course .NET Aspire belongs to the .NET ecosystem. Nonetheless, it is design to be able to integrate with different languages and frameworks.

Distributed Calculator

Let's consider a simple example. We have a distributed calculator that is composed of different services: a frontend service and a backend for each operation.

fronted: react
subtraction: dotnet
addition: go
multiplication: python
division: nodejs

We also have a redis cache to store the state of the calculator.

In this scenario we can use .NET Aspire to build the distributed calculator, leveraging all the tool and integrations that come with it. Aspire offers native support for certain languages, but we can also extend it to support other languages using containers.

At the time of writing, .NET Aspire supports the following languages:

Here's how we can configure all the backend services in the AppHost project:

Golang

Golang isn't natively supported, so we will add it as a container. Note that we can decide to use different images for the scenarios in which we are running the AppHost for publishing the manifest or for local development.

// Configure Adder in Go
var add = (builder.ExecutionContext.IsPublishMode
    ? builder.AddContainer("addapp", "acrt6xtihl2b3uxe.azurecr.io/addapp")
    : builder.AddContainer("addapp", "addapp"))
        .WithHttpEndpoint(targetPort: 6000, env: "APP_PORT", name: "http")
        .WithOtlpExporter()
        .WithEnvironment("OTEL_SERVICE_NAME", "addapp")
        .PublishAsContainer();
var addEnpoint = add.GetEndpoint("http");

Python

Python is natively supported, so we can use the AddPythonProject method to configure the multiplier service. Please follow this tutorial to correctly configure the Python project.

// Configure Multiplier in Python
var multiply = builder.AddPythonProject("multiplyapp", "../../python-multiplier", "app.py")
    .WithHttpEndpoint(targetPort: 5001, env: "APP_PORT", name: "http")
    .WithEnvironment("OTEL_SERVICE_NAME", "multiplyapp")
    .PublishAsDockerFile();

NodeJS

NodeJS is natively supported, so we can use the AddNodeApp method to configure the divider service.

// Configure Divider in NodeJS
var divide = builder.AddNodeApp(name: "divideapp", scriptPath: "app.js", workingDirectory: "../../node-divider")
    .WithHttpEndpoint(targetPort: 4000, env: "APP_PORT", name: "http")
    .WithEnvironment("OTEL_SERVICE_NAME", "divideapp")
    .PublishAsDockerFile();

.NET

No surprise here, .NET Aspire natively supports .NET, so we can use the AddProject method to configure the subtractor service.

// Configure Subtractor in .NET
var subtract = builder.AddProject<Projects.dotnet_subtractor>("subtractapp")
    .WithReference(insights)
    .WithEnvironment("OTEL_SERVICE_NAME", "subtractapp");

Redis

The redis cache can be easily configured using the AddRedis method, or in this scenario using Dapr via the AddDaprStateStore method.

// Configure Dapr State Store
var stateStore = builder.AddDaprStateStore("statestore");

Dapr is not the focus of this article, but it is worth mentioning that .NET Aspire can be used in conjunction with Dapr to build distributed applications. For further information regarding Dapr, please reference the official documentation.

React

Lastly, we can configure the frontend service using the AddNpmApp method.

// Configure Frontend in React
builder.AddNpmApp(name: "calculator-front-end", workingDirectory: "../../react-calculator")
    .WithDaprSidecar(new DaprSidecarOptions
    {
        AppPort = 3000,
        AppProtocol = "http",
        DaprHttpPort = 3500
    })
    .WithEnvironment("DAPR_HTTP_PORT", "3500")
    .WithReference(addEnpoint)
    .WithReference(multiply)
    .WithReference(divide)
    .WithReference(subtract)
    .WithReference(stateStore)
    .WithReference(insights)
    .WithHttpEndpoint(targetPort: 3000, env: "PORT")
    .WithExternalHttpEndpoints()
    .WithEnvironment("OTEL_SERVICE_NAME", "calculator-front-end")
    .PublishAsDockerFile();

Since we are referencing all the previously configured services, we can easily connect them to the frontend service. When we need to invoke the adder from the frontend, we can easily do so by using the environment variable that has been injected by Aspire:

app.post('/calculate/add', async (req, res) => {
  try {
      const serviceUrl = process.env.services__addapp__http__0;

      const appResponse = await axios.post(`${serviceUrl}/add`, req.body);

      // Return expected string result to client
      return res.send(`${appResponse.data}`); 
  } catch (err) {
      console.log(err);
  }
});

Further considerations

The entire application can be deployed to Azure using the Azure Developer CLI. The CLI will read the manifest generated by the AppHost project and deploy the application to Azure, creating all the necessary resources. To learn how to integrate Aspire with the Azure Developer CLI, please reference the official tutorial.

All the code for the distributed calculator can be found in the Aspire Multilanguage repository.

GitHub Copilot vs ChatGPT

Tommaso Stocchi — Sat, 25 May 2024 06:30:00 +0000

In this article I would like to analyze thedifferences between two tools that seem to overlap: GitHub Copilot and ChatGPT. What are the fundamental differences between the two? Which one to choose? And do you really have to choose?

ChatGPTChatGPT

Let's start by analyzing ChatGPT. It is a web portal where you can start a chat with a Large Language Model (LLM). There are several ChatGPT tiers:

With the free tier, we have a fairly minimal experience that allows us to interact with the GPT-3.5 model.

Things start to get interesting from the Plus tier, which already offers us the possibility to interact with the GPT-4 model. This also gives us access to the web. Access to the web is important, because when we ask a question to the model, it is able to answer based on "native knowledge" derived from its training. Let's say that the model has been trained with data from the web up to 2021, and we ask it who the Prime Minister of the United Kingdom is, it would answer Boris Johnson (the Prime Minister in office at that time). If we gave the same model access to the web, it would be able to give us the exact answer: Rishi Sunak (Prime Minister in office at the time of writing this article).

The third team, in addition to interacting with other models such as DALL-E, adds the possibility that the data sent through the requests will not be used to retrain the model.

GitHub Copilot

GitHub Copilot is a fine-tuning of the GPT-4 model for code. Fine-tuning refers to the ability to train a model by specializing it for a specific scenario, in this case working on code. The basic capabilities are therefore the same as GPT-4, which is already highly capable of working on code, with a specific specialization on this feature.

Just like ChatGPT, GitHub Copilot also offers different pricing tiers.

It can be observed how for the tiers Individual and Business the difference in features is mostly related to "Management and policies". The Individual tier is aimed at individual users, while the Business tier targets more corporate scenarios, where centralized user and policy management provides a significant advantage for tool administrators.

I will dedicate a separate paragraph to the Enterprise tier later in this article.

Terms and Conditions

Another fundamental difference between the two tools can be found in the Terms & Conditions (T&C). GitHub Copilot's terms and conditions of use ensure that the underlying model will not be retrained using user-inputted data. Essentially, even in the Individual tier, when GitHub Copilot analyzes your code to provide you with answers and suggestions, it does not use the portions of code analyzed to retrain its algorithm, thereby preserving intellectual property.

Regarding ChatGPT, this applies starting from the Team tier.

From the perspective of Copilot in Edge, however, Commercial Data Protection is guaranteed for the types of accounts listed in the first paragraph of this link, and only when accessing with the company account and not the personal account.

Due to data protection concerns, for professional use, I would never recommend a tier that does not offer data protection functionalities, for this purpose from now on we will consider a comparison between the different tiers of GitHub Copilot and ChatGPT Team.

IDE Integration

The main advantage of GitHub Copilot is the integration with the IDE: it is in fact born as a tool to suggest code in real time to the developer as he or she writes code. It infers from the context of what has already been written and what is being written to suggest entire portions of code in a pro-active way.

Over time, GitHub Copilot has evolved by adding several features, in addition to the Code Completion we just talked about: Chat and Smart Actions.

We can imagine the Chat feature as an implementation of a ChatGPT-like scenario. However, being a model specialized in code, the field is therefore restricted: asking GitHub Copilot who the Prime Minister of the United Kingdom is, it will respond:

If ChatGPT can answer both code and general questions, what's the advantage of using GitHub Copilot over ChatGPT?

Keeping in mind that this question compares a feature of a larger product (GitHub Copilot is not just its chat) to a complete product (ChatGPT is a chat service), the strengths of GitHub Copilot lie in its integration with the IDE.

Without leaving the Visual Studio Code screen (or Visual Studio or JetBrains), we can select portions of code and ask direct questions to our pair programming assistant. From the backend perspective, the question posed to the model will therefore contain:

our context: the selected code
our question, for example "explain to me what this portion of code does"
the system prompt: the system prompt is a basic prompt that has been defined and written on the backend and surrounds the question we have asked. They are the basic instructions. In the simplest cases, we can think of the system prompt as a series of basic indications such as "You are a very helpful virtual assistant in helping. You always try to explain things in a complete but never verbose way and you are able to schematize complex concepts in order to make their understanding easier". This is a remarkably simple system prompt, GitHub Copilot's will clearly be more complex and will contain instructions such as "Only respond to questions related to the programming world", which generates responses like the one in the screenshot above.

This system prompt is one of the strengths of GitHub Copilot: the code generated by the tool is not directly passed on to the end user, but is filtered and double-checked in order to avoid scenarios of prompt-injection (a concept similar to SQL injection, but which applies to prompt engineering scenarios).

Even more important than the system prompt is GitHub Copilot's ability to access the extended context of our IDE. The context is formed by the files open in tabs and the files present in the open folder. In fact, there is the possibility, through the @workspace keyword, to ask broader questions about the entire repository that is open.

In the two screenshots above, we can see how GitHub Copilot Chat is able to analyze the entire structure of the folders, without having to specifically select portions of code, and provide me with the exact answers. In the second case, it is even able to understand the intended usage of certain services that I have described and how the APIs that I have defined work. It can also generate links to files so that they can be accessed directly without having to navigate the structure of my repository.

Taking other Visual Studio Code extensions that integrate a GPT model with our IDE as an example, functions are inherently more limited: Visual chatGPT Studio - Visual Studio Marketplace. As we can see in this case, the offered features only cover a subset of the functionalities related to the selection of code sections and the ability to ask questions about them.

But let's analyze an even more complex scenario than what we have seen so far: let's assume that we have two microservices communicating with each other through a queue. In addition to the "@workspace" tag, I also use the tag "#file:" to enrich the context of the chat by inserting, in addition to the selected code, another file. This way, I can ask how the event managed by microservice 2 is formatted inside microservice 1:

What is meant by "programming questions"?

It is interesting to focus on GitHub Copilot's narrowed operational context. When it's said that the tool is able to answer programming questions, we should not only think about the world of code in the narrowest sense.

We are also able to ask questions about frameworks, third-party tools we use in the code, and the hosting architecture of an application, such as "How can I create a secure AKS cluster?"

Therefore, general questions such as "What are the general principles of quantum mechanics?" are excluded. Questions that do not pertain in any way, even from an architectural point of view, to writing code.with the writing of code are therefore excluded.

However, if we need answers on certain topics in order to conduct an analysis for the code we are writing, do we have alternatives to obtain such answers? We can safely use more general tools such as ChatGPT or tools that natively have access to the web to provide answers such as Copilot on Edge.

GitHub Copilot EnterpriseGitHub Copilot Enterprise

In Enterprise tier, the gap in terms of features grows. In addition to normal Chat and Code completion (also present in the first two tiers), some truly interesting features are added:

Knowledge of my organization's knowledge base: GitHub Copilot is able to index the repositories in my organization and answer questions related to all the files contained within it. An RAG scenario, or Chat With Your Data, focused on code that offers the possibility to ask the chat questions about the use of other projects compared to the ones I am currently working on (for example a shared function library used within my organization).
Pull Request Diff Analysis: here we have the ability to have GitHub Copilot analyze the individual commits that make up a Pull Request highlighting the fundamental changes that impact it
Web Search Powered by Bing (beta): We can ask questions about the latest updates to a framework and the answer will be generated by searching online content.Attention: we are also talking here about code-related questions.
Fine-Tuned Models (coming soon): it is a feature that has not yet been made available in General Availability, but it will allow fine-tuning of the GitHub Copilot model based on our repositories. How does it differ from indexing? Simply put: the model will not have to search for answers based on user questions in an indexed information database, it will have the answer built into its model. Just as ChatGPT in the free tier can tell us who the Prime Minister of the United Kingdom is without searching for the answer, GitHub Copilot will be able to natively know which shared libraries exist within our organization. Suppose we have a library that is used to interact with OpenAI APIs, when GitHub Copilot suggests (even proactively!) code to call OpenAI APIs, it will not suggest a generic HTTP request, but rather suggest the appropriate library invocation. ## Conclusion If from a proactive point of view, GitHub Copilot appears to be the only tool that offers the service of suggesting entire portions of code without explicit requests, the native integration of the Chat with the IDE makes the use of the tool significantly easier compared to ChatGPT integrated services, which require a more manual intervention for context construction.

From a cost standpoint, ChatGPT Team has a higher costcompared to GitHub Copilot Business, which offers more advanced programming features. What is lost by not having the ability to do online searches - replaceable as we have seen with other tools - is gainedin terms of proactivity and ease of use of the tool. GitHub Copilot Enterprise, on the other hand, has a cost higher than ChatGPT Team but offers a series of truly interesting additional features for Enterprise scenarios.

Serverless or AKS in an OpenAI solution: let's go against the tide

Tommaso Stocchi — Tue, 21 May 2024 18:00:00 +0000

Working as a Cloud Solution Architect, one of the most common scenarios encountered in the world ofserverless migration to the enterprise. Usually, a serverless architecture is built to support a PoC or pilot project, and then migrated to enterprise-level environments such as Azure App Service or Azure Kubernetes Service.
Does it make sense to consider a reverse path? As always, when it comes to architectures, the answer depends on the scenario we are considering.

Retrieval Augmented Generation

Let's consider an architecture based on OpenAI as a scenario, specifically an implementation of Retrieval Augmented Generation. By Retrieval Augmented Generation (or RAG) we refer to an architecture that allows us to implement a "chat with your data" project. We use a Large Language Model (LLM) to generate responses based on our specific data.
The steps to create this architecture are simple: we first need to upload our data (simple pdf files, for example), divide them into smaller text portions, and for each portion, calculate a numerical vector (embedding) and save it in a vector database. This is to ensure that based on the question asked by the user, we can conduct vector search starting from the question, understand in which indexed text portion the answer to the question is located, and provide that text portion to the LLM, in our case ChatGPT, along with the question itself so that the model can give us the answer.
Semantic search is important in this case because we need the user interaction to be as "human-like" as possible. We need to retrieve one or more paragraphs that may contain an answer to a question, so a simple word-by-word check of the paragraphs would not be sufficient. For example, my paragraph might say, "to obtain an identity card at the Como municipality, you need to apply at the police station, bringing with you [...]”, and the question might be asked as, "How to renew an identity document." The difference is minimal, but a word-by-word search would not give me a positive result. Vector search, on the other hand, is based on the semantic distance between words: "identity card" and "identity document" are very close together.

On Azure, we have different options to implement the various components of this architecture. In our case, we will use Azure Form Recognizer for dividing the PDF into paragraphs, Azure Search for saving the paragraphs and performing vector searches, Azure OpenAI for computing paragraph vectors and formulating responses, and finally an Azure Blob Storage for storing our files. As a code hosting service, if we want to build an architecture capable of scaling to multiple files uploaded simultaneously, we will have four microservices on Azure Kubernetes Service:

Frontend: simple applicationBlazor that allows you to upload PDFs and ask questions
Document Processing: API that uses Form Recognizer to divide the uploaded PDF into its paragraphs
Knowledge Processing: API that takes a paragraph and calculates its vector to be saved in memory
Search Knowledge: API that receives a user's question, calculates its embedding and searches the vector database, and then invokes Azure OpenAI to generate the answer

Document Processing and Knowledge Processing are the two microservices that will need to scale up to cope with multiple files uploaded together and the resulting number of paragraphs to save. To achieve this, the two services are not directly invoked through API calls, but are subscribed to a queue service, in our case Azure Service Bus.

Why do we use AKS in this architecture? There could be several reasons behind this choice: corporate culture, the need for high scalability capacity,possibility of customizing the cluster or simply the assurance of being able to migrate from one installation to another with ease.

Communication among microservices, subscription to Service Bus, and reading secrets from Azure Key Vault are entrusted to Dapr. The authorization for the various resources involved is instead managed, where possible, through Azure Identity, leveraging “User-assigned Identities”.

The repository of reference for this architecture is available here.

How much does it cost me?

So now we have our active solution and we can chat with our data. But how much does it cost to keep it active?

Assuming a cluster consisting of three B2alsv2 virtual machines (2 vCPUs, 4GB RAM), the monthly cost would be around $94.

After the initial upload of our so-called knowledge base (the set of files we want to query until we release the solution), as we move forward in time, we will need to upload a few files per month. Regardless of the reasons that prompted us to adopt AKS as a starting point, it might now make sense to migrate to a serverless.

Azure Container Apps to the rescue!

The Azure Container Apps (ACA) come into play. It is an environment that is fully managed by Azure. No node management, infrastructure management, or networking is required. In addition to the significant advantage of having an infrastructure that is fully managed by the cloud provider, ACA also offers a consumption-based billing plan, i.e. serverless. To make a comparison, while the cost plan for AKS is similar to that of a Virtual Machine Scale Set, ACA's plan is similar to that of AzureFunction consumption. Instances are not paid for the time they remain active, but for the number of requests and the execution time used to process each request.

Assuming 1 million monthly requests with 20 concurrent requests managed for each Container App and a total execution time of 2 seconds per request, the monthly cost for a Container App (4vCPUs, 8GB RAM) would be around $6.60.

For details on the billing plan of Azure Container Apps please refer to this link.

How do Azure Container Apps work from a resource perspective? The first thing to do is to create a Container Apps Environment. This resource has two main functions: managing resource allocation (and therefore billing) for our apps and configuring DNS, network, components. Dapr, certificates. You can think of the Container Apps Environment as the App Service Plan for Web Apps. Every Web App must have an App Service Plan that manages scaling based on available resources, and each ASP can have one or more Web Apps to allocate those resources to. The more the base resources assigned to this plan grow, the higher the cost. However, the two resources differ in one way. While an App Service Plan has a fixed cost that depends on how long it remains on and the chosen tier, a Container Apps Environment has a cost dependent on the tier chosen multipled by the number of requests processed per month.
To make a comparison with AKS, we can see the Container Apps Environment as the namespace of Kubernetes and the individual Azure Container App as the deployments.

In a scenario like the one described, where after the initial intensive use we expect to have a low number of monthly requests for our product, the savings generated by a migration choice of this kind are significant.

What are the operational implications of this migration? If we think about moving a REST API from a Web App to Azure, we know we will have to rewrite part of the code. On the other hand, when it comes to migrating a workload from AKS to ACA, none of this is required. We just need to create an Azure Container App and point it to the Docker image that needs to be hosted. For instance, we know we will have to rewrite part of the code. When it comes to migrating a workload from AKS to ACA, none of this is required. We just need to create an Azure Container App and point it to the Docker image that needs to be hosted. User-Assigned Identity that were already used by AKS can also be assigned to the same Container Apps.

Dapr is natively supported in Azure Container Apps and the same User-Assigned Identity that were already used by AKS can also be assigned to the same Container Apps.

To present one of our Apps externally, we just need to check the box in the Ingress section.

Conclusions

We have seen how to achieve an internal namespace of Kubernetes on a completely serverless resource, greatly reducing costs, and how this trend of migration can make sense. However, we can also go further: we can ask ourselves if, in certain scenarios, it makes sense to start with a hosting environment rather than another. The motivations that lead us to choose one hosting mode over another are various and complex, but often when we talk about microservices we are driven by habit to immediately and only talk about Kubernetes. However, it is often necessary to invest in training courses for both developers and operations departments to approach solutions of this type with my clients on the world of Kubernetes, both for the client himself and for developers. The paradigm of considering the many serverless solutions on the market can lead us to great advantages. Therefore, it is worth knowing them and taking them into consideration.