DEV Community: Borys Generalov

GitHub Agentic Workflows: Building Self-Healing CI for .NET

Borys Generalov — Thu, 21 May 2026 16:10:11 +0000

Agentic Platform Engineering: Self-Healing CI/CD Pipelines

Demo Repository: Check the complete project on GitHub to see the full setup.

My CI failures are usually not dramatic. But they are still annoying.

A test breaks with a NullReferenceException. A Helm chart release failed. I open the logs, trace the problem, fix a tiny mistake, push, and wait for CI again. That is a lot of delay for bugs that are often small.

So I built a workflow for that exact loop. When CI fails, a GitHub Agentic Workflow reads the logs and uploaded artifacts, traces the root cause, and asks for a draft pull request with the fix. I still review it. I still merge it. The agent does the investigation work that normally takes the first 15 minutes.

In this article, I will show you how I built that setup in a standard .NET project, how I fed the agent the evidence it needed, and what happened when I tested it with two deliberate bugs.

What Are GitHub Agentic Workflows?

Getting started: Install the CLI and set up your first workflow by following the official quick start guide.

GitHub Agentic Workflows let you define automation in Markdown with YAML frontmatter. That sounds really simple. The YAML part tells GitHub when the workflow runs, which permissions it gets, and which safe write actions it may request. The Markdown body tells the agent what job to do.

Here is a tiny example:

---
on:
  issues:
    types: [opened]
permissions: read-all
safe-outputs:
  add-comment:
---

# Issue Clarifier

Analyze the current issue and ask for additional details if the issue is unclear.

You compile that file with:

gh aw compile

This creates a .lock.yml file that GitHub Actions can execute. The Markdown file is the source you maintain. The compiled workflow is what runs in CI.

Here, I am not using agentic workflows for summaries or changelogs. Those are straightforward to automate. I care about practical use cases where a failure is easy to address, yet if the CI build fails, someone still has to dig through it.

Setting Up the Project

I used a plain .NET 10 Web API and an xUnit test project. No custom starter kit. Just the templates you already know.

Scaffold the Project

dotnet new sln -n DemoAiPipelines
dotnet new webapi -n OrdersApi -o src/OrdersApi
dotnet new xunit -n OrdersApi.Tests -o tests/OrdersApi.Tests

dotnet sln add src/OrdersApi/OrdersApi.csproj
dotnet sln add tests/OrdersApi.Tests/OrdersApi.Tests.csproj

cd tests/OrdersApi.Tests
dotnet add reference ../../src/OrdersApi/OrdersApi.csproj

Then I added two deliberate bugs.

Bug One: Guest Checkout Crash

This service throws when Customer is null:

public class OrderService
{
    public decimal CalculateDiscount(Order order)
    {
        // BUG: throws NullReferenceException when Customer is null
        var rate = order.Customer.LoyaltyTier switch
        {
            "gold" => 0.15m,
            "silver" => 0.10m,
            _ => 0.05m
        };
        return order.Total * rate;
    }
}

And the test that exposes it:

[Fact]
public void CalculateDiscount_GuestCheckout_ReturnsZero()
{
    var order = new Order(200m, 1, Customer: null);
    var result = _sut.CalculateDiscount(order); // crash here
    Assert.Equal(0m, result);
}

Bug Two: Wrong Port in Helm

The app listens on 8080, but the chart still points at 80:

containers:
  - name: orders-api
    ports:
      - containerPort: 80
    readinessProbe:
      httpGet:
        port: 80

That is enough to make Kubernetes restart the pod forever.

Capture the Evidence

If you want an agent to investigate failures, you need to upload the same logs you would need as a human. For test failures, that means the test output. For deploy failures, that means enough cluster state to explain why the pod never became healthy.

Here is the CI workflow:

# .github/workflows/ci.yml
jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Test
        id: test
        run: dotnet test --logger "trx;LogFileName=results.trx" 2>&1 | tee test-output.txt
        continue-on-error: true
      - name: Upload test evidence
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: "**/test-output.txt"

  deploy-to-kind:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Create KinD cluster
        uses: helm/kind-action@v1.12.0
      - name: Deploy with Helm
        id: helm
        run: |
          helm upgrade --install orders-api ./deploy/helm \
            --wait --timeout 2m 2>&1 | tee helm-output.txt
        continue-on-error: true
      - name: Capture Kubernetes state
        if: steps.helm.outcome == 'failure'
        run: |
          kubectl get pods -o wide > k8s-debug.txt
          kubectl describe pods >> k8s-debug.txt
          kubectl logs -l app=orders-api --tail=50 >> k8s-debug.txt
      - name: Upload deploy evidence
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: deploy-results
          path: k8s-debug.txt

Writing the Self-Healing Workflow

Now we can define the workflow that reacts to a failed CI run.

Create .github/workflows/self-heal.md:

---
engine:
  id: copilot
  version: latest # defaults to latest
  model: gpt-5
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]
permissions:
  contents: read
  actions: read
safe-outputs:
  create-pull-request:
    title-prefix: "fix: "
    labels: [ai-fix, self-healing]
    draft: true
    expires: 7
---

# Self-Healing: Fix Failed CI

You are a .NET and DevOps engineer. A CI run has just failed.

## Your Mission

Analyze the failure, find the root cause, and submit a fix as a
pull request.

## Step-by-Step Instructions

1. Check which job failed: `build-and-test` or `deploy-to-kind`.
2. Download the relevant artifact (`test-results` or `deploy-results`).
3. Read the logs to identify the root cause.
4. For test failures: find the exception and fix the source code.
5. For deploy failures: read `k8s-debug.txt` to trace the issue.
   - Cross-reference with the Dockerfile and Helm chart.
6. Open a PR explaining what went wrong and why the fix is correct.

You do not need to overdo the prompt. You do need to tell the agent where the failure lives, which artifact to read, and what kind of output you want back.

Watching It Fix Real Failures

The normal CI workflow ran first. That workflow built the code, ran tests, tried the Helm deployment, and uploaded evidence whether it passed or failed. After that finished, GitHub triggered the compiled agentic workflow through workflow_run.

So the order looked like this:

the regular CI workflow ran
one job failed: test or deploy
CI uploaded the relevant artifact
the compiled self-heal workflow started
the agent downloaded the artifact and investigated
the agent proposed a draft PR

That means the self-healing workflow only woke up after the normal pipeline had already failed and produced evidence. It was not polling. It was not scanning on a schedule. It reacted to a failed run automatically.

For the test failure, the agent downloaded the test-results artifact, found the NullReferenceException, followed the stack trace into OrderService.cs, and proposed the missing null check.

For the deploy failure, it downloaded the deployment evidence, read k8s-debug.txt, saw that the app was listening on 8080 while the probe was still hitting 80, and changed the Helm config to match.

In both cases the result was a draft PR, not a silent commit to the branch.

That was important for me because I wanted to see the exact diff, the explanation, and the reasoning path. I was not trying to hide the process. I wanted the same review surface I would expect from a teammate.

This also made testing the idea straightforward. Break the pipeline on purpose, let the normal CI fail, and watch whether the follow-up workflow can read the evidence and get back to the right fix.

The Real Cost of Self-Healing

The extra cost starts only after CI fails and the self-heal workflow wakes up. You pay for the agent run, the model tokens used to read logs and repo files, and whatever artifact storage and transfer that investigation needs.

So the real way to think about it is cost per failed run. If failures are rare and the artifacts are small, the cost stays low. If builds fail often and every failure uploads huge logs, the bill grows.

A Practical Rollout Plan

If I were rolling this out for a real team, I would keep the scope narrow:

run only after failed CI workflows
restrict it to one or two common failure types
require uploaded evidence before the workflow can act
allow only draft pull requests as output
review every proposed diff manually

That keeps the workflow predictable. It also gives you a clean way to measure the return on cost. I would track three basic numbers from day one:

how many failed runs triggered the workflow
how many proposed PRs were actually correct
how much engineer time those investigations would normally have taken

If the workflow costs a few dollars but saves hours of senior engineering time on repeatable failures, the tradeoff is obvious. If it produces noisy PRs that nobody merges, the token bill is a waste of money.

Where Self-Healing Gets More Interesting

I would not jump straight to "AI fixes everything." I would expand the triggers one by one.

For example:

after deployment, scan pod logs for restart loops or obvious startup exceptions
after a health check job, inspect the logs if the app never became ready
after a scheduled smoke test, investigate if an endpoint starts failing

That is where self-healing gets interesting. Not a magic system that pushes to production on its own, but a continuous investigator that notices a broken deployment, reads the evidence, and hands you a draft PR.

I still would not let it merge for me. But I would absolutely let it do the boring first pass on failures.

If you want to try the full setup yourself, the demo repo and workflow files are here:

Tip:
Demo Repository:
github.com/bgener/demo-ai-github-pipelines

This article is part of The Modern DevEx Stack series. The next post looks at using MegaLinter in a polyglot repo without turning every pull request into a waiting game.

Build an AI-Powered Developer Portal with Backstage and .NET

Borys Generalov — Thu, 21 May 2026 15:58:15 +0000

Build an AI-Powered Developer Portal with Backstage and .NET

Want to apply AI, not just read about it? Most tutorials stop at a "Hello World" chatbot. We are going to build something that actually solves a common engineering headache: stale documentation.

Who this is for

This guide is for platform engineers and .NET developers who need to organize a growing software landscape without forcing teams to manually write YAML files.

What you will build

You will build a dynamic developer portal using Backstage that automatically populates its service catalog. We will use a .NET CLI tool to scan source code and use local AI (Ollama) to generate summaries.

Source repo: demo-backstage-catalog-generator
Constraint: We use local inference only. No source code ever leaves your machine.

Have you ever needed to update a service, but forgot what it does? Or spent time trying to understand code you have not touched in months? We usually solve this with a README.md that nobody updates, or a wiki that rots.

An Internal Developer Portal (IDP) solves this by making the software landscape visible, but only if the data is fresh. Automation is the only way to avoid the "stale metadata" trap.

Tip:
Want to skip ahead? Check out the complete working demo on
GitHub with all
the code ready to run!

Prerequisites

Before we start, make sure you have the following installed:

.NET SDK 8+
Node.js (includes npx and yarn)
Ollama with the llama3:8b model pulled
A GitHub account (for hosting and deployment)

Why Does an Internal Developer Portal Matter?

The term "Internal Developer Portal" (IDP) can be a little misleading, since it sounds like a tool exclusively for developers only. In reality, it functions as an internal organizational portal focused entirely on your software portfolio. Unlike a general-purpose SharePoint site where everything is dumped in one place and nothing is easy to find, an IDP is deliberately narrow in scope. It covers your software landscape and nothing else, which is exactly what makes it powerful.

An IDP becomes the single source of truth for your engineering organization. It answers critical questions across every role:

Engineers: Which services exist? Who owns them? What do they do? What are the APIs and how do I call them?
Team leads and architects: What is the team composition? Which squad owns which set of services? What architectural decisions have been made and why?
New joiners: How do I get up to speed on a codebase I have never seen before?
Platform and operations teams: What is running in production, who is responsible, and what is the lifecycle status?

Beyond just listing services, a mature IDP centralizes Architecture Decision Records (ADRs), arguably one of its most valuable features. ADRs capture why a decision was made, not just what was decided. Without a central place to surface them, they rot in forgotten wiki pages or git repositories that no one thinks to check.

The challenge is keeping all of this populated and accurate. If you rely on engineers to manually maintain metadata YAML files, the data grows stale within weeks. Automation is the only sustainable path.

Formulating the Architecture

Here's the plan to extract metadata from source code and present it visually:

Backstage: the UI layer where engineers browse and discover services, APIs, documentation, and team ownership
.NET Core: a CLI tool that scans project folders, extracts metadata, and generates Backstage-compatible YAML
Ollama: runs AI inference locally, so your source code never leaves your machine
Static hosting: deploy to Netlify, Azure Static Web Apps, or any provider of your choice

Info:
Why Ollama? Because it runs locally and you do not want to expose your code
to the AI agents over the public internet. You do not know what and how they
use it for, and it does not feel safe. If your employer finds out, you're
done.

Setting Up the Project Infrastructure

You can either follow along and build everything from scratch, or clone the demo repository to get started immediately.

To clone the demo:

git clone https://github.com/bgener/demo-backstage-catalog-generator.git
cd demo-backstage-catalog-generator

To build from scratch, start by downloading and running the LLM model we will use in this guide:

ollama pull llama3:8b
ollama serve

Tip:
We use llama3:8b specifically. It is significantly faster for local
inference than the full-size model and produces more consistent, concise
output for our use case. If you have a powerful GPU, feel free to use llama3
instead.

Next, scaffold the baseline .NET services. We’ll create one Web API and one MVC project:

mkdir Backstage-Dev-Portal
cd Backstage-Dev-Portal

dotnet new webapi -n ServiceA
dotnet new mvc -n ServiceB

dotnet new sln -n Backstage-Dev-Portal
dotnet sln add ServiceA/ServiceA.csproj
dotnet sln add ServiceB/ServiceB.csproj

You can replace the default controllers with real logic later. These raw services represent the uncataloged microservices in your organization.

Building a Smart Catalog Generator in .NET

We will build a .NET CLI tool using OllamaSharp. It scans each project, sends relevant files to the local AI model, and generates a single catalog-info.yaml file containing all services, ready for Backstage to consume.

Create the tool and add the required package:

dotnet new console -n ProjectSummarizer
cd ProjectSummarizer
dotnet add package OllamaSharp

Instead of sending every file to the AI, we take a smarter approach to avoid token limits and save compute time. We will send only *.csproj, Program.cs, and the folder structure. This is all the context the AI needs to understand the project structure and purpose.

Replace Program.cs with the implementation below. The full version is in the demo repository. Here we focus on the key parts.

First, set up the Ollama client and configure the system prompt. This is the most fragile part of the chain: the system prompt has to force the model into a YAML-safe format without it hallucinating markdown backticks:

var ollamaApiClient = new OllamaApiClient(
    new Uri("http://localhost:11434")) { SelectedModel = "llama3:8b" };

var chat = new Chat(ollamaApiClient, systemPrompt:
    "You are a technical documentation assistant. " +
    "You produce concise, YAML-safe summaries of .NET projects. " +
    "Output only plain text, no markdown, no bullet points, no quotes, no colons, no newlines.");

Instead of sending every file to the AI, we only send *.csproj, Program.cs, and the folder structure. This is all the context the model needs.

Warning:
Prompt sanitization is critical. If your Program.cs contains complex
string literals or nested colons, the AI might pass them through to your YAML,
breaking the Backstage parser. Always sanitize the output before writing the
file.

var sb = new StringBuilder();
sb.AppendLine($"Project: {projectName}");
sb.AppendLine("Folder structure:");
AppendFolderStructure(projectDir, sb, "");
sb.AppendLine(File.ReadAllText(csprojPath));

var programPath = Directory.GetFiles(projectDir, "Program.cs", SearchOption.AllDirectories)
    .FirstOrDefault();
if (programPath != null)
    sb.AppendLine(File.ReadAllText(programPath));

The prompt itself uses few-shot examples to guide the model toward the output format we want:

var prompt = "Summarize the project in 1-2 sentences based on the files provided. " +
             "Do not output anything else. " +
             "Examples of good output: " +
             "REST API service providing weather forecasts with temperature data\n" +
             "ASP.NET MVC application with React frontend for managing todo items\n\n"
             + sb.ToString();

await foreach (var token in chat.SendAsync(prompt, cts.Token))
    summaryBuilder.Append(token);

Finally, each summary is sanitized and assembled into a Backstage-compatible YAML entry:

var summary = summaryBuilder.ToString().Trim()
    .Replace("\n", " ").Replace(":", " -").Replace("\"", "'");

var yamlEntry = $@"
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: {projectName.ToLowerInvariant()}
  description: ""{summary}""
spec:
  type: service
  lifecycle: production
  owner: group:default/engineering";

Run the generator against the target directory (use . if you are already in the project root):

dotnet run --project ProjectSummarizer -- .

You should see the AI streaming its summaries in real time:

Most of the real work here is figuring out the prompt. Even a tiny change can produce a completely different output. I encourage you to experiment with the system prompt and the user prompt to see how it affects quality. That is the real learning here.

Integrating the AI Catalog with Backstage

With the catalog-info.yaml ready, we can integrate it into a Backstage instance. Install Backstage:

npx @backstage/create-app

Follow the prompts to name it dev-portal.

Now point Backstage to your generated catalog file. Open app-config.yaml in the dev-portal directory and add the following under the catalog section:

catalog:
  locations:
    - type: file
      target: ../Backstage-Dev-Portal/catalog-info.yaml
      rules:
        - allow: [Component]

This tells Backstage where to find the AI-generated service metadata. The target path is relative to the Backstage root directory. Adjust it to point to wherever your generator wrote the catalog-info.yaml.

To run it locally:

cd dev-portal
yarn dev

Open http://localhost:3000 in your browser. You should see all your services listed in the Software Catalog with AI-generated summaries visible in the description column.

Deploying the Portal

To host this, build your portal as a static site:

yarn build:static

Push the output to GitHub and deploy to any static hosting provider: Netlify, Azure Static Web Apps, Vercel, or even self-hosted on Kubernetes. Set the build command to yarn build:static and the publish directory to dist.

Info:
A static Backstage build is great for read-only catalogs. If you need dynamic
features like authentication, real-time plugin backends, or write
operations, you will need to deploy the full Backstage backend as a Node.js
service instead.

Automating with CI/CD

The real value comes from running the catalog generator automatically. Here is a GitHub Actions workflow that regenerates summaries on every push to main:

name: Update Backstage Catalog
on:
  push:
    branches: [main]

jobs:
  generate-catalog:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: ‘8.0.x’

      - name: Install and start Ollama
        run: |
          curl -fsSL https://ollama.com/install.sh | sh
          ollama serve &
          sleep 5
          ollama pull llama3:8b

      - name: Generate catalog
        run: dotnet run --project ProjectSummarizer -- "$GITHUB_WORKSPACE"

      - name: Commit updated catalog
        run: |
          git config user.name "github-actions"
          git config user.email "github-actions@github.com"
          git add catalog-info.yaml
          git diff --cached --quiet || git commit -m "chore: regenerate AI catalog summaries"
          git push

Warning:
CI Performance: Running Ollama in CI uses CPU-only inference by default. A
llama3:8b summary takes about 20-30 seconds per project on a standard GitHub
runner. For a large monorepo, your CI bill will spike. Consider using a
persistent self-hosted runner with a GPU if you scale this.

Final thoughts

The practical rule is simple: automate the metadata generation where the code lives, but keep the UI (Backstage) as a thin, static client. This prevents the "stale documentation" problem without adding a heavy runtime dependency to your production environment.

Use narrow context: Don't send the whole repo to the AI. Files like Program.cs and *.csproj are usually enough.
Sanitize strictly: AI output is non-deterministic. Always strip colons and newlines before writing to YAML.
Start static: A read-only static portal is 10x easier to maintain than a dynamic one.

FAQ

Can I use OpenAI instead of Ollama?
Yes, but you will be sending your source code (or at least your Program.cs) to a third party. Use a local model if security is a concern.

Does this replace README files?
No. It replaces the "Service Directory" that usually lives in a spreadsheet. It points engineers to the README they actually need.

How do I handle project renames?
The generator uses the folder or .csproj name. If you rename them, Backstage will see it as a new component unless you map the identity stable-ly.

I help teams build exactly this kind of internal tooling, from developer portals to platform engineering. See my work or get in touch.

SQLMesh for dbt users: the migration path

Borys Generalov — Thu, 14 May 2026 19:39:36 +0000

How to move from dbt to SQLMesh, understand the plan/apply cycle, leverage native column-level lineage, and run dev environments without schema suffixing.

What you get: an introduction to SQLMesh's core differences, the exact commands to run, how tests became audits, how dev environments work, and screenshots from a running demo.

What is dbt

Data Build Tool (dbt) is an open-source framework that allows analysts and engineers to transform data in their warehouses by writing SQL SELECT statements enriched with Jinja templating. This special syntax adds dynamic logic—like loops, variables, and dependency referencing—directly into your SQL. Instead of manually writing CREATE TABLE statements, dbt takes your templated queries and automatically materializes them in the database for you. When a data platform scales, nested Jinja templates become impossible to debug, and copying production data to build staging environments gets expensive fast.

What is SQLMesh

This is where SQLMesh comes in. As a modern framework challenging the status quo for the transformation (the 'T') phase of ETL/ELT pipelines, SQLMesh is designed specifically to address these growing pains. It provides a more robust, scalable way to manage data transformations by bringing strict software engineering practices—like stateful dry runs and zero-copy environments—directly into the data warehouse. Because it solves the exact operational bottlenecks that slow down mature data teams, it has rapidly grown in popularity, ultimately leading to its contribution to the Linux Foundation in March 2026.

Under the Hood: Text Templating vs. AST Parsing

The fundamental difference between dbt and SQLMesh comes down to how they read your code.

dbt is a text templater. It uses Jinja to stitch strings of SQL together and sends the final block to the data warehouse to execute. It doesn't semantically understand the SQL it generates.

SQLMesh is a semantic parser. It reads your code and builds an Abstract Syntax Tree (AST) using its underlying engine, SQLGlot. Because SQLMesh understands your SQL before it hits the warehouse, it knows exactly which columns are being selected, aliased, or joined at compile time.

Because SQLMesh actually reads your code, it does three things that dbt cannot do natively:

A Terraform-like plan/apply cycle that guarantees you know the exact impact before spending warehouse compute.
Column-level lineage natively, without requiring expensive third-party data catalogs.
Virtual Data Environments that allow developers to test changes in isolation without physically copying production data.

The plan/apply cycle: seeing the impact

In dbt, you run dbt run and wait to see what breaks. In SQLMesh, you use a stateful workflow inspired by Terraform: plan and apply.

sqlmesh plan

This command evaluates your local SQL files against the current state of the database and generates an execution plan. It outputs exactly what will be built, whether it's a full refresh or incremental, and asks for confirmation:

======================================================================
Plan: prod
======================================================================
New environment `prod` will be created from `empty`

Added Models:
├── jaffle_shop.customers    (Full Refresh)
└── jaffle_shop.orders (Full Refresh)

Apply - Create prod environment and backfill models [y/n]:

When you add a column to stg_payments.sql, the next sqlmesh plan highlights the real dependency chain before any warehouse compute is consumed. In this demo, orders reads from stg_payments, and customers reads from orders, so both appear as indirectly modified:

You review the impact, then type y to apply. You stop paying for blind warehouse queries just to see if your code works.

The plan and apply execution from the running demo looks like this:

Column-level lineage out of the box

Because SQLMesh uses SQLGlot to parse your SQL, it natively understands how data flows from source to destination at the column level. It traces dependencies through complex aliases, window functions, and joins without you ever writing a YAML definition.

SQLMesh does have a dbt-docs-style DAG view, but it is exposed through the data catalog and lineage graph instead of a separate static docs site. The useful difference is that the graph is tied to the selected SQLMesh environment and can show model fields, not only model boxes.

The lineage view for the demo model looks like this. The point is not that SQLMesh has a DAG. dbt already has that. The useful part is column visibility: you can inspect which upstream columns feed a downstream model, which is what makes impact analysis and PII tracing practical.

In this small demo, the graph shows seed_model feeding incremental_model, then full_model, with the columns visible on each node.

Use Case 1: Safe Deprecation of Columns

Say you need to drop a deprecated user_phone column from the production database.

In a dbt setup, understanding the full impact is problematic. You would have to do a global text search for user_phone across hundreds of SQL files. Even then, if the column was aliased (SELECT user_phone AS phone_number), a simple text search might miss downstream models that rely on phone_number.

SQLMesh solves this instantly. By parsing the AST, it tracks the column through every alias and CTE. You simply open the built-in UI editor and click the column name.

It visually traces and highlights exactly which downstream reporting tables or BI dashboards will break if you drop the source column. You can confidently deprecate columns without causing massive data outages.

Use Case 2: Tracing PII Data for Compliance

Data privacy regulations (like GDPR or CCPA) require strict tracking of Personally Identifiable Information (PII). If you mistakenly join a table containing an email_address into a public-facing metrics aggregate, it could result in a massive compliance violation.

With SQLMesh, you can visually trace the flow of sensitive data using the built-in UI:

sqlmesh ui

This spins up a local web server displaying an interactive lineage graph. For a sensitive field such as email, the lineage should make the path easy to inspect:

`raw_customers.email` (PII) -> `stg_customers.email` -> `customers.email`

Because the lineage is native, data governance teams can instantly verify that PII is masked or excluded before it reaches downstream aggregates.

Dev environments without schema suffixing (Virtual Environments)

This is how SQLMesh directly cuts warehouse compute costs.

In dbt, a development environment is physical. It's a target schema (e.g., analytics_dev_yourname). When you build your models to test a change, you physically copy or rebuild the data into your dev schema.

In SQLMesh, a dev environment is a Virtual Data Environment.

When you run sqlmesh plan dev, SQLMesh doesn't copy data. Instead, it creates lightweight, virtualized database views that simply point to the existing physical tables from prod.

The SQLMesh UI keeps those environments visible in the same toolbar used by the editor, plan view, and data catalog. In this demo, prod and dev both exist, and prod is marked as the production environment.

The model is simple:

prod.customers can point to customers_v1, while prod.orders points to orders_v1.
dev.customers can point to a new customers_v2, while dev.orders still points to the existing orders_v1.

Only the changed model needs new physical storage. Everything unchanged can keep pointing at the already-built production tables.

Use Case 1: Multi-Developer Collaboration (Dev)

In a fast-moving data team, multiple developers are working on different features simultaneously. Alice is updating the customers model, and Bob is updating the orders model.

In traditional setups, Alice and Bob either step on each other's toes in a shared staging schema, or they both have to spend 20 minutes copying gigabytes of data into alice_dev and bob_dev schemas before they can start working.

With SQLMesh, Alice simply creates her own virtual environment (sqlmesh plan alice_feature). SQLMesh builds only her modified customers table physically, while her orders view points to the production physical table. Bob does the same for his feature. They get perfect isolation instantly, with zero duplicated data and zero wasted compute cost.

Use Case 2: Instant Rollbacks and Blue/Green Deployments (Prod)

Deploying pipeline changes to production is risky. A flawed query can corrupt downstream tables and break executive BI dashboards.

SQLMesh handles production deployments using Blue/Green deployments natively via virtual environments.

When you merge your code to main and deploy, SQLMesh builds the new physical tables in the background (the "Green" state). Your prod environment views still point to the old tables (the "Blue" state).

Once the new tables are fully built, populated, and automatically audited for quality, SQLMesh simply updates the prod views to point to the new physical tables. This pointer swap takes milliseconds. If a bug is discovered after deployment, you can instantly rollback by swapping the view pointers back to the previous physical tables. No data needs to be rebuilt, providing a massive safety net for data teams.

Upgrading a Model: dbt vs SQLMesh

Here is what this looks like in practice. We need to build a customers model that cleans up user data from jaffle_shop.stg_customers and joins it with jaffle_shop.orders to calculate lifetime value.

In dbt, you first write the SQL template (models/customers.sql):

{{ config(
    materialized='table'
) }}

WITH customer_orders AS (
  SELECT
    customer_id,
    MIN(order_date) AS first_order_date,
    MAX(order_date) AS most_recent_order_date,
    COUNT(*) AS number_of_orders,
    SUM(amount) AS customer_lifetime_value
  FROM {{ ref('orders') }}
  WHERE status <> 'returned'
  GROUP BY customer_id
)

SELECT
  c.customer_id,
  c.customer_name,
  c.email,
  c.signup_date,
  o.first_order_date,
  o.most_recent_order_date,
  COALESCE(o.number_of_orders, 0) AS number_of_orders,
  COALESCE(o.customer_lifetime_value, 0) AS customer_lifetime_value
FROM {{ ref('stg_customers') }} AS c
LEFT JOIN customer_orders AS o
  ON c.customer_id = o.customer_id

But you're not done. In dbt, your SQL file only contains the logic. To validate that the output data is actually correct—for example, to test that your customer_id is unique and never null—you have to leave your SQL file, open a completely separate YAML configuration file (models/schema.yml), and write your validation tests there:

version: 2
models:
  - name: customers
    columns:
      - name: customer_id
        tests:
          - unique
          - not_null

The dbt docs graph for this looks like this:

When you migrate this same model to SQLMesh, the scattered configuration disappears. Everything—metadata, dependencies, materialization logic, and testing—consolidates into the SQL file itself via the MODEL block.

MODEL (
  name jaffle_shop.customers,
  kind FULL,
  grain customer_id
);

WITH customer_orders AS (
  SELECT
    customer_id,
    MIN(order_date) AS first_order_date,
    MAX(order_date) AS most_recent_order_date,
    COUNT(*) AS number_of_orders,
    SUM(amount) AS customer_lifetime_value
  FROM jaffle_shop.orders
  WHERE status <> 'returned'
  GROUP BY customer_id
)

SELECT
  c.customer_id::INT AS customer_id,
  c.customer_name::TEXT AS customer_name,
  c.email::TEXT AS email,
  c.signup_date::DATE AS signup_date,
  o.first_order_date::DATE AS first_order_date,
  o.most_recent_order_date::DATE AS most_recent_order_date,
  COALESCE(o.number_of_orders, 0)::INT AS number_of_orders,
  COALESCE(o.customer_lifetime_value, 0)::DOUBLE AS customer_lifetime_value
FROM jaffle_shop.stg_customers AS c
LEFT JOIN customer_orders AS o
  ON c.customer_id = o.customer_id

Notice what happened here:

No Jinja Refs: You simply query jaffle_shop.stg_customers and jaffle_shop.orders using standard SQL. Because SQLMesh parses the AST, it automatically detects the dependencies for both tables.
No YAML Tests: The grain customer_id property inside the MODEL block automatically declares the natural key. You do not need to write explicit tests. When you want to validate the data, you simply run sqlmesh audit in the terminal, and SQLMesh will automatically generate and execute the uniqueness and not-null validation checks for you based on that grain.

The generated DAG for this model in SQLMesh looks like this:

This is the closest match to the familiar dbt docs graph: a model dependency DAG generated from parsed SQL. The browser UI goes one step further by letting you inspect the same dependency chain with environment context and column metadata.

If you want to try this yourself locally (using DuckDB, so no warehouse account needed), installation is simple:

pip install "sqlmesh[duckdb]"
mkdir sqlmesh-demo && cd sqlmesh-demo
sqlmesh init -t empty

Automating this with CI/CD (GitHub Actions)

The plan/apply cycle and virtual environments are impressive locally, but they become a superpower when integrated into your CI/CD pipeline.

SQLMesh provides a native GitHub Action (TobikoData/sqlmesh-action) that automates this entire workflow:

The PR Plan: When a developer opens a Pull Request, the GitHub Action automatically creates a temporary Virtual Environment (e.g., pr_123) and runs a plan.
Automated Review: The bot posts the exact execution plan as a comment directly on the PR. The reviewer instantly sees which models are modified and which downstream models will be impacted, without having to run anything locally.
Zero-Copy Deployment: Upon merging to main, the deployment job doesn't need to rebuild the data. It simply swaps the pointers in the prod environment to reference the physical tables that were already built and tested during the PR.

This guarantees that main is always in a deployable state and developers never accidentally merge breaking schema changes into production.

Final thoughts

Moving from text-based templating to semantic AST parsing is the structural shift required to solve data environment bloat and lineage blindness.

To evaluate this transition yourself:

Audit your current warehouse spend strictly related to developer sandbox schemas.
Run sqlmesh init --template dbt on a local branch of your existing dbt project.
Migrate 5 connected models to the MODEL block syntax.
Execute sqlmesh plan dev to observe the virtual pointer creation in your warehouse.

When the tool understands the code, the infrastructure manages itself.

Q: Do I need to rewrite all my dbt SQL immediately?
No, SQLMesh can parse and run your existing Jinja-based dbt models directly using its dbt adapter.

Q: Does SQLMesh require DuckDB?
No, DuckDB is just used here for a fast local demo; SQLMesh natively supports Snowflake, BigQuery, Databricks, and Redshift.

Q: How does SQLMesh handle dbt macros?
It executes existing dbt macros perfectly during the transition, though rewriting them as native Python macros eventually unlocks deeper AST-level validation.

Q: Is this relevant if my team only has three data models?
The immediate value for small teams is the automated testing and plan visibility, but the cost savings of virtual environments truly compound at scale.

OpenTelemetry custom spans in .NET: seeing what your code decided

Borys Generalov — Thu, 07 May 2026 13:55:09 +0000

How to instrument business decisions that auto-instrumentation cannot see. Events, span kinds, cross-process context propagation over RabbitMQ, and baggage” with a working .NET 10 demo.

What you get: when to use custom spans, what to put on them, and how to avoid the traps. How to name spans, what to tag, when to use events instead, context propagation between services over RabbitMQ, and baggage that travels without any method parameters.

Demo: github.com/bgener/otel-custom-spans” two .NET 10 services, RabbitMQ, Aspire Dashboard and Jaeger. docker compose up --build and you have a working trace across two processes.

OpenTelemetry spans

A trace is the full path of a request through your system. Each step in that path is a span: a named, timed unit of work with attributes attached. OpenTelemetry's auto-instrumentation creates spans for the calls it understands: inbound HTTP requests, outbound database queries, gRPC calls, message queue operations. For each of those you see the operation name, duration, and whether it succeeded.

But auto-instrumentation stops at the call boundary. Your .NET service runs business logic and makes decisions. It returns a result without throwing anything. Something is off, but none of that is visible.

You probably recognize this from the era of heavy log.Debug and log.Verbose. One log statement after every if clause, every internal state worth knowing. The code got noisy and barely readable.

Custom spans are the same idea done properly. Structured attributes, timestamped, sitting in the trace right next to the auto-instrumented calls.

ActivitySource and your first span

In .NET, custom spans are built with ActivitySource. You create one per service or logical area of code, start activities from it, and attach attributes that describe what your code decided. The pattern works the same whether you are in an API handler, a domain service, or a background worker.

When you call StartActivity(), the new span automatically becomes the active span for the current async execution context. Nested calls can reach it via Activity.Current without you threading a variable through every method signature. Auto-instrumented spans work the same way: the inbound HTTP span created by ASP.NET Core middleware is also an Activity, and your custom spans nest inside it automatically.

The demo centralizes source creation in a static class:

internal static class Observability
{
    public const string ServiceName = "weather-api";
    public static readonly ActivitySource ActivitySource = new(ServiceName);
}

One source per service. The name matches the service name so spans are grouped correctly in your APM backend.

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(Observability.ServiceName))
    .WithTracing(tracing => tracing
        .AddSource(Observability.ServiceName)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter(e => { e.Endpoint = new Uri(otelOptions.AspireDashboardEndpoint); }));

Caution:
Skip AddSource and the spans are created but never exported. No error. No warning. Just silence. I have stared at an empty trace viewer for an embarrassing amount of time before catching this.

Here is the weather forecast service from the demo:

internal sealed class WeatherService
{
    private static readonly ActivitySource _source = Observability.ActivitySource;

    public WeatherForecast[] GetForecast(int days = 5)
    {
        using var activity = _source.StartActivity("generate forecast", ActivityKind.Internal);

        activity?.SetTag("forecast.days", days);
        activity?.SetTag("client.id", Baggage.GetBaggage("client.id"));

        var forecast = Enumerable.Range(1, days)
            .Select(index => new WeatherForecast(
                DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
                Random.Shared.Next(-20, 55),
                _summaries[Random.Shared.Next(_summaries.Length)]))
            .ToArray();

        activity?.SetTag("forecast.min_temp_c", forecast.Min(f => f.TemperatureC));
        activity?.SetTag("forecast.max_temp_c", forecast.Max(f => f.TemperatureC));

        return forecast;
    }
}

Three things are on this span: a name, a kind, and attributes.

The name is how this operation appears in the trace viewer. Keep it low-cardinality: name the class of operation, not the instance. OTel recommends {verb} {noun} with spaces: generate forecast, process payment, write record. This mirrors how HTTP (GET /forecast) and database (SELECT weather) name operations. generate forecast is a span name. generate forecast london is not â€” the city belongs in an attribute. See the OTel naming guide.

The attribute forecast.days travels with every trace for this operation. The client.id comes from baggage set at the API edge â€” covered in the cross-process section. Record inputs before the work, outcomes after it. That is the pattern.

Events

Attributes describe the final state of the span. But some operations have decision points inside them that attributes cannot capture.

The forecast can include sub-zero days. That is a fact worth recording at the moment it was detected, not as a summary after the fact. An attribute set after the loop collapses the timeline into a single value. An event preserves when inside the span the thing happened.

var coldDays = forecast.Count(f => f.TemperatureC < 0);
if (coldDays > 0)
{
    activity?.AddEvent(new ActivityEvent("forecast.cold_days_detected",
        tags: new ActivityTagsCollection
        {
            ["cold.day_count"] = coldDays,
            ["cold.threshold_c"] = 0
        }));
}

Events are timestamped. In the trace viewer they appear pinned at the exact millisecond they fired, inside the span timeline. If the span was slow, you can see whether the detection happened early or late and whether it correlates with the latency. An attribute cannot tell you that.

OTel event names follow the general naming conventions: lowercase, dot-separated namespaces, snake_case for multi-word parts. forecast.cold_days_detected is correct. forecastColdDaysDetected is not.

The rule: use an attribute for facts about the operation as a whole. Use an event for things that happened at a specific moment inside it.

Custom span roles and error signals

When you add a custom span, two things determine where it shows up and what it feeds: the kind and the status.

The kind tells your backend what role the span plays. There are five:

_source.StartActivity("handle request",        ActivityKind.Server);   // receiving work
_source.StartActivity("generate forecast",     ActivityKind.Internal); // internal logic
_source.StartActivity("call stored procedure", ActivityKind.Client);   // calling something
_source.StartActivity("publish event",         ActivityKind.Producer); // sending to a queue
_source.StartActivity("process event",         ActivityKind.Consumer); // reading from a queue

Pick the wrong one and your span shows up in the wrong dashboard. Backends that derive metrics from spans emit different time series depending on kind. Server spans become incoming-request RED metrics. Client spans become outbound-dependency metrics. Producer and Consumer feed messaging throughput. Internal appears only inside traces.

Mark a stored proc call as Internal and the dependency dashboard goes blank for that call. Mark a queue publish as Client and the messaging panel never sees it. Kind is not cosmetic.

Status is separate from exceptions. A business rejection is not an exception. Nothing throws. But it is still an Error outcome from the trace's perspective.

if (!CoverageMap.Includes(city))
{
    activity?.SetStatus(ActivityStatusCode.Error, "city outside coverage area");
}

Your error rate dashboard now reflects business failures, not just crashes. This is the gap between SRE error rates and product success rates that you usually learn to live with.

For real exceptions, the demo consumer records them with full structured context:

catch (Exception ex)
{
    activity?.AddException(ex);
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    await _channel!.BasicNackAsync(result.DeliveryTag, multiple: false, requeue: false);
}

AddException records the exception as a span event with exception.type, exception.message, and exception.stacktrace as structured fields. Searchable, filterable, aggregatable in any OTel-compatible backend.

Span links

Most spans have one parent. Batch consumers do not. A consumer drains 50 messages from a queue in one poll. Each message came from a different upstream request, each with its own trace ID. A span needs a parent to belong to a trace, but with 50 different upstream traces there is no single parent.

Span links solve this. A span can carry references to spans in other traces. The consumer span becomes the root of its own trace and holds pointers back to all 50 producer spans. From the consumer trace you click through to any of them.

var links = batch
    .Select(msg => new ActivityLink(msg.ExtractParentContext()))
    .ToArray();

using var activity = _source.StartActivity(
    "consume forecast batch",
    ActivityKind.Consumer,
    parentContext: default,
    links: links);

Without links, batch consumers either pick one arbitrary parent and silently lose the rest, or appear as orphan traces with no context.

Common cases: batch processors, dead-letter retries linking back to the original failure, fan-out workflows where one request kicks off many async jobs.

Cross-process tracing

When both ends of a call know about OTel, trace context flows automatically. HTTP, gRPC, async/await: the .NET runtime handles all of it. The hard case is any transport the SDK has no built-in instrumentation for. RabbitMQ is one of those.

Baggage

Before the producer and consumer, it helps to understand baggage. Baggage is a set of key-value pairs the propagator carries across every process boundary automatically. Set it once at the API edge and it appears on every span in every downstream service without any method parameters.

The demo sets client.id from the X-Client-Id request header:

app.Use(async (context, next) =>
{
    var clientId = context.Request.Headers["X-Client-Id"].FirstOrDefault() ?? "anonymous";
    Baggage.SetBaggage("client.id", clientId);
    await next();
});

Three hops downstream, in the worker process, without passing anything as a parameter:

activity?.SetTag("client.id", Baggage.GetBaggage("client.id"));

Wire a BaggageActivityProcessor into your tracer (see the OTel contrib repo) and you never call SetTag for baggage values at all. The processor copies them onto every span automatically as it closes. Every dashboard you build gets client.id as a free filter.

Caution:
Do not put sensitive data in baggage. It travels in plain text in headers. If your OTel collector exports to a third-party SaaS, you have shipped that data across your trust boundary.

Producer to queue

The publisher creates a Producer span, tags it with messaging attributes, then injects W3C trace context and baggage into the message headers:

internal sealed class WeatherEventPublisher(IConnection connection)
{
    private static readonly ActivitySource _source = Observability.ActivitySource;

    public async Task PublishAsync(WeatherForecastServedEvent evt, CancellationToken ct)
    {
        using var activity = _source.StartActivity(
            "publish weatherforecast event",
            ActivityKind.Producer);

        activity?.SetTag("messaging.system", "rabbitmq");
        activity?.SetTag("messaging.destination.name", QueueNames.WeatherForecastServed);
        activity?.SetTag("messaging.operation.type", "publish");
        activity?.SetTag("client.id", evt.ClientId);

        await using var channel = await connection.CreateChannelAsync(cancellationToken: ct);

        var props = new BasicProperties
        {
            Headers = new Dictionary<string, object?>(),
            DeliveryMode = DeliveryModes.Persistent,
            MessageId = Guid.NewGuid().ToString()
        };

        // Inject W3C trace context + baggage into message headers.
        // The worker extracts these to continue this trace across processes.
        Propagators.DefaultTextMapPropagator.Inject(
            new PropagationContext(Activity.Current!.Context, Baggage.Current),
            props.Headers,
            static (carrier, key, value) => carrier![key] = Encoding.UTF8.GetBytes(value));

        var body = JsonSerializer.SerializeToUtf8Bytes(evt);
        await channel.BasicPublishAsync(
            exchange: string.Empty,
            routingKey: QueueNames.WeatherForecastServed,
            mandatory: false,
            basicProperties: props,
            body: body,
            cancellationToken: ct);

        activity?.SetTag("messaging.message.id", props.MessageId);
    }
}

Two things worth distinguishing. The SetTag calls attach attributes to this specific span â€” they describe this publish operation and appear in the trace detail view. The Propagators.Inject call encodes the W3C trace context and baggage into the message headers so the worker can restore the trace on the other side. Span attributes stay on the span. Baggage travels forward.

connection is IConnection from RabbitMQ.Client v7, which added async channel creation.

The headers on the wire after Inject runs:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
baggage:     client.id=my-client

The traceparent format is the W3C Trace Context spec: {version}-{trace-id}-{parent-id}-{flags}. The 32-character trace ID identifies the whole distributed trace. The 16-character parent ID is the span ID of the publisher span. 01 means sampled. This is the same format HTTP headers carry automatically â€” here you are writing it manually into AMQP message headers.

Consumer

The worker is a completely separate process with its own Program.cs and its own OTel registration. It extracts context from the message headers and passes it as the parent when starting the consumer span:

private async Task ProcessMessageAsync(BasicGetResult result, CancellationToken ct)
{
    var parentContext = Observability.Propagator.Extract(
        default,
        result.BasicProperties.Headers,
        static (headers, key) =>
        {
            if (headers is null || !headers.TryGetValue(key, out var raw) || raw is null)
                return [];
            return raw switch
            {
                byte[] bytes => [Encoding.UTF8.GetString(bytes)],
                string text  => [text],
                _            => [raw.ToString() ?? string.Empty]
            };
        });

    Baggage.Current = parentContext.Baggage;

    using var activity = Observability.ActivitySource.StartActivity(
        "process weatherforecast event",
        ActivityKind.Consumer,
        parentContext.ActivityContext);  // links consumer span to producer span

    activity?.SetTag("messaging.system", "rabbitmq");
    activity?.SetTag("messaging.operation.type", "process");
    activity?.SetTag("messaging.message.id", result.BasicProperties.MessageId);
    activity?.SetTag("client.id", Baggage.GetBaggage("client.id"));

    var evt = JsonSerializer.Deserialize<WeatherForecastServedEvent>(result.Body.Span)!;
    await SimulateAnalyticsWriteAsync(evt, ct);
    await _channel!.BasicAckAsync(result.DeliveryTag, multiple: false);
}

Two lines connect the processes:

parentContext.ActivityContext            // restores trace ID + parent span ID from message headers
Baggage.Current = parentContext.Baggage // restores baggage that travelled with the message

The consumer creates a nested child span for the analytics write â€” a separate Client span showing the DB operation as a distinct step:

private static async Task SimulateAnalyticsWriteAsync(WeatherForecastServedEvent evt, CancellationToken ct)
{
    using var activity = Observability.ActivitySource.StartActivity(
        "write analytics record",
        ActivityKind.Client);

    activity?.SetTag("db.system", "postgresql");
    activity?.SetTag("db.operation", "INSERT");
    activity?.SetTag("db.sql.table", "weather_analytics");
    activity?.SetTag("client.id", evt.ClientId);

    await Task.Delay(Random.Shared.Next(10, 50), ct);
}

What you see in Aspire Dashboard or Jaeger after one request:

GET /weatherforecast                          (Server, ForecastApi)
  generate forecast                           (Internal, ForecastApi)
  publish weatherforecast event               (Producer, ForecastApi)
                    ----- RabbitMQ boundary -----
process weatherforecast event                 (Consumer, ForecastWorker)
  write analytics record                      (Client, ForecastWorker)

Two processes. One trace. The consumer span hangs off the producer span as if they were in the same call stack.

Without Inject and Extract: two completely disconnected traces. The forecast request appears to end at the publish. The worker appears to start a job for no apparent reason. The first time a customer reports a bug you discover the only thing connecting the API request to the analytics record is a wall-clock timestamp.

Spans inside the database

This section covers a scenario not in the demo: stored procedures and how to get visibility into what happens inside the database.

When .NET calls Postgres via Npgsql, EF Core records a SQL client span. Postgres has no idea your trace exists. The proc executes in isolation. The bridge is SQLCommenter, a format that encodes trace context as a SQL comment the database can read.

Info:
Database support: SQLCommenter works with PostgreSQL and MySQL. SQL Server and Oracle have no equivalent. If your stack runs a database without SQLCommenter support, wrap the call in a custom span and capture output parameters as attributes instead.

using var activity = _source.StartActivity("call stored procedure", ActivityKind.Client);

var traceparent =
    $"00-{activity!.TraceId.ToHexString()}-{activity.SpanId.ToHexString()}-01";

await using var cmd = _connection.CreateCommand();
cmd.CommandText =
    $"/*traceparent='{traceparent}'*/ CALL get_forecast_for_city(@city, @tier)";
cmd.Parameters.AddWithValue("@city", city);
cmd.Parameters.AddWithValue("@tier", tier);
await cmd.ExecuteNonQueryAsync();

EF Core 9 with Npgsql ships a built-in EnableSqlCommenter() option that does this for you. Use it if your driver supports it.

Beyond SQLCommenter, there is pg_tracing: a Postgres extension open-sourced by Datadog that reads the traceparent comment and emits OTLP spans for the engine work it can see: parse, plan, execute, nested function calls, trigger execution. Postgres 16+ only.

call stored procedure  (Client, your app)
  pg.parse
  pg.plan
  pg.execute
    pg.function: get_forecast_for_city
      pg.query: SELECT FROM sensor_readings
      pg.query: SELECT FROM city_forecast

Caveats: Postgres 16 or higher only. You build the extension yourself or use a Datadog prebuilt image. It does not let you write start_span() inside PL/pgSQL. Instrumentation happens at the engine level, not in user code. MySQL and SQL Server have nothing comparable.

For any team running Postgres, this is the most direct way to stop treating stored procedures as a blackbox.

The cardinality trap

Every major APM backend converts span names into metric labels. Put a dynamic value in the span name and you get a unique time series for every unique value. At production scale that exhausts cardinality limits and your backend starts dropping data. It looks completely innocent when you write it.

// Do NOT do this
using var activity = _source.StartActivity($"generate forecast {city}");

SigNoz, Datadog APM, Honeycomb, Grafana Tempo all run a spanmetrics processor that emits time series keyed on span_name. A few hundred cities is fine. A few thousand customer IDs in the span name is not. You find out at 2am when dashboards stop refreshing.

// Correct
using var activity = _source.StartActivity("generate forecast");
activity?.SetTag("forecast.city", city);

Same signal. No explosion. The span name stays generate forecast. The city is a filterable attribute. If it changes per request: attribute. If it names the kind of operation: span name.

The real cost

Custom spans have a maintenance cost nobody mentions in the enthusiastic how-to posts.

Business logic changes. When it does, the instrumentation has to change with it. If the attribute you added six months ago no longer reflects what the code does, you now have misleading data in traces. Not missing data. Wrong data. I would rather have an empty trace than a trace that confidently tells me the wrong thing.

I would not use this in production unless your team treats span attributes with the same review discipline as API contracts. They are observable behavior. When they drift silently they break dashboards, misfire alerts, and send engineers in the wrong direction during the worst possible moment.

There is also the coordination overhead. You need a naming convention the whole team follows. You need someone who knows what sources are registered, what each attribute means, and why it was added. A bit of documentation here, kept up to date, is not glamorous but it is real engineering infrastructure.

On a team of three with two services, nobody will notice if you skip that. On a platform with twenty services and four teams it becomes a governance problem before it becomes an engineering one.

The alternative is debugging production with a green trace and a prayer. I have done both. Pick your poison.

Final thoughts

Auto-instrumentation covers infrastructure. Custom spans cover intent: what the code chose to do, not just what it called.

Patterns to take from here:

One ActivitySource per service, centralized in a static Observability class. Register it by name in AddSource() before you have ten sources.
Span name for the operation kind, attributes for variable data. If it changes per request, it belongs in a tag.
Match ActivityKind to the role. Wrong kind, wrong dashboard.
Inject and extract everywhere your runtime cannot. Queues, SQL comments, file metadata, custom protocols.
Set baggage at the API edge once. It travels to every span in every downstream service automatically.
Review span attributes in the same PR as the business logic they describe. They are observable behavior. When they drift silently they break dashboards.

The portable rule: if it names the kind of thing that happened, it is the span name. If it describes the operation, it is an attribute. If it happened at a specific moment inside the operation, it is an event.

What is the difference between a span event and a span attribute? Attributes describe the span as a whole. Events are timestamped points inside the span with their own attributes. Use an attribute for forecast.days = 5. Use an event for forecast.cold_days_detected at 12ms with cold.day_count = 3. The event tells you when inside the span the thing happened.

What does ActivityKind actually change? It changes which dashboards your span shows up in. Backends that derive metrics from spans emit different time series for Server vs Client vs Producer vs Consumer. Internal appears only inside traces. Wrong kind, wrong dashboard.

How do I propagate trace context through RabbitMQ? Use Propagators.DefaultTextMapPropagator.Inject on the producer side, passing Activity.Current!.Context and Baggage.Current. On the consumer side use Extract with a getter that reads the message headers, then pass parentContext.ActivityContext to StartActivity and restore Baggage.Current = parentContext.Baggage.

What is baggage and when should I use it? Baggage is a set of key-value pairs the propagator carries across every process boundary automatically. Use it for cross-cutting context: tenant ID, client ID, correlation tokens. Do not put sensitive data in baggage â€” it travels in plain text in headers.

Does this work with .NET Aspire? Yes. Register each ActivitySource via AddSource() inside ConfigureOpenTelemetry() in ServiceDefaults. Aspire does not auto-discover custom sources.

What happens if I put high-cardinality values in the span name? Backends that derive metrics from spans turn the span name into a metric label. Each unique name becomes a unique time series. At production scale this exhausts cardinality limits and causes data drops.

Can I use Activity.Current instead of a local variable? Yes. Activity.Current returns the ambient span for the current async context. Useful in nested calls where you want to attach an attribute without threading the activity reference through every method signature.

Can I get spans from inside a stored procedure? Only on PostgreSQL 16+, using the pg_tracing extension. It reads the traceparent from a SQLCommenter comment and emits OTLP spans for parse, plan, execute, function calls, and triggers. MySQL and SQL Server have no equivalent.

Build an AI-Powered Developer Portal with Backstage and .NET

Borys Generalov — Fri, 01 May 2026 21:17:40 +0000

Build an AI-Powered Developer Portal with Backstage and .NET

Want to apply AI, not just read about it? Most tutorials stop at a "Hello World" chatbot. We are going to build something that actually solves a common engineering headache: stale documentation.

Who this is for

This guide is for platform engineers and .NET developers who need to organize a growing software landscape without forcing teams to manually write YAML files.

What you will build

Source repo: demo-backstage-catalog-generator
Constraint: We use local inference only. No source code ever leaves your machine.

An Internal Developer Portal (IDP) solves this by making the software landscape visible, but only if the data is fresh. Automation is the only way to avoid the "stale metadata" trap.

Want to skip ahead? Check out the complete working demo on
GitHub with all
the code ready to run.

Prerequisites

Before we start, make sure you have the following installed:

.NET SDK 8+
Node.js (includes npx and yarn)
Ollama with the llama3:8b model pulled
A GitHub account (for hosting and deployment)

Why Does an Internal Developer Portal Matter?

An IDP becomes the single source of truth for your engineering organization. It answers critical questions across every role:

Engineers: Which services exist, who owns them, what they do, and where the APIs are.
Team leads and architects: Team composition, squad ownership, and architectural decisions with their rationale.
New joiners: How to get up to speed on a codebase they have never seen.
Platform and operations teams: What is running in production, who is responsible, and the lifecycle status.

Beyond just listing services, a mature IDP centralizes Architecture Decision Records (ADRs). I'd argue those are more valuable than the service catalog itself, because they capture why a decision was made. Without a central place to surface them, they end up in forgotten wiki pages or git repositories nobody checks.

Formulating the Architecture

Here's the plan to extract metadata from source code and present it visually:

Backstage: the UI layer where engineers browse and discover services, APIs, documentation, and team ownership
.NET Core: a CLI tool that scans project folders, extracts metadata, and generates Backstage-compatible YAML
Ollama: runs AI inference locally. No source code leaves the machine, no API costs.
Static hosting: deploy to Netlify, Azure Static Web Apps, or any provider of your choice

Why Ollama? Because it runs locally and you do not want to expose your code
to the AI agents over the public internet. You do not know what and how they
use it for, and it does not feel safe. If your employer finds out, you're
done.

Setting Up the Project Infrastructure

You can either follow along and build everything from scratch, or clone the demo repository to get started immediately.

To clone the demo:

git clone https://github.com/bgener/demo-backstage-catalog-generator.git
cd demo-backstage-catalog-generator

To build from scratch, start by downloading and running the LLM model we will use in this guide:

ollama pull llama3:8b
ollama serve

We use llama3:8b specifically. It is significantly faster for local
inference than the full-size model and produces more consistent, concise
output for our use case. If you have a powerful GPU, feel free to use llama3
instead.

Next, scaffold the baseline .NET services. We’ll create one Web API and one MVC project:

mkdir Backstage-Dev-Portal
cd Backstage-Dev-Portal

dotnet new webapi -n ServiceA
dotnet new mvc -n ServiceB

dotnet new sln -n Backstage-Dev-Portal
dotnet sln add ServiceA/ServiceA.csproj
dotnet sln add ServiceB/ServiceB.csproj

You can replace the default controllers with real logic later. These raw services represent the uncataloged microservices in your organization.

Building a Smart Catalog Generator in .NET

Create the tool and add the required package:

dotnet new console -n ProjectSummarizer
cd ProjectSummarizer
dotnet add package OllamaSharp

Replace Program.cs with the implementation below. The full version is in the demo repository. Here we focus on the key parts.

var ollamaApiClient = new OllamaApiClient(
    new Uri("http://localhost:11434")) { SelectedModel = "llama3:8b" };

var chat = new Chat(ollamaApiClient, systemPrompt:
    "You are a technical documentation assistant. " +
    "You produce concise, YAML-safe summaries of .NET projects. " +
    "Output only plain text, no markdown, no bullet points, no quotes, no colons, no newlines.");

Instead of sending every file to the AI, we only send *.csproj, Program.cs, and the folder structure. This is all the context the model needs.

Prompt sanitization is critical. If your Program.cs contains complex
string literals or nested colons, the AI might pass them through to your YAML,
breaking the Backstage parser. Always sanitize the output before writing the
file.

var sb = new StringBuilder();
sb.AppendLine($"Project: {projectName}");
sb.AppendLine("Folder structure:");
AppendFolderStructure(projectDir, sb, "");
sb.AppendLine(File.ReadAllText(csprojPath));

var programPath = Directory.GetFiles(projectDir, "Program.cs", SearchOption.AllDirectories)
    .FirstOrDefault();
if (programPath != null)
    sb.AppendLine(File.ReadAllText(programPath));

The prompt itself uses few-shot examples to guide the model toward the output format we want:

var prompt = "Summarize the project in 1-2 sentences based on the files provided. " +
             "Do not output anything else. " +
             "Examples of good output: " +
             "REST API service providing weather forecasts with temperature data\n" +
             "ASP.NET MVC application with React frontend for managing todo items\n\n"
             + sb.ToString();

await foreach (var token in chat.SendAsync(prompt, cts.Token))
    summaryBuilder.Append(token);

Finally, each summary is sanitized and assembled into a Backstage-compatible YAML entry:

var summary = summaryBuilder.ToString().Trim()
    .Replace("\n", " ").Replace(":", " -").Replace("\"", "'");

var yamlEntry = $@"
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: {projectName.ToLowerInvariant()}
  description: ""{summary}""
spec:
  type: service
  lifecycle: production
  owner: group:default/engineering";

Run the generator against the target directory (use . if you are already in the project root):

dotnet run --project ProjectSummarizer -- .

You should see the AI streaming its summaries in real time:

Integrating the AI Catalog with Backstage

With the catalog-info.yaml ready, we can integrate it into a Backstage instance. Install Backstage:

npx @backstage/create-app

Follow the prompts to name it dev-portal.

Now point Backstage to your generated catalog file. Open app-config.yaml in the dev-portal directory and add the following under the catalog section:

catalog:
  locations:
    - type: file
      target: ../Backstage-Dev-Portal/catalog-info.yaml
      rules:
        - allow: [Component]

To run it locally:

cd dev-portal
yarn dev

Open http://localhost:3000 in your browser. You should see all your services listed in the Software Catalog with AI-generated summaries visible in the description column.

Deploying the Portal

To host this, build your portal as a static site:

yarn build:static

A static Backstage build is great for read-only catalogs. If you need dynamic
features like authentication, real-time plugin backends, or write
operations, you will need to deploy the full Backstage backend as a Node.js
service instead.

The part that is easy to miss is that a static Backstage portal has no live catalog refresh. The catalog is baked at build time, so there is always a lag between a code change and the portal showing it.

Automating with CI/CD

The catalog only stays current if the generator runs automatically. Here is a GitHub Actions workflow that regenerates summaries on every push to main:

name: Update Backstage Catalog
on:
  push:
    branches: [main]

jobs:
  generate-catalog:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: ‘8.0.x’

      - name: Install and start Ollama
        run: |
          curl -fsSL https://ollama.com/install.sh | sh
          ollama serve &
          sleep 5
          ollama pull llama3:8b

      - name: Generate catalog
        run: dotnet run --project ProjectSummarizer -- "$GITHUB_WORKSPACE"

      - name: Commit updated catalog
        run: |
          git config user.name "github-actions"
          git config user.email "github-actions@github.com"
          git add catalog-info.yaml
          git diff --cached --quiet || git commit -m "chore: regenerate AI catalog summaries"
          git push

CI Performance: Running Ollama in CI uses CPU-only inference by default. A
llama3:8b summary takes about 20-30 seconds per project on a standard GitHub
runner. For a large monorepo, your CI bill will spike. Consider using a
persistent self-hosted runner with a GPU if you scale this.

Final thoughts

Automate the metadata generation where the code lives, and keep the UI (Backstage) as a thin, static client. That is the whole trick. Once those two are separate, the "stale documentation" problem mostly goes away.

Use narrow context: Don't send the whole repo. Files like Program.cs and *.csproj are usually enough context for the model.
Sanitize strictly: AI output is non-deterministic. Always strip colons and newlines before writing to YAML.
Start static: Read-only is much easier to maintain. Add a backend only when you actually need write operations or auth.

I would not use this in production unless I had tested the sanitization logic against a few real codebases first. The Replace calls that strip colons and newlines are intentionally minimal. They break on YAML with complex string values, like connection strings or environment variable blocks.

FAQ

Can I use OpenAI instead of Ollama?
Yes, but you will be sending your source code (or at least your Program.cs) to a third party. Use a local model if security is a concern.

Does this replace README files?
No. It replaces the "Service Directory" that usually lives in a spreadsheet. It points engineers to the README they actually need.

How do I handle project renames?
The generator uses the folder or .csproj name. If you rename them, Backstage will see it as a new component unless you map the identity stably.

I help teams build exactly this kind of internal tooling, from developer portals to platform engineering. See my work or get in touch.