DEV Community: Thang Chung

MCP: Programmatic Tool Calling (Code Mode) with OpenSandbox

Thang Chung — Fri, 03 Apr 2026 12:16:28 +0000

Introduction

Model Context Protocol or MCP enables AI agents to access external systems they cannot reach by default, including authenticated APIs, CI/CD pipelines, live process streams, and IDE integrations. It acts as a structured bridge between the model and real-world environments, allowing controlled interaction with tools and infrastructure.

However, MCP does not automatically make interactions efficient or intelligent. Traditional MCP implementations often inject large JSON payloads into the model context, which increases token consumption and reduces efficiency. MCP also does not eliminate the need for proper tool selection and orchestration; if poorly structured, it can introduce unnecessary abstraction and overhead. In environments where agents can directly execute commands or interact with systems natively, MCP may become redundant, as it primarily serves as an access layer rather than an execution engine.

In many software products, having hundreds of APIs is normal. With a traditional MCP approach, all tool definitions are loaded into the model’s context upfront. This inflates the context window with large OpenAPI schemas (adapt to MCP tools), even when most tools are irrelevant to the current request. The result is unnecessary token consumption, higher cost, slower reasoning, and increased risk of hallucination due to context bloat. In this post, I use the OpenAPI schemas from https://www.openbrewerydb.org/documentation and https://petstore3.swagger.io, load them into application memory, adapt them into MCP tools, and dynamically search and invoke the relevant endpoints.

Statistics shared by Anthropic illustrate the scale of the problem under the current MCP pattern:

Consider a five-server setup:

GitHub: 35 tools (~26K tokens)
Slack: 11 tools (~21K tokens)
Sentry: 5 tools (~3K tokens)
Grafana: 5 tools (~3K tokens)
Splunk: 2 tools (~2K tokens)

That's 58 tools consuming approximately 55K tokens before the conversation even starts. Add more servers like Jira (which alone uses ~17K tokens) and you're quickly approaching 100K+ token overhead. At Anthropic, they've seen tool definitions consume 134K tokens before optimization.

This means that before any meaningful task begins, thousands of tokens may already be consumed simply to describe available OpenAPI tools in many enterprise systems.

To address this inefficiency, Anthropic introduced two optimization techniques: Tool Search Tool and Programmatic Tool Calling (Code Mode). These approaches avoid preloading every tool definition and instead retrieve or execute tools only when required, significantly reducing token overhead.

I have been researching and implementing these techniques in .NET/C# for enterprise use. But this article focuses specifically on applying Code Mode with both a local sandbox and OpenSandbox, exploring how executable tool invocation can replace large static tool context injection.

Code Mode with OpenSandbox and Local Sandbox runners

At a high level, the flow operates as follows. At application startup, all available OpenAPI specifications are discovered and loaded into a tool registry. When a request arrives, the system performs a search over the registered OpenAPI endpoint metadata (treated as tools). The LLM then inspects the selected tool’s schema via get_schema, enabling it to understand the available operations, parameters, and data structures.

Using this schema, the LLM generates Python code that correctly invokes the chosen endpoint. The generated code is sent to a sandbox environment—either a local sandbox or OpenSandbox—through execute. The sandbox executes the code, which issues outbound HTTP requests to the target system via a request client.

After execution, the sandbox returns the raw result to the host. The LLM analyzes this output and converts it into a structured, human-readable response for the end user.

A sandbox is required because the Python script is generated dynamically by the LLM. Executing model-generated code directly in the host environment introduces security risks, including arbitrary file access, network misuse, privilege escalation, and system compromise. A sandbox isolates execution, enforces resource limits, controls outbound network access, and restricts filesystem and process permissions. This containment ensures that generated code can perform necessary API calls without exposing the host system to uncontrolled behavior.

If you want to explore OpenSandbox, refer to the official repository by Alibaba: OpenSandbox

In short:

OpenSandbox is a general-purpose sandbox platform for AI applications, offering multi-language SDKs, unified sandbox APIs, and Docker/Kubernetes runtimes for scenarios like Coding Agents, GUI Agents, Agent Evaluation, AI Code Execution, and RL Training.

OpenSandbox is now listed in the CNCF Landscape.

Walk through Code Mode flow

Access to this >> Mermaid, if you want to see the flow in detail

In the core, there are 3 main tools: search, get_schema, and execute.

[McpServerTool(Name = "search"), Description("Discover tools by query with optional detail, tag filter, and limit. Use before get_schema and execute in a multi-step code-mode workflow.")]
public static DiscoverySearchResponse Search(
  [FromServices] DiscoveryTools discoveryTools,
  [FromServices] UserContext context,
  [FromServices] ILoggerFactory loggerFactory,
  [Description("Search query for tool discovery")] string query,
  [Description("Detail level: Brief (default), Detailed, or Full. Also accepts 0, 1, or 2.")] JsonElement ? detail = null,
  [Description("Optional tag filters. When omitted, all tags are included.")] IReadOnlyList < string > ? tags = null,
  [Description("Optional maximum results. Defaults to server discovery default.")] int ? limit = null) 
  {
      SchemaDetailLevel resolvedDetail = detail.HasValue ? ParseSchemaDetailLevel(detail.Value, SchemaDetailLevel.Brief) : SchemaDetailLevel.Brief;
      DiscoverySearchResponse response = discoveryTools.Search(query, context, resolvedDetail, tags, limit);

  return response;
}

[McpServerTool(Name = "get_schema"), Description("Retrieve input schemas for tool names. Pass a list via toolNames or a single name via name. Default detail is Detailed markdown, and missing tool names are reported.")]
public static SchemaLookupResponse GetSchema(
  [FromServices] DiscoveryTools discoveryTools,
  [FromServices] UserContext context,
  [FromServices] ILoggerFactory loggerFactory,
  [Description("List of tool names to retrieve schemas for")] IReadOnlyList < string > ? toolNames = null,
  [Description("Single tool name shorthand - equivalent to toolNames with one entry")] string ? name = null,
  [Description(
    "Schema verbosity: Brief name-only, Detailed compact markdown, Full full JSON schema. Also accepts 0, 1, or 2.")] JsonElement ? detail = null) 
{
  IReadOnlyList < string > requestedToolNames = ResolveSchemaToolNames(toolNames, name);
  SchemaDetailLevel resolvedDetail = detail.HasValue ?
    ParseSchemaDetailLevel(detail.Value, SchemaDetailLevel.Detailed) :
    SchemaDetailLevel.Detailed;
  SchemaLookupResponse response = discoveryTools.GetSchema(requestedToolNames, context, resolvedDetail);

  return response;
}

[McpServerTool(Name = "execute"), Description("Execute constrained code and return the final result. IMPORTANT: call search, then get_schema, then get_execute_syntax before Execute.The runner will reject code written in the wrong style with an error message.")]
public static async Task < object ? > Execute(
  [Description("Code string written in the syntax returned by get_execute_syntax.")] string code,
  [FromServices] ExecuteTool executeTool,
  [FromServices] ILoggerFactory loggerFactory,
  CancellationToken ct) 
{
    try 
    {
       ExecuteResponse response = await executeTool.ExecuteAsync(code, ct);
       return response.FinalValue;
    }
    catch (Exception ex) when(ex is not OperationCanceledException) {
       logger.LogWarning(ex, "[codemode] {HandlerName} failed: {Error}.", nameof(Execute), ex.Message);
       return $"[Execute error] {ex.Message}";
  }
}

And the sandbox interface (which will support 2 approaches: local sandbox and OpenSandBox) as following

public interface ISandboxRunner
{
    string SyntaxGuide { get; }

    Task<RunnerResult> RunAsync(string code, CancellationToken ct);
}

And with local sandbox implementation:

public sealed class LocalConstrainedRunner : ISandboxRunner
{
    private static readonly Regex AbsoluteHttpUrlRegex = new(
        "https?://[^\\s\"'\\)\\]]+",
        RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.CultureInvariant);

    private readonly TimeSpan timeout;
    private readonly int maxToolCalls;

    // Single source of truth for the URL allowlist. String representations are derived on use.
    private readonly IReadOnlyList<Uri> allowedBaseUris;

    private static readonly JsonSerializerOptions PayloadJsonOptions = new()
    {
        PropertyNameCaseInsensitive = true,
    };

    public LocalConstrainedRunner(
        TimeSpan timeout,
        int maxToolCalls,
        IReadOnlyList<string>? allowedBaseUrls = null)
    {
        this.timeout = timeout;
        this.maxToolCalls = maxToolCalls;
        allowedBaseUris = NormalizeAllowedBaseUris(allowedBaseUrls);
    }

    public string SyntaxGuide
    {
        get
        {
            string defaultBaseUrl = allowedBaseUris.Count > 0
                ? allowedBaseUris[0].AbsoluteUri.TrimEnd('/')
                : string.Empty;

            string baseUrlSection = defaultBaseUrl.Length > 0
                ? $"Use the injected `BASE_URL` variable for API calls. Default `BASE_URL` is \"{defaultBaseUrl}\"."
                : "Use the injected `BASE_URL` variable for API calls. No default URL was discovered from OpenAPI sources.";

            string allowedSection = allowedBaseUris.Count > 0
                ? $"Only call API URLs under configured OpenAPI bases via `BASE_URL`: {string.Join(", ", allowedBaseUris.Select(static u => $"\"{u.AbsoluteUri.TrimEnd('/')}\""))}."
                : "No API base allowlist is currently configured, so URL host restrictions are not enforced.";

            return $$"""
        Runner: local (Python)
        Write pure Python code.
        You may call HTTP APIs directly from Python code when needed.
        A lightweight `requests`-compatible shim is available for basic HTTP requests; set a timeout (for example: timeout=10).
        Prefer assigning the final value to `result`.
        If `result` is not set, captured stdout is returned when available.
        {{baseUrlSection}}
        {{allowedSection}}
        Code mode is isolated from tool-Search tools.
        Do NOT use: SearchTools, CallTool, Search, GetSchema, or Execute in this code.
        Example:
            import requests
            response = requests.get(f"{BASE_URL}/pet/findByStatus", params={"status": "sold"}, timeout=10)
            response.raise_for_status()
            result = response.json()
        The final `result` value (or stdout fallback) is returned as tool output.
        """;
        }
    }

    public async Task<RunnerResult> RunAsync(string code, CancellationToken ct)
    {
        ArgumentException.ThrowIfNullOrWhiteSpace(code);

        // remove for the brevity
        // ...

        using CancellationTokenSource timeoutCts = new(timeout);
        using CancellationTokenSource linkedCts = CancellationTokenSource.CreateLinkedTokenSource(ct, timeoutCts.Token);

        object? finalValue = await ExecutePythonLocallyAsync(code, linkedCts.Token);
        return new RunnerResult(finalValue, 0);
    }

    private bool ContainsForbiddenHardcodedApiUsage(string code)
    {
        // remove for the brevity
        // ...
    }

    private async Task<object?> ExecutePythonLocallyAsync(string code, CancellationToken ct)
    {
        string encodedCode = Convert.ToBase64String(Encoding.UTF8.GetBytes(code));
        string serializedDefaultBaseUrl = JsonSerializer.Serialize(
            allowedBaseUris.Count > 0 ? allowedBaseUris[0].AbsoluteUri.TrimEnd('/') : string.Empty);
        string wrapper = $$"""
import base64
import contextlib
import io
import json
import sys
import traceback
import types
import urllib.error
import urllib.parse
import urllib.request

code = base64.b64decode("{{encodedCode}}".encode("ascii")).decode("utf-8")
BASE_URL = {{serializedDefaultBaseUrl}}

# Install a default opener with explicit headers. Some public APIs reject
# Python's default urllib user-agent and return HTTP 403.
_opener = urllib.request.build_opener()
_opener.addheaders = [
    ("User-Agent", "mcp-experiments-local-runner/1.0"),
    ("Accept", "application/json"),
]
urllib.request.install_opener(_opener)

class _RequestsError(Exception):
    pass

class _RequestsHttpError(_RequestsError):
    pass

class _RequestsResponse:
    def __init__(self, status_code, content, headers):
        self.status_code = status_code
        self.content = content
        self.headers = dict(headers.items()) if headers is not None else {}
        self.text = content.decode("utf-8", errors="replace")

    def json(self):
        if not self.text:
            return None
        return json.loads(self.text)

    def raise_for_status(self):
        if self.status_code >= 400:
            raise _RequestsHttpError(f"HTTP {self.status_code}")

def _append_query(url, params):
    if not params:
        return url

    query = urllib.parse.urlencode(params, doseq=True)
    separator = "&" if "?" in url else "?"
    return f"{url}{separator}{query}"

def _normalize_body(data, json_payload, headers):
    if json_payload is not None:
        headers.setdefault("Content-Type", "application/json")
        return json.dumps(json_payload).encode("utf-8")

    if isinstance(data, str):
        return data.encode("utf-8")

    return data

def _requests_request(method, url, params=None, data=None, json=None, headers=None, timeout=None):
    request_headers = dict(headers or {})
    request_url = _append_query(url, params)
    request_body = _normalize_body(data, json, request_headers)
    request = urllib.request.Request(request_url, data=request_body, headers=request_headers, method=method.upper())

    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return _RequestsResponse(response.getcode(), response.read(), response.headers)
    except urllib.error.HTTPError as ex:
        return _RequestsResponse(ex.code, ex.read(), ex.headers)

requests = types.ModuleType("requests")
requests.RequestException = _RequestsError
requests.HTTPError = _RequestsHttpError
requests.request = _requests_request
requests.get = lambda url, **kwargs: _requests_request("GET", url, **kwargs)
requests.post = lambda url, **kwargs: _requests_request("POST", url, **kwargs)
requests.put = lambda url, **kwargs: _requests_request("PUT", url, **kwargs)
requests.delete = lambda url, **kwargs: _requests_request("DELETE", url, **kwargs)
requests.patch = lambda url, **kwargs: _requests_request("PATCH", url, **kwargs)
sys.modules["requests"] = requests

scope = {
    "BASE_URL": BASE_URL
}
captured_stdout = io.StringIO()
captured_stderr = io.StringIO()
try:
    with contextlib.redirect_stdout(captured_stdout), contextlib.redirect_stderr(captured_stderr):
        exec(code, scope, scope)
    payload = {
        "ok": True,
        "finalValue": scope.get("result"),
        "stdout": captured_stdout.getvalue(),
        "stderr": captured_stderr.getvalue()
    }
except Exception as ex:
    payload = {
        "ok": False,
        "error": str(ex),
        "traceback": traceback.format_exc(),
        "stdout": captured_stdout.getvalue(),
        "stderr": captured_stderr.getvalue()
    }

print(json.dumps(payload, ensure_ascii=False, default=str))
""";

        ProcessStartInfo startInfo = new()
        {
            FileName = "python3",
            Arguments = "-",
            RedirectStandardInput = true,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
            CreateNoWindow = true,
        };

        using Process process = new() { StartInfo = startInfo };
        if (!process.Start())
        {
            throw new InvalidOperationException("Failed to start local Python process.");
        }

        await process.StandardInput.WriteAsync(wrapper.AsMemory(), ct);
        await process.StandardInput.FlushAsync();
        process.StandardInput.Close();

        Task<string> stdoutTask = process.StandardOutput.ReadToEndAsync(ct);
        Task<string> stderrTask = process.StandardError.ReadToEndAsync(ct);
        await process.WaitForExitAsync(ct);

        string stdout = await stdoutTask;
        string stderr = await stderrTask;

        if (process.ExitCode != 0)
        {
            throw new InvalidOperationException($"Local Python execution failed with exit code {process.ExitCode}: {stderr}");
        }

        string? payloadLine = stdout
            .Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
            .LastOrDefault();

        if (string.IsNullOrWhiteSpace(payloadLine))
        {
            throw new InvalidOperationException("Local Python execution produced no parseable output.");
        }

        LocalExecutionPayload? payload = JsonSerializer.Deserialize<LocalExecutionPayload>(payloadLine, PayloadJsonOptions);
        if (payload is null)
        {
            throw new InvalidOperationException("Local Python execution payload is empty.");
        }

        if (!payload.Ok)
        {
            // remove for the brevity
            // ...
        }

        object? finalValue = payload.FinalValue;
        if (finalValue is null && !string.IsNullOrWhiteSpace(payload.Stdout))
        {
            finalValue = payload.Stdout.TrimEnd();
        }

        return finalValue;
    }

    // remove for the brevity
    // ...
}

And the OpenSandbox implementation:

public sealed class OpenSandboxRunner : ISandboxRunner
{
    private static readonly JsonSerializerOptions PayloadJsonOptions = new()
    {
        PropertyNameCaseInsensitive = true,
    };

    private readonly OpenSandboxRunnerOptions options;
    private readonly Func<string, CancellationToken, Task<RunnerResult>> remoteExecutor;

    public string SyntaxGuide =>
        """
        Runner: OpenSandbox (Python)
        Write pure Python code.
        You may call HTTP APIs directly from Python code when needed.
        A lightweight `requests`-compatible shim is available for basic HTTP requests; set a timeout (for example: timeout=10).
        Assign the final value to a variable named exactly `result` (lowercase).
        If `result` is not set, captured stdout is returned when available.
        CRITICAL: Do NOT use `Result`, `RESULT`, or any other casing — only lowercase `result` is captured.
        Do NOT use bare identifiers as section dividers (e.g., `Result # comment`). Use only `#` comments.
        Code mode is fully isolated from tool-Search tools.
        Do NOT use: SearchTools, CallTool, Search, GetSchema, or Execute in this code.
        HTTP example:
            import requests
            response = requests.get("https://api.example.com/data", timeout=10)
            result = response.json()
        Compute example:
            data = [1, 2, 3]
            result = sum(data)
        The final `result` value (or stdout fallback) is returned as tool output.
        """;


    public OpenSandboxRunner(
        OpenSandboxRunnerOptions options,
        Func<string, CancellationToken, Task<RunnerResult>>? remoteExecutor = null)
    {
        ArgumentNullException.ThrowIfNull(options);
        ArgumentNullException.ThrowIfNull(loggerFactory);

        if (string.IsNullOrWhiteSpace(options.Domain))
        {
            throw new InvalidOperationException("OpenSandbox:Domain is required when CodeMode:Runner=opensandbox.");
        }

        this.options = options;
        this.remoteExecutor = remoteExecutor ?? ((code, ct) => ExecuteInSandboxAsync(this.options, this.logger, code, ct));
    }

    public async Task<RunnerResult> RunAsync(string code, CancellationToken ct)
    {
        // Code mode must stay isolated from tool-Search meta-tools.
        if (SandboxCodeGuard.ContainsForbiddenMetaToolUsage(code))
        {
            throw new InvalidOperationException(
            "Code mode is isolated from tool-Search tools. " +
            "Do not call SearchTools, CallTool, Search, GetSchema, or Execute inside code mode; use pure Python compute only.");
        }

        try
        {
            using CancellationTokenSource timeoutCts = new(options.Timeout);
            using CancellationTokenSource linkedCts = CancellationTokenSource.CreateLinkedTokenSource(ct, timeoutCts.Token);

            RunnerResult remoteResult = await remoteExecutor(code, linkedCts.Token);
            return remoteResult;
        }
        catch (OperationCanceledException ex) when (!ct.IsCancellationRequested)
        {
            // remove for the brevity
        }
        catch (Exception ex) when (!ct.IsCancellationRequested)
        {
            // remove for the brevity
        }
    }

    private static async Task<RunnerResult> ExecuteInSandboxAsync(
        OpenSandboxRunnerOptions options,
        ILogger logger,
        string code,
        CancellationToken ct)
    {
        var config = ConnectionConfigBuilder.Build(options);
        string encodedCode = Convert.ToBase64String(Encoding.UTF8.GetBytes(code));
        string command = BuildPythonCommand(encodedCode);

        // Suppress HTTP auto-instrumentation for SDK internal polling and RPC chatter.
        using var suppressScope = SuppressInstrumentationScope.Begin();

        // Retry sandbox creation with exponential backoff
        await using Sandbox sandbox = await RetryHelper.RetryAsync(
            async () => await Sandbox.CreateAsync(new SandboxCreateOptions
            {
                ConnectionConfig = config,
                Image = options.Image,
                TimeoutSeconds = options.TimeoutSeconds,
                ReadyTimeoutSeconds = options.ReadyTimeoutSeconds,
            }).WaitAsync(ct),
            maxAttempts: 3,
            initialDelay: TimeSpan.FromSeconds(0.5),
            cancellationToken: ct);

        Execution execution;
        try
        {
            // Retry sandbox command execution
            execution = await RetryHelper.RetryAsync(
                async () => await sandbox.Commands.RunAsync(command, cancellationToken: ct).WaitAsync(ct),
                maxAttempts: 2,
                initialDelay: TimeSpan.FromSeconds(0.5),
                cancellationToken: ct);
        }
        finally
        {
            try
            {
                await sandbox.KillAsync();
                logger.LogInformation("OpenSandbox instance killed.");
            }
            catch (Exception killEx)
            {
                logger.LogWarning(killEx, "Failed to kill OpenSandbox instance cleanly.");
            }
        }

        if (execution.Error is not null)
        {
            throw new InvalidOperationException($"OpenSandbox execution failed: {execution.Error.Name}: {execution.Error.Value}");
        }

        string stdout = string.Join(
            '\n',
            execution.Logs.Stdout.Select(message => message.Text).Where(text => !string.IsNullOrWhiteSpace(text)));

        string? payloadLine = stdout
            .Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
            .LastOrDefault();

        if (string.IsNullOrWhiteSpace(payloadLine))
        {
            throw new InvalidOperationException("OpenSandbox execution produced no parseable output.");
        }

        SandboxExecutionPayload? payload = JsonSerializer.Deserialize<SandboxExecutionPayload>(payloadLine, PayloadJsonOptions);
        if (payload is null)
        {
            throw new InvalidOperationException("OpenSandbox execution payload is empty.");
        }

        if (!payload.Ok)
        {
            throw new InvalidOperationException(
                $"OpenSandbox script error: {payload.Error ?? "Unknown error"}. {payload.Traceback} {payload.Stderr}");
        }

        object? finalValue = payload.FinalValue;
        if (finalValue is null && !string.IsNullOrWhiteSpace(payload.Stdout))
        {
            finalValue = payload.Stdout.TrimEnd();
        }

        return new RunnerResult(finalValue, 0);
    }



    private static string BuildPythonCommand(string encodedCode)
    {
        return $$"""
            python3 - <<'PY'
            import base64
            import contextlib
            import io
            import json
            import sys
            import traceback
            import types
            import urllib.error
            import urllib.parse
            import urllib.request

            code = base64.b64decode("{{encodedCode}}".encode("ascii")).decode("utf-8")

            # Install a default opener with explicit headers. Some public APIs reject
            # Python's default urllib user-agent and return HTTP 403.
            _opener = urllib.request.build_opener()
            _opener.addheaders = [
                ("User-Agent", "mcp-experiments/1.0"),
                ("Accept", "application/json"),
            ]
            urllib.request.install_opener(_opener)

            class _RequestsError(Exception):
                pass

            class _RequestsHttpError(_RequestsError):
                pass

            class _RequestsResponse:
                def __init__(self, status_code, content, headers):
                    self.status_code = status_code
                    self.content = content
                    self.headers = dict(headers.items()) if headers is not None else {}
                    self.text = content.decode("utf-8", errors="replace")

                def json(self):
                    if not self.text:
                        return None
                    return json.loads(self.text)

                def raise_for_status(self):
                    if self.status_code >= 400:
                        raise _RequestsHttpError(f"HTTP {self.status_code}")

            def _append_query(url, params):
                if not params:
                    return url

                query = urllib.parse.urlencode(params, doseq=True)
                separator = "&" if "?" in url else "?"
                return f"{url}{separator}{query}"

            def _normalize_body(data, json_payload, headers):
                if json_payload is not None:
                    headers.setdefault("Content-Type", "application/json")
                    return json.dumps(json_payload).encode("utf-8")

                if isinstance(data, str):
                    return data.encode("utf-8")

                return data

            def _requests_request(method, url, params=None, data=None, json=None, headers=None, timeout=None):
                request_headers = dict(headers or {})
                request_url = _append_query(url, params)
                request_body = _normalize_body(data, json, request_headers)
                request = urllib.request.Request(request_url, data=request_body, headers=request_headers, method=method.upper())

                try:
                    with urllib.request.urlopen(request, timeout=timeout) as response:
                        return _RequestsResponse(response.getcode(), response.read(), response.headers)
                except urllib.error.HTTPError as ex:
                    return _RequestsResponse(ex.code, ex.read(), ex.headers)

            requests = types.ModuleType("requests")
            requests.RequestException = _RequestsError
            requests.HTTPError = _RequestsHttpError
            requests.request = _requests_request
            requests.get = lambda url, **kwargs: _requests_request("GET", url, **kwargs)
            requests.post = lambda url, **kwargs: _requests_request("POST", url, **kwargs)
            requests.put = lambda url, **kwargs: _requests_request("PUT", url, **kwargs)
            requests.delete = lambda url, **kwargs: _requests_request("DELETE", url, **kwargs)
            requests.patch = lambda url, **kwargs: _requests_request("PATCH", url, **kwargs)
            sys.modules["requests"] = requests

            scope = {}
            captured_stdout = io.StringIO()
            captured_stderr = io.StringIO()
            try:
                with contextlib.redirect_stdout(captured_stdout), contextlib.redirect_stderr(captured_stderr):
                    exec(code, scope, scope)
                payload = {
                    "ok": True,
                    "finalValue": scope.get("result"),
                    "stdout": captured_stdout.getvalue(),
                    "stderr": captured_stderr.getvalue()
                }
            except BaseException as ex:
                payload = {
                    "ok": False,
                    "error": str(ex),
                    "traceback": traceback.format_exc(),
                    "stdout": captured_stdout.getvalue(),
                    "stderr": captured_stderr.getvalue()
                }

            try:
                print(json.dumps(payload, ensure_ascii=False, default=str))
            except Exception as serialization_ex:
                fallback_payload = {
                    "ok": False,
                    "error": f"Failed to serialize execution payload: {serialization_ex}",
                    "traceback": traceback.format_exc(),
                    "stdout": captured_stdout.getvalue(),
                    "stderr": captured_stderr.getvalue()
                }
                print(json.dumps(fallback_payload, ensure_ascii=False, default=str))
            PY
            """;
    }
}

Test it all

The steps below shall run OpenSandbox runner by Alibaba.

At the root project run:

dotnet run --project src/AppHost/AppHost.csproj

Open the testweb project:

Now, let's input Use code mode to find breweries with moon in the name, return the top 5 as name and city

Back to Aspire dashboard logs, we should see some important logs:

...
info: McpServer.CodeMode.CodeModeHandlers[0]
      [codemode] Search returned 10 tools out of 27 matches for query brewery search by name: get_user_by_name, brewery_search, search_breweries, get_single_brewery, brewery_get, brewery_list, find_pets_by_tags, get_random_brewery, brewery_random, find_pets_by_status.
...
info: McpServer.CodeMode.CodeModeHandlers[0]
      [codemode] GetSchema returned 1 schemas for tools search_breweries. Detail: Detailed. Missing: (none).
...
info: McpServer.CodeMode.CodeModeHandlers[0]
      [codemode] Execute executing code (253 chars).
...
info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       Generated Python code for OpenSandbox Execute:

       import requests

       response = requests.get(
           "https://api.openbrewerydb.org/v1/breweries/search",
           params={"query": "moon", "per_page": 5},
           timeout=10
       )
       data = response.json()
       result = [{"name": b["name"], "city": b["city"]} for b in data[:5]]

 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       Sending generated Python code to OpenSandbox server:

       import requests

       response = requests.get(
           "https://api.openbrewerydb.org/v1/breweries/search",
           params={"query": "moon", "per_page": 5},
           timeout=10
       )
       data = response.json()
       result = [{"name": b["name"], "city": b["city"]} for b in data[:5]]

 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       Creating OpenSandbox instance for domain localhost:8080 with image python:3.12-slim.
...

  OpenSandbox info:     2026-04-03 10:57:15+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=inspect image python:3.12-slim | duration=29.24
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=create sandbox container | duration=108.47
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=ensure directory /opt/opensandbox | duration=21.35
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=copy execd archive to sandbox | duration=246.19
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=ensure directory /opt/opensandbox | duration=19.67
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=install bootstrap script | duration=11.49
  OpenSandbox info:     2026-04-03 10:57:16+0000 [7af6d2f6e67b46a3b4ad911d949ea9fa] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=start sandbox container | duration=486.26
  OpenSandbox info:     2026-04-03 10:57:16+0000 [-] uvicorn.access: 192.168.117.1:47168 - "POST /v1/sandboxes HTTP/1.1" 202
  OpenSandbox info:     2026-04-03 10:57:16+0000 [-] uvicorn.access: 192.168.117.1:47168 - "GET /v1/sandboxes/50f59a2e-24c4-4eb0-9356-515359352d0f/endpoints/44772?use_server_proxy=true HTTP/1.1" 200
  OpenSandbox info:     2026-04-03 10:57:16+0000 [-] uvicorn.access: 192.168.117.1:47168 - "GET /v1/sandboxes/50f59a2e-24c4-4eb0-9356-515359352d0f/endpoints/18080?use_server_proxy=true HTTP/1.1" 200
...
 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       OpenSandbox instance is ready. Running generated Python command.
...
  OpenSandbox info:     2026-04-03 10:57:16+0000 [-] uvicorn.access: 192.168.117.1:47168 - "GET /sandboxes/50f59a2e-24c4-4eb0-9356-515359352d0f/proxy/44772/ping HTTP/1.1" 200
  OpenSandbox info:     2026-04-03 10:57:16+0000 [-] uvicorn.access: 192.168.117.1:42344 - "POST /sandboxes/50f59a2e-24c4-4eb0-9356-515359352d0f/proxy/44772/command HTTP/1.1" 200
...
 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       OpenSandbox command execution completed.
...
  OpenSandbox info:     2026-04-03 10:57:20+0000 [c4cc5cb1084747e9aebddae865d3bf33] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=kill sandbox container | duration=130.02
  OpenSandbox info:     2026-04-03 10:57:20+0000 [c4cc5cb1084747e9aebddae865d3bf33] src.services.docker: sandbox=50f59a2e-24c4-4eb0-9356-515359352d0f | action=remove sandbox container | duration=22.65
  OpenSandbox info:     2026-04-03 10:57:20+0000 [-] uvicorn.access: 192.168.117.1:47168 - "DELETE /v1/sandboxes/50f59a2e-24c4-4eb0-9356-515359352d0f HTTP/1.1" 204
...
 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       OpenSandbox instance killed.
 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]
       OpenSandbox execution raw output: {"ok": true, "finalValue": [{"name": "Moon Under Water Brewery", "city": "Victoria"}, {"name": "Moon Dog Craft Brewery", "city": "Abbotsford"}, {"name": "Moon Tower Sudworks", "city": "Houston"}, {"name": "Moon Hill Brewing Co., Inc.", "city": "Gardner"}, {"name": "Moon River Brewing Co", "city": "Savannah"}], "stdout": "", "stderr": ""}
 info: McpServer.CodeMode.OpenSandbox.OpenSandboxRunner[0]

You can see LLM model generates the Python code and send to OpenSandbox for execution:

""" with the prompt: Use code mode to find breweries with moon in the name, return the top 5 as name and city
The code generated by LLM model as: 
"""
import requests

response = requests.get(
   "https://api.openbrewerydb.org/v1/breweries/search",
   params={"query": "moon", "per_page": 5},
   timeout=10)

data = response.json()
result = [{"name": b["name"], "city": b["city"]} for b in data[:5]]

Then on the UI:

And even more complex query like:

Write Python code for execute mode.

Requirements:
1) Get breweries in San Diego.
2) Return the total number of breweries found for San Diego.
3) Search for breweries matching "moon" and return top 5 with only:
   - name
   - city
4) Also compute a San Diego summary:
   - count by brewery_type
   - top 3 cities by number of breweries (if city is missing, use "Unknown")
5) Deduplicate breweries by id before counting.
6) Sort outputs deterministically:
   - type summary: count desc, then type asc
   - top cities: count desc, then city asc
   - moon top 5: name asc
7) Return one JSON object in `result` with this exact shape:
{
  "san_diego_total": number,
  "san_diego_type_summary": [{"brewery_type": string, "count": number}],
  "san_diego_top_cities": [{"city": string, "count": number}],
  "moon_top_5": [{"name": string, "city": string}]
}
8) If any tool call fails, set a top-level "errors" array with readable messages but still return partial data.

"""
The code generated by LLM model as: 
"""
import requests
from collections import defaultdict

errors = []
BASE = "https://api.openbrewerydb.org/v1/breweries"

# --- 1. Fetch all San Diego breweries (paginate) ---
sd_breweries = {}
page = 1
while True:
   try:
       r = requests.get(BASE, params={"by_city": "san_diego", "per_page": 200, "page": page}, timeout=10)
       r.raise_for_status()
       page_data = r.json()
   except Exception as e:
       errors.append(f"San Diego fetch page {page} failed: {e}")
       break
   if not page_data:
       break
   for b in page_data:
       bid = b.get("id")
       if bid and bid not in sd_breweries:
           sd_breweries[bid] = b
   if len(page_data) < 200:
       break
   page += 1

sd_list = list(sd_breweries.values())
san_diego_total = len(sd_list)

# --- 4a. Count by brewery_type ---
type_counts = defaultdict(int)
for b in sd_list:
   type_counts[b.get("brewery_type") or "unknown"] += 1

san_diego_type_summary = sorted(
   [{"brewery_type": t, "count": c} for t, c in type_counts.items()],
   key=lambda x: (-x["count"], x["brewery_type"])
)

# --- 4b. Top 3 cities ---
city_counts = defaultdict(int)
for b in sd_list:
   city = b.get("city") or "Unknown"
   city_counts[city] += 1

all_cities_sorted = sorted(
   [{"city": c, "count": n} for c, n in city_counts.items()],
   key=lambda x: (-x["count"], x["city"])
)
san_diego_top_cities = all_cities_sorted[:3]

# --- 3. Search for "moon" breweries ---
moon_top_5 = []
try:
   r = requests.get(f"{BASE}/search", params={"query": "moon", "per_page": 50}, timeout=10)
   r.raise_for_status()
   moon_data = r.json()
   moon_deduped = {}
   for b in moon_data:
       bid = b.get("id")
       if bid and bid not in moon_deduped:
           moon_deduped[bid] = b
   moon_sorted = sorted(moon_deduped.values(), key=lambda x: x.get("name") or "")
   moon_top_5 = [{"name": b["name"], "city": b.get("city") or ""} for b in moon_sorted[:5]]
except Exception as e:
   errors.append(f"Moon search failed: {e}")

result = {
   "san_diego_total": san_diego_total,
   "san_diego_type_summary": san_diego_type_summary,
   "san_diego_top_cities": san_diego_top_cities,
   "moon_top_5": moon_top_5,
}
if errors:
   result["errors"] = errors

And the response UI:

The code executed in the local sandbox should mirror the same behavior as in OpenSandbox to ensure consistency during development and testing. However, the local sandbox setup is strictly for development purposes. It must not be used in a production environment.

Conclusion

Programmatic Tool Calling (Code Mode) with OpenSandbox significantly reduces token usage by avoiding large upfront tool schema injection. Instead of loading every API definition into the context window, the LLM generates a Python script that executes inside a sandbox, performs the required API calls, and returns only the relevant result. This approach is especially effective for complex queries and multi-step operations that are difficult to express through traditional MCP-style tool invocation.

However, this model introduces new risks. Prompt injection becomes more dangerous when the model is allowed to generate executable code. The generated script may attempt to access sensitive files, exfiltrate secrets, or communicate with unauthorized external hosts. Without strict controls, execution can become unpredictable. For this reason, code must run inside a sandbox with explicit network whitelisting, filesystem restrictions, resource limits, and controlled environment variables. Execution isolation is mandatory, not optional.

I will continue exploring this approach in the coming weeks. I will present a deeper technical breakdown of Tool Search Tool and Code Mode at AI Tinkerers in Ho Chi Minh City on April 18, 2026.

Disclaimer:

The referenced codebase was generated primarily using GitHub Copilot in Visual Studio Code. Minimal manual coding was required, but debugging and architectural correction were necessary (human). Early generations contained confusing control flows and suboptimal structure. Manual debugging and iterative steering were required to align the system with the intended architecture (human).
Letting AI generate and execute code autonomously often leads to architectural drift, lack of optimization, or incorrect abstractions. Explicit guardrails are required (human).

Embedding principles such as KISS, YAGNI, DRY, Boy Scout Rule, and TDD (red–green cycle) into copilot-instructions.md materially improves output quality.
AI-native development enables building ideas that were previously constrained by time or knowledge. However, overreliance without verification can distort judgment and engineering rigor. The responsibility for correctness, architecture, and security remains with the engineer (human).

At times, though, it leaves me feeling what some describe as "AI psychosis." (https://www.youtube.com/watch?v=kwSVtQ7dziU)

All source code can be found at https://github.com/thangchung/agent-engineering-experiment/tree/main/mcp-experiments

References:

Auto-Scaling ComfyUI-API and ComfyUI on Azure Container Apps

Thang Chung — Mon, 16 Feb 2026 17:08:26 +0000

Introduction

Last month, I published “Auto-Scaling ComfyUI-API and ComfyUI: Orchestrating GPU Workloads with Azure Kubernetes Service and KEDA” post. That work focused on cluster-level control, event-based autoscaling, and production-grade GPU orchestration.

After reviewing the official GPU workload guidance for Azure Container Apps, I evaluated whether the same ComfyUI and ComfyUI-API containers could run effectively in a fully managed, serverless container model—without owning cluster operations.

I implemented and validated this approach over several days. The result is a clean deployment model for running ComfyUI and a companion ComfyUI-API backend on Azure Container Apps with GPU-enabled workload profiles.

The architecture removes VM lifecycle management, node pool configuration, and CUDA driver maintenance from your responsibility. You package ComfyUI and the API as Docker images, push them to a container registry, and deploy them to Azure Container Apps configured with GPU profiles. Azure provisions NVIDIA GPUs on demand and scales replicas based on traffic. When idle, the environment can scale to zero, eliminating unnecessary GPU cost.

This produces a production-ready, HTTP-accessible ComfyUI platform that:

Exposes REST endpoints for automated image generation through ComfyUI-API
Utilises NVIDIA T4-class GPUs for inference acceleration
Scales dynamically based on workload
Scales to zero when inactive to control cost

This guide focuses on the concrete deployment steps and the API-level submission required to run ComfyUI-API/ComfyUI on Azure Container Apps with GPU support for image-generation workloads.

Create Azure Container Apps (ACA) on Azure Portal

Go to the https://portal.azure.com, and do some authentication with your Azure account.

Step 1:

(1): Input your Azure resource group
(2): Your ACA name
(3): The region you want to host your ACA

Step 2:

(1) Input the Registry Server, in this case, ghcr.io and input the image and tag, in this case, ghcr.io/thangchung/agent-engineering-experiment/comfyui-api:qwenvl-1
(2) We need to choose GPU, and the GPU type is Consumption - GPU NC8as-T4

Step 3:

(1) We need to enable Ingress
(2) And Accepting traffic from anywhere - because of dev/test
(3) Target port, in this case 3000, because our container is exposed at port 3000

Step 4:

Finally, we click the Create button and wait a bit for ACA provisioning.

Test it out

Go to https://<your aca name>.westus2.azurecontainerapps.io/docs, you should see:

You should input some prompt (Happy New Year of the Fire Horse! The image depicts a white horse leaping into the sky.) as shown in the picture below.

Then, click the execute button, wait a bit, then go to https://webhook.site/1ec2e965-3d92-4e2d-9714-ce7368b80995, you should see all the webhook requests, pick base64, and convert it to a picture, then you will see:

Bonus: .NET Aspire on ACA

(1) At the Overview page, you can enable Aspire Dashboard.

And now you can see Resources, Console logs, Traces, and Metrics of the ACA resources you run.

How cool is that? <3

Conclusion

This experiment confirms that ComfyUI and ComfyUI-API run reliably on Azure Container Apps with GPU-enabled workload profiles. The containers deploy cleanly, GPU allocation is handled transparently, ingress is straightforward, and scale behaviour aligns with real image generation demand. There is no cluster management, no node pool tuning, and no driver maintenance. The operational surface area is significantly smaller than a Kubernetes-based approach while still delivering GPU acceleration, HTTP exposure, and scale-to-zero cost control.

For teams that need GPU-backed inference without owning orchestration complexity, this model is viable and production-ready. The trade-off is reduced low-level control compared to AKS, but the gain in simplicity and operational efficiency is substantial.

Happy New Lunar Year 2026. May the Year of the Horse bring speed, endurance, and decisive execution.

AI-Driven Software Architecture Diagram Generation: Automating Excalidraw and draw.io with MCP Apps

Thang Chung — Fri, 13 Feb 2026 08:57:14 +0000

Introduction

In my role as a Solutions Architect and consultant, I create architecture diagrams constantly to communicate high-level system design, software flows, and component interactions.

Previously, the process was entirely manual. I had to read documentation, review source code, analyze the current state of products or projects, and then translate that understanding into diagrams using draw.io or Excalidraw. The work is intellectually engaging, but it is time-consuming and demands precision for every element and connection inside the web editor.

Last week, I came across two MCP app tools introduced in the architecture community: drawio-mcp and excalidraw-mcp. They immediately captured my attention. I wanted to understand how far they could go in supporting real architecture workflows.

After several days of hands-on use, the impact was clear. These tools fundamentally change how diagrams can be created and evolved. The shift from manual drawing to prompt-driven, programmatic diagram generation is substantial.

This post provides a practical guide to setting up and running simple prompts to visualize a hypothetical TODO application. The goal is to help developers and architects quickly get started with drawio-mcp and excalidraw-mcp and understand their real potential in day-to-day architecture work.

Prerequisites

MacOS
vscode (I used 1.109.3)
GitHub Copilot in Agent mode (Claude Opus 4.5)
Any source codes

Setup drawio-mcp and excalidraw-mcp on vscode

Press Shift + Command + P on the vscode, then type mcp to search, and choose add MCP server as following:

The choose HTTP option:

Next, assign a file name of your choice and repeat the same process with both the drawio-mcp and excalidraw-mcp servers.

The result should resemble the following:

That is the setup. Now proceed with experimentation.

In VS Code, enter read the project structure and visualise all main components using excalidraw in the GitHub Copilot chat window. After a short processing period, the generated diagram will appear:

Focus on the red box. It reads the entire source code, sends it to the LLM, and then uses the Excalidraw MCP tools to generate the diagram. This ensures the output reflects the actual system structure rather than a rough interpretation.

Ref Todo App architecture using Excalidraw

Now switch to draw.io by typing: now with draw.io pls:

Ref: Todo App architecture using draw.io

Conclusion

MCP app tools redefine how architecture diagrams are created and maintained. By allowing AI to read source code, infrastructure definitions, and system constraints, diagrams can be generated and updated programmatically rather than drawn manually.

This approach shifts architecture from static documentation to an adaptive, automation-ready asset. It provides a practical blueprint for integrating AI directly into architecture design, improving accuracy, consistency, and delivery speed across projects.

Happy hacking!!!

Auto-Scaling ComfyUI-API and ComfyUI: Orchestrating GPU Workloads with Azure Kubernetes Service and KEDA

Thang Chung — Wed, 21 Jan 2026 16:49:06 +0000

Introduction

I am currently working on a project that relies heavily on multiple AI features built on Stable Diffusion models. The AI team uses ComfyUI and ComfyUI-api as the workflow orchestrator for Stable Diffusion. This allows them to design and optimise workflows visually in ComfyUI, then export each AI feature as a workflow definition (in JSON format), for example:
https://github.com/SaladTechnologies/comfyui-api/blob/main/example-workflows/sd1.5/txt2img.json

All model training and fine-tuning work (including LLM and VLM models) is currently performed on GPU-enabled virtual machines.

After joining the team as a Solutions Architect, I worked closely with the backend engineers to refactor large parts of the application layer. This included applying the Asynchronous Request–Reply pattern, agent workflow orchestration patterns, webhooks, Azure Blob Storage integration, and other architectural improvements. As a result, the application tier now scales significantly better.

However, a major bottleneck remained: all AI workloads were still running on a single GPU VM, exposed via docker-compose up. The same machine was used for training, fine-tuning, and inference, creating tight coupling, poor isolation, and limited scalability.

Since this domain was relatively new to me, I spent time surveying the ecosystem for tools that could support scalable inference. I evaluated Ollama, Foundry Local, vLLM, KServe, and KAITO. In practice, none of these solutions fit the requirement, because ComfyUI and ComfyUI-api are not simple LLM inference endpoints—they are a workflow-based Stable Diffusion orchestrator with their own API server.

In the end, I settled on a straightforward, production-ready solution: Azure Kubernetes Service (AKS) with KEDA enabled. I spent multiple nights provisioning, deploying, validating, and stress-testing each component.

This blog consolidates that deployment work. I will walk through:

Provisioning an AKS cluster with GPU nodes (Standard_NC4as_T4_v3)
Containerising ComfyUI and ComfyUI-api, including workflow and model downloads
Enabling KEDA-based autoscaling with HTTP add-on
Demonstrating the final results

Let’s get started.

Build the comfyui-api execution file

My working environment is Windows 11 + WSL2 with Ubuntu 24

First of all, you need to check out the comfyui-api to build the execution file.

git clone https://github.com/SaladTechnologies/comfyui-api.git
cd comfyui-api
npm install
npm run build-binary

Dockerize the comfyui and comfyui-api app

Package the comfyui and comfyui-api as well as copy all workflows and pull all LLM + VLM models (in this case, dreamshaper5.safetensors, dreamshaper_8.safetensors, and Qwen 3 VL) to the Dockerfile.

# Custom ComfyUI API image with QwenVL support
# Base image: comfyui-api with ComfyUI 0.7.0, API 1.16.1, PyTorch 2.8.0, CUDA 12.8
FROM ghcr.io/saladtechnologies/comfyui-api:comfy0.7.0-api1.16.1-torch2.8.0-cuda12.8-runtime

# Set environment variables
ENV WORKFLOW_DIR=/workflows
ENV STARTUP_CHECK_MAX_TRIES=30

# Install ComfyUI-QwenVL custom node
WORKDIR /opt/ComfyUI/custom_nodes
RUN git clone https://github.com/1038lab/ComfyUI-QwenVL && \
    pip install --no-cache-dir -r ComfyUI-QwenVL/requirements.txt

# Optional: Install llama-cpp-python for GGUF support (uncomment if needed)
# RUN pip install --no-cache-dir llama-cpp-python

# Copy workflows into the image
COPY example-workflows/sd1.5 /workflows

# Copy the comfyui-api binary
COPY bin/comfyui-api /app/comfyui-api
RUN chmod +x /app/comfyui-api

# Copy the model download script (downloads models at runtime)
COPY docker/download-models.sh /app/download-models.sh
RUN chmod +x /app/download-models.sh

# Set working directory
WORKDIR /app

# Run the entrypoint script (downloads models then starts API)
CMD ["/app/download-models.sh"]

And download-models.sh:

#!/bin/bash
set -e

MODELS_DIR="/opt/ComfyUI/models/checkpoints"
mkdir -p "$MODELS_DIR"

# Download dreamshaper_8 if not exist
if [ ! -f "$MODELS_DIR/dreamshaper_8.safetensors" ]; then
  echo "Downloading dreamshaper_8.safetensors..."
  wget -q --show-progress -O "$MODELS_DIR/dreamshaper_8.safetensors" \
    "https://civitai.com/api/download/models/128713?type=Model&format=SafeTensor&size=pruned&fp=fp16"
  echo "✓ dreamshaper_8.safetensors downloaded"
else
  echo "✓ dreamshaper_8.safetensors already exists, skipping download"
fi

# Download dreamshaper5 if not exist
if [ ! -f "$MODELS_DIR/dreamshaper5.safetensors" ]; then
  echo "Downloading dreamshaper5.safetensors..."
  wget -q --show-progress -O "$MODELS_DIR/dreamshaper5.safetensors" \
    "https://huggingface.co/Lykon/DreamShaper/resolve/main/DreamShaper_5_beta2_noVae_half_pruned.safetensors?download=true"
  echo "✓ dreamshaper5.safetensors downloaded"
else
  echo "✓ dreamshaper5.safetensors already exists, skipping download"
fi

echo "All models ready. Starting comfyui-api..."

# Start the API server
exec /app/comfyui-api

The reason we use CMD ["/app/download-models.sh"] because we don't want to pull all the big models into a Docker image, which makes the image really big and start-up time is slow sequentially.

Now you go to GitHub developer settings to create a personal access token.

Then, you can build and push the Docker image to a GitHub artefact like

docker login ghcr.io -u thangchung # it will ask for the GitHub access token just created
docker build -t ghcr.io/thangchung/agent-engineering-experiment/comfyui-api:qwenvl-1 -f docker/qwenvl.dockerfile .
docker push ghcr.io/thangchung/agent-engineering-experiment/comfyui-api:qwenvl-1

See the file I have pushed before at https://github.com/thangchung/agent-engineering-experiment/pkgs/container/agent-engineering-experiment%2Fcomfyui-api/650944015?tag=qwenvl-1

Setup AKS cluster with GPU in-place

We follow the guidance at https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu?tabs=add-ubuntu-gpu-node-pool#manually-install-the-nvidia-device-plugin to set up the AKS cluster (remember we need to have a node pool with a GPU, which will run the AI workload, and another node pool is the normal one with CPU, and we will use this normal node pool to run the rest of the workloads).

And the node pool with GPU (because we are experimenting, so we use Standard_NC4as_T4_v3, in reality, we might need to consider the bigger one, and we also need to request Microsoft Azure supporting team for more quota to be able to create this GPU option):

Run comfyui and comfyui-api with normal workload on AKS

After you finish provisioning the AKS cluster, just run

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: comfyui
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: comfyui-api
  namespace: comfyui
  labels:
    app: comfyui-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: comfyui-api
  template:
    metadata:
      labels:
        app: comfyui-api
    spec:
      containers:
        - name: comfyui-api
          # GitHub Container Registry image (includes download-models.sh entrypoint)
          image: ghcr.io/thangchung/agent-engineering-experiment/comfyui-api:qwenvl-latest
          imagePullPolicy: Always
          ports:
            - name: api
              containerPort: 3000
              protocol: TCP
            - name: comfyui
              containerPort: 8188
              protocol: TCP
          env:
            - name: LOG_LEVEL
              value: "debug"
            - name: WORKFLOW_DIR
              value: "/workflows"
            - name: STARTUP_CHECK_MAX_TRIES
              value: "30"
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "16Gi"
              cpu: "4"
            requests:
              nvidia.com/gpu: 1
              memory: "8Gi"
              cpu: "2"
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
      # GPU node pool tolerations
      tolerations:
        - key: "sku"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"
      # Schedule on GPU nodes
      nodeSelector:
        accelerator: nvidia
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: comfyui-api-service
  namespace: comfyui
  labels:
    app: comfyui-api
spec:
  type: LoadBalancer
  ports:
    - name: api
      port: 3000
      targetPort: 3000
      protocol: TCP
    - name: comfyui
      port: 8188
      targetPort: 8188
      protocol: TCP
  selector:
    app: comfyui-api
EOF

Now you can port-forward the comfyui-api POD and do some curl to test it, but wait a minute, we will do the test when we have auto-scaling in place. Now moving on.

Run autoscaling comfyui and comfyui-api on AKS and KEDA

But with the really expensive GPU workload, we need to think about how we can save the cost if no one uses it. So we can set up the KEDA to auto scale to zero incase no traffic usage.

To be able to do auto scaling, we need to install KEDA:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

Enable KEDA HTTP add-on (because we are based on the HTTP traffic to scale it up and down):

helm install http-add-on kedacore/keda-add-ons-http --namespace keda

Finally, we can run the app with KEDA enabled:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: comfyui
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: comfyui-api
  namespace: comfyui
  labels:
    app: comfyui-api
spec:
  replicas: 0  # KEDA controls replicas
  selector:
    matchLabels:
      app: comfyui-api
  template:
    metadata:
      labels:
        app: comfyui-api
    spec:
      containers:
        - name: comfyui-api
          # GitHub Container Registry image (includes download-models.sh entrypoint)
          image: ghcr.io/thangchung/agent-engineering-experiment/comfyui-api:qwenvl-1
          imagePullPolicy: Always
          ports:
            - name: api
              containerPort: 3000
              protocol: TCP
            - name: comfyui
              containerPort: 8188
              protocol: TCP
          env:
            - name: LOG_LEVEL
              value: "debug"
            - name: WORKFLOW_DIR
              value: "/workflows"
            - name: STARTUP_CHECK_MAX_TRIES
              value: "30"
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "16Gi"
              cpu: "4"
            requests:
              nvidia.com/gpu: 1
              memory: "8Gi"
              cpu: "2"
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
      # GPU node pool tolerations
      tolerations:
        - key: "sku"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"
      # Schedule on GPU nodes
      nodeSelector:
        accelerator: nvidia
      restartPolicy: Always
---
# Internal service for KEDA HTTP Add-on to route traffic to
apiVersion: v1
kind: Service
metadata:
  name: comfyui-api-service
  namespace: comfyui
  labels:
    app: comfyui-api
spec:
  type: ClusterIP
  ports:
    - name: api
      port: 3000
      targetPort: 3000
      protocol: TCP
    - name: comfyui
      port: 8188
      targetPort: 8188
      protocol: TCP
  selector:
    app: comfyui-api
---
# KEDA HTTP Add-on - scales based on incoming HTTP requests
# Traffic flows: Client -> KEDA Interceptor -> comfyui-api-service -> Pod
apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: comfyui-api-scaler
  namespace: comfyui
spec:
  hosts:
    - comfyui.local  # Use this hostname in requests (Host header)
  targetPendingRequests: 1   # Scale up on any pending request
  scaledownPeriod: 300       # 5 minutes idle before scale to zero
  scaleTargetRef:
    name: comfyui-api
    kind: Deployment
    apiVersion: apps/v1
    service: comfyui-api-service
    port: 3000
  replicas:
    min: 0   # Scale to ZERO when idle
    max: 2   # Maximum replicas
EOF

In this case, if after 5 minutes there is no HTTP traffic, then KEDA will scale it down to zero automatically.

Make it a test

We need to do a port-forward for the KEDA HTTP interceptor proxy as

kubectl port-forward svc/keda-add-ons-http-interceptor-proxy -n keda 3000:8080

Now try to curl as below

curl -H "Host: comfyui.local" http://localhost:3000/health

You will got

But when you go back to the GPU's node pool, you will see it provisioning automatically.

And the POD will be running (in pending status):

Waiting around 2 minutes, then the POD will be up and running:

Now you should see:

If you go to http://localhost:3000/docs, you should see the API document of the comfyui-api:

If you notice in the red box, you would see 3 workflows that we deploy: img2img, qwen3-vl3-experiment, and txt2img

Now, let test the txt2img worklow with a payload like:

{
  "id": "7f350df6-49a9-4cd0-88de-5b53df870003",
  "webhook_v2": "https://webhook.site/b59a7434-c944-4897-91b8-5cd808219094",
  "input": {
    "prompt": "Create a photorealistic image of a woman standing outdoors on what appears to be a sunny autumn day. She has shoulder-length black hair and is wearing a Vietnamese Ao Dai. The background features blurred trees in the Tet holiday season. The lighting suggests natural sunlight, casting soft shadows that highlight her figure. The scene includes other people in casual attire, suggesting a public or social setting. The camera angle is at eye level, focusing on the woman's upper body and face while slightly capturing the background to provide context. The whole picture in Ho Chi Minh City",
    "negative_prompt": "",
    "checkpoint": "dreamshaper_8.safetensors"
  }
}

And do a curl request:

curl -X POST http://localhost:3000/workflow/txt2img \
  -H "Content-Type: application/json" \
  -H "Host: comfyui.local" \
  -d @payload.json

After a minute, I go to https://webhook.site/#!/view/b59a7434-c944-4897-91b8-5cd808219094/bfab5db6-c8e9-463e-8936-9bfe3dd4d289/1, copy the base64 picture there and paste it to base64-to-image-converter:

But after 5 minutes, if there is no HTTP traffic, then the POD is gone:

Conclusion

This brings me to the end of my experiment: scaling ComfyUI and ComfyUI-api on AKS using KEDA. From my perspective, the result is both practical and satisfying.

If you’ve read through this post and see opportunities for improvement or have alternative approaches to scaling a workflow-based Stable Diffusion orchestrator like ComfyUI. I’d be interested to hear your thoughts in the comments.

Happy hacking.

Appendix - comfyui and comfyui-api logs

✓ dreamshaper5.safetensors downloaded
All models ready. Starting comfyui-api...
{"level":20,"time":1769010999301,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/img2img.js","msg":"Evaluating workflow file"}
{"level":40,"time":1769010999370,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","module":"RemoteStorageManager","error":{},"msg":"Error initializing storage provider S3StorageProvider"}
{"level":30,"time":1769010999302,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"img2img","file":"/workflows/img2img.js","msg":"Loaded workflow"}
{"level":20,"time":1769010999302,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/img2img.ts","msg":"Transpiling TypeScript workflow"}
{"level":20,"time":1769010999332,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/img2img.js","msg":"Evaluating workflow file"}
{"level":30,"time":1769010999333,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"img2img","file":"/workflows/img2img.js","msg":"Loaded workflow"}
{"level":20,"time":1769010999333,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/qwen3-vl-experiment.js","msg":"Evaluating workflow file"}
{"level":30,"time":1769010999334,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"qwen3-vl-experiment","file":"/workflows/qwen3-vl-experiment.js","msg":"Loaded workflow"}
{"level":20,"time":1769010999334,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/qwen3-vl-experiment.ts","msg":"Transpiling TypeScript workflow"}
{"level":20,"time":1769010999353,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/qwen3-vl-experiment.js","msg":"Evaluating workflow file"}
{"level":30,"time":1769010999353,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"qwen3-vl-experiment","file":"/workflows/qwen3-vl-experiment.js","msg":"Loaded workflow"}
{"level":20,"time":1769010999354,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/txt2img.js","msg":"Evaluating workflow file"}
{"level":30,"time":1769010999354,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"txt2img","file":"/workflows/txt2img.js","msg":"Loaded workflow"}
{"level":20,"time":1769010999354,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/txt2img.ts","msg":"Transpiling TypeScript workflow"}
{"level":20,"time":1769010999367,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","file":"/workflows/txt2img.js","msg":"Evaluating workflow file"}
{"level":30,"time":1769010999367,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","workflow":"txt2img","file":"/workflows/txt2img.js","msg":"Loaded workflow"}
{"level":40,"time":1769010999370,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","module":"RemoteStorageManager","error":{},"msg":"Error initializing storage provider AzureBlobStorageProvider"}
{"level":30,"time":1769010999370,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","module":"RemoteStorageManager","msg":"Initialized with 2 storage providers"}
{"level":30,"time":1769010999371,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","module":"RemoteStorageManager","msg":"Cache populated with 0 files, total size: 0.00 B"}
{"level":30,"time":1769010999371,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"Starting ComfyUI API 1.16.1 with ComfyUI 0.7.0"}
2082600K .......... .......... .......... .......... ..        100%  218M=9.4s2026-01-21 15:56:39 - root - DEBUG - Tracking command: launch with arguments: {'extra': ['--listen', '*'], 'background': False, 'frontend_pr': None}
2026-01-21 15:56:39 - root - DEBUG - tracking event called with event_name: launch and properties: {'extra': ['--listen', '*'], 'background': False, 'frontend_pr': None}
╭──────────────────────────── 🔔 Update Available! ────────────────────────────╮
│ ✨ Newer version of comfy-cli is available: 1.5.4.                           │
│ Current version: 1.5.3                                                       │
│ Update by running: 'pip install --upgrade comfy-cli' ⬆                       │
╰──────────────────────────────────────────────────────────────────────────────╯

Launching ComfyUI from: /opt/ComfyUI

[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2026-01-21 15:56:40.917
** Platform: Linux
** Python version: 3.11.13 | packaged by conda-forge | (main, Jun  4 2025, 14:48:23) [GCC 13.3.0]
** Python executable: /opt/conda/bin/python
** ComfyUI Path: /opt/ComfyUI
** ComfyUI Base Folder Path: /opt/ComfyUI
** User directory: /opt/ComfyUI/user
** ComfyUI-Manager config path: /opt/ComfyUI/user/__manager/config.ini
** Log path: /opt/ComfyUI/user/comfyui.log

Prestartup times for custom nodes:
   1.7 seconds: /opt/ComfyUI/custom_nodes/ComfyUI-Manager

Checkpoint files will always be loaded safely.
Total VRAM 15931 MB, total RAM 28063 MB
pytorch version: 2.8.0+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 Tesla T4 : cudaMallocAsync
Using async weight offloading with 2 streams
Enabled pinned memory 26659.0
Using pytorch attention
Python version: 3.11.13 | packaged by conda-forge | (main, Jun  4 2025, 14:48:23) [GCC 13.3.0]
ComfyUI version: 0.7.0
ComfyUI frontend version: 1.35.9
[Prompt Server] web root: /opt/conda/lib/python3.11/site-packages/comfyui_frontend_package/static
Total VRAM 15931 MB, total RAM 28063 MB
pytorch version: 2.8.0+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 Tesla T4 : cudaMallocAsync
Using async weight offloading with 2 streams
Enabled pinned memory 26659.0
### Loading: ComfyUI-Manager (V3.39)
[ComfyUI-Manager] network_mode: public
[ComfyUI-Manager] ComfyUI per-queue preview override detected (PR #11261). Manager's preview method feature is disabled. Use ComfyUI's --preview-method CLI option or 'Settings > Execution > Live preview method'.
### ComfyUI Revision: 1 [f59f71cf] *DETACHED | Released on '2025-12-30'
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
Error loading module AILab_QwenVL_GGUF_PromptEnhancer: No module named 'llama_cpp'

Import times for custom nodes:
   0.0 seconds: /opt/ComfyUI/custom_nodes/websocket_image_save.py
   0.1 seconds: /opt/ComfyUI/custom_nodes/ComfyUI-Manager
   0.5 seconds: /opt/ComfyUI/custom_nodes/ComfyUI-QwenVL

Context impl SQLiteImpl.
Will assume non-transactional DDL.
No target revision found.
Starting server

To see the GUI go to: http://*:8188
{"level":30,"time":1769011009412,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"Comfy UI started"}
{"level":30,"time":1769011009412,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"ComfyUI 0.7.0 started."}
{"level":30,"time":1769011009477,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"Registered workflow /workflow/img2img"}
{"level":30,"time":1769011009477,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"Registered workflow /workflow/qwen3-vl-experiment"}
{"level":30,"time":1769011009477,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"Registered workflow /workflow/txt2img"}
{"level":30,"time":1769011009561,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","msg":"ComfyUI fully ready in 10.19s"}
FETCH ComfyRegistry Data: 5/120
{"level":30,"time":1769011010791,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-1","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":35108},"msg":"incoming request"}
{"level":30,"time":1769011010793,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-1","res":{"statusCode":200},"responseTime":1.678925999905914,"msg":"request completed"}
FETCH ComfyRegistry Data: 10/120
FETCH ComfyRegistry Data: 15/120
FETCH ComfyRegistry Data: 20/120
{"level":30,"time":1769011020790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-2","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":45030},"msg":"incoming request"}
{"level":30,"time":1769011020791,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-2","res":{"statusCode":200},"responseTime":0.429451999720186,"msg":"request completed"}
FETCH ComfyRegistry Data: 25/120
FETCH ComfyRegistry Data: 30/120
FETCH ComfyRegistry Data: 35/120
{"level":30,"time":1769011030790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-3","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":41768},"msg":"incoming request"}
{"level":30,"time":1769011030790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-3","res":{"statusCode":200},"responseTime":0.3221490001305938,"msg":"request completed"}
FETCH ComfyRegistry Data: 40/120
FETCH ComfyRegistry Data: 45/120
FETCH ComfyRegistry Data: 50/120
FETCH ComfyRegistry Data: 55/120
{"level":30,"time":1769011040789,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-4","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":54420},"msg":"incoming request"}
{"level":30,"time":1769011040790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-4","res":{"statusCode":200},"responseTime":0.3270169999450445,"msg":"request completed"}
FETCH ComfyRegistry Data: 60/120
FETCH ComfyRegistry Data: 65/120
FETCH ComfyRegistry Data: 70/120
{"level":30,"time":1769011050790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-5","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":48618},"msg":"incoming request"}
{"level":30,"time":1769011050790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-5","res":{"statusCode":200},"responseTime":0.33953100023791194,"msg":"request completed"}
FETCH ComfyRegistry Data: 75/120
FETCH ComfyRegistry Data: 80/120
FETCH ComfyRegistry Data: 85/120
{"level":30,"time":1769011059836,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-6","req":{"method":"GET","url":"/health","host":"comfyui.local","remoteAddress":"::ffff:10.244.0.245","remotePort":36602},"msg":"incoming request"}
{"level":30,"time":1769011059836,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-6","res":{"statusCode":200},"responseTime":0.3211170001886785,"msg":"request completed"}
{"level":30,"time":1769011060790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-7","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":59568},"msg":"incoming request"}
{"level":30,"time":1769011060790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-7","res":{"statusCode":200},"responseTime":0.35903899976983666,"msg":"request completed"}
FETCH ComfyRegistry Data: 90/120
FETCH ComfyRegistry Data: 95/120
FETCH ComfyRegistry Data: 100/120
FETCH ComfyRegistry Data: 105/120
{"level":30,"time":1769011070790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-8","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":56480},"msg":"incoming request"}
{"level":30,"time":1769011070791,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-8","res":{"statusCode":200},"responseTime":0.5209230002947152,"msg":"request completed"}
FETCH ComfyRegistry Data: 110/120
FETCH ComfyRegistry Data: 115/120
FETCH ComfyRegistry Data: 120/120
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
{"level":30,"time":1769011080790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-9","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":44198},"msg":"incoming request"}
{"level":30,"time":1769011080791,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-9","res":{"statusCode":200},"responseTime":0.5331540000624955,"msg":"request completed"}
{"level":30,"time":1769011090790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-a","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":37554},"msg":"incoming request"}
{"level":30,"time":1769011090790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-a","res":{"statusCode":200},"responseTime":0.4365039998665452,"msg":"request completed"}
{"level":30,"time":1769011100790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-b","req":{"method":"GET","url":"/health","host":"10.244.1.233:3000","remoteAddress":"::ffff:10.224.0.5","remotePort":43552},"msg":"incoming request"}
{"level":30,"time":1769011100790,"pid":1,"hostname":"comfyui-api-786bdbc-zz4nd","reqId":"req-b","res":{"statusCode":200},"responseTime":0.33777799969539046,"msg":"request completed"}

Microsoft Agent Framework with Ollama (.NET/C#)

Thang Chung — Wed, 31 Dec 2025 11:05:08 +0000

Introduction

This week, I published a blog post explaining how to use Microsoft Agent Framework with Foundry Local. Because Foundry Local does not support native tool calling, we must implement the tool-calling loop manually. Specifically, the LLM is first given a list of available tools and asked to decide whether a tool invocation is required. We then parse the model’s response, execute the selected tool, and feed the results back into the LLM for final generation. These extra steps are necessary because Foundry Local lacks built-in tool-calling support, meaning the UseFunctionInvocation capability on the ChatClient in Microsoft.Extensions.AI cannot handle this automatically.

Just yesterday, I read the excellent post Building Multi-Agent Workflows in .NET with AgentFactory and Handoff by El Bruno. In that article, he highlights using Microsoft Agent Framework with Ollama via the native Ollama provider included in the framework (sample here). This immediately sparked an idea: we can extend my previous sample at
https://github.com/thangchung/agent-engineering-experiment/tree/main/foundry-local-agent-fx
by adding Ollama as an additional provider, allowing us to switch seamlessly between Foundry Local and Ollama depending on the use case.

This post walks through the design, implementation, and code required to make that approach work in practice.

Prerequisites

Install Ollama on Windows 11: https://ollama.com/download/windows or read the guidance at https://docs.ollama.com/windows

> ollama --version
ollama version is 0.13.5

Now I have ollama run on my local machine at http://localhost:11434/
Because I will use mistral LLM model, I have to run:

> ollama pull mistral
pulling manifest
pulling f5074b1221da: 100% ▕██████████████████████████████████████████████████████████▏ 4.4 GB
pulling 43070e2d4e53: 100% ▕██████████████████████████████████████████████████████████▏  11 KB
pulling 1ff5b64b61b9: 100% ▕██████████████████████████████████████████████████████████▏  799 B
pulling ed11eda7790d: 100% ▕██████████████████████████████████████████████████████████▏   30 B
pulling 1064e17101bd: 100% ▕██████████████████████████████████████████████████████████▏  487 B
verifying sha256 digest
writing manifest
success

Setup the code structure

Aspire's AppHost

// Active provider: "FoundryLocal" or "Ollama"
var activeProvider = builder.AddParameter("agent-provider", "Ollama");

// Ollama configuration
var ollamaEndpoint = builder.AddParameter("ollama-endpoint", "http://localhost:11434/");
var ollamaModel = builder.AddParameter("ollama-model", "mistral");

var mcpToolServer = builder.AddProject<Projects.McpToolServer>("mcp-tools")
    .WithExternalHttpEndpoints();

var agentService = builder.AddProject<Projects.AgentService>("agentservice")
    .WithReference(mcpToolServer)
    // Provider selection
    .WithEnvironment("AGENT_PROVIDER", activeProvider)
    // Ollama settings
    .WithEnvironment("OLLAMA_ENDPOINT", ollamaEndpoint)
    .WithEnvironment("OLLAMA_MODEL", ollamaModel)
    // MCP Tools
    .WithEnvironment("MCP_TOOLS", mcpToolServer.GetEndpoint("http"))
    .WithExternalHttpEndpoints()
    .WaitFor(mcpToolServer);

builder.Build().Run();

Check the full code at https://github.com/thangchung/agent-engineering-experiment/blob/main/foundry-local-agent-fx/AppHost/AppHost.cs

Create Microsoft Agent Framework's Ollama factory

public static ChatClientAgent Create(
    string endpoint,
    string model,
    IList<McpClientTool> mcpTools,
    string? instructions = null,
    string? name = null,
    string? description = null,
    ILoggerFactory? loggerFactory = null,
    bool? enableSensitiveData = null)
{
    var ollamaClient = new OllamaApiClient(new Uri(endpoint), model);

    var tools = mcpTools.Cast<AITool>().ToList();

    // Build the chat client pipeline with OpenTelemetry instrumentation
    IChatClient chatClient = new ChatClientBuilder(ollamaClient)
        .UseOpenTelemetry(
            loggerFactory: loggerFactory,
            sourceName: OtelSourceName,
            configure: otel =>
            {
                // Enable sensitive data capture if explicitly set, otherwise respect env var
                // OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
                if (enableSensitiveData.HasValue)
                {
                    otel.EnableSensitiveData = enableSensitiveData.Value;
                }
            })
        .UseFunctionInvocation(loggerFactory: loggerFactory)
        .Build();

    // Create the agent with the instrumented chat client
    return new ChatClientAgent(
        chatClient: chatClient,
        instructions: instructions ?? "You are a helpful assistant with access to tools.",
        name: name ?? "OllamaAgent",
        description: description,
        tools: tools);
}

The full source code can be found at https://github.com/thangchung/agent-engineering-experiment/blob/main/foundry-local-agent-fx/AgentService/Providers/OllamaAgentFactory.cs

Create AgentFroviderFactory to choose Ollama or Foundry Local

return providerType switch
        {
            // Ollama uses official ChatClientAgent with FunctionInvokingChatClient
            AgentProviderType.Ollama => services.CreateOllamaAgent(
                ollamaEndpoint: config.Endpoint,
                model: config.Model,
                mcpToolsUrl: config.McpToolsUrl,
                mcpTools: mcpTools,
                instructions: instructions,
                name: name ?? "OllamaAgent",
                description: description ?? "AI Agent powered by Ollama"),

            AgentProviderType.FoundryLocal => services.CreateFoundryLocalAgent(
                foundryEndpoint: config.Endpoint,
                model: config.Model,
                mcpToolsUrl: config.McpToolsUrl,
                mcpTools: mcpTools,
                instructions: instructions,
                name: name ?? "FoundryLocalAgent",
                description: description ?? "AI Agent powered by Foundry Local"),

            _ => throw new ArgumentException($"Unknown provider type: {providerType}", nameof(providerType))
        };

Full source code at https://github.com/thangchung/agent-engineering-experiment/blob/main/foundry-local-agent-fx/AgentService/Providers/AgentProviderFactory.cs

Wire it up

In the Program.cs, we can wire it all up as.

var agent = AgentProviderFactory.Create(
    services: app.Services,
    providerType: providerType,
    config: providerConfig,
    mcpTools: mcpTools,
    instructions: instructions,
    name: $"{providerType}Agent",
    description: $"AI Agent powered by {providerType} with MCP tools");

And /chat endpoint can be easy to use it:

app.MapPost("/chat", async (ChatRequest request) =>
{
    try
    {
        logger.LogInformation("Processing chat: '{Message}'", request.Message);

        // Get or create thread
        AgentThread agentThread;
        string threadId;

        if (!string.IsNullOrEmpty(request.ThreadId) && threads.TryGetValue(request.ThreadId, out var existingThread))
        {
            agentThread = existingThread;
            threadId = request.ThreadId;
            logger.LogDebug("Using existing thread: {ThreadId}", threadId);
        }
        else
        {
            agentThread = agent.GetNewThread();
            threadId = GetThreadId(agentThread);
            threads[threadId] = agentThread;
            logger.LogDebug("Created new thread: {ThreadId}", threadId);
        }

        // Run the agent
        var messages = new[] { new ChatMessage(ChatRole.User, request.Message) };
        var response = await agent.RunAsync(messages, agentThread);

        // Get the assistant's response
        var responseText = response.Messages.LastOrDefault()?.Text ?? "No response generated.";

        return Results.Ok(new ChatResponse(responseText, threadId));
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Error processing chat request");
        return Results.Problem(ex.Message);
    }
});

Some results

We run the application (AppHost), and open up the AgentService at https://localhost:15000/scalar/#tag/agentservice/POST/chat

Open https://localhost:17222/traces, and click into the detail trace of this request:

The special one is, if you click on the star icon (Semantic conventions for generative AI systems), you will see:

I have tested with:
- mistral
- deepseek-v3.1:671b-cloud
- gpt-oss:20b
There is some unsupported native function calling like llama3.2:3b, then we need to do a hack like a previous blog at https://dev.to/thangchung/microsoft-agent-framework-with-foundry-local-in-netc-3l63. The code for it can be found at https://gist.github.com/thangchung/7be39c7db2e75b89be11d6e373fe1780 (if you want to use it consistently).

Conclusion

This wraps up what I explored using Ollama with Microsoft Agent Framework. As you can see, the integration is quite straightforward and does not require significant effort to get up and running.

Happy New Year to everyone reading this post. I’m looking forward to seeing Microsoft Agent Framework reach GA very soon and to the innovations the community will build on top of it.

Full source code of this post: https://github.com/thangchung/agent-engineering-experiment/tree/main/foundry-local-agent-fx

Microsoft Agent Framework with Foundry Local in .NET/C#

Thang Chung — Sun, 28 Dec 2025 15:24:06 +0000

Introduction

Last month, I had the honour of speaking at .NET Conf 2025 Vietnam, where I presented on “Multi-Agent Workflows in .NET with Microsoft Agent Framework, A2A, MCP, and AG-UI.”
For those who couldn’t attend the session live, you can watch it here:

At the same event, I also had the chance to watch an excellent talk by anh Phi Huynh on "Self-Sovereign AI with Azure AI Foundry Local." The recording is available here:

In his session, anh Phi introduced Foundry Local and several LLMs that can run locally, such as Phi, Qwen, and ChatGPT-OSS. What immediately sparked my curiosity was this question:
Can we combine Foundry Local with the Microsoft Agent Framework (AgentFx) in .NET/C# to build fully local, self-sovereign agentic AI systems?

If the answer is yes, we could run both data and LLMs entirely on local machines, which is especially compelling for sensitive domains such as banking or insurance, where data must never be exposed to public LLM services.

Motivated by this idea, I started experimenting — and quickly discovered a number of interesting challenges and insights along the way. This post captures those findings, based on my hands-on experience trying to make Microsoft AgentFx work seamlessly with Foundry Local. It also highlights the gaps that exist today and where the Microsoft AgentFx team may eventually enable native support within the framework itself.

Explore Foundry Local

At the beginning, I installed Foundry Local on my machine by following the docs at https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/get-started?view=foundry-classic.

I used winget, then I typed:

winget install Microsoft.FoundryLocal

After that, I typed:

foundry service start
🟢 Service is Started on http://127.0.0.1:57591/, PID 35620!

And,

foundry service status
🟢 Model management service is running on http://127.0.0.1:57591/openai/status
EP autoregistration status: Successfully downloaded and registered the following EPs: NvTensorRTRTXExecutionProvider, CUDAExecutionProvider.
Valid EPs: CPUExecutionProvider, WebGpuExecutionProvider, NvTensorRTRTXExecutionProvider, CUDAExecutionProvider

Make sure you got to http://127.0.0.1:57591/openai/status, and see below.

Make Microsoft AgentFx work with Foundry Local

The next logical step was to make this work with Microsoft AgentFx, right? After spending about 30 minutes or more reviewing the code and documentation in the Microsoft AgentFx repository (https://github.com/microsoft/agent-framework),
I discovered that native support for Foundry Local currently exists only in the Python implementation (see https://github.com/microsoft/agent-framework/pull/2915). Unfortunately for me, what I was really looking for was a .NET/C# equivalent.

So, I decided to prototype my own solution in .NET/C#, taking the Python implementation as a reference. The results so far have been very promising. The setup works smoothly using the OpenAIClient SDK, and the LLM model I’m running locally is qwen2.5-14b-instruct-generic-cpu:4.

The code looks like:

var openAiClient = new OpenAIClient(
    new ApiKeyCredential("not-needed"),
    new OpenAIClientOptions { Endpoint = new Uri("http://127.0.0.1:57591/v1") });

var chatClient = openAiClient.GetChatClient("qwen2.5-14b-instruct-generic-cpu:4");

var openAiMessages = new List<OpenAI.Chat.ChatMessage>
{
    new SystemChatMessage("You are a helpful assistant.")
};

openAiMessages.Add(new UserChatMessage("Write the poem about .NET Conf 2025 in Vietnam"))

var response = await chatClient.CompleteChatAsync(openAiMessages, cancellationToken: cancellationToken);

Now I think we can use a function call with the OpenAI client with the standard function-calling format (and this is actually a main building block when we build agentic AI app, right?). After trying with the prototype, it didn't work. After struggling a bit with a ton of knowledge in Foundry Local, I found out that Foundry Local used a different format than what OpenAI does. See

Foundry Local uses a non-standard function calling format. Instead of returning function calls in the standard OpenAI tool_calls field, Qwen models return the function call as JSON text in the response content.

For example, when you ask about the weather, instead of:

# Standard OpenAI format message.tool_calls = [ {"name": "get_weather", "arguments": {"location": "Birmingham"}} ]

You get:

# Foundry Local format message.content = '{"name": "get_weather", "arguments": {"location": "Birmingham"}}'

See the article from Microsoft's Foundry Local team: https://techcommunity.microsoft.com/blog/educatordeveloperblog/function-calling-with-small-language-models/4472720

Now, we need to do a hack by writing the code to parse the tool_calling, which returns at the first LLM Model calling, then call the tools, gather all results and send again to the LLM model for generation. The full workflow is just exactly the same with OpenAI docs:

Look what I did below.

var openAiClient = new OpenAIClient(
    new ApiKeyCredential("not-needed"),
    new OpenAIClientOptions { Endpoint = new Uri(_foundryEndpoint) });

var chatClient = openAiClient.GetChatClient(_model);

// Build system prompt with tool definitions
var toolDefs = BuildToolDefs();
var jsonExample = @"{""name"": ""tool_name"", ""arguments"": {""param"": ""value""}}";

var systemPrompt = _mcpTools.Count > 0
    ? $"""
        {_instructions ?? "You are a helpful assistant with access to tools."}

        When you need to use a tool, respond ONLY with a JSON object in this exact format:
        {jsonExample}

        Available tools:
        {toolDefs}

        If you don't need a tool, respond normally with text.
        """
    : _instructions ?? "You are a helpful assistant.";

// Convert messages to OpenAI format
var openAiMessages = new List<OpenAI.Chat.ChatMessage>
{
    new SystemChatMessage(systemPrompt)
};

foreach (var msg in chatMessages)
{
    if (msg.Role == ChatRole.User)
        openAiMessages.Add(new UserChatMessage(msg.Text ?? ""));
    else if (msg.Role == ChatRole.Assistant)
        openAiMessages.Add(new AssistantChatMessage(msg.Text ?? ""));
}

_logger?.LogDebug("[FoundryLocalAgent] Sending request to model...");

var response = await chatClient.CompleteChatAsync(openAiMessages, cancellationToken: cancellationToken);
var content = response.Value.Content.FirstOrDefault()?.Text ?? "";

_logger?.LogDebug("[FoundryLocalAgent] Model response: {Content}", content);

// Parse tool calls from content (Foundry Local format)
var toolCalls = ToolCallParser.Parse(content);
var validToolCalls = toolCalls
    .Where(tc => _mcpTools.Any(t => t.Name == tc.Name))
    .ToList();

if (validToolCalls.Count == 0)
{
    // No tool calls, return the response as-is
    return content;
}

_logger?.LogInformation("[FoundryLocalAgent] Executing {Count} tool(s): {Tools}",
    validToolCalls.Count,
    string.Join(", ", validToolCalls.Select(tc => tc.Name)));

// Execute tools in parallel
var toolTasks = validToolCalls.Select(async toolCall =>
{
    try
    {
        var result = await CallMcpToolAsync(toolCall.Name, toolCall.Arguments, cancellationToken);
        return new { toolCall.Name, Result = result ?? "", Error = (string?)null };
    }
    catch (Exception ex)
    {
        return new { toolCall.Name, Result = "", Error = (string?)ex.Message };
    }
});

var toolResults = await Task.WhenAll(toolTasks);

// Build tool results message
var toolResultsText = string.Join("\n\n", toolResults.Select(r =>
    r.Error is null
        ? $"Tool '{r.Name}' result:\n{r.Result}"
        : $"Tool '{r.Name}' error: {r.Error}"));

// Add tool results to the conversation and get the final response
openAiMessages.Add(new AssistantChatMessage(content));
openAiMessages.Add(new UserChatMessage($"Tool execution results:\n{toolResultsText}\n\nPlease provide your final response based on these results."));

var finalResponse = await chatClient.CompleteChatAsync(openAiMessages, cancellationToken: cancellationToken);
var finalResult = finalResponse.Value.Content.FirstOrDefault()?.Text ?? toolResultsText;

The full source code version (Microsoft AgentFx + Foundry Local) can be found at FoundryLocalAgent.cs

Some results

I set up the agent (FoundryLocalAgent.cs) and 2 MCP tools:

DateTimeTools - GetDateTime tool (https://github.com/thangchung/agent-engineering-experiment/blob/27d0668e82f39b1991b8981f96bb7f1c1017a36e/foundry-local-agent-fx/McpToolServer/Tools/DateTimeTools.cs)
WeatherTools - GetWeather and GetTypicalWeather tools (https://github.com/thangchung/agent-engineering-experiment/blob/27d0668e82f39b1991b8981f96bb7f1c1017a36e/foundry-local-agent-fx/McpToolServer/Tools/WeatherTools.cs)

When I do a query on the chat API like "London time, and weather in New York after 2PM?", then it responds

{
  "response": "According to the London time (Europe/London), it is currently Sunday, December 28, 2025, 2:53:55 PM. Since you asked about the weather",
  "threadId": "ca0e12c91b02478482d71356bc8069e9"
}

See the Scalar UI as below.

And with Aspire, we can see even more details of how the workflow is running:

Local LLM models and the disadvantage of my implementation

At the time of writing this blog, I have tested with:
- qwen2.5-14b-instruct-generic-cpu:4
- Phi-4-mini-instruct-cuda-gpu:5
- Unluckily, I couldn't run Llama-3.2-1B-Instruct (https://learn.microsoft.com/en-gb/azure/ai-foundry/foundry-local/how-to/how-to-compile-hugging-face-models?view=foundry-classic&tabs=Bash) due to some failed setup when working with HuggingFace Model (https://github.com/thangchung/agent-engineering-experiment/issues/1) on my laptop. I will have a look at it later and give you an update.
And the current implementation with FoundryLocalAgent is only working correctly with RunAsync. RunStreamingAsync is still not working very effectively (call to RunAsync instead of implementing the async stream version). I have posted the comment on https://github.com/microsoft/agent-framework/issues/2963, and hope Microsoft AgentFx and Foundry Local team will support it natively.

Conclusion

I’ve walked you through my journey of getting Foundry Local (with Qwen and Phi models) working with Microsoft AgentFx — highlighting what works today, what doesn’t yet, and the workaround I used to bridge the gap. While this solution is admittedly a hack rather than a perfect implementation, it’s fully functional — and honestly, it feels pretty great to see it running end to end.

Looking ahead, I’m optimistic that the Microsoft AgentFx and Foundry Local teams will collaborate to deliver native Foundry Local support in AgentFx as part of an upcoming GA release (.NET/C# version), which I believe is coming very soon.

I hope you find this post useful.
The full source code is available here:
https://github.com/thangchung/agent-engineering-experiment/tree/27d0668e82f39b1991b8981f96bb7f1c1017a36e/foundry-local-agent-fx

Agent Skill in VS Code

Thang Chung — Sun, 21 Dec 2025 14:21:55 +0000

Introduction

Agent Skill is a collection of MD files, scripts, examples, and test scenarios that your Agent can use to perform specialised tasks. Think of them like a set of instructions.

Originally, Agent Skill was a standard used solely by Anthropic Models/Agents. Now it's an open standard, much like MCP. For more information, you can check out agentskills.io.

And now, it is adopted by several editors/CLIs, and in this post, I would like to explore using it with Visual Studio Code (insider version).

Give it a try

Make sure you have Visual Studio Insiders 1.107 or higher. Then you can go to Preferences -> Settings -> Find Anthropic Agent Skills, then you should see as below (check Use Agent Skills):

Step 1: Create a calculator utility's Agent Skill

Then, I would like to extract add/subtract/multiply/divide operators to a Python script to demonstrate the agent's ability to detect and call the script.

All the source code can be found at

thangchung / agent-engineering-experiment

Agent Engineering experiment, practices, codes, specs...

You can check out the calculator-utils/SKILL.md, and the calculator-utils/scripts/calculator.py

Step 2: Discover the calculator utility's Agent Skill

I type the prompt give me the results of ((1*6.5/2)+(10-12))/4?. It will know that I want to do some kind of calculator, then VSCode will discover the appropriate agent skill (in this case is calculator utils at .github/skills):

Step 3: Execute the script based on the SKILL.md guidance

Each execution step will divide the complex calculation into a simple one, and call the Python script to execute it as follows:

How many editor/CLI supports?

Currently, Claude, VS Code + GitHub Copilot, Codex CLI, Amp, and Factory already support Anthropic Agent Skill natively. Look at the picture below from
Eleanor Berger twit

Happy hacking!

Upgrade Projects Easily with CentralisedPackageConverter!

Thang Chung — Mon, 09 Jun 2025 11:10:51 +0000

Context: The SK_MCP repository should utilize ManagePackageVersionsCentrally to facilitate the easy upgrading of NuGet packages used in both client and server projects.

We have found a very cool tool called CentralisedPackageConverter

First, let's install the tool by using:

dotnet tool install CentralisedPackageConverter --global

Now, let's run it in the root of SK_MCP project:

git clone git@github.com:thangchung/mcp-labs.git
cd SK_MCP
central-pkg-converter .

Then it will confirm and convert all of csproject to use ManagePackageVersionsCentrally, as shown in the picture below.

And Directory.Packages.props created with content as:

<Project>
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <CentralPackageTransitivePinningEnabled>false</CentralPackageTransitivePinningEnabled>
  </PropertyGroup>
  <ItemGroup>
    <PackageVersion Include="Aspire.Hosting.AppHost" Version="9.2.0" />
    <PackageVersion Include="Azure.AI.Projects" Version="1.0.0-beta.7" />
    <PackageVersion Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="9.0.4" />
    <PackageVersion Include="Microsoft.Extensions.Configuration.UserSecrets" Version="9.0.4" />
    <PackageVersion Include="Microsoft.Extensions.Hosting" Version="8.0.1" />
    <PackageVersion Include="Microsoft.Extensions.Logging" Version="9.0.4" />
    <PackageVersion Include="Microsoft.Extensions.Logging.Console" Version="9.0.4" />
    <PackageVersion Include="Microsoft.Extensions.Logging.Debug" Version="9.0.4" />
    <PackageVersion Include="Microsoft.SemanticKernel" Version="1.47.0" />
    <PackageVersion Include="Microsoft.SemanticKernel.Abstractions" Version="1.47.0" />
    <PackageVersion Include="Microsoft.SemanticKernel.Agents.AzureAI" Version="1.47.0-preview" />
    <PackageVersion Include="Microsoft.SemanticKernel.Agents.Core" Version="1.47.0" />
    <PackageVersion Include="Microsoft.SemanticKernel.Connectors.AzureOpenAI " Version="1.47.0" />
    <PackageVersion Include="Microsoft.SemanticKernel.Connectors.InMemory" Version="1.47.0-preview" />
    <PackageVersion Include="Microsoft.SemanticKernel.Core" Version="1.47.0" />
    <PackageVersion Include="Microsoft.SemanticKernel.PromptTemplates.Handlebars" Version="1.47.0" />
    <PackageVersion Include="ModelContextProtocol" Version="0.1.0-preview.10" />
    <PackageVersion Include="ModelContextProtocol.AspNetCore" Version="0.1.0-preview.10" />
    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.11.2" />
    <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.11.2" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.11.1" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.11.0" />
  </ItemGroup>
</Project>

Server and Client projects are modified to use ManagePackageVersionsCentrally as well. Look at the below.

# MCPServer
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>5ee045b0-aea3-4f08-8d31-32d1a6f8fed0</UserSecretsId>
    <NoWarn>$(NoWarn);VSTHRD111;CA2007;CA1054;SKEXP0001;SKEXP0010;SKEXP0110</NoWarn>
  </PropertyGroup>

  <ItemGroup>
    <Content Remove="Prompts\getCurrentWeatherForCity.json" />
  </ItemGroup>

  <ItemGroup>
    <None Remove="ProjectResources\EmployeeBirthdaysAndPositions.png" />
    <None Remove="ProjectResources\SalesReport2014.png" />
  </ItemGroup>

  <ItemGroup>
    <EmbeddedResource Include="ProjectResources\EmployeeBirthdaysAndPositions.png" />
    <EmbeddedResource Include="ProjectResources\getCurrentWeatherForCity.json" />
    <EmbeddedResource Include="ProjectResources\cat.jpg">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </EmbeddedResource>
    <EmbeddedResource Include="ProjectResources\SalesReport2014.png" />
    <EmbeddedResource Include="ProjectResources\semantic-kernel-info.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </EmbeddedResource>
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" />
    <PackageReference Include="ModelContextProtocol" />
    <PackageReference Include="ModelContextProtocol.AspNetCore" />
    <!-- Core Semantic Kernel packages -->
    <PackageReference Include="Microsoft.SemanticKernel" />
    <PackageReference Include="Microsoft.SemanticKernel.Abstractions" />
    <PackageReference Include="Microsoft.SemanticKernel.Core" />
    <!-- Connectors -->
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.AzureOpenAI " />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.InMemory" />
    <!-- Prompting -->
    <PackageReference Include="Microsoft.SemanticKernel.PromptTemplates.Handlebars" />
    <!-- Agent -->
    <PackageReference Include="Microsoft.SemanticKernel.Agents.Core" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" />
  </ItemGroup>

</Project>

And,

# MCPClient
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>5ee045b0-aea3-4f08-8d31-32d1a6f8fed0</UserSecretsId>
    <NoWarn>$(NoWarn);VSTHRD111;CA2007;CA1054;SKEXP0001;SKEXP0010;SKEXP0110</NoWarn>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="ModelContextProtocol" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" />
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" />
    <PackageReference Include="Microsoft.Extensions.Logging" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" />
    <!-- Core Semantic Kernel packages -->
    <PackageReference Include="Microsoft.SemanticKernel" />
    <PackageReference Include="Microsoft.SemanticKernel.Abstractions" />
    <PackageReference Include="Microsoft.SemanticKernel.Core" />
    <!-- Connectors -->
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.AzureOpenAI " />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.InMemory" />
    <!-- Prompting -->
    <PackageReference Include="Microsoft.SemanticKernel.PromptTemplates.Handlebars" />
    <!-- Agent -->
    <PackageReference Include="Microsoft.SemanticKernel.Agents.Core" />
    <PackageReference Include="Microsoft.SemanticKernel.Agents.AzureAI" />
  </ItemGroup>

  <ItemGroup Label="NotUsed">
    <PackageReference Include="Azure.AI.Projects" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" />
  </ItemGroup>

</Project>

Bonus tip: migrate sln to slnx in .NET 9

dotnet sln migrate <your_solution.sln>

Now, we can upgrade the NuGet packages in Visual Studio. Happy tips, happy working!

Time for enhancing the development tools with GenAI, first PostgreSQL IDE

Thang Chung — Mon, 26 May 2025 09:37:06 +0000

Introduction

Not long ago, I discovered how to query databases using natural language with a tool on VS Code, utilizing Agent Mode combined with PostgreSQL's MCP server. Since then, I've had a strong feeling that the era of GenAI-enhanced development tools is imminent. And now, it's here...

Microsoft recently announced a new IDE for PostgreSQL in VS Code. You can read more about it here. Additionally, you can find the GitHub repository for the VS Code extension and some helpful guidance here.

Today's post aims to demonstrate the connection between my work on querying PostgreSQL using natural language and achieving the same results with Microsoft's PostgreSQL VS Code extension.

Prepare and connect to the PostgreSQL DB server

Install PostgreSQL's VSCode extensions

Make sure you install the VS Code extension as the picture below.

Provisioning a PostgreSQL database server

git clone git@github.com:thangchung/mcp-labs.git
cd mcp-labs/vscode-mcp-servers
dotnet build
dotnet run --project AppHost1/AppHost1.csproj

This process will provision a PostgreSQL database, create a complete copy of the Northwind database, and seed it with data. Once you've completed these steps, you should see the following view in .NET Aspire.

Wire it up

Open VS Code IDE, and click on the PostgreSQL icon (just installed in the previous steps), and try to input the information of server name, username, password, and database port. Make sure you click Test Connection and it should work as expected.

After connecting, you should see all table schemas as the picture below.

Database schema visualizer

Now, let's explore the cool feature in this extension. At the root of the database server, you can right-click on postgres, and choose Visualize Schema:

You should see all the database table schemas displayed in the beautiful visualizer as follows:

Query in VS Code with MCP tools in Agent mode

Overview of how we can make a prompting and smart query in GitHub Copilot in VS Code (Agent Mode with Claude Sonnet 4 model) as below:

Let's deep dive a little bit into it:

Look at my simple prompt: Show me the top 10 orders with the best prices in T-SQL, and query it out

And it generated as below:

SELECT
    o.order_id,
    o.customer_id,
    c.company_name,
    o.order_date,
    o.shipped_date,
    -- Calculate total order value considering discount
    ROUND(
        SUM(
            od.unit_price * od.quantity * (1 - od.discount)
        )::numeric, 
        2
    ) AS total_order_value,
    -- Additional details
    COUNT(od.product_id) AS total_items,
    o.freight,
    o.ship_country
FROM orders o
    INNER JOIN order_details od ON o.order_id = od.order_id
    LEFT JOIN customers c ON o.customer_id = c.customer_id
GROUP BY 
    o.order_id, 
    o.customer_id, 
    c.company_name,
    o.order_date, 
    o.shipped_date, 
    o.freight, 
    o.ship_country
ORDER BY total_order_value DESC
LIMIT 10;

If you look at the T-SQL above, you'll notice it's quite intelligent compared to my previous setup. With just a zero-shot prompt, it accurately identifies the valid tables in the database. In the previous setup, I had to specify the correct table names and the price column.

Now, let's examine the result in detail:

Asking for confirmation to use PostgreSQL's query MCP tool.
Using this tool to query to database server and get data out.
Submit to the LLM model to enrich the result and add more insightful views.

Conclusion

In conclusion, the integration of GenAI into development tools is revolutionizing the way we work, making tasks more intuitive and efficient (this time is PostgreSQL IDE). As we stand on the brink of this technological evolution, it's clear that the future holds even more exciting advancements. The proliferation of GenAI-enhanced tools will undoubtedly transform our workflows, driving innovation and productivity to new heights. Stay tuned, as this journey is just beginning, and the best is yet to come.

Visual Studio Code + Model Context Protocol (MCP) Servers - the first look

Thang Chung — Fri, 28 Mar 2025 11:01:08 +0000

Introduction

Model Context Protocol (MCP) has gained traction, and we can see everybody talked about it on X, YouTube, LinkedIn, FaceBook and everywhere.

This post is going to have a look and set up MCP servers such as @modelcontextprotocol/server-filesystem, Custom MCP server with .NET SDK, and @modelcontextprotocol/server-postgres on Visual Studio Code with Copilot (MCP client).

Prerequisites

https://code.visualstudio.com/insiders/
http://aka.ms/copilot-free
python Python 3.12.9
Node v22.14.0
npx 11.1.0

Make sure you have @modelcontextprotocol/server-filesystem@2025.1.14, @modelcontextprotocol/server-postgres@0.6.2. If not, you have to install it by using

npm -g @modelcontextprotocol/server-filesystem
npm -g @modelcontextprotocol/server-postgres

You should see as below:

> npm list -g --depth=0
C:\Program Files\nodejs -> .\
+-- @modelcontextprotocol/server-filesystem@2025.1.14
+-- @modelcontextprotocol/server-postgres@0.6.2
+-- corepack@0.31.0
+-- npm@11.1.0
`-- uv@1.4.0

Set it up

Open vscode insiders version, then on the top of this editor click on the Copilot icon as below:

Then, we open up the Chat windows, then choose Agent mode as follows:

In the next step, we open the vscode settings at File -> Preferences -> Settings -> on Search settings, we type mcp -> and on Mcp panel choose Edit in settings.json, we will see:

{
    ...
    "chat.mcp.discovery.enabled": true,
    "mcp": {
        "inputs": [],
        "servers": {
            "mcp-server-time": {
                "command": "python",
                "args": [
                    "-m",
                    "mcp_server_time",
                    "--local-timezone=America/Los_Angeles"
                ],
                "env": {}
            }
        }
    }
}

When you start the mcp-server-time, it will throw an error if you don't have Python on your machine or mcp_server_time package.

Solution:
If we have got an error "No module named mcp_server_time"
Then I found the issue here: https://github.com/microsoft/vscode/issues/244637#issuecomment-2760114961

Make sure you have python on your machine, and then run python -m pip install mcp-server-time, then you're good to go.

If everything is okay, now we can use this tool to look at the top of the chat windows. See

Now, let's use it (I typed What's time in Saigon?, then I typed The gaps with London?)

Look at it. It can use 2 tools that we have to get the current time zone (Saigon) and convert it (London). It's very cool, right?

MCP with our custom code

Now, let's imagine that we want to develop our own MCP server using C# SDK for Model Context Protocol servers and clients. The code for it is below

var builder = Host.CreateEmptyApplicationBuilder(settings: null);
builder.Services
    .AddMcpServer()
    .WithStdioServerTransport()   
    .WithTools();
await builder.Build().RunAsync();

[McpToolType]
public static class TimeTool
{
    [McpTool, Description("Get the current time for a city")]
    public static string GetCurrentTime(string city) => 
        $"It is {DateTime.Now.Hour}:{DateTime.Now.Minute} in {city}.";
}

Build the solution, then we configure it with VS Code - Copilot as below:

"mcp": {
    "servers": {
        "my-own-mcp-server": {
            "command": "C:\\Users\\thangchung\\source_code\\ai_labs\\mcp-labs\\vscode-mcp-servers\\MCPServer\\bin\\Debug\\net9.0\\MCPServer.exe"
        }
    }
}

Starting our own server, then back to the chat window:

Let's do some chat:

MCP file server

Now, let's configure another MCP Server for @modelcontextprotocol/server-filesystem as below

"mcp": {
    "servers": {
        // remove for brevity
        "filesystem": {
            "command": "npx",
            "args": [
                "-y",
                "@modelcontextprotocol/server-filesystem",
                "C:\\temp"
            ],
            "env": {
                "DEBUG": "*"
            }
        }
    }
}

We create a folder on C:\ with the named temp, then create a test.txt file inside it.

Start the server, then we type as:

MCP Postgres server

One day last month, my colleague challenged me whether I could use the MCP client to connect with the MCP Postgres server, and then use some kind of natural language (English) to command and let the foundation model (GPT-4o) analyse, plan, and generate the T-SQL, then it can connect to the database to query the data out. Challenge accepted 😂. I keep thinking about it, and now I will show you how can it work.

Firstly, we need the Postgres database and seed the Northwind database for trying some experiments. I use .NET Aspire to set it up (you can run docker image or docker-compose, or whatever it is). The code is as below:

var builder = DistributedApplication.CreateBuilder(args);

// !External params
var username = builder.AddParameter("db-username");
var password = builder.AddParameter("db-password");

var postgresQL = builder.AddPostgres("postgresQL", userName: username, password: password)
                        .WithLifetime(ContainerLifetime.Persistent)
                        .WithBindMount("./Scripts", "/docker-entrypoint-initdb.d")
                        .WithEndpoint("tcp", (e) =>
                        {
                            e.Port = 5432;
                            e.IsProxied = false;
                        })
                        .WithPgWeb();

var postgres = postgresQL.AddDatabase("postgres");

builder.Build().Run();

Then, run it:

> dotnet run

The Northwind data is in place:

"mcp": {
    "inputs": [],
    "servers": {
        // remove for brevity
        "postgres": {
            "command": "npx",
            "args": [
                "-y",
                "@modelcontextprotocol/server-postgres",
                "postgresql://postgres:P@ssW0rd@123@localhost:5432/postgres?connect_timeout=20"
            ],
        }
    }
}

Now, it is a show time:

The T-SQL script is generated as:

SELECT od.*, o.*, p.* 
FROM order_details od JOIN orders o ON od.order_id = o.order_id JOIN products p ON od.product_id = p.product_id 
ORDER BY od.unit_price DESC 
LIMIT 10;

We can do even better like convert it into LINQ:

The source code to get started can be found at https://github.com/thangchung/mcp-labs/tree/main/vscode-mcp-servers

Conclusion

In this post, we do many things to experiment with MCP servers. When use it correctly, then we can improve a lot in DevEx and Vibe coding actually.
In the next step, I will try to think about how can we build the Agentic AI architecture on the coffee shop apps. Happy hacking.

References

Deploying Distributed Apps (Spin + Dapr) on SpinKube (AKS)

Thang Chung — Thu, 20 Feb 2025 12:41:15 +0000

Introduction

In the last part, we walk through the journey of how to build and run Spin Apps in a local environment with the support of .NET Aspire. If you haven't read it, I recommend you to check it out at Part 1: Orchestrating Distributed Apps (Spin/Rust and .NET/C#) with .NET Aspire/Dapr.

In this part, we will continue to deploy the apps in the previous part using SpinKube (Azure Kubernetes Service - AKS). To make it more easy to follow, I draw the deployment model below.

Source code: https://github.com/thangchung/dapr-labs/tree/main/aspire

Prerequisites

az CLI

az version

{
  "azure-cli": "2.67.0",
  "azure-cli-core": "2.67.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}

dapr CLI

dapr version

CLI version: 1.14.1
Runtime version: 1.14.4

spin CLI

cargo --version
cargo 1.83.0 (5ffbef321 2024-10-29)

spin --version
spin 3.1.1 (aa919ce 2024-12-20)

.NET 9 CLI

dotnet --list-sdks
9.0.100 [C:\Program Files\dotnet\sdk]

kubectl CLI

kubectl version

Client Version: v1.32.0
Kustomize Version: v5.5.0

helm CLI

helm version

version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"clean", GoVersion:"go1.22.7"}

Provisioning AKS cluster

Follow the guidance at Deploy an Azure Kubernetes Service (AKS) cluster using Azure CLI to create the AKS cluster. Let's say its name is spinkube-azure-globalazure24

Provisioning SpinKube components

In the past, to make Spin work on the AKS cluster, we needed to install several components and did that manually. Its work might be hard and error-prone. Now, the Fermyon team has introduced SpinKube-Azure, it wraps every component into just one Helm chart, so we can run only 1 command to install those components into AKS as below.

git clone git@github.com:spinframework/azure.git
cd azure
helm install --wait spinkube .

Let's wait a second, then we are ready to go.

Provisioning Dapr components

If you remember in the previous part, we used RabbitMQ for Dapr's pub/sub. So let's work on it on AKS.

RabbitMQ cluster provisioning

helm install rabbit --set service.type=NodePort stable/rabbitmq

# Because of the demo, then we get it password and use it for the next step, not store it in the KeyVault.
echo $(kubectl get secret --namespace rabbit mu-rabbit-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)

Ref: https://gist.github.com/enesusta/e2ac956104dc0fa4130c795e7270bfc2

Dapr's RabbitMQ pubsub component

Create a Dapr's pub-sub component with the RabbitMQ configuration above.

kubectl apply -f - <<EOF
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
  namespace: default
spec:
  type: pubsub.rabbitmq
  version: v1
  metadata:
  - name: protocol
    value: amqp
  - name: hostname
    value: rabbit-rabbitmq
  - name: username
    value: user
  - name: password
    value: <password above step>
EOF

Scaffolding SpinApp and .NET app

Spin App

spin registry push ttl.sh/thangchung-test-spin:1h --build

.NET App

dotnet publish ./WebApp/WebApp.csproj --os linux --arch x64 /t:PublishContainer -c Release

Deploy our apps into AKS

kubectl apply -f - <<EOF
apiVersion: core.spinkube.dev/v1alpha1
kind: SpinApp
metadata:
  name: test-spin
spec:
  image: "ttl.sh/thangchung-test-spin:1h"
  executor: containerd-shim-spin
  replicas: 1
  podAnnotations:
    dapr.io/enabled: "true"
    dapr.io/app-id: "test-spin"
    dapr.io/app-port: "80"
    dapr.io/enable-api-logging: "true"
  variables:
  - name: dapr_url
    value: http://localhost:3500
  - name: loglevel
    value: info
EOF

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "webapp"
        dapr.io/app-port: "8080"
        dapr.io/enable-api-logging: "true"
    spec:
      containers:
      - name: webapp
        image: ttl.sh/thangchung-webapp:1h
        imagePullPolicy: Always
        env:
        - name: TestSpinApp
          value: "test-spin"
        resources:
          # limit the resources to 128Mi of memory and 100m of CPU
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: webapp
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 8080
  selector:
    app: webapp
EOF

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store-front
spec:
  ingressClassName: webapprouting.kubernetes.azure.com
  rules:
  - http:
      paths:
      - backend:
          service:
            name: webapp
            port:
              number: 5000
        path: /
        pathType: Prefix
EOF

Notes: Make sure you enable approuting in AKS, by using:
az aks approuting enable --resource-group globalazure24 --name spinkube-azure-globalazure24

AKS Dashboard

Running the distributed apps

Port forward the web-app

kubectl port-forward services/webapp 5000:5000 -n default

Notes: if you use the ingress setup above, then you can curl directly into the public IP:
kubectl get ingress # it will show the public IP

Create some traffic

curl http://localhost:5000/item-types
curl -d '{}' http://localhost:5000/ping

Tail the test-spin-app logs

Tail the web app logs

Conclusion

This post shows you that we can deploy and run hybrid apps (Spin App and .NET App) with Dapr in place. It is a good start to build up more complex apps in the next posts which I will work on. Thanks for reading it through. Happy deploying ^_^

Orchestrating Distributed Apps (Spin/Rust and .NET/C#) with .NET Aspire/Dapr

Thang Chung — Fri, 10 Jan 2025 10:32:41 +0000

Introduction

If you have followed me long enough, then you know that I strongly believe that WebAssembly (WASI) will become the popular choice for running and development workloads in the future (and it is happening now). It is more efficient and usable if it can work with other apps built with current container technology (on Docker or Podman), see https://dev.to/thangchung/series/24617.

In the journey of discovering all possibilities in this mechanism, we look to enhance the DevEx when developing the Spin app on the local machine. To experiment with some innovation ideas, I have a look at .NET Aspire (just released a new version with .NET 9 a couple of months ago).

We built this sample because .NET Aspire is very cool in enhancing DevEx for developers, and my question is whether we can combine .NET Aspire to develop .NET apps with Spin Apps altogether. And one more thing, let's say we build a custom action on .NET Aspire to allow re-build the Spin app on the dashboard, then it should be very cool, right?

That being said, the Fermyon team built something very cool at https://github.com/fermyon/Aspire.Spin, but we want something simpler, just for running and building Spin apps in the .NET Aspire dashboard. This post will explore the ability to do that and add a new custom task on .NET Aspire dashboard to rebuild the Spin app without the need to stop and start solution again (.NET Aspire community has worked very actively to bring up the ability to work on various programming languages as well as web frameworks at https://github.com/CommunityToolkit/Aspire/tree/main/examples, but again we would like to re-build the task on .NET Aspire to do just very simple spin build command, then reload the app state).

Let's get started...

High level of the sample distributed apps

.NET app calls to Spin app using service invocation (Dapr)
.NET app publishes an event (PingEvent) to Spin App (pub-sub via Dapr)
Spin app subscribes to the event (PingEvent) from step 2, processing, and then publishes an event (PongEvent) back to the .NET app.

Preprequisites

.NET 9

dotnet --list-sdks
9.0.100 [C:\Program Files\dotnet\sdk]

Rust & Spin CLI

cargo --version
cargo 1.83.0 (5ffbef321 2024-10-29)

spin --version
spin 3.1.1 (aa919ce 2024-12-20)

Dapr CLI

dapr version
CLI version: 1.14.1
Runtime version: 1.14.4

The solution setup

Source code for the whole solution: https://github.com/thangchung/dapr-labs/tree/main/aspire

.NET app - Minimal API (.NET 9)

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddCors();

builder.AddServiceDefaults();

builder.Services.AddOpenApi();

builder.Services.AddDaprClient();
builder.Services.AddSingleton(new JsonSerializerOptions()
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    PropertyNameCaseInsensitive = true,
});

var app = builder.Build();

app.MapDefaultEndpoints();

if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
    app.MapScalarApiReference();
}

app.UseCors(x => x.AllowAnyOrigin().AllowAnyMethod().AllowAnyHeader());

app.UseRouting();

app.UseCloudEvents();

app.MapSubscribeHandler();

app.MapGet("/", () => Results.Redirect("/scalar/v1"))
    .ExcludeFromDescription();

app.MapGet("/item-types", async (DaprClient client, IConfiguration config) =>
{
    var res = await client.InvokeMethodAsync<List<ItemTypeDto>>(
        HttpMethod.Get, 
        config.GetValue<string>("TestSpinApp"), 
        "v1-get-item-types");

    var curActivity = Activity.Current;
    curActivity?.AddBaggage("method-name", "item-types");

    return res;
})
.WithName("GetItemTypes");

app.MapPost("/ping", async (DaprClient client) =>
{
    await client.PublishEventAsync("pubsub", "pinged", new { Id = Guid.NewGuid() });
    return Results.Ok();
});

app.MapPost("/pong", [Topic("pubsub", "ponged")] async (Pong pong) =>
{
    Console.WriteLine($"Pong received: {pong.Id}");
    return Results.Ok();
}).ExcludeFromDescription();

app.Run();

internal record ItemTypeDto(string Name, int ItemType, float Price, string Image);
internal record Pong(Guid Id);

[ExcludeFromCodeCoverage]
public partial class Program;

Spin app (Rust)

const PUB_SUB_NAME: &str = "pubsub";

#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct Pinged {
    pub id: Uuid,
}

#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct Ponged {
    pub id: Uuid,
}

#[derive(Debug, Serialize, Clone)]
#[serde(rename_all = "camelCase")]
struct ItemType {
    name: String,
    item_type: i8,
    price: f32,
    image: String,
}

impl TryFrom<&[u8]> for Pinged {
    type Error = anyhow::Error;

    fn try_from(value: &[u8]) -> std::result::Result<Self, Self::Error> {
        serde_json::from_slice::<Pinged>(value)
            .with_context(|| "Could not deserialize value into Pinged model")
    }
}

fn init_logger() -> Result<()> {
    const LOG_LEVEL_CONFIG_VARIABLE: &str = "loglevel";

    let level: LevelFilter = variables::get(LOG_LEVEL_CONFIG_VARIABLE)?
        .parse()
        .map_err(|e| anyhow!("parsing log level: {e}"))?;

    SimpleLogger::new()
        .with_level(level)
        .init()?;

    Ok(())
}

/// A simple Spin HTTP component.
#[http_component]
fn handle_test_spin(req: Request) -> anyhow::Result<impl IntoResponse> {
    info!("method={}, uri={}", req.method(), req.uri());
    init_logger()?;
    let mut router = Router::default();
    router.get("/", get_home_handler);
    router.get("/v1-get-item-types", get_item_types_handler);
    router.post_async("/pinged", post_ping_handler);
    router.get("/dapr/subscribe", get_dapr_subscribe_handler);
    Ok(router.handle(req))
}

fn get_home_handler(_: Request, _: Params) -> Result<impl IntoResponse> {
    Ok(Response::builder()
        .status(200)
        .header("content-type", "text/plain")
        .body("Hello, Fermyon")
        .build())
}

fn get_item_types_handler(_: Request, _: Params) -> Result<impl IntoResponse> {
    let items = json!(get_item_types());
    let result = bytes::Bytes::from(items.to_string());
    Ok(Response::builder()
        .status(200)
        .header("content-type", "application/json")
        .body(Some(result))
        .build())
}

fn get_item_types() -> Vec<ItemType> {
    vec![
        ItemType {
            name: "CAPPUCCINO".to_string(),
            item_type: 0,
            price: 4.5,
            image: "img/CAPPUCCINO.png".to_string(),
        },
        ItemType {
            name: "COFFEE_BLACK".to_string(),
            item_type: 1,
            price: 3.0,
            image: "img/COFFEE_BLACK.png".to_string(),
        }
    ]
}

fn get_dapr_subscribe_handler(_: Request, _params: Params) -> Result<impl IntoResponse> {
    let model = json!([
        {
            "pubsubname": PUB_SUB_NAME,
            "topic": "pinged",
            "routes": {
              "rules": [
                {
                  "match": "event.type == 'pinged'",
                  "path": "/pinged"
                },
              ],
              "default": "/pinged"
            }
        }
    ]);

    let result = bytes::Bytes::from(model.to_string());

    Ok(Response::builder()
        .status(200)
        .header("content-type", "application/json")
        .body(Some(result))
        .build())
}

async fn post_ping_handler(req: Request, _params: Params) -> Result<impl IntoResponse> {
    let dapr_url = variables::get("dapr_url")?;
    info!("# dapr_url: {}", dapr_url);

    let Ok(model) = Pinged::try_from(req.body()) else {
        return Ok(Response::builder()
            .status(400)
            .body(Some("Something wrong."))
            .build());
    };

    info!("post_ping_handler: {:?}", json!(model).to_string());

    pub_ponged(
        dapr_url.as_str(),
        PUB_SUB_NAME,
        "ponged",
        Ponged { id: model.id },
    ).await;

    Ok(Response::builder()
        .status(200)
        .header("content-type", "application/json")
        .body(Some(""))
        .build())
}

async fn pub_ponged(dapr_url: &str, pubsub_name: &str, topic: &str, e: Ponged) {
    let url = format!("{}/v1.0/publish/{}/{}", dapr_url, pubsub_name, topic);
    info!("pub_ponged: {:?}", url.to_string());
    info!("pub_ponged: {:?}", json!(e).to_string());

    let body = bytes::Bytes::from(json!(e).to_string());
    let result = spin_sdk::http::send::<_, Response>(
        RequestBuilder::new(Method::Post, url)
            .header("content-type", "application/json")
            .body(Some(body))
            .build(),
    );

    let result_unwrapped = result.await.unwrap();
    info!("pub_ponged result: {:?}", result_unwrapped.body());
}

Orchestrating apps with .NET Aspire + Dapr host

var builder = DistributedApplication.CreateBuilder(args);

var rabbitUser = builder.AddParameter("RabbitUser");
var rabbitPass = builder.AddParameter("RabbitPassword", true);
var rmq = builder.AddRabbitMQ("rabbitMQ", rabbitUser, rabbitPass)
                   .WithManagementPlugin()
                   .WithEndpoint("tcp", e => e.Port = 5672)
                   .WithEndpoint("management", e => e.Port = 15672);

var stateStore = builder.AddDaprStateStore("statestore");
var pubSub = builder.AddDaprPubSub(
    "pubsub",
    new DaprComponentOptions
    {
        LocalPath = Path.Combine("..", "dapr" , "components", "pubsub-rabbitmq.yaml")
    }).WaitFor(rmq);

var testSpinApp = builder.AddSpinApp("test-spin", workingDirectory: Path.Combine("..", "test-spin"), 
    args: ["--env", $"dapr_url=http://localhost:3500"])
    .WithHttpEndpoint(name: "http", targetPort: 3000, port: 8080)
    .WithDaprSidecar()
    .WithReference(stateStore)
    .WithReference(pubSub);

var webapp = builder.AddProject<Projects.WebApp>("webapp")
    .WithDaprSidecar(o => o.WithOptions(new DaprSidecarOptions { DaprHttpPort = 3500 }))
    .WithReference(stateStore)
    .WithReference(pubSub)
    .WaitFor(testSpinApp);

builder.Build().Run();

Up and running apps

dotnet run --project SpinAppHost/SpinAppHost.csproj

if you are using Visual Studio, then you can press F5 (make sure SpinAppHost is a default running project), and you are on set.

.NET Aspire Dashboard

Look at it. We have all Daprized apps 😍😍😍

Observability

Structure logs

Traces

Overall tracing

PingRequest and PongRequest (pubsub tracing)

curl curl -d '{}' http://localhost:5000/ping/

Service invocation

curl http://localhost:5000/item-types/

Develop a .NET Aspire's custom resource command to re-build SpinApp (Rust)

Our purpose is to re-build the Spin app in the .NET Aspire dashboard and to do that we have a look at https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/custom-resource-commands. The idea is to add a custom resource command to Re-build Spin App.

The code is below

internal class SpinAppResource(string name, string workingDirectory) 
    : ExecutableResource(name, "spin", workingDirectory)
{
}

internal static class SpinAppExtensions
{
    public static IResourceBuilder<SpinAppResource> AddSpinApp(
        this IDistributedApplicationBuilder builder, string name, string command = "up", string workingDirectory = "", string[]? args = null)
    {
        var resource = new SpinAppResource(name, workingDirectory);

        return builder.AddResource(resource)
            .BuildAppCommand(workingDirectory)
            .WithArgs(context =>
            {
                context.Args.Add(command);

                if (args is not null)
                {
                    foreach (var arg in args)
                    {
                        context.Args.Add(arg);
                    }
                }
            })
            .WithOtlpExporter()
            .ExcludeFromManifest();
    }

    public static IResourceBuilder<SpinAppResource> BuildAppCommand(
        this IResourceBuilder<SpinAppResource> builder, string workingDirectory)
    {
        builder.WithCommand(
            name: "build",
            displayName: "Build Spin App",
            executeCommand: context => OnRunReBuildSpinAppCommandAsync(builder, context, workingDirectory),
            iconName: "BuildingFactory",
            iconVariant: IconVariant.Filled);

        return builder;
    }

    private static async Task<ExecuteCommandResult> OnRunReBuildSpinAppCommandAsync(
        IResourceBuilder<SpinAppResource> builder,
        ExecuteCommandContext context,
        string workingDirectory)
    {
        var logger = context.ServiceProvider.GetRequiredService<ResourceLoggerService>().GetLogger(builder.Resource);
        var notificationService = context.ServiceProvider.GetRequiredService<ResourceNotificationService>();

        await Task.Run(async () =>
        {
            var cmd = Cli.Wrap("spin").WithArguments(["build"]).WithWorkingDirectory(workingDirectory);
            var cmdEvents = cmd.ListenAsync();

            await foreach (var cmdEvent in cmdEvents)
            {
                switch (cmdEvent)
                {
                    case StartedCommandEvent:
                        await notificationService.PublishUpdateAsync(builder.Resource, state => state with { State = "Running" });
                        break;
                    case ExitedCommandEvent:
                        await notificationService.PublishUpdateAsync(builder.Resource, state => state with { State = "Finished" });
                        break;
                    case StandardOutputCommandEvent stdOut:
                        logger.LogInformation("{ResourceName} stdout: {StdOut}", builder.Resource.Name, stdOut.Text);
                        break;
                    case StandardErrorCommandEvent stdErr:
                        logger.LogInformation("{ResourceName} stderr: {StdErr}", builder.Resource.Name, stdErr.Text);
                        break;
                }
            }
        });

        return CommandResults.Success();
    }
}

Adding it into .NET Aspire AppHost:

var testSpinApp = builder.AddSpinApp("test-spin", workingDirectory: Path.Combine("..", "test-spin"), 
    args: ["--env", $"dapr_url=http://localhost:3500"])

Then, run the solution again. Look at the picture below:

After clicking on that command, the spin build will trigger, and we can re-build the Spin app.

It's very cool. We can hack it 😎

Conclusion

During the post, we show you how can we make the Spin app and .NET app work together with each other via Dapr's service invocation and pub-sub mechanisms. That's very cool, right?

Next post, we will explore the ability to deploy this sample app on AKS with SpinKube. Stay stunned, we can do even more. Thanks for reading it ❤️

Source code: https://github.com/thangchung/dapr-labs/tree/main/aspire

Part 2: Deploying Distributed Apps (Spin + Dapr) on SpinKube (AKS)