DEV Community: Kamil Mrzygłód

What AMQP compatibility means for a local Azure emulator

Kamil Mrzygłód — Tue, 26 May 2026 12:51:46 +0000

"Supports AMQP" covers a lot of ground. A broker that accepts a TCP connection on port 5671 and exchanges OPEN/BEGIN frames with the client technically speaks AMQP. So does a broker that handles hundreds of thousands of messages per second with full PeekLock semantics, dead-letter routing, and session state. When an emulator claims AMQP compatibility, the interesting question is how far that compatibility actually goes — and specifically, whether it is deep enough for a real message-processing framework to drive it.

This post is about what that means in practice, and why we know Topaz passes the test: because we got it wrong first and had to fix it.

⭐ Star Topaz on GitHub

The two layers of Service Bus compatibility

Most discussions of Azure Service Bus compatibility focus on the control plane: can you create namespaces, queues, and topics through ARM or the Azure CLI? That layer is important — it is what makes az servicebus queue create and azurerm_servicebus_queue work locally — but it is not the interesting layer for message-processing code.

The interesting layer is the AMQP data plane, and it breaks down into two sub-layers:

SDK compatibility — does the Azure Service Bus SDK connect, authenticate, send, and receive? This is the easier bar. The SDK connects through CBS (Claims-Based Security), opens a sender link for sending and a receiver link for receiving, and uses basic settled transfers for most operations. A passable emulator can get this right with a few hundred lines of AMQP link handling code.

Framework compatibility — does a message-processing framework like MassTransit, NServiceBus, or Rebus actually work on top of it? Frameworks drive a more complete subset of the AMQP specification. They open management links alongside receive links, use $management request-response to perform operations the SDK does not surface directly, expect unsettled transfers with explicit client-side settlement, and rely on correct credit replenishment to maintain throughput. These behaviors are not optional features — they are the normal operating path.

The gap between these two bars is larger than it looks.

What MassTransit actually does over AMQP

MassTransit's Azure Service Bus transport (MassTransit.Azure.ServiceBus.Core) uses the Azure SDK as its underlying client but adds a layer of messaging conventions on top. When a ReceiveEndpoint starts, it:

Opens an AMQP session and a receiver link to the queue.
Immediately opens a second link to <queue>/$management — a request-response link used for management operations like com.microsoft:update-disposition and com.microsoft:renew-lock.
Sends an AMQP FLOW frame on the receiver link with initial link credit, indicating how many messages it is prepared to accept.
For every message it processes, sends a DISPOSITION frame to complete or abandon it, then expects the broker to update the session's delivery state and replenish credit.

Step 2 is where most partial AMQP implementations break down. The Azure Service Bus SDK does not surface queue-level $management directly to callers; it is an internal transport detail. The root $management link is handled by IRequestProcessor in most AMQP server implementations, but queue-scoped $management links are distinct — they attach to the queue's link processor, not the root processor. An emulator that routes all $management traffic to one handler will complete the CBS authentication but silently drop every queue management request, causing MassTransit's CompleteAsync to wait 60 seconds for a response that never arrives.

Step 4 is where the second class of failures appears. If the broker sends transfers as sender-settled (the settled bit set in the TRANSFER frame), the receiver never adds the delivery to its unsettled map. When MassTransit calls CompleteAsync, the SDK sees no pending delivery with that lock token, settlement happens locally without waiting for broker confirmation, and no DISPOSITION frame is sent. The broker never gets the acknowledgement it expects. Credit is consumed but never replenished. After the first message, the consumer stops receiving.

The bugs we found

Running MassTransit against an early version of Topaz exposed exactly these two failure modes.

Bug 1: Missing queue $management handler. Topaz already handled the root $management link for CBS token validation. Queue-scoped management links were being attached to the link processor without a handler. MassTransit's CompleteAsync calls timed out after 60 seconds with an amqp:internal-error.

The fix required intercepting ATTACH frames addressed to <anything>/$management inside LinkProcessor and routing them to a dedicated request-response endpoint — separate from the root management handler. On the sender side, the endpoint registers a request processor that reads the operation property from incoming application properties, builds the appropriate response (status code, correlation ID, operation-specific payload for com.microsoft:renew-lock), and sends it back on the paired response link.

Bug 2: Wrong management response property names. Once queue management requests were being answered, MassTransit's completion path started working — but threw amqp:internal-error (GeneralError) instead of returning. Decompiling the Azure SDK revealed the issue:

public static AmqpResponseStatusCode GetResponseStatusCode(this AmqpMessage responseMessage)
{
    // reads responseMessage.ApplicationProperties.Map["statusCode"]
}

The initial management responses used status-code and status-description — the property names from the CBS specification. Service Bus management uses camelCase: statusCode and statusDescription. The SDK parsed the response, found no statusCode key, and treated the reply as a failure. Changing two string literals fixed it.

Bug 3: Sender-settled transfers. After the response key fix, the amqp:internal-error disappeared. But only the first message was consumed; the consumer then sat idle. Looking at the AMQP frame trace showed: one FLOW frame from the consumer link, one outgoing TRANSFER, then silence. No new FLOW frame, no credit replenishment.

The cause was in OutgoingLinkEndpoint. Topaz was setting the Settled property on outgoing deliveries to true — sender-settled transfers. The SDK never put the delivery in the unsettled map. CompleteAsync short-circuited. No DISPOSITION came back. No credit was restored. The fix was a one-line change:

// Before: sender-settled — credit consumed without replenishment
DeliverySettledProperty.SetValue(delivery, true);

// After: unsettled — let the receiver settle explicitly via DISPOSITION
DeliverySettledProperty.SetValue(delivery, false);

The consumer now receives a message, completes it, the broker sees the DISPOSITION and returns credit — standard PeekLock behaviour.

What the working example looks like

The Topaz.Examples.MassTransit project in the repository is a minimal ASP.NET Core app that starts a Topaz container via Testcontainers, provisions a Service Bus namespace and queue via the ARM API, and wires up MassTransit with a consumer:

builder.Services.AddMassTransit(x =>
{
    x.AddConsumer<MessageConsumer>();
    x.UsingAzureServiceBus((context, cfg) =>
    {
        cfg.Host(TopazResourceHelpers.GetServiceBusConnectionStringWithTls("sbnamespace"));
        cfg.ReceiveEndpoint("sbqueue", e =>
        {
            e.ConfigureConsumer<MessageConsumer>(context);
            e.PrefetchCount = 1;
        });
    });
});

MassTransit uses the TLS endpoint (port 5671) because it expects a standard Azure Service Bus connection string without UseDevelopmentEmulator=true. The non-TLS endpoint (port 8889) uses pre-settled receive-and-delete semantics — compatible with the Azure SDK's development emulator mode, but not with how MassTransit drives the receive path. For any framework that manages its own PeekLock cycle, the TLS endpoint is the right choice.

The worker sends one message per second. With the three bugs above fixed, the output is exactly one Message dispatched and one Message consumed per second — sustained indefinitely:

Message dispatched: {"Timestamp":"2026-05-26T09:14:35.86...","Message":"The time is ..."}
Message consumed:   {"Timestamp":"2026-05-26T09:14:35.86...","Message":"The time is ..."}
Message dispatched: {"Timestamp":"2026-05-26T09:14:36.97...","Message":"The time is ..."}
Message consumed:   {"Timestamp":"2026-05-26T09:14:36.97...","Message":"The time is ..."}

Why MassTransit support is a meaningful signal

MassTransit is not the only framework that exercises this part of the AMQP specification, but it is a widely-used one in the .NET ecosystem and it drives all three of the failure modes described above simultaneously. If Topaz runs MassTransit end-to-end, it means:

Queue-scoped $management request-response is implemented and returns correct responses.
Transfers are unsettled, so receivers that manage their own settlement cycle work correctly.
Credit replenishment is functional, so sustained throughput is possible without the consumer stalling.

NServiceBus's Azure Service Bus transport and any other library that implements the full PeekLock cycle over the Azure SDK should follow the same path.

What this does not guarantee: dead-letter queues, message sessions, topic subscription rules, and partitioned entities are not yet implemented. If your application depends on those features, the Azure Service Bus Emulator still has the advantage on that specific surface. Topaz's roadmap tracks when those features are coming.

The comparison with the Azure Service Bus Emulator

The Microsoft emulator ships as two Docker containers (emulator plus SQL Server), configures entities via a static config.json at startup, and does not implement the ARM control plane. What it does have is a more complete messaging feature set at the moment: dead-letter queues, message sessions, and topic filters work today.

The trade-off is concrete. If you need az servicebus queue create to work locally, Terraform azurerm_servicebus_queue to apply locally, or multiple namespaces in the same environment, the emulator cannot help — it has no ARM API. If you need dead-letter queues or message sessions, Topaz cannot yet help.

For teams in between — who need a real PeekLock consumer with sustained receive throughput and ARM-level infrastructure tooling in the same local process — Topaz is the current answer.

Let Copilot handle your local Azure setup via MCP

Kamil Mrzygłód — Thu, 21 May 2026 08:15:21 +0000

The best GitHub Copilot experience for local Azure development is not "write me a bash script that calls az group create". It is telling the assistant what you need to build and having it wire up the local infrastructure while you write the application code. Instead of tab-switching to a terminal, remembering the right parameter names, and sequencing five CLI commands in the right order, you describe the stack in natural language and the assistant handles it — Key Vault, Storage Account, Service Bus namespace, all provisioned, connection strings returned, ready to paste into your configuration file.

Topaz ships a Model Context Protocol (MCP) server that makes this possible against the local emulator. This post covers how the MCP server works, why the Docker networking setup is non-trivial, and what the full setup looks like end to end.

⭐ Star Topaz on GitHub

What MCP actually does here

The Model Context Protocol is a standard that lets AI assistants call external tools and receive structured results — creating real side effects rather than generating commands for you to copy-paste and run manually. When Copilot calls a tool, the call goes to the MCP server process, the server executes it against a live Topaz instance, and the result comes back to the assistant. The assistant sees real data: the vault URI of the Key Vault it just created, the connection string for the Service Bus namespace, the login server for the Container Registry.

This matters because the assistant can chain those results. After creating a Storage Account, it can pass the connection string directly to the next tool call that creates a Key Vault secret — without you manually copying values between terminal windows.

The Topaz MCP server exposes two kinds of capabilities:

Tools — individual operations: create a resource group, provision a Key Vault, fetch all connection strings in a subscription, check emulator health.
Prompts — pre-defined multi-step recipes that wire tools together into complete scenarios: bootstrap a full dev environment, set up a Functions-ready stack, provision a document processing pipeline.

The Docker connectivity problem

When you run the MCP server as a Docker container — which is the recommended way to distribute it, so clients do not need the .NET runtime — it needs to reach the Topaz host over the network. The Topaz host also runs as a Docker container. Two containers running on the same machine are not on the same network by default; they cannot reach each other by hostname.

The first instinct is --network host. The Docker documentation describes this as "the container shares the host's network namespace". On Linux this is exactly what it means — the container sees localhost, all published ports, and everything else on the host's network stack. In practice, this works when Topaz runs directly on the host machine. But it breaks as soon as both Topaz and the MCP server are running as containers, because localhost inside the MCP container is the Linux VM's loopback, not a path to the Topaz container.

Even on Linux, where --network host does give the MCP container access to the host network stack, there is a second problem: wildcard subdomains. The Topaz MCP tools do not only call topaz.local.dev:8899 for ARM operations. When the Key Vault tool calls the Key Vault data-plane, it calls <vault-name>.vault.topaz.local.dev:8898. When the Container Registry tool calls the registry endpoint, it calls <registry-name>.cr.topaz.local.dev:8892. These subdomains are dynamic — they depend on resource names the user provides at runtime. --network host gives network access but does nothing for DNS; topaz.local.dev might resolve if the host has it in /etc/hosts, but no host-machine /etc/hosts entry covers my-vault.vault.topaz.local.dev.

The correct solution has two parts: a shared Docker network, and a wildcard DNS resolver.

The network and DNS setup

The RunTopazAsContainer tool returns a shell command that sets up the full environment in one step. The command does three things:

1. Create a user-defined bridge network with a fixed subnet:

docker network create --subnet 172.28.0.0/16 topaz-net

User-defined networks have Docker's built-in DNS — containers can resolve each other by name. The fixed subnet allows assigning stable IP addresses, which the dnsmasq configuration depends on.

2. Start a lightweight DNS resolver at a fixed IP:

docker run -d --name topaz-dns \
  --network topaz-net --ip 172.28.0.53 \
  alpine sh -c "apk add -q --no-cache dnsmasq && \
    dnsmasq --no-daemon --no-resolv --server=8.8.8.8 \
            --address=/.topaz.local.dev/172.28.0.10"

The --address=/.topaz.local.dev/172.28.0.10 directive tells dnsmasq to resolve every hostname that ends in .topaz.local.dev — at any depth — to 172.28.0.10. That covers topaz.local.dev itself, my-vault.vault.topaz.local.dev, stdev.blob.storage.topaz.local.dev, any registry name, any Service Bus namespace. The resolver is at 172.28.0.53 following DNS convention; all queries it does not know about forward to 8.8.8.8.

3. Start the Topaz host container at the fixed IP:

docker run -d --name topaz.local.dev \
  --network topaz-net --ip 172.28.0.10 \
  -p 8899:8899 -p 8898:8898 ... \
  thecloudtheory/topaz-host:<version>

The Topaz container is named topaz.local.dev. On a user-defined network, Docker DNS resolves container names, so even without the dnsmasq sidecar, the base topaz.local.dev hostname would resolve by container name alone. The dnsmasq sidecar handles everything else.

The MCP container connects to the same network and points at the DNS sidecar:

docker run -i --network topaz-net --dns 172.28.0.53 thecloudtheory/topaz-mcp:<version>

The --dns flag is set at container creation time — it controls what Docker writes into the container's /etc/resolv.conf before any process starts. Once the MCP container is running, topaz.local.dev and all its subdomains resolve to 172.28.0.10, which is the Topaz host.

The certificate

Every Topaz endpoint is HTTPS. The Topaz host uses a self-signed certificate that covers *.topaz.local.dev and all its subdomain patterns. The MCP container image has this certificate pre-installed in the Ubuntu system CA store — it is baked in during the Docker image build the same way the Compose example app handles it:

FROM ubuntu:noble AS cert-builder
RUN apt-get update -qq && apt-get install -y --no-install-recommends ca-certificates
COPY certificate/topaz.crt /usr/local/share/ca-certificates/topaz.crt
RUN update-ca-certificates

FROM mcr.microsoft.com/dotnet/runtime-deps:10.0-noble-chiseled AS final
COPY --from=cert-builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

The cert-builder stage runs update-ca-certificates (not available in the minimal chiseled final image) and the updated CA bundle is copied across. The MCP server process inherits the system CA store and trusts Topaz's certificate without any code-level bypass.

Setting it up in VS Code

The one-time setup is:

# Create the shared Docker network once — survives reboots until you remove it manually
docker network create --subnet 172.28.0.0/16 topaz-net

Then add the server to .vscode/mcp.json:

{
  "servers": {
    "Topaz": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "--network", "topaz-net",
        "--dns", "172.28.0.53",
        "thecloudtheory/topaz-mcp:<version>"
      ]
    }
  }
}

After saving, VS Code prompts you to start the server. It appears in the MCP Servers panel, and GitHub Copilot picks up the tools automatically.

The first time you use the MCP server, ask Copilot to start Topaz:

"Start Topaz using the latest beta image for Apple Silicon."

Copilot calls RunTopazAsContainer with platform=linux/arm64, gets back a shell command, and tells you to run it. After that, every provisioning tool has a live Topaz instance to talk to. Depending on the LLM you're using you can just skip the platform architecture part:

"Start Topaz using the latest beta image."

Most of the models will be able to infer the architecture automatically and pass the correct platform parameter.

What a real session looks like

Once the emulator is running, provisioning a complete local dev environment is a single chat message:

"Create a subscription called dev-local with ID 10000000-0000-0000-0000-000000000001, a resource group rg-dev in westeurope, a storage account stdevlocal001, a Service Bus namespace sbdevlocal with a queue called orders, and a Key Vault kv-dev with a secret connection-string set to the Service Bus connection string. Use superadmin access."

Copilot issues seven tool calls in sequence:

CreateSubscription(subscriptionId=10000000-..., subscriptionName=dev-local, objectId=00000000-...)
CreateResourceGroup(subscriptionId=10000000-..., resourceGroupName=rg-dev, location=westeurope, ...)
CreateStorageAccount(...)
CreateServiceBusNamespace(...)
CreateServiceBusQueue(namespaceName=sbdevlocal, queueName=orders, ...)
CreateKeyVault(...)
  → vaultUri = https://kv-dev.vault.topaz.local.dev:8898
  → seededSecret = connection-string
GetConnectionStrings(subscriptionId=10000000-..., ...)

The final GetConnectionStrings call returns a structured list:

Storage account stdevlocal001:
  connectionString: DefaultEndpointsProtocol=https;AccountName=stdevlocal001;...
  blobServiceUri:   https://stdevlocal001.blob.storage.topaz.local.dev:8891/
  queueServiceUri:  https://stdevlocal001.queue.storage.topaz.local.dev:8893/

Service Bus namespace sbdevlocal:
  connectionString: Endpoint=sb://sbdevlocal.servicebus.topaz.local.dev:5671;...

Key Vault kv-dev:
  vaultUri: https://kv-dev.vault.topaz.local.dev:8898

Everything in that output is a real, reachable endpoint backed by a live Topaz instance. You can paste the connection strings directly into appsettings.Development.json or a .env file and start the application.

Prompts: pre-defined stacks

For common scenarios, the MCP server exposes prompts — multi-step recipes that wire the individual tools together so you do not have to specify the sequence yourself. Invoke them by name in the chat:

bootstrap-topaz — first-time setup. Starts the container, registers a subscription, creates a resource group, confirms health. This is the entry point before any provisioning prompt.

setup-functions-local-dev — provisions a Storage Account (for AzureWebJobsStorage), a Service Bus namespace and queue (for trigger), and a Key Vault with the storage connection string already stored as a secret. Returns a ready-to-paste local.settings.json snippet.

setup-event-driven-microservice — creates a Service Bus namespace with a command queue and an event topic (with a subscription), plus a Key Vault with the connection string. Models the write-side/read-side split directly.

setup-multi-tenant-fixtures — takes a list of tenant names and a naming prefix, then creates an isolated subscription, resource group, storage account, and Key Vault for each tenant. Useful for testing tenant isolation or seeding multi-tenant integration tests.

inspect-environment — runs a health check, lists subscriptions, and returns connection strings for every provisioned resource in one pass. Useful when you return to a session and want to know the current state.

Each prompt is a structured instruction message that tells Copilot exactly which tools to call and in which order. You supply the parameter values; the prompt handles the sequencing.

When this is most useful

Onboarding. A new developer cloning a repository does not need to know which Azure services the application uses or how to set them up locally. They ask Copilot to bootstrap the environment, Copilot reads the project context, picks the right prompt, and the infrastructure is ready before they have finished reading the README.

Integration testing setup. Instead of maintaining a shared dev Azure subscription with manually-created test resources, each developer runs their own full local stack. Resources are created fresh per session, so tests never share state and there is nothing to clean up in a shared environment.

Infrastructure experimentation. Testing a new Service Bus topology or a multi-tenant naming convention against a real API before committing to it is fast when the infrastructure is local. Create it, test the application behaviour, tear it down, adjust the design, repeat — all in one chat session.

CI. The same MCP workflow that provisions resources locally can provision them inside a CI job. The GetConnectionStrings output feeds directly into the test runner environment. Topaz starts in a Docker container in the CI step, the MCP server provisions what the tests need, the tests run against real emulated endpoints.

Tearing down

When you are done with a session:

"Stop Topaz and clean up the containers."

Copilot calls StopTopazContainer, which stops the Topaz host container, the topaz-dns resolver container, and removes the topaz-net network. The next session starts fresh.

If you just want to check what is running before deciding whether to stop it:

"Is Topaz running and which services are up?"

GetTopazStatus hits the health endpoint and probes all service ports, returning a per-service reachability report. Useful when something is not behaving as expected and you want to rule out an infrastructure problem before looking at the application code.

The design constraint that shaped everything

The most important thing the MCP server does differently from a simple "generate az commands" approach is that it calls real APIs and returns real data. When CreateKeyVault returns a vault URI, that URI resolves to a live endpoint. When GetConnectionStrings returns a connection string, it connects to a live Service Bus namespace. The assistant is not simulating infrastructure — it is creating it.

That distinction is what makes the DNS and certificate setup matter. A generated az keyvault create command does not care whether the domain resolves inside a container. A real SecretClient call to https://kv-dev.vault.topaz.local.dev:8898 does. The dnsmasq sidecar and the pre-installed certificate are not operational overhead — they are the thing that makes the data the assistant returns actually usable by an application.

Building a devcontainer for Topaz: workspace mounts, DNS wildcards, and why /etc/resolv.conf always wins

Kamil Mrzygłód — Tue, 12 May 2026 14:23:29 +0000

The "Open in Dev Container" badge is one of the highest-friction-to-value improvements you can make to a developer tool. Someone lands on your README, clicks the badge, waits for a container to build, and starts using the tool — no install steps, no certificate trust ceremony, no "is port 8899 available on this machine" questions. The conversion from "interesting project" to "I am actually running this" happens in the time it takes Docker to pull an image.

Getting there, however, requires navigating a specific class of problems that are not particularly hard once you understand them but are completely opaque until you do. This post is a technical account of building the Topaz devcontainer: the three services that ended up in the Docker Compose file, why /etc/resolv.conf defeated two consecutive DNS approaches, and what the working architecture looks like.

By the way - if you're interested in further development of Topaz or just want to appreciate the idea, feel free to ⭐ Star Topaz on GitHub

What we were building

Topaz is a single binary that emulates Azure Storage, Key Vault, Service Bus, Event Hubs, Container Registry, Managed Identity, RBAC, ARM, and Entra ID. The devcontainer goal was straightforward: when a developer opens the repository in VS Code, Topaz Host should already be running, all the service ports should be reachable, TLS certificates should be trusted, and DNS wildcards for *.topaz.local.dev should resolve — without any manual setup steps.

The target end state was a developer terminal like this:

vscode ➜ /workspaces/Topaz $ topaz health
{"workingDirectory":"/app","version":"...","status":"Healthy"}

vscode ➜ /workspaces/Topaz $ curl https://topaz.local.dev:8899/health
{"workingDirectory":"/app","version":"...","status":"Healthy"}

vscode ➜ /workspaces/Topaz $ curl https://my-vault.vault.topaz.local.dev:8898/secrets
{"value":[],"nextLink":null}

The second and third commands — using hostnames rather than IP addresses — are the ones that turned out to be the interesting engineering problem.

The initial architecture: three services in Docker Compose

The devcontainer uses Docker Compose mode, which VS Code's Dev Containers extension supports directly. The natural structure maps to three services:

services:
  devcontainer:   # the VS Code workspace container
  topaz:          # the Topaz Host sidecar
  dns-sidecar:    # wildcard DNS resolver (more on this later)

All three share a bridge network with a fixed subnet (172.28.0.0/16) and static IP assignments:

devcontainer at 172.28.0.2
topaz at 172.28.0.10
dns-sidecar at 172.28.0.53

The fixed IPs are important. DNS configuration needs to point at an address that is stable before any container starts, and the address=/.topaz.local.dev/172.28.0.10 dnsmasq rule needs to know where Topaz lives without dynamic lookup.

Problem one: workspace mount in Compose mode

The first thing that breaks when you move a devcontainer to Docker Compose mode is workspace mounting. In a single-container devcontainer, VS Code automatically bind-mounts your local workspace folder using the workspaceMount property. In Compose mode, VS Code tries to inject a workspace mount into a generated override compose file, but the injection is unreliable — particularly when the workspace is on an external drive or a path the container runtime does not have in its file-sharing configuration.

The first sign of the problem is an error like:

OCI runtime exec failed: exec failed: unable to start container process:
chdir to cwd ("/workspaces/Topaz") set in config.json failed: no such file or directory

The /workspaces/Topaz directory does not exist because the bind mount never happened. Adding workspaceMount to devcontainer.json looks like it should help, but the Dev Containers schema explicitly does not allow it in Compose mode — VS Code will flag it as an unknown property and ignore it.

The reliable fix is to declare the workspace mount explicitly in the Docker Compose file:

devcontainer:
  image: mcr.microsoft.com/devcontainers/base:ubuntu
  volumes:
    - ../:/workspaces/Topaz:cached
  command: sleep infinity

The .. is relative to .devcontainer/, so it resolves to the repository root. workspaceFolder in devcontainer.json must match the target path exactly:

"workspaceFolder": "/workspaces/Topaz"

This works when the container runtime can access the source path. When it cannot — because you are using Colima and an external drive is not in its mount list — Docker creates the bind-mount target as an empty directory rather than failing loudly, which is how we ended up with topaz.crt and topaz.key appearing as empty directories inside the container instead of files. Solving that for certificates is what led to the second problem.

Problem two: distributing TLS certificates without bind mounts

Topaz requires a TLS certificate to start. The certificate files live in certificate/topaz.crt and certificate/topaz.key in the repository. The straightforward approach — bind-mounting them into the Topaz sidecar — fails on Colima (and on Docker Desktop when the path is outside the configured file-sharing list) for the same reason workspace mounts fail: the bind mount silently becomes an empty directory.

The Docker Compose example in the repository solved this a different way: using docker cp to populate a named volume, the same mechanism Testcontainers' WithResourceMapping uses internally. A named volume is always accessible to containers regardless of which paths the runtime has permission to bind-mount from the host.

The devcontainer uses the same pattern. A shell script at .devcontainer/init-certs.sh runs via initializeCommand in devcontainer.json:

"initializeCommand": "bash .devcontainer/init-certs.sh"

initializeCommand runs on the host machine before any container starts, which means it has access to the certificate files at their actual paths:

VOLUME="topaz-devcontainer-certs"
CONTAINER=$(docker create -v "$VOLUME:/certs" alpine)
docker cp "$SCRIPT_DIR/topaz.crt" "$CONTAINER:/certs/topaz.crt"
docker cp "$SCRIPT_DIR/topaz.key" "$CONTAINER:/certs/topaz.key"
docker rm "$CONTAINER"

The Topaz sidecar then mounts the named volume rather than bind-mounting the files directly:

topaz:
  volumes:
    - topaz-devcontainer-certs:/certs:ro
    - topaz-data:/app/.topaz
  command:
    - --certificate-file
    - /certs/topaz.crt
    - --certificate-key
    - /certs/topaz.key

The named volume is declared as external: true in the compose file so Docker Compose does not try to create it (the initializeCommand already did that).

Problem three: DNS for *.topaz.local.dev

This was the longest part of the investigation. Topaz services are reached at subdomains of topaz.local.dev — my-vault.vault.topaz.local.dev, myaccount.blob.storage.topaz.local.dev, myregistry.cr.topaz.local.dev — and these need to resolve to the Topaz sidecar IP inside the devcontainer. There is no wildcard support in /etc/hosts, so a single extra_hosts entry for topaz.local.dev is not enough; every named resource would need a manual entry.

The approach we tried first, failed, tried second, and failed in a different way before arriving at the working solution is worth describing in order.

Attempt 1: extra_hosts in docker-compose.yml

Docker Compose supports extra_hosts, which injects entries into /etc/hosts. The initial configuration added topaz.local.dev to the devcontainer service:

devcontainer:
  extra_hosts:
    - "topaz.local.dev:172.28.0.10"

This did not work — not because the mechanism is wrong, but because VS Code generates a compose override file when it starts the devcontainer, and the override appears to discard or override extra_hosts from the base file. The host entry never appeared in /etc/hosts inside the container.

Even if it had worked, it would only have solved topaz.local.dev. Every vault, every storage account, every registry would still need a manual entry. The extra_hosts approach was the wrong layer entirely.

Attempt 2: dnsmasq installed inside the devcontainer

The next approach was to install dnsmasq inside the devcontainer via postCreateCommand and configure it to resolve *.topaz.local.dev to 172.28.0.10. This is how the existing install-linux.sh script works for non-container Linux installs.

The first failure was a port 53 conflict. The devcontainer base image runs systemd-resolved or has a stub resolver already listening on UDP port 53. Killing it with systemctl stop systemd-resolved produced:

"systemd" is not running in this container due to its overhead.
Use the "service" command to start services instead.

systemd does not run in devcontainers. The next attempt used fuser -k 53/udp to kill whatever was on the port, which worked as a one-time fix.

The second failure was /etc/resolv.conf. To make the container query our dnsmasq instance first, we needed to prepend nameserver 127.0.0.1 to /etc/resolv.conf. The file appeared writable, and our tee overwrote it — but when a new shell opened, the change was gone. Docker bind-mounts /etc/resolv.conf from a file managed by the container runtime. You can overwrite it with tee within a single process's lifetime, but the next container process re-reads it from the bind-mounted source. You cannot delete it (rm fails with "Device or resource busy"). Any content written to it via tee in a postCreateCommand is effectively transient.

The third failure was timing. postCreateCommand runs once after container creation. postStartCommand runs on every start. dnsmasq installed during postCreateCommand would not be running when the container was restarted — and postStartCommand cannot install dnsmasq (apt is not available until postCreateCommand has run). The lifecycle ordering makes in-container dnsmasq unreliable after a restart regardless of the resolv.conf problem.

The working solution: dnsmasq as a sidecar service

The correct layer for this problem is not inside the devcontainer — it is in Docker Compose. Docker sets /etc/resolv.conf based on the compose dns: directive before any container process starts, and the value persists for the lifetime of the container without anything inside the container being able to overwrite it.

The solution is a dedicated DNS sidecar — a minimal alpine container running dnsmasq — and a dns: entry on the devcontainer service pointing at it:

dns-sidecar:
  image: alpine:latest
  command: >
    sh -c "apk add --no-cache dnsmasq -q &&
           echo 'address=/.topaz.local.dev/172.28.0.10' > /etc/dnsmasq.d/topaz.conf &&
           dnsmasq --no-daemon --server=1.1.1.1 --server=8.8.8.8"
  networks:
    topaz-net:
      ipv4_address: "172.28.0.53"
  restart: unless-stopped

devcontainer:
  dns:
    - 172.28.0.53    # topaz DNS sidecar (resolves *.topaz.local.dev)
    - 1.1.1.1        # fallback internet DNS
  depends_on:
    - dns-sidecar

When Docker Compose starts the devcontainer service, it writes:

nameserver 172.28.0.53
nameserver 1.1.1.1

into the container's /etc/resolv.conf. This is a file Docker controls, not one we are trying to modify from inside the container. Any DNS lookup that reaches 172.28.0.53 gets the address=/.topaz.local.dev/172.28.0.10 answer for Topaz subdomains and forwards everything else to 1.1.1.1.

The depends_on ensures the DNS sidecar is up before the devcontainer tries to use it. The restart: unless-stopped keeps it running across container restarts. Because it is a separate service, the dnsmasq lifecycle is entirely independent of anything happening in postCreateCommand or postStartCommand.

After this change:

vscode ➜ /workspaces/Topaz $ curl https://topaz.local.dev:8899/health
{"workingDirectory":"/app","version":"...","status":"Healthy"}

vscode ➜ /workspaces/Topaz $ curl https://my-vault.vault.topaz.local.dev:8898/secrets
{"value":[],"nextLink":null}

No hosts-file entries. No manual entries per resource. Any resource created in Topaz with any name resolves automatically.

What postCreate.sh does now

With DNS and certificate distribution handled at the compose layer, postCreateCommand is left with a smaller, cleaner set of responsibilities:

Trust the Topaz TLS certificate in the Ubuntu system CA store. This is a one-time operation — update-ca-certificates ingests the cert so that curl, the Azure SDK, and any other TLS client that uses the system store trusts Topaz's self-signed certificate without --insecure.
Inject the certificate into the Azure CLI certifi bundle. The Azure CLI ships its own bundled cacert.pem separate from the system store. Without this step, az rest --url https://topaz.local.dev:8899/... fails with an SSL error even though curl works fine. The script finds the bundle at the path the azure-cli devcontainer feature installs it (/opt/az/lib/python*/site-packages/certifi/cacert.pem) and appends the Topaz cert.
Install the Topaz CLI. The CLI (topaz) and host binary (topaz-host) are downloaded from the GitHub release. The version is resolved from /releases rather than /releases/latest because Topaz is currently in beta and beta releases do not appear at the latest endpoint. The install is best-effort — if it fails (wrong architecture, network issue), the script continues rather than aborting the entire postCreateCommand.
Write shell environment variables. AZURE_TENANT_ID (the Topaz default tenant) and REQUESTS_CA_BUNDLE (pointing at the system certificate store for Python-based tools) are appended to ~/.bashrc idempotently.

The final compose structure

networks:
  topaz-net:
    driver: bridge
    ipam:
      config:
        - subnet: "172.28.0.0/16"

volumes:
  topaz-data: {}
  topaz-devcontainer-certs:
    external: true

services:
  devcontainer:
    image: mcr.microsoft.com/devcontainers/base:ubuntu
    volumes:
      - ../:/workspaces/Topaz:cached
    command: sleep infinity
    dns:
      - 172.28.0.53
      - 1.1.1.1
    depends_on:
      - dns-sidecar
    networks:
      topaz-net:
        ipv4_address: "172.28.0.2"

  topaz:
    image: thecloudtheory/topaz-host:latest
    platform: linux/amd64
    networks:
      topaz-net:
        ipv4_address: "172.28.0.10"
    ports:
      - "8899:8899"
      - "8898:8898"
      - "8892:8892"
      - "8891:8891"
      # ... all service ports
    volumes:
      - topaz-devcontainer-certs:/certs:ro
      - topaz-data:/app/.topaz
    command:
      - --certificate-file
      - /certs/topaz.crt
      - --certificate-key
      - /certs/topaz.key

  dns-sidecar:
    image: alpine:latest
    command: >
      sh -c "apk add --no-cache dnsmasq -q &&
             echo 'address=/.topaz.local.dev/172.28.0.10' > /etc/dnsmasq.d/topaz.conf &&
             dnsmasq --no-daemon --server=1.1.1.1 --server=8.8.8.8"
    networks:
      topaz-net:
        ipv4_address: "172.28.0.53"
    restart: unless-stopped

Three services. Two volumes. One network. The certificate distribution happens before any container starts (initializeCommand). The DNS configuration happens when Docker Compose starts the devcontainer service (dns:). The certificate trust and CLI install happen once after container creation (postCreateCommand).

The constraints that shaped the design

Most of the intermediate failures came from the same underlying constraint: inside a running container, the parts of /etc/resolv.conf and /etc/hosts that Docker manages are not ours to own. Docker bind-mounts both files. You can read them, you can overwrite them with a new process, but the bind mount source is what Docker wrote when it started the container — and that is what survives restarts, new shells, and exec sessions.

The only way to control what nameservers a Docker container queries is to set dns: in the compose service definition before the container starts. Once it is running, you are working around a constraint rather than owning the solution. Every in-container approach — dnsmasq in postCreateCommand, tee to /etc/resolv.conf, fuser to free port 53 — is a workaround with a lifecycle problem attached to it. The sidecar approach is not.

The same logic applies to the workspace mount. In Compose mode, the workspace bind mount is injected by VS Code into a generated override file. The injection is sometimes unreliable — particularly on external drives or when the container runtime has not been configured to share those paths. Declaring the mount explicitly in the base compose file makes it deterministic. The explicit declaration wins over whatever VS Code's override would have injected.

What this enables

The devcontainer is the easiest path to a Topaz environment for anyone who does not want to go through the certificate trust and DNS configuration steps manually. Click the badge, wait for the build, open a terminal:

# DNS wildcard resolves automatically — no hosts file editing
vscode ➜ /workspaces/Topaz $ topaz health
{"status":"Healthy"}

# HTTPS with the Topaz cert trusted — no --insecure needed
vscode ➜ /workspaces/Topaz $ az rest --method get \
  --url "https://topaz.local.dev:8899/subscriptions?api-version=2020-01-01"

# Named resources work immediately after creation
vscode ➜ /workspaces/Topaz $ az keyvault create \
  --name my-vault --resource-group rg-dev --location westeurope
vscode ➜ /workspaces/Topaz $ curl https://my-vault.vault.topaz.local.dev:8898/secrets?api-version=7.4

The devcontainer is available at the root of the repository and as a standalone copy-into-your-project template in Examples/Devcontainer/. The standalone template bundles its own certificate copy so it has no dependency on the certificate/ directory — teams can drop .devcontainer/ into any project, add their resource names as needed, and get the same environment.

Topaz vs Azurite: what actually works locally and what doesn't

Kamil Mrzygłód — Tue, 05 May 2026 11:59:04 +0000

If you have ever written a line of Azure code on a laptop, you have used Azurite. It is the official local emulator for Azure Storage, ships in every Visual Studio install, and runs unchanged in tens of thousands of CI pipelines. For Storage-only workloads it is an excellent tool — Microsoft maintains it, Azure SDKs target it, and the parity with the real Azure Storage REST API is strong.

The problem is that real applications stop at Azure Storage roughly never. The moment you reach for a secret in Key Vault, publish a message to Service Bus, push an image to a Container Registry, or want a DefaultAzureCredential chain that does not silently fall back to interactive browser auth, Azurite has nothing to offer. You are left bolting together a Service Bus emulator from a community Docker image, mocking the Key Vault SDK in tests, and hoping that the way your CI fakes Entra tokens does not drift away from how production behaves.

Topaz is a single .NET 8 (and when 1.3 version is released - .NET 10) binary that emulates Azure Storage, Key Vault, Service Bus, Event Hubs, Container Registry, Managed Identity, RBAC, ARM, and a working Entra ID layer in one process. This post is an honest comparison between the two, focused on what developers who already know Azurite actually run into.

Run the Azure SDK, Azure CLI, Terraform, and docker push against a local emulator — no Azure subscription, no service principal, no cloud charges.
brew tap thecloudtheory/topaz && brew install topaz && topaz-host   # macOS
curl -fsSL https://raw.githubusercontent.com/TheCloudTheory/Topaz/main/install/get-topaz.sh | bash   # Linux
Getting started →

Where Topaz and Azurite agree

It is worth being clear about this up front: for Azure Storage, Azurite is good. Anyone telling you otherwise is selling something. Topaz does not exist because Azurite is bad at Storage — it exists because Azurite is only Storage, and most real Azure applications are not.

For Blob, Queue, and Table data plane operations, the two emulators are at full parity on the cases that matter for day-to-day development:

Storage feature	Topaz	Azurite
Blob basic operations (put, get, delete, head, list)	Yes	Yes
Blob metadata, container metadata	Yes	Yes
Block blobs (`Put Block`, `Put Block List`, `Get Block List`)	Yes	Yes
Page blobs (`Put Page`, `Get Page Ranges`)	Yes	Yes
Container ACLs and stored access policies	Yes	Yes
Container and blob leases (acquire / renew / change / release / break)	Yes	Yes
Blob copy and snapshots	Yes	Yes
Table create / delete / query / entity CRUD	Yes (stable)	Yes (preview)
Table ACL (stored access policies)	Yes	Yes
Queue messages (enqueue, dequeue, peek, update, delete, clear)	Yes	Yes
Queue ACL	Yes	Yes
RA-GRS secondary endpoints	Partial (roadmap)	Yes

The two practical Storage gaps in Topaz today are Account / Service SAS token validation on the data plane (the ARM endpoints that generate SAS strings exist; the data plane providers do not yet recognise the ?sv=...&sig=... parameters in incoming requests) and full RA-GRS secondary endpoint emulation. Both are scheduled — SAS validation is on the v1.4-beta milestone, RA-GRS is on v1.4-beta as well. More on that further down.

The Storage feature that Azurite still does not have is multiple named storage accounts as a first-class concept. Azurite's default account is devstoreaccount1. Adding more requires editing your hosts file, exporting AZURITE_ACCOUNTS with name:key1:key2;name:key1:key2, and restarting the emulator. Topaz has a real ARM control plane: az storage account create --name sa-orders --resource-group rg-local works the same way it works against Azure, registers a DNS entry automatically, and gives you a real connection string you can paste into Azure Storage Explorer or hand to Terraform.

Where Azurite stops: Key Vault

This is the largest single delta between the two emulators, and it is the reason most developers eventually try to replace Azurite.

Azurite has no Key Vault. Not a partial one, not a stubbed one — none. The standard workarounds are:

Mock the SecretClient in tests (loses any coverage of the actual auth layer).
Read secrets from appsettings.Development.json instead (creates a code-path divergence between local and production).
Spin up a real Key Vault per developer (works, but now CI needs Azure credentials and the developer onboarding doc has a "before you can run tests, ask the platform team for..." section).

Topaz has the most complete Key Vault emulation of any service it ships. The control plane covers the full vault lifecycle — create, update, soft-delete, list deleted, recover, purge, name availability, access policy management — and the data plane runs on its own port (8898) reachable at https://<vault-name>.vault.topaz.local.dev:8898/, which is the URL pattern the Azure SDK already constructs.

What works today on the data plane:

Surface	Operations
Secrets	Set, Get (by name and version), List, Update, Delete, Get Versions, Backup, Restore, full soft-delete surface (Get Deleted, List Deleted, Recover, Purge)
Keys	Create, Import (RSA + EC), Get, List, Update, Delete, Backup, Restore, Rotate, Get/Update Rotation Policy, full soft-delete surface, `encrypt`, `decrypt`, `sign`, `verify`, `wrapKey`, `unwrapKey`, `release`, `Get Random Bytes`, `Get Key Attestation`
Certificates	Not yet — full surface scheduled for v1.3-beta

The soft-delete and recovery surface deserves a specific call-out because it is what most teams accidentally depend on. If your application catches KeyVaultErrorException on a soft-deleted secret, recovers it, and continues — that code path is unreachable in any other local emulator. Topaz exercises it end-to-end. Tokens are real signed JWTs. The vault URL works with DefaultAzureCredential exactly as in production.

What is not there yet is the certificates data plane (CRUD, soft-delete surface, issuers, contacts, merge — all scheduled for v1.3-beta). If your application reads certificates from Key Vault — for example, an ASP.NET Core app loading its TLS cert via the Key Vault provider — that path is not yet covered. For secrets and keys, including the cryptographic operations, it is.

Where Azurite stops: Service Bus

Service Bus is the second most common reason teams outgrow Azurite. The community workarounds — RabbitMQ, ActiveMQ, in-memory test doubles — all break the same invariant: the AMQP wire format is not the same, so any code that reaches into broker-specific behaviour (dead-letter queues, peek-lock vs receive-and-delete, deferred messages, session state) drifts away from Service Bus reality.

Topaz runs a real AMQP 1.0 broker on ports 8889 (plain) and 5671 (AMQP/TLS), implemented natively in the host process. The Azure Service Bus SDK connects to it without modification. MassTransit, NServiceBus, and any other library that speaks AMQP work because they are speaking AMQP, not because Topaz pretends to be Service Bus through a thin shim.

Surface	What works
Namespaces (control plane)	Create / Update / Delete / Get / List By Resource Group
Queues (control plane)	Create / Update / Delete / Get / List By Namespace
Topics (control plane)	Create / Update / Delete / Get / List By Namespace
Subscriptions	Create / Update / Delete / Get (via AMQP management node)
Messaging (data plane, AMQP)	Send / Receive on queues and topics, Complete / Abandon / Dead-letter
AMQP entity management	Create / Get / Delete queues, topics, subscriptions — used by MassTransit

Specifically not implemented yet: subscription rule management (filters / actions), namespace authorization rules, list-keys / regenerate-keys, disaster recovery configs, migration configs, private endpoints. For most application development workflows this is a non-issue; if you are building tooling that programmatically rotates Service Bus SAS keys or manages DR pairing, real Azure is still the right target for those tests.

Where Azurite stops: Container Registry

docker push myregistry.azurecr.io/myapp:latest against a local emulator is the kind of thing that historically required either a real ACR (and therefore Azure credentials in CI) or running a generic Docker registry and pretending it is ACR (works for pull, breaks the moment anything touches the ARM control plane or the ACR-specific OAuth2 exchange).

Topaz emulates both layers:

Control plane (ARM): create, update, delete, list registries; manage admin credentials; toggle adminUserEnabled.
Data plane (OCI Distribution Spec, port 8892): manifest CRUD, blob upload (POST / PATCH / PUT chunked uploads), blob download and existence checks, repository catalog, tag listing, and the ACR-specific /oauth2/exchange endpoint that makes az acr login work.

The full authentication flow — GET /v2/ returning a Www-Authenticate challenge, POST /oauth2/exchange swapping the Entra token for an ACR refresh token, and the bearer token round-trip — is implemented end-to-end. There is a separate post on how this works that goes into the design tradeoffs.

What this unlocks: you can run docker push, helm push, and any OCI-compliant client against Topaz without modifying anything in your build pipeline. The registry hostname follows real Azure conventions (myregistry.cr.topaz.local.dev:8892). Image promotion workflows, CI builds that publish images, and Terraform configurations that create ACRs all work locally. docker pull and end-to-end image pull-through is on the roadmap.

Where Azurite stops: Entra ID and Managed Identity

This is the silent one. Azurite does not need an identity layer because it accepts SharedKey for authentication and falls back to "no auth" when nothing is provided. The moment your application uses DefaultAzureCredential — which is the Microsoft-recommended pattern for everything except Storage — Azurite cannot help. Your local code either talks to a real Entra tenant (forcing every developer onto a corporate identity) or you replace DefaultAzureCredential with a custom test double (creating a code-path divergence with production).

Topaz ships a working Entra ID emulation layer. A local tenant (topaz.local.dev, tenant ID 50717675-3E5E-4A1E-8CB5-C62D8BE8CA48) is provisioned at startup with a built-in superadmin account. Every token is a real, signed JWT — same format Azure issues, same claims layout, signed with HMAC-SHA256, one-hour lifetime. The OIDC discovery endpoint (/.well-known/openid-configuration) is fully functional, both /organizations/ and /{tenantId}/ variants are served, and four grant types are supported: client_credentials, password, authorization_code, and refresh_token.

Tied to that:

Microsoft Graph API for users, applications, service principals, and groups — enough to script a full identity setup.
Managed Identity — user-assigned and system-assigned, including federated identity credentials. The same DefaultAzureCredential chain that works in production works locally, no special credential type required.
RBAC — role assignments and role definitions at any ARM scope, with the standard built-in roles (Owner, Contributor, Reader, plus the service-specific data-plane roles like Storage Blob Data Contributor) pre-loaded.

This is the difference between "I can read a blob locally" and "I can run my entire authn/authz path locally". The latter catches a class of bug — token audience mismatches, role assignment drift, scope errors — that mocks never catch.

There is a dedicated post on the Entra emulation layer if you want the design details.

Where Azurite stops: ARM, Terraform, and the Azure CLI

Azurite has no ARM control plane, which means there is no az group create, no azurerm_resource_group, no Bicep deployment, no way to express the resources your application needs in the same infrastructure-as-code language you use in production. Local development and production end up using two different definitions of "what infrastructure exists".

Topaz has a working ARM emulation. The Resource Manager port (8899) accepts az and azurerm provider traffic, exposes a metadata document at the discovery endpoint that points every Azure API URL at the local emulator, and accepts ARM template / Bicep deployments end-to-end. The same terraform apply that creates resources in Azure can create them in Topaz with one provider setting:

provider "azurerm" {
  features {}
  metadata_host                   = "topaz.local.dev:8899"
  resource_provider_registrations = "none"
}

That is the entire integration. metadata_host redirects endpoint discovery; the AzureRM provider then constructs every subsequent URL — Storage, Key Vault, Service Bus, Container Registry — from Topaz's metadata document. Detail in the Terraform integration post.

The Azure CLI works the same way: register Topaz as a cloud environment with az cloud register, switch to it, and az login issues a token from the local Entra layer. From there, az keyvault secret set, az servicebus queue create, az acr login, az group deployment create all work as they do against Azure. Azurite supports az storage and only az storage.

What Azurite still does better

A genuinely honest post needs this section. Azurite is not strictly worse — it is narrower, more mature in its narrow scope, and the right call for some workloads.

Microsoft maintenance. Azurite is shipped and supported by Microsoft. Topaz is open source and maintained by an independent team. If your organisation requires vendor-supported software for local development tooling, that distinction matters.
RA-GRS secondary endpoints. Azurite emulates the read-access geo-redundant secondary URL pattern today. Topaz has partial support; full secondary endpoint semantics — secondary DNS hostnames, GeoReplicationStats payloads, and read-only enforcement — are scheduled for v1.4-beta.
The UseDevelopmentStorage=true shortcut. Azurite is hardcoded to the Azure SDK's UseDevelopmentStorage=true connection string. Topaz uses real connection strings — the same format Azure issues — which is more flexible but loses the one-line shortcut.
Visual Studio integration. Azurite ships with Visual Studio and Storage Explorer has a built-in "Local Emulator" entry. Topaz works with Storage Explorer, but you connect via a connection string rather than the dedicated emulator option.
Maturity. Azurite has been in production CI pipelines for years. Topaz is in beta as of v1.0 and is moving fast — that is also the reason this post needs to list which surfaces are not yet covered.

If your application uses only Storage, a single account is enough, and you do not need ARM or Terraform integration locally, Azurite is genuinely the simpler choice and there is no reason to switch.

What is coming for Storage in Topaz

The Storage roadmap is publicly tracked in BACKLOG.md and mirrored on the website roadmap. The near-term Storage work in flight:

v1.4-beta — SAS token validation on the data plane.
The control plane already generates SAS tokens via ListAccountSas and ListServiceSas. The data-plane security providers (Blob, Queue, Table) currently only recognise the Authorization: header — incoming requests with ?sv=...&sig=... query strings hit the missing-header path and return 401. The work in progress adds:

Account SAS validation: detects sv, ss, srt, sp, se, st, spr, sip, sig, builds the canonical Account SAS string-to-sign, HMAC-SHA256 against the account key, validates expiry / service / resource type / HTTP method.
Service SAS validation: per-service string-to-sign, including the canonicalized resource and the response-header overrides (rscc, rscd, etc.), with stored access policy lookup when si=<policyId> is present.
Stored access policy enforcement: the Container ACL / Queue ACL / Table ACL endpoints already round-trip <SignedIdentifiers> XML to disk; v1.4 wires them into the SAS validation path so revoking a named policy actually revokes the tokens that reference it.

v1.4-beta — anonymous / public-access reads for Blob containers.
Real Azure allows containers created with x-ms-blob-public-access: container (list + read) or blob (read only) to permit unauthenticated reads. Topaz currently rejects every request without an Authorization header. The fix is to store the public-access level on the container, look it up in the security provider when no auth is present, and permit the request when the level allows the operation.

v1.4-beta — RA-GRS secondary endpoint semantics.
Secondary DNS hostnames ({accountName}-secondary.*), secondaryEndpoints.blob/.table/.queue/.file in the storage account ARM response, the ?restype=service&comp=stats endpoint returning a realistic GeoReplicationStats payload, and read-only enforcement that returns 403 WriteOperationNotSupportedOnSecondary on mutating requests against a secondary endpoint.

v1.5-beta — User Delegation SAS for Blob.
The Entra-derived SAS variant. Two coordinated pieces: an ARM endpoint that mints a user delegation key bounded by (start, expiry, oid, tid), and Blob data-plane validation that recomputes the key from the stored fields and validates the signed token. This is the only SAS variant that needs the local Entra layer to be coherent with the storage layer — which is part of why Topaz is built as one process rather than five.

v1.6-beta — unified data-plane port.
Real Azure exposes Blob / Table / Queue / File on port 443 with subdomain-based routing ({account}.blob.core.windows.net, etc.). Topaz currently uses separate ports per sub-service (8891 / 8890 / 8893 / 8894) for routing simplicity. Some Azure CLI / SDK code paths construct storage URLs via get_account_url(), which builds a single https:// URL from the cloud-suffix storage_endpoint — encoding only one port. Consolidating onto a single port behind subdomain routing fixes that and makes the local URL pattern identical to production.

These are all in the public backlog with milestone labels — there is no roadmap kept in someone's head.

Developer experience: where the differences compound

The feature comparison above is the headline. The day-to-day experience matters more.

Single binary, single process, single working directory

Azurite is one process for Storage. Replicating Topaz's surface with separate emulators means orchestrating Azurite + a Service Bus emulator + a registry + a mock identity server, all writing to different directories, all logging in different formats, all started from different scripts. Every team that goes down this path eventually writes a Docker Compose file that nobody is happy with.

Topaz is one binary or one Docker image. State lives in .topaz/ next to your project. Stop the host, the state is preserved. Start it again, the state comes back. Delete the directory, the slate is clean. There is one log stream and one health check (GET /health).

MCP server for AI tooling

Topaz ships an MCP server (Topaz.MCP) that exposes the host as a set of tools any MCP-compatible client can call. The intended use case is GitHub Copilot in VS Code — drop a .vscode/mcp.json in your workspace, point it at the MCP binary, and Copilot can run RunTopazAsContainer, CreateSubscription, CreateResourceGroup, CreateKeyVault, CreateStorageAccount, CreateServiceBusNamespace, CreateContainerRegistry, and a GetConnectionStrings tool that returns ready-to-paste connection strings for everything it just provisioned.

In practice this means you can describe a local environment in natural language:

Start Topaz. Create a subscription, a resource group rg-dev in westeurope, then provision a Storage account, a Service Bus namespace with a queue named orders, and a Key Vault with a secret db-password. Output the connection strings.

…and the assistant runs the whole sequence. There is also a GetTopazStatus diagnostics tool for the case where a setup fails partway through and you want to know which ports are bound. Full details in the MCP server documentation.

Azurite has no equivalent. There is no MCP integration, no programmatic way for an AI assistant to provision a Storage account, no notion of provisioning at all — the tool exists at the data plane and stops there.

Editor and IDE integration

GitHub Copilot in VS Code works through the MCP server. The Topaz CLI (topaz) is a thin client over the host process — every command is topaz <verb> with consistent JSON output, which means it scripts cleanly from a terminal, a shell hook, a Makefile, or a CI step. A native VS Code extension is on the roadmap; for the moment, the MCP path covers the AI-assisted workflows and the CLI covers everything else.

Scaling out: CI and Docker

Topaz publishes a Docker image (topaz/host) that runs as a sidecar in any CI environment. The pattern in GitHub Actions is:

services:
  topaz:
    image: topaz/host:latest
    ports:
      - 8899:8899
      - 8898:8898
      - 8891:8891
      - 8889:8889
      - 8892:8892
    options: --health-cmd="curl -f http://localhost:8899/health || exit 1"

Then terraform apply against metadata_host = "topaz:8899" from any step. No Azure credentials in the pipeline secrets, no rate-limited subscription, no per-run cost. The same image runs as a Testcontainer inside .NET integration tests via the Topaz Testcontainers helper. The CI/CD integration guide covers GitHub Actions and Azure DevOps.

Azurite has a Docker image too. The difference is what the image emulates — Topaz's image covers the same surface this post described (Storage + Key Vault + Service Bus + Event Hubs + ACR + Entra + ARM); Azurite's covers Storage. If your CI only needs Storage, that is fine. If it needs more, Azurite forces you back to the multi-emulator orchestration problem.

ASP.NET Core integration

AddTopaz() is an extension method on IServiceCollection that provisions local Azure infrastructure at application startup — declaratively, in the same Program.cs where you wire up DI. Spin up a resource group, a Service Bus namespace, a Key Vault with seed secrets, a Storage account, all in code, all conditional on environment so the same Program.cs runs unchanged in production. Detail in the ASP.NET Core integration guide.

When to keep Azurite

Your application uses only Azure Storage and a single account is sufficient.
You need RA-GRS secondary endpoint emulation today.
You need a Microsoft-supported emulator with vendor backing.
Your toolchain is built around Azurite and migration is not worth the effort.

When to switch to Topaz

Your application uses any service beyond Storage — Key Vault, Service Bus, Event Hubs, Container Registry, Managed Identity, RBAC.
You need multiple named storage accounts in local or CI environments without manual hosts file edits.
You want a single process to replace multiple emulators and the Docker Compose file that holds them together.
You use Terraform with the azurerm provider and want a local target for terraform apply.
You want the full Azure CLI (az keyvault, az servicebus, az acr, az deployment) to work locally, not just az storage.
You want ARM-level resource management (resource groups, subscriptions, ARM templates, Bicep) in CI without a real subscription.
You want DefaultAzureCredential to work end-to-end locally without code-path divergence from production.
You want AI-assisted environment provisioning through MCP.

Migrating

For Storage specifically, Topaz implements the same data-plane APIs Azurite does. Point your existing Azure SDK clients at Topaz's endpoints and they connect without code changes — only the endpoint hostname, port, and credentials change. The one item to check during migration is authentication: Topaz always enforces SharedKey signatures on Table and Queue requests, so any request that Azurite silently accepted with a missing or invalid signature will be rejected. This is intentional — it is the same behaviour real Azure has, and catching the divergence locally is the whole point.

Beyond Storage, the API coverage docs list which operations are implemented per service. If you hit something that is not yet supported, open an issue — the backlog is publicly tracked and feedback shapes priorities.

Summary

Azurite is a good Storage emulator. Topaz is a Storage emulator that also covers Key Vault, Service Bus, Event Hubs, Container Registry, Managed Identity, RBAC, ARM, and Entra ID — in one binary, with one log stream, one working directory, and one Docker image. The Storage parity is essentially complete; the gaps that remain (SAS validation, RA-GRS, public-access reads) are scheduled and tracked in the open backlog.

If your application is Storage-only, stay on Azurite. If it is anything else — and most real Azure applications are — Topaz exists to remove the orchestration tax of running five different local emulators that were never designed to work together.

Two days chasing a SharedKey signature mismatch: fixing azurerm_storage_table_entity in Topaz

Kamil Mrzygłód — Thu, 30 Apr 2026 10:42:05 +0000

Some bugs announce themselves loudly. A null pointer in a hot path, a missing route that returns 404 to every request — the kind of thing that fails immediately and points straight at the cause. Others are quieter. They let most of the stack work correctly and only reveal themselves at the intersection of two independently correct but mutually incompatible assumptions. The azurerm_storage_table_entity failure was the second kind.

This post is an account of a two-day investigation into a persistent 401 Unauthorized response on Terraform table entity operations, the four separate bugs we found along the way, and how pairing with GitHub Copilot shaped the investigation. The fix touched authentication, HTTP routing, upsert semantics, and stream lifecycle — each uncovered only after the previous one was resolved.

You can read more about Topaz here: https://topaz.thecloudtheory.com/

The starting point

Topaz already had table storage support: accounts, tables, entity insert and query. The azurerm_storage_table Terraform resource worked. What did not work was azurerm_storage_table_entity. Any Terraform run that tried to create a table entity failed immediately:

Error: creating Entity (Partition Key "pk1" / Row Key "rk1" / ...):
  executing request: unexpected status 401 (401 Unauthorized) with EOF

The 401 was puzzling because the same storage account, created by the same Terraform run, authenticated correctly for every ARM-level operation. Listing keys worked. Creating the table worked. Only the data-plane entity operation failed.

The investigation begins: is the key the problem?

The first hypothesis was key mismatch. Terraform calls listKeys to get a storage account key and then uses it to sign data-plane requests with HMAC-SHA256. If Topaz returned a different key than it used to verify the signature, every request would fail.

To confirm or rule this out, we added diagnostic logging across the authentication path. The TableStorageSecurityProvider was modified to emit the full stored keys (as base64 and as raw bytes in hex), the received signature, the computed signature for both keys, the full Authorization header, every request header with its raw bytes, and the exact string-to-sign in hex. ListStorageAccountKeysEndpoint was modified to log a prefix of both keys on every call, so we could correlate what Terraform received with what Topaz verified against.

The first finding was that the keys were stable. Terraform's listKeys calls (there were four of them per apply — the provider is aggressive about refreshing credentials) consistently received the same key1 prefix. The key logged at verification time matched. The key mismatch hypothesis was eliminated.

Ruling out Topaz restarts

One suspicious pattern appeared in the logs: at a certain point during a test run, ARM requests started returning no route to host on port 8899. This looked like the Topaz container crashing and restarting mid-test — which would regenerate the storage keys and make Terraform's cached key stale.

We checked the Docker container lifecycle carefully. The logs showed two AMQP listener errors at startup (a known benign issue with port reuse in the Docker test environment) but no crash or restart. The no route to host was a red herring — it appeared after Terraform had already failed and was in its cleanup phase, not before. Cross-referencing the listKeys timestamps with the entity request timestamps confirmed: the key prefix was identical across all four calls in the same test run. No restart, no key regeneration.

Narrowing the scope: the isolated test

At this point we were spending time waiting for the full storage batch test to complete — roughly thirty minutes per run because Terraform retries failing requests with exponential backoff before giving up. To eliminate noise from other resources in the same test, we created a minimal isolated scenario: a single resource group, a single storage account, a single table, and a single entity. Nothing else.

resource "azurerm_storage_table_entity" "entity" {
  storage_table_id = azurerm_storage_table.tbl.id
  partition_key    = "pk1"
  row_key          = "rk1"
  entity = {
    name = "isolated-entity"
  }
}

This reduced the iteration cycle and made the logs dramatically easier to read.

The actual bug: URL encoding

With a clean log, the problem became visible. Here is what Topaz logged as the string-to-sign it computed:

GET\n\n\nThu, 30 Apr 2026 06:43:27 GMT\n/tfisoentityacct/isoentities(PartitionKey='pk1',%20RowKey='rk1')

And the TF debug log showed the URL Terraform actually sent the request to:

GET https://tfisoentityacct.table.storage.topaz.local.dev:8890/isoentities%28PartitionKey=%27pk1%27,%20RowKey=%27rk1%27%29

The entity key lookup path contains parentheses and single quotes. Terraform's go-azure-sdk encodes those: ( → %28, ' → %27. It signs the request using the raw encoded URL. ASP.NET Core receives the request and makes the decoded path available through HttpRequest.Path. When Topaz called ToString() on that path, it got the decoded form — (PartitionKey='pk1', — which is not what was signed. The HMAC computation was based on a different string than what the client used.

The fix was to read the raw request target before ASP.NET Core's decoding step using IHttpRequestFeature.RawTarget, which preserves the wire-format path exactly as the client sent it:

var rawTarget = context.Features.Get<IHttpRequestFeature>()?.RawTarget
                ?? context.Request.Path.Value
                ?? string.Empty;
var queryIndex = rawTarget.IndexOf('?');
var rawPath = queryIndex >= 0 ? rawTarget[..queryIndex] : rawTarget;

This was a one-line conceptual fix that required updating the IsRequestAuthorized call in all fourteen table endpoint classes. The base class TableDataPlaneEndpointBase was refactored to accept an HttpContext and extract the raw path internally, so all call sites became a single-argument call.

Bug two: the MERGE verb

With auth fixed, the next run reached Terraform's actual entity creation step — and hit a new error:

Request MERGE /isoentities(PartitionKey='pk1',%20RowKey='rk1') has no corresponding endpoint assigned.

The Azure Table Storage REST API uses HTTP MERGE for Insert-or-Merge operations. Topaz's InsertOrMergeTableEntityEndpoint only declared POST in its Endpoints array:

public string[] Endpoints => [@"POST /^.*?\(PartitionKey='.*?',(%20|\s)?RowKey='.*?'\)$"];

Terraform's go-azure-sdk uses MERGE as the verb. The fix was straightforward — add MERGE to the array:

public string[] Endpoints =>
[
    @"POST /^.*?\(PartitionKey='.*?',(%20|\s)?RowKey='.*?'\)$",
    @"MERGE /^.*?\(PartitionKey='.*?',(%20|\s)?RowKey='.*?'\)$",
];

Bug three: Insert-or-Merge semantics

With routing fixed, the next run returned 404 Not Found. The HandleUpdateEntityRequest path calls DataPlane.UpdateEntity, which throws EntityNotFoundException when the entity file does not exist on disk:

if(File.Exists(entityPath) == false)
{
    throw new EntityNotFoundException();
}

For a PUT (Replace) or PATCH (Merge), throwing here is correct — updating a non-existent entity should fail. But MERGE in Table Storage semantics means Insert-or-Merge: create the entity if it does not exist, merge the properties if it does. The existing code had no path for that case.

We added an UpsertEntity method to TableServiceDataPlane that writes a new entity unconditionally, and a upsert parameter to HandleUpdateEntityRequest that catches EntityNotFoundException and falls through to the upsert path:

catch (EntityNotFoundException) when (upsert)
{
    buffered!.Position = 0;
    DataPlane.UpsertEntity(buffered, subscriptionIdentifier, resourceGroupIdentifier,
        tableName, storageAccountName, partitionKey, rowKey);
    response.StatusCode = HttpStatusCode.NoContent;
}

Bug four: the disposed stream

That buffered!.Position = 0 line is significant. The first implementation of the upsert fallback passed the original input stream directly to UpsertEntity. The next test run returned 500 Internal Server Error with Cannot access a disposed object in the Topaz logs.

UpdateEntity reads the body with a StreamReader before throwing EntityNotFoundException. StreamReader disposes its underlying stream by default when it is disposed — which happens when the using var sr block exits after the throw. By the time the catch (EntityNotFoundException) when (upsert) block ran, the stream was already closed.

The fix was to buffer the request body into a MemoryStream before entering the UpdateEntity call when upsert is true, and use leaveOpen: true in UpdateEntity's StreamReader to avoid closing the memory stream on an unexpected throw:

MemoryStream? buffered = null;
if (upsert)
{
    buffered = new MemoryStream();
    input.CopyTo(buffered);
    buffered.Position = 0;
    input = buffered;
}

The test assertion

After all four bugs were fixed, Terraform reported Apply complete! Resources: 4 added, 0 changed, 0 destroyed. The test itself still failed:

System.InvalidOperationException : The node must be of type 'JsonValue'.

The terraform output -json response wraps each output in an envelope: { "partition_key": { "value": "pk1", "type": "string" } }. The initial test assertion called .GetValue<string>() directly on outputs["partition_key"], which is a JsonObject, not a JsonValue. The fix was a one-character change — navigate to ["value"] first, matching every other test in the suite.

The role of Copilot in the investigation

Copilot assisted throughout both days: adding the diagnostic logging, generating the isolated Terraform scenario and test class, and suggesting fixes as each root cause was identified. The investigation benefited from being able to express hypotheses in natural language — "could the key bytes differ even if the base64 strings match?" led immediately to logging the raw bytes in hex — and from having a second reader of the logs who could cross-reference the wire trace in the TF debug log against the canonicalization logic in the Topaz source.

The most valuable contribution was narrowing the investigation down to the URL encoding issue. The logs included a DecodedPathSTS variant that computed the signature using Uri.UnescapeDataString(absolutePath). When that also failed to match, it confirmed the issue was not simply percent-encoded spaces — it was the full encoding of the path characters. Tracing IHttpRequestFeature.RawTarget was the correct escape hatch once the root cause was understood.

What changed

Area	Change
`TableStorageSecurityProvider`	Signs against `IHttpRequestFeature.RawTarget` (raw wire path) instead of `HttpRequest.Path` (decoded)
`InsertOrMergeTableEntityEndpoint`	Added `MERGE` verb alongside `POST`
`TableServiceDataPlane`	Added `UpsertEntity`; `UpdateEntity` uses `leaveOpen: true`
`TableDataPlaneEndpointBase`	`HandleUpdateEntityRequest` accepts `upsert` flag; buffers body into `MemoryStream` for fallback
All 14 table endpoints	Updated to pass `HttpContext` to `IsRequestAuthorized`

The storage API coverage page now marks azurerm_storage_table_entity create, read, and delete as implemented. The full Terraform scenario — resource group, storage account, table, entity — runs end-to-end in roughly two minutes with no manual steps.

Takeaway

Each of the four bugs was independently plausible and independently fixable. But they were invisible until the previous one was resolved. You cannot see that MERGE is unrouted if you never get past the 401. You cannot see the upsert semantics gap if MERGE returns 404 before reaching the data plane. You cannot see the disposed stream if the upsert path is never exercised. The debugging process was necessarily sequential — each fix revealed the next layer.

The URL encoding issue is worth calling out specifically. The Azure Table Storage SharedKey algorithm requires signing the canonicalized resource, which is derived from the request URL. Whether that URL is in its raw percent-encoded form or its decoded form is not a detail the specification makes obvious. ASP.NET Core's routing infrastructure silently decodes the path before most code ever sees it. IHttpRequestFeature.RawTarget is the correct place to read the unmodified wire path, and it is not the first thing you reach for. Getting there required enough diagnostic signal to rule out every other explanation first.

Running Terraform against Azure locally, without a subscription

Kamil Mrzygłód — Tue, 14 Apr 2026 10:16:23 +0000

This posts explains how Terraform is integrated with Topaz - Azure emulator for local development and testing. You can check it here.

Every Terraform workflow that targets Azure needs the same things before it can do anything useful: an Azure subscription, a service principal or user account with the right permissions, and a network path to the Azure APIs. In a team setting you also need to make sure those credentials are available wherever terraform apply runs — local machines, CI agents, staging pipelines. The feedback loop is slow, and the blast radius for a misconfigured apply is real.

Topaz removes all of that. The same terraform apply that would create resources in Azure can instead create them in a local emulator, with no subscription, no credentials to rotate, and no cloud charges. This post explains how the integration works and how to set it up.

Why the standard AzureRM provider works at all

The key insight is that the AzureRM provider does not have Azure's API endpoints hardcoded. When it initialises, it fetches a metadata document from a discovery endpoint that describes where each Azure API lives. In a normal setup, that discovery endpoint is the Azure Resource Manager metadata endpoint at management.azure.com. Once the provider has that document, it constructs every subsequent request URL from it.

The metadata_host setting exists precisely to point that discovery step somewhere else. Set it to Topaz's ARM port, and the provider fetches Topaz's metadata document instead. That document points every API URL — authentication, resource management, Key Vault, Storage — at the local emulator. The provider never knows it is not talking to Azure.

provider "azurerm" {
  features {}
  metadata_host                    = "topaz.local.dev:8899"
  resource_provider_registrations  = "none"
}

Two settings are doing the work here. metadata_host redirects endpoint discovery to Topaz. resource_provider_registrations = "none" tells the provider not to attempt registering resource providers on startup — Topaz does not emulate the full registration flow, and it is not needed for local development anyway.

DNS setup

The hostname topaz.local.dev needs to resolve to 127.0.0.1 on your machine. Topaz ships install scripts that configure this using dnsmasq, which handles the wildcard subdomains that services like Container Registry depend on. The getting started guide covers installation and the one-time DNS and certificate setup.

For containerised environments, the same configuration can be handled at the container network level — the Terraform integration guide covers that path as well.

Authentication

AzureRM needs a credential to authenticate against whatever it thinks Azure is. With Topaz, that means authenticating against Topaz's local Entra ID emulation layer, which responds to the same OAuth2 endpoints the real Azure uses.

The simplest approach for local development is to use the Azure CLI configured for Topaz's local cloud. Once configured, az login issues a token from the local Entra layer, and the AzureRM provider picks it up through DefaultAzureCredential exactly as it would in production. No service principal, no environment variables, no secrets.

A fixed subscription ID avoids drift between runs:

topaz start \
  --default-subscription 00000000-0000-0000-0000-000000000001 \
  --log-level Information

Pair that with:

export ARM_SUBSCRIPTION_ID=00000000-0000-0000-0000-000000000001

And every terraform plan and apply targets the same subscription, the same resource IDs, and the same state — which matters when you want your CI runs to behave identically to your laptop.

A complete example

With Topaz running, this is all it takes to create a resource group and a Key Vault locally:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "= 4.67.0"
    }
  }
}

provider "azurerm" {
  features {}
  metadata_host                   = "topaz.local.dev:8899"
  resource_provider_registrations = "none"
}

resource "azurerm_resource_group" "example" {
  name     = "rg-local"
  location = "westeurope"
}

resource "azurerm_key_vault" "example" {
  name                = "kv-local"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  tenant_id           = "50717675-3e5e-4a1e-8cb5-c62d8be8ca48"
  sku_name            = "standard"
}

terraform init
terraform apply -auto-approve
terraform destroy -auto-approve

The tenant ID above is Topaz's built-in local tenant — the same one az login uses when configured for the local cloud.

AzAPI provider

If your Terraform configuration uses azapi resources alongside azurerm, the setup is straightforward. Keep the azurerm provider configured for Topaz metadata and add the azapi provider declaration:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "= 4.67.0"
    }
    azapi = {
      source = "Azure/azapi"
    }
  }
}

provider "azurerm" {
  features {}
  metadata_host                   = "topaz.local.dev:8899"
  resource_provider_registrations = "none"
}

provider "azapi" {}

Because azapi inherits its endpoint configuration from the same environment, and Topaz's metadata document covers the resource management endpoints that azapi targets, no additional configuration is needed.

Using it in CI

The setup above works identically in a CI pipeline. Run Topaz as a service container or a background step, set ARM_SUBSCRIPTION_ID, and run terraform apply as normal. No Azure credentials in the pipeline, no cost per run, no rate limiting from the Azure APIs. The CI/CD integration guide has ready-to-use examples for GitHub Actions and Azure DevOps.

What works today

Topaz currently supports Terraform workflows for Azure Storage, Key Vault, Service Bus, Event Hubs, Container Registry, and Resource Manager operations including resource groups and ARM template deployments. The API coverage docs list which operations are implemented per service.

Not every AzureRM resource type is emulated yet. If you hit a resource that Topaz does not support, the provider will return a 404 or an unsupported operation error. Check the API coverage page for current status, and open an issue if something you need is missing.