DEV Community: Martin Oehlert

Infrastructure as Code: Terraform for Your New Stack

Martin Oehlert — Fri, 25 Sep 2026 06:46:19 +0000

The first terraform apply against a fresh subscription created the Container Apps environment, created the app, and then the revision failed to pull its image. The second apply, with nothing changed, succeeded. The AcrPull role assignment existed both times; what was missing on the first run was an edge in Terraform's dependency graph, because the container app references the identity and the registry login server and has no reason to reference the grant. Terraform orders what you reference, Azure requires things you would never reference, and that gap is where most of the production failures in a Container Apps stack live.

Part 6 described that architecture as a chain of hops and asked what each one trusts. Written down as Terraform it is fifteen resources in the root module before a single module block runs, and four modules under it holding the VNet, the private endpoints, the environment, the apps, the Dapr components, the identities and their grants.

Every HCL block below is lifted out of a tree that terraform validates on Terraform 1.16.3 and hashicorp/azurerm 5.6.0, published 2026-09-17, and the whole tree is in the companion repo. The pair is pinned in one place:

terraform {
  required_version = ">= 1.16.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "5.6.0"
    }
  }

  # Partial configuration. The rest comes from envs/<env>.backend.hcl at init time:
  #   terraform init -reconfigure -backend-config=envs/prod.backend.hcl
  backend "azurerm" {}
}

provider "azurerm" {
  features {}

  subscription_id = var.subscription_id

  # v5 defaults this to "none". On a fresh subscription the first apply then fails on an
  # unregistered Microsoft.App, and the error reads like a permissions problem.
  resource_provider_registrations = "legacy"
}

One thing about that validation matters before you trust your own: terraform init without -upgrade happily resolved an azurerm 4.x build out of the local plugin cache with a 5.x constraint sitting in required_providers, and the three v4-shaped errors that followed read as though the configuration was wrong. terraform version -json prints provider_selections; check it before you start editing code that was fine.

Four modules, drawn where the lifetime changes

A module boundary is a lifetime boundary, not a service boundary. Here is the root module wiring the four of them together, with the arguments that only carry data trimmed out:

module "networking" {
  source = "./modules/networking"

  address_space = var.address_space

  # 7 newbits on a /16 gives a /23. /27 is the legal minimum for workload profiles, but the
  # subnet cannot be resized afterwards and a /27 caps at 9 Dedicated nodes.
  aca_subnet_newbits = 7

  tags = local.required_tags
}

module "container_apps" {
  source = "./modules/container-apps"

  log_analytics_workspace_id = azurerm_log_analytics_workspace.this.id
  infrastructure_subnet_id   = module.networking.container_apps_subnet_id
  internal_only              = true

  container_registry_login_server = azurerm_container_registry.this.login_server

  app_identities = {
    (local.orders_app_id) = {
      id        = module.identity.identity_ids[local.orders_app_id]
      client_id = module.identity.client_ids[local.orders_app_id]
    }
  }

  tags = local.required_tags

  depends_on = [module.identity, module.networking]
}

module "dapr_components" {
  source = "./modules/dapr-components"

  container_app_environment_id = module.container_apps.environment_id
  dapr_identity_client_id      = module.identity.client_ids[local.orders_app_id]

  state_store_scopes = [module.container_apps.dapr_app_ids[local.orders_app_id]]
  pubsub_scopes      = [module.container_apps.dapr_app_ids[local.orders_app_id]]
}

The VNet outlives the architecture: you will still be running that address space after the apps inside it have been rewritten twice. The environment is near-permanent for reasons the next section is entirely about. The apps change every time somebody merges. Cut along those rates and each module has one reason to be applied. Cut by Azure service instead and you get a cosmos module that has to be planned every time an image tag moves.

What crosses each boundary is an output feeding a var, and that reference is the only thing Terraform needs to order the two. module.networking exports the delegated subnet ID; module.container_apps exports the environment ID, its default domain, its static IP, the app FQDNs and the Dapr app ids; module.identity exports identity IDs, client IDs and principal IDs as three separate maps, because they are not interchangeable and a swap fails at runtime with nothing visible at plan time. module.dapr_components exports nothing anyone consumes, which is a fine thing for a leaf module to do.

The two dashed edges are the ones Terraform cannot draw, and the depends_on on module.container_apps is where you write them down by hand. module.identity is there for the failed image pull in the first paragraph of this article. module.networking is there because the environment references the subnet and never the NSG association, so without the edge the environment can come up against an unprotected subnet. The sample puts both on the module call rather than on the resource, since the identities and the apps live in different modules; inside a single module the same fix is depends_on = [azurerm_role_assignment.acr_pull] on the azurerm_container_app resource itself. The identity chain gets taken apart properly further down.

Three patterns would fail a module review at a stricter shop, and they are worth naming because most Terraform repos contain all three. A module wrapping a single resource with a rename is a rename. A resource you configure once and never vary does not need a variable, let alone a module. And a passthrough module, where every variable maps one-to-one onto a provider argument, has added a file and taken away the provider documentation. dapr-components survives that test by a narrow margin: it hard-codes the component types and versions, it refuses an empty scopes list, and it hides the azureClientId plumbing. A generic "any Dapr component" module would do none of those things and would be worse than writing the resource directly.

The environment: every interesting argument is a one-way door

The resource that decides the entire network posture of this stack has no required network argument at all. Omit infrastructure_subnet_id and you get a public environment outside your VNet, and it plans clean:

locals {
  missing_identities = setsubtract(keys(var.apps), keys(var.app_identities))
}

resource "azurerm_container_app_environment" "this" {
  name                = "cae-${var.workload}-${var.environment}"
  location            = var.location
  resource_group_name = var.resource_group_name

  logs_destination           = "log-analytics"
  log_analytics_workspace_id = var.log_analytics_workspace_id

  infrastructure_subnet_id           = var.infrastructure_subnet_id
  infrastructure_resource_group_name = "rg-${var.workload}-${var.environment}-aca-infra"
  internal_load_balancer_enabled     = var.internal_only
  zone_redundancy_enabled            = var.environment == "prod"

  dapr_application_insights_connection_string = var.dapr_application_insights_connection_string

  workload_profile {
    name                  = "Consumption"
    workload_profile_type = "Consumption"
  }

  dynamic "workload_profile" {
    for_each = var.dedicated_workload_profiles

    content {
      name                  = workload_profile.key
      workload_profile_type = workload_profile.value.profile_type
      minimum_count         = workload_profile.value.minimum_count
      maximum_count         = workload_profile.value.maximum_count
    }
  }

  tags = var.tags

  lifecycle {
    precondition {
      condition     = length(local.missing_identities) == 0
      error_message = "Every app needs an identity: ${join(", ", local.missing_identities)} are in apps but not in app_identities."
    }
  }
}

Nearly every argument above forces replacement when it changes: infrastructure_subnet_id, infrastructure_resource_group_name, internal_load_balancer_enabled, zone_redundancy_enabled, dapr_application_insights_connection_string, the workload_profile blocks, and the size of the subnet the first one points at. Replacing a Container Apps environment takes every container app and every Dapr component inside it along, so those are decisions you make once, on day one, with the same weight you would give a database engine choice.

workload_profile is the trap, because nothing about it looks final. It is an optional repeatable block with optional arguments, the shape Terraform uses everywhere for things you tune later. The provider note is explicit: "Environments created without an initial Workload Profile cannot have them added at a later time and must be recreated. Similarly, an environment created with Profiles must always have at least one defined Profile, removing all profiles will force a recreation of the resource." That is why the static Consumption block sits above the dynamic one rather than inside it. A Consumption profile must be named Consumption, there can only be one, and keeping it out of the map means an operator who empties dedicated_workload_profiles removes a dedicated profile instead of destroying the environment.

logs_destination = "log-analytics" is there because v5 stopped computing it. A v4 configuration that set only log_analytics_workspace_id and got log analytics by default now drops to Streaming Only on upgrade, with no error, and you find out when you go looking for a query result that is not there.

The dedicated_workload_profiles variable is where the environment's rules get written down, and the third validation on it is a different kind of rule from the first two:

variable "dedicated_workload_profiles" {
  type = map(object({
    profile_type  = string
    minimum_count = optional(number, 0)
    maximum_count = optional(number, 3)
  }))
  default     = {}
  nullable    = false
  description = "Dedicated profiles keyed by profile name. A Consumption profile is always added, because removing the last profile recreates the environment."

  validation {
    condition = alltrue([
      for name, profile in var.dedicated_workload_profiles :
      can(regex("^(D4|D8|D16|D32|E4|E8|E16|E32)$", profile.profile_type))
    ])
    error_message = "profile_type must be one of D4, D8, D16, D32, E4, E8, E16, E32."
  }

  validation {
    condition = alltrue([
      for name, profile in var.dedicated_workload_profiles :
      profile.maximum_count >= profile.minimum_count
    ])
    error_message = "maximum_count must be greater than or equal to minimum_count."
  }

  validation {
    condition     = !contains(keys(var.dedicated_workload_profiles), "Consumption")
    error_message = "The Consumption profile is added by the module; do not declare it here."
  }
}

The first two encode Azure rules, and Azure would have rejected the apply anyway; all the validation buys you is a readable error twenty minutes earlier. The third encodes a rule Azure has never heard of. It exists because this module chose to add the Consumption profile itself, and a caller who declares one would get a duplicate name from a design decision they had no way to see. Validation blocks are most useful for exactly that: the constraints your module invented, which no provider schema and no API will ever check for you.

The app: one variable is the whole interface

Callers of the container-apps module never touch a resource. They fill in one map, and the key of that map is the Dapr app id:

variable "apps" {
  type = map(object({
    image                 = string
    target_port           = number
    cpu                   = optional(number, 0.5)
    memory                = optional(string, "1Gi")
    min_replicas          = optional(number, 1)
    max_replicas          = optional(number, 10)
    workload_profile_name = optional(string, "Consumption")
    key_vault_secrets     = optional(map(string), {})

    ingress = optional(object({
      external_enabled  = optional(bool, false)
      allowed_ip_ranges = optional(map(string), {})
    }))

    service_bus_scale_rule = optional(object({
      queue_name    = string
      namespace     = string
      message_count = optional(number, 5)
    }))
  }))
  nullable    = false
  description = "Apps keyed by Dapr app id. The map key becomes the dapr app_id and the container app name suffix, so a Dapr component can never be scoped to a string that does not exist."

  validation {
    condition     = alltrue([for id, app in var.apps : can(regex("^[a-z][a-z0-9-]{1,30}[a-z0-9]$", id))])
    error_message = "App keys become Dapr app ids and container app names; use lower-case alphanumeric and hyphens, 3-32 chars."
  }

  validation {
    condition     = alltrue([for id, app in var.apps : app.min_replicas <= app.max_replicas])
    error_message = "min_replicas must be less than or equal to max_replicas."
  }

  validation {
    condition     = alltrue([for id, app in var.apps : contains([0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0], app.cpu)])
    error_message = "cpu must be one of the allocations the Consumption profile accepts: 0.25 through 2.0 in 0.25 steps."
  }

  # KEDA's namespace metadata is the bare name. Dapr's namespaceName, two modules over, is the
  # FQDN. Same concept, two formats, and neither one fails at plan time.
  validation {
    condition = alltrue([
      for id, app in var.apps :
      app.service_bus_scale_rule == null || !strcontains(try(app.service_bus_scale_rule.namespace, ""), ".")
    ])
    error_message = "The Service Bus scale rule namespace is the bare namespace name, not the FQDN. Drop the .servicebus.windows.net suffix."
  }
}

Keying on the Dapr app id rather than on the container app name is the one design decision in this module that changes what can go wrong later. A Dapr component's scopes list matches on app ids, the app id is otherwise just a string somebody typed twice, and the next section is about what happens when the two copies drift. Making it the map key means it is typed once, in tfvars, and everything else is derived.

Of the four validations, the character-set rule is the one that pays for itself. Azure enforces a name format on the container app and Dapr enforces its own on the app id, and the map key has to satisfy both, so checking it at the interface beats reading two different rejections from two different services. The CPU one covers an allocation set the provider does not check at all. The last one is the namespaceName versus namespace split: Dapr's Service Bus component wants the fully qualified namespace and KEDA's scaler wants the bare name, the two live two modules apart in the same stack, and neither one fails at plan time if you supply the other format.

Cross-variable validation is why the module needs Terraform 1.9 as a floor. The networking module's subnet check reads tonumber(split("/", var.address_space)[1]) + var.aca_subnet_newbits <= 27, and a validation block that references a second variable was an error before 1.9.

Then the resource, which is mostly each.value plumbing plus four things worth stopping on:

resource "azurerm_container_app" "this" {
  for_each = var.apps

  name                         = "ca-${each.key}-${var.environment}"
  container_app_environment_id = azurerm_container_app_environment.this.id
  resource_group_name          = var.resource_group_name
  revision_mode                = "Single"
  workload_profile_name        = each.value.workload_profile_name
  max_inactive_revisions       = 3
  tags                         = var.tags

  identity {
    type         = "UserAssigned"
    identity_ids = [var.app_identities[each.key].id]
  }

  registry {
    server   = var.container_registry_login_server
    identity = var.app_identities[each.key].id
  }

  # A Key Vault reference, not a copy of the value into state. The identity here has to be the
  # same one the AcrPull grant went to, or the revision fails to start on secret resolution.
  dynamic "secret" {
    for_each = each.value.key_vault_secrets

    content {
      name                = secret.key
      identity            = var.app_identities[each.key].id
      key_vault_secret_id = secret.value
    }
  }

  # app_id is the string Dapr component scopes match on. It is deliberately the map key, so the
  # dapr-components module can read it back off this resource instead of guessing a convention.
  dapr {
    app_id       = each.key
    app_port     = each.value.target_port
    app_protocol = "http"
  }

  dynamic "ingress" {
    for_each = each.value.ingress == null ? [] : [each.value.ingress]

    content {
      external_enabled = ingress.value.external_enabled
      target_port      = each.value.target_port
      transport        = "auto"

      traffic_weight {
        latest_revision = true
        percentage      = 100
      }
    }
  }

  template {
    min_replicas                     = each.value.min_replicas
    max_replicas                     = each.value.max_replicas
    polling_interval_in_seconds      = 30
    cooldown_period_in_seconds       = 300
    termination_grace_period_seconds = 30

    container {
      name   = each.key
      image  = each.value.image
      cpu    = each.value.cpu
      memory = each.value.memory

      # The client ID, not the principal ID. Both are GUIDs on the same identity and swapping
      # them fails at runtime with nothing to see at plan time.
      env {
        name  = "AZURE_CLIENT_ID"
        value = var.app_identities[each.key].client_id
      }

      readiness_probe {
        transport = "HTTP"
        port      = each.value.target_port
        path      = "/healthz/ready"
      }
    }

    dynamic "http_scale_rule" {
      for_each = each.value.ingress == null ? [] : [1]

      content {
        name                = "http-rps"
        concurrent_requests = "50"
      }
    }

    # identity_id on custom_scale_rule landed in azurerm 4.69.0. Below that floor a Service Bus
    # scale rule needs a connection string secret instead of the workload identity.
    dynamic "custom_scale_rule" {
      for_each = each.value.service_bus_scale_rule == null ? [] : [each.value.service_bus_scale_rule]

      content {
        name             = "sb-queue-depth"
        custom_rule_type = "azure-servicebus"
        identity_id      = var.app_identities[each.key].id

        metadata = {
          queueName    = custom_scale_rule.value.queue_name
          namespace    = custom_scale_rule.value.namespace
          messageCount = tostring(custom_scale_rule.value.message_count)
        }
      }
    }
  }

  lifecycle {
    precondition {
      condition     = each.value.ingress != null || each.value.service_bus_scale_rule != null
      error_message = "App ${each.key} has neither ingress nor a queue scale rule, so nothing can ever wake it."
    }
  }
}

The precondition catches what no validation block can reach, because it depends on two optional attributes of the same object rather than on one variable. An app with no ingress and no scale rule is legal HCL, legal Azure, and applies green. It then sits at zero replicas forever, because nothing can wake it: no ingress means no HTTP traffic, and no queue rule means no backlog signal. The failure is silence, which is the worst shape a failure can take in infrastructure code.

AZURE_CLIENT_ID carries the client ID. The role assignments that give this identity its Cosmos, Service Bus and Key Vault access take the principal ID. Both are GUIDs hanging off the same azurerm_user_assigned_identity, they look identical in a plan diff, and swapping them produces a clean apply and a 401 at runtime. That is the reason the identity module exports client_ids and principal_ids as two separately named maps instead of one convenient map of identity objects.

Three more things in that resource only start mattering after the first apply.

Revisions. revision_mode = "Single" here pairs with a traffic_weight block that just says latest_revision = true at 100 percent. Switch to Multiple and that block becomes a list you own: percentages have to sum to exactly 100, nothing is defaulted for you, and traffic_weight has no meaning at all in Single mode. max_inactive_revisions = 3 is the knob almost nobody sets. Container Apps keeps inactive revisions around, they show up in every listing and every diff, and three is enough history to roll back to while staying readable.

Scaling. What concurrent_requests = "50" actually counts is the thing Part 6 settled with arithmetic, against a Microsoft page that contradicts itself two paragraphs apart; this article is about the packaging, so take the number from there.

The packaging detail is custom_scale_rule.identity_id, which landed in azurerm 4.69.0. Below that floor a Service Bus scale rule has to authenticate with a connection string, which means a shared access key in a secret, which means the one credential the rest of this architecture went out of its way not to have. Copy that argument name out of the registry page and you will also get Error: Unsupported argument: the rendered docs spell it ìdentity_id, with a Latin i-grave, and in most editors it looks exactly like identity_id.

polling_interval_in_seconds and cooldown_period_in_seconds are template arguments rather than scale-rule ones, and 4.51.0 itself had neither; ~> 4.51 floats up to 4.81.0, which has both. They decide how quickly a scaled-to-zero worker notices a message and how long it lingers afterwards, so set them rather than inherit 30 and 300 by accident.

Secrets. The secret block takes either a literal value, which puts the secret in state in cleartext, or a key_vault_secret_id, which does not. The reference form needs an identity alongside it, and it has to be an identity that actually holds a secrets-read grant on the vault, or the revision fails to start on secret resolution rather than on the image pull. The sample passes versionless_id so that rotating the secret in Key Vault does not require a Terraform run.

Dapr components: `scopes` is the security boundary, and both ways of getting it wrong are silent

Get scopes wrong in either direction and terraform apply is green. Microsoft states the rule plainly: component scopes "correspond to the Dapr application ID of a container app, not the container app name", and "by default, all Dapr-enabled container apps in the same environment load the full set of deployed components" (dapr-components). Those are the two failure shapes.

Put the container app name in the list, or the Terraform resource name, or the app id with the environment suffix somebody added to the naming convention last quarter, and the component matches nothing. It is deployed, it is visible in the portal, it loads into no sidecar, and the first POST /v1.0/state/statestore comes back 500 at runtime. Leave scopes out entirely and the component loads into every Dapr-enabled app in the environment, including the ones with no business holding a connection to the orders database. The security argument for the second case is obvious; the cost argument is not. Dapr's Cosmos reference warns that Cosmos enforces a metadata request rate limit shared across the whole account and that new connections eat a large slice of it, then recommends scoping components to specific applications. Every sidecar in the environment opens its connection at startup whether it will ever read state or not.

The fix inside one module is to make the scope an attribute reference, scopes = [azurerm_container_app.orders.dapr[0].app_id], so the two strings cannot drift. Across module boundaries the reference has to survive the trip, which is why the container-apps module reads its own resource back rather than reusing the input map:

output "dapr_app_ids" {
  value       = { for id, app in azurerm_container_app.this : id => app.dapr[0].app_id }
  description = "Dapr app ids read back off the container app resource. This is what the scopes list on a Dapr component expects; the container app name is not."
}

keys(var.apps) would have produced the same strings today and would have gone on producing them after someone added a prefix inside the module. Reading the attribute makes it an edge in the graph as well as a value.

On the receiving side the components module refuses to accept the omission at all:

variable "state_store_scopes" {
  type        = list(string)
  description = "Dapr app ids allowed to load the state store. Not container app names, and not Terraform resource names."

  validation {
    condition     = length(var.state_store_scopes) > 0
    error_message = "Set scopes explicitly; an unscoped component is loaded by every Dapr-enabled app in the environment."
  }
}

There is no default. An unscoped component is a decision, and a module that lets you make it by forgetting to type something is a module that will make it for you.

The three components themselves carry no credentials:

# init_timeout is documented as an ISO 8601 string with the example "5s". Real ISO 8601 durations
# look like PT30S, and PT30S is not what the provider accepts. The Go-style form is correct.
resource "azurerm_container_app_environment_dapr_component" "state_cosmos" {
  name                         = "statestore"
  container_app_environment_id = var.container_app_environment_id
  component_type               = "state.azure.cosmosdb"
  version                      = "v1"
  init_timeout                 = "30s"
  ignore_errors                = false
  scopes                       = var.state_store_scopes

  metadata {
    name  = "url"
    value = var.cosmos_endpoint
  }

  metadata {
    name  = "database"
    value = var.cosmos_database
  }

  metadata {
    name  = "collection"
    value = var.cosmos_container
  }

  metadata {
    name  = "azureClientId"
    value = var.dapr_identity_client_id
  }

  metadata {
    name  = "actorStateStore"
    value = "true"
  }
}

# v2, not the v1 in the provider's own registry example. v1 strips the state key prefix as though
# keyPrefix were always none, and there is no migration path from v1 to v2.
resource "azurerm_container_app_environment_dapr_component" "state_blob" {
  count = var.blob_state_store == null ? 0 : 1

  name                         = "checkpoints"
  container_app_environment_id = var.container_app_environment_id
  component_type               = "state.azure.blobstorage"
  version                      = "v2"
  scopes                       = var.state_store_scopes

  metadata {
    name  = "azureClientId"
    value = var.dapr_identity_client_id
  }
}

resource "azurerm_container_app_environment_dapr_component" "pubsub_servicebus" {
  name                         = "orders-pubsub"
  container_app_environment_id = var.container_app_environment_id
  component_type               = "pubsub.azure.servicebus.topics"
  version                      = "v1"
  scopes                       = var.pubsub_scopes

  metadata {
    name  = "namespaceName"
    value = var.service_bus_namespace_fqdn
  }

  metadata {
    name  = "azureClientId"
    value = var.dapr_identity_client_id
  }

  metadata {
    name  = "consumerID"
    value = "{appID}"
  }
}

No masterKey, no connectionString, no accountKey, and no secret block backing any of them. The whole authentication story is azureClientId plus the client ID of a user-assigned identity that holds data-plane role assignments, which is the point at which this stops being a Terraform question and becomes the role assignment question further down the page.

Two lines in that sample disagree with the documentation on purpose. The blob component is version = "v2" where the provider's own registry example writes v1; Dapr's reference says users "should always use v2 by default" and that there is no migration path from v1 to v2, because v1 strips the state key prefix as though keyPrefix were always none. Copy the registry example into a new module and you have inherited a legacy component you cannot upgrade in place. And init_timeout is documented as "an ISO8601 formatted string. e.g. 5s, 2h, 1m", which is not what ISO 8601 durations look like. Write PT30S and it fails. 30s is correct and the doc text is wrong about the name of the format it is describing.

The subnet you cannot resize

infrastructure_subnet_id has been an input everywhere above. The networking module is where it gets made, carved out of one address space alongside two other prefixes, each with a different job:

locals {
  aca_prefix  = cidrsubnet(var.address_space, var.aca_subnet_newbits, 0)
  pe_prefix   = cidrsubnet(var.address_space, 8, 8)
  apim_prefix = cidrsubnet(var.address_space, 8, 9)
}

resource "azurerm_virtual_network" "this" {
  name                = "vnet-${var.workload}-${var.environment}"
  location            = var.location
  resource_group_name = var.resource_group_name
  address_space       = [var.address_space]
  tags                = var.tags
}

# Size is immutable once the environment exists. Changing this prefix is a rebuild of the
# environment and of every app and Dapr component in it.
resource "azurerm_subnet" "container_apps" {
  name                 = "snet-aca-infra"
  resource_group_name  = var.resource_group_name
  virtual_network_name = azurerm_virtual_network.this.name
  address_prefixes     = [local.aca_prefix]

  delegation {
    name = "aca-environment"

    service_delegation {
      name    = "Microsoft.App/environments"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

resource "azurerm_subnet" "private_endpoints" {
  name                              = "snet-pe"
  resource_group_name               = var.resource_group_name
  virtual_network_name              = azurerm_virtual_network.this.name
  address_prefixes                  = [local.pe_prefix]
  private_endpoint_network_policies = "Disabled"
}

resource "azurerm_subnet" "apim" {
  name                 = "snet-apim"
  resource_group_name  = var.resource_group_name
  virtual_network_name = azurerm_virtual_network.this.name
  address_prefixes     = [local.apim_prefix]

  delegation {
    name = "apim"

    service_delegation {
      name = "Microsoft.ApiManagement/service"
    }
  }

  # v5 shape. v4 wrote service_endpoints = ["Microsoft.Storage", "Microsoft.EventHub"].
  service_endpoint {
    service = "Microsoft.Storage"
  }

  service_endpoint {
    service = "Microsoft.EventHub"
  }
}

aca_subnet_newbits = 7 against a /16 produces a /23, and that number is the only one in this article you cannot revise later. Learn's rule for a workload profiles environment is that "the minimum subnet size required for virtual network integration is /27", and a few lines further down on the same page, "you can't modify subnet sizes after you create a Container Apps environment" (custom-virtual-networks#subnet). An undersized subnet is not a resize. It is a new environment, a new set of apps, new Dapr components, and a cutover.

/27 is legal and it caps hard:

Nine Dedicated nodes is a small production environment on a good day, and a rollout in revision_mode = "Single" runs the old and new revisions side by side, so the real ceiling during a deploy is roughly half the column. A /23 out of a 10.0.0.0/16 costs you address space you were never going to allocate to anything else.

Do not compute the size yourself from a reserved-IP count, because Microsoft states that count three ways. The body text on custom-virtual-networks#subnet says Container Apps "automatically reserves 12 IP addresses". The footnote under the sizing table on that same page says 14, "which includes 5 IP addresses that the subnet reserves". The workload profiles CLI page says 11, and concludes that a /27 has 21 available addresses, which contradicts the 18 in the table. Nobody has reconciled these. The sizing table is the source to trust, because it is the only one that resolves to node and replica limits rather than to an IP count, and its arithmetic agrees with itself. Read the replica column, double it for the rollout, and pick the row above that.

The /21 figure still circulating comes from the azurerm 4.51 docs and is stale by a factor of 64. The /23 minimum that sits next to it belongs to the legacy Consumption-only environment, and that is also where the delegation rule inverts rather than relaxes: a workload profiles subnet must be delegated to Microsoft.App/environments, and a Consumption-only subnet "must not be delegated to any services, including Microsoft.App/environments". Copying a Consumption-era module into a workload profiles environment gets both halves wrong at once. The actions list under service_delegation is the one Azure fills in for that delegation anyway; write it or leave it out, you cannot change it.

azurerm_virtual_network also accepts an inline subnet block, and a VNet that uses the inline form and standalone azurerm_subnet resources at the same time fights itself on every plan. Pick one, and pick the standalone resources.

The NSG rules everyone pastes are the Consumption rules

# Workload profiles rules, not the Consumption-only set. The Consumption environment needs
# an outbound allow to AzureCloud on 443 and to 1.1.1.1/1.0.0.1 on 53; a workload profiles
# environment reaches its control plane over the same 443 rules the app already needs.
resource "azurerm_network_security_group" "container_apps" {
  name                = "nsg-aca-${var.workload}-${var.environment}"
  location            = var.location
  resource_group_name = var.resource_group_name
  tags                = var.tags

  security_rule {
    name                       = "allow-apim-to-edge-proxy"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_ranges    = ["443", "31443"]
    source_address_prefix      = local.apim_prefix
    destination_address_prefix = local.aca_prefix
  }

  security_rule {
    name                       = "allow-lb-health-probe"
    priority                   = 110
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "30000-32767"
    source_address_prefix      = "AzureLoadBalancer"
    destination_address_prefix = local.aca_prefix
  }

  # An NSG is evaluated on both NICs, so the catch-all deny below would otherwise override the
  # default AllowVnetInBound at priority 65000 and break pod to pod traffic inside the subnet.
  security_rule {
    name                       = "allow-intra-subnet-inbound"
    priority                   = 120
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "*"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = local.aca_prefix
    destination_address_prefix = local.aca_prefix
  }

  security_rule {
    name                       = "deny-other-inbound"
    priority                   = 4000
    direction                  = "Inbound"
    access                     = "Deny"
    protocol                   = "*"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  security_rule {
    name                       = "allow-entra"
    priority                   = 130
    direction                  = "Outbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "443"
    source_address_prefix      = local.aca_prefix
    destination_address_prefix = "AzureActiveDirectory"
  }

  # Never deny 168.63.129.16. The environment stops working, and nothing in the portal says why.
  security_rule {
    name                       = "allow-azure-platform-dns"
    priority                   = 150
    direction                  = "Outbound"
    access                     = "Allow"
    protocol                   = "*"
    source_port_range          = "*"
    destination_port_range     = "53"
    source_address_prefix      = local.aca_prefix
    destination_address_prefix = "168.63.129.16"
  }

  security_rule {
    name                       = "deny-other-outbound"
    priority                   = 4000
    direction                  = "Outbound"
    access                     = "Deny"
    protocol                   = "*"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_subnet_network_security_group_association" "container_apps" {
  subnet_id                 = azurerm_subnet.container_apps.id
  network_security_group_id = azurerm_network_security_group.container_apps.id
}

The full module adds four more outbound allows on 443, to MicrosoftContainerRegistry, AzureFrontDoor.FirstParty, AzureMonitor and the private endpoint prefix. What is missing is the interesting part. Most Container Apps NSGs in the wild were copied off the Consumption-only tab of firewall-integration, and on a workload profiles environment those extra rules grant reachability nobody needs:

AzureActiveDirectory on 443 outbound sits on the workload profiles outbound list with a condition attached: "If you're using a managed identity, it's required" (firewall-integration). In this architecture that condition is never false, because every identity here is a managed identity and every one of them fetches its token over that path. Remove that rule and you do not break the platform; you break every grant the previous section set up, some hours later, when a cached token expires.

Two more rules are not about ports. Never deny 168.63.129.16: the docs carry an explicit warning and the environment stops working. And the moment a catch-all inbound deny exists, an explicit intra-subnet inbound allow becomes mandatory, because an NSG is evaluated on both the source and the destination NIC and a rule at priority 4000 wins over the default AllowVnetInBound at 65000. One more shows up only at apply time: service tags work in source_address_prefix and not in source_address_prefixes, so the plural form takes your AzureLoadBalancer string and fails on the API call.

The sharper point is that on an external workload profiles environment, none of the inbound rules do anything at all. Traffic arrives through a public IP in the environment's managed resource group and reaches the app without ever transiting your subnet, so the NSG sees it only on the way out. internal_only = true is what turns this file from documentation into enforcement, which is the Terraform-shaped version of the argument Part 6 made about Private Link at the gateway.

None of those rules apply until the association exists, which is the second dashed edge on the module diagram near the top of this article. The networking module exports nsg_association_id for callers who would rather have that ordering as a reference than as a hand-written prerequisite; the root module takes the blunter route and lists module.networking in depends_on.

Private DNS: the zone the private endpoint will not create for you

Four services go behind private endpoints and they need five zones:

locals {
  # ACR Premium needs the region-specific data zone as well as the registry zone.
  # Without it the manifest resolves privately and the layer download falls back to the public endpoint.
  private_dns_zone_names = {
    cosmos     = "privatelink.documents.azure.com"
    servicebus = "privatelink.servicebus.windows.net"
    keyvault   = "privatelink.vaultcore.azure.net"
    acr        = "privatelink.azurecr.io"
    acr_data   = "${var.location}.data.privatelink.azurecr.io"
  }
}

resource "azurerm_private_dns_zone" "this" {
  for_each = local.private_dns_zone_names

  name                = each.value
  resource_group_name = var.resource_group_name
  tags                = var.tags
}

# private_dns_zone_group on the endpoint writes the records. It does not link the zone to the
# VNet, and an unlinked zone resolves to the public IP with no DNS error anywhere.
resource "azurerm_private_dns_zone_virtual_network_link" "this" {
  for_each = azurerm_private_dns_zone.this

  name                 = "link-${each.key}-${var.workload}-${var.environment}"
  private_dns_zone_id  = each.value.id
  virtual_network_id   = azurerm_virtual_network.this.id
  registration_enabled = false
  tags                 = var.tags
}

resource "azurerm_private_endpoint" "container_registry" {
  name                = "pe-acr-${var.workload}-${var.environment}"
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = azurerm_subnet.private_endpoints.id
  tags                = var.tags

  private_service_connection {
    name                           = "psc-acr"
    private_connection_resource_id = var.container_registry_id
    subresource_names              = ["registry"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name = "acr"
    private_dns_zone_ids = [
      azurerm_private_dns_zone.this["acr"].id,
      azurerm_private_dns_zone.this["acr_data"].id,
    ]
  }
}

A private endpoint does exactly one DNS job for you, and it is not the one people assume. private_dns_zone_group writes the A record for the endpoint's private IP into the zones you list. It does not link those zones to a VNet. Skip the link and every lookup falls through to public DNS, which answers with the public IP, and nothing reports a DNS failure anywhere: the application gets a connection refused or a 403 from a firewall, and the first four people who look at it will look at the firewall. The for_each pairing zones with links is worth more than the typing it saves, because it makes an unlinked zone impossible to create.

Three details in that file will only ever be checked by the service. subresource_names is case-sensitive at the API and the casing is not consistent between services: Sql for Cosmos NoSQL, namespace for Service Bus, vault for Key Vault, registry for the registry. Cosmos in particular has one answer per API, so a Table or Gremlin account needs a different subresource and a different zone than the privatelink.documents.azure.com above. Second, privatelink.servicebus.windows.net is shared by Service Bus, Event Hubs, Relay and IoT Hub, so four unrelated services collide in one zone and one for_each key. Third, ACR Premium needs two zones. The registry zone resolves the manifest; the region-specific {region}.data.privatelink.azurecr.io zone resolves the data endpoint that serves the layers. Omit the second and a pull starts privately, then stalls or falls back partway through the download, which looks like a network problem and is a missing zone.

The zone nobody remembers belongs to the environment itself, and it lives in the root module because the environment is not a private endpoint at all:

# A private endpoint gets its records from private_dns_zone_group. The environment is not a
# private endpoint, so its default_domain zone is hand-built: wildcard for the apps, apex for
# the environment itself, both pointing at the internal load balancer address.
resource "azurerm_private_dns_zone" "container_apps_environment" {
  name                = module.container_apps.environment_default_domain
  resource_group_name = azurerm_resource_group.this.name
  tags                = local.required_tags
}

resource "azurerm_private_dns_zone_virtual_network_link" "container_apps_environment" {
  name                 = "link-cae-${local.name_suffix}"
  private_dns_zone_id  = azurerm_private_dns_zone.container_apps_environment.id
  virtual_network_id   = module.networking.vnet_id
  registration_enabled = false
  resolution_policy    = "Default"
  tags                 = local.required_tags
}

resource "azurerm_private_dns_a_record" "container_apps_wildcard" {
  name                = "*"
  private_dns_zone_id = azurerm_private_dns_zone.container_apps_environment.id
  ttl                 = 300
  records             = [module.container_apps.environment_static_ip]
  tags                = local.required_tags
}

Both halves of that are exported attributes, default_domain and static_ip_address, so no data source and no hard-coded domain is involved. The apex record is the same shape with name = "@". This zone only belongs on an internal environment: static_ip_address is the internal load balancer address when internal_load_balancer_enabled is true and a public IP when it is not, and publishing a public IP into a private zone is a confusing way to achieve nothing.

Coming from azurerm v4

Four v5 changes hit this stack, and the private DNS one is why the diff is large rather than interesting:

The link and record rows are mechanical: five zones against one VNet means five links, plus every A record resource. The row to slow down on is the Cosmos one, because it is a rename and a polarity flip. Carry the boolean across unchanged and you have written local_authentication_enabled = true, which turns key authentication back on in the same commit where you believed you turned it off, and the plan diff reads like a rename. The neighbouring services spell the same idea three more ways for good measure: Service Bus uses local_auth_enabled = false and Key Vault uses rbac_authorization_enabled = true.

None of the container_app resources themselves carry a breaking change across either major. The whole tree validated on ~> 4.51 and on ~> 5.6. Everything that hurt was in the network and the data services around them.

The identity chain, and the edge Terraform cannot see

One identity per app, and every grant it needs hanging off it:

locals {
  # Role GUIDs, not names. A renamed role keeps its ID, and a name costs a role-definitions
  # lookup on every plan.
  role_ids = {
    acr_pull         = "7f951dda-4ed3-4680-a7ca-43fe172d538d"
    kv_secrets_user  = "4633458b-17de-408a-b874-0445c86b69e6"
    sb_data_sender   = "69a216fc-b8fb-44d8-bc22-1f3c2cd27a39"
    sb_data_receiver = "4f6d3b9b-027b-4f4c-9142-0e5a2a2247e0"
  }

  role_definition_prefix = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/providers/Microsoft.Authorization/roleDefinitions"

  # Grants are scoped to the individual queue, never to the namespace.
  app_queue_pairs = {
    for pair in setproduct(tolist(var.app_ids), keys(var.service_bus_queue_ids)) :
    "${pair[0]}.${pair[1]}" => {
      app_id = pair[0]
      queue  = pair[1]
    }
  }
}

resource "azurerm_user_assigned_identity" "app" {
  for_each = var.app_ids

  name                = "id-${each.value}-${var.environment}"
  location            = var.location
  resource_group_name = var.resource_group_name
  tags                = var.tags
}

# principal_id is unknown until apply, so Terraform orders the identity ahead of every grant
# below without any help. The edge it cannot see is the one from the container app to this
# grant; see the depends_on on the container-apps module call in the root module.
resource "azurerm_role_assignment" "acr_pull" {
  for_each = var.app_ids

  scope              = var.container_registry_id
  role_definition_id = "${local.role_definition_prefix}/${local.role_ids.acr_pull}"
  principal_id       = azurerm_user_assigned_identity.app[each.value].principal_id

  # Correct for a managed identity and wrong for a user or a group: it suppresses the existence
  # check that fails with PrincipalNotFound while Entra is still replicating the new principal.
  principal_type                   = "ServicePrincipal"
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "service_bus_sender" {
  for_each = local.app_queue_pairs

  scope                            = var.service_bus_queue_ids[each.value.queue]
  role_definition_id               = "${local.role_definition_prefix}/${local.role_ids.sb_data_sender}"
  principal_id                     = azurerm_user_assigned_identity.app[each.value.app_id].principal_id
  principal_type                   = "ServicePrincipal"
  skip_service_principal_aad_check = true
}

# Cosmos data-plane access is not azurerm_role_assignment. A Cosmos DB Built-in Data Contributor
# handed out through the control plane grants nothing at the data plane, and the apply is green.
# 0001 is Data Reader, 0002 is Data Contributor.
resource "azurerm_cosmosdb_sql_role_assignment" "data_contributor" {
  for_each = var.app_ids

  resource_group_name = var.cosmosdb_resource_group_name
  account_name        = var.cosmosdb_account_name
  role_definition_id  = "${var.cosmosdb_account_id}/sqlRoleDefinitions/00000000-0000-0000-0000-000000000002"
  principal_id        = azurerm_user_assigned_identity.app[each.value].principal_id
  scope               = "${var.cosmosdb_account_id}/dbs/${var.cosmosdb_database_name}"
}

Two edges in that file run in opposite directions and only one of them exists. principal_id is unknown until apply, so Terraform has to create the identity before it can even render the role assignment, and it orders them correctly with no help. The container app in the next module over references the identity and the registry login server, never the grant, so Terraform is equally free to create the app first and let the first revision fail its pull. That is the failed pull from the opening paragraph, and the depends_on on the module call is the whole of the fix.

The same grant carries two more failure modes that have nothing to do with ordering. A brand new service principal can return PrincipalNotFound from the role assignment API while Entra is still replicating it, which skip_service_principal_aad_check = true suppresses. That argument is correct for every managed identity and wrong for a user or a group, where the check is the only thing standing between a typo and a grant to nothing. And the one nothing in Terraform can fix: managed identity backends cache tokens per resource "for around 24 hours", and Cosmos data-plane assignments propagate on their own schedule, so a green apply is evidence that a grant exists and not that it works yet. Removing a grant lags the same way, which matters more.

The GUID map is not premature optimisation: Microsoft's own advice is to use IDs because "even if the role is renamed, the role ID does not change", and role_definition_name makes the provider list role definitions on every plan. app_queue_pairs comes from the same instinct. setproduct over apps and queues produces one grant per app per queue, scoped to azurerm_servicebus_queue.orders.id. Scoping to the namespace instead is the most common over-grant in this architecture, and it is one shorter scope argument away.

The Cosmos resource at the bottom breaks the intuition of anyone who already knows Azure RBAC. azurerm_role_assignment with a Cosmos DB Built-in Data Contributor role does exist, does apply, and grants nothing at the data plane, because Cosmos keeps its own role system underneath the account. azurerm_cosmosdb_sql_role_assignment takes a sqlRoleDefinitions/... path, not a subscription-level role definition ID, and its scope is a data-plane path like <account>/dbs/orders. Built-in ...0001 is Data Reader and ...0002 is Data Contributor. This is what makes local_authentication_enabled = false on the account survivable: turn keys off without the data-plane assignment and every read returns 403, so the two changes belong in the same commit.

State, and the lease nothing expires

One root module, one backend config per environment, one tfvars file per environment, and no workspaces:

use_oidc             = true
use_azuread_auth     = true
storage_account_name = "sttfstateprod01"
container_name       = "tfstate"
key                  = "container-apps/prod.tfstate"

subscription_id = "00000000-0000-0000-0000-000000000000"

workload      = "orders"
environment   = "prod"
location      = "westeurope"
address_space = "10.70.0.0/16"

cost_center = "GAZE"
owner       = "AZE"
project     = "orders"

image_tag    = "1.4.2"
min_replicas = 2
max_replicas = 30

log_retention_days = 90

dedicated_workload_profiles = {
  D4 = {
    profile_type  = "D4"
    minimum_count = 1
    maximum_count = 5
  }
}

# orders_api_key has no default and is deliberately absent here. Supply it as
# TF_VAR_orders_api_key from the pipeline, not from a file in git.

cd infra
terraform init -reconfigure -backend-config=envs/prod.backend.hcl
terraform plan -var-file=envs/prod.tfvars -out=tfplan
terraform apply tfplan

use_azuread_auth = true is now a required option rather than an opt-in when you authenticate to the state account with Entra ID, and resource_group_name moved from required to optional on the backend; it is needed only alongside lookup_blob_endpoint. The -reconfigure on init lets the same working directory point at a different state blob without Terraform offering to migrate the old one.

Refusing workspaces is a decision with a cost, so here are both halves. HashiCorp's own position is that "CLI workspaces within a working directory use the same backend", which makes them unsuitable for environments that differ in blast radius. The Azure-specific reason is uglier: the azurerm backend builds the workspace blob name by string concatenation, so a staging workspace on key = "container-apps/prod.tfstate" writes its state to a blob literally named container-apps/prod.tfstateenv:staging. What you give up is per-environment configuration divergence. Anything dev needs that prod does not has to become a variable, and a variable that exists only to make dev cheaper is one somebody will eventually set in prod.

The lock deserves more attention than the docs give it. All the backend documentation says is "supports state locking", and the mechanism is in the implementation: Lock() acquires a blob lease on the state file with LeaseDuration: -1, the Terraform lock ID travels as the proposed lease ID in the x-ms-lease-id header, and the holder is recorded in the blob's terraformlockid metadata. There is no lock table to provision, which is the pleasant half.

The unpleasant half is that -1 means infinite, and infinite means nothing expires it. A runner that is hard-killed mid-apply leaves the state locked until a human runs terraform force-unlock <LOCK_ID> or breaks the lease in the portal. That is a runbook entry, not a piece of folklore, and it is the reason every plan and apply in the next section passes -lock-timeout=5m and the workflow sets cancel-in-progress: false. The default reflex of cancel-in-progress: true is precisely the wrong setting here: it cancels a running apply halfway through and strands the lease it was holding.

Correct reasoning gets you the wrong answer on one RBAC detail. Storage Blob Data Reader is not enough for terraform plan, because plan takes the lock and taking the lock writes blob metadata. Grant Contributor scoped to the container, or run read-only plans with -lock=false and accept that you can read a state file somebody else is halfway through writing.

Tags: the provider argument that does not exist

cost_center, owner and project are in that tfvars file because the policy in this tenant denies any resource missing cost-center, owner, environment and project. That is a provider-level problem in every IaC tool that has provider-level tags. azurerm is not one of them:

# azurerm has no provider-level default_tags, so the policy-required four go in a locals map and
# get assigned on every resource. merge() where a resource needs extras.
locals {
  required_tags = {
    "cost-center" = var.cost_center
    "owner"       = var.owner
    "environment" = var.environment
    "project"     = var.project
  }
}

resource "azurerm_log_analytics_workspace" "this" {
  name                = "log-${local.name_suffix}"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  sku                 = "PerGB2018"
  retention_in_days   = var.log_retention_days
  tags                = local.required_tags
}

tags = local.required_tags on every resource, tags = merge(local.required_tags, { ... }) where one needs extras, and a review that catches the resource where somebody forgot. The tracker will tell you otherwise if you let it: issue #11682 asking for default_tags has 48 reactions and is closed as completed, while PR #31108 adding the argument has been open and unmerged since November 2025. A closed-as-completed issue is not a released feature.

The tooling will not correct you either. Write default_tags { tags = local.required_tags } into a provider block, run terraform validate, and you get Success!, because validate never decodes the provider block, and neither does plan when no resource needs a provider instance: a configuration whose only content was that invented block also returned No changes. "It validated" is not evidence that an argument exists. terraform providers schema -json is the answer: the azurerm provider block carries no tag attribute and exactly one nested block, features.

The pipeline: plan on the PR, apply behind an environment gate

name: Terraform

on:
  pull_request:
    branches: [main]
    paths: ["TerraformContainerAppsDemo/infra/**"]
  push:
    branches: [main]
    paths: ["TerraformContainerAppsDemo/infra/**"]

permissions:
  contents: read

concurrency:
  group: terraform-${{ github.ref }}
  # Never true here. The azurerm backend lock is an infinite blob lease, so a cancelled apply
  # strands the state locked until someone runs terraform force-unlock.
  cancel-in-progress: false

env:
  TF_IN_AUTOMATION: "true"
  ARM_USE_OIDC: "true"
  ARM_USE_AZUREAD: "true"
  ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }}
  ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
  ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }}

jobs:
  plan:
    name: Plan (prod)
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write
    defaults:
      run:
        working-directory: TerraformContainerAppsDemo/infra
    steps:
      - uses: actions/checkout@v5

      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: 1.16.3

      - name: Init
        id: init
        run: terraform init -input=false -backend-config=envs/prod.backend.hcl

      - name: Validate
        id: validate
        run: terraform validate -no-color

      - name: Plan
        id: plan
        env:
          TF_VAR_orders_api_key: ${{ secrets.ORDERS_API_KEY }}
        run: terraform plan -input=false -no-color -lock-timeout=5m -var-file=envs/prod.tfvars -out=tfplan
        continue-on-error: true

      - name: Fail if plan errored
        if: steps.plan.outcome == 'failure'
        run: exit 1

      # A saved plan carries sensitive values in cleartext and any user with repo read can
      # download an artifact. Keep the retention short.
      - name: Upload plan
        if: github.event_name == 'push'
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: TerraformContainerAppsDemo/infra/tfplan
          retention-days: 1

  apply:
    name: Apply (prod)
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    # The approval gate is this key and nothing else. Required reviewers, wait timers and
    # deployment branch rules are configured on the environment in repo settings.
    environment: prod
    permissions:
      contents: read
      id-token: write
    defaults:
      run:
        working-directory: TerraformContainerAppsDemo/infra
    steps:
      - uses: actions/checkout@v5

      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: 1.16.3
          terraform_wrapper: false

      - uses: actions/download-artifact@v4
        with:
          name: tfplan
          path: TerraformContainerAppsDemo/infra

      - name: Init
        run: terraform init -input=false -backend-config=envs/prod.backend.hcl

      # No -var-file and no -auto-approve: a saved plan takes no further planning options, and
      # passing the file is the approval.
      - name: Apply
        run: terraform apply -input=false -lock-timeout=5m tfplan

The trimmed step is a github-script block that posts the plan as a PR comment and updates that comment on every push instead of stacking new ones. Four smaller choices in the file get a line each. id-token: write sits on each job and never at workflow level, because at workflow level every job in the file holds a token exchangeable for Azure credentials, including the ones that only run a linter. azure/login is absent because the azurerm provider performs the OIDC exchange itself. The apply job passes a saved plan with no -var-file and no -auto-approve, since a saved plan accepts no further planning options and handing it over is the approval. And terraform validate needs no Azure credentials at all, because subscription_id is required for plan and apply and not for validate.

Everything that actually breaks in this file breaks on one string. The federated credential matches the sub claim on the token GitHub mints, and that claim changes shape per trigger:

repo:OWNER/REPO:pull_request
repo:OWNER/REPO:ref:refs/heads/main
repo:OWNER/REPO:environment:prod

Three subjects, so three federated credentials on the identity. The precedence rule is what makes this a trap rather than an inconvenience: GitHub uses the environment name whenever the job references an environment, pull_request only if the job references no environment and the trigger was a pull request, and the branch only when neither applies (subject claim examples). Add environment: prod to the plan job so it can read a secret and you have silently rewritten that job's own subject, its credential stops matching, and the AADSTS700213-class error you get back says nothing about environments.

The segment key is singular ref:. HashiCorp's own azurerm OIDC guide prints refs:refs/heads/main; GitHub mints the token, GitHub uses ref:, and the plural form is a typo that will cost you an afternoon because the failure looks like a trust configuration problem rather than a spelling one.

Then there is the change that made every older tutorial wrong. "Repositories created after July 15, 2026 now use an immutable default subject format that includes both the owner ID and repository ID" (immutable subject claims), which looks like repo:OWNER@OWNER-ID/REPO@REPO-ID:ref:refs/heads/BRANCH. Repos created before that date keep the old format until they opt in, and a rename or a transfer after it flips them. The IDs cannot be stripped back out with include_claim_keys. Copy a subject string out of any pre-2026 blog post, including a good one, and it will not match on a new repository. Read the real sub off a token, or build the string from the owner and repo IDs the API gives you. There is a ceiling on that string, too: an application or user-assigned identity accepts a maximum of 20 federated identity credentials and wildcards are not supported, so one credential per subject is a budget rather than a pattern.

Drift detection that is still believed in week three

on:
  schedule:
    - cron: "17 6 * * 1-5"
  workflow_dispatch:

jobs:
  drift:
    runs-on: ubuntu-latest
    # If the prod environment has required reviewers, this scheduled run sits waiting for
    # approval. Give drift its own read-scoped environment with no reviewers when that happens.
    environment: prod
    permissions:
      contents: read
      id-token: write
      issues: write
    steps:
      # -detailed-exitcode is what makes this possible: 0 no changes, 1 error, 2 changes.
      # Without it a plan with changes and a plan without both exit 0.
      - name: Plan with detailed exit code
        id: plan
        env:
          TF_VAR_orders_api_key: ${{ secrets.ORDERS_API_KEY }}
        run: terraform plan -input=false -no-color -lock-timeout=5m -detailed-exitcode -var-file=envs/prod.tfvars
        continue-on-error: true

      - name: Fail the job on a real error
        if: steps.plan.outputs.exitcode == '1'
        run: exit 1

      # A detector that only opens issues trains everyone to ignore it within two weeks.
      - name: Close the drift issue when clean
        if: steps.plan.outputs.exitcode == '0'
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh issue list --state open --label drift --json number --jq '.[].number' \
            | xargs -r -I{} gh issue close {} --comment "Plan is clean again."

The checkout, setup and init steps are the same three as the plan job, and the step that opens or comments on the issue on exit code 2 is in the companion repo. continue-on-error: true on the plan step is not optional: exit code 2 is a success for our purposes and a failure as far as Actions is concerned, so without it the step fails, every conditional after it is skipped, and you get a red run that tells you nothing about whether prod drifted.

The step most drift workflows are missing is the last one. Opening an issue when the plan is dirty is the easy half; closing it when the plan comes back clean keeps anyone reading the issues in week three. A detector that only ever adds to a pile teaches the team to filter the label, and after that it is a scheduled job that costs money and detects nothing.

Whether this ever runs comes down to scheduling rules the docs bury. Scheduled workflows only ever run from the default branch, so you cannot test a schedule change on a branch and workflow_dispatch is how you try it. GitHub delays scheduled runs during high load at the top of the hour, hence 17 rather than 0. And a public repository has its scheduled workflows disabled after 60 days without activity, which lands exactly when drift has had time to accumulate and nobody is watching. The environment: prod on this job exists to match the third federated credential, and it brings the approval gate with it: if that environment has required reviewers, the scheduled run sits waiting for a human who is not expecting to approve anything. Give drift a read-scoped environment of its own with its own credential.

Conclusion

A grant before a pull. An NSG before an environment. A zone link before a lookup. A data-plane role before a read. None of those four are things your configuration has any reason to reference, which is why depends_on and a handful of precondition blocks are where you write them down, and why every silent failure in this article is the same failure.

The full tree is in the companion repo, including the two workflow files and the steps trimmed out of the samples above. It validates and fmts clean on Terraform 1.16.3 with azurerm 5.6.0, and it deploys nothing on its own.

One question, and it splits teams cleanly: does your Terraform run plan on every pull request, or only after the merge?

Azure Front Door, API Management and Container Apps: What Each Hop Trusts

Martin Oehlert — Fri, 18 Sep 2026 09:29:39 +0000

You can put Front Door, API Management and an internal Container Apps environment in front of orders-api and still ship an app that takes a token from any workload in your tenant. Container Apps authentication checks that API Management's managed identity token came from your tenant, and checks nothing else, so a workload with any identity calls the app directly and names whichever end user it likes. Entra closes that with one setting. The four hops in front of the app each run on a different answer to the same question, and only two of those answers are a credential.

Front Door takes TLS, the WAF and a per-client rate limit off the gateway's hands, which leaves API Management deciding who may call and Container Apps running the code. The scenario is the order API from Part 5, with the Function App gone.

The shape you land on, and the tiers it forces

Each tier below is the cheapest one with a feature the topology cannot do without.

Front Door Premium. On Standard, "only custom rules are supported": the managed Default Rule Set and Bot Manager are Premium, and so are Private Link origins. Either reason alone settles it.

API Management Standard v2. The gateway has to accept Front Door's traffic privately and send its own traffic into the VNet that holds the Container Apps environment. Those are separate networking features, and the tier table decides which instances get both:

Two constraints choose for you, and both bite late. Classic Premium looks like the enterprise answer and is the tier that blocks you: "In the classic API Management tiers, private endpoints aren't supported in instances injected in an internal or external virtual network." For the v2 tiers Microsoft's private endpoint guide says to "combine inbound private endpoints to Standard v2 instances with outbound virtual network integration to provide end-to-end network isolation", which leaves Standard v2 as the lowest tier that does both.

The other is an ordering rule, and it breaks pipelines. Integration on its own leaves the gateway and developer portal endpoints publicly reachable, so you switch public network access off once the private endpoint exists. You can only do that on an instance that already exists, never in the deployment that creates it, which means the gateway is public for as long as it takes you to run the second deployment. Give it an integration subnet of its own: at least /27, delegated to Microsoft.Web/serverFarms, with a network security group attached.

Container Apps: a workload profiles environment, internal, in your own VNet. You pick accessibility once, when you create the environment, and an internal one has no public endpoint. The subnet is fixed at creation too, and you cannot resize it afterwards: /27 minimum, delegated to Microsoft.App/environments. Size it for the most revisions you expect to keep active at once. Single revision mode briefly doubles address use during a revision change, and orders-api runs in multiple revision mode, where every active revision keeps its own minReplicas running on the same subnet (one address per 10 replicas on the Consumption profile, one per node on a dedicated one).

Service Bus Premium. Service Bus private endpoints are a Premium feature, and the data hop at the end of this article runs over them.

Front Door: what the edge actually does for you

WAF on a JSON API

The WAF policy is a separate resource from the profile, and a security policy attaches it to the custom domain:

resource wafPolicy 'Microsoft.Network/FrontDoorWebApplicationFirewallPolicies@2025-03-01' = {
  name: 'wafOrdersApi'
  location: 'global'
  sku: { name: 'Premium_AzureFrontDoor' }   // defaults to Classic_AzureFrontDoor when omitted
  properties: {
    policySettings: {
      enabledState: 'Enabled'
      mode: 'Detection'
    }
    managedRules: {
      managedRuleSets: [
        {
          ruleSetType: 'Microsoft_DefaultRuleSet'
          ruleSetVersion: '2.2'
          ruleSetAction: 'Block'
        }
      ]
    }
    customRules: {
      rules: [
        {
          name: 'ApiPerClientRateLimit'
          priority: 10
          enabledState: 'Enabled'
          ruleType: 'RateLimitRule'
          rateLimitDurationInMinutes: 5
          rateLimitThreshold: 3000
          matchConditions: [
            {
              matchVariable: 'RequestUri'
              operator: 'Contains'
              matchValue: [ '/api/' ]
            }
          ]
          action: 'Block'
        }
      ]
    }
  }
}

resource wafAttachment 'Microsoft.Cdn/profiles/securityPolicies@2025-06-01' = {
  parent: frontDoorProfile
  name: 'waf-api-contoso'
  properties: {
    parameters: {
      type: 'WebApplicationFirewall'
      wafPolicy: { id: wafPolicy.id }
      associations: [
        {
          domains: [ { id: apiCustomDomain.id } ]
          patternsToMatch: [ '/*' ]
        }
      ]
    }
  }
}

Start in mode: 'Detection' on purpose. DRS 2.x uses anomaly scoring: a request is acted on once its score reaches 5, which one Critical match does on its own, and in Detection mode that action is only a log entry. Microsoft's rollout guidance is detection, then tuning, then prevention, and "the whole process might take several weeks." Do the tuning in Bicep or the CLI: changing the rule set version in the portal "resets all previous customizations".

Expect Authorization to be your first exclusion, because Entra access tokens in a request header "can contain special characters that trigger a false positive detection". The WAF also "doesn't support content-encoding", so a client that gzips its POST body gets no meaningful body inspection.

The rate limit rule is narrower than its property names suggest. It counts requests "from each socket IP address" over a fixed window of one or five minutes, and once a client crosses the threshold, "all traffic matching that rate limiting rule is blocked for the remainder of the fixed window." The action in the rule above is one of only two a rate limit rule accepts, Log or Block. Counters live on individual Front Door servers, so low thresholds leak: below about 200 requests a minute "you might see some requests above the threshold get through". The five-minute window with a larger threshold is the more accurate shape, and you take the number itself from a week of Detection logs.

Layer 3, 4 and 7 DDoS protection arrives with the profile whether you write a WAF policy or not.

One origin, and it is the gateway

The origin is API Management: Front Door cannot see apps behind an internal environment, and the per-path routing it did during the migration now happens in the gateway. If you leave API Management out, the Private Link origin is the environment itself, and Part 5 covered why that takes every app in it private at once.

resource apimOriginGroup 'Microsoft.Cdn/profiles/originGroups@2025-06-01' = {
  parent: frontDoorProfile
  name: 'apim-gateway'
  properties: {
    loadBalancingSettings: {
      sampleSize: 4
      successfulSamplesRequired: 3
    }
    healthProbeSettings: {
      probePath: '/status-0123456789abcdef'
      probeRequestType: 'GET'
      probeProtocol: 'Https'
      probeIntervalInSeconds: 30
    }
  }
}

resource apimOrigin 'Microsoft.Cdn/profiles/originGroups/origins@2025-06-01' = {
  parent: apimOriginGroup
  name: 'apim-primary'
  properties: {
    hostName: '${apim.name}.azure-api.net'
    originHostHeader: '${apim.name}.azure-api.net'
    httpsPort: 443
    priority: 1
    weight: 1000
    enforceCertificateNameCheck: true
    sharedPrivateLinkResource: {
      privateLink: { id: apim.id }
      groupId: 'Gateway'
      privateLinkLocation: apim.location
      requestMessage: 'Front Door Premium to the orders gateway'
    }
  }
}

resource apiEndpoint 'Microsoft.Cdn/profiles/afdEndpoints@2025-06-01' = {
  parent: frontDoorProfile
  name: 'orders-api'
  location: 'global'
  properties: { enabledState: 'Enabled' }
}

resource apiRoute 'Microsoft.Cdn/profiles/afdEndpoints/routes@2025-06-01' = {
  parent: apiEndpoint
  name: 'api'
  dependsOn: [ apimOrigin ]
  properties: {
    customDomains: [ { id: apiCustomDomain.id } ]
    originGroup: { id: apimOriginGroup.id }
    patternsToMatch: [ '/api/*' ]
    supportedProtocols: [ 'Https' ]
    forwardingProtocol: 'HttpsOnly'
  }
}

Deploying this does not connect anything yet. Front Door creates a private endpoint that shows up as a pending connection on the API Management instance, and traffic flows only after someone approves it with az network private-endpoint-connection approve. Put that step in the pipeline, or the deployment succeeds and serves nothing. A Private Link origin can't share an origin group with public origins, and it has a ceiling worth writing down before you size anything: "each Front Door regional cluster has a limit of 7200 RPS (requests per second) per Front Door profile", and requests past it come back 429.

The route has no cacheConfiguration, and leaving it out is how you disable caching for an API. The managed certificate on apiCustomDomain has a condition of its own: it renews by itself only if the domain's CNAME "points directly to a Front Door endpoint".

The probe tests the gateway

You are configuring this probe for a day that has not arrived yet. With one gateway region there is one origin, and a single-origin group routes to it whatever the probe reports. The probe starts deciding anything on the day you add a second region, which is also the day you would least like to discover you pointed it at the wrong path.

/status-0123456789abcdef is the path Microsoft's Front Door and API Management guide configures. It is built into every managed gateway outside the Consumption tier and answers 200 when the gateway is up, which is all a probe choosing between gateways needs to know. On a Private Link origin, health probes take the same network path as traffic.

An app route would fail as a probe target. The gateway rejects a probe that carries no subscription key or token, and the /health endpoint from Part 4 is mapped in Development only. When every origin fails, Front Door "considers all origins unhealthy and routes traffic in a round robin distribution across all of them": the probe fails open. App health is decided one layer down anyway, by readiness probes on the container apps and by a circuit breaker on the API Management backend.

API Management: the only layer that knows the caller

Front Door knows a socket address and Container Apps knows a token issuer. Only API Management knows which person or partner is calling and on which product, so most of this architecture's configuration lives in its policies.

A backend per app, reached through the VNet

There is no backend for "the Container Apps environment". An environment has no single URL: each app has its own FQDN and the environment's proxy routes on the host header. API Management gets one backend entity per app, and the VNet gets a private DNS zone that resolves those names to the environment's internal address:

param acaDefaultDomain string   // az containerapp env show --query properties.defaultDomain
param acaStaticIp string        // az containerapp env show --query properties.staticIp

resource ordersBackend 'Microsoft.ApiManagement/service/backends@2024-05-01' = {
  parent: apim
  name: 'orders-api'
  properties: {
    url: 'https://orders-api.${acaDefaultDomain}'
    protocol: 'http'
  }
}

resource acaZone 'Microsoft.Network/privateDnsZones@2024-06-01' = {
  name: acaDefaultDomain
  location: 'global'
}

resource acaRecords 'Microsoft.Network/privateDnsZones/A@2024-06-01' = [for recordName in [ '*', '@' ]: {
  parent: acaZone
  name: recordName
  properties: {
    ttl: 3600
    aRecords: [ { ipv4Address: acaStaticIp } ]
  }
}]

resource acaZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2024-06-01' = {
  parent: acaZone
  name: 'vnet-orders'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: { id: vnet.id }
  }
}

The zone is named after the environment's default domain, and it has to be linked to the VNet that API Management integrates with, or the gateway resolves the public name and gets nowhere. Leave the host header alone: "by default, API Management overrides the host name that's sent to the back end" with the host of the URL it calls, which is the app FQDN the environment proxy routes on. Reference it from policy with backend-id rather than base-url, because Part 5 covered the trap in mixing the two across scopes.

The failure that survives all of this is a 404. Aspire maps an internal endpoint to internal-only ingress, so azd infra gen writes ingress.external: false for every project the AppHost doesn't mark with WithExternalHttpEndpoints(), and on an internal environment that setting hides the app from the rest of the VNet as well: the DNS name resolves and the TLS handshake succeeds, but the proxy rejects the request with a 404. It reads as a routing bug and is a visibility setting. Add WithExternalHttpEndpoints() to orders-api in the AppHost; on an internal environment, external ingress still keeps the app off the internet.

One inbound policy, in the order it has to run

The policy on the orders API does the rest of this section's work, and the order of the elements is load-bearing:

<policies>
  <inbound>
    <base />
    <check-header name="X-Azure-FDID" failed-check-httpcode="403"
                  failed-check-error-message="Invalid request." ignore-case="false">
      <value>{{FrontDoorId}}</value>
    </check-header>
    <validate-azure-ad-token tenant-id="{{TenantId}}" output-token-variable-name="caller">
      <audiences>
        <audience>{{OrdersPublicAudience}}</audience>
      </audiences>
    </validate-azure-ad-token>
    <rate-limit-by-key calls="120" renewal-period="60"
                       counter-key="@("orders-" + ((Jwt)context.Variables["caller"]).Subject)" />
    <set-header name="X-End-User-Id" exists-action="override">
      <value>@(((Jwt)context.Variables["caller"]).Claims.GetValueOrDefault("oid", ""))</value>
    </set-header>
    <set-backend-service backend-id="orders-api" />
    <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />
    <authentication-managed-identity resource="{{OrdersApiAppId}}" />
  </inbound>
  <backend><base /></backend>
  <outbound><base /></outbound>
  <on-error><base /></on-error>
</policies>

check-header goes first, so nothing downstream spends a token validation on a request that did not come through your profile. Front Door adds X-Azure-FDID to every request it forwards, and anyone with portal access can read the value, so it proves only that a request passed through your profile. That still matters, because Front Door's address ranges are shared: "IP address filtering alone isn't sufficient ... because other Azure customers use the same IP addresses", and someone else's profile pointed at your gateway passes an IP filter and fails this check. Keep it once Private Link is in place, as the backstop for the day public network access is switched back on. ignore-case is required on this policy.

validate-azure-ad-token is the reason the Front Door hop uses Private Link. Front Door can authenticate to an origin with a managed identity, but it "overwrites an existing Authorization header with its origin authentication token", so the caller's token would never reach this line unless a Front Door rule copied it into a different header first. The feature is still in preview: its authentication block exists only in a preview API version of the origin group, which also adds a tokenDestinationHeader that can move the token out of Authorization. Neither helps here, because the feature "isn't currently supported for origins with Private Link enabled". output-token-variable-name keeps the validated token as a Jwt object for the elements after it.

Microsoft's example for rate-limit-by-key keys on context.Request.IpAddress, which is the wrong key here. Behind Front Door, the connection API Management sees comes from Front Door, so an IP key throttles Front Door itself. The policy above keys on the validated subject instead, and the per-client-IP limit stays where it already works, in the WAF. Counters are tracked per gateway ("It doesn't aggregate call data across the entire instance") and shared by every scope that uses the same key, hence the orders- prefix. v2 tiers count with a token bucket where classic tiers use a sliding window, and Microsoft warns that inconsistent limits on a shared key there "can cause unpredictable behavior", so give every policy that shares a key the same numbers.

The last element, authentication-managed-identity, is the reason the two set-header elements above it are there at all. It requests a token for the orders-api app registration and writes it into Authorization, over the top of the caller's own token, so the end user's identity reaches orders-api only if you copy it out first: here as the oid claim in X-End-User-Id, with exists-action="override" so that a client sending its own X-End-User-Id gets it replaced rather than trusted. The subscription key goes in the same pass, because by default it is "passed to the backend and might be exposed in backend monitoring logs".

The gateway then caches that token until it expires, and the element "doesn't validate which backend the token is sent to". Anyone who can write an API policy can point this element at a resource of their choosing and collect the gateway identity's token, so treat policy-edit rights as token rights.

Subscription keys, per product

A partner's key is issued against a product that requires a subscription, and the API it contains is the one the policy above is attached to:

resource ordersHttpApi 'Microsoft.ApiManagement/service/apis@2024-05-01' = {
  parent: apim
  name: 'orders'
  properties: {
    displayName: 'Orders'
    path: 'api/orders'
    protocols: [ 'https' ]
    subscriptionRequired: true
  }
}

resource partnerProduct 'Microsoft.ApiManagement/service/products@2024-05-01' = {
  parent: apim
  name: 'orders-partners'
  properties: {
    displayName: 'Orders for partners'
    subscriptionRequired: true
    state: 'published'
  }
}

resource partnerProductOrders 'Microsoft.ApiManagement/service/products/apis@2024-05-01' = {
  parent: partnerProduct
  name: ordersHttpApi.name
}

A subscription key tells you which product a caller bought, and it is a shared secret with no lifecycle: API Management "doesn't provide built-in features to manage the lifecycle of subscription keys, such as setting expiration dates or automatically rotating keys". That is why the policy validates a token as well, and because tenant-id pins the issuer to your tenant, a partner needs a token your tenant issues: from an app registration you create for them, or from their multitenant app once it is consented into your tenant.

The same page describes the configuration that quietly disables every key you have issued. "If the key isn't valid but a product exists that includes the API without requiring a subscription (an open product), API Management ignores the key and handles the request as an API request without a subscription key." One open product left over from a demo turns a revoked partner key into anonymous access. Scope is the other trap: requests made with an API-scoped, all-APIs or built-in all-access subscription skip product-scope policies entirely. Keep the checks in the policy above at API scope, and use product scope for what legitimately differs per product, such as rate-limit and quota, which only apply when a subscription key is used.

Managed identity to Container Apps, and who else gets a token

Every check so far runs in the gateway, which holds up until a caller skips it.

Container Apps authentication, the Easy Auth sidecar, validates the token that the last policy element attaches, and on its own it accepts far more than the gateway. Microsoft's page for this exact flow says it "allows any client application in your Microsoft Entra tenant to request an access token and authenticate to the target app." Any workload in your tenant with a managed identity could call orders-api directly, set X-End-User-Id to any user's object ID, and look like API Management vouching for that user.

That is worth a second read, because it inverts the usual reading of this diagram. The hop that gets pointed at when someone calls this architecture secure is the hop where nothing about the caller is checked beyond which tenant it lives in.

Close it in Entra. Define an app role, assign it to the gateway's identity, and set assignment required on the orders-api service principal. The client credentials documentation says that setting "will block users and applications without assigned roles from being able to get a token for this application", so a workload without the role never gets as far as the sidecar:

az ad app update --id "$ORDERS_API_APP_ID" --set api.requestedAccessTokenVersion=2
az ad app update --id "$ORDERS_API_APP_ID" --app-roles \
  '[{"allowedMemberTypes":["Application"],"value":"Orders.Gateway","displayName":"Orders gateway","description":"API Management calls orders-api","isEnabled":true}]'
GATEWAY_ROLE_ID=$(az ad app show --id "$ORDERS_API_APP_ID" --query "appRoles[?value=='Orders.Gateway'].id" -o tsv)

# Role assignments and the assignment requirement live on the service principal, which may not exist yet.
ORDERS_SP_ID=$(az ad sp show --id "$ORDERS_API_APP_ID" --query id -o tsv 2>/dev/null \
  || az ad sp create --id "$ORDERS_API_APP_ID" --query id -o tsv)
az ad sp update --id "$ORDERS_SP_ID" --set appRoleAssignmentRequired=true

# The Azure CLI has no command for assigning an app role to a managed identity.
APIM_PRINCIPAL_ID=$(az resource show --ids "$APIM_ID" --query identity.principalId -o tsv)
az rest -m POST \
  -u "https://graph.microsoft.com/v1.0/servicePrincipals/$APIM_PRINCIPAL_ID/appRoleAssignments" \
  -b "{\"principalId\":\"$APIM_PRINCIPAL_ID\",\"resourceId\":\"$ORDERS_SP_ID\",\"appRoleId\":\"$GATEWAY_ROLE_ID\"}"

az containerapp auth microsoft update \
  --resource-group rg-orders --name orders-api \
  --client-id "$ORDERS_API_APP_ID" \
  --issuer "https://login.microsoftonline.com/$TENANT_ID/v2.0" \
  --yes
az containerapp auth update \
  --resource-group rg-orders --name orders-api \
  --enabled true --unauthenticated-client-action Return401

The first command exists because of the issuer line. The token version API Management receives is chosen by the orders-api registration: a requestedAccessTokenVersion of null or 1 results in v1.0 tokens, and the manifest reference is explicit that null is where a registration starts, since "if the value is null, this parameter defaults to 1, which corresponds to the v1.0 endpoint". A v1.0 token's issuer is https://sts.windows.net/<tenant>/, which never matches the /v2.0 URL Microsoft tells you to configure. Set the registration to 2 and the two match by construction.

Assignment required decides who gets a token. Any role on the registration satisfies it, and Easy Auth "doesn't perform the validation steps" for roles, so a role check in code is the second line, for the day someone gives another identity a different role on the same registration:

using System.Text.Json;

public static class GatewayCaller
{
    private sealed record PrincipalClaim(string Typ, string Val);
    private sealed record ClientPrincipal(string? RoleTyp, PrincipalClaim[]? Claims);

    private static readonly JsonSerializerOptions Json =
        new() { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower };

    public static bool HasRole(HttpRequest request, string role)
    {
        string? header = request.Headers["X-MS-CLIENT-PRINCIPAL"];
        if (string.IsNullOrEmpty(header)) return false;

        if (JsonSerializer.Deserialize<ClientPrincipal>(Convert.FromBase64String(header), Json)
            is not { Claims: { } claims, RoleTyp: var roleType })
            return false;

        return claims.Any(c => (c.Typ is "roles" || c.Typ == roleType) && c.Val == role);
    }
}

var orders = app.MapGroup("/api/orders")
    .AddEndpointFilter(async (context, next) =>
        GatewayCaller.HasRole(context.HttpContext.Request, "Orders.Gateway")
            ? await next(context)
            : Results.StatusCode(StatusCodes.Status403Forbidden));

The identity headers come from the sidecar, and "external requests aren't allowed to set these headers". The Base64 X-MS-CLIENT-PRINCIPAL header and its role_typ field are documented for App Service's Easy Auth, whose claims "undergo a default claims-mapping process", which is why the filter accepts roles or whatever role_typ names. Role changes also arrive late. Managed identity back ends cache tokens per resource "for around 24 hours", and API Management caches its own until expiry, so a token issued before the assignment can keep arriving without the roles claim, and the filter answers 403 until it ages out.

Container Apps: the layer that decides who gets in and how many answer

Which ingress settings let the gateway in

Whether API Management can reach orders-api at all depends on the environment's accessibility, fixed when the environment is created, and on the app's external flag, which you can change on any deploy without creating a revision.

The bottom-right cell is the Aspire 404 from the backend section, and the bottom-left cell is where orders-api belongs. inventory-api sits outside the table, because it has no ingress at all. Other Dapr-enabled apps still invoke it through Dapr service invocation, which is the only way anything in this design calls it.

This is orders-api as the rest of the section configures it:

param blueSuffix string
param greenSuffix string = ''
param greenWeight int = 0
param apimIntegrationSubnetPrefix string

var latestSuffix = !empty(greenSuffix) ? greenSuffix : blueSuffix

resource ordersApp 'Microsoft.App/containerApps@2026-01-01' = {
  name: 'orders-api'
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    environmentId: acaEnvironment.id
    configuration: {
      activeRevisionsMode: 'Multiple'
      ingress: {
        external: true
        targetPort: 8080
        transport: 'auto'
        ipSecurityRestrictions: [
          {
            name: 'apim-outbound'
            ipAddressRange: apimIntegrationSubnetPrefix
            action: 'Allow'
          }
        ]
        traffic: !empty(blueSuffix) && !empty(greenSuffix) ? [
          { revisionName: 'orders-api--${blueSuffix}', label: 'blue', weight: 100 - greenWeight }
          { revisionName: 'orders-api--${greenSuffix}', label: 'green', weight: greenWeight }
        ] : [
          { revisionName: 'orders-api--${blueSuffix}', label: 'blue', weight: 100 }
        ]
      }
      dapr: { enabled: true, appId: 'orders-api', appPort: 8080 }
    }
    template: {
      revisionSuffix: latestSuffix
      containers: [
        { name: 'orders-api', image: 'acrorders.azurecr.io/orders-api:${latestSuffix}' }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 20
        rules: [
          { name: 'http-rate', http: { metadata: { concurrentRequests: '40' } } }
        ]
      }
    }
  }
}

The only caller with ingress access is API Management, leaving from its integration subnet, so the allow rule is that subnet's prefix; Front Door's addresses never reach this app. Rules take CIDR ranges only, and all of one type ("You can't combine allow rules and deny rules"), and they cannot read a header, so the X-Azure-FDID check stays in the gateway. If a browser client ever appears, CORS goes in API Management's cors policy as the first inbound element, because during a preflight "only the cors policy is evaluated on the OPTIONS request" and the request never reaches the app.

The HTTP rule counts a rate, not concurrency

concurrentRequests: '40' reads as "add a replica when 40 requests are in flight". It means something else. "Every 15 seconds, the number of concurrent requests is calculated as the number of requests in the past 15 seconds divided by 15": requests per second, averaged over a 15-second window, and Microsoft's own example describes its setting as "100 concurrent requests per second". The replica count then follows desiredReplicas = ceil(currentMetricValue / targetMetricValue).

Request duration appears nowhere in that formula, so one number can be wrong in both directions. Take the rule above on paper. An order lookup answering in 25 ms at 400 requests per second holds about 10 requests in flight across the whole app, and the rule asks for ceil(400 / 40) = 10 replicas, each doing one thing at a time. A report export taking 4 seconds at 5 requests per second holds 20 requests open and gets ceil(5 / 40) = 1 replica. The first overspends, the second queues, and the replica chart for both looks like autoscaling working.

Set the value from throughput. Load one replica until latency starts to climb, note the request rate it sustained, and set concurrentRequests below that. If an app serves both shapes of request, the rule can be right for one of them only, which is an argument for moving the slow endpoint into its own app. maxReplicas is the cost ceiling either way: the scale limits table puts its default at 10 with 1,000 configurable, and replicas arrive in a scale up step of "1, 4, 8, 16, 32, ... up to configured maximum replicas".

minReplicas: 1 matters more here than it did during the migration. Nothing sends orders-api traffic on a schedule any more, because Front Door's probe stops at the gateway's status path, so at zero replicas the first call after a quiet spell waits for a replica to start while API Management holds the request open. scale.cooldownPeriod does not soften that: "The cool down period only takes effect when scaling in from the final replica to 0."

Queue depth for the app nobody calls over HTTP

inventory-api scales on the backlog of the orders topic instead:

resource inventoryApp 'Microsoft.App/containerApps@2026-01-01' = {
  name: 'inventory-api'
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    environmentId: acaEnvironment.id
    configuration: {
      activeRevisionsMode: 'Single'
      dapr: { enabled: true, appId: 'inventory-api', appPort: 8080 }
    }
    template: {
      containers: [
        { name: 'inventory-api', image: 'acrorders.azurecr.io/inventory-api:${releaseSuffix}' }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'orders-backlog'
            custom: {
              type: 'azure-servicebus'
              identity: 'system'
              metadata: {
                namespace: serviceBusNamespaceName
                topicName: 'orders'
                subscriptionName: 'inventory-api'
                messageCount: '30'
              }
            }
          }
        ]
      }
    }
  }
}

There is no ingress block, and activeRevisionsMode: 'Single' is deliberate: the scale rules page asks for single revision mode whenever a scale rule is not an HTTP one, which is one reason the canary lives on orders-api. messageCount is a target per replica, so 150 waiting messages ask for five replicas, and subscriptionName has to name the subscription the Dapr component drains, the pairing Part 5 covered. Keep namespace even though that Microsoft sample leaves it out: KEDA's Service Bus scaler requires it when it authenticates with an identity, and no Container Apps page says the platform fills it in.

identity: 'system' authenticates the rule with the app's own identity. Container Apps documents neither the Service Bus role that identity needs nor whether the scaler reaches a namespace with public access disabled. The nearest answers are both too generous: the AKS guide for KEDA assigns Azure Service Bus Data Owner, and KEDA's own reference wants a Manage policy for a connection string, either of them more than a consumer should hold. That is why minReplicas is 1: a replica that exists keeps draining the subscription whether or not the scaler can read it, and it is also awake for Dapr invocations from orders-api, which don't pass through ingress and so can't wake an app from zero.

Revisions: the rollout Front Door no longer does

In Part 5, Front Door origin weights moved traffic between two runtimes. With one origin and one gateway, the rollout axis is the app's revisions, two layers below anything Front Door can see.

The traffic block follows Microsoft's blue-green template. The first deploy sets only blueSuffix, and the single entry sends everything to the one revision that deploy creates. The next deploy sets greenSuffix, creates the green revision and keeps it at greenWeight: 0. Weights split the requests that arrive on the app's own FQDN, which is the URL API Management's backend calls, so a deploy with greenWeight: 10 is a canary for every caller of the API with no gateway change. Labels give a revision its own address, orders-api---green.<default domain>, and a label keeps its URL when you move it to another revision. The zone's * record already resolves that name, but the smoke test still has to come from an address ipSecurityRestrictions allows: add a test operation in API Management that targets the label address, or add your build agents' subnet to the rule for the duration of the rollout.

Promotion follows the template too: az containerapp ingress traffic set --label-weight blue=0 green=100, then a deploy with greenWeight: 100 so the next template run doesn't put the old split back. Rollback is the same with the numbers reversed. Neither creates a revision, because ingress changes apply to every revision at once. Keeping both revisions active has a price: minReplicas counts per revision, so each one holds its own warm replica, and sticky sessions are single revision mode only, which Part 5 ran into from the Front Door side.

The trust chain, hop by hop

Here are the four hops from the introduction again, with what the receiving side of each one actually relies on:

You settled the first two in the gateway section. From Front Door to API Management, trust rests on a network path and an identifier, and neither of those is a credential, which is why check-header stays in the policy. From API Management to orders-api, the assignment requirement decides which identities can get a token at all, and your role check confirms that the one which arrived carries the gateway's role.

Sidecar to app: the token you no longer set

Container Apps generates APP_API_TOKEN for you. The platform injects it into every Dapr-enabled app, "unique per each app and app revision", and the supported dapr settings have no field for it or for DAPR_API_TOKEN. Part 3's endpoint filter still does the checking, and its fail-open branch never runs in Azure: the variable is always set when Dapr is enabled. The token "can also change at any time", so never copy it into Key Vault or another app's settings.

Placement matters with two filters in play. The token filter goes on routes the sidecar calls: inventory-api's invoked methods and its pub/sub handler. It never goes on the orders-api routes API Management calls, because those requests carry no dapr-api-token and every one of them would get a 401. The GatewayCaller role check is the mirror image and belongs only on those gateway routes.

The token proves that a call came through inventory-api's own sidecar. It does not say which app sent it. The sidecars secure that call with mutual TLS on their own, but the Dapr access control policies that would restrict which app IDs may invoke inventory-api live in the Configuration spec, which Container Apps does not support. Any Dapr-enabled app in the environment can call it, so on this hop the boundary is who is allowed to deploy into the environment.

App to data, and the network under all four hops

On the last hop managed identity carries the trust with no secret anywhere, and what you add is plumbing. Both apps reach Service Bus and Cosmos DB with their own identities and data-plane role assignments, wired as in Part 2.

The VNet in the diagram carries four subnets with one owner each: API Management's inbound private endpoint, its /27 integration subnet, the environment's /27, and snet-pe for the data services. With public network access disabled on both services, each app resolves them through private DNS zones, privatelink.servicebus.windows.net and privatelink.documents.azure.com, linked to the same VNet as the environment's own zone. Miss a link and the name resolves to a public IP instead of the private one, so the failure shows up as a refused or timed-out connection from the app, with no DNS error anywhere.

Verify on first deploy

Each item below rests on documentation that describes the pieces separately and never shows them working together, so check it once in your own subscription:

The Front Door origin reports healthy once the private endpoint connection is approved.
A managed identity without the app role is refused a token for orders-api. The assignment requirement is documented for applications in general, never for a managed identity.
One decoded X-MS-CLIENT-PRINCIPAL from a real gateway call tells you the claim type the role arrives under.
A call to orders-api from a VM in another subnet is refused. That is the ingress proxy comparing the source address you expect it to compare.
The inventory-api replica count follows a test backlog with public network access disabled on the namespace, which proves the scaler reaches Service Bus with the role you granted.

Conclusion

This architecture gets sold on one line: each layer authenticates to the next with managed identity. On the data hop that line is true. On the hop from API Management to orders-api it is true only after Entra refuses tokens to identities without the app role, and false every minute before that. The other two hops never claimed it in the first place. Front Door to API Management runs on a network path and an identifier, and sidecar to app runs on a token the platform generates for you.

It is still a sound design, as long as the checks carrying the trust that managed identity does not carry are actually deployed. The expensive mistake is dropping the header check or the assignment requirement because "it's managed identity". The other price is on the invoice: no public origin means a Premium Front Door profile, a v2 gateway and a Premium namespace, all billed before the first container starts, so price them for your region before the design review rather than after it.

Part 7 puts this stack into Terraform, where the steps this article ran by hand (the private endpoint approval, the app role, the assignment requirement) have to become code or be forgotten in the next environment.

Have you set assignment required on your API's service principal, or are you relying on the role check in app code?

Azure Functions to Container Apps, One Endpoint at a Time

Martin Oehlert — Fri, 11 Sep 2026 06:06:01 +0000

You have /api/orders/* running on a Dapr-enabled Container App in staging and /api/reports/* nowhere near ready, and Azure Front Door will move production traffic for you either way. The two ways are different objects and you cannot get one from the other: a route maps a set of path patterns to exactly one origin group, while a weight splits traffic between origins inside a single origin group. Take the route and the cutover happens one path at a time, reverting by deleting a pattern. Take the weight and both runtimes have to serve every path on that route, and the ratio you typed does nothing at all until you change a load balancing setting almost nobody mentions. The four phases below are the order those two levers go in, and what each one costs you when it goes wrong.

Phase 1: deploy alongside and route nothing

The running scenario for the rest of this article: an order API on api.contoso.com, fronted by a Front Door Standard profile, with a single route whose patterns are ['/*'] pointing at a Function App origin. Everything is on that one route today. /api/orders/* moves to the Container App first, /api/reports/* moves last.

Phase 1 adds two things and changes nothing else. The Container App gets deployed and registered in the Front Door profile as an origin in its own origin group. No route points at that group yet, so it serves zero production requests, and the only claim phase 1 makes is that Front Door can reach it and gets a 200 back from a probe.

The /* route stays exactly where it is, for the whole migration. It is the catch-all: the fallback for every endpoint you have not moved yet, and the thing that catches a path back when you revert one. Front Door has no implicit default, and a request matching no route errors out with one of two status codes depending on which doc you read, so do not build anything that inspects the code.

Everything here is Front Door Standard/Premium, resource type Microsoft.Cdn/profiles. Front Door classic (Microsoft.Network/frontDoors) retires on 2027-03-31: no new profiles, no new domain onboarding, no new managed certificates. If your existing profile is classic, the migration in front of you is a different one from this article's.

The ordering decision worth making now is which path goes last, and /api/reports/* is last here for a reason that is not "reports matter less". Front Door's origin response timeout is 16 to 240 seconds, and "this timeout value is applied to all endpoints in the Azure Front Door profile". A long-running report path cannot be given a longer timeout than the order path next to it. Whichever endpoint has the widest latency spread sets the number for the whole profile, so you move it when you have the most information.

The probe is the first thing that breaks

Front Door's health probe has a rule that reads as a detail and behaves as an outage: only 200 OK counts as healthy. Not 204, not 301. Put a Function App whose /healthz returns 204, which plenty do on purpose, in an origin group next to a container that returns 200, and you have manufactured an outage in the component you added to prevent one.

The second rule contradicts the first place you would look: the probe path is case sensitive, while route path patterns are case insensitive, to the point that Front Door rejects /FOO and /foo as duplicates in the same field.

The Container Apps side is quieter and worse. With ingress enabled and no probes defined, Container Apps adds defaults that are TCP checks against the ingress target port, and a TCP probe passes as long as something is bound to it. A process that is listening and broken, whose database connection died on startup or whose configuration failed to bind, passes every default probe it gets: false green, at the exact moment you are looking for evidence that the new runtime works.

On top of startup, readiness and liveness, a Dapr-enabled app adds a fourth opinion: the sidecar's own app health probe (dapr.appHealth, with its own path, interval, timeout and threshold), which decides whether the sidecar considers your app up. That is a separate question from whether the platform considers the replica up, and from whether Front Door considers the origin healthy.

Collapse it onto one answer: one HTTP health endpoint returning exactly 200, the same path with the same casing on both origins, and explicit probes replacing the TCP defaults so the container's readiness reflects the check Front Door is making:

probes:
  - type: Readiness
    httpGet:
      path: /healthz      # same string, same casing, as the Front Door probePath
      port: 8080

Then have the Function App serve /healthz at 200 too, before you point a single probe at it, and check which verb you are probing with while you are there. New profiles default to HEAD, which is the right choice for load, and a MapGet answers it with 405 while an [HttpTrigger(..., "get")] answers it with 404. Either add the verb on both origins or set probeRequestType: 'GET', because the rule at the top of this aside does not care why the response was not a 200.

Phase 2: send the new endpoints and nothing else

Phase 2 is the first config that carries traffic, and it is two resources: an origin group for the orders path, and a route pointing at it. Everything else in the profile is untouched.

resource ordersOriginGroup 'Microsoft.Cdn/profiles/originGroups@2021-06-01' = {
  name: 'orders-origin-group'
  parent: frontDoorProfile
  properties: {
    loadBalancingSettings: {
      sampleSize: 4
      successfulSamplesRequired: 3
      // Load-bearing, and it does nothing in phase 2. Leave it at the default 0 and the
      // weights you set in phase 3 are ignored entirely. See "Your 90/10 split is a
      // 100/0 split" below.
      additionalLatencyInMilliseconds: 500
    }
    healthProbeSettings: {
      probePath: '/healthz'          // case sensitive, unlike patternsToMatch
      probeRequestType: 'HEAD'
      probeProtocol: 'Https'
      probeIntervalInSeconds: 30     // set it; the documented default varies by source
    }
  }
}

resource ordersRoute 'Microsoft.Cdn/profiles/afdEndpoints/routes@2021-06-01' = {
  name: 'orders-route'
  parent: frontDoorEndpoint
  dependsOn: [ ordersOrigin ]   // the origin group must not be empty when the route is created
  properties: {
    originGroup: { id: ordersOriginGroup.id }
    supportedProtocols: [ 'Https' ]
    // Both patterns, deliberately. The wildcard rules do not settle whether
    // '/api/orders/*' also matches a bare '/api/orders', and POST /api/orders is the
    // create-order call. Listing both is correct either way and costs nothing.
    patternsToMatch: [ '/api/orders', '/api/orders/*' ]
    forwardingProtocol: 'HttpsOnly'
    linkToDefaultDomain: 'Enabled'
    httpsRedirect: 'Enabled'
  }
}

The /* route from phase 1 is still deployed and still points at the Function App, and nothing about this new route mentions it. That omission is why the strangler fig shape works in Front Door: there is no priority field on a route and declaration order is irrelevant. Front Door "always matches to the most-specific request by evaluating the left-hand side properties: protocol, domain, and path, in that order", with the frontend host matched exactly and the path matched against exact patterns first, wildcard patterns second. /api/orders/12345 hits orders-route and /api/reports/monthly falls through to the catch-all, and you did not order anything to make that happen.

The wildcard syntax has three rules and they are all about the end of the string: * is valid only as the last character, with nothing after it, preceded by /. A pattern with no wildcard is an exact match, and so is a pattern ending in /, which is the one that catches people:

The CLI equivalent, if your Front Door is not in Bicep yet:

az afd route create \
  --resource-group rg-orders --profile-name afd-orders \
  --endpoint-name orders-endpoint --route-name orders-route \
  --origin-group orders-origin-group \
  --patterns-to-match "[/api/orders,/api/orders/*]" \
  --supported-protocols "[Https]" \
  --forwarding-protocol HttpsOnly \
  --https-redirect Enabled --link-to-default-domain Enabled

There is no --cache-configuration flag on that command, and its absence is the setting: omitting it leaves caching off for the route. A migrating order API wants it off badly enough that phase 3 spends an aside on what happens if you forget.

Doing this one path at a time has a ceiling, and it is far away: a Standard profile allows 100 routes and 100 origin groups (Premium doubles both), 100 path patterns per route, and a composite limit of 5000 routes per profile counted as domains multiplied by paths. An API with thirty endpoints is not close to any of those.

A route is a switch, not a dial

Read the route above as a percentage control and you will misread the whole migration. A route maps its patterns to exactly one origin group and 100% of matching requests go there. There is no weight, ratio or percentage anywhere in a route's properties, and no partial state between "the Function App serves /api/orders" and "the Container App serves it".

The percentage lever lives one level down: origin weights, integers from 1 to 1000, on the origins inside a single origin group. That is a different object with different behaviour, and the two do not compose the way most migration write-ups imply. You cannot put 10% of /api/orders/* on the Container App by editing a route, because what the route selects is the group, not the origin. Which leaves a design fork that decides the rest of your migration:

Two origin groups plus two routes. Per-path cutover, no percentage available. You revert by removing a pattern from patternsToMatch, and the catch-all route re-catches the path. Blast radius is one path pattern.
One origin group plus two origins. Percentage rollout by weight, reverted by disabling the new origin. The cost is that both origins are eligible for every path on that route, so the Function App and the Container App each have to serve the whole pattern set.

Phase 2 is the first shape, deliberately: a path you have never sent a request to is the cheapest thing in the world to move, and the revert is one line of config. Phase 3 is the second shape, where the weights come out and turn out not to work at their defaults.

The host header decides which app answers

Both origins need one property that has no obvious symptom when it is missing, because Front Door is documented to fill it in for you: "If you use Azure Resource Manager templates or another method without explicitly setting this field, Front Door sends the incoming host name as the value for the host header." Omit originHostHeader in Bicep and the Container App receives Host: api.contoso.com.

That breaks on both sides, and harder on the new one. App Service and Functions want the host header to match the backend domain unless the custom domain is bound to the app. Container Apps goes further: the environment's ingress proxy routes traffic to the correct app, revision and replica based on host headers, so a wrong header is not a mismatched certificate, it is the environment being unable to work out which of your apps you meant. Set it explicitly on each origin, and keep both lines next to each other so the asymmetry is obvious in review:

originHostHeader: 'orders-api.orangeplant-77e5875b.westeurope.azurecontainerapps.io'  // Container App
originHostHeader: 'orders-func.azurewebsites.net'                                     // Function App

The same property decides which app answers a probe, so an unset host header can turn phase 1's green probe into a phase 2 outage without either config changing in between.

Both runtimes on one Service Bus topic

Phase 2 moved an HTTP path. The order topic is the first Azure resource both runtimes hold at once, and it is where a migration stops being a routing exercise. Three files decide whether that goes well, and the same string appears in all three.

The Dapr component on the Container App:

# components/orders-pubsub.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orders-pubsub
spec:
  type: pubsub.azure.servicebus.topics
  version: v1
  metadata:
    - name: namespaceName
      value: "sb-orders.servicebus.windows.net"
    - name: azureClientId
      value: "<user-assigned-identity-client-id>"
    # The line that decides whether this is a cutover or a duplicate.
    # consumerID is the Service Bus SUBSCRIPTION NAME, not a label.
    - name: consumerID
      value: "orders-fn"
    # The Function App's bicep already created the topic and the subscription.
    - name: disableEntityManagement
      value: "true"

The Function App, unchanged, still deployed, still consuming:

[Function(nameof(ProcessOrder))]
public Task ProcessOrder(
    [ServiceBusTrigger("orders", "orders-fn", Connection = "ServiceBus")] Order order)

And the scale rule on the Container App, which is the file people forget is part of the same decision:

scale: {
  minReplicas: 0
  maxReplicas: 10
  rules: [ { name: 'orders-topic', custom: {
    type: 'azure-servicebus'
    identity: 'system'
    metadata: {
      topicName: 'orders'
      subscriptionName: 'orders-fn'   // the same literal as consumerID above
      messageCount: '30'
    }
  } } ]
}

orders-fn three times, in three different formats, validated by nothing.

That shared string buys you the second row of a two-row decision, and that decision is the whole coexistence story. A subscription on a Service Bus topic "resembles a virtual queue that receives copies of the messages that are sent to the topic", and every subscription gets its own copy of every published message. The number of subscriptions is the number of times each order gets processed.

The second row is a supported configuration rather than a hack. Subscriptions "support the same patterns described earlier in this section regarding queues: competing consumer, temporal decoupling, load leveling, and load balancing." Dapr already does it to itself: replicas sharing an app-id get each message delivered to "only one instance of that application", so adding the Function widens an existing race rather than starting a new one. Which runtime wins any given race is not something you get to know, so do not write a handler whose correctness depends on which side processed order 41.

One default changes load rather than behaviour, so nothing will page you about it. Dapr's maxConcurrentHandlers defaults to 0, meaning unlimited; the Functions host's maxConcurrentCalls defaults to 16. Point both runtimes at the same downstream database and the new side applies materially more concurrent pressure than the side it replaces, on a number nobody typed.

Sessions are the one mismatch here that fails loudly, at component init. Session properties belong to the subscription metadata rather than the component metadata (requireSessions, default false; maxConcurrentSessions, default 8), and the Functions equivalent is IsSessionsEnabled. Point a default Dapr subscriber at a session-enabled subscription the Function has been draining for two years and the component source returns subscription %s already exists but session requirement doesn't match, with nothing in the portal mentioning sessions. If you enable sessions on the Dapr side, check the concurrency number too: 8 against the Functions host's 2000 is a drop of two and a half orders of magnitude.

consumerID is the subscription name, and it defaults to your app ID

The Dapr reference page describes consumerID as a grouping label. Verbatim: "Consumer ID (consumer tag) organizes one or more consumers into a group. Consumers with the same consumer ID work as one virtual consumer ... If the consumerID is not provided, the Dapr runtime set it to the Dapr application ID (appID) value."

Read that as a label and you will leave it unset. The component source says what the page does not, which is that the value is a name of a real Azure entity:

// pubsub/azure/servicebus/topics/servicebus.go
r, rErr := a.client.GetClient().NewReceiverForSubscription(req.Topic, a.metadata.ConsumerID, nil)
...
err := a.client.EnsureSubscription(subscribeCtx, a.metadata.ConsumerID, req.Topic, opts)

ConsumerID goes straight into NewReceiverForSubscription and EnsureSubscription as the subscription argument. The consumer ID is the Service Bus subscription name. That mapping comes from components-contrib, not from the reference page, so cite the source if you have to defend it in review.

Follow the default through the migration and you get a specific outcome. Deploy the Container App with app ID orders-api and no consumerID, and Dapr creates a Service Bus subscription literally named orders-api and consumes from it. Nothing about orders-fn changes. The Function keeps working perfectly. You are now in the first row of that table: every order processed twice, by two runtimes, with no error raised anywhere, and the only visible symptom is downstream. Duplicate confirmation emails, doubled inventory decrements, two rows where you expected one.

The second half of the pairing is the scale rule. subscriptionName in the KEDA azure-servicebus rule is the entity KEDA polls to decide how many replicas you need. consumerID is the entity Dapr drains. Both are naming the same kind of thing, so if the two strings differ, KEDA is measuring a backlog that the app it is scaling will never touch. The failure is quiet in the direction that matters: with minReplicas: 0, KEDA watches a subscription that stays empty, keeps the app at zero replicas, and the subscription that is actually filling up has no consumer at all. Nothing validates the pair, in either resource.

Entity management is the third thing the component does that a migration does not want it doing. disableEntityManagement defaults to "false", meaning create-if-missing. The source shows what "automatically" costs: the component builds a second, administrative client, and EnsureSubscription calls EnsureTopic and then GetSubscription before deciding whether to create anything. A failed GetSubscription is an error rather than a shrug:

res, err := c.adminClient.GetSubscription(ctx, topic, subscription, nil)
if err != nil {
    return false, fmt.Errorf("could not get subscription %s: %w", subscription, err)
}

GetSubscription is a management-plane call. A Container App whose managed identity holds only Azure Service Bus Data Receiver has no path to the management plane, so on that reading subscription setup fails at init while the identity holds every right it needs to actually receive messages. The permission that looks correct on a least-privilege review is the one that stops the sidecar from starting. That chain is read off the source rather than watched, so treat the exact failure text as unconfirmed until you have the sidecar log in front of you.

One related waste of time. Four fields in the component reference (lockDurationInSec, maxDeliveryCount, defaultMessageTimeToLiveInSec, autoDeleteOnIdleInSec) each carry the qualifier "Used during subscription creation only", and the source confirms there is no update path. In a migration you are always pointing at a subscription that already exists, so all four are inert: setting maxDeliveryCount in your component YAML changes nothing about how many times a message is delivered. Set it on the Service Bus entity.

What to do instead, and it is the file at the top of this section. Set consumerID explicitly to the Function's existing subscription name. Set disableEntityManagement: "true". Pre-create the topic and the subscription in the Function App's existing bicep, which already owns them. Make the KEDA subscriptionName the same literal, and if your infrastructure is in Bicep, make it the same variable so the compiler enforces what neither service does. Then grep for that string before you deploy: it should appear in the component, the trigger attribute, and the scale rule, and nowhere else.

The envelope goes one way cleanly and one way silently

Dapr "uses the CloudEvents 1.0 specification as its message format" and wraps outgoing messages automatically. Publish an order from the Container App and this is what lands on the topic:

{
  "topic": "orders",
  "pubsubname": "orders-pubsub",
  "data": { "orderId": "order-123", "customerId": "cust-42", "total": 89.50 },
  "id": "5929aaac-a5e2-4ca1-859c-edfe73f11565",
  "specversion": "1.0",
  "datacontenttype": "application/json; charset=utf-8",
  "source": "orders-api",
  "type": "com.dapr.event.sent",
  "time": "2026-09-11T06:23:21Z",
  "traceid": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01",
  "traceparent": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01",
  "tracestate": ""
}

Your order is in there, one level down, under data. Going the other way costs nothing: the Function publishes a bare order body with the SDK, and "the subscribing Dapr process still wraps these raw messages in a CloudEvent before delivering them to the subscribing application", so the handler receives an envelope whose data is the body the Function sent. The direction that breaks is the one you will deploy second, where the Function's existing trigger binds to Order:

[Function(nameof(ProcessOrder))]
public Task ProcessOrder(
    [ServiceBusTrigger("orders", "orders-fn", Connection = "ServiceBus")] Order order)
{
    // order.OrderId    is null
    // order.CustomerId is null
    // order.Total      is 0
    logger.LogInformation("Processing order {OrderId}", order.OrderId);
    return Task.CompletedTask;
}

The binding deserializes the message body, and the body is now the envelope. topic and specversion are not properties of Order, and OrderId is not a property of a CloudEvent, so every field lands on its default, with no exception thrown, because the JSON is valid and the type is populated with nothing. The message completes, the delivery count never increments, nothing dead-letters, and DeadletteredMessages stays flat while every order silently evaporates.

There is a documented switch that makes the problem go away, and it costs more than it looks. Setting rawPayload: "true" on publish stops Dapr wrapping the message, but the same page states that disabling CloudEvents "disables support for tracing, event deduplication per messageId, content-type metadata, and any other features built using the CloudEvent schema", and the subscribe side is then "always base64 encoded with content type application/octet-stream". Trading tracing away is a bad deal in a migration whose whole argument is that a single trace spans both sides of it.

So teach the Function about the envelope rather than teaching Dapr to stop producing it. The shim is a record and one property access:

public sealed record CloudEvent<T>(
    [property: JsonPropertyName("data")] T Data,
    [property: JsonPropertyName("id")] string Id,
    [property: JsonPropertyName("type")] string Type,
    [property: JsonPropertyName("traceparent")] string? TraceParent);

[Function(nameof(ProcessOrder))]
public Task ProcessOrder(
    [ServiceBusTrigger("orders", "orders-fn", Connection = "ServiceBus")] CloudEvent<Order> message)
{
    var order = message.Data;
    logger.LogInformation("Processing order {OrderId} from event {EventId}", order.OrderId, message.Id);
    return Task.CompletedTask;
}

That keeps tracing, keeps messageId deduplication, and hands the Function the traceparent it needs to join the Container App's trace. It also survives the Function App being turned off in phase 4. Reach for rawPayload only when the Function App is genuinely untouchable, and write down what you gave up.

How many requests one call actually makes

Phase 3 is about to put production traffic on the new runtime, so this is the last comfortable moment to count what one request there actually costs. The same orders-api that now shares that subscription also asks inventory-api whether an order can be filled, and the call site is one call:

using var response = await inventory.PostAsJsonAsync(
    "/stock/check", new StockCheckRequest(order.OrderId, order.Lines), cancellationToken);

inventory is an HttpClient built by DaprClient.CreateInvokeHttpClient("inventory-api") with AddStandardResilienceHandler() on it, which is what the templates put there. That client, the /stock/check endpoint it calls and the service defaults wrapping both are in DaprAspireDemo in azure-functions-samples, the same projects Part 4 built. Count the HTTP requests inventory-api receives for that one call while it is having a bad minute:

Layer 1  AddStandardResilienceHandler   1 initial + 3 retries = 4 attempts
Layer 2  Dapr service invocation        1 initial + 3 retries = 4 attempts, per layer-1 attempt
                                                                ------------
         delivered to inventory-api                 4 x 4    =   16

Layer 0  Service Bus redelivery, MaxDeliveryCount 10
         before one message dead-letters           10 x 16   =  160

Layer 1 is Microsoft.Extensions.Http.Resilience. AddStandardResilienceHandler() chains five strategies from outermost to innermost: a rate limiter, a 30 second total timeout, a retry strategy with max retries 3 and exponential backoff with jitter on a 2 second base delay, a circuit breaker, and a 10 second per-attempt timeout. It retries on HTTP 500 and above, 408 and 429, and by default it retries every HTTP method including POST. Three retries means four attempts leave your process.

Layer 2 is the sidecar those four attempts pass through. Dapr's built-in service-invocation retries use a 1 second backoff interval with a threshold of 3, so each attempt becomes four sidecar-to-sidecar requests, on a number nobody wrote down in your repository. The version repeated in write-ups goes one further and is wrong: attach your own Dapr retry policy and you would expect 4 x 4 x 4 = 64, but "a user defined retry policy replaces default retries. Targets rely solely on the applied policy." The number stays 16.

What makes sixteen a migration number rather than a Dapr number is the message underneath it. orders-api is woken up by a message on the orders-fn subscription the Function App still reads, and throwing out of the handler abandons that message for redelivery up to MaxDeliveryCount, which defaults to 10 and replays the whole fan-out each time. That last multiplication is arithmetic on documented defaults rather than a number anyone publishes, so read it as the shape of the problem. MaxDeliveryCount belongs to the subscription rather than the consumer. That is why the component field of the same name was inert back in the topic section, and why the count only bites during coexistence: every delivery the Container App burns on a failure is one the Function App does not get. Two runtimes, one budget, sized when only one existed.

The timing interaction is worse than the multiplication, and it is what turns a slow dependency into an outage. Take inventory-api degraded to 3 seconds per request, not down, just slow:

sidecar budget for ONE layer-1 attempt   4 x 3 s + 3 backoffs x 1 s  = 15 s
layer-1 per-attempt timeout                                          = 10 s

The outer timeout is shorter than the inner retry budget. Polly cancels every attempt at 10 seconds, waits its backoff, and starts a fresh one, so the sidecar's sequence never runs to the end, and the 30 second total timeout leaves room for roughly three of those attempts. The caller pushes seven or eight real requests into an already-degraded dependency and returns a timeout, having succeeded zero times. The defaults in that chain are documented; the interaction between them is reasoning, not a measured run. The rule: the outermost timeout must exceed the innermost retry budget, or the inner layer never finishes and the outer layer only adds load. Either raise the attempt timeout above (maxRetries + 1) * expected_latency + total_backoff, or take a layer out.

The instinct at this point is to port the Function App's retry configuration across. Functions runtime retry policies ([FixedDelayRetry], [ExponentialBackoffRetry]) support four trigger types: Cosmos DB, Event Hubs, Kafka and Timer. Service Bus is not one of them, so a Service Bus trigger's retry behaviour has always come from the broker's delivery count, and clientRetryOptions in host.json looks like the answer and is not: those settings "only apply to interactions with the Service Bus service. They don't affect retries of function executions." Where a policy does carry across, on a Timer or Event Hubs trigger you move later, the retry count "is stored in the memory of the instance", so "the maximum retry count is a best effort". Either way a like-for-like port is a category error: "we retry ten times" meant one thing when a broker counted the deliveries, and on a chained in-process pipeline sitting on a sidecar it means 10 x 16.

The one lever you own everywhere, including on Container Apps, is layer 1:

httpClientBuilder.AddStandardResilienceHandler(options =>
{
    options.Retry.DisableForUnsafeHttpMethods();   // POST, PATCH, PUT, DELETE, CONNECT
});

DisableFor(params HttpMethod[]) is the targeted form if you want to keep retrying one unsafe method you know is idempotent. The Dapr layer below retries a POST regardless of what you think about idempotency, and the app layer is the only place the multiplication can be stopped.

resiliency.yaml is the right answer and it does not reach your deployment

Dapr has a proper answer, and it is one resource. policies names the strategies, targets binds them to an app, a component or an actor type, and both blocks are required:

apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: orders-resiliency
version: v1alpha1
scopes:
  - orders-api
spec:
  policies:
    timeouts:
      inventoryTimeout: 5s
    retries:
      inventoryRetry:
        policy: exponential
        maxInterval: 10s
        maxRetries: 2
        matching:
          httpStatusCodes: "429,500-599"
    circuitBreakers:
      inventoryBreaker:
        maxRequests: 1
        timeout: 30s
        trip: consecutiveFailures > 5
  targets:
    apps:
      inventory-api:
        timeout: inventoryTimeout
        retry: inventoryRetry
        circuitBreaker: inventoryBreaker

Two caveats before you copy it. The apiVersion is dapr.io/v1alpha1 and the resource is not on Dapr's alpha/beta API maturity page, so do not plan around it as a stable surface. And "currently, resiliency policies are not supported for service invocation via gRPC", which, since sidecar-to-sidecar traffic is gRPC whatever the caller speaks, means your app must call its sidecar over HTTP rather than that the wire must be HTTP end to end.

The larger caveat is that on Azure Container Apps the block above does not apply: the Dapr Resiliency resource is not exposed there. ACA has two unrelated preview features instead, Dapr component resiliency, which covers outbound and inbound hops and has no targets.apps concept, and service discovery resiliency, whose documentation states that "you can't apply resiliency policies to requests made by using the Dapr Service Invocation API".

That closure makes the much-repeated claim "Dapr's built-in retries cannot be disabled" true in a narrower way than it is usually written. The global override is genuinely capped: overrides "are not applied to specific targets", and "you cannot override with lesser values than the provided default value, or completely remove default retries", so maxRetries: 0 under DaprBuiltInServiceRetries does nothing. A per-target policy replaces the built-in rather than lowering it, and maxRetries: 0 there does take, which a maintainer thread (dapr/dapr#9625) documents as the supported workaround. On ACA that per-target route is the one thing you cannot reach: resiliency "is not enabled for Dapr service invocation. And for components it is only enabled if you explicitly create a component resiliency policy" (microsoft/azure-container-apps#585, February 2024).

So pick one layer to own retries and neuter the others. On Container Apps the choice is made for you: layer 2 is not yours, so layer 1 is where the policy lives.

Own layer 1 deliberately. AddStandardResilienceHandler is per-HttpClient, so leave it off the client that goes through the sidecar and keep it on the clients that call external HTTP APIs directly. If you keep it on the sidecar client, apply DisableForUnsafeHttpMethods() and raise AttemptTimeout above the sidecar's budget so an attempt can complete.
Configure the layer ACA does give you. Component resiliency covers the sidecar-to-component and sidecar-to-app hops that pub/sub and state run over:

   az containerapp env dapr-component resiliency create \
     --name orders-pubsub-resiliency \
     --dapr-component-name orders-pubsub \
     --environment aca-env --resource-group rg-orders \
     --out-http-retries 2 --out-http-delay 500 --out-http-interval 5000

The CLI defaults if you leave those out are 3 retries, a 1000 ms delay and a 10000 ms interval, and responseTimeoutInSeconds "includes all retries". Applying a policy requires restarting your Dapr applications, so it is a deployment step, not a hot config change.

Turn the message-level multiplier down while both runtimes share a subscription. Judgement rather than documentation: MaxDeliveryCount at 10 was sized for a single consumer whose only retry mechanism was redelivery, and with a pipeline and a sidecar underneath it, a lower count dead-letters faster and costs the shared dependency less.

Write the number down for one endpoint before you cut it over. Sixteen is fine if you decided on sixteen.

Phase 3: one endpoint at a time

Everything still sitting on the /* route has live clients on it, so phase 3 is a loop rather than a deploy. One path leaves the catch-all per iteration, and every point inside the iteration has to be somewhere you would be willing to stop.

The two levers stack rather than blend: the route decides which origin group serves a path, and the weights inside that group decide which runtime serves a request. So one path moves in two motions, carved out of the catch-all onto its own route pointing at a group that holds both origins, then reweighted. /api/customers/* is the one moving here; /api/reports/* is still last.

// Step 1. One origin group for the path, holding both runtimes.
// customersOriginGroup is name: 'customers-origin-group', and its
// loadBalancingSettings and healthProbeSettings are the same block as
// orders-origin-group in phase 2, additionalLatencyInMilliseconds: 500 included.
// That line is the whole reason the weights below do anything: see the aside.

resource customersFunctionOrigin 'Microsoft.Cdn/profiles/originGroups/origins@2021-06-01' = {
  name: 'customers-functions'
  parent: customersOriginGroup
  properties: {
    hostName: 'orders-func.azurewebsites.net'
    originHostHeader: 'orders-func.azurewebsites.net'
    httpsPort: 443
    priority: 1          // both origins share a priority, deliberately
    weight: 75
    enabledState: 'Enabled'
  }
}

resource customersContainerOrigin 'Microsoft.Cdn/profiles/originGroups/origins@2021-06-01' = {
  name: 'customers-containerapp'
  parent: customersOriginGroup
  properties: {
    hostName: 'orders-api.orangeplant-77e5875b.westeurope.azurecontainerapps.io'
    originHostHeader: 'orders-api.orangeplant-77e5875b.westeurope.azurecontainerapps.io'
    httpsPort: 443
    priority: 1
    weight: 25
    enabledState: 'Disabled'   // deploy dark; step 3 is the only step that moves traffic
  }
}

The route is the phase 2 shape with different values: patternsToMatch: [ '/api/customers', '/api/customers/*' ], pointing at customersOriginGroup. That is the whole configuration, and the order you apply it in is what makes it reversible:

Deploy the group with the new origin disabled. The Function App is the only enabled member, so the group behaves exactly like the catch-all did.
Deploy the route. The path is carved out of /* and 100% of it still lands on the Function App. Nothing a client can observe changes, which is why it is worth deploying on its own and leaving alone for a day.
Enable the Container App origin, at the low ratio you already deployed. First contact between new-runtime code and a production request on this path.
Shift weight and watch. Raise the Container App, lower the Function App, repeat.
Revert by disabling the origin, not by undoing the deployment. "If any issues arise with the new origin, disable it to route all traffic back to the old origin", and "when an origin is disabled, both routing and health probes to the origin are also disabled".

That is Microsoft's own blue/green procedure with a migration in the middle of it, and Microsoft lists the use case by name under weighted routing: "Application migration to Azure: ... Adjust weights to prefer new origins ... then disable and remove less preferred origins."

Reversibility is worth costing before you need it, because the five levers are not equally cheap:

Every row is config only: no redeploy, no pipeline, and any of them can be done by whoever is holding the pager. What none of them is, is a keystroke with an immediate effect. The first four are control-plane changes to Front Door, and a control-plane change reaches the points of presence on Front Door's schedule rather than yours. The last row is the exception, and it exists only if API Management is already in the path. Measure the delay in your own profile, and write the runbook with a window in it rather than a promise.

Which path goes next is judgement rather than documentation. Move the paths that are neither cacheable nor session-affine first, because both change behaviour when the origin behind a path changes, and neither is something you want to debug on the same afternoon as a runtime change. That is a second reason /api/reports/* goes last, on top of the profile-wide timeout argument from phase 1.

One more weighting mechanism shows up in this phase, and it is not on this axis at all. Container Apps splits traffic between revisions:

az containerapp ingress traffic set -n orders-api -g rg-orders \
  --revision-weight orders-api--knowngood=80 orders-api--newrevision=20

Weights there must total 100, it needs multiple revision mode, and every name in that command is a revision of orders-api: no argument accepts a Function App. Front Door weights answer "old runtime or new runtime"; revision weights answer "old version or new version of the service that already moved". The trick worth stealing early is a label on a revision carrying weight 0, a stable URL that takes no production traffic.

Your 90/10 split is a 100/0 split

Set 75 and 25 on the two origins above, deploy, and watch every single request land on one of them. Nothing fails. The portal shows the weights you typed, on the origins you set them on, in the group the route points at, and the traffic ignores all of it.

Weights are the last of four filters, and the third one has already discarded an origin by the time weighting runs. Front Door picks an origin in this order:

Available. Enabled origins that are passing their health probe.
Priority. Only origins at the best priority value survive.
Latency. Only origins within the latency sensitivity range of the fastest one survive.
Weight. Round robin across whatever is left, in your ratio.

Stage 3 reads additionalLatencyInMilliseconds on the origin group, and it defaults to 0. Verbatim, from the routing methods page: "By default, the latency sensitivity property is set to 0 ms. With this setting, requests are always forwarded to the fastest available origins. Weights on the origins only take effect if two origins have the same network latency." Two origins having the same measured network latency is the condition, and a Function App and a Container App do not meet it, ever. Whichever one measures faster from a given point of presence is the only origin still standing after stage 3, and stage 4 round-robins across a set of one. Your 90/10 deploys as 100/0.

The same pipeline has a second way to do this to you, one stage earlier. If the Function App origin is priority 1 and the Container App origin is priority 2, the Container App gets nothing at all while the Function App is healthy, whatever the weights say. Both origins have to carry the same priority value, which is why the Bicep above sets priority: 1 on both and says so in a comment.

And the honest caveat for anyone running an internal API at a handful of requests per second: "For customers with very low RPS, due to the distributed nature of Azure Front Door points of presence (POPs) and machines, Azure Front Door can't guarantee that the weights you configure are strictly followed and the load balancing might appear skewed." A clean 90/10 is not something to promise a change board.

The fix is one property on the origin group, and it is the blue/green guide's value:

// before: the default. 90/10 deploys as 100/0.
loadBalancingSettings: {
  sampleSize: 4
  successfulSamplesRequired: 3
  additionalLatencyInMilliseconds: 0
}

// after: both origins stay eligible at stage 3, and 90/10 means 90/10.
loadBalancingSettings: {
  sampleSize: 4
  successfulSamplesRequired: 3
  additionalLatencyInMilliseconds: 500
}

Confirming it took effect is two checks, not one, because the property being deployed and the split being live are different claims. Read the value back off the deployed resource:

az afd origin-group show -g rg-orders --profile-name afd-orders \
  --origin-group-name customers-origin-group \
  --query loadBalancingSettings.additionalLatencyInMilliseconds

Then count requests at the two origins over a window and compare the ratio to the one you configured, rather than assuming it. Both apps log their own hostname, and Front Door's probes announce themselves with the Edge Health Probe user agent, so exclude those before you count or a low-traffic path will look busier on both sides than it is. Expect the ratio to be roughly right rather than exactly right, and treat "roughly 75/25" as the property working and "everything on one origin" as it not working.

Caching and affinity change what a cutover means

A path you cut over on Tuesday is still answering from the Function App on Friday, and every piece of routing config is correct.

Only GET requests are cacheable, so POST /api/orders was never at risk and GET /api/customers/{id} is, from the moment anyone enables caching on the route. The part that outlives the cutover is what Front Door does when your origin is quiet about it: if the response carries no Cache-Control, Front Door invents a TTL "between one and three days" at random. A migration-era endpoint that forgot its cache headers can serve the old runtime's response for days after the origin behind it changed, and none of the five reversibility levers touches it: they all move traffic, not what is already cached.

Caching also rewrites the request the origin sees: Content-Length, Transfer-Encoding, Accept, Accept-Charset, Accept-Language and Vary are not forwarded, so an API that content-negotiates behaves differently on the same code. And Set-Cookie is stripped from cacheable responses, which is how caching breaks session affinity without either feature being touched: Front Door's own affinity is not established at all if the origin sends a cacheable response.

Affinity has a harder constraint underneath it, a dead end rather than a trap. Container Apps sticky sessions (ingress.stickySessions.affinity: "sticky") are supported in single revision mode only, which is exactly what revision traffic splitting cannot run in, so the smoke-test lever from earlier in this phase is not available on an affine path.

So: leave caching off for /api/* for the whole migration, which is what omitting --cache-configuration already did in phase 2, and prove it rather than believe it. Curl a migrated GET and read the X-Cache header: CONFIG_NOCACHE is caching being off, TCP_HIT is a response you are no longer choosing the origin for. Move session-affine paths by path rather than by weight, because a weighted origin group is free to send the second request of a session to the other runtime, and if a path needs sticky sessions, pick between affinity and revision splitting before the cutover window starts.

Going private takes the whole environment

The two origin types stop being interchangeable at network lockdown:

The load-bearing row is the target sub-resource. The Container Apps private endpoint attaches to managedEnvironments, so enabling it takes the entire environment private, every app in it, not the one path you migrated last week: you plan the environment, not the app, and you plan it before the first cutover. The row underneath costs you a pattern you probably already have on the Function App, since the AzureFrontDoor.Backend service tag paired with an x-azure-fdid header match has no Container Apps ingress equivalent. Origin lockdown on the new runtime is either Private Link or an X-Azure-FDID check in your own middleware.

What you do when it goes wrong

Two hours after /api/orders/* starts serving from the Container App, the 5xx rate on it triples. You have two questions and only one of them has a good answer: how fast you can put the traffic back, and what putting it back leaves behind. The second one is the expensive one.

The fast answer depends on which surface owns the routing. With only Front Door in front of the two runtimes, the levers are the ones phase 3 costed. If API Management sits between Front Door and your backends, which is the steady-state shape Part 6 builds, the fastest lever in the migration is a policy edit, because it takes effect at the gateway rather than at the edge:

<policies>
  <inbound>
    <base />
    <choose>
      <when condition="@(context.Request.Url.Path.StartsWith("/api/orders"))">
        <choose>
          <!-- {{orders-backend}} is a named value: "containerapp" or "functions".
               One edit in the portal or one az apim nv update, no redeploy. -->
          <when condition="@("{{orders-backend}}" == "containerapp")">
            <set-backend-service backend-id="orders-containerapp" />
          </when>
          <otherwise>
            <set-backend-service backend-id="orders-functions" />
          </otherwise>
        </choose>
      </when>
      <otherwise>
        <set-backend-service backend-id="orders-functions" />
      </otherwise>
    </choose>
    <!-- Type is Single or Pool. This is how you prove from logs which runtime
         served request X, rather than inferring it from response latency. -->
    <set-header name="X-Backend-Type" exists-action="override">
      <value>@(context.Backend?.Type ?? "n/a")</value>
    </set-header>
  </inbound>
  <backend><base /></backend>
  <outbound><base /></outbound>
</policies>

context.Backend exposes Id, Type and AzureRegion, so the X-Backend-Type header turns "which runtime served this request" from a guess into a field you can group by. During a mixed week that header is the difference between an incident review and an argument.

There is one way to get that wrong that you will not enjoy at 3am. set-backend-service takes either base-url or backend-id, and the two do not mix across scopes: if a base policy sets the backend with backend-id, "it can only be overridden with a policy using the backend-id attribute, not the base-url attribute." Pick backend-id everywhere, including the rollback branch.

And the backend pool is the one place in this migration where a Function App and a Container App can be given real percentages against each other: up to 30 backends with round-robin, weighted or priority-based balancing and per-backend circuit breakers, with no latency-sensitivity setting standing between the weights you typed and the traffic you get. The docs are honest that "because of the distributed nature of the API Management architecture, backend load balancing is approximate", which is a much smaller caveat than the one phase 3 ran into.

The runbook is mostly about the writes

Write this before the cutover, one per migrated endpoint, and keep it to six lines:

Route revert. The named value edit above, or az afd route update --route-name orders-route --enabled-state Disabled, with the resource group already filled in.
Consumer re-enable. The app setting that disabled the Function's orders-fn trigger (phase 4 names it), and the value to put it back to.
Stores the new path wrote to, by name: the Dapr state store component, its container, and anything the handler touched directly.
The reconciliation query for each. Counts and checksums on both sides rather than a row-by-row diff, written out, tested, and dated.
Dead-letter disposition. Drain, dead-letter or accept, decided now rather than during the incident.
The delivery-count assumption. What you believe MaxDeliveryCount is on orders-fn, so the person reverting knows what budget they are inheriting.

Lines 1 and 2 take minutes to write. Lines 3 to 5 are why the document exists, because reverting a route is a control-plane change and a control-plane change has no opinion about data. Three things the new path did while it was live survive the revert intact.

State it committed. The write through the sidecar succeeded, and flipping the named value does not un-write it. Dapr does not store your object as your object: it stores an envelope, your document nested under a value property and the application ID prefixed onto the key. So the old Function path, still reading that container with the Cosmos SDK, finds documents under IDs it never wrote, in a shape it does not deserialize. Do not point both runtimes at one state store during the migration; give each its own and keep them in step through the topic.

Messages it published. An order event the Container App put on the orders topic is in the broker. The HTTP revert does not recall it, and the subscriber will process it minutes after you believe the new path is off. This is the line teams skip, and the one that produces the second incident an hour after the first closed.

Delivery attempts it burned. If the Container App consumed a message from orders-fn and failed, the count is already incremented broker-side, and "the delivery count is increased when a message is received in PeekLock mode and didn't complete the message before the message lock expired." Re-enable the Function consumer and it inherits a partially spent budget on messages it has never seen. The counter does not reset because you changed your mind.

Re-enabling is not the mirror image of disabling either. It puts a cold app back on a subscription that accumulated backlog for the duration of the failed experiment, with no way to ask whether it came back healthy, which is the next problem.

The health signal is not symmetric

You will want a dashboard with the old runtime on the left and the new one on the right. The left column does not exist.

App Service Health check is the feature you would use, and for the app most likely to be strangled it is not available: "Health check isn't an option for the Flex Consumption and Consumption plans." Where it does exist, it needs an anonymous HTTP trigger answering on the configured path with a 200, which a queue-triggered Function App does not have, and even then a failure only surfaces once ten consecutive one-minute pings cross the load-balancing threshold.

The Container App side is the opposite problem: startup, readiness and liveness probes plus the sidecar's own dapr.appHealth, the four-opinion pile-up from phase 1, with readiness at its default failure threshold of 48 on a 5-second period giving a signal roughly four minutes wide. Four health opinions and a four-minute signal on one side, against nothing and a ten-minute signal on the other.

Front Door does not close that gap, and for the shape phase 2 built it is documented not to: "If you have only a single origin, Azure Front Door always routes traffic to that origin even if its health probe reports an unhealthy status." Microsoft's guidance is to disable probes when a group has one origin, so every probe in orders-origin-group is telemetry rather than protection. Probes are not free either: volume is per point of presence, so a freshly cut-over Container App running minReplicas: 0 is kept warm, and billed, by probe traffic alone.

So build for one side. The rollback trigger is the new runtime's error rate and readiness state, grouped by the X-Backend-Type header, plus the reconciliation queries from line 4 of the runbook. Do not wait for a comparison that the old runtime cannot participate in.

There is no feature flag in API Management

The policy above says {{orders-backend}} rather than anything that looks like a feature flag, and that is not a stylistic choice. API Management has no feature-flag primitive. What it has is a named value (plain, secret, or a Key Vault reference) read as {{name}} inside a choose, and a backend pool with per-backend weights. Everything written as "check the feature flag in your APIM policy" is one of those two with a nicer noun on it.

If the flag genuinely has to live in Azure App Configuration, price the gateway version before you build it. A bare send-request per inbound request is a full HTTP round trip in the inbound pipeline before the backend has been chosen, with a timeout that defaults to 60 seconds. Caching the lookup is the only workable shape, and the cache is its own project: cache-store-value is asynchronous, so every expiry lets a burst through to App Configuration at once; the built-in cache is volatile and per-region, so a flag flip lands region by region; it does not exist at all in the Consumption tier; and cache-lookup-value "is not supported inside a policy fragment", which closes the obvious refactor.

Then the number that decides the design. In-app, Microsoft.FeatureManagement with UseFeatureFlags() refreshes App Configuration feature flags on a 30-second default interval, with no cache to design, no tier requirement and no regional skew.

What to do instead: use the named value from the first policy for the routing switch, because it is one edit that applies at the gateway, with no cache in front of it. Reach for the App Configuration lookup only when the rollback genuinely has to beat 30 seconds, and be able to say why it does. One security sentence, because migrations hand out policy-edit rights broadly: anyone holding Microsoft.ApiManagement/service/apis/policies/write can use authentication-managed-identity to authenticate as the service identity and take the token with them.

The cross-runtime trace is two settings away from existing

The part everyone worries about works. Dapr uses W3C trace context on service invocation and pub/sub: "When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain", and "Dapr always propagates trace spans to an application." The traceparent in the CloudEvent envelope earlier is that contract, in a message. Dapr's docs then warn that you must carry the context from your app's inbound request onto its outbound calls yourself, with no SDK helpers for it. Do not repeat that warning unchanged to a .NET audience: System.Diagnostics.Activity plus the HttpClient diagnostics handler already injects traceparent from the ambient activity, and W3C TraceContext has been the default ID format since .NET 5.

What actually breaks, in both directions, breaks by omission. Open Application Insights during a mixed week with nothing configured and the Function App contributes no host telemetry, because the isolated worker emits OpenTelemetry only when host.json says so, while the sidecar hop between the runtimes is missing because Dapr's documented default sample rate is 0.0001, one span in ten thousand. Two islands of application-level logging, with a hole where the interesting hop was.

The obvious fix is closed on Container Apps. Sampling lives at spec.tracing.samplingRate in a Dapr Configuration resource, and the first entry in ACA's list of unsupported Dapr capabilities is "Dapr Configuration spec: Any capabilities that require use of the Dapr configuration spec." Neither doc set says whether ACA's managed sidecar keeps the 0.0001 default or sets its own rate once export is enabled, so turn export on and count spans rather than trusting a number from anywhere, including this article.

Two settings, one per runtime. On the Function App:

{
  "version": "2.0",
  "telemetryMode": "OpenTelemetry"
}

with APPLICATIONINSIGHTS_CONNECTION_STRING set alongside it. On the Container Apps environment, the property that replaces the sampling knob you cannot reach:

properties: {
  daprAIConnectionString: appInsights.properties.ConnectionString
  openTelemetryConfiguration: {
    tracesConfiguration: {
      includeDapr: true                 // exports the sidecar's own spans
      destinations: [ 'appInsights' ]
    }
  }
}

After both, one request that enters through the Function App and continues into orders-api is one operation_Id, because the W3C to Application Insights mapping is fixed: trace-id becomes operation_Id and the span's parent-id becomes the id of the request or dependency. The Application Map draws both runtimes and the sidecar hop between them.

Set telemetryMode on the Function App, set includeDapr on the environment, and point both runtimes at the same Application Insights resource. That last one is not a detail. Two workspaces give you a trace that is technically continuous and practically invisible, because Transaction Diagnostics and the Application Map join on data they can both see. A migration whose main argument is that one trace spans both runtimes should be able to show the trace.

Phase 4: turning the Function App off

Phase 4 has no new code in it, and the only question it asks is how you know it is safe to stop. One endpoint is done when nothing routes to its Function implementation and nothing triggers it, which is what the end of each pass through phase 3 looks like. The app is done when the last endpoint is done, and that is not the same as the app being gone. Disable, do not delete.

The route goes first, because it is the only thing here that reverses without touching state. For every path you already moved, the Front Door work finished in phase 3; what is left is the /* catch-all, still pointing at the Function App origin and still catching everything nobody has looked at in six weeks. Do it in two moves. Set enabledState: 'Disabled' on the Function App origin, which stops routing and probes together and leaves the route there to re-enable. Then, only after a full business cycle passes without anyone noticing, remove the /* route. That second move changes what an unrecognised path does, from quietly served by the old runtime to erroring at the edge, which is both the point of it and the reason it goes last.

The consumer goes next, one function at a time. The message-triggered half of the app has nothing to do with Front Door and has its own switch, an app setting named after the function: "You can disable a function in place by creating an app setting in the format AzureWebJobs.<FUNCTION_NAME>.Disabled set to true." One function stops and the rest of the app keeps serving.

az functionapp config appsettings set \
  --name orders-func --resource-group rg-orders \
  --settings AzureWebJobs.ProcessOrder.Disabled=true

Two caveats travel with that command. The setting is not honoured for Functions running on Container Apps, so if you already moved the Function App into a container this lever does not exist for you. And "changing application settings causes your function app to restart by default across all hosting plans", so disabling one function restarts every other function in the app. Schedule it as a restart, not as a config tweak. Then wait longer than feels reasonable: the grace period for draining in-flight invocations "can extend up to 10 minutes for Consumption plan apps and up to 60 minutes for Flex Consumption and Premium plan apps". Budget an hour on Premium, and check that the Service Bus extension is version 4.2.0 or later before you rely on the drain rather than after.

The app itself you stop and keep. Stop it if you want the compute line gone, and leave the code deployed, because reverting this step is AzureWebJobs.ProcessOrder.Disabled back to false: the shortest rollback anywhere in this migration, though not a free one, since it costs the same restart and puts a cold Function App back onto whatever backlog built up. Delete the app and that rollback becomes a redeploy from a repository whose pipeline you disabled two weeks ago, at the exact moment you are least interested in fixing a build. Keep it deployed and disabled for one full retention window, long enough that a bug report about an order processed on the old path can still be answered by the app that processed it. The CloudEvent<T> shim from the topic section stays in that code for the same reason: the Container App goes on publishing envelopes whether or not anything is subscribed.

One deletion needs planning, and it does not look like it belongs to the Function App at all. The topic and the orders-fn subscription were pre-created in the Function App's bicep, and the Container App now consumes from them, so deleting that resource group takes away the subscription the migrated app is reading from. Move those two resources into the template that owns the new runtime before you delete anything on the old side, and treat the move as its own change with its own rollback, because a Service Bus subscription that gets deleted and recreated does not bring its messages with it.

ActiveMessageCount zero does not mean the old consumer is gone

Disabling the consumer ends in "wait, then confirm", and "confirm" is the step people replace with a metric. The metric they reach for cannot answer the question, twice over.

The first reason is shape. The Azure Monitor ActiveMessages metric carries the dimension EntityName and nothing else, and on a topic the topic-level figure is not a backlog at all: "the active message count on the topic itself is 0, as those messages have been successfully forwarded to the subscription." Per-subscription backlog is not available from Azure Monitor metrics.

The second survives getting the number right. An active message count counts messages nobody is currently holding, and a message received in PeekLock and not yet completed is not active, so orders-fn can read zero while a Function instance is halfway through the last three orders it will ever process. Zero proves the subscription is empty right now, and says nothing about whether anything is still attached to it.

Take three signals instead, in increasing order of how much they prove:

Function invocation count in Application Insights. Necessary, not sufficient: zero invocations is also what a broken trigger looks like, so it tells you when to keep waiting, not when to proceed.
Per-subscription CountDetails, from the administration API rather than from metrics. Get-AzServiceBusSubscription ... | Select CountDetails returns ActiveMessageCount, DeadLetterMessageCount, ScheduledMessageCount and the two transfer counts; in .NET, SubscriptionRuntimeProperties. Poll it gently, because "the acquisition of the message counters is an expensive operation inside the message broker, and executing it frequently directly and adversely impacts the entity performance".
ActiveConnections on the namespace or entity. The receiver's AMQP link disappearing is the closest thing to proof that the old consumer let go. It will not drop to zero, because the Container App holds links of its own, so write down the count while both consumers are attached, or the drop you are waiting for has nothing to be a drop from.

Conclusion

By path. The route is the lever you reach for first, and the reason is what the revert costs: taking /api/orders/* back off the Container App means deleting one string from patternsToMatch and letting the catch-all catch it again, with a blast radius of exactly one path pattern and no arithmetic to get wrong at three in the morning. Weights are the second lever rather than the competing one. They belong inside a single origin group, and they mean nothing until additionalLatencyInMilliseconds stops handing every request to whichever origin measured fastest. Use them when one path is too large to move in a single piece, and not to avoid making the per-path decision.

None of that is the hard part. The hard part is that almost nothing in this migration fails loudly: a weight that is ignored, a second subscription that appears under your app ID because you left a field blank, a message that deserializes into an object with every field at its default and nothing thrown. Nothing raises and nothing pages, and all of it looks like working software right up until a customer counts their confirmation emails. Most of the work in the four phases is not moving traffic. It is making those failures audible while you are still watching for them.

Part 6 takes the same three components (Front Door, API Management, Container Apps) and asks what they look like when they are not a transition: the steady-state architecture, sized and secured for the shape you land on rather than the one you pass through.

.NET Aspire: Orchestrating Cloud-Native Apps

Martin Oehlert — Fri, 04 Sep 2026 05:48:18 +0000

Part 3 started two services and two sidecars from a Dapr multi-app run file, one dapr run per app, and that file never leaves your laptop. Aspire replaces it with a C# program that also never leaves your laptop: aspire.dev states that "The AppHost isn't a production runtime. It's a development-time orchestration tool that simplifies the process of running and debugging your application locally." Both artifacts are deleted at the deployment boundary. So the question is what the C# one does in the hours before that boundary that the run file does not. The four projects that answer it are in DaprAspireDemo in azure-functions-samples: an AppHost, a ServiceDefaults library, and the two services, building clean at 0 warnings and 0 errors.

What the AppHost actually owns

The local loop is still one command, and that command is the only line of the developer workflow that changed:

dapr run -f .    # Part 3: one run file, one dapr run per app
aspire run       # Part 4: one C# program, the same four processes

The second one came back with four resources in the dashboard, all Running. Two are the services, listed against their .csproj. The other two are order-service-dapr-cli and inventory-service-dapr-cli, executable resources whose Source column holds the entire dapr run command line the integration assembled: the app ID, the app port, the three sidecar ports, the app channel address, the app protocol, and whichever components flag it chose to pass. That column is the best debugging surface in this whole setup. "Which component folder did this sidecar actually load" becomes one row in a browser instead of a log grep, and the in-memory state store further down is nothing but an exercise in reading it.

The run file had no equivalent. dapr run -f . gives you four processes interleaving their stdout into one terminal and a dapr stop -f . to take the set back down. Under the AppHost each sidecar is a resource in its own right: its own console log pane, its own recorded start time, its own Stop and Restart actions, and its own OTLP stream arriving in the same dashboard as the app it sits beside. The sidecar stops being something you remember to start and becomes something you can point at.

Startup order is the second thing the AppHost owns, and it is where people arriving from Docker Compose lose an afternoon. Two methods look interchangeable and are not. WithReference(x) is wiring: it injects the configuration a consumer needs in order to find x, and the documentation is explicit that it says nothing whatsoever about who starts first. WaitFor(x) is ordering: it holds a resource back until x is running and its registered health checks report healthy. Read WithReference as compose's depends_on and you have written down a dependency the runtime will not honour, which surfaces as a startup race rather than as an error message. aspire.dev's own compose migration guide names this as the top gotcha for people coming from compose, and it earns the title.

One piece of housekeeping before any of this reproduces on your machine. Ask a box set up today which Aspire version it is on and you get three answers: one from the CLI, one from the installed project templates, and one from the packages that actually resolve at build time. aspire update reconciles them. Everything below is pinned to Aspire.AppHost.Sdk 13.5.3 and CommunityToolkit.Aspire.Hosting.Dapr 13.0.0, and version-qualifying your own transcripts is worth the ten seconds when a stack moves this fast.

It is not docker-compose, and it does not replace it

The reflex on first reading an AppHost.cs is that this is a compose file with C# syntax, and half of that reflex is correct. The compose file that exists purely to stand up Postgres, Redis, RabbitMQ or Azurite and then run your services does go away: AddPostgres plus AddProject covers it, and adds service discovery, health-gated startup and the dashboard on top. The part that goes away with it is the workaround where you containerise a .NET service you have no intention of shipping in a container, just so compose can see it. AddProject runs it as a host process, so breakpoints and the ordinary build loop keep working.

The other half of the reflex is wrong in a way that matters, because a compose file is a deployment artifact and the AppHost is not. Aspire does not delete your compose file. It writes one: publishing to a Docker Compose environment emits docker-compose.yaml, a .env, per-environment .env files, and a Dockerfile per resource, and aspire deploy then runs docker compose up -d --remove-orphans over the result. The C# is the model; the YAML is the output.

Three gaps matter before you delete anything, and all three come from that same migration guide rather than from community complaints. deploy.resources.limits.memory and cpus are documented as "Not supported", so you cannot reproduce a memory-starved container locally the way compose lets you. Restart policies exist at publish time only. And network isolation has no direct equivalent at all: Microsoft's own wording is that "If your Docker Compose setup relies on network isolation (for example, preventing a frontend service from directly accessing the database), Aspire doesn't provide a direct equivalent." That last one is the one to read twice if your compose file uses a private network as a dev-time security boundary, because the boundary does not survive the move and nothing tells you it is gone.

One file that starts everything

The AppHost is a console application whose whole job is to describe the other four resources. Its project file comes first, because the shape of it changed on the 13.x line and the version most tutorials show no longer matches what the template writes.

<Project Sdk="Aspire.AppHost.Sdk/13.5.3">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <IsAspireHost>true</IsAspireHost>
    <AspireUseCliBundle>true</AspireUseCliBundle>
    <UserSecretsId>dapraspiredemo-apphost-w36</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="CommunityToolkit.Aspire.Hosting.Dapr" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\DaprAspireDemo.OrderService\DaprAspireDemo.OrderService.csproj" />
    <ProjectReference Include="..\DaprAspireDemo.InventoryService\DaprAspireDemo.InventoryService.csproj" />
  </ItemGroup>

</Project>

The Aspire version lives on the Sdk attribute of the Project element, and there is no Aspire.Hosting.AppHost package reference anywhere to match it: the only PackageReference in the file is the Dapr integration. The entry point is AppHost.cs, not Program.cs. Material written against Aspire 9.x describes a different file, with a nested <Sdk Name="Aspire.AppHost.Sdk" Version="..." /> element and a hosting package reference, so a PropertyGroup copied from a 9.x post lands in a project that no longer expects it.

<AspireUseCliBundle> is the line worth stopping on. Setting it to true takes the dashboard and the orchestrator (DCP) from the installed Aspire CLI bundle rather than from NuGet, which is also what keeps Aspire.Dashboard.Sdk and the DCP packages out of the restore graph, and it needs the aspire CLI on PATH. Build the project on 13.5 without it and MSBuild answers:

warning ASPIRE010: DaprAspireDemo.AppHost is configured with AspireUseCliBundle=false. Some Aspire
features require the Aspire CLI bundle. Set AspireUseCliBundle=true to enable those features, or
suppress ASPIRE010 to continue without the bundle. See https://aka.ms/aspire/diagnostics/aspire010
for more information.

Aspire.Hosting.AppHost.props defaults the property to false and Aspire.Hosting.AppHost.targets then warns that it is false, which is an odd pairing until you reach the detail that makes it dangerous: TreatWarningsAsErrors=true does not promote ASPIRE010. It comes out of an MSBuild <Warning> task rather than out of a Roslyn diagnostic, so a CI gate that turns warnings into errors passes it straight through. Only a rule that counts warnings catches it, which means a repository with a strict-looking gate can ship an AppHost missing whatever the bundle provides and never see a red build. The property is new in 13.5; it did not exist in 13.3.5, so an older sample carries no clue that it is missing.

The one package reference is the one to get right, and the obvious search result is the wrong package. Aspire.Hosting.Dapr is dead: it stopped at 9.1.0, and every version of it is deprecated on nuget.org with the notice "We will no longer be publishing new versions of this package. We recommend using the CommunityToolkit.Aspire.Hosting.Dapr package going forward." "Deprecated" reads as "abandoned", and that is not what happened here. The move was an ownership transfer, argued out in the comments of CommunityToolkit/Aspire#349: David Fowler laid out three options, keep it in core, move it to the Toolkit, or hand it to the Dapr .NET client team, and the thread settled on the Toolkit. The reason on the record is maintainer bandwidth, that Dapr was not a priority for the core team and every PR would still need core review, not a technical fault in the package. The confusing result is that the Dapr integration docs are first-party on aspire.dev while the code is community-maintained, so the page you read and the package you install do not share a name. CommunityToolkit.Aspire.Hosting.Dapr 13.0.0 is the newest stable and depends on Aspire.Hosting >= 13.0.0, so it rides the same 13.x train as the AppHost SDK above.

That is the whole project file. Here is the program it builds:

using CommunityToolkit.Aspire.Hosting.Dapr;

var builder = DistributedApplication.CreateBuilder(args);

builder.AddDapr();

var stateStore = builder.AddDaprStateStore(
    "statestore",
    new DaprComponentOptions
    {
        LocalPath = Path.Combine(builder.AppHostDirectory, "..", "components", "statestore.yaml"),
    });

var inventory = builder.AddProject<Projects.DaprAspireDemo_InventoryService>("inventory-service")
    .WithDaprSidecar(sidecar => sidecar.WithOptions(new DaprSidecarOptions
    {
        AppId = "inventory-service",
    }));

var orders = builder.AddProject<Projects.DaprAspireDemo_OrderService>("order-service")
    .WithDaprSidecar(sidecar => sidecar
        .WithOptions(new DaprSidecarOptions
        {
            AppId = "order-service",
        })
        .WithReference(stateStore));

orders.WaitFor(inventory);

builder.Build().Run();

builder.AddDapr() is optional, which is the kind of claim worth distrusting, so I commented the line out and ran it again: both *-dapr-cli resources still started, POST /orders still returned 201, and the dapr run command line the integration assembled came back character-identical modulo ports. WithDaprSidecar() calls builder.ApplicationBuilder.AddDapr() itself, and AddDapr registers its lifecycle hook through TryAddEventingSubscriber, so the second call is a no-op. What writing the line buys you is the AddDapr(Action<DaprOptions>) overload, and DaprOptions has exactly three members: DaprPath for a Dapr CLI that is not on PATH, EnableTelemetry for turning the sidecars' dashboard telemetry off, and PublishingConfigurationAction for the publish step. Called with no callback, as here, it is the line that names the dependency: documentation rather than wiring.

The lifecycle hook it registers does not check whether Dapr is initialised. It probes for the dapr CLI binary, Homebrew prefix included, so a machine where somebody ran brew install dapr and stopped there passes the check, the AppHost starts clean, and the order-service-dapr-cli resource fails on its own some seconds later because dapr run has no daprd to launch. The tell is dapr --version answering with a CLI version and a runtime version of n/a.

AddDaprStateStore gets the second aside below, because the argument that looks optional is the one that decides whether your data outlives a process. The name is doing three jobs in the meantime: statestore is the Aspire resource name, the metadata.name in the component YAML, and the first argument order-service passes to SaveStateAsync. All three have to be the same string, and nothing checks that they are.

Both WithDaprSidecar calls take a callback rather than an options object, and the callback receives a sidecar builder carrying its own WithOptions and WithReference. DaprSidecarOptions has 32 properties; this file sets one of them, twice. What the integration fills in when you set nothing is the part to know before you start overriding things: --app-id falls back to the Aspire resource name, --app-port to the HTTP endpoint Aspire allocated for the app, and --config is not passed at all. The two AppId values above are the fallback spelled out, which is what stops a later rename of the Aspire resource from silently breaking every caller. The Source column shows you the resulting dapr run command line without a debugger.

One number in that command line will not match the one your application sees. On the run behind this article, order-service was handed DAPR_HTTP_PORT=57679 while its own daprd started with --dapr-http-port 57685. DCP puts a proxy in front of the sidecar, so the port the app talks to is not the port daprd binds. Both answer.

orders.WaitFor(inventory) is the last line and it is not a Dapr feature at all. Part 3 dealt with the same window from inside the caller: when the sidecar cannot route to an app ID it answers HTTP 500 with an ERR_DIRECT_INVOKE body, and InventoryClient translated that into a retryable failure rather than a flat 502. WaitFor addresses it from the other end, by refusing to start the caller until the callee reports healthy. They are not alternatives. The ordering is there so the startup case cannot arise; the error translation stays because the target can also fall over at three in the morning, when nothing is starting up.

Three ways to write that reference, one that compiles

The form printed in the package's own README does not build in this repository:

builder.AddProject<Projects.DaprAspireDemo_OrderService>("order-service")
    .WithDaprSidecar()
    .WithReference(stateStore);   // <- [Obsolete]; fails the build

AppHost.cs(10,1): error CS0618: 'IDistributedApplicationResourceBuilderExtensions.WithReference<TDestination>(IResourceBuilder<TDestination>, IResourceBuilder<IDaprComponentResource>)' is obsolete: 'Add reference to the sidecar resource instead of the project resource'

Build FAILED.

The error is harder to act on than it looks. The position is (10,1), the start of the whole chained expression, not the .WithReference token three lines further down, so an editor puts the squiggle on builder.AddProject and the method it is complaining about is off the highlighted line entirely. And the parameterless .WithDaprSidecar() in the middle is fine: it carries no diagnostic of its own, which sends you looking at the wrong call first. Only WithReference on the project builder is obsolete.

Under TreatWarningsAsErrors this fails the build, which is the good outcome. Without that setting it is a warning you can live with for months, and it is not cosmetic. Published both ways, the obsolete form emits a dapr.v0 resource with no components array at all, while the sidecar-callback form emits "components": ["statestore", "pubsub"]. Same C# intent, different published output, and what that difference costs you comes back at the end, when this model meets aspire publish.

Two other spellings look plausible and are not. builder.AddDaprSidecar("order-service") appears in enough posts to feel like an API:

AppHost.cs(5,9): error CS1061: 'IDistributedApplicationBuilder' does not contain a definition for 'AddDaprSidecar' and no accessible extension method 'AddDaprSidecar' accepting a first argument of type 'IDistributedApplicationBuilder' could be found (are you missing a using directive or an assembly reference?)

Reading the assembly explains why: the builder-level Dapr surface is exactly four methods, AddDapr, AddDaprComponent, AddDaprPubSub and AddDaprStateStore, and AddDaprSidecar is not among them. A sidecar attaches to a resource; it is never declared standalone.

The third one is the using at the top of AppHost.cs, which is easy to read as decoration and is not:

AppHost.cs(11,26): error CS0246: The type or namespace name 'DaprSidecarOptions' could not be found (are you missing a using directive or an assembly reference?)

The extension methods are declared in the Aspire.Hosting namespace, which the AppHost's implicit usings already bring in, so AddDapr and WithDaprSidecar resolve with no using directive at all. DaprSidecarOptions and DaprComponentOptions are not in that namespace. Write the callback form without using CommunityToolkit.Aspire.Hosting.Dapr; and the calls compile while the options types do not, which is exactly the kind of half-working state that sends you back to the package reference. The sample on aspire.dev omits the using.

What compiles, and what the file above already uses, is the reference on the sidecar builder: .WithDaprSidecar(sidecar => sidecar.WithOptions(...).WithReference(stateStore)). One callback, everything Dapr-shaped inside it.

The state store you did not write is in memory

Delete the LocalPath argument from AddDaprStateStore and everything keeps working. That is the problem.

The integration does not go looking for a component you might already have. It writes its own, into a temp directory, and this is what it writes:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
spec:
  type: state.in-memory
  version: v1
  metadata: []

Then it passes that directory to the sidecar as --resources-path. The mechanism is in that flag, and it is the part the advice you will find online gets backwards: --resources-path replaces --components-path, it does not add to it. Whatever is in the machine's ~/.dapr/components, including the Redis state store dapr init put there, is invisible to a sidecar started that way.

The proof is an asymmetry inside a single launch. Two daprd processes, one ps:

daprd ... --components-path /Users/martino/.dapr/components  --app-id inventory-service
daprd ... --resources-path /var/folders/.../T/aspire-dapr.2X7RwV/statestore --app-id order-service

inventory-service has no WithReference, so it keeps the default --components-path and sees the machine's real components: its /v1.0/metadata returns two, pubsub on pubsub.redis and statestore on state.redis. order-service, the one service in the solution that actually stores anything, has the reference, gets --resources-path, and returns exactly one component: statestore on state.in-memory. The service that needs a database is the service that lost it.

Nothing about this announces itself, because the generated store is fully functional. SaveStateAsync succeeds, GetStateAsync hands the value back, the dashboard trace has the same shape either way, and no log line anywhere mentions that the store is in memory. It fails on one event only: the sidecar restarting. Which is to say never on a laptop, and eventually in production.

Pointing LocalPath at the component file in the repository is the fix, and Redis can be asked whether it took:

$ docker exec dapr_redis redis-cli KEYS '*'
order-service||ORD-2001

With LocalPath set, /v1.0/metadata on the order sidecar reports state.redis and that key appears after a write; without it, KEYS '*' comes back empty and the metadata says state.in-memory. The key is also Part 3's point arriving intact: the application passed ORD-2001, and order-service|| in front of it was added by the sidecar, not by the SDK, so the app ID is still part of the physical key even though nothing in the AppHost or the service ever wrote it there.

Check it before you trust it. The sidecar will tell you what it loaded:

curl -s http://localhost:<daprHttpPort>/v1.0/metadata | jq '.components'
# [{"name":"statestore","type":"state.in-memory","version":"v1", ...}]   <- the bug, visible

The daprHttpPort is the one on the dapr run line in the dashboard, not the one in the app's DAPR_HTTP_PORT, though as noted above both answer. If you would rather not leave the browser, the same dashboard Source column answers the question without a request: look at which of --resources-path and --components-path the sidecar was given.

ServiceDefaults: what every service inherits

Both services start with the same line, and that line is the entire footprint Aspire has inside a service process. All four calls it makes:

public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
{
    builder.ConfigureOpenTelemetry();

    builder.AddDefaultHealthChecks();

    builder.Services.AddServiceDiscovery();

    builder.Services.ConfigureHttpClientDefaults(http =>
    {
        // Turn on resilience by default
        http.AddStandardResilienceHandler();

        // Turn on service discovery by default
        http.AddServiceDiscovery();
    });

    return builder;
}

Nothing in this sample was customised. Extensions.cs is byte-identical between the 13.3.5 and 13.5.3 templates and byte-identical again to the copy sitting in the repository's older AspireDemo solution, so the file above is current for the whole 13.x line and you can read someone else's copy as if it were your own.

It can stay identical because there is nothing to version. No Aspire.ServiceDefaults package exists to install. The template stamps a class library into your solution and from then on the code is yours, which is why you read the four calls above rather than trusting them. Look at what the project file references and the ownership gets clearer: Microsoft.Extensions.Http.Resilience, Microsoft.Extensions.ServiceDiscovery, and five OpenTelemetry.* packages. Not one Aspire.* reference anywhere. Resilience and service discovery ship out of dotnet/extensions on the 10.x line while the AppHost next door is pinned to 13.5.3, so upgrading Aspire does not upgrade this, and the three-way version skew from earlier does not reach it. First-party guidance is firm about keeping it that way: "Don't include other shared functionality or models in this project."

One call is missing from the list, and it is the one people assume is there. MapDefaultEndpoints is not invoked by AddServiceDefaults; you write it yourself in Program.cs. The reason is a type mismatch rather than an oversight. Everything inside AddServiceDefaults is generic over TBuilder : IHostApplicationBuilder, so it works on a worker host or a MAUI builder as happily as on a web app, while MapDefaultEndpoints takes a concrete WebApplication. A background worker can register health checks and has nothing to map them on, so registration and mapping had to be two calls. The aside below is about what those endpoints do once you have mapped them, which is less than the name suggests.

The interesting default is one method deeper than the list above, in the private AddOpenTelemetryExporters that ConfigureOpenTelemetry calls on its last line, and it is a conditional:

var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

if (useOtlpExporter)
{
    builder.Services.AddOpenTelemetry().UseOtlpExporter();
}

Everything ConfigureOpenTelemetry does before reaching that call is unconditional: logging with formatted messages and scopes, ASP.NET Core, HttpClient and runtime metrics, ASP.NET Core and HttpClient tracing. The exporter is the only part that asks a question first, and the question is whether OTEL_EXPORTER_OTLP_ENDPOINT has a value. If it does not, no exporter is registered at all. The instrumentation still runs, still allocates, still builds every span, and then drops the lot. Nothing warns, because from the SDK's point of view nothing is wrong.

Under the AppHost that variable is always set, along with OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_SERVICE_NAME (literally the string you passed to AddProject), a per-run service.instance.id in OTEL_RESOURCE_ATTRIBUTES, and an x-otlp-api-key header for the dashboard's ingest endpoint. That is why nothing in either Program.cs names a collector: the orchestrator names it, on every child process it starts, including the two dapr run executables. Two of the injected values are Development-only, and they explain a mismatch people blame on their APM vendor: locally you get OTEL_TRACES_SAMPLER=always_on and one-second export intervals, and in production Aspire sets neither, leaving you on the OTel SDK's own defaults. Dashboards that feel instant on a laptop lag in production for that reason and no other.

Take the same service and deploy it to App Service or a plain container without that variable and you get zero telemetry with zero warnings. Setting the variable is the fix, not editing Extensions.cs. And resist the reflex of adding your vendor's OTLP exporter beside the defaults: UseOtlpExporter is single-shot, so a second call, or a signal-specific AddOtlpExporter() on the same service collection, throws NotSupportedException at startup rather than warning.

That is the whole inheritance, and from the consuming side it is one line in OrderService/Program.cs:

using Dapr.Client;
using DaprAspireDemo.OrderService.Inventory;
using DaprAspireDemo.OrderService.Orders;

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

builder.Services.AddDaprClient();

// Keyed singleton rather than AddHttpClient, to keep the app ID out of reach of service discovery.
builder.Services.AddKeyedSingleton<HttpClient>(
    InventoryClient.AppId,
    (_, key) => DaprClient.CreateInvokeHttpClient(appId: (string)key!));

builder.Services.AddSingleton<InventoryClient>();

var app = builder.Build();

app.MapDefaultEndpoints();

app.MapOrderEndpoints();

await app.RunAsync();

Two frameworks, two lines, and neither knows the other exists. AddServiceDefaults never mentions Dapr. AddDaprClient never mentions Aspire: it reads DAPR_HTTP_PORT and DAPR_GRPC_PORT out of its own environment, exactly as it did in Part 3, and the only thing that changed is who set them. There it was dapr run from the multi-app file. Here it is the hosting integration, one layer further out. The defaults 3500 and 50001 appear nowhere in this file precisely because nothing in it assumes a port.

Service invocation itself runs over an ordinary HttpClient. CreateInvokeHttpClient gives the client a base address whose host is the app ID, inventory-service over plain http, and installs the handler that rewrites each request into {daprEndpoint}/v1.0/invoke/inventory-service/method/{path}, and that factory is the supported surface now that DaprClient.InvokeMethodAsync has carried [Obsolete] since the Dapr .NET SDK 1.17.

The keyed registration in the middle is the one line in the file that is a judgement call, and it deserves a more honest defence than "the factory would break", because the factory does not break. Registering the same client through AddHttpClient, same inventory-service base address, same Dapr InvocationHandler, resolved through IHttpClientFactory, returns HTTP 201 on POST /orders. Two independent launches, no exception at CreateClient and none at request time.

It works because of what AddServiceDiscovery() registers by default: a pass-through provider. A host name with no matching configuration entry is not an error, it is a name that gets handed onward untouched to ordinary DNS. A pair of probe clients in one launch proves both halves of that sentence live. The one with a configuration entry resolved and then failed at connect with Connection refused (localhost:59999), so service discovery is genuinely in the pipeline and genuinely rewriting. The one without an entry failed at the socket with nodename nor servname provided, or not known (inventory-service:80), so the URI reached DNS with the app ID still in it. On the real invoke client the Dapr handler intercepts before any socket opens and rewrites the authority to the sidecar, so the DNS failure never happens.

The trap is on the other side of that condition, and it is one line in the AppHost away. Add .WithReference(inventory) to the order-service resource, the ordinary service-discovery overload and not the obsolete component one from the AppHost earlier, and Aspire hands the order-service process four new environment variables: the lowercase http and https service-discovery entries, and the uppercase INVENTORY_SERVICE_HTTP and INVENTORY_SERVICE_HTTPS forms of the same thing. A full environment diff against a control launch found those four and nothing else. Service discovery now has something to resolve, so it rewrites the request before Dapr ever sees it:

services__inventory-service__http__0=http://localhost:5049

http://inventory-service/stock/check  ->  http://localhost:5049/stock/check

The handler then reads uri.Host off the rewritten URI, which drops the port, and the sidecar is asked to invoke a service called localhost:

{"errorCode":"ERR_DIRECT_INVOKE","message":"failed to invoke, id: localhost, err: couldn't find service: localhost"}

POST /orders comes back 503. I ran that rather than inferring it from the error message: same AppHost, the factory registration in place, 503 on the order and that body verbatim out of the sidecar. Seeing the rewrite behind it took a separate run. A second client in the same launch, same base address, service discovery in its pipeline and no InvocationHandler, reached inventory-service's own Kestrel and came back 200, which is only possible if something had already replaced the authority.

Run that same AppHost against the sample as it ships and the reference changes nothing: 201, 200, 409, byte-identical to a control launch without it. The registration style, not the AppHost, is what decides the outcome. DaprClient.CreateInvokeHttpClient builds its client directly, so it never passes through IHttpClientFactory, ConfigureHttpClientDefaults never reaches it, and the injected configuration sits there inert. The two mechanisms are not exclusive, they are ordered, and service discovery is first. Keeping the invoke client out of the factory keeps the app ID out of the pipeline service discovery reads, which is a smaller claim than "the factory would break" and the one the runs actually support.

AddStandardResilienceHandler in the shared defaults is quietly wrapping every factory-created client in retries, a circuit breaker and two layers of timeout, which is a good thing to know before you write your own retry loop on top of it; Part 5 pulls that apart properly, because it belongs to the migration story rather than to this one.

InventoryService/Program.cs is the shortest argument in the whole sample. It calls AddServiceDefaults, maps its endpoints, and stops. Its project file references DaprAspireDemo.ServiceDefaults and nothing else: no Dapr.Client, no Dapr.AspNetCore, no Dapr type anywhere in its source. It is reachable as inventory-service because the AppHost attached a sidecar with that app ID, and Part 3's asymmetry survives the move intact. Being callable by app ID still costs the callee zero lines.

The health endpoints disappear outside Development

MapDefaultEndpoints is nine lines of code, and the first of them is a condition:

public static WebApplication MapDefaultEndpoints(this WebApplication app)
{
    // Adding health checks endpoints to applications in non-development environments has security implications.
    // See https://aka.ms/aspire/healthchecks for details before enabling these endpoints in non-development environments.
    if (app.Environment.IsDevelopment())
    {
        // All health checks must pass for app to be considered ready to accept traffic after starting
        app.MapHealthChecks(HealthEndpointPath);

        // Only health checks tagged with the "live" tag must pass for app to be considered alive
        app.MapHealthChecks(AlivenessEndpointPath, new HealthCheckOptions
        {
            Predicate = r => r.Tags.Contains("live")
        });
    }

    return app;
}

Outside Development neither /health nor /alive is mapped. Not "returns 503", not "returns an empty body": there is no route. Point an AKS readiness probe or a Container Apps health probe at /health, get a 404, and the rollout stalls with a pod that never goes ready and not one log line anywhere mentioning health checks. aspire.dev states it plainly: "In non-development environments, the /health and /alive endpoints are disabled by default."

The reason is defensible. An unauthenticated /health that fans out to your database, cache and message broker on every request is a DoS amplifier that anyone on the internet can aim at your dependencies, and its response body enumerates those dependencies by name.

Before you go rewriting them, get the tag semantics right, because the version repeated in most posts is backwards. /health is mapped with no predicate, so it runs every registered check, including the ones tagged live. /alive filters to live only, so untagged checks never reach it. A probe app registering the template's "self" check plus a second untagged check returning Unhealthy answers 503 on /health and 200 on /alive at the same moment, which is exactly the Kubernetes semantic you want: a dead dependency should fail readiness and pull the pod out of the load balancer, and must not fail liveness and trigger a restart that cannot possibly help. Every Aspire client integration that registers a check (AddNpgsqlDataSource, AddRedisClient) registers it untagged, so this is the behaviour you inherit whether you thought about it or not.

The fix is not deleting the if. The documented alternative keeps the endpoints cheap and quiet: a 5-second request timeout policy, a 10-second output cache so a probe storm hits the cache rather than the database, both applied through a MapGroup, and host filtering or authorization on that group so only the platform's probe can reach it. That is a handful of lines in the same method, and it is the variant to copy.

One consequence reaches back into the AppHost. WithHttpHealthCheck("/health") combined with WaitFor is what turns "start B after A" into "start B after A is actually answering", which is the mechanism orders.WaitFor(inventory) leaned on back in the AppHost. If the endpoint it probes exists only in Development, that gate exists only in Development too, and nothing tells you when it stops applying.

One request, seven spans

POST /orders does three things: it takes an order, asks inventory-service whether the lines can be filled, and writes the accepted order to the state store. Two of those cross a process boundary, and nothing in the order-service source names a host, a port or a URL to make them happen.

curl -X POST http://localhost:5037/orders \
  -H "Content-Type: application/json" \
  -d '{"orderId":"ORD-1001","customerId":"CUST-42","lines":[{"sku":"AZ-KEYBOARD","quantity":2,"unitPrice":79.99},{"sku":"AZ-MOUSE","quantity":1,"unitPrice":24.50}]}'

{"orderId":"ORD-1001","customerId":"CUST-42","lines":[...],"total":184.48,"status":0,"placedAt":"2026-08-28T06:47:16.941436+00:00"}

HTTP 201, with the total computed from the lines the inventory service agreed to reserve. Order something the warehouse does not have and the same endpoint answers 409 with the shortfall rather than a stack trace:

{"orderId":"ORD-1002","shortfalls":[{"sku":"AZ-DOCK","requested":1,"onHand":0}]}

Part 3's run file could produce both of those responses. What it could not produce is the next block. The dashboard's trace detail for order-service: POST /orders/ reports Duration 0.18s, Resources 4, Depth 5, Total spans 7, and the tree behind those numbers is this:

POST /orders/                                     order-service
  HTTP POST 200                                   order-service
    HTTP POST                                     order-service-dapr-cli
      CallLocal/inventory-service/stock/check     inventory-service-dapr-cli
        POST /stock/check                         inventory-service
  DATA state /dapr.proto.runtime.v1.Dapr/SaveState  order-service-dapr-cli
  HTTP POST 200 order-service-dapr-cli            order-service

The right-hand column is the resource each span came from, and it is the column that makes this worth printing. Both *-dapr-cli resources are in it. The sidecar is not a black box that the trace jumps over: order-service-dapr-cli records the outbound HTTP POST it received, and inventory-service-dapr-cli records CallLocal/inventory-service/stock/check, the sidecar-to-sidecar hop, as a real span with the callee's own ASP.NET Core span nested underneath it. The SaveState call is there too, as its own span on the order sidecar, tagged with the gRPC method the SDK actually invoked (/dapr.proto.runtime.v1.Dapr/SaveState) rather than with anything the application wrote. Four resources, one trace, and not a line of telemetry configuration in either Program.cs.

Two separate things have to be true for that tree to exist, and only one of them is about the dashboard. The first is instrumentation: the single AddServiceDefaults call in each service registers inbound ASP.NET Core instrumentation on both services and outbound HttpClient instrumentation on the caller, so order-service writes a traceparent on the way out and inventory-service reads one on the way in. The second is delivery: the AppHost sets OTEL_EXPORTER_OTLP_ENDPOINT on the two service processes and on the two dapr run executables, along with the protocol, a per-resource OTEL_SERVICE_NAME and the dashboard's OTLP API key, so all four processes ship to the same collector without any of them knowing where it is. Miss the difference between those two and you will draw the wrong conclusion the first time a trace comes back fragmented: curl a callee directly, with no caller upstream, and you get a separate trace with a single span in it, because spans link on propagated context and not on a shared destination.

Before you build a habit around that dashboard, know what it is and is not. Its telemetry is capped at 10,000 traces and 10,000 logs shared across every resource, and Microsoft's own framing of the scope is that "the dashboard is designed as a development and short-term diagnostic tool", which "persists telemetry in-memory" and where "no telemetry is persisted when the dashboard is restarted". And you reach it through a one-time ?t= token, which is regenerated on every launch, on whichever port the template wrote into the AppHost's launchSettings.json when it scaffolded the project. That port is randomised once, at scaffold time, and then pinned: 17004 for this sample on every run, and some other five-digit number for yours. Either way, any instruction that tells you to browse to localhost:18888 is describing the standalone dashboard container, not the one aspire run just printed a URL for.

One open risk belongs here rather than in a footnote, because it is specifically a multi-sidecar risk. CommunityToolkit/Aspire#1509, open since August 2026, reports one sidecar per launch dying while the dashboard continues to show it as Finished, with a rotating victim and the dapr CLI's serialised startup as the suspected cause. I never saw it across three launches on Aspire 13.5.3 with toolkit 13.0.0: both sidecars reached Running, stayed there, answered /v1.0/metadata and contributed spans every time. Three launches against a race, with no attempt on my part to provoke it, is evidence of absence and not much more. Read it as a reason to check the resource list rather than as a reason to assume the issue is gone.

Restarting the app proves nothing about durability

The dashboard puts a Restart button next to every resource, which makes it the obvious way to answer "does my data survive a restart?". It answers a different question.

Restart order-service from the dashboard and the app process is genuinely replaced: PID 54221 became PID 60531, and the start time in the resource list moved with it. Neither daprd moved. order-service-dapr-cli stayed on PID 54223 with the dashboard still showing its original start time, and inventory-service-dapr-cli stayed on 54222. The sidecar is a top-level resource in its own right, and restarting its app does not touch it.

That is a useful property most of the time and a trap once. Because the sidecar survives, anything the sidecar was holding survives with it, and a state store living inside daprd's own memory is exactly that. GET /orders/ORD-1001 on an order written before the restart came back HTTP 200 on the configuration with no LocalPath set, the one that resolves to state.in-memory. The store that loses everything the moment the sidecar dies passes the restart-the-app test with full marks. Restart the *-dapr-cli resource instead and the two configurations finally disagree: 404 with the generated in-memory component, 200 with the repository's Redis one. That is the test worth writing down.

Two small details from the same menu. An app resource offers Stop, Restart and Rebuild; a *-dapr-cli resource offers Stop and Restart with no Rebuild, which is one more reminder that the sidecar is an executable Aspire launched and not a project it built. And Restart is not Rebuild even where both exist: the product's own description of it is "Source code is not recompiled."

If you switch to the http profile

The AppHost template writes two launch profiles, https and http, and on Aspire 13.5.3 the second one does not start:

Unhandled exception. System.AggregateException: One or more errors occurred. (The 'applicationUrl'
setting must be an https address unless the 'ASPIRE_ALLOW_UNSECURED_TRANSPORT' environment variable
is set to true. ...)
 ---> Microsoft.Extensions.Options.OptionsValidationException: The 'applicationUrl' setting must be
an https address unless the 'ASPIRE_ALLOW_UNSECURED_TRANSPORT' environment variable is set to true.

The template-generated http profile does not set that variable, so the profile the template ships is dead on arrival. This is the good kind of failure: loud, immediate, and naming the variable it wants.

Getting far enough to see it is the awkward half. aspire run has no --launch-profile flag at all, and with AspireUseCliBundle=true the dotnet run --launch-profile http you would reach for next is routed through aspire run and the profile is dropped on the floor: the AppHost came up with DOTNET_LAUNCH_PROFILE=https regardless, with nothing in the output admitting the switch was ignored. Any instruction of the form "run it with the http profile" has to say how, which in this case meant running the built AppHost binary directly with the profile's environment variables set by hand.

Once ASPIRE_ALLOW_UNSECURED_TRANSPORT=true is set and the app comes up, the interesting part is that nothing about the trace changes. Same tree, 4 resources, depth 5, 7 spans, both sidecars contributing, over a plain http OTLP endpoint, with OTEL_EXPORTER_OTLP_INSECURE never set on either daprd process. The insecure-endpoint story you may have read about Dapr and OTLP is not what bites you on this stack; the launch profile is.

One more observation from reading those environments, because it explains where the sidecars' configuration is not coming from. Both daprd processes were started with --config ~/.dapr/config.yaml, and that flag arrives the same way --components-path does: it is the dapr CLI's own default, not something the integration passed. The file it points at declares only a Zipkin exporter and has no OTLP section anywhere in it, yet both sidecars exported to the dashboard anyway. daprd 1.18.3 honours the OTEL_EXPORTER_OTLP_* variables independently of the Dapr Configuration resource. I did not check whether the spans reached Zipkin at the same time.

Where the model stops

Everything above is one machine. The introduction called the AppHost a development-time orchestrator on Microsoft's own authority, and this is the section that says what that costs you, because the parts of the model that do not cross the deployment boundary do not announce that they are staying behind.

Start with the components, because that is the one people find last. An AppHost carrying AddAzureContainerAppEnvironment("cae").WithDaprComponents() alongside an AddDaprStateStore and an AddDaprPubSub publishes cleanly: aspire publish -o ./out reported all five steps succeeded. The cae.bicep it wrote contains a managed identity, a container registry, a role assignment, a Log Analytics workspace, the Microsoft.App/managedEnvironments resource itself, the dashboard's dotNetComponents resource, and the outputs. It contains no Microsoft.App/managedEnvironments/daprComponents resource of any kind. The two components declared in C# have no representation anywhere in the emitted infrastructure.

Reading the publishing code explains that rather than excusing it. WithDaprComponents invokes a component's publishing action only when that component carries an AzureDaprComponentPublishingAnnotation, and a plain AddDaprStateStore is never given one. The annotation comes from the Azure helper packages, in practice from CommunityToolkit.Aspire.Hosting.Azure.Dapr.Redis, and only when you call WithReference on the component with an AddAzureManagedRedis resource as the argument; AddRedis and AddAzureRedis do not qualify, and neither does the local YAML file that made everything work on your laptop. The place not to look for the missing components is azd. It deploys the bicep it was handed, and the bicep never had them.

Sidecar enablement does survive publish, and one value in it needs checking before anything reaches a cluster:

dapr: { enabled: true, appId: 'svc', appProtocol: 'http', appPort: 8080, logLevel: 'info', enableApiLogging: false }

That appPort was not inferred. The probe container behind that line listens on 80, and the bicep still said 8080, because the publishing code falls back to a literal 8080 whenever DaprSidecarOptions.AppPort is unset. Port inference from the app's allocated endpoint happens at run time only, and has no publish-mode equivalent. A container that does not happen to listen on 8080 gets a deployed sidecar that cannot reach it, and the first symptom is a health check rather than an error about ports.

AppPort is also one of the survivors, which is the more useful way to read that line. Of the 32 properties on DaprSidecarOptions, the publishing path translates five into an ACA sidecar: AppId, AppPort, EnableApiLogging, LogLevel and AppProtocol. The other 27 have no destination in the emitted bicep, including every AppHealth* property, the port overrides, ResourcesPaths, and Config. Config being dropped costs nothing in practice, because Azure Container Apps lists the Dapr Configuration spec first among the things it does not support: the capability is not available on that platform whether Aspire forwards the setting or not.

Which matters mostly because Aspire never writes a Dapr Configuration resource in the first place. The integration passes --config only when you point DaprSidecarOptions.Config at a file you wrote yourself, and there is no C# surface that produces one. Tracing sampling rates, mTLS, access control lists, resiliency policies, middleware pipelines: all of it is hand-written YAML on the local side, and a separate question again about what the target platform accepts.

Line those up and they resolve into one uncomfortable symmetry. components/statestore.yaml, the one file you had to write by hand because the generated alternative was an in-memory store, is the artifact from this whole exercise that crosses the boundary intact. It is a real file, in source control, describing a real Redis instance, and getting it onto Container Apps or Kubernetes is an ordinary deployment problem with ordinary answers. The component the AppHost wrote for you lived in a temp directory that was deleted at shutdown and has no counterpart on the far side of publish. The half you had to do by hand is the half that ships.

Conclusion

Both files are deleted at the deployment boundary, so whatever the C# one is worth had to show up before then, and it does. One aspire run started four processes, held the caller back until the callee's health check answered rather than until it happened to be up, and put seven spans from four resources into one trace, with no line in either Program.cs naming a collector, a port, or the other service. dapr run -f . starts the same four processes and hands you back a terminal.

What you give up for it is one thing rather than a list. Everything the AppHost inferred for you it inferred at run time and nowhere else: the component YAML, the wiring, the ports, the health-gated ordering. None of it is waiting on the far side of aspire publish, and the one Dapr artifact the AppHost could not generate, because the generated version would have quietly lost your data, is the one you can actually deploy. Treat that as the rule for how much of this model to trust: whatever the AppHost saved you from writing, you will write eventually, and the sooner you write it the less of it is a surprise.

Part 5 takes the same boundary from the other side, moving an existing Functions app across it one piece at a time, and works out what happens when Dapr's built-in service-invocation retries, the standard resilience handler ServiceDefaults installs on every HttpClient, and your own retry loop all fire on the same failed request.

Building Your First Dapr + Web API Service

Martin Oehlert — Fri, 28 Aug 2026 06:08:23 +0000

Part 2 ended on a trade: Dapr removes the part of your code that names a backing service, and a second process in every replica is the price. The interesting question is what the first half costs on the C# side, because "no vendor names" is not the same as "no vendor constraints". Two projects are enough to find out: one that calls through Dapr, and one that only gets called.

Two services, two app IDs

Start with the two project files, because the difference between them is the argument.

<!-- OrderApi/OrderApi.csproj -->
<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" />
    <PackageReference Include="Dapr.Client" />
  </ItemGroup>
</Project>

<!-- InventoryApi/InventoryApi.csproj -->
<Project Sdk="Microsoft.NET.Sdk.Web">
  <!-- No Dapr package reference, on purpose. inventory-api is reached BY app ID, it does not
       reach anything by app ID, so it never touches the SDK. Being callable through Dapr costs
       a target service exactly zero lines of code. -->
</Project>

orders-api owns order state and calls out. inventory-api answers stock checks and calls nothing, so it carries no Dapr package, no Dapr type, and no Dapr configuration. Being addressable as inventory-api is a property of how the process was started, not of how it was written: the app ID is a runtime label the sidecar owns.

That is also why this is two projects rather than one binary started twice under two app IDs: running one assembly twice would hide the asymmetry that matters, which is that only one of the two services is coupled to Dapr at all. Versions are pinned centrally at Dapr.Client and Dapr.AspNetCore 1.18.5, the same pair Part 2 used; AddDaprClient lives in Dapr.AspNetCore and the DaprClient it registers in Dapr.Client, which is why orders-api, which only wants the client, still takes both packages. Both projects, the run file and the components folder are in DaprWebApiDemo in azure-functions-samples, and its README carries the local run loop.

The wiring on the calling side is two registrations in Program.cs:

var builder = WebApplication.CreateBuilder(args);

// Source-generated JSON, front of the chain. The reflection resolver stays behind it because
// the framework serialises types this app never declares (ProblemDetails, most obviously).
builder.Services.ConfigureHttpJsonOptions(options =>
    options.SerializerOptions.TypeInfoResolverChain.Insert(0, OrderApiJsonContext.Default));

// The same generated metadata, handed to Dapr. AddDaprClient reads DAPR_HTTP_PORT /
// DAPR_GRPC_PORT from the environment, so 3500 and 50001 are never written down here.
builder.Services.AddDaprClient(dapr =>
    dapr.UseJsonSerializationOptions(OrderApiJsonContext.Default.Options));

AddDaprClient is the whole registration, and Part 2 covered the two environment variables behind it. The remaining lines of Program.cs build the HttpClient that reaches inventory-api.

The callee's Program.cs is the same file with the Dapr line removed:

var builder = WebApplication.CreateBuilder(args);

// The only wiring this service needs. No AddDaprClient, because it never calls out through a
// sidecar; it only gets called through one.
builder.Services.ConfigureHttpJsonOptions(options =>
    options.SerializerOptions.TypeInfoResolverChain.Insert(0, InventoryApiJsonContext.Default));

var app = builder.Build();

app.MapStockEndpoints();

await app.RunAsync();

One JSON context, wired two different ways

Both services declare their wire types up front, which on orders-api looks like this:

[JsonSourceGenerationOptions(JsonSerializerDefaults.Web)]
[JsonSerializable(typeof(CreateOrderRequest))]
[JsonSerializable(typeof(Order))]
[JsonSerializable(typeof(OrderLine))]
[JsonSerializable(typeof(OutOfStockResponse))]
[JsonSerializable(typeof(ErrorResponse))]
[JsonSerializable(typeof(ConfirmConflictResponse))]
[JsonSerializable(typeof(CancelOrderRequest))]
[JsonSerializable(typeof(CancelOrderResponse))]
[JsonSerializable(typeof(OrderCancellation))]
[JsonSerializable(typeof(CancelConflictResponse))]
[JsonSerializable(typeof(StockCheckRequest))]
[JsonSerializable(typeof(StockCheckResponse))]
[JsonSerializable(typeof(StockLine))]
[JsonSerializable(typeof(StockShortfall))]
[JsonSerializable(typeof(DaprErrorBody))]
public sealed partial class OrderApiJsonContext : JsonSerializerContext;

The two registrations above hand that context to their serializers in different shapes. Dapr gets UseJsonSerializationOptions(OrderApiJsonContext.Default.Options), a hard swap: the only things the client serialises are state values and invocation payloads, every one of them is on the list above, and there is nothing left for a fallback to catch. Minimal API instead gets the context inserted at the front of the resolver chain, because the framework serialises types this file will never list: ProblemDetails, the bare strings that come out of Results.BadRequest, and whatever the next middleware decides to write. Swap the reflection resolver out there and you break responses you never wrote.

The asymmetry decides how a mistake reaches you. Add a type to a response, forget the matching [JsonSerializable] line, and the Dapr path fails on it immediately while the HTTP path quietly falls through to reflection and works. The quiet side is the one that breaks when somebody publishes trimmed or AOT.

JsonSerializerDefaults.Web on the attribute is the other line to read twice. It is what keeps the generated metadata in step with ASP.NET Core's own defaults: camelCase names, case-insensitive reads, numbers accepted from strings. Leave it off and nothing fails to compile; a property just silently stays null at runtime, on the far side of a service call, where you will look for it last.

Starting both, with a sidecar each

Two apps means two sidecars, and the CLI has a multi-app run file for exactly that:

version: 1
common:
  env:
    ASPNETCORE_ENVIRONMENT: Development
apps:
  - appID: orders-api
    appDirPath: .
    appProtocol: http
    appPort: 5100
    resourcesPaths: ./components/local
    command: ["dotnet", "run", "--project", "./OrderApi", "--no-build", "--", "--urls", "http://localhost:5100"]
  - appID: inventory-api
    appDirPath: .
    appProtocol: http
    appPort: 5101
    resourcesPaths: ./components/local
    command: ["dotnet", "run", "--project", "./InventoryApi", "--no-build", "--", "--urls", "http://localhost:5101"]

dotnet build
dapr run -f .

dapr run -f . reads that file, starts both projects with a daprd sidecar each, and points both at components/local. dapr stop -f . takes the set back down. Two details in the YAML cost time if you skip them: every relative path is resolved against appDirPath, which is why both apps keep appDirPath: . and select their project on the command line instead, and appPort is a declaration rather than an instruction. It tells the sidecar which port to call your app on; the --urls argument after the -- is what makes Kestrel actually listen there. The two are set separately and nothing checks that they agree, so keep them in view of each other.

The component behind resourcesPaths is unchanged from Part 2: a state.redis component named orderstore locally, a state.azure.cosmosdb component with the same metadata.name in components/azure, and no C# that knows the difference. One line in it belongs to this section's point rather than to Part 2's:

scopes:
  - orders-api

inventory-api is deliberately absent. It holds no state, so it has no reason to be able to reach the store, and without a scopes list every Dapr app in the environment loads every component. The service that costs zero lines of Dapr code also gets zero Dapr permissions.

Calling a service you cannot name

Service invocation in the current SDK is a client you register rather than a method you call. DaprClient.CreateInvokeHttpClient hands back a configured HttpClient, and from there the call site is plain ASP.NET Core: a path, a DTO, PostAsJsonAsync. (The InvokeMethodAsync overloads that used to do this are marked [Obsolete] as of 1.17 and point at the same factory, so new code and migrated code land in the same place.)

// Service invocation runs over an ordinary HttpClient. CreateInvokeHttpClient sets BaseAddress
// to http://inventory-api and installs the handler that rewrites the request into
// {daprEndpoint}/v1.0/invoke/inventory-api/method/{path}. One client per target app ID is the
// documented pattern (an app ID containing an uppercase letter only works when it is passed
// here), so the app ID doubles as the DI key.
builder.Services.AddKeyedSingleton<HttpClient>(
    InventoryClient.AppId,
    (_, key) => DaprClient.CreateInvokeHttpClient(appId: (string)key!));

builder.Services.AddSingleton<InventoryClient>();

InventoryClient.AppId is the constant "inventory-api", and under Dapr's service invocation model that is the entire address. No host, no port, no service discovery configuration anywhere in orders-api.

A second factory does the same thing, which matters when you are reading someone else's code. DaprClient.CreateInvokeHttpClient(appId) is static. daprClient.CreateInvokableHttpClient(appId) is an instance method, and its implementation is a one-line delegation to the static one: it passes along the DaprClient's own HTTP endpoint and API token header. The two are alternatives rather than an old form and a new one. Reach for the instance form when the DaprClient you already hold was configured with a non-default endpoint or an API token, which saves repeating both at every call site. For a client built once at startup, as above, the static form is the one that fits.

What the handler does, and what it refuses to do

The InvocationHandler that factory installs is a DelegatingHandler with a short job list. It reads the request URI, treats the host as the target app ID, and rewrites the request onto the sidecar's invoke route. If an API token is configured it adds a dapr-api-token header for the duration of the call. Then it passes the response back to you completely unmodified: no status inspection, no exception, no wrapping of any kind.

That last part is the one to plan around. A non-2xx from the target arrives as an ordinary HttpResponseMessage, an unreachable sidecar as an ordinary HttpRequestException, and nothing in between knows what an app ID is. Error translation is your code's job, and the sample gives it a home. (Teams arriving from InvokeMethodAsync give up InvocationException and its .AppId, .MethodName and .Response in the move; the SDK does not offer a replacement.)

public async Task<Result<StockCheckResponse>> CheckStockAsync(
    StockCheckRequest request,
    CancellationToken cancellationToken)
{
    try
    {
        // A plain POST. "/stock/check" is the route on the target app; the app ID lives in
        // the BaseAddress, and the handler turns the pair into
        // {daprEndpoint}/v1.0/invoke/inventory-api/method/stock/check.
        using var response = await http.PostAsJsonAsync(
            "/stock/check",
            request,
            OrderApiJsonContext.Default.StockCheckRequest,
            cancellationToken);

        if (!response.IsSuccessStatusCode)
        {
            return await TranslateFailureAsync(response, cancellationToken);
        }

        var body = await response.Content.ReadFromJsonAsync(
            OrderApiJsonContext.Default.StockCheckResponse, cancellationToken);

        return body is null
            ? Fail(InvocationFailure.InvalidResponse, (int)response.StatusCode, "inventory-api returned an empty body.")
            : new Result<StockCheckResponse>.Success(body);
    }
    catch (TaskCanceledException ex) when (!cancellationToken.IsCancellationRequested)
    {
        // HttpClient reports its own timeout as a cancellation, so the guard is what
        // separates "we gave up" from "the caller went away".
        logger.LogWarning(ex, "Stock check for {OrderId} timed out.", request.OrderId);

        return Fail(InvocationFailure.Timeout, null, "inventory-api did not answer in time.");
    }
    catch (HttpRequestException ex)
    {
        // Connection-level: daprd itself is not listening. Nothing was routed.
        logger.LogError(ex, "Could not reach the Dapr sidecar for a stock check on {OrderId}.", request.OrderId);

        return Fail(InvocationFailure.SidecarUnreachable, null, "The Dapr sidecar is not reachable.");
    }
}

InventoryClient is the only class in orders-api that knows inventory-api exists. It owns the wire contract and the error translation, and Fail is a one-liner that wraps an InvocationError into Result<StockCheckResponse>.Failure.

The when (!cancellationToken.IsCancellationRequested) guard on the first catch earns its keep. HttpClient reports its own timeout as a TaskCanceledException, the same type you get when the incoming request is abandoned and ASP.NET Core cancels the token. Without the guard, a caller who closed their browser and an upstream service that stopped answering produce the same log line and the same response.

Failures as values

public abstract record Result<TValue>
{
    private Result()
    {
    }

    public sealed record Success(TValue Value) : Result<TValue>;

    public sealed record Failure(InvocationError Error) : Result<TValue>;
}

public sealed record InvocationError(InvocationFailure Kind, int? StatusCode, string Message);

public enum InvocationFailure
{
    TargetUnreachable,
    SidecarUnreachable,
    UpstreamError,
    InvalidResponse,
    Timeout,
}

The private constructor on Result<TValue> is what makes this worth typing out rather than reaching for a bool and an out parameter. No type outside the file can derive from it, so Success and Failure are the only cases that can ever exist, and a switch over them is exhaustive by construction rather than by convention. The create-order handler leans on that directly:

var stock = await inventory.CheckStockAsync(stockRequest, cancellationToken);

switch (stock)
{
    case Result<StockCheckResponse>.Failure(var error):
        return ToProblem(error);

    case Result<StockCheckResponse>.Success({ Available: false } answer):
        logger.LogInformation("Rejected order {OrderId}: inventory short on {Count} line(s).",
            request.OrderId, answer.Shortfalls.Count);

        return Results.Conflict(new OutOfStockResponse(request.OrderId, answer.Shortfalls));
}

"Inventory said no" and "inventory could not be asked" sit side by side in one statement instead of being split across a try and a catch twenty lines apart. That is the argument for the result pattern here: through Dapr, a call failing is not exceptional. The target app ID has no address yet, the target is mid-restart, the sidecar is still warming up. Those are Tuesday afternoon, not an incident.

The five InvocationFailure cases are deliberately coarser than HTTP status codes, because the caller's question is never "was that a 502 or a 504". It is "retry, give up, or blame the payload". Which is exactly why the client bothers to open the sidecar's error body at all:

private async Task<Result<StockCheckResponse>> TranslateFailureAsync(
    HttpResponseMessage response,
    CancellationToken cancellationToken)
{
    var status = (int)response.StatusCode;
    var raw = await response.Content.ReadAsStringAsync(cancellationToken);
    var daprError = TryReadDaprError(raw);

    if (string.Equals(daprError?.ErrorCode, DirectInvokeErrorCode, StringComparison.Ordinal))
    {
        logger.LogWarning(
            "Sidecar could not route to {AppId}: {Detail}", AppId, daprError?.Message ?? raw);

        return Fail(InvocationFailure.TargetUnreachable, status, $"Dapr could not route to '{AppId}'.");
    }

    logger.LogWarning("inventory-api answered {Status}: {Body}", status, raw);

    return Fail(InvocationFailure.UpstreamError, status, $"inventory-api answered {status}.");
}

When the sidecar cannot route to an app ID it answers HTTP 500 with a JSON body whose errorCode is ERR_DIRECT_INVOKE. Read only the status and that is indistinguishable from inventory-api throwing on your request. The distinction is the whole point: in the first case nothing ran on the far side, so the call is worth making again once the target registers; in the second, the target ran and rejected you, and repeating it changes nothing. DirectInvokeErrorCode is a constant on the client holding that string, and TryReadDaprError is best-effort, catching JsonException and returning null, because a target app is free to return whatever it likes in an error body and non-JSON there is normal.

Those two categories then have to reach the HTTP caller as different answers:

private static IResult ToProblem(InvocationError error) => error.Kind switch
{
    InvocationFailure.TargetUnreachable or InvocationFailure.SidecarUnreachable =>
        Results.Problem(
            title: "Inventory is unavailable.",
            detail: error.Message,
            statusCode: StatusCodes.Status503ServiceUnavailable),

    InvocationFailure.Timeout =>
        Results.Problem(
            title: "Inventory did not answer in time.",
            detail: error.Message,
            statusCode: StatusCodes.Status504GatewayTimeout),

    _ => Results.Problem(
        title: "Inventory rejected the stock check.",
        detail: error.Message,
        statusCode: StatusCodes.Status502BadGateway),
};

A routing failure becomes 503 and an error the target produced becomes 502, which is that same distinction reaching your caller as advice about whether to try again. Both were HTTP 500 on the wire a moment ago. Retries, backoff, and circuit breakers on top of this belong to Dapr's own resiliency policies rather than to your handler, and Part 5 covers them where the migration story makes the trade-offs concrete.

If you would rather speak gRPC

Calls between sidecars go over gRPC no matter what, so choosing gRPC for your own leg changes how your process talks to the process next to it and nothing about the hop between services. The factory is DaprClient.CreateInvocationInvoker(appId), which is static and returns a Grpc.Core.CallInvoker that injects the target app ID (and the API token) into outgoing gRPC metadata. You hand that invoker to a client generated from the target service's .proto, which is the real .NET-side cost: HTTP invocation needs a DTO and an HttpClient, gRPC invocation needs a contract both sides agree on and a build step that generates from it. The sample stays on HTTP.

The token nobody checks

The header everyone reaches for first is the wrong one. dapr-app-id is an outbound routing header. You put it on a request to your own sidecar to say which app you want to reach, as an alternative to rewriting the URL. It is an instruction, not a claim, and Dapr never verifies it on the way in. An endpoint filter that reads dapr-app-id and rejects anything unexpected blocks precisely nobody, because any caller who can reach your app port can set that header to any string they like.

The primitive that does mean something is dapr-api-token, and the part that catches people out is the token asymmetry. One header name, two directions, two environment variables:

DAPR_API_TOKEN secures your app's outbound calls to its own sidecar. The SDK reads that variable itself and attaches the header, including on the HttpClient from the previous section. No application code in orders-api touches it.
APP_API_TOKEN secures the sidecar's inbound calls into your app: service invocation arriving from another app, pub/sub deliveries, input bindings. The runtime sends it on the same dapr-api-token header, and checking it is entirely your problem.

Dapr's guidance for the second one is a sentence: look for the header. There is no ASP.NET Core code on the page, which is presumably why so few services have any check at all.

In a Minimal API, an endpoint filter is where it goes, and a class rather than a lambda, because the check needs configuration and a logger.

var orders = app.MapGroup("/orders")
    .AddEndpointFilter<DaprApiTokenFilter>();

orders.MapCreateOrder();
orders.MapGetOrder();
orders.MapConfirmOrder();
orders.MapCancelOrder();

Applying it at the group rather than per endpoint is what makes it hard to bypass: AddEndpointFilter covers every endpoint added to the group afterwards, so a new slice cannot forget the token check by forgetting a line. ASP.NET Core constructs the filter through ActivatorUtilities, so IConfiguration and ILogger<T> arrive by constructor injection with no registration of the filter type anywhere.

public sealed class DaprApiTokenFilter : IEndpointFilter
{
    private const string HeaderName = "dapr-api-token";

    private readonly byte[]? expectedToken;
    private readonly ILogger<DaprApiTokenFilter> logger;

    public DaprApiTokenFilter(IConfiguration configuration, ILogger<DaprApiTokenFilter> logger)
    {
        var token = configuration["APP_API_TOKEN"];

        expectedToken = string.IsNullOrEmpty(token) ? null : Encoding.UTF8.GetBytes(token);
        this.logger = logger;

        if (expectedToken is null)
        {
            logger.LogWarning("APP_API_TOKEN is not set; inbound Dapr calls are not authenticated.");
        }
    }

    public async ValueTask<object?> InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
    {
        ArgumentNullException.ThrowIfNull(context);
        ArgumentNullException.ThrowIfNull(next);

        if (expectedToken is null)
        {
            return await next(context);
        }

        var presented = context.HttpContext.Request.Headers[HeaderName];

        // Exactly one value. A repeated header is a caller trying something.
        if (presented.Count != 1 || !Matches(presented[0], expectedToken))
        {
            logger.LogWarning(
                "Rejected {Method} {Path}: missing or invalid dapr-api-token.",
                context.HttpContext.Request.Method,
                context.HttpContext.Request.Path);

            return Results.Unauthorized();
        }

        return await next(context);
    }

    private static bool Matches(string? candidate, byte[] expected)
    {
        if (string.IsNullOrEmpty(candidate))
        {
            return false;
        }

        // FixedTimeEquals is false for a length mismatch without leaking where the difference
        // is. A token is a shared secret, so the ordinary string comparison is the wrong tool.
        return CryptographicOperations.FixedTimeEquals(Encoding.UTF8.GetBytes(candidate), expected);
    }
}

Two decisions in there will make you stop and squint.

CryptographicOperations.FixedTimeEquals instead of == is the smaller one. An ordinary string comparison returns as soon as two bytes differ, and that difference in timing is a signal a patient caller can measure. The presented.Count != 1 test next to it rejects a request carrying the header twice: a client with a legitimate token sends it once, and a duplicated header is somebody probing which value the framework picks.

The larger decision is that the filter fails open. With APP_API_TOKEN unset there is no expected token, and every request goes through with a warning logged once. That looks backwards for a security filter, and it is the only behaviour that works: with no token configured, the sidecar has nothing to send, so a filter that closed would reject Dapr itself and leave you debugging 401s from your own runtime. It also keeps a plain curl against the app port working on a laptop. Anywhere that is not a laptop, set the variable.

The filter is constructed once per endpoint rather than once per request, so that "not authenticated" warning appears at startup and never again. Four routes in the group, four warnings, then silence: a registration receipt, not a repeated alarm.

What this prevents is not "a caller with the wrong app ID". Your application listens on its own port, and the sidecar is one more client of that port. Anything else with network access to the process, a misconfigured ingress rule, a pod in the same namespace, a colleague's port-forward, reaches your endpoints directly and bypasses Dapr entirely: no mTLS, no access control policy, no trace. The one thing such a caller cannot produce is a token it was never given.

Order state, and the write that loses

Every order endpoint touches state, and the calls look nearly alike without being alike. Creating an order writes a value that did not exist a second ago. Confirming one reads a value, changes a field, and writes it back. The second shape is the one a key-value store cannot make safe on its own, and the SDK's method names tell you so before the documentation does.

The create handler ends on a single call:

// State management. SaveStateAsync has no ETag parameter at all: an unconditional
// write is last-write-wins by construction. That is the right call for a create, and
// the wrong call for the confirm in the sibling slice.
//
// The metadata is the Cosmos DB partition, and it is on all four slices or none of
// them. OrderPartition says why; this is the write that decides where the document
// lands, so it is the one to get right first.
await dapr.SaveStateAsync(
    StateStore.Name,
    order.OrderId,
    order,
    metadata: OrderPartition.For(order.CustomerId),
    cancellationToken: cancellationToken);

StateStore.Name is the constant "orderstore", which is the component's metadata.name and the only string this application knows about its own database. The key it hands over is order.OrderId; what lands in Redis locally and in Cosmos DB in Azure is orders-api||ORD-1001, with the app-ID prefix Part 2 covered added by the sidecar rather than by the SDK.

SaveStateAsync has no ETag parameter, in any overload: not an optional one, not a nullable one. Its signature is store, key, value, StateOptions, metadata, cancellation token, and nothing in there can carry a version, which makes an unconditional save last-write-wins by construction. Optimistic concurrency in this SDK is opt-in by choosing a different method rather than by passing an extra argument. For a create that is the behaviour you want: the value being written was computed from the request, not from prior state, so there is no earlier version whose contents you could be silently discarding.

That metadata argument is a decision rather than an API, and the transactions section argues it out. The rule it follows: the partition value is the same on every call that touches an order key, which is why it appears on the create, the read, and the confirm alike.

Reading it back is the same shape in reverse, and the route is the first place the decision shows:

private static async Task<IResult> GetOrderAsync(
    string customerId,
    string orderId,
    DaprClient dapr,
    CancellationToken cancellationToken)
{
    ArgumentNullException.ThrowIfNull(dapr);

    var order = await dapr.GetStateAsync<Order>(
        StateStore.Name,
        orderId,
        metadata: OrderPartition.For(customerId),
        cancellationToken: cancellationToken);

    // Cosmos DB enforces the second half of this on its own: an order from another customer
    // is in another partition and comes back null. Redis ignores the metadata and hands over
    // whatever sits under the key, so the check is what keeps both stores answering alike.
    return order is null || !string.Equals(order.CustomerId, customerId, StringComparison.Ordinal)
        ? Results.NotFound()
        : Results.Ok(order);
}

The route is /orders/{customerId}/{orderId} rather than /orders/{orderId}, and that is not a REST preference. The partition value lives inside the document you have not read yet, so a caller has to name the partition to address the order at all. Committing to a partition key reaches the URL, not just the storage layer.

A missing key is not an error in Dapr. GetStateAsync returns the default value for TValue, so a key that was never written and a key holding nothing look the same from inside the handler. Turning either into a 404 is your application's decision; the runtime has no opinion. The customer check beside it collapses "no such order" and "not your order" into the same 404 on purpose, since answering them differently would tell an unauthenticated caller which order IDs exist.

The confirm is a loop, not a call

Confirming an order is a read-modify-write against a key other replicas of orders-api are free to write at the same moment. The conditional save is TrySaveStateAsync, which takes an ETag and returns bool: false means the store's current ETag no longer matches the one you presented. No exception, no status code to unpick. Part 2 spent real space on the fact that an ETag mismatch on the raw HTTP API is not a 409 but a 500-class body with ERR_STATE_SAVE and "possible etag mismatch" buried in the text. The .NET client absorbs that and gives you a branch.

private static async Task<IResult> ConfirmOrderAsync(
    string customerId,
    string orderId,
    DaprClient dapr,
    ILogger<Order> logger,
    CancellationToken cancellationToken)
{
    ArgumentNullException.ThrowIfNull(dapr);

    var partition = OrderPartition.For(customerId);

    for (var attempt = 1; attempt <= MaxAttempts; attempt++)
    {
        var (order, etag) = await dapr.GetStateAndETagAsync<Order>(
            StateStore.Name,
            orderId,
            ConsistencyMode.Strong,
            partition,
            cancellationToken);

        // A missing key comes back as a default value plus some ETag rather than throwing,
        // so the null check has to come before anything reads the order. The customer check
        // beside it is what Redis will not do for you: on Cosmos DB an order belonging to
        // someone else is simply in another partition and never arrives.
        if (order is null || !string.Equals(order.CustomerId, customerId, StringComparison.Ordinal))
        {
            return Results.NotFound();
        }

        if (order.Status is OrderStatus.Confirmed)
        {
            // Already done. Re-confirming is a no-op, which is what makes this endpoint
            // safe to retry from the outside as well as the inside.
            return Results.Ok(order);
        }

        var confirmed = order with
        {
            Status = OrderStatus.Confirmed,
            ConfirmedAt = DateTimeOffset.UtcNow,
        };

        var saved = await dapr.TrySaveStateAsync(
            StateStore.Name,
            orderId,
            confirmed,
            etag,
            new StateOptions
            {
                // With an ETag attached the store already behaves first-write-wins; saying
                // so explicitly keeps the intent readable next to the ETag itself.
                Concurrency = ConcurrencyMode.FirstWrite,
                Consistency = ConsistencyMode.Strong,
            },
            partition,
            cancellationToken);

        if (saved)
        {
            logger.LogInformation("Confirmed order {OrderId} on attempt {Attempt}.", orderId, attempt);

            return Results.Ok(confirmed);
        }

        logger.LogInformation(
            "Lost the ETag race on order {OrderId}, attempt {Attempt} of {MaxAttempts}.",
            orderId, attempt, MaxAttempts);

        // A little backoff. A tight loop under contention just restates the race at speed.
        await Task.Delay(TimeSpan.FromMilliseconds(25 * attempt), cancellationToken);
    }

    logger.LogWarning("Gave up confirming order {OrderId} after {MaxAttempts} attempts.", orderId, MaxAttempts);

    return Results.Conflict(new ConfirmConflictResponse(orderId, MaxAttempts));
}

MaxAttempts is a private constant set to 5, and the bound is not decoration. An unbounded version of this loop is a livelock with good manners: under sustained contention it never returns and never errors, which is worse for whoever is holding the request open than a conflict would be.

The line that carries the weight is the position of the read. GetStateAndETagAsync sits at the top of the loop body, not above it. Hoist it out to save a round trip and every retry re-presents the ETag the store has already moved past, which is precisely the ETag that cannot match. The save then fails five times for the same reason it failed once, the backoff makes each failure slower than the last, and the endpoint returns a conflict that no amount of retrying would ever have resolved. Re-reading is not an optimisation on top of the retry; re-reading is what a retry consists of.

The early return on an already-confirmed order does something separate and easy to read past: it makes the endpoint idempotent before it makes it concurrency-safe. Inside the loop, a lost race whose winner also confirmed the order terminates on the next pass instead of burning attempts on a write that would be a no-op anyway. Outside the loop, a caller who never saw the response (a timeout, a dropped connection, a queue redelivering) can send the same request again and get the same 200 back. Without that check the second confirm would either overwrite a ConfirmedAt timestamp somebody else already committed or fight for the right to do so.

Then there is ConsistencyMode.Strong, passed twice. The StateOptions on the save draws the eye, but it is the third positional argument on the read that decides whether this loop means what it says. Dapr assumes data stores are eventually consistent by default, and under ConsistencyMode.Eventual a read can come back from a replica that has not caught up. The ETag you were handed is then already behind the store's authoritative copy, and a conditional save that returns true has told you something narrower than "nobody changed this while I was working": it has told you that a comparison against whichever copy answered went through. Strong on the read is what makes the ETag a claim about the current value.

One gap the sample steps around rather than solves. What the ETag in that tuple contains when the key does not exist is not something the code relies on: it inspects the value, returns 404, and never looks at the string. If you are tempted to branch on an empty ETag to mean "this key is new", verify that against the store you actually run, because the SDK's contract does not promise it.

Bulk is not a transaction

Dapr's own overview describes bulk operations as submitting multiple requests individually to the underlying store and returning the results as a single batch. Read that sentence for what it withholds. A batch is a round-trip optimisation: one call from your process to the sidecar instead of N, with the sidecar doing the fan-out. It is not a unit of work. SaveBulkStateAsync with five items can leave you with three items written and two not, and the SDK's return type has no room to tell you which. On the wire it is not even a distinct operation; it is the same POST /v1.0/state/<store> array body Part 2 described for a single save, with more entries in the array.

The atomic one is ExecuteStateTransactionAsync, and only against a store that declares transaction support. Against Blob Storage or Table Storage the call fails outright rather than partially, which is the better failure of the two. Its operations are Upsert or Delete only; there is no read inside a transaction, so every value you commit was read before the transaction was built, which puts you back on the ETag machinery from the previous section if the read matters. The retry shape is not the same one, though. ExecuteStateTransactionAsync returns a bare Task, so there is no false to branch on the way TrySaveStateAsync gives you, and a conflict can only reach you as an exception.

The bulk signatures carry a second surprise, visible only if you go looking for a parameter that is not there.

public abstract Task SaveBulkStateAsync<TValue>(
    string storeName, IReadOnlyList<SaveStateItem<TValue>> items,
    CancellationToken cancellationToken = default);

public abstract Task DeleteBulkStateAsync(
    string storeName, IReadOnlyList<BulkDeleteStateItem> items,
    CancellationToken cancellationToken = default);

Every other state method on DaprClient takes StateOptions and a metadata dictionary as top-level parameters. These two take neither, because consistency, concurrency, ETag and metadata all live per item, on SaveStateItem<TValue> and BulkDeleteStateItem. That is a fair reflection of what the call is: N independent writes that happen to travel together, each with its own ETag and its own opinion about consistency, and nothing global left to configure.

Which brings up the part nobody warns you about. Non-actor state in Cosmos DB partitions by the item's own state key, so two orders under two keys land on two partitions, by design and by default. Cosmos DB also requires every item in one transaction to share a partition. Put those together and a multi-key ExecuteStateTransactionAsync against Cosmos DB can fail on partitioning alone, with data that is entirely consistent and code that is entirely correct in its own terms. The fix is the same metadata bag every state method carries, with an explicit partitionKey set to the same value on every operation in the set:

// The serialisation the transactional API does not do for you.
var orderBytes = JsonSerializer.SerializeToUtf8Bytes(cancelled, OrderApiJsonContext.Default.Order);
var auditBytes = JsonSerializer.SerializeToUtf8Bytes(cancellation, OrderApiJsonContext.Default.OrderCancellation);

await dapr.ExecuteStateTransactionAsync(
    StateStore.Name,
    [
        new StateTransactionRequest(order.OrderId, orderBytes, StateOperationType.Upsert, metadata: partition),
        new StateTransactionRequest(CancellationKey(order.OrderId), auditBytes, StateOperationType.Upsert, metadata: partition),
    ],
    cancellationToken: cancellationToken);

StateTransactionRequest takes the value as byte[]?, which is why the two SerializeToUtf8Bytes calls are sitting there in the open. Every other state method on the client serialises for you; this one hands the job back.

Nothing in the .NET signature hints at the partition constraint either. metadata is an opaque string dictionary forwarded to the component verbatim, and which keys are legal is entirely store-specific, so the compiler cannot help and neither can IntelliSense.

Then comes the part that makes this a design decision rather than a parameter. The default partition value is the key the component sees, which is the prefixed orders-api||ORD-1001 from Part 2, and application code cannot reproduce that without hardcoding its own app ID. An explicit partitionKey therefore never equals the default: it always moves the key somewhere else. The override is not scoped to the transaction, it is a property of where that document now lives, so every later read and write of those keys has to carry the same override or Cosmos DB looks in the partition the state key implies and finds nothing there. Adding one transaction to an existing key relocates it, permanently, for every code path that touches it. Decide the partitioning before the first write, not on the day you need two keys to commit together.

Redis has no equivalent constraint, running the set through MULTI/EXEC in one keyspace, which is exactly what makes the trap quiet: the transaction that works all week on your laptop is the one that fails on the store you deploy to. Actor state avoids it because Dapr derives the partition key from the app ID, actor type and actor ID rather than the state key, which guarantees one actor's writes share a partition.

Whether any of this is available to you is a property of the component, not of the API, and the four Azure-relevant stores differ more than the common DaprClient surface suggests:

Actor eligibility is not a flag anyone sets. Dapr's rule is that a store can back actors if it supports both transactions and ETag, which is why the last column is derivable from the two before it rather than being independent information. The practical reading of the bottom two rows: Blob Storage and Table Storage will take the SaveStateAsync and TrySaveStateAsync calls from the previous section and reject ExecuteStateTransactionAsync, so a component swap that looks like a one-line change in YAML can remove an operation your code depends on without any C# changing at all. That is the same leak Part 2 flagged, arriving through a different door.

What I would not build on yet

DaprClient has a QueryStateAsync, and it reads like the answer to "give me every order whose status is pending". I would keep it away from anything that has to be on call.

The endpoint behind it is POST /v1.0-alpha1/state/<store>/query, and the how-to page carries an alpha banner to match. That is the polite version of the warning. The impolite version is the release history: the API has been alpha since 1.5, no release note for 1.15, 1.16, 1.17, or 1.18 mentions it moving, and alpha work beside it graduated on schedule, Bulk PubSub in 1.17 and the Jobs API in 1.18. An API that has not moved through four minor releases of a project that demonstrably does promote things is not queued behind the next one. That, rather than the word "alpha", is the reason to stay off it.

The PostgreSQL situation is sharper still. The v2 component stores values as BYTEA where v1 used JSONB, and JSONB was what made SQL-level filtering possible, so v2 shipped with no query support and still has none; Dapr's own reference page says so outright. v1 keeps it and is not deprecated, which leaves the component version you pick deciding whether a .NET method you already wrote does anything at all.

Which stores implement it is the second reason. Query support is not a documented capability flag: Dapr's own state store table carries CRUD, transactions, ETag, TTL, actors, and workflow, and there is no query column in it. The only reliable signal is in components-contrib, where a store supports the alpha API if its Go package ships a *_query.go. Cosmos DB has cosmosdb_query.go, Redis has redis_query.go, and Blob Storage has neither, which fits a store that holds one opaque blob per key. Having to read Go source to find out whether a .NET method does anything against your configured store says plainly where this feature sits in the project.

Searching for it hands you the wrong page as well. Dapr's docs have a Query state store section with a page for Cosmos DB and a page for Redis, and neither one is about the Query API. They show you how to query the underlying store with its own tools, the Cosmos DB SQL API in Data Explorer and redis-cli against the keys, with the sidecar out of the picture entirely. Read at speed that looks like documented per-store support for the thing you went looking for, when it is the docs telling you to go around Dapr.

Which is also the recommendation. When a read path needs "all orders where X", query the store natively behind a repository type that is the one place the vendor SDK appears, or keep an index you own: a key per status holding the order IDs, maintained by the same code that writes the order. Both are more code than one QueryStateAsync call, and both still work after a component upgrade.

Conclusion

Part 2 priced Dapr as a trade: the vendor names leave your code, and a second process joins every replica. Two services later, the first half holds up, and the second half is not where the cost showed up.

inventory-api is the evidence for the first half. Another service reaches it by app ID, through a sidecar, and none of that reached its source code: the project file carries a comment where the package references would be, and that is the whole of it. On the calling side, the component name is the only place a backing store gets named, and moving from Redis to Cosmos DB is a different file under components/, not a different using.

What the component name does not hide is the shape of the store behind it. /orders/{customerId}/{orderId} has a customer in it because Cosmos DB needs a partition before it can find a document, and an explicit partitionKey on one transaction moves those keys for every code path that touches them afterwards. No vendor type appears anywhere in that, and the vendor's constraints still reach the route table.

The rest of the bill is the code between your service and its sidecar. The HttpClient the SDK hands you routes and does nothing else, so telling "the sidecar could not route to that app ID" apart from "the target ran and refused you" is a string constant and a best-effort JSON read you maintain yourself. The inbound token check has the same shape: Dapr defines APP_API_TOKEN and sends it on every call into your app, and whether anything compares it is a filter you write from a docs page one sentence long.

So the second process is not the expensive part. The expense is the code between your service and it, and how much of that code the documentation leaves to you. Part 4 hands the run file and the components folder to .NET Aspire, where the sidecar stops being something you remember to start.

Introduction to Dapr for Azure Developers

Martin Oehlert — Fri, 21 Aug 2026 06:44:07 +0000

Part 1 handed the job of triggers and bindings to Dapr and then stopped, which leaves the obvious question sitting there: what does a second process in every replica give you that a NuGet package cannot? The honest answer is narrower than the pitch. Dapr does not remove your HTTP client, your message schema, or your working knowledge of Cosmos DB, and the .NET SDK makes that point better than any blog post can, having deprecated its own service invocation methods in favour of a plain HttpClient. It removes exactly one thing: the part of your code that names a backing service. Whether that trade pays for the sidecar it arrives in is what I want to settle before Part 3.

The sidecar, and what it costs you

Dapr does not arrive as a library your application calls. It arrives as a separate process named daprd running beside yours: a second container in the same Container Apps replica or Kubernetes pod, a second OS process under dapr run on your laptop. Your code never links an assembly that knows how to reach Cosmos DB. It makes a loopback call to daprd, and daprd makes the call that leaves the machine.

Two ports carry that loopback traffic: 3500 for the Dapr HTTP API and 50001 for gRPC. Both are defaults rather than guarantees, and the environment variable reference says to read DAPR_HTTP_PORT "instead of hardcoding the port value".

// 3500 holds for exactly as long as nobody passes -H to dapr run.
// The CLI sets DAPR_HTTP_PORT for self-hosted runs, and the
// dapr-sidecar-injector sets it on every container in the pod.
var port = Environment.GetEnvironmentVariable("DAPR_HTTP_PORT") ?? "3500";
var sidecar = new Uri($"http://localhost:{port}");

DAPR_GRPC_PORT is the same contract for 50001. Container Apps documents both ports inside the replica, so the constant would survive there. Read the variable anyway: the same binary runs on your laptop, where -H and -G are one flag away from moving both.

Whichever protocol you picked to reach your own sidecar, sidecar-to-sidecar traffic is always gRPC. The choice changes only how your process talks to a process on the same host.

What it costs

The numbers Dapr publishes come from AKS, three Standard_D2s_v6 nodes, and they spread wider than the single figure that gets quoted in every Dapr thread. At roughly 1,000 iterations per second the sidecar holds 44 to 51 MB of memory, while CPU swings by more than an order of magnitude depending on what you are calling: 4 millicores for a state get over gRPC, 17 for service invocation over HTTP, 106 for a pub/sub publish. Actors and workflow are measured at lower throughputs and cost more memory per replica, up to 241 MB on the actor stress test. Treat 50 MB as a floor, not a budget.

There is a question those numbers do not answer, and I could not find anyone who does: whether the sidecar in Container Apps eats your app's allocated CPU and memory, adds to it, or comes free. No Learn page says either way. So the figures above are the only concrete ones in reach, and they were measured on AKS rather than on the platform this series is heading for.

The startup race

The sidecar is not ready when your container starts. It reaches readiness "once the application is accessible on its configured port", and until then "the application cannot access the Dapr components".

That ordering breaks the obvious pattern of loading configuration from a Dapr secret store in Program.cs. The escape hatch is a second health endpoint: GET /v1.0/healthz checks components, the HTTP port, and the app channel, while GET /v1.0/healthz/outbound runs the same check without the app channel, so it answers while your own startup is still in progress. Both return 204 when healthy and 500 when not, and code that tests for 200 reports a healthy sidecar as broken. Do not depend on the first one in application code: it fails for apps using the Actor and Workflow APIs and creates a circular dependency everywhere else.

The .NET SDK ships CheckOutboundHealthAsync and WaitForSidecarAsync for this. The docs limit both to secrets and configuration retrieval, promise to remove them in a future release, and attach a caveat that will cost you an afternoon: an application that waits on WaitForSidecarAsync without using actors, secret management, configuration retrieval, or workflows "will indefinitely lock up during startup", because the runtime never opens an outbound connection for it to wait on. Read "indefinitely" literally. DAPR_HEALTH_TIMEOUT caps the runtime's own 60-second wait, but the SDK method is a poll loop with no timeout in it: it ends when the sidecar answers or when you cancel the token, and on the path above it does neither.

The reverse race has its own answer: app health checks, off by default, enabled with --enable-app-health-check, hold back every pub/sub subscription, input binding, and inbound invocation until the first probe of your app succeeds.

What the five building blocks actually remove

Every building block has the same two parts, and no third: an HTTP or gRPC surface on localhost, and a YAML file naming the implementation behind it. Your code calls a path, the path names a component, and a type field in that YAML decides whether the name resolves to Redis or to Cosmos DB.

Part 1 made the case that a Functions binding is a property of the host. This is the same contract moved out to the environment, and each of the five below bites in its own way.

Service invocation: you keep the HttpClient

The surface is one route with every verb on it, /v1.0/invoke/<appID>/method/<method-name>, where <appID> is a logical name, not a host. Cross-namespace targets use <appID>.<namespace>, and targets that are not Dapr apps put an HTTPEndpoint resource name or an FQDN in the same slot. A second form matters more for migration, because it leaves existing URLs alone and moves the routing into a dapr-app-id header.

The .NET SDK has already walked away from its own helper for this.

// Before: the call that docs.dapr.io still teaches.
// Compiles with warning CS0618: ... is obsolete: 'Recommended guidance is to
// use a native HTTP or gRPC client for service invocation'
await daprClient.InvokeMethodAsync<ReserveStock>("inventory", "reserve", request);

// After: an ordinary HttpClient whose BaseAddress is http://inventory.
// One client per target app ID, because the app ID is the base address.
builder.Services.AddSingleton(_ =>
    new InventoryClient(DaprClient.CreateInvokeHttpClient(appId: "inventory")));

public sealed class InventoryClient(HttpClient http)
{
    public async Task<Reservation?> ReserveAsync(ReserveStock request, CancellationToken ct)
    {
        var response = await http.PostAsJsonAsync("/stock/reserve", request, ct);
        response.EnsureSuccessStatusCode();

        return await response.Content.ReadFromJsonAsync<Reservation>(ct);
    }
}

CreateInvokeHttpClient installs an invocation handler and sets BaseAddress to http://<app-id>, throwing when that is not a legal hostname. Because the app ID is the base address, you get one client per target app whether you wanted the ceremony or not, and the XML docs add that passing appId stops being optional the moment your app ID contains an uppercase letter.

Strip the pitch back and the claim gets smaller, and more interesting for it. Dapr does not replace your HTTP client. It replaces the service discovery, the mTLS, the retries, the tracing, and the round-robin load balancing behind it. The hostname, the port, the certificate handling, and the Polly policy all go.

State management: the key you write is not the key that lands

The API is asymmetric in a way that shows up the first time you read a trace: a save puts the key in a body array, a get and a delete put it in the path. Three .NET methods cover almost everything, and none is deprecated:

public sealed record Order(string OrderId, string CustomerId, decimal Total, string Currency);

// "statestore" is the component's metadata.name. Your code never names the type.
const string StoreName = "statestore";
var order = new Order("order-1041", "cust-8802", 149.95m, "EUR");

// POST /v1.0/state/statestore   (the key travels in the body)
await daprClient.SaveStateAsync(StoreName, order.OrderId, order);

// GET /v1.0/state/statestore/order-1041   (and DELETE on the same path)
var stored = await daprClient.GetStateAsync<Order>(StoreName, order.OrderId);

Concurrency is optimistic and ETag-based. concurrency takes first-write or last-write, consistency takes strong or eventual, and on a save both ride in an options object next to the value. Send no ETag and first-write behaves as last-write-wins. Eventual is the default, because "Dapr assumes data stores are eventually consistent by default": read that again if you just configured a Cosmos DB account with strong consistency and assumed the component inherited it.

Do not go looking for a status code on an ETag mismatch, and do not trust the 409 that circulates online. A save, a delete, and a get for a missing key all answer 204; a concurrency failure arrives as an error body carrying ERR_STATE_SAVE and "possible etag mismatch". The 409 traces back to dapr/dapr#2619, a proposal that never shipped.

The next one shows up in the portal, not the debugger. Dapr prefixes every state key with your app ID, using || as the separator. The key your code passes is order-1041; the document that lands in Cosmos DB has the id orders-api||order-1041. Nothing in your code mentions the prefix and nothing in the portal explains it. Two consequences: two Dapr apps cannot read each other's state, and any existing system reading that container looks for order-1041 and finds nothing. The keyPrefix metadata field is the lever, taking appid (the default), name, namespace, or none.

Two constraints here turn into component decisions later: /transaction depends on a store that supports transactions, and actor state additionally requires actorStateStore: "true" on a store that can do those transactions, with the docs adding that a distributed database behind it has to provide strong consistency. That is the first crack in the abstraction.

Pub/sub: the envelope you did not ask for

Publishing is one call, POST /v1.0/publish/<pubsubname>/<topic>, with any metadata as query parameters. Delivery is at-least-once, so your handler has to tolerate seeing the same order twice. The subscriber acknowledges with a 2xx and a body of {"status": "<status>"}, where the status is SUCCESS, RETRY, or DROP.

In .NET, PublishEventAsync(pubsubName, topicName, payload) sends the order. What arrives at the other end is not the order.

{
  "id": "5929aaac-a5e2-4ca1-859c-edfe73f11565",
  "source": "orders-api",
  "type": "com.dapr.event.sent",
  "specversion": "1.0",
  "datacontenttype": "application/json",
  "time": "2026-08-21T09:14:02Z",
  "topic": "orders",
  "pubsubname": "orderpubsub",
  "traceid": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "tracestate": "",
  "data": {
    "orderId": "order-1041",
    "customerId": "cust-8802",
    "total": 149.95,
    "currency": "EUR"
  }
}

CloudEvents is the default, and it is the number one first-timer surprise. Your subscriber's model binder sees id, source, and data, binds nothing useful, and hands your handler an Order with every property null. The bug report writes itself, and the fix is one line:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers().AddDapr();

var app = builder.Build();

app.UseCloudEvents();       // unwraps the envelope above, so binding sees `data`
app.MapControllers();
app.MapSubscribeHandler();  // serves GET /dapr/subscribe from the [Topic] attributes

app.Run();

[ApiController]
public sealed class OrderEventsController : ControllerBase
{
    [Topic("orderpubsub", "orders")]
    [HttpPost("/orders/placed")]
    public async Task<IActionResult> OnOrderPlaced(Order order, OrderStore store)
    {
        await store.RecordAsync(order);
        return Ok(new { status = "SUCCESS" });   // RETRY redelivers, DROP discards
    }
}

Remove app.UseCloudEvents() and the same code compiles, runs, and silently records empty orders.

The subscription itself can be declared three ways, and the operational difference matters more than the mechanics. Declarative subscriptions are a YAML Subscription resource that "removes the Dapr dependency from your code" and hot reloads by default. Programmatic subscriptions are the [Topic] attributes above, "only read once during application start-up", so a new topic means a restart. Streaming subscriptions invert the flow: the application pulls from Dapr through Dapr.Messaging.PublishSubscribe, and a subscription can start and stop at runtime.

You can opt out of the envelope with rawPayload, and the bill is longer than it looks: raw mode "disables support for tracing, event deduplication per messageId, content-type metadata, and any other features built using the CloudEvent schema". There is also a naming asymmetry with no defence: the publish-side key is rawPayload, while .NET and the Kubernetes CRDs spell it isRawPayload.

Dapr provides dead letter topics for every pub/sub component, "even if the underlying system does not support this feature natively", set with deadLetterTopic on the subscription. Then the default that catches everyone:

By default, when a dead letter topic is set, any failing message immediately goes to the dead letter topic.

No retry, no backoff, no second chance: one transient timeout against your database and the order is in the dead letter topic. Configure a retry resiliency policy first, then enable the dead letter topic.

Message TTL is ttlInSeconds in metadata, enforced by the runtime, so every component supports it. Azure Service Bus is the one component with native entity-level TTL, where Dapr hands the metadata to the broker.

Bindings: the trigger, minus the host

Direction is the part everyone gets backwards. An output binding is your application calling out through the sidecar; an input binding is Dapr calling in, to a route on your own app port.

Output binding calls go through DaprClient.InvokeBindingAsync, with a required operation field. The generic set is create, update, delete, and exec, but the legal values are whatever the component implements, so get, list, and query show up on some and not others; the component's reference page is the only list that binds. On the input side there is no DaprClient method at all.

// Output binding: your app calls out through the sidecar.
await daprClient.InvokeBindingAsync("order-archive", "create", order);

// Input binding: Dapr calls in. The route name is the component's metadata.name.
app.MapPost("order-received", async (Order order, OrderService orders) =>
{
    await orders.HandleAsync(order);
    return Results.Ok();   // anything other than 200 OK schedules redelivery
});

// Dapr probes every input binding route with OPTIONS on startup and expects
// 2xx or 405. A 404 here means no events, ever.
app.MapMethods("order-received", [HttpMethods.Options], () => Results.Ok());

That OPTIONS probe is the failure mode to remember, because it produces no error anywhere: a router that answers 404 never subscribes, never receives an event, and never logs a reason. The component behind the route is an ordinary bindings.azure.storagequeues resource named order-received, and its direction property, while not required, "is highly recommended" for input bindings.

Compared against [QueueTrigger], this is more ceremony, not less: an endpoint, a component file, and a name that has to match in two places, where an attribute used to do it. In exchange you get an ordinary HTTP endpoint, one that runs in a test, a console host, or any service with no Functions host around to supply the trigger. You traded brevity for portability.

Secrets: one API with a dangerous default

Two routes, /v1.0/secrets/<store-name>/<name> for one secret and /bulk for everything, and a .NET method behind each of them. GetSecretAsync is the one you will use:

// The declared return type is Dictionary<string, string>, not your secret.
Dictionary<string, string> secret =
    await daprClient.GetSecretAsync("orderssecrets", "orders-cosmos-key");

// Azure Key Vault is a name/value store, so the field is the secret name:
//   { "orders-cosmos-key": "..." }
var cosmosKey = secret["orders-cosmos-key"];

// A Kubernetes store returns the inner data keys instead, and no secret name:
//   { "accountKey": "...", "accountEndpoint": "..." }

The return type is the gotcha. GetSecretAsync hands back a dictionary rather than your secret, because the response shape depends on the store type. Code written against a local file store in development still compiles against Kubernetes, and then throws KeyNotFoundException on the first lookup. In exchange the API takes away the vendor SDK, the chicken-and-egg of needing a secret in order to fetch secrets, and secrets in appsettings.json.

The default it ships with is the one to change:

Once you configure a secret store for your application, any secret defined within that store is accessible by default from the Dapr application.

Every secret. Not the ones your app asked for, not the ones it was scoped to. Narrowing it means a secrets.scopes policy on the Dapr Configuration resource, where allowedSecrets or deniedSecrets take priority over defaultAccess.

That fix has a sting attached. Azure Container Apps does not expose the Dapr Configuration spec at all, so secrets.scopes is not available there. Scope moves down to the Azure layer instead: a Key Vault holding only what this application needs, RBAC on that vault, and component scopes limiting which app IDs load the component at all.

Where the component model leaks

A component is a YAML resource, and four lines of it do the work:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore              # <- what your code says
spec:
  type: state.azure.cosmosdb    # <- what actually runs
  version: v1
  metadata:                     # <- how that implementation is configured
    - name: url
      value: "https://acct.documents.azure.com:443/"
scopes:
  - orders-api                  # <- which app IDs may load it

Your code addresses statestore and never mentions state.azure.cosmosdb, which is the entire proposition: change spec.type from state.redis to state.azure.cosmosdb, redeploy, and the SaveStateAsync call above runs unchanged. Since Dapr v1.18 you do not even restart for it: components hot reload by default, going briefly unavailable while they close and re-initialize. Actor state stores and workflow backends are carved out of that, so the component whose swap you would most like to be quiet is the one that still needs a restart.

That promise holds at compile time. It leaks at the level of behaviour, in five places.

Capability is per component, and the code that used it stops working.

Swap statestore from Cosmos DB to Blob Storage without touching a line of C# and your build stays green while your /transaction calls fail, your TTLs stop expiring, and your actors refuse to activate. Note the Cosmos DB row too: every capability except workflow, which is not the gap anybody predicts.

Support level is per component as well. Azure Service Bus Topics pub/sub is Stable since runtime 1.0; the Queues variant is still Beta. Same building block, same publish call, different answer to "is this supported in production". Container Apps then disagrees with the project that printed the label and lists that same Beta queues component in its own fully-supported tier, so the answer also depends on who you ask.

The backing store's own constraints show through anyway. Cosmos DB requires the container's partition key to be named exactly /partitionKey, and the Service Bus subscription Dapr creates is named after your consumerID. Neither fact appears in the Dapr API surface, and both are the first thing you see in the Azure portal.

Emulation is not equivalence. Dead letter topics exist "even if the underlying system does not support this feature natively", which on Service Bus leaves you owning two dead letter mechanisms with different semantics. And stores without native ETags "are expected to simulate ETags", so your optimistic concurrency is as strong as somebody else's emulation of it.

The physical data is not what you wrote. orders-api||order-1041 is still in that container under a key no other system will look for.

The YAML swap does what it says at compile time. The behavioural equivalence is not real, and nothing in the API surface tells you which capabilities you gave up. Treat a component change as a dependency change with a review and a test pass behind it, not as a configuration change you make on a Friday.

Mapping the blocks onto Azure

None of the three services below is new to you. A state store component is the [CosmosDBOutput] binding you have already configured, a pub/sub component is the [ServiceBusTrigger] you have already debugged, and a secret store is the Key Vault reference already in your app settings. What changes is ownership: the connection detail leaves local.settings.json for a resource that outlives the app, and the identity belongs to the workload rather than to the Functions host.

Each mapping has one detail that fails your first deploy. Cosmos DB fails on a role assignment you cannot make in the portal, Service Bus on a permission nobody documents, Key Vault on a secret name that was never legal.

State store to Azure Cosmos DB

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore                 # the string your SaveStateAsync call passes
spec:
  type: state.azure.cosmosdb
  version: v1
  initTimeout: 5m                  # sits on spec, not inside metadata
  metadata:
    - name: url
      value: "https://acct-orders.documents.azure.com:443/"
    - name: database
      value: "orders"
    - name: collection             # the field is collection, not container
      value: "orderstate"
    - name: azureClientId          # user-assigned identity
      value: "8f3c1d2a-...-9b4e"
    # no masterKey: dropping it is what selects Entra ID authentication
scopes:
  - orders-api

# The data plane wants the identity's OBJECT id, not the client ID above.
PRINCIPAL_ID=$(az identity show \
  --name id-orders-api --resource-group rg-orders \
  --query principalId --output tsv)

# ...0002 is Cosmos DB Built-in Data Contributor (...0001 is Data Reader).
# There is no portal equivalent of this command.
az cosmosdb sql role assignment create \
  --account-name acct-orders --resource-group rg-orders \
  --scope "/" \
  --principal-id "$PRINCIPAL_ID" \
  --role-definition-id "00000000-0000-0000-0000-000000000002"

The field names are where the first hour goes. It is collection on a service whose portal has called it a container for years, and masterKey is documented as required "only when not using Microsoft Entra ID authentication", so dropping that one line is the switch onto the shared azure* credential fields. Raise initTimeout to 5m as the docs recommend, because Cosmos DB rate-limits metadata requests account-wide, exactly the request every cold sidecar makes at once.

Then the requirement that rejects a container you created last week:

The partition key for the collection must be named /partitionKey (note: this is case-sensitive).

For ordinary state, the Dapr key doubles as the partition key value, which is why orders-api||order-1041 shows up in both fields of the same document. Actor state differs: the partition key comes from app ID plus actor type plus actor ID, so everything one actor owns lands on one physical partition, because actor operations are transactional and Cosmos DB transactions are single-partition.

The second code block catches people who have done Azure RBAC a hundred times. Cosmos DB for NoSQL does not use standard Azure RBAC for data-plane access. Contributor on the account buys management-plane rights and no ability to read a single document. You need a native Cosmos role assignment, identified by a fixed GUID, and two things about it will trip you:

--principal-id is the service principal object ID, while azureClientId takes the client ID. Different GUIDs for the same identity, and the wrong one produces an assignment that exists and never applies.
Data-plane role assignments cannot be managed in the Azure portal. CLI, PowerShell, or Bicep, or it does not happen.

Pub/sub to Azure Service Bus

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
spec:
  type: pubsub.azure.servicebus.topics       # Stable. The queues variant is Beta.
  version: v1
  metadata:
    - name: namespaceName                    # FQDN, and required for Entra ID
      value: "sb-orders.servicebus.windows.net"
    - name: azureClientId
      value: "8f3c1d2a-...-9b4e"
    # Leave the next two out in development and Dapr creates the topic and the
    # subscription on first use. Keep them for production, where the entities
    # come from your IaC and the sidecar never needs admin rights.
    - name: disableEntityManagement
      value: "true"
    - name: consumerID
      value: "orders-fulfilment"
scopes:
  - orders-api

connectionString and namespaceName are mutually exclusive: setting both fails the component with "connectionString and namespaceName cannot both be specified".

Leave those last two fields out, publish once, and the portal shows you a subscription you never created, named after your application. Dapr creates one Service Bus subscription per topic per consumerID, and consumerID defaults to your Dapr app ID. That default decides your delivery semantics without appearing in any config file: two applications with different app IDs each get their own subscription and both see every message, while two replicas of one application share a subscription and compete for it. Setting it explicitly, as the YAML above does, is how you make the choice visible to whoever reads it next.

Auto-creation carries a permission requirement nobody advertises. Dapr creates topics and subscriptions through the Service Bus admin client, which it constructs only when disableEntityManagement is false. I want to be straight about where that requirement comes from: the Dapr docs never state it. It is derived from the component source plus Microsoft's role definitions. Creating an entity is a management operation, and neither Data Sender nor Data Receiver includes management rights, so as it ships the component needs Azure Service Bus Data Owner. Verify that against your own namespace instead of taking it on trust, and know the shape of the failure: a subscriber whose sidecar cannot start, not a message that goes missing.

The least-privilege path is to create the entities in your IaC, set disableEntityManagement: "true", and assign scoped Data Sender and Data Receiver. Two Learn caveats come with it: an assignment for a topic subscription needs effective scope over that topic subscription resource, which the portal cannot assign at, and propagation takes up to five minutes, long enough that your first test run afterwards fails for a reason unrelated to your configuration.

Secrets to Azure Key Vault

secretstores.azure.keyvault has the shortest spec of the three: vaultName and the shared auth fields. No key mode, no connection string mode, so Entra ID is the only way in. Dapr's own setup creates the vault with --enable-rbac-authorization true and assigns Key Vault Secrets User. Dapr's page describes that role as "Get secrets", which undersells it by one action: it also reads secret metadata, which is exactly what the /bulk route needs in order to list names before it fetches values. No write, no delete, no keys, no certificates. Take the definition from Learn rather than from the Dapr page, and stop there.

The naming rule is where two separate facts usually get merged into one:

Azure constrains a secret name to alphanumerics and hyphens, so a configuration key like ConnectionStrings:Orders cannot be a Key Vault secret name at all.
Dapr does no name translation and no JSON flattening for the Key Vault store. GetSecret returns one entry keyed by the name you asked for, so JSON in the value is an opaque string your own code parses.

The Section--Name to Section:Name convention that makes fact 1 survivable belongs to the ASP.NET Core Key Vault configuration provider, not to Dapr; through Dapr, a double hyphen is two hyphens in a name. The only Dapr secret store with a separator is the local file store, via nestedSeparator.

Keeping connection strings out of component YAML closes the loop: a metadata entry takes secretKeyRef in place of value, and auth.secretStore names the store as a sibling of spec:

spec:
  metadata:
    - name: connectionString
      secretKeyRef:
        name: orders-servicebus-connection    # for Key Vault these two
        key: orders-servicebus-connection     # must be identical
auth:
  secretStore: orderssecrets

The two differ only for a multi-key store such as Kubernetes, where name picks the secret and key picks a field inside it.

One asymmetry changes a runbook rather than a code file. Kubernetes secrets are re-read when they change; every other secret store, Key Vault included, is resolved once at component init. Rotate a secret in the vault and the running sidecar keeps the old value until it restarts, or until you touch the component manifest and let hot reload pick it up. Rotation is a deployment step here, not a vault operation.

The auth chain that hides a misconfiguration

Every Azure-backed component shares one credential model, and with none of its fields set Dapr walks a fixed chain where the first success wins: client credentials, client certificate, workload identity on AKS, SPIFFE, managed identity, and finally the Azure CLI.

That last entry is the problem. On your laptop, where az login succeeded weeks ago, a component with no identity configured authenticates through your own user account and the demo runs. Nothing in the logs suggests that the managed identity the component will use in Azure was never created, never assigned a Cosmos data-plane role, and never granted Key Vault Secrets User. You find all three at once, in a deployment, where the sidecar fails to initialize the component.

The fix is one metadata entry. azureAuthMethods: "managedidentity" restricts the chain to the listed methods, so the CLI credential is never tried and the local run fails the same way the deployment would. A failure you see at dapr run is a config change; the same failure in a deployment is a rollback.

Running it on your machine

dapr init

# Everything after -- is your application's own command line.
dapr run --app-id orders-api --app-port 5000 --resources-path ./components -- dotnet run

dapr init leaves four containers running, not the three that older walkthroughs show: Redis as your default state store and message broker, Zipkin for traces, the placement service for actors, and the scheduler service for jobs, which only arrived in Dapr 1.14. The companion sample is a .NET 10 minimal API that exercises service invocation, state management, and pub/sub against a sidecar: DaprDemo in azure-functions-samples. Its README carries the full local loop, and it holds the local Redis component YAML next to the Azure one so you can see exactly what changes between them.

Treat the CLI as a way to see the moving parts rather than as your inner loop: Part 4 replaces most of it with .NET Aspire, which models the sidecar and the components as part of the app host instead of as a YAML folder you remember to point at.

What Azure Container Apps gives you, and what it takes away

az containerapp create -n orders-api -g rg-orders \
  --environment cae-orders --image acrorders.azurecr.io/orders-api:2026.08.21 \
  --enable-dapr --dapr-app-id orders-api --dapr-app-port 8080 --dapr-app-protocol http

One flag turns it on, the others describe your application, and the sidecar is there, patched and upgraded without you. No Helm chart, no control plane, no injector, no placement or scheduler service to keep alive. For most teams that is the whole argument for Container Apps over AKS.

The Dapr settings are application-scope, so changing them restarts every existing revision; no new one is created. Components change shape too: in open-source Dapr a component belongs to an app or a namespace, while here it is an environment-level resource deployed once and visible to everything in the environment, with YAML simplified to match:

# statestore.yaml for Container Apps. The name is not in the file: it comes from
# az containerapp env dapr-component set --dapr-component-name statestore ...
componentType: state.azure.cosmosdb
version: v1
initTimeout: 5m       # root level here, because there is no spec wrapper
metadata:
  - name: url
    value: "https://acct-orders.documents.azure.com:443/"
  # database, collection, and azureClientId exactly as before
scopes:
  - orders-api

No apiVersion, no kind, no spec wrapper, and componentType where spec.type used to be, which is why initTimeout sits at the root here and on spec in the open-source component earlier. The name comes from --dapr-component-name, not the file, so a component copied out of a docs.dapr.io tutorial is a rewrite, not a paste.

scopes carries more weight here, because every Dapr-enabled app in the environment loads every deployed component unless a scope says otherwise. And the values are Dapr app IDs, not container app names: the same string often enough that nobody notices until a --dapr-app-id diverges from the resource name and a component silently stops loading.

Identity wiring loses a moving part, because the component has no identity of its own. It uses the managed identity of the container apps in its scope, so the Cosmos role assignment from earlier is made against the app.

The constraints below are trades, not defects.

The Dapr Configuration spec is not available, stated exactly once, in a bullet list, as "any capabilities that require use of the Dapr configuration spec". If you carry one limitation out of this article, carry that one. Everything in the open-source Configuration resource goes at once: tracing sampling rates, access control policies between apps, API allowlists, feature flags, spec.mtls, and secrets.scopes. Tracing itself survives the cut, wired at the environment level instead through daprAIConnectionString or an OpenTelemetry configuration with includeDapr set; the sampling rate is the one knob with no documented replacement.

secrets.scopes is the item to sit with, because it is the fix the secrets section pointed you at and it is not here. Your Key Vault store hands every secret in the vault to every app scoped to the component, and your only boundary is the vault itself and the RBAC on it.

mTLS between sidecars is on, and there is no documented way off. No Learn page says you cannot disable it; the conclusion follows from the exclusion above. Do not confuse it with the environment-level peerAuthentication.mtls setting or with ingress.clientCertificateMode, which sit at different layers and solve different problems.

You cannot pin the Dapr version. The only lever is timing, through planned maintenance. Version strings carry an Azure suffix such as -msft.N whose numbers skip values, so watch the update feed and not a version number.

Resiliency exists, in a different place than you would look for it, and still in preview. daprComponents/resiliencyPolicies carries the timeouts, HTTP retries, and circuit breakers, with independent outboundPolicy (sidecar to component) and inboundPolicy (sidecar to your app) settings. The portal exposes only timeout and retry, you restart your Dapr apps after applying one, and this is where the retry policy goes that has to exist before you enable deadLetterTopic.

Then the coverage gaps, which matter when you are choosing NuGet packages: the Dapr server extension, actor, and workflow SDK packages are not compatible with Azure Container Apps. Dapr is not supported for Container Apps jobs either, so a scheduled job runs without a sidecar, and actor reminders require minReplicas of at least 1, which takes scale-to-zero off the table.

Which leaves the question from the top of the article where it started. Nobody has published what the sidecar costs inside a replica, so size it on the AKS figures and then measure your own.

Conclusion

What you buy is a code base with no vendor name in it: no CosmosClient, no ServiceBusClient, no SecretClient, no connection string reaching your application at all. The bill arrives as a second process in every replica, an envelope you did not design, a key prefix you did not write, and a capability matrix to read before every component swap. Most of that price is paid in things you have to know rather than things you have to run, which is the part nobody puts on a slide.

For a service with one queue and one table, that price buys nothing: a process, a component model, and a class of failure that exists only because Dapr is there, in exchange for portability you will never exercise.

It starts paying when you have several services that call each other, share state, and need to prove which one is calling. Service discovery, mTLS, and retry policy stop being things you write and start being things the platform holds, and on Container Apps it is already holding the sidecar.

Part 3 builds it: .NET 10 minimal APIs with service invocation and state management wired end to end, against the Cosmos DB and Service Bus components from this article rather than the Redis defaults dapr init handed you.

Why Move Beyond Azure Functions? The Enterprise Ceiling

Martin Oehlert — Fri, 14 Aug 2026 05:58:47 +0000

The case for leaving Azure Functions used to fit in one sentence, and Flex Consumption deleted it: on the plan Microsoft now recommends for new serverless work, the maximum execution timeout is unbounded. What that leaves behind is the harder question. Which limits survived the plan that was supposed to remove them, and are those the kind you engineer around or the kind you migrate away from?

None of the survivors are about execution time. They are 230 seconds for any HTTP response, on every plan; 4 GB of memory across exactly three instance sizes; a scale-out rate that gets slower the busier your app already is; and a plan boundary that ends up deciding your team topology whether you wanted it to or not. Those are structural limits, and functionTimeout moves none of them.

This is written about apps that already work. Nothing below argues Functions was the wrong call, and most of the Functions code running in production today should stay exactly where it is. The argument starts at the point where the hosting model begins making architectural decisions on your behalf. Where this series goes when that happens is Azure Container Apps with Dapr and .NET Aspire; whether you are actually there yet is the thing worth settling first.

The plan everyone is arguing about no longer exists

If your mental model of Functions hosting is "Consumption or Premium," it is a generation out of date. The Consumption plan now carries a legacy banner in its own documentation, Flex Consumption is the recommended serverless plan for new function apps, and Linux Consumption hosting retires on 30 September 2028, receiving no new features or language versions before then. Apps running on Windows in a Consumption plan are explicitly not affected yet, so keep the Linux qualifier when you repeat this to your platform team.

That changes the maximum execution time picture completely:

Flex Consumption: 30 minutes by default, unbounded maximum
Premium (Elastic Premium): 30 minutes by default, unbounded maximum
Dedicated (App Service): 30 minutes by default, unbounded with Always On
Container Apps: 30 minutes by default, unbounded when min replicas is 1 or more
Consumption (legacy): 5 minutes by default, 10 minutes maximum

The famous ten-minute wall sits on exactly one row of that list, and it is the row Microsoft is retiring. If you are still hitting it, the answer is a hosting change rather than an architecture change. Moving to Flex is not a plan toggle (there is no in-place migration, a point that comes back later), but it is a redeploy, not a rewrite.

Everything after this section is about the limits that a redeploy will not fix.

The enterprise ceiling

Timeouts that are unbounded until they aren't

"Unbounded" carries two asterisks, and both of them bite in production.

The first is that no HTTP response can take longer than 230 seconds, on any plan, ever. That number is not a Functions setting: it is the Azure Load Balancer idle timeout sitting in front of your app, so functionTimeout cannot raise it and no host.json value will help. A function that runs for 25 minutes is fine. A function that runs for 25 minutes and answers an HTTP request at the end of it is not, and the failure is asymmetric: the caller's connection is dropped while your function carries on executing, entirely unaware that nobody is listening.

The documented way around it is the Durable async HTTP pattern: accept the request, start the work, hand back somewhere to poll.

[Function("StartMonthlyReport")]
public static async Task<HttpResponseData> Start(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "reports/monthly")]
        HttpRequestData req,
    [DurableClient] DurableTaskClient client)
{
    var request = await req.ReadFromJsonAsync<ReportRequest>()
        ?? throw new InvalidOperationException("Report request body was empty.");

    string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
        nameof(MonthlyReportOrchestrator), request);

    // 202 Accepted plus statusQueryGetUri, terminatePostUri, and friends.
    // The HTTP request finishes in milliseconds. The report does not.
    return await client.CreateCheckStatusResponseAsync(req, instanceId);
}

This works, and it is the right pattern. Count what it costs, though. You have added a Durable task hub with its own storage account and transaction bill, a status contract every caller now has to implement, a polling loop in each of those callers, and an orchestration whose replay semantics you have to test differently from ordinary code (the previous series spent a whole article on exactly that). A 40-second synchronous call became a distributed workflow because of a load balancer setting.

The second asterisk is that unbounded execution is bounded anyway. Scale-in gives a running instance a 60-minute grace period on Flex and Premium; platform updates give it 10 minutes. A three-hour job is not safe on Functions even where the docs print the word "unbounded" next to your plan. Premium stacks three more interrupts on top: an idle timer stops the worker after 60 minutes with no new executions, scale-in can shut a worker down after the same 60 minutes, and a slot swap terminates executions on both the source and target slots.

Read together, those give you long-running execution, not guaranteed-to-finish execution. If your workload cannot tolerate being killed mid-flight, you owe it checkpointing regardless of which plan you are on.

4 GB is the ceiling, and there are only three rungs

Flex Consumption offers three instance sizes. Not a range, not a slider. Three:

512 MB with 0.25 CPU cores
2,048 MB with 1 core (the documented default recommendation)
4,096 MB with 2 cores

There is a further 272 MB of platform buffer memory on top of each size, which you are not billed for. Nothing exists between 512 MB and 2,048 MB, so an app whose working set peaks at 700 MB buys the 2 GB rung and pays for the other 1.3 GB on every billed second of execution.

Container Apps allocates the same two resources in fixed pairs, from 0.25 vCPU / 0.5 GiB up to 4.0 vCPU / 8.0 GiB in quarter-core steps, with memory always at 2 GiB per vCPU. That is 16 rungs where Flex has three, and it goes twice as high on both axes. (A Consumption-only Container Apps environment caps at 2 cores / 4 GiB, which lands it back at parity with Flex, so the workload profile you choose matters.)

The consequence is worth stating without hedging: a serverless workload that needs more than 4 GB has no Functions answer. Your options are Premium EP2 (2 cores / 7 GB) or EP3 (4 cores / 14 GB), or Dedicated. All of them bill continuously whether or not anything is executing, which means the memory limit quietly converts a serverless cost model into an always-on one. That is not a scaling decision anymore. It is a budget decision, made for you by an instance-size table.

Cold start is a scale-out problem, not an idle problem

Almost every cold-start discussion frames it as an idle problem: nobody called your app for a while, the platform reclaimed the instance, the next caller pays for a fresh start. That framing is why cold start gets dismissed as a low-traffic concern, and it hides the case that actually breaks a latency SLA.

Reframe it. Every instance the platform adds during a spike is a cold start. A busy app that goes from 20 instances to 60 in ninety seconds just paid 40 cold starts, and it paid them at the exact moment its p99 was already under pressure.

That would be tolerable if the platform added instances at a constant rate. It does not. Flex Consumption applies a documented scale-out rate curve where the per-interval allowance shrinks as your app grows:

When an app is running only a few instances, the allowance is at its largest... As an app grows to run more and more instances, the platform grants each additional batch more gradually.

Microsoft deliberately does not publish per-interval numbers and tells you not to design against them, so take the shape rather than the arithmetic: scale-out is fastest when you need it least, and most measured when you need it most. That is the inverse of what a burst-shaped traffic profile asks for.

There is one documented exemption. Always-ready instances bypass the on-demand scale-out rate entirely, which makes them the supported way to pre-provision for a burst you can predict. The default is 0, so a Flex app that nobody has configured behaves like legacy Consumption at idle, and plenty of Flex apps in production have never been configured.

# Two HTTP instances stay running ahead of the burst. They are exempt from the
# on-demand scale-out rate curve, and they bill whether traffic arrives or not.
az functionapp create \
  --name orders-api \
  --resource-group rg-orders \
  --storage-account stordersapi \
  --flexconsumption-location eastus \
  --runtime dotnet-isolated \
  --instance-memory 2048 \
  --always-ready-instances http=2

# Push per-instance HTTP concurrency past the size-derived default of 16,
# so each warm instance absorbs more of the spike before a new one is needed.
az functionapp scale config set \
  --name orders-api \
  --resource-group rg-orders \
  --trigger-type http \
  --trigger-settings perInstanceConcurrency=32

Those two commands are the whole mitigation, and they are also the whole trade. Always-ready capacity has its own billing meter that runs continuously, and unlike on-demand execution it gets no free grant at all. You are buying your way off the scale curve, which is a reasonable thing to do; it is just no longer scale-to-zero.

Premium's equivalent works differently enough to be worth naming, because the word "prewarmed" oversells it. A prewarmed instance only becomes active once all your currently active instances are in use, so it is a buffer covering the next instance, not a reserve that absorbs a burst. Always-ready is Premium's floor; prewarmed is a single step of headroom above it.

One number is missing from all of this on purpose. Microsoft publishes no cold-start latency figure for any plan, and the independent measurements floating around are browser-timed, meaning they include connection setup and TLS on top of whatever the platform actually did. If you need a number for an SLA conversation, measure your own app on your own plan and label it as your measurement. Anything else you have read is somebody else's network.

The scaling gaps nobody reads until they page you

The headline scale-out numbers look generous. Flex Consumption reaches 1,000 instances; Container Apps reaches 1,000 replicas, or 300 if you created the app from the portal; Premium reaches 100 on Windows and somewhere between 20 and 100 on Linux depending on the region; Dedicated manages 10 to 30, or 100 on an App Service Environment; legacy Consumption does 200 on Windows and 100 on Linux.

Five things sit underneath those numbers, and each one has paged somebody.

The defaults are nowhere near the maximums. Flex ships with a default max of 100, not 1,000. Container Apps ships with a default max of 10 replicas. Neither of those is the number in the marketing table, and neither changes because your traffic grew.

Your configured maximum is a request, not a promise. Flex Consumption enforces a regional subscription memory quota of 250 cores and 512,000 MB per region per subscription, shared across every Flex app you run there. Do the arithmetic on a single app configured at the recommended 2,048 MB size: 250 instances multiplied by 2,048 MB is 512,000 MB, and the quota is gone. One 512 MB app at its full 1,000 instances exhausts it just as completely. Your app can be sized correctly, configured correctly, and still stop scaling because a different team's Flex app in the same region and subscription got there first.

Private endpoints cap you at 20 instances. HTTP triggers behind a private endpoint scale no further than that, whatever your plan says. It is a networking decision, usually made by a different team, that silently becomes a throughput decision.

Concurrency is derived from a size you may have picked for memory reasons. Default per-instance HTTP concurrency on Flex is 4 requests at 512 MB, 16 at 2,048 MB, and 32 at 4,096 MB. (Python is 1 at every size.) The tie between the two settings is the trap: set concurrency explicitly and it stops tracking instance size forever, so the next engineer who resizes the app for memory gets none of the concurrency change they expected.

The last one is the least known and the most likely to invalidate a capacity plan. Per-function scaling is really per-group: all your http triggers scale together on shared instances, all your Event Grid blob triggers scale together, all your durable triggers scale together, and everything else scales individually. Your configured maximum then applies per group, not per app. An app set to 100 instances with HTTP, blob, and Durable triggers is not budgeted for 100 instances. It is budgeted for up to 300, and the bill arrives shaped accordingly.

Organizational challenges

The function app is the unit of scale, deployment, and blame

Resource limits are negotiable. You can buy your way past most of them, and the previous section was mostly about what that costs. The limits in this section are different, because no amount of money moves them: they are shape, not size.

Start with the one that surprises people who came to Functions from microservices. The function app, not the function, is the unit of scale. On Consumption and Premium every function inside an app shares the host instance and therefore shares its memory allocation. One PDF-rendering function that peaks at 3 GB drags every lightweight HTTP endpoint in the same app up onto a larger SKU with it, and those endpoints pay the higher rate on every instance, forever, for a function they never call.

The same boundary is also the unit of deployment. Two teams sharing a function app ship together and roll back together, which means one team's bad Friday afternoon is the other team's incident. They also share a single host.json, so logging levels, concurrency settings, retry policy, and extension bundle versions are one negotiation rather than two decisions.

And it is the unit of blame, which is the part that shows up in retrospectives rather than architecture diagrams. When the app throttles, the telemetry says the app throttled. Attributing that to the function that consumed the instance is work you do by hand, after the fact, under time pressure.

The obvious fix is to split the app, one per team. That fix has a bill attached, because splitting apps multiplies plans, and on Premium each new plan brings its own always-on floor. The organizational boundary you want costs real money to draw.

Premium couples every team through the plan

Premium lets you put up to 100 function apps in one App Service Plan, which reads like a cost optimization and behaves like a coupling.

The mechanic is in the Premium plan documentation, stated plainly. A plan's minimum instance count is the maximum of its apps' always-ready counts. Put three apps in one plan, set two of them to 1 always-ready instance and the third to 5, and the plan minimum becomes 5: "the minimum number of instances for which your plan is billed."

Follow that through to the org chart. The checkout team has a latency requirement and sets 5 always-ready instances to meet it. The reporting team, sharing the plan, is now billed against a floor of 5 as well, and nothing in the reporting team's own configuration explains why their number went up. Their always-ready setting still reads 1. The cost lives in a resource neither team owns, driven by a value neither team can see from where they work.

Flex Consumption inverts the whole arrangement: one function app per plan, full stop. That kills the noisy-neighbour problem outright, and it kills the shared-floor billing with it. It also removes a capability in the same stroke, which is the subject of the next section.

Deployment coupling and the doors that only open one way

Flex Consumption has no deployment slots. Premium gives you 3, legacy Consumption 2, Dedicated up to 20, and the recommended plan gives you none. If your zero-downtime story was "deploy to staging, warm it, swap," that story needs rewriting. The replacement is the rolling update site strategy, and two facts about it shape your release process before its behaviour does. It is generally available in East Asia, West Central US, North Central US, and West US 2 while the rollout to every other region continues, so check your region. And it can only be configured from Bicep or ARM templates, not the CLI, not the portal, and not VS Code, so whoever owns your deployment templates owns this decision.

There is also no in-place migration into or out of Flex Consumption. You create a new function app, redeploy, and cut over: DNS, keys, private endpoints, managed identity role assignments, the lot. That makes the plan choice a one-way door in both directions, which is an unusual property for a setting that looks like a dropdown.

The third door is the one that turns a runtime concern into a deployment failure. Flex Consumption times out app initialization after 30 seconds, the value is not configurable, and when you exceed it the symptom is a gRPC System.TimeoutException rather than anything mentioning startup. Everything your Program.cs does eagerly happens inside that window.

var builder = FunctionsApplication.CreateBuilder(args);

builder.ConfigureFunctionsWebApplication();

// Cheap to register, expensive to build. This one opens a blob connection and
// blocks the startup thread on it, inside the same 30-second budget.
builder.Services.AddSingleton<IOrderPricingCache>(sp =>
    OrderPricingCache.LoadFromBlobAsync(sp.GetRequiredService<BlobServiceClient>())
        .GetAwaiter().GetResult());

builder.Services.AddDbContext<OrderDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("Orders")));

builder.Build().Run();

A pricing cache that loads from blob storage in 8 seconds locally can take considerably longer from a cold instance against a throttled account, and there is no knob to buy more time. The fix is to make startup lazy (resolve the cache on first use, not at registration) rather than to make it faster, because "faster" has no floor you control. Discussion and workarounds are tracked in azure-functions-host#10482.

None of these three is a scaling limit. They are all deployment shape, and deployment shape is what your release process is built out of.

Debugging across triggers

One order arrives on an HTTP trigger, drops a message that a queue trigger picks up to reserve inventory, and finishes inside a Durable orchestration that handles fulfilment. That is one business transaction and three execution models. Split those functions across apps for the team-boundary reasons above and you have also split them across Application Insights resources, at which point no single query can see the whole transaction.

The Durable half of this is solvable, and the testing and monitoring article from the previous series covers the KQL for reconstructing one instance's history, including why isReplay has to be in the filter. That technique still works here. It just stops at the app boundary, because correlation across a queue hop is plumbing you write yourself and keep writing yourself at every trigger transition.

When migration makes sense

Every limit in the two sections above has a workaround, and most of those workarounds are defensible engineering. That is what makes the decision awkward. No single number tells you it is time to leave. What you get instead is a set of signals, and the useful question is not whether one of them is true but how many are true at once.

Signals you can check against your own system

You are chaining Durable Functions to dodge a limit rather than to model a workflow. An orchestration that exists because the business process genuinely has steps, compensations, and a lifetime measured in hours is Durable Functions doing the job it was built for. An orchestration that exists because a 40-second call cannot answer over HTTP is the 230-second constraint wearing a workflow costume, and you keep paying for the disguise in task hubs, polling contracts, and replay-aware tests.

A single function is choosing the instance size for every function beside it. If your size is set by one report renderer and the other eleven endpoints in the app are billed at that size on every instance they run on, the app boundary has stopped describing anything about your system except its worst-case memory.

Your working set has passed 4 GB, or your dependency list has passed what you can install without a container. Neither of those is a slope you can climb with configuration. There is no larger Flex instance to buy, and there is no package-install step in the sandbox.

Your p99 degrades during scale-out and always-ready capacity costs more than the latency is worth to the business. Always-ready instances work. The question is whether the number of them you need has quietly turned a serverless app into a fleet you are sizing by hand.

Team boundaries and function app boundaries have stopped lining up. The split that would fix it is the one that multiplies plans and billed floors, which is why it keeps getting deferred to next quarter. When the org chart and the resource graph disagree, one of them wins, and it is rarely the one on the wiki.

You need a capability the sandbox does not have. A GPU, a sidecar, an init container, mTLS between services. These are not limits you tune around. They are absences.

Counting matters more than any individual signal. One is a configuration problem: raise the instance size, buy always-ready capacity, split the app. Two or three usually mean a redesign inside Functions is still cheaper than leaving it. Once four or more are true, and the true ones are the structural kind (memory, boundaries, deployment shape), the choice is no longer between Functions and containers. It is between an accumulating pile of workarounds and one migration.

One more item belongs on that scale, and it is the point the plan section left hanging. Because Flex Consumption supports no in-place migration in either direction, any plan change is already a new function app, a redeploy, and a full cutover: DNS, keys, private endpoints, and every managed identity role assignment. Price the two moves next to each other. Getting from legacy Consumption onto Flex costs a new resource and a cutover. Getting onto Container Apps costs a new resource, a cutover, and a Dockerfile. Most of the migration people hesitate over is work they committed to the moment Flex became the recommended plan.

The case for staying

Most Functions code in production should not move, and the reasons are better than inertia.

Event-driven glue is what the programming model is for, and nothing about a container improves it. A queue trigger that validates a message and writes a row is a dozen lines with a binding and a hundred with a hosted service, a client, a lease policy, and a retry loop you now own and now have to test. Scheduled cleanup jobs, webhook receivers, and low-traffic internal APIs land in the same place: the trigger is the interesting part of the code, and Functions hands it to you already written.

The free grants are not a rounding error either. Flex Consumption includes 250,000 executions and 100,000 GB-seconds per subscription per month. A 2 GB function that runs for two seconds burns 4 GB-seconds, so the grant absorbs 25,000 of those runs before execution time costs anything at all. Container Apps has a grant too, 180,000 vCPU-seconds, 360,000 GiB-seconds, and 2,000,000 requests, but a single replica pinned at one vCPU consumes 180,000 vCPU-seconds in about 50 hours. A grant sized against bursts and a grant sized against a continuously running replica behave very differently by the end of the month.

Then there is duty cycle, which the next section turns into arithmetic. If your workload is spiky, short, and idle for most of the day, Functions is not a compromise you are tolerating. It is the correct answer, and a Container Apps replica held at one min replica would be a downgrade you paid extra for.

The last reason never appears on a pricing page. A migration costs your team its next two quarters: a new deployment pipeline, new observability wiring, new runbooks, and a set of failure modes nobody has met yet. Spending that to escape a limit you could have configured your way past is the most expensive mistake available anywhere in this decision.

The cost crossover, honestly

The memory section left a claim unpaid. Every escape hatch above 4 GB bills continuously; so does every serious mitigation for scale-out latency; so does a Container Apps replica with min replicas set to 1. Once scale-to-zero has left both sides of the comparison, the argument stops being serverless against containers and becomes arithmetic about what fraction of the month your code is actually running.

The anchor number first. One always-on Premium EP1 instance (1 core, 3.5 GB) costs about $157.71 a month in East US before a single execution runs, at $0.173 per vCPU-hour and $0.0123 per GiB-hour across 730 hours. A Container Apps replica of comparable shape (1 vCPU / 2 GiB), billed at active rates every second of that month, is $78.84. A Flex always-ready baseline instance at 2,048 MB is $21.02, plus execution time on top of it. Those three numbers are most of the reason Premium keeps showing up in cost reviews.

The crossover is more useful than any of them, because it tells you which side of the line your workload sits on. Here is the whole calculation, using pay-as-you-go rates pulled from the Azure Retail Prices API on 10 August 2026:

East US
  Container Apps, 1 vCPU / 2 GiB, billed every second the replica exists:
    1 x $0.000024  +  2 x $0.000003   =  $0.000030 per second

  Flex Consumption, 2,048 MB instance, billed only while code executes:
    2 x $0.000026                     =  $0.000052 per second of execution

  Break-even duty cycle: 0.000030 / 0.000052 = 57.7%

Above roughly 58% duty cycle in East US, an always-on Container Apps replica costs less than Flex Consumption on demand for the same compute shape. Below it, scale-to-zero wins.

Now run the identical calculation in another region, because this is the part that catches people out. Flex Consumption is priced the same in West Europe as in East US, to the sixth decimal place. Container Apps is not: vCPU is 42% more expensive there.

West Europe
  Container Apps, 1 vCPU / 2 GiB:
    1 x $0.000034  +  2 x $0.000004   =  $0.000042 per second

  Flex Consumption, 2,048 MB:         =  $0.000052 per second of execution

  Break-even duty cycle: 0.000042 / 0.000052 = 80.8%

The same two services, the same workload shape, and the crossover moves from 58% to 81%. A quoted crossover figure with no region attached is not a rough number; it is a wrong one.

Both of those percentages are my arithmetic, not a Microsoft-published figure, and they are only as good as their inputs. They use list pay-as-you-go rates for two regions on one date. They exclude the free grants, per-execution and per-request charges, storage, Application Insights, networking, and the Container Apps idle rate (vCPU idle is eight times cheaper than active in East US, while memory is billed at the same rate idle or active in both regions). They also assume the two platforms get equivalent throughput from a core, which is a property of your workload rather than of Azure. Treat 58% as a starting point for modelling your own numbers, not a constant to quote in a design review.

One billing mechanic deserves its own sentence, because it wrecks the intuition behind these percentages for short functions. Flex Consumption's minimum billable execution period is 1,000 ms, rounding up to the nearest 100 ms after that. A function that finishes in 40 ms bills as if it took a full second, which means an app made of very short executions is running at a much higher effective duty cycle than its telemetry suggests.

The cloud-native alternative

Naming the destination matters less than naming what it fixes. Container Apps is the runtime, Dapr is the distributed-systems toolkit that replaces what bindings were doing, and Aspire is how you keep the local development loop from getting worse. Only the first of those is a hosting decision.

Container Apps

Take the limits from the first half of this article and check them off one at a time. Execution time is unbounded when min replicas is 1 or more. CPU and memory come in sixteen steps from 0.25 vCPU / 0.5 GiB to 4.0 vCPU / 8.0 GiB rather than three, so an app that peaks at 700 MB buys a 0.5 vCPU / 1 GiB pairing instead of the 2 GB rung. Scaling is KEDA, which means HTTP and TCP rules, CPU and memory rules, and any custom scaler in the KEDA catalogue, instead of platform heuristics you cannot inspect. Deployment safety comes from revisions with traffic splitting, which does blue/green and A/B rather than the single staging swap slots gave you. Custom images and GPU exist. Sidecars and init containers exist.

Three things go the other way and should be said in the same breath. The 230-second HTTP wall does not vanish, it moves: Container Apps times a request out at its own ingress after 240 seconds, and the only supported way past that is premium ingress on a dedicated workload profile, where the limit becomes a configurable idle timeout of up to 30 minutes. Ten extra seconds is not an answer to the problem that sent you looking. An app that scales on CPU or memory load cannot scale to zero at all, so the KEDA flexibility buys you a floor in exactly the cases where you were hoping it would not. And the container image limit is 8 GB per replica on a Consumption workload profile, which is generous until somebody bakes a model into the image.

The fact that de-risks this whole conversation belongs here rather than at the end: function apps run inside Container Apps. You deploy the same app as a container app with --kind functionapp, and the platform generates KEDA scale rules from the triggers you already have instead of making you rewrite them as scale rules by hand.

# The same function app, hosted on Container Apps. The triggers stay; the platform
# translates them into KEDA scale rules unless allowScalingRuleOverride opts out.
az containerapp create \
  --name orders-api \
  --resource-group rg-orders \
  --environment cae-orders \
  --kind functionapp \
  --storage-account stordersapi \
  --image acrorders.azurecr.io/orders-api:2026.08.14 \
  --ingress external \
  --target-port 80 \
  --min-replicas 1 \
  --max-replicas 50

That makes migration incremental rather than a cutover weekend, which matters more than any capability in the list above.

It is not free of holes, and the gaps are documented. Durable Functions autoscaling only works with the MSSQL or Durable Task Scheduler storage providers, so a Durable app on the default Azure Storage provider scales on the rules you write yourself. Blob-trigger autoscaling requires Event Grid. There is no autoscaling at all for the Azure Cache for Redis or Azure SQL triggers. Deployment slots do not exist, and neither do portal-generated Functions access keys. If your release process or your on-call tooling depends on any of those, the container-hosted version is a step backwards until you replace them.

Dapr

Triggers and bindings are an excellent deal right up to the day you want the same message-publishing code to run inside a service, a console app, and a test. Then the binding turns out to be a property of the host rather than of your code, and there is no version of that code which runs anywhere else unchanged.

Dapr moves that contract into a sidecar your app talks to over HTTP or gRPC. Which broker, which state store, which secret store: all of it becomes component configuration attached to the environment instead of an attribute on a method signature. The building blocks that are GA in Container Apps cover service invocation, state management, publish/subscribe, bindings, actors, secrets, and configuration, which is most of what a bindings-heavy function app was doing, plus a virtual actor model that Functions has no answer for outside Durable entities.

The sidecar is not everywhere, though. Dapr is not supported for Container Apps jobs, so anything you were going to model as a scheduled job runs without it. Actor reminders are the other edge: they require min replicas of at least 1, which removes scale-to-zero from any actor-based design before you have written a line of it.

.NET Aspire

Aspire is the least dramatic piece here and the easiest to justify, because adopting it does not require you to have containerized anything. Aspire integrates with Azure Functions directly, so an existing function app can join an AppHost alongside the queue, the database, and whichever service you are building next, and a single F5 starts all of them with the connection wiring already generated. That makes Aspire the bridge into this stack rather than the reward at the end of it: you can run it against the architecture you have while you are still deciding whether to change it.

What the rest of this series does

Seven articles follow this one.

Introduction to Dapr for Azure Developers: the building blocks, and how they map onto Azure services you already run.
Building Your First Dapr + Web API Service: .NET 10 Minimal APIs, service invocation, and state management end to end.
.NET Aspire: Orchestrating Cloud-Native Apps: AppHost, service defaults, the local loop, and wiring Dapr into it.
Migration Strategy: Incremental Transition Patterns: strangler fig, Functions and containers running side by side, and the data migration nobody budgets for.
Enterprise Architecture: Front Door, API Management, Container Apps: traffic management, gateway policy, and the Managed Identity and RBAC patterns underneath.
Infrastructure as Code: Terraform for Your New Stack: module structure for Container Apps, networking, and CI/CD.
The Complete Picture: A Reference Architecture: the assembled stack, what it costs to operate, and what did not survive contact with production.

Conclusion

The ceiling moved, and it moved somewhere harder to see. When the argument was about time you could tell the moment you hit it: a job died, the log said so, and the fix had a name. What replaced it never announces itself. An instance-size table quietly sets your monthly bill. A plan boundary decides which two teams share an incident, and whether you have a deployment slot determines what your release process is allowed to promise. None of those throw an exception, and all of them compound.

That is also why the honest answer for most readers is to stay put. If your functions are event-driven glue running for a few hundred milliseconds a few thousand times a day, the free grant probably covers the bill, everything above is theoretical, and migrating would buy you a Dockerfile and a worse on-call rota. The case for moving arrives when the workarounds stop being individual decisions and start being an architecture, when what you built to stay inside the hosting model has grown larger than the thing it hosts.

The rest of this series assumes you got there. Dapr for the building blocks, Aspire for the local loop, Container Apps for the runtime, Terraform to make it reproducible. None of it requires switching everything off first, which is the whole reason the migration article keeps Functions running alongside the containers rather than replacing them.

When your Functions app hit its ceiling, was it a resource limit (memory, timeout, scale-out) or an organizational one (two teams stuck inside one deployment)?

Testing Durable Functions and Production Realities

Martin Oehlert — Fri, 17 Jul 2026 09:30:23 +0000

An orchestrator that calls three activities is not a function you can test like any other, and the reason is one word the rest of this article keeps returning to: replay. The runtime re-runs the orchestrator body from the top every time the instance wakes up, rebuilding progress from recorded history, so the question a test has to answer is not "did it return the right value once" but "does it return the right value every time the same code runs against the same history." That reframes two demands at once. A mocked activity has to hand back the same result on every call, and the only way to prove the real replay engine agrees with your mock is to run that engine.

Those two demands split testing into two layers, and the split organizes everything below. Unit tests mock the orchestration context, run in milliseconds, and prove your branch and aggregation logic; no engine runs and nothing replays. Integration tests register the real orchestrator and activities with an in-memory host, so real replay, real serialization, and real activity execution all happen against your code. The root cause underneath both is determinism: an orchestrator body has to make the same decisions on every replay, and your tests either honour that constraint or they quietly lie about whether the workflow is correct.

The code under test is the order workflow from the DurableFunctionsDemo companion sample (isolated worker, .NET 10); the entity examples reuse the DurableEntitiesDemo sample from the previous article. Tests use xUnit and Moq, the stack Microsoft's own Durable testing guidance assumes.

Why replay makes orchestrator tests different

The tests target this orchestrator. It validates an order, creates it, sends a confirmation, and folds a trace id and a start time into the result.

[Function(nameof(OrderOrchestrator))]
public static async Task<string> RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<OrderRequest>()!;

    // Replay-safe substitutes: the runtime records these once and replays the
    // recorded value, so every replay sees the same timestamp and the same id.
    DateTime startedUtc = context.CurrentUtcDateTime;
    Guid traceId = context.NewGuid();

    var validated = await context.CallActivityAsync<bool>(
        nameof(ValidateOrderActivity), order);
    if (!validated)
        return "Order validation failed";

    var orderId = await context.CallActivityAsync<string>(
        nameof(CreateOrderActivity), order);

    await context.CallActivityAsync(nameof(SendConfirmationActivity), orderId);

    return $"{orderId} (trace {traceId:N}, started {startedUtc:O})";
}

Two properties of this function decide how it can be tested.

It replays. The runtime re-executes the body from the top after every awaited step completes, and again after a timer fires, an event arrives, or the host recycles. Awaits that already finished return their recorded results instead of running a second time, but the lines between them run again on each pass. The stretch from GetInput to the final return may execute many times over one instance's life.

It must be deterministic. Because every pass has to reach the same decisions, the body cannot read wall-clock time, mint random GUIDs, or touch I/O directly. That is why the sample reaches for context.CurrentUtcDateTime and context.NewGuid() instead of DateTime.UtcNow and Guid.NewGuid(): the runtime records each on the first pass and feeds the recorded value back on every later pass. A raw DateTime.UtcNow would produce a fresh timestamp each replay and diverge the history, at which point the runtime raises NonDeterministicOrchestrationException, though it does not detect every form of divergence.

Both properties land on your tests. Since the body can run more than once, any value a test feeds in has to stay stable across calls; a mock that returns a new GUID or timestamp each time makes the same test pass or fail depending on how many replays happened. And because the return string folds in traceId and startedUtc, a test that stubs those with changing values has nothing stable to assert against. The fix is the discipline the orchestrator already follows: pin every value the body sees to a constant.

Prevention comes before the test. The Durable Functions Roslyn Analyzer flags the common determinism violations (DateTime.Now, Guid.NewGuid(), direct environment or I/O access) at edit time, so a whole class of replay bugs never reaches a test run.

Unit testing orchestrators: replay-safe mocks

A unit test mocks TaskOrchestrationContext, stubs each activity call, invokes the orchestrator method directly, and asserts on what it returns. The happy path runs a valid order all the way to its confirmation.

[Fact]
public async Task Valid_order_runs_to_confirmation()
{
    var order = new OrderRequest(CustomerId: "cust-1", Sku: "sku-9", Quantity: 2);
    var context = new Mock<TaskOrchestrationContext>();

    context.Setup(x => x.GetInput<OrderRequest>()).Returns(order);
    context.Setup(x => x.CurrentUtcDateTime)
        .Returns(new DateTime(2026, 7, 17, 9, 0, 0, DateTimeKind.Utc));
    context.Setup(x => x.NewGuid())
        .Returns(Guid.Parse("00000000-0000-0000-0000-0000000000ab"));

    context.Setup(x => x.CallActivityAsync<bool>(
        It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync(true);
    context.Setup(x => x.CallActivityAsync<string>(
        It.Is<TaskName>(n => n.Name == nameof(CreateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync("ORD-cust-1-sku-9");
    context.Setup(x => x.CallActivityAsync(
        It.Is<TaskName>(n => n.Name == nameof(SendConfirmationActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).Returns(Task.CompletedTask);

    string result = await OrderOrchestrator.RunOrchestrator(context.Object);

    Assert.StartsWith("ORD-cust-1-sku-9", result);
}

The matcher that trips everyone is the activity name. CallActivityAsync does not take a string; it takes a TaskName struct, and the orchestrator's nameof(ValidateOrderActivity) converts to one implicitly. Write It.IsAny<string>() in the setup and it never matches, the mock returns default (here false), and the order "fails validation" for no visible reason. Match the struct instead: It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)). This is the single most common cause of a Durable orchestrator mock that "returns null." The third argument is TaskOptions; set it up as It.IsAny<TaskOptions>() even for calls that pass no options, because the overload Moq binds against always carries that parameter.

The values you return matter as much as the names you match. ReturnsAsync("ORD-cust-1-sku-9") is replay-safe: the same result on every call. Compare the version that reaches for real non-determinism.

// Flaky: a fresh GUID on every access, so the returned string is never the same twice.
context.Setup(x => x.NewGuid()).Returns(Guid.NewGuid);

Guid.NewGuid here is a method group, so Moq invokes it on each call and hands back a new value. The orchestrator folds that GUID into its return, so the test has no stable string to assert on; worse, the same stub under the real replay engine (the integration host in the next section) records a different history on each pass and breaks the run outright. Pin it to a fixed value and both problems vanish. The rule is the orchestrator's own rule pointed back at the test: every value the body sees has to be a function of its input, not of when or how often it ran.

The second thing this layer buys you is cheap branch coverage. Flip the validation result and assert the workflow short-circuits without ever creating an order.

[Fact]
public async Task Invalid_order_stops_before_creation()
{
    var order = new OrderRequest("cust-1", Sku: "", Quantity: 0);
    var context = new Mock<TaskOrchestrationContext>();
    context.Setup(x => x.GetInput<OrderRequest>()).Returns(order);
    context.Setup(x => x.CurrentUtcDateTime).Returns(DateTime.UnixEpoch);
    context.Setup(x => x.NewGuid()).Returns(Guid.Empty);
    context.Setup(x => x.CallActivityAsync<bool>(
        It.Is<TaskName>(n => n.Name == nameof(ValidateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>())).ReturnsAsync(false);

    string result = await OrderOrchestrator.RunOrchestrator(context.Object);

    Assert.Equal("Order validation failed", result);
    context.Verify(x => x.CallActivityAsync<string>(
        It.Is<TaskName>(n => n.Name == nameof(CreateOrderActivity)),
        It.IsAny<object>(), It.IsAny<TaskOptions>()), Times.Never);
}

The Verify(..., Times.Never) is the assertion that earns its keep: it proves the guard actually skipped order creation, rather than the test landing on a matching string by luck.

What this layer cannot do is run the real thing. The mocked context never replays, never serializes OrderRequest into history, never executes an activity. A test can pass here while the deployed orchestrator diverges on replay or fails to round-trip its state. Closing that gap is the job of the in-memory host, and it is the next section.

Testing activities and entities

Activities are where the replay rules stop applying. An activity runs once per invocation, does the real I/O, and has no history to reconstruct, so it tests like any other method: call it, assert on what it returns. The sample's validation activity is a pure function of its input.

[Fact]
public void Validate_rejects_empty_sku()
{
    Assert.False(ValidateOrderActivity.Run(new OrderRequest("cust-1", "", 1)));
    Assert.True(ValidateOrderActivity.Run(new OrderRequest("cust-1", "sku-9", 1)));
}

No mock, no context, no harness. That is the reward for pushing logic into activities: the moment code leaves the orchestrator it becomes ordinary and testable.

Real activities rarely stay pure. They call a database, an HTTP API, a blob container, and that dependency is exactly what you mock. Inject it through the constructor of an activity class (the isolated worker supports standard DI on [Function] classes) and hand the test a fake. The sample registers IPaymentGateway in Program.cs, and the worker resolves it into the activity's constructor.

public class PaymentActivities(IPaymentGateway gateway)
{
    [Function(nameof(ChargeCard))]
    public Task<bool> ChargeCard([ActivityTrigger] OrderRequest order)
        => gateway.ChargeAsync(order.CustomerId, order.Quantity);
}

[Fact]
public async Task ChargeCard_delegates_to_gateway()
{
    var gateway = new Mock<IPaymentGateway>();
    gateway.Setup(g => g.ChargeAsync("cust-1", 2)).ReturnsAsync(true);

    var activities = new PaymentActivities(gateway.Object);

    Assert.True(await activities.ChargeCard(new OrderRequest("cust-1", "sku-9", 2)));
}

Entities test differently again, and the difference is worth pausing on. An entity is never replayed, so its operations run arbitrary code, which makes the logic itself the most ordinary thing in this article to check. The catch is access. On a class-based TaskEntity<TState>, the State property is protected (verified against Microsoft.DurableTask.Abstractions 1.24.1), so a test cannot reach in and seed it before calling an operation. A small test-only subclass exposes the seam.

private sealed class TestableCounter : Counter
{
    // Counter.State is protected; widen it to the test so we can seed prior state.
    public int Seed { get => State; set => State = value; }
}

[Fact]
public void Add_accumulates_onto_existing_state()
{
    var counter = new TestableCounter { Seed = 7 };
    counter.Add(5);
    Assert.Equal(12, counter.Seed);
}

Counter.Add is the previous article's public void Add(int amount) => this.State += amount;; the subclass only widens State for the test.

That same test names a design fact worth asserting on directly: Add is not idempotent. Run it twice and you get 12 then 17, not 12 both times. Entity delivery is at-least-once, so an accumulating operation that runs twice on a redelivery double-counts. An operation you can prove idempotent (Reset, or a SetQuantity(x) that assigns instead of adds) survives redelivery unchanged, and that property deserves a test of its own.

What none of these unit tests touch is serialization. Setting Seed in memory never round-trips State through System.Text.Json, so the case-sensitivity and dictionary-key-casing traps from the previous article stay invisible here. Proving the state survives a real round-trip needs the in-memory host, which runs the actual serializer against your entity. That is where the next section goes.

Integration testing with DurableTaskTestHost

The mocked-context test proved the branch logic and never ran the engine. This test does the opposite: it starts the real orchestration engine in-process, schedules the workflow, and waits for it to finish. DurableTaskTestHost (from Microsoft.DurableTask.InProcessTestHost) hosts that engine in memory, with no Azure Storage, no emulator, and no sidecar to install.

[Fact]
public async Task Valid_order_completes_with_order_id()
{
    await using DurableTaskTestHost host = await DurableTaskTestHost.StartAsync(tasks =>
    {
        // Register the real orchestrator. The typed-input overload deserializes the
        // scheduled input to OrderRequest, so context.GetInput<OrderRequest>() inside
        // the orchestrator round-trips through the real serializer, exactly as in production.
        tasks.AddOrchestratorFunc<OrderRequest, string>(
            nameof(OrderOrchestrator),
            (ctx, _) => OrderOrchestrator.RunOrchestrator(ctx));

        // Register the production activity logic by name so CallActivityAsync executes it.
        tasks.AddActivityFunc<OrderRequest, bool>(
            nameof(ValidateOrderActivity), (_, order) => ValidateOrderActivity.Run(order));
        tasks.AddActivityFunc<OrderRequest, string>(
            nameof(CreateOrderActivity), (_, order) => CreateOrderActivity.Run(order));
        tasks.AddActivityFunc<string>(
            nameof(SendConfirmationActivity), (_, _) => { /* side-effect only */ });
    });

    string instanceId = await host.Client.ScheduleNewOrchestrationInstanceAsync(
        nameof(OrderOrchestrator), new OrderRequest("cust-1", "sku-9", 2));

    OrchestrationMetadata result = await host.Client.WaitForInstanceCompletionAsync(
        instanceId, getInputsAndOutputs: true);

    Assert.Equal(OrchestrationRuntimeStatus.Completed, result.RuntimeStatus);
    Assert.StartsWith("ORD-cust-1-sku-9", result.ReadOutputAs<string>());
}

The shape is register, schedule, wait, assert. StartAsync takes a registration callback where you add the production orchestrator and activities; the standalone SDK registers them by name with AddOrchestratorFunc/AddActivityFunc (the Azure Functions worker discovers [Function] methods for you at runtime, so this manual step is the test-side equivalent). host.Client is a full DurableTaskClient, the same type the production HTTP starter uses, so scheduling, status queries, terminate, and raise-event all work here too. WaitForInstanceCompletionAsync(getInputsAndOutputs: true) blocks until the instance reaches a terminal state and pulls the output back so ReadOutputAs<string>() can deserialize it.

This is the layer that earns its runtime. The typed registration forces OrderRequest through System.Text.Json on the way in, GetInput<OrderRequest>() deserializes it on the way out, and if a [JsonPropertyName] or a case mismatch broke that round-trip the orchestration would land in Failed, not Completed. That is exactly the class of bug the mocked-context test in the previous section cannot see, because its GetInput handed back the in-memory object it was told to return.

The failure path is one assertion away. Pull that registration block into a StartHostAsync() helper so both tests share it, then schedule an order with an empty SKU: the real ValidateOrderActivity returns false, the orchestrator short-circuits, and the instance completes with the validation message rather than throwing.

[Fact]
public async Task Invalid_order_completes_with_validation_failure()
{
    await using DurableTaskTestHost host = await StartHostAsync();

    string instanceId = await host.Client.ScheduleNewOrchestrationInstanceAsync(
        nameof(OrderOrchestrator), new OrderRequest("cust-1", "", 0));

    OrchestrationMetadata result = await host.Client.WaitForInstanceCompletionAsync(
        instanceId, getInputsAndOutputs: true);

    Assert.Equal(OrchestrationRuntimeStatus.Completed, result.RuntimeStatus);
    Assert.Equal("Order validation failed", result.ReadOutputAs<string>());
}

Two layers, not two choices. Unit tests stay for speed and branch coverage; DurableTaskTestHost covers whole-workflow replay and serialization. One honest caveat on lineage: the classic DTFx LocalOrchestrationService (in DurableTask.Emulator) offered a similar in-memory host for the older API, but it is legacy. New isolated-worker code targets DurableTaskTestHost.

Storage in production: Azure Storage vs Netherite

The tests above ran against an in-memory engine. Production runs against a storage provider, and the provider you pick decides both your throughput ceiling and your bill. The question is narrower than "which is faster": it is at what volume the cheaper-per-event provider is worth its fixed cost floor.

Start with where the money goes on the default provider. Azure Storage drives execution through queues, records status and history in Tables, and distributes instances across workers with blobs and blob leases. Payloads over roughly 45 KB serialized are compressed and offloaded to a blob in <taskhub>-largemessages, with a reference left in the row. The cost mechanic is the part worth internalizing: every orchestration event is Storage transactions. Each activity call is a queue put and a queue get; each state change is a Table write. A three-activity orchestration generates well over a dozen billable transactions, and a fan-out that schedules two hundred activities multiplies that. At low volume the transaction bill is noise. At high fan-out it becomes the dominant line item, and per-transaction billing is what accumulates. For performance-sensitive apps, give Durable its own storage account rather than sharing AzureWebJobsStorage, and prefer a general-purpose v1 account (v2 can bill Durable's transaction-heavy pattern at a higher rate).

Netherite attacks that transaction bill directly. Built by Microsoft Research for high throughput, it can raise the ceiling by more than an order of magnitude. It does so by changing the storage shape: an Azure Event Hubs namespace moves messages between partitions, and a Storage account holds durable state through FASTER logs. Many small Storage transactions become batched Event Hubs traffic plus fewer, larger Storage operations. That trade buys better price-performance at scale, but it adds a fixed Event Hubs cost floor (the provisioned throughput units) that a low-volume app pays whether or not it uses the capacity.

The constraints are the part not to bury. Netherite is retiring: support ends March 31, 2028. It is not supported on the Flex Consumption plan, and it does not support identity-based connections, so a managed-identity requirement rules it out. Switching providers does not migrate task-hub data; you start from a fresh, empty task hub. For those reasons Microsoft steers new high-throughput work toward the Durable Task Scheduler, a managed, push-based gRPC backend with the highest throughput of the three, a built-in instance dashboard, managed-identity support, and no storage account to babysit.

The decision rule is measure, do not guess. Netherite pays off only above the throughput threshold where its cheaper per-event handling outweighs the Event Hubs floor; below that line Azure Storage is both cheaper and simpler. There is no published ops-per-second figure to copy, because the crossover depends on your fan-out width, payload sizes, and event rate. Benchmark your own workload before switching an existing app, and for anything greenfield, evaluate the Durable Task Scheduler rather than adopting a backend with a retirement date.

Monitoring in Application Insights

A test proves the workflow was correct when you shipped it. Monitoring answers the harder production question: which of ten thousand running instances failed last night, and where. Application Insights is the recommended surface, because the Durable extension already emits a tracking event for every lifecycle transition (scheduled, started, awaited, completed, failed), and those events land in the traces table where KQL can slice them.

The first instinct in production is usually the wrong one: find every failed orchestration in a window. Each tracking event carries the orchestration's state in a custom dimension, so the filter is a single predicate.

traces
| where timestamp > ago(24h)
| where customDimensions["prop__runtimeStatus"] == "Failed"
| project timestamp,
          instanceId = customDimensions["prop__instanceId"],
          functionName = customDimensions["prop__functionName"],
          reason = customDimensions["prop__reason"]
| order by timestamp desc

Once you have an instance ID, the next query reconstructs what that one instance actually did. This is where a Durable-specific trap shows up. Because the orchestrator replays, the same log line appears many times, so a raw query returns the history several times over. Two dimensions clean it up: isReplay filters out the replayed passes, and ordering by sequenceNumber after timestamp puts the events in execution order.

traces
| where customDimensions["prop__instanceId"] == "the-instance-id"
| where customDimensions["prop__isReplay"] == "False"
| project timestamp,
          functionType = customDimensions["prop__functionType"],
          functionName = customDimensions["prop__functionName"],
          state = customDimensions["prop__state"],
          sequenceNumber = toint(customDimensions["prop__sequenceNumber"])
| order by timestamp asc, sequenceNumber asc

Sort by timestamp first and sequenceNumber second, in that order. sequenceNumber resets to zero when the host restarts, so it disambiguates events inside one host lifetime but cannot order events across a restart on its own.

A few configuration facts change what these queries can see. The relevant log category is Host.Triggers.DurableTask in host.json; its default level emits non-replay tracking events, and raising it to Warning trims volume while logReplayEvents: true turns the replay noise back on only when you are debugging a replay. Inputs and outputs are not logged by default, for cost and PII reasons; only byte counts appear unless you opt in with traceInputsAndOutputs. The Functions runtime also samples telemetry by default, which can silently drop lifecycle events during a burst, so tune sampling in host.json if instances go missing from your results.

For latency rather than failure, turn on distributed tracing V2: set distributedTracingEnabled: true and version: "V2" under durableTask.tracing in host.json (isolated worker needs the extension at v1.4.0 or later). App Insights then draws an end-to-end Gantt chart in Transaction Search that correlates the orchestration with each activity and sub-orchestration, which is the fastest way to spot the one slow step in a long workflow.

The thread back to the rest of this article is the log line inside the orchestrator. Because the body replays, a plain logger.LogInformation(...) fires on every pass and floods the trace with duplicates. context.CreateReplaySafeLogger("Name") suppresses the replayed emissions so each line logs once. The honest limit: it suppresses replay duplicates, not duplicates from a full re-execution (a host restart or an expired queue visibility timeout runs the body again from scratch, and those lines log again). Alongside App Insights, the Durable Functions Monitor gives a UI over instance state, and the Roslyn Analyzer remains the cheapest observability of all: it catches the determinism bug at edit time, before there is anything to monitor.

Conclusion

One word carried this whole article: replay. It is why a mocked activity has to return the same value on every call, why the mocked-context test can prove branch logic but never catches a serialization bug, and why the in-memory host has to exist at all. Testing Durable code is not one activity but a stack: fast mocked-context units for branch coverage, DurableTaskTestHost for the replay and serialization the mocks cannot exercise, and plain method calls for activities and entities where replay never applied. Production sits on top of that stack, where the storage provider decides your bill and App Insights tracking events decide how fast you find the instance that broke.

This was the last of five. The series started with triggers and the isolated worker, moved through orchestration patterns and fan-out, spent an article on entities and their at-least-once delivery, and ends here on proving the whole thing works before and after it ships. The connective tissue was always determinism: every constraint, from context.NewGuid() to the replay-safe logger, exists so the runtime can re-run your code and get the same answer.

When an orchestrator misbehaves in production, which do you reach for first: an App Insights tracking-event query to find where it failed, or a local DurableTaskTestHost run to reproduce the replay on your machine?

Durable Entities: Stateful Actors Inside Azure Functions

Martin Oehlert — Fri, 10 Jul 2026 06:10:38 +0000

Parts 1 to 3 gave you a workflow with a beginning and an end: an orchestration starts, runs its ordered steps, and completes. So what do you reach for when the thing you are modeling never ends, when it is just a piece of keyed state that a shopping cart, a game session, or a per-user counter keeps mutating over its whole lifetime, and every one of those mutations has to be race-free without you writing a single lock? That is the other half of Durable Functions. An entity is a tiny stateful actor: long-lived, addressable by a string key, reactive to operations, and serialized so exactly one operation touches its state at a time. Orchestrations are code-as-workflow; entities are code-as-state, and the determinism rules that constrained your orchestrators do not apply here.

Defining an entity: from a counter to a cart

Three shapes define an entity in the isolated worker, and the class-based one is the shape to reach for first. You derive from TaskEntity<TState>, write your operations as public methods, and wire up a single [Function] method with an [EntityTrigger] to route incoming operations to them. Every code sample below is from the companion sample (isolated worker, .NET 10). Here is the canonical counter.

public class Counter : TaskEntity<int>
{
    public void Add(int amount) => this.State += amount;
    public Task Reset() { this.State = 0; return Task.CompletedTask; }
    public Task<int> Get() => Task.FromResult(this.State);

    [Function(nameof(Counter))]
    public static Task Run([EntityTrigger] TaskEntityDispatcher dispatcher)
        => dispatcher.DispatchAsync<Counter>();
}

Three things carry this. The persisted state is the inherited this.State property, and only State is serialized: any other field you hang off the class is scratch space that does not survive between operations. The operations are the public methods (Add, Reset, Get); each one reads or mutates this.State and returns void, a Task, or a Task<T> when it hands a value back. The [Function(nameof(Counter))] method is the registration itself: dispatcher.DispatchAsync<Counter>() takes the operation name off the incoming message and invokes the public method whose name matches. That trigger method is the whole wiring. There is no host.json opt-in and no attribute beyond [Function] and [EntityTrigger].

One naming rule crashes you at runtime if you miss it: never call the trigger method RunAsync. ITaskEntity already defines an instance RunAsync(TaskEntityOperation), confirmed against the compiled Microsoft.DurableTask.Entities types, and the collision throws an ambiguous-match error the moment an operation dispatches. It compiles without complaint (your trigger is a static method, so C# sees no clash), which is exactly why the failure waits until runtime. Name the trigger Run, or anything else, and the problem is gone.

Two lifecycle operations come for free. Override InitializeState to seed state on first access, before any operation has set it: protected override int InitializeState(TaskEntityOperation entityOperation) => 10; runs only while the state is still null. Deletion is setting state back to null; TaskEntity<TState> hands you an implicit Delete operation, and you override it by declaring your own Delete() method that sets this.State = null (which needs a nullable TState).

An int is the smallest interesting state. Entities earn their keep when the state is a whole object. Model a shopping cart and the same shape holds: the operations become AddItem, RemoveItem, and GetTotal, and this.State becomes a collection of line items that lives across every operation the cart ever receives.

public record CartLine(string Sku, int Quantity, decimal UnitPrice);

public class ShoppingCart : TaskEntity<List<CartLine>>
{
    protected override List<CartLine> InitializeState(TaskEntityOperation entityOperation)
        => new();

    public void AddItem(CartLine line) => this.State.Add(line);

    public void RemoveItem(string sku) => this.State.RemoveAll(l => l.Sku == sku);

    public Task<decimal> GetTotal() =>
        Task.FromResult(this.State.Sum(l => l.UnitPrice * l.Quantity));

    [Function(nameof(ShoppingCart))]
    public static Task Run([EntityTrigger] TaskEntityDispatcher dispatcher)
        => dispatcher.DispatchAsync<ShoppingCart>();
}

The cart reads almost the same as the counter, and that is the point: the state grew from an int to a List<CartLine>, but the operation-per-method model and the single trigger did not change. InitializeState returns an empty list on first access instead of relying on a null state, AddItem and RemoveItem each take their one argument, and GetTotal returns a Task<decimal> so an orchestration can call it for the value.

Which brings up the rule that bites hardest once state stops being a primitive: entity state serializes with System.Text.Json, not Newtonsoft. Property names are case-sensitive on the way back in, a Newtonsoft [JsonProperty] attribute is ignored (use [JsonPropertyName]), and dictionary keys come out camelCased by default. The operation methods carry their own constraints: at most one argument each, no overloads, no generic type parameters, and every argument and return value has to be JSON-serializable. Design the state type and the operation signatures around those limits from the start, because a serialization mismatch shows up as silently-empty state after a round-trip, not a compile error.

Talking to an entity: signal, call, and entity IDs

Every entity is addressed by an ID, and you build one with new EntityInstanceId(name, key). The name is the entity type, and it matches the entity function name (case-insensitively); the key is the string that picks one instance out of every entity of that type, so a user id, an order id, or a GUID. new EntityInstanceId(nameof(Counter), "myCounter") addresses the one counter keyed myCounter, which internally resolves to the form @Counter@myCounter. The ID is all you need to reach an instance, and if none with that ID exists yet, the first operation you send creates it.

There are two ways to reach an entity, and the difference decides your whole design. A signal is one-way and fire-and-forget: the task you await completes when the message is reliably enqueued, not when the entity has processed it, so there is no return value and no error to observe on the sending side. A call is a two-way round-trip: the sender waits for the operation to run and gets back its result, or the exception it threw.

The catch is that not every surface can do both. From a client you can signal an entity and you can read its state, but you cannot call one for a return value.

var id = new EntityInstanceId(nameof(Counter), "myCounter");
await client.Entities.SignalEntityAsync(id, "Add", 5);

SignalEntityAsync enqueues an Add operation carrying 5 and returns the instant that message is durably queued. The increment happens later, on the entity's own turn. That is the entire vocabulary a client has for changing an entity: fire an operation and move on, with no value coming back.

To pull a value out of an operation you have to be inside an orchestration. The orchestrator's entity feature can both call and signal, and the call is the only place in Durable Functions where you get request/response with an entity.

var id = new EntityInstanceId(nameof(Counter), "myCounter");
int current = await context.Entities.CallEntityAsync<int>(id, "Get");
if (current < 10)
    await context.Entities.SignalEntityAsync(id, "Add", 1);

CallEntityAsync<int>(id, "Get") sends the Get operation and awaits its return value, so current holds the counter's actual state; the conditional SignalEntityAsync then fires a one-way Add only if it is under the threshold. Read the value with a call, change it with a signal. From inside an entity the vocabulary shrinks again: an entity can signal other entities but never call one and wait for a reply.

Reading state without running an operation at all is the client's other move. GetEntityAsync<T> hands back the entity's state without dispatching anything to it.

EntityMetadata<int>? entity = await client.Entities.GetEntityAsync<int>(id);
int value = entity?.State ?? 0;

GetEntityAsync<int> returns an EntityMetadata<int>? that is null when the entity has never been created, and the read does not create it (unlike a signal or a call). entity.State is the typed value. The qualifier that matters is that this state is committed but can be stale: the query hits the durable tracking store and returns the most recently persisted state, never a half-applied intermediate, but it can lag the entity's in-memory state. Only an orchestration sees an entity's live in-memory state; a client read is always a snapshot of the last commit.

Two traps close out the surface. The first is API shape: in the isolated worker, entity access lives on context.Entities.* and client.Entities.*. The bare context.SignalEntity(...) and context.CallEntityAsync(...) forms belong to the in-process model, and reaching for them is the single most common isolated-migration error. The second follows straight from the signal semantics: because awaiting SignalEntityAsync only means "enqueued," a client GetEntityAsync fired right after a signal can still return the pre-signal state. Signal and then immediately read, and you should expect the old value; the write is on its way, not applied.

One operation at a time: the single-threaded guarantee

The Counter above has no lock around Add, and that is not an oversight. It is the guarantee that makes an entity worth reaching for: a single entity runs its operations serially, one after another, and only one operation runs at a time for a given instance. Two callers can signal Add at the same instant, and the runtime queues both and applies them in turn, so this.State += amount never races another copy of itself. You get race-free updates with no lock, no semaphore, no compare-and-swap; serializing the operations is the platform's job, not yours.

That serialization is scoped to one entity id, and this is what keeps it from being a bottleneck by default. The guarantee is per instance: @Counter@userA and @Counter@userB are scheduled independently, so a thousand distinct user counters make progress without ever queueing behind each other. This is the scale-out mechanism the actor model runs on, many entities each holding modest state, each serialized only against itself. Microsoft frames it as distributing work across many entities rather than promising that distinct ids run literally in parallel, so read it as "scheduled independently" and design for that, not for a parallelism guarantee.

Underneath, delivery is reliable and in order: messages from one sender arrive FIFO. The honest qualifier is that on rare restart, scale, or crash paths an operation can be delivered more than once (at-least-once), so an Add that runs twice should leave the same result as an Add that runs once wherever you can arrange it. Idempotent operations are the design default here, not an edge-case hardening step you bolt on later.

One property ties this back to the virtual-actor model from earlier: an entity has no create step. The first operation you signal or call to an id materializes it, and a read does not, which is why GetEntityAsync on an id nothing has ever written to comes back null instead of a fresh zero-state instance.

When one entity is a bottleneck: sharding for throughput

The serial guarantee has a limit, and you hit it the moment one entity id gets popular. Point every increment in the system at a single @Counter@global and you have rebuilt a single-threaded queue: every write lines up behind the one before it, and the entity works through them one at a time no matter how many workers you are running. There is no published per-entity operations-per-second figure to design against, so do not cite one, but the shape is clear enough. Throughput on one id is whatever one serialized entity can commit, and because entities prioritize durability over latency, they are a poor fit for latency-sensitive writes even before you hit that ceiling.

The fix is to stop funneling. Replace the one id with N sub-entities, @Counter@global-0 through @Counter@global-{N-1}, and each write picks one shard and signals only that shard. Up to N increments can now be in flight at once against N independently-scheduled entities, so the contention that a single id created spreads N ways. Reading the total means fanning out across every shard and summing what each one holds.

int shard = Random.Shared.Next(ShardCount);
var shardId = new EntityInstanceId(nameof(Counter), $"global-{shard}");
await client.Entities.SignalEntityAsync(shardId, "Add", 1);

int total = 0;
for (int i = 0; i < ShardCount; i++)
{
    var id = new EntityInstanceId(nameof(Counter), $"global-{i}");
    EntityMetadata<int>? m = await client.Entities.GetEntityAsync<int>(id);
    total += m?.State ?? 0;
}

The write path picks a shard (random here; a hash of the caller or plain round-robin works as well) and signals just that one, so no two writers are forced through the same entity unless they happen to land on the same index. The read path pays for that spread with a fan-out: it visits every shard and adds up the committed state.

The cost lands on consistency, and it is worth stating rather than hiding. Each GetEntityAsync<int> returns that shard's last committed state on its own, and under active writes some shards will have applied increments that others have not caught up to yet, so the summed total is eventually consistent: correct once the writes quiesce, approximate while they are in flight. If you need a total that is exact against a single instant while writes are still landing, you have to lock the shards together in a critical section, which serializes them again and hands back the throughput you sharded for in the first place. Shard when an approximate running total is acceptable, and keep a single id (or a database) for when the count has to be exact.

Entities, orchestrations, or a database

The decision that decides everything above is when to reach for an entity at all. Microsoft's documented "good for" list is short and specific: aggregating data from multiple sources, distributed locks and semaphores, and stateful objects like shopping carts or game sessions, with the counter as the canonical worked example. If what you are modeling is a named thing that accumulates state over its lifetime and reacts to updates, that is the entity's home ground.

The first fork is entity versus orchestration, and it is the code-as-state and code-as-workflow line from the intro made concrete. An orchestration is a workflow: ordered steps, a defined start and end, deterministic replay, activity retries. An entity is long-lived keyed state with no predefined lifecycle, and because it is not replayed the way an orchestrator is, an operation can run any code, including the nondeterministic or long-running work that would be illegal inside an orchestrator. They are not rivals; you combine them, and the combination is the only place you get request/response with an entity, since a call has to originate in an orchestration.

The second fork is entity versus a database row, and this is the one to be honest about, because the entity does not always win. The entity wins when the alternative is writing the locking code yourself: updates are serialized race-free, the state survives restarts, and the durable backend manages it with no connection pool or transaction of yours involved. The database wins on everything the entity was not built for: higher throughput on a single hot key, rich queries and joins, lower read latency, and state larger than the cap. Remember that a client read of an entity is committed-but-stale and that entities are not richly queryable, so a question like "every cart abandoned for more than an hour" is a query, and a query belongs to a database, not an entity.

Some limits are worth naming before you commit. There is the serial-per-id throughput ceiling from the previous section; a maximum entity state of 1 MB on the Durable Task Scheduler backend (that cap is backend-specific, not a universal number, so check the one you run); durability prioritized over latency; and request/response available only from an orchestration. One thing that is not a documented limit: there is no rule against long-running work inside an operation. Current docs allow it. Keeping operations short is still worth doing, because a slow operation blocks the entity's whole queue and long work usually belongs in a child orchestration, but treat that as a design habit of your own, not a platform rule you can cite.

When you do need to touch several entities atomically, the classic case being a bank transfer that debits one account and credits another with no observable state in between, you open a durable critical section from an orchestration over the entities you are locking.

public static class TransferOrchestrator
{
    [Function(nameof(TransferOrchestrator))]
    public static async Task<bool> Run(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        TransferRequest request = context.GetInput<TransferRequest>()!;
        var from = new EntityInstanceId(nameof(BankAccount), request.From);
        var to = new EntityInstanceId(nameof(BankAccount), request.To);

        await using (await context.Entities.LockEntitiesAsync(from, to))
        {
            bool withdrew = await context.Entities.CallEntityAsync<bool>(
                from, nameof(BankAccount.Withdraw), request.Amount);
            if (!withdrew)
                return false;

            await context.Entities.CallEntityAsync(
                to, nameof(BankAccount.Deposit), request.Amount);
            return true;
        }
    }
}

The signature is now pinned to the compiled SDK: context.Entities.LockEntitiesAsync(...) has a params EntityInstanceId[] overload (and an IEnumerable<EntityInstanceId> one) and returns Task<IAsyncDisposable>, so await using (await context.Entities.LockEntitiesAsync(from, to)) is the exact isolated shape. One note trips people up: entities and critical sections both work in the isolated worker, yet Microsoft's own orchestration docs still carry a stale line claiming critical sections are not available in isolated. That line is out of date; the mechanism moved to a renamed API, it did not disappear. Note too that a critical section gives you isolation, not a transaction: there is no automatic rollback, so any compensating undo on a mid-transfer failure is code you write.

One deployment gotcha before you build on any of this: the MSSQL storage provider does not support entities in the isolated worker. If your task hub runs on MSSQL, entities are off the table until you move backends, and learning that now beats learning it from a failed deploy.

There is also a client-side listing API for walking every entity of a type, for enumerating your shards or every open cart rather than addressing one by id. The isolated signature is confirmed against the compiled SDK: client.Entities.GetAllEntitiesAsync<T>(EntityQuery) returns an AsyncPageable<EntityMetadata<T>> you enumerate with await foreach. The EntityQuery filter carries InstanceIdStartsWith (matching the @name@key id form), IncludeState, and paging via PageSize.

var query = new EntityQuery
{
    InstanceIdStartsWith = $"@{nameof(ShoppingCart).ToLowerInvariant()}@",
    IncludeState = true,
};

await foreach (EntityMetadata<List<CartLine>> cart in
    client.Entities.GetAllEntitiesAsync<List<CartLine>>(query))
{
    decimal total = cart.State.Sum(l => l.UnitPrice * l.Quantity);
}

Conclusion

You now own the decision the series has been circling. Orchestrations were code-as-workflow: a thing with a start, an end, and steps that replay deterministically. Entities are code-as-state: a named, addressable thing that outlives any single request, mutates across its lifetime, and hands you race-free updates without a line of locking code. The choice between them is not about which is newer or nicer; it is whether the thing you are modeling has a lifecycle or only a state.

And when the answer is "only a state," one fork remains, and it is the one to sit with. Keyed state that mutates over a lifetime, needs serialized race-free updates, and can tolerate a committed-but-possibly-stale read points squarely at an entity. A need for high single-key throughput, rich queries, or strict read latency points just as squarely at a database row you manage yourself. Neither is the default answer. You pick on which set of constraints is actually yours.

For the next piece of keyed state in your system, a per-user counter, a cart, a session: do you reach for a Durable Entity so the runtime serializes every update for you, or a database row you lock yourself?

Human Interaction and External Events: Approval Workflows

Martin Oehlert — Fri, 03 Jul 2026 05:59:36 +0000

An expense report needs a manager's sign-off before the reimbursement goes out, and that manager might click approve in five minutes or come back from a trip in five days. The question Parts 1 and 2 never had to answer is how the workflow waits that long: a chain or a fan-out runs to completion in seconds, but an approval step has to suspend on a person and stay suspended without holding a thread or billing you for the idle time in between. Durable Functions answers the waiting half with a single await that costs no compute while it is parked. The half most guides skip is the other one: the runtime hands you an instance ID when the workflow starts, and finding that one paused instance again when the approval finally arrives is your code's job, not the platform's.

WaitForExternalEvent pattern

The mechanic at the center of every approval workflow is one line: await context.WaitForExternalEvent<T>(eventName). It suspends the orchestration until something outside the function raises an event with that name, and the typed payload it returns carries whatever the approver decided.

Every code sample below is from the companion sample (isolated worker, .NET 10). Here is the expense-approval orchestrator: it parks on the wait, then hands the decision to an activity that settles or rejects the report.

using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;

public record ExpenseReport(string ReportId, string Employee, decimal Amount, string Category);
public record ApprovalDecision(string DecisionId, bool Approved, string Approver, string? Note);
public record SettlementInput(ExpenseReport Report, ApprovalDecision Decision);

public static class ExpenseApprovalOrchestrator
{
    [Function(nameof(ExpenseApprovalOrchestrator))]
    public static async Task<string> Run(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        var report = context.GetInput<ExpenseReport>()!;

        // Suspends here until an "ApprovalDecision" event is raised for this instance.
        // No thread is held and no compute is billed while the orchestration waits.
        ApprovalDecision decision =
            await context.WaitForExternalEvent<ApprovalDecision>("ApprovalDecision");

        return await context.CallActivityAsync<string>(
            nameof(SettleExpenseActivity), new SettlementInput(report, decision));
    }
}

The WaitForExternalEvent<ApprovalDecision>("ApprovalDecision") call is the whole pause. When the orchestrator reaches it, the runtime checkpoints the instance and unloads it; execution does not resume until an event named ApprovalDecision is raised against this instance ID, at which point the raised JSON is deserialized into an ApprovalDecision and the await returns it. The first argument is the event name, the contract both ends agree on, and matching is case-insensitive: a wait on "ApprovalDecision" is satisfied by an event raised as "approvaldecision". The type argument is the payload shape; if the raised JSON cannot be converted to it, the wait throws rather than returning a half-filled object.

The activity is an ordinary function that acts on the decision.

public static class SettleExpenseActivity
{
    [Function(nameof(SettleExpenseActivity))]
    public static string Run([ActivityTrigger] SettlementInput input)
    {
        var (report, decision) = input;
        if (!decision.Approved)
            return $"Report {report.ReportId} rejected by {decision.Approver}.";

        // Real work belongs here: queue the reimbursement, post to the ledger, notify the employee.
        return $"Report {report.ReportId} approved by {decision.Approver}; {report.Amount:C} scheduled for payment.";
    }
}

What makes this safe to wait on for days is that the suspension costs no compute. Once the orchestrator yields at the wait, there is no thread blocked, no instance kept warm, nothing to bill. On the Consumption and Flex plans you pay for actual execution, not idle wait time, so an approval parked for a week is free until the event arrives. The wait is also replay-safe in exactly the sense Part 1 set up: the pending event is part of the orchestration's durable state, so the worker can be stopped, scaled in, or recycled, and the instance is reawakened when the event shows up. An event that arrives early is not a problem either; if it is raised before the orchestrator has reached the wait, it is buffered in the instance state and dispatched the moment the wait is reached, so a fast approver who beats the orchestration to the wait line does not lose their decision.

One honest gotcha to design for up front: event delivery on the Azure Storage backend is at-least-once, so a restart or scale event can deliver the same approval twice. That is why the payload here carries a DecisionId. If the downstream activity is not naturally idempotent, dedupe on that ID so a duplicate delivery does not pay the same expense report twice. (The MSSQL provider consumes events transactionally and does not produce duplicates, but coding for at-least-once keeps the orchestrator portable across backends.)

This wait is indefinite: nothing here ever gives up. A real approval workflow needs a deadline so a report that no one ever touches does not sit parked forever, and bounding the wait with a durable timer is the next section.

Approval endpoint design

The orchestrator only knows how to wait. Two HTTP endpoints surround it: one to start the workflow and one to deliver the approver's answer. The start endpoint is the async HTTP pattern from Part 2, schedule the orchestration and hand back a status URL without blocking.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask.Client;

public static class StartExpenseApprovalClient
{
    [Function(nameof(StartExpenseApprovalClient))]
    public static async Task<HttpResponseData> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = "expenses")]
            HttpRequestData req,
        [DurableClient] DurableTaskClient client)
    {
        ExpenseReport report = await req.ReadFromJsonAsync<ExpenseReport>()
            ?? throw new InvalidOperationException("Expense report body is required.");

        string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
            nameof(ExpenseApprovalOrchestrator), report);

        // 202 + management URLs; the orchestration is now parked on its WaitForExternalEvent.
        return await client.CreateCheckStatusResponseAsync(req, instanceId);
    }
}

ScheduleNewOrchestrationInstanceAsync enqueues the orchestration and returns its instance ID without waiting for it to run, and CreateCheckStatusResponseAsync builds the HTTP 202 response: a Location header pointing at the status-query endpoint and a JSON body of management URLs for the instance. By the time the caller has that 202, the orchestration is already parked on its WaitForExternalEvent. Hold on to that returned instanceId; it is the only handle that reaches the parked instance, and the approval endpoint is useless without it.

The approval endpoint is the other half. It reads the manager's decision and raises the event the orchestrator is blocked on.

public record ApprovalRequest(bool Approved, string Approver, string? Note);

public static class SubmitApprovalClient
{
    [Function(nameof(SubmitApprovalClient))]
    public static async Task<HttpResponseData> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = "expenses/{instanceId}/decision")]
            HttpRequestData req,
        string instanceId,
        [DurableClient] DurableTaskClient client)
    {
        ApprovalRequest body = await req.ReadFromJsonAsync<ApprovalRequest>()
            ?? throw new InvalidOperationException("Approval body is required.");

        var decision = new ApprovalDecision(
            DecisionId: Guid.NewGuid().ToString("N"),
            Approved: body.Approved,
            Approver: body.Approver,
            Note: body.Note);

        // Raises the event the orchestrator is waiting on. Returns when the event is
        // enqueued, not when the orchestrator has consumed it.
        await client.RaiseEventAsync(instanceId, "ApprovalDecision", decision);

        return await client.CreateCheckStatusResponseAsync(req, instanceId);
    }
}

RaiseEventAsync(instanceId, "ApprovalDecision", decision) is the mirror image of the wait. The event name has to match the orchestrator's wait name (again, case-insensitively), and the decision object is JSON-serialized and deserialized into the orchestrator's ApprovalDecision on the other side. The returned task completes when the event is enqueued, not when the orchestrator wakes up and consumes it, so a 202 here means "your decision is on its way," not "the report is settled." The caller polls the status URL to see the workflow reach Completed. Generating the DecisionId server-side is what makes the earlier dedupe work: it gives every raised decision a stable identity even if the platform delivers it twice.

There are two ways to raise this event, and the choice is the reason this endpoint exists at all. The 202 body from the start call already contains a sendEventPostUri, the built-in raise-event API: a caller can POST the decision straight to .../instances/{instanceId}/raiseEvent/ApprovalDecision with no code from you. It is the quickest path, and it returns useful status codes (404 for an unknown instance, 410 for one that already finished). What it does not give you is a place to put your own concerns. The custom endpoint above exists so you can own the route, authentication, validation, and audit: check that this approver is allowed to sign off this report, record who decided and when, reject a malformed body before it ever reaches the orchestration. If none of that matters for your case, the built-in webhook is less to maintain.

The honest gotcha lives in the failure mode of the SDK call. A bare RaiseEventAsync to a completed or non-existent instance is silently discarded: no exception, no error, nothing. Raise ApprovalDecision against a stale or mistyped instance ID and the call returns happily while the event evaporates, and the approver sees a success they did not get. If you need to tell an approver that the workflow they are signing off no longer exists, pre-check with GetInstanceAsync and inspect the runtime status before raising, or use the built-in HTTP API and surface its 404 and 410 to the caller. The silent path is convenient until the instance ID is wrong.

Which raises the question this endpoint quietly assumes away: it takes instanceId from the route as if the caller already knows it. Where that ID comes from, how the approval link in the manager's email ends up carrying the right one, and how you avoid losing it, is its own problem, and the section on instance ID storage is where it gets solved.

Timeout and escalation

A wait that never gives up is a workflow you cannot operate. The previous section left the orchestration parked on WaitForExternalEvent with no exit; a report that no manager ever touches stays Running until someone terminates it by hand. The fix is to race the event against a durable timer and let whichever finishes first decide the outcome.

Start the timer and the wait, then await Task.WhenAny on the pair. The orchestration wakes on the first of the two to complete.

public static class ExpenseApprovalOrchestrator
{
    [Function(nameof(ExpenseApprovalOrchestrator))]
    public static async Task<string> Run(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        var report = context.GetInput<ExpenseReport>()!;

        // Deadline comes off the orchestration clock, NOT DateTime.UtcNow. Part 1's
        // determinism rule: every replay must compute the same instant, and
        // CurrentUtcDateTime is frozen to the original execution time on replay.
        DateTime deadline = context.CurrentUtcDateTime.AddDays(3);

        using var cts = new CancellationTokenSource();
        Task<ApprovalDecision> approvalTask =
            context.WaitForExternalEvent<ApprovalDecision>("ApprovalDecision");
        Task timeoutTask = context.CreateTimer(deadline, cts.Token);

        Task winner = await Task.WhenAny(approvalTask, timeoutTask);

        if (winner == approvalTask)
        {
            // The approval landed first. Cancel the timer before moving on.
            cts.Cancel();
            return await context.CallActivityAsync<string>(
                nameof(SettleExpenseActivity),
                new SettlementInput(report, approvalTask.Result));
        }

        // The timer won: nobody decided in time. Fail closed by auto-rejecting.
        var timedOut = new ApprovalDecision(
            DecisionId: $"timeout-{report.ReportId}",
            Approved: false,
            Approver: "system (timeout)",
            Note: $"No decision by {deadline:o}.");
        return await context.CallActivityAsync<string>(
            nameof(SettleExpenseActivity), new SettlementInput(report, timedOut));
    }
}

Two details carry the weight here. The first is the deadline source. context.CurrentUtcDateTime is the replay-safe clock from Part 1: on the first execution it is the real time, and on every replay after a checkpoint it returns that same original value, so AddDays(3) resolves to one fixed instant no matter how many times the orchestrator re-runs. Use DateTime.UtcNow instead and the deadline drifts forward on every replay, which breaks determinism and can move the timer past where it should have fired.

The second is the cts.Cancel() on the approval branch, and it is easy to read as optional cleanup when it is not. CreateTimer registers a durable timer in the instance state, and the framework will not let an orchestration reach Completed while a timer it created is still outstanding. Skip the cancel and your approved report settles its activity, then sits in Running for the rest of the three days until the abandoned timer finally fires. Cancelling the token does not abort anything in flight; it tells the runtime to drop the pending timer so the orchestrator can finish now. The using on the CancellationTokenSource disposes it when the method exits.

Fail-closed is a choice, not a rule. Auto-rejecting on timeout is the conservative default: an expense nobody approved should not be paid. The richer variant is to escalate rather than reject. Instead of returning, the timeout branch notifies a second approver (in the expense case, the manager's manager) and waits again with a fresh timer.

// Escalation variant for the timeout branch: re-notify, then wait once more.
await context.CallActivityAsync(nameof(NotifyEscalationApproverActivity), report);

using var escalationCts = new CancellationTokenSource();
Task<ApprovalDecision> escalatedApproval =
    context.WaitForExternalEvent<ApprovalDecision>("ApprovalDecision");
Task escalationTimeout =
    context.CreateTimer(context.CurrentUtcDateTime.AddDays(2), escalationCts.Token);

if (await Task.WhenAny(escalatedApproval, escalationTimeout) == escalatedApproval)
{
    escalationCts.Cancel();
    return await context.CallActivityAsync<string>(
        nameof(SettleExpenseActivity),
        new SettlementInput(report, escalatedApproval.Result));
}
// Still nothing after the second window: now fail closed.

Each escalation round is the same race with a new deadline and a new CancellationTokenSource, so the same two rules apply every time: derive the deadline from CurrentUtcDateTime, cancel the timer when the event wins. You can wrap the round in a loop to escalate up a chain of approvers, but give it a hard ceiling; an unbounded escalation loop is the indefinite wait you just removed, wearing a different hat.

Instance ID storage patterns

The approval endpoint took instanceId straight from its route, as if the caller already had it. The start endpoint, meanwhile, returned that ID inside a 202 and then dropped it. So when the manager opens an email three days later and clicks approve, what fills in the {instanceId} segment of expenses/{instanceId}/decision? This is the half the intro flagged: Durable Functions hands you the instance ID at start time and keeps no index from a business entity to its instance ID. Mapping report R-2048 back to the orchestration that is waiting on it is your code's job, and there are three ways to do it.

Option 1: an external store keyed by the business entity. Write a row at start time (ReportId to instanceId) into whatever database or lookup service you already run, then read it back in the approval endpoint. This is the production default. It is authoritative and queryable, it supports many runs mapping to one entity and full history, and it survives instance purging. The cost is that you now own a second piece of state: an extra write on start, an extra read on approval, and the consistency between that row and the orchestration is yours to keep (make the write idempotent or transactional with the start so you cannot end up with a row pointing at an instance that never scheduled, or an instance with no row).

Option 2: make the instance ID the business key, and store nothing. ScheduleNewOrchestrationInstanceAsync lets you supply the ID instead of taking an autogenerated GUID, via StartOrchestrationOptions.InstanceId. If the ID is the report key, the approval endpoint reconstructs it from the route with no lookup at all.

string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
    nameof(ExpenseApprovalOrchestrator),
    report,
    new StartOrchestrationOptions { InstanceId = $"expense-{report.ReportId}" });

Now the email link is just expenses/expense-{ReportId}/decision, built from data you already have, and there is no table to keep in sync. The constraints are real, though, and the runtime enforces them unevenly across storage providers, so honor them regardless: the ID must be unique within the task hub, 1 to 100 characters, must not start with @, and must not contain /, \, #, ?, or control characters. Raw GUIDs are fine; emails and file paths usually need encoding first. The mapping is strictly one-to-one, so this fits short, naturally unique, single-run keys and not much else. A report ID like expense-R-2048 qualifies; a customer who can file many reports does not.

Option 2 also inherits a gotcha worth stating plainly: scheduling an instance ID that already exists is not a safe atomic create-if-absent. The documented pattern is check-then-start (call GetInstanceAsync, inspect RuntimeStatus, and start only if the instance is missing or in a terminal state), and Microsoft flags a concurrency race even then: two requests for the same key can both pass the check and both report success while only one orchestration actually runs. If a duplicate submit must never double-schedule, you need a lock outside Durable Functions, which starts to erode the "store nothing" advantage.

Option 3: a Durable Entity as a registry. Entities are supported in the .NET isolated worker (the "not in isolated" caveat you may have read refers to in-orchestration critical sections, not entities), their operations run serially so there is no intra-entity race, and the index stays inside Durable Functions instead of a separate database. Treat this as viable but unproven: no official guidance endorses an entity as the business-key index, entities favor durability over latency so client reads can be stale, and routing every lookup through one hot registry entity serializes all traffic into a throughput bottleneck. Reach for it only if you have a specific reason to avoid an external store.

One temptation to rule out: the query APIs are not a reverse lookup. GetInstanceAsync needs the ID you are trying to find. GetAllInstancesAsync(OrchestrationQuery) filters only on runtime status, time range, and orchestration name, with no predicate for an arbitrary business key, so finding R-2048 that way means scanning every instance client-side, an O(n) walk that degrades as history grows and breaks once completed instances are purged. Custom status (SetCustomStatus) is for surfacing progress and caps at 16 KB; tags are queryable only in the scheduler dashboard, not from code. None of them is a point lookup. A business key to instance ID mapping always comes back to an app-owned index (option 1) or a derivable ID (option 2).

For the expense workflow the call is short. If a report ID is already a clean single-run key, option 2 removes a whole moving part: the email link encodes the ID and there is nothing to persist or reconcile. The moment you need many approvals per report, audit history, or a key that does not survive the ID character rules, option 1's extra row pays for itself.

Wrapping up

The pause is the easy half. WaitForExternalEvent suspends for days on a single await and bills you nothing while it waits, and a CreateTimer race keeps that wait from becoming a leak. The half that decides whether this works in production is the one most walkthroughs skip: the instance ID is a handle the platform hands you once and never indexes, so reuniting a parked workflow with the human who finally answers it is a design decision you make on purpose, at start time. Get that wrong and the approver clicks a link that raises an event into the void.

So make the call deliberately on your next approval workflow: do you persist the instance ID in an external table keyed by the business entity, or derive it from the business key so there is nothing to store?

Fan-Out/Fan-In and the Async HTTP Pattern

Martin Oehlert — Fri, 26 Jun 2026 05:00:00 +0000

You have 500 order line items to process and no ordering dependency between them, so the question is why the workflow takes as long as 500 activities run back to back when nothing forces them to. The chaining pattern from Part 1 awaits each activity before scheduling the next, which is exactly right when step two needs step one's output and exactly wasteful when the items are independent. The fix is fan-out/fan-in: schedule all the activities at once, then aggregate the results, and the only real trick is doing it in a way that survives the orchestrator replaying. There is also a one-character version of this code that compiles, runs, and silently throws the parallelism away, which is the bug this article spends the most time on.

When sequential processing isn't enough

Chaining is the pattern you reach for when the steps form a line: validate the order, then create it, then send the confirmation, each one feeding the next. The await between steps is load-bearing there, because CreateOrderActivity genuinely cannot start until ValidateOrderActivity has returned. That dependency is what makes the sequence correct.

Now change the shape of the work. A batch order arrives with 500 line items, and each one needs the same per-item processing: check stock, price it, reserve inventory. No line item depends on any other. If you write that as a chain, awaiting each item's activity before scheduling the next, the total wall-clock time is the sum of all 500 activities. At a rough sequential ceiling of about 5 activities per second on a single instance, that batch takes roughly a minute and a half, and every second of it is one item waiting on the item before it for no reason.

The independence is the whole point. When the items have no ordering relationship, the latency you actually care about is not the sum of the activities but the slowest single one, because there is nothing stopping them from running at the same time. Fan out across the available workers and the same batch finishes in the time of its longest item, plus a little aggregation overhead. The documented fan-out throughput target is around 100 activities per second per instance, an order of magnitude over the sequential figure, and that gap is entirely the difference between running the work in series and running it in parallel.

Fan-out/fan-in is the orchestration pattern for that situation: fan out by scheduling many activities at once, fan in by waiting for all of them and collecting the results. The replay engine from Part 1 is what makes it safe, and the next section shows the exact shape.

Fan-out/fan-in with Task.WhenAll

The shape has two halves. Fan-out means projecting your inputs into activity calls and collecting the tasks without awaiting any of them. Fan-in means a single await Task.WhenAll(tasks) that completes once every activity has finished, handing you the results.

Every code sample below is from the companion sample (isolated worker, .NET 10). Here is the batch processor as an orchestrator, with an aggregation step at the end.

using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;

public record OrderItem(string Sku, int Quantity);
public record OrderResult(string Sku, bool Reserved, decimal LineTotal);
public record BatchSummary(int Processed, int Reserved, decimal Total);

public static class ProcessBatchOrchestrator
{
    [Function(nameof(ProcessBatchOrchestrator))]
    public static async Task<BatchSummary> Run(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        var items = context.GetInput<OrderItem[]>()!;

        // Fan-out: schedule every item at once, collect the tasks unawaited.
        Task<OrderResult>[] tasks =
            [.. items.Select(item => context.CallActivityAsync<OrderResult>(
                nameof(ProcessItemActivity), item))];

        // Fan-in: one await blocks until all of them complete.
        OrderResult[] results = await Task.WhenAll(tasks);

        // Optional: hand the collected results to a final aggregation activity.
        return await context.CallActivityAsync<BatchSummary>(
            nameof(SummarizeBatchActivity), results);
    }
}

The fan-out is the collection expression. items.Select(...) projects each OrderItem into a CallActivityAsync<OrderResult> call, and because nothing awaits those calls, each one schedules an activity and returns its Task<OrderResult> immediately. The [.. ...] spread materializes them into a Task<OrderResult>[]. After that line, all 500 activities are scheduled; none has been waited on.

await Task.WhenAll(tasks) is the fan-in. It returns a single task that completes only when every task in the array has completed, and its result is an OrderResult[]. The element order matches the order of the tasks array, not the order the activities happened to finish in, so results[0] is always the result for items[0] regardless of which line item processed fastest. That positional guarantee is standard Task.WhenAll<TResult> behavior, and it is why you can return the array directly without sorting or correlating anything back to its input.

The activity itself is an ordinary function doing the per-item work.

public static class ProcessItemActivity
{
    [Function(nameof(ProcessItemActivity))]
    public static OrderResult Run([ActivityTrigger] OrderItem item)
    {
        // Real work and I/O belong here: check stock, price, reserve inventory.
        bool reserved = item.Quantity > 0;
        decimal lineTotal = item.Quantity * 9.99m;
        return new OrderResult(item.Sku, reserved, lineTotal);
    }
}

Task.WhenAll is not one of the nondeterministic APIs banned inside an orchestrator. It is replay-safe, and the reason traces straight back to Part 1's replay engine. The durable calls it aggregates are the replay-checkpointed operations: the runtime records each scheduled CallActivityAsync in the History table the moment the orchestrator yields, and records each result as it arrives. On a replay the orchestrator reaches the same fan-out line, the framework sees those activities already scheduled (and any already completed), and reconstructs the same task array from history rather than re-running the work. The whole parallel batch therefore survives a host recycle the same way a chain does: the activities run across multiple workers concurrently, and the end-to-end execution is resilient to the orchestrator being unloaded and replayed.

One behavior to know before you ship this. When several activities fail, await Task.WhenAll(tasks) throws only the first exception, even though more than one task faulted. If you need to see every failure (to log all of them, or decide based on how many failed) inspect the Exception property on the task that Task.WhenAll returns, which holds the full AggregateException with one inner exception per faulted activity.

The loop mistake (and the fix)

The previous section showed the correct shape. This section is about the version that looks just as correct, compiles cleanly, runs without error, and quietly runs everything in series anyway. It is the most common fan-out bug, and it is worth being able to spot in a code review on sight, because nothing else will flag it for you.

Here is the broken orchestrator. A reviewer skimming it sees a loop over the items, an activity call per item, and a list of results. It reads like a fan-out.

// BROKEN: awaiting inside the loop runs the activities one after another.
[Function(nameof(ProcessBatchOrchestrator))]
public static async Task<OrderResult[]> Run(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var items = context.GetInput<OrderItem[]>()!;
    var results = new List<OrderResult>();
    foreach (var item in items)
        results.Add(await context.CallActivityAsync<OrderResult>(
            nameof(ProcessItemActivity), item));   // awaits before scheduling the next
    return [.. results];
}

The await is in the wrong place. Each iteration calls CallActivityAsync, then await suspends the orchestrator until that one activity returns, and only after the result is added to the list does the loop come back around to schedule the next item. So the activities are scheduled one at a time, each waiting on the one before it. This is the chaining pattern wearing a loop, and it has the chaining pattern's latency: the sum of all 500 activities, back at roughly 5 per second. There is no compiler warning, no runtime exception, no log line that looks wrong. The only symptom is that a batch that should take a couple of seconds takes a minute and a half, and you usually only notice once the batch sizes grow in production.

The fix is to move the await out of the loop. Schedule everything first, collect the tasks, then fan in with a single Task.WhenAll.

// FIXED: collect every task first, then fan in with one Task.WhenAll.
[Function(nameof(ProcessBatchOrchestrator))]
public static async Task<OrderResult[]> Run(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var items = context.GetInput<OrderItem[]>()!;
    Task<OrderResult>[] tasks =
        [.. items.Select(item => context.CallActivityAsync<OrderResult>(
            nameof(ProcessItemActivity), item))];   // scheduled, not awaited
    return await Task.WhenAll(tasks);                // all run in parallel
}

The difference is whether await sits on each individual call or once on the whole array. In the broken version every CallActivityAsync is awaited the instant it is made, which serializes the schedule-and-wait. In the fixed version the Select schedules all of them without awaiting any, and the single await Task.WhenAll(tasks) is the only suspension point, so the activities are in flight together and the worker fans them out across the pool. Same activity, same input, same result array (Task.WhenAll preserves the order of the tasks array, so you can return it directly), and at batch scale the difference is roughly 100 activities per second instead of 5.

The review heuristic: if you see await on a durable call inside a foreach or for loop, stop and ask whether those iterations actually depend on each other. If iteration N needs iteration N-1's output, awaiting in the loop is correct, that is a chain. If they are independent, the await belongs on a Task.WhenAll after the loop, and leaving it inside is the silent serialization trap.

The async HTTP pattern

A 500-item batch that fans out to activities can run for a minute and a half, and no HTTP client should be holding a socket open that long. Browsers, load balancers, and API gateways all time out well before that, so a client function that started the orchestration and blocked on its result would fail the caller before the batch even finished. The async HTTP pattern solves this by separating starting the work from collecting its result: the HTTP trigger kicks off the orchestration and returns immediately with a set of URLs the caller can poll.

Here is the client function that starts ProcessBatchOrchestrator and hands the caller back a status endpoint.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask.Client;

public static class StartBatchClient
{
    [Function(nameof(StartBatchClient))]
    public static async Task<HttpResponseData> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = "batches")]
            HttpRequestData req,
        [DurableClient] DurableTaskClient client)
    {
        OrderItem[] items = await req.ReadFromJsonAsync<OrderItem[]>() ?? [];

        string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
            nameof(ProcessBatchOrchestrator), items);

        // 202 Accepted + Location + the management URLs, without blocking on the result.
        return await client.CreateCheckStatusResponseAsync(req, instanceId);
    }
}

ScheduleNewOrchestrationInstanceAsync enqueues the orchestration and returns its instance id without waiting for it to run. CreateCheckStatusResponseAsync then builds the response: HTTP 202 Accepted, a Location header pointing at the status-query endpoint, and a Retry-After header (10 seconds by default) telling the caller how long to wait before polling again. Prefer the awaited CreateCheckStatusResponseAsync over the synchronous CreateCheckStatusResponse: under the ASP.NET Core integration the synchronous form can throw InvalidOperationException ("Synchronous operations are disallowed"), which is why the async overload exists.

The JSON body carries the management URLs for the instance:

{
  "id": "7f3c1e9a4b8d4f0e9c2a6b5d8e1f0a3c",
  "statusQueryGetUri": "https://.../runtime/webhooks/durabletask/instances/7f3c...?...",
  "sendEventPostUri": "https://.../instances/7f3c.../raiseEvent/{eventName}?...",
  "terminatePostUri": "https://.../instances/7f3c.../terminate?reason={text}&...",
  "suspendPostUri": "https://.../instances/7f3c.../suspend?reason={text}&...",
  "resumePostUri": "https://.../instances/7f3c.../resume?reason={text}&...",
  "purgeHistoryDeleteUri": "https://.../instances/7f3c...?..."
}

statusQueryGetUri is the one the caller polls (it is the same URL as the Location header); the others handle raising an external event, terminating, suspending, resuming, and purging the instance's history. A preview rewindPostUri shows up too on supported plans.

From the caller's side it is a poll loop against statusQueryGetUri. While the instance is still running the status endpoint returns 202, with its own Location header pointing back at itself; once the instance reaches a terminal state it returns 200 with the full status body, and the body's output field carries the orchestration's return value. The runtimeStatus field tells you which state you landed in: Running, Pending, Failed, Canceled, Terminated, Completed, or Suspended.

A curl-style poll honoring Retry-After is the whole client protocol:

using HttpClient http = new();
HttpResponseMessage start = await http.PostAsJsonAsync(
    "https://.../api/batches", items);

// 202 came back; Location is the status-query URL.
Uri statusUri = start.Headers.Location!;

while (true)
{
    HttpResponseMessage status = await http.GetAsync(statusUri);
    if (status.StatusCode == HttpStatusCode.OK)
    {
        BatchSummary? summary = await status.Content
            .ReadFromJsonAsync<StatusResponse>() is { Output: var o } ? o : null;
        // runtimeStatus == "Completed"; the result is in the body's output field.
        break;
    }

    // 202: still running. Wait the Retry-After the server asked for.
    TimeSpan wait = status.Headers.RetryAfter?.Delta ?? TimeSpan.FromSeconds(10);
    await Task.Delay(wait);
}

If your caller can tolerate a short synchronous wait (a small batch that usually finishes in a second or two), WaitForCompletionOrCreateCheckStatusResponseAsync collapses the round trip: it waits for the instance to complete and returns its output with a 200, and if the wait elapses first it falls back to the same 202 + management-URL payload. The isolated-worker signature is worth a careful read, because it differs from the in-process one: it takes a retryInterval for the internal poll cadence and a CancellationToken that bounds the overall wait, but there is no timeout parameter. You cap the wait by cancelling the token, not by passing a TimeSpan.

Memory and concurrency limits

Two things bite at batch scale that never show up on a three-item demo: where all those activity results go, and how parallel the fan-out actually runs.

Start with memory. Every activity output is serialized into the orchestration's history in the <TaskHubName>History table, and Part 1's replay engine loads that full history into memory each time the orchestrator replays. With a 500-wide fan-out, the fan-in array is 500 serialized results sitting in history and getting rehydrated on every replay. If each result is small that is fine; if each activity returns a multi-megabyte payload, history balloons. Past 45 KB serialized, a result spills over to a <taskhub>-largemessages blob container automatically (the underlying Azure Queue message hard cap is 64 KB, and the 45 KB threshold leaves headroom for the compressed form). That spillover is correctness-preserving, but it is not free: it costs CPU and IO for the compress-and-round-trip, and the rehydrated payloads still bloat replay memory.

The fix is to return a reference, not the bytes. Have the activity write the heavy payload to blob storage and return a small id the fan-in can carry cheaply.

// BROKEN: the multi-MB image is serialized into history on fan-in.
[Function(nameof(RenderPageActivity))]
public static RenderedPage Run([ActivityTrigger] int page)
{
    byte[] image = Renderer.ToPng(page);     // multi-MB payload
    return new RenderedPage(page, image);    // every byte lands in history
}

// FIXED: write the payload to blob storage, return a small reference.
[Function(nameof(RenderPageActivity))]
public static async Task<PageRef> Run([ActivityTrigger] int page)
{
    byte[] image = Renderer.ToPng(page);
    string blobName = await Blobs.UploadAsync($"pages/{page}.png", image);
    return new PageRef(page, blobName);      // history carries a reference, not the bytes
}

Whatever needs the bytes later reads them back from blob storage by name. The history stays small, replay stays fast, and you never go near the spillover threshold.

Now concurrency. Fan-out is not unbounded parallelism. Scheduling 500 activities at once does not mean 500 run at once; the host caps how many activities execute concurrently per instance through maxConcurrentActivityFunctions, which defaults to 10 on the Consumption plan (10x the processor count on Dedicated and Premium). Orchestrators have their own ceiling, maxConcurrentOrchestratorFunctions, defaulting to 5. Both live under extensions.durableTask in host.json:

{
  "extensions": {
    "durableTask": {
      "maxConcurrentActivityFunctions": 10,
      "maxConcurrentOrchestratorFunctions": 5
    }
  }
}

Both limits are per-instance, so scale-out multiplies them: ten workers at the default give you up to 100 activities in flight. The surplus past the concurrency limit does not fail, it queues, and the runtime drains it as slots free up. So a 500-wide fan-out on a single instance runs about 10 at a time with the rest waiting their turn, and the batch is only as parallel as your concurrency setting times your worker count allows. Size the fan-out against that ceiling rather than assuming the width you scheduled is the width that runs.

Closing

The whole pattern turns on where one await sits. Move it off the individual CallActivityAsync calls and onto a single Task.WhenAll, and the same orchestrator that ran 500 items in series now runs them in parallel, replay-safe, across every worker the platform gives you. The async HTTP pattern lets a caller start that batch and walk away with a status URL instead of a held-open socket, and the memory and concurrency caps are the guardrails that keep a wide fan-out from quietly bloating history or pretending to be more parallel than it is.

Part 3 picks up the other half of orchestration: workflows that pause and wait on something outside the function, like a human approval or an external event, without burning compute while they wait.

Do you cap your fan-out width with a host.json concurrency limit, or let the platform scale it and size the batch to fit?

Intro to Durable Functions: Orchestrations and the Chaining Pattern

Martin Oehlert — Fri, 19 Jun 2026 06:01:50 +0000

An order comes in, and you need to validate it, create it, then send a confirmation, three steps that have to run in order and survive a crash halfway through. A single Azure Function can't do that, because the moment it returns it forgets everything, so the usual fix is a chain of queue-triggered functions wired together with a correlation ID, a status table, and your own retry logic. That hand-rolled state machine is exactly what Durable Functions replaces, and the price of admission is learning to write an orchestrator: normal-looking C# that the runtime is allowed to run more than once.

Why a stateless function can't run a workflow

A plain Azure Function is a single invocation. It receives a trigger, runs, returns, and the worker that ran it can be recycled the instant it finishes. Nothing in the function body survives to the next invocation: no local variables, no "where was I" pointer, no record that step two of a three-step process already succeeded. That design is what makes Functions cheap to scale, and it's the right model for the bulk of event handlers you write.

It stops being enough the moment one logical unit of work spans more than one step. Take the order example: validate the request, create the order record, send a confirmation. You want those to run in sequence, you want the second step to use the output of the first, and you want the whole thing to pick up where it left off if the host restarts between steps. None of that is possible inside one function, so the standard pattern is to split each step into its own queue-triggered function and pass a message down the chain.

That works, but look at what you end up owning. You need a correlation ID so you can tell which messages belong to the same order. You need a status table so you can answer "is order 4815 done yet" and so a retry doesn't redo a step that already completed. You need poison-queue handling, timeout logic, and some way to fan results back together if any step branches. You have hand-built a state machine, and state machines spread across five queues are where the 2 a.m. pages come from.

Durable Functions takes over the state. It records every step your workflow completes in durable storage, and it reconstructs your workflow's position from that record after any interruption. You write the sequence as ordinary C# with await between the steps; the runtime makes the sequence survive crashes, scale-ins, and host upgrades. The correlation ID, the status table, and the retry bookkeeping all move from your code into the framework.

The three roles: orchestrator, activity, client

Durable Functions splits a workflow into three kinds of function, each identified by a trigger or binding type. Keeping them straight is most of the battle when you're starting out, because the rules about what code is legal where depend entirely on which role you're in.

The orchestrator is the workflow itself. It's a function marked with the [OrchestrationTrigger] binding, and its job is to coordinate: call this step, wait for the result, decide what to call next. It contains the control flow (if, loops, sequencing) but does no real work of its own. The orchestrator is the one role with a hard constraint attached. Its body can be re-executed many times over the life of a single workflow instance, so the code in it must be deterministic. That single fact (the orchestrator replays) is what the rest of this series keeps coming back to; the replay mechanics and the exact list of rules are covered later in this article.

An activity is where the actual work happens. It's a function marked with [ActivityTrigger], and it's the only one of the three roles allowed to touch the outside world: database writes, HTTP calls, sending email, reading a blob. Activities can bind directly to their input type, so an activity that validates an order can take an OrderRequest parameter and nothing else. The guarantee Durable Functions gives you on activities is at-least-once execution, and that has a real consequence. An activity can run more than once for the same logical step (after a transient failure and retry, for instance), so activity logic should be idempotent. Sending a confirmation email twice because the worker died after sending but before recording success is the kind of bug this guarantee invites if you're not careful.

The client (often called the starter) is the entry point that kicks a workflow off and lets you query it. In the isolated worker model the client is the injected DurableTaskClient, supplied through the [DurableClient] binding on a normal trigger such as an HTTP function. It is not something you call from inside an orchestrator; it lives in regular functions that start instances with ScheduleNewOrchestrationInstanceAsync and hand back a status response the caller can poll.

Behind all three sits the task hub: the set of Azure Storage resources (queues, tables, and a couple of blob containers) that the default storage provider creates in your function app's storage account to hold the workflow's messages and history. You don't provision it or write to it directly. It's enough to know it exists and that it's where your workflow's durable state actually lives; the history table inside it is the star of the replay section below.

The chaining pattern

Function chaining is the simplest orchestration and the one you'll reach for most. You run a sequence of activities in order, where each step's output feeds the next. Here is the order-processing workflow as an orchestrator.

using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;

public record OrderRequest(string CustomerId, string Sku, int Quantity);

public static class OrderOrchestrator
{
    [Function(nameof(OrderOrchestrator))]
    public static async Task<string> RunOrchestrator(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        var order = context.GetInput<OrderRequest>()!;

        var validated = await context.CallActivityAsync<bool>(
            nameof(ValidateOrderActivity), order);
        if (!validated)
            return "Order validation failed";

        var orderId = await context.CallActivityAsync<string>(
            nameof(CreateOrderActivity), order);

        await context.CallActivityAsync(
            nameof(SendConfirmationActivity), orderId);

        return orderId;
    }
}

Read it top to bottom and it's the sequence from the opening, written as plain C#. context.GetInput<OrderRequest>() pulls the input the client passed when it started the instance; the ! is there because the input is typed as nullable and you know this orchestrator always gets one. Each await context.CallActivityAsync(...) schedules an activity and waits for its result before moving on, which is what gives you the chain: validated gates whether the order is created, and orderId (the output of CreateOrderActivity) becomes the input to SendConfirmationActivity.

The call comes in two shapes. When an activity returns a value you use the generic CallActivityAsync<TResult>, which gives you back a Task<TResult>: CallActivityAsync<bool> for the validation result, CallActivityAsync<string> for the new order ID. When an activity is fire-the-step with nothing to return, you use the non-generic CallActivityAsync, which returns a plain Task; that's the confirmation send. The first argument is the activity name as a TaskName, and since there's an implicit conversion from string, nameof(ValidateOrderActivity) works directly and keeps the name refactor-safe.

The activity is an ordinary function that does the real work. This is where I/O is allowed, so it's where validation against your database or rules engine actually happens.

public static class ValidateOrderActivity
{
    [Function(nameof(ValidateOrderActivity))]
    public static bool Run([ActivityTrigger] OrderRequest order)
    {
        // Real work and I/O belong here, never in the orchestrator:
        // check inventory, validate the customer, hit the database.
        return order.Quantity > 0 && !string.IsNullOrWhiteSpace(order.Sku);
    }
}

The activity binds straight to OrderRequest, the same type the orchestrator passed as input, so there's no manual deserialization. Keep in mind the at-least-once guarantee from earlier. If ValidateOrderActivity did something with side effects, you'd want running it twice to be safe. Pure validation like this is naturally idempotent, which is one reason it's a good first step in the chain.

Something has to start the workflow. That's the client, here an HTTP-triggered function that schedules a new instance and returns a status response the caller can poll.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask.Client;

public static class OrderClient
{
    [Function(nameof(StartOrder))]
    public static async Task<HttpResponseData> StartOrder(
        [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req,
        [DurableClient] DurableTaskClient client)
    {
        var order = await req.ReadFromJsonAsync<OrderRequest>();

        string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
            nameof(OrderOrchestrator), order);

        return client.CreateCheckStatusResponse(req, instanceId);
    }
}

ScheduleNewOrchestrationInstanceAsync starts the orchestrator with the deserialized order as input and returns the new instance's ID. CreateCheckStatusResponse then builds an HttpResponseData that's an HTTP 202 (Accepted) carrying a set of management URLs (status, terminate, and so on) keyed to that instance ID. The workflow runs asynchronously; the HTTP caller gets an immediate 202 and uses the status URL to find out when the order is done. The orchestrator never starts itself, and the client never contains workflow logic; each role stays in its lane.

One pin before you copy this into a project: the Durable extension is Microsoft.Azure.Functions.Worker.Extensions.DurableTask version 1.16.5 (the 1.x line, even on .NET 10), and TaskOrchestrationContext lives in Microsoft.DurableTask while DurableTaskClient lives in Microsoft.DurableTask.Client. These are the isolated-worker types; the in-process model used different names (IDurableOrchestrationContext, a different client), and mixing the two is the most common reason a copied snippet won't compile.

Replay mechanics

The orchestrator code reads like it runs once, top to bottom. It doesn't. To understand every rule that follows, you have to start from the one mechanic that drives them all: the orchestrator function body runs many times over the life of a single workflow instance.

Durable Functions doesn't snapshot the orchestrator's current state and resume it. It uses event sourcing. Every action the orchestrator takes (an activity scheduled, an activity completed, a timer created, a result returned) is appended to an append-only log in the History table, which lives in the task hub from the previous section. That log, not the in-memory state of the function, is the source of truth for where the workflow is.

Here is what actually happens when the order orchestrator runs. The first time, it executes from the top, hits await context.CallActivityAsync<bool>(nameof(ValidateOrderActivity), order), and yields. The dispatcher commits "ValidateOrderActivity scheduled" to the History table and unloads the orchestrator from memory entirely. There is now no thread, no stack, nothing in RAM waiting; the workflow exists only as rows in storage. When the validation activity finishes, its result is written to history and the orchestrator is woken back up.

On that wake-up, the orchestrator runs again from the very first line. It reaches the same CallActivityAsync call, but this time the framework checks the History table, sees that ValidateOrderActivity already completed, and replays the result from history instead of re-running the activity. The activity does not execute a second time; the recorded true (or false) is read straight out of storage and handed back, the validated local gets the value it had on the first run, and execution fast-forwards to the first step that hasn't completed yet, CreateOrderActivity. That step is now scheduled, the orchestrator yields again, and the cycle repeats until SendConfirmationActivity returns and the orchestrator runs to completion.

This is precisely what lets a workflow survive a crash. If the host dies after creating the order but before sending the confirmation, the History table still holds "order created" with its result. When a new worker picks the instance up, it replays from the top, fast-forwards past validation and creation using the recorded results, and resumes at exactly the confirmation step. No step that already succeeded runs twice as part of recovery, because recovery is just replay against the same history.

One practical consequence shows up the first time you add a log line to an orchestrator. It fires on every replay, so a single workflow can emit the same log message several times. The context exposes context.IsReplaying so you can suppress noise from replayed execution.

[Function(nameof(OrderOrchestrator))]
public static async Task<string> RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context,
    ILogger logger)
{
    var order = context.GetInput<OrderRequest>()!;

    if (!context.IsReplaying)
        logger.LogInformation("Starting order for customer {CustomerId}", order.CustomerId);

    var validated = await context.CallActivityAsync<bool>(
        nameof(ValidateOrderActivity), order);
    // ... rest of the chain
}

The if (!context.IsReplaying) guard means the "Starting order" line is written once, on the genuine first pass, and skipped on every replay. Without it the line would appear once for every time the orchestrator is dispatched, roughly once per activity in the chain.

Determinism rules

Replay is also why the orchestrator is the one role with code restrictions. If the body re-executes from the top every time, then any line that produces a different value on the second run than it did on the first will make the workflow take a different path during replay than it took originally, and the state reconstructed from history no longer matches the code's decisions. So the rule is blunt: orchestrator code must be deterministic. The same inputs and the same history must always produce the same sequence of calls.

These restrictions apply only to orchestrators. Activities can do anything; that's the point of them. It's the orchestrator, and only the orchestrator, that has to behave identically on every pass.

Here is the trap, written the way it usually gets written:

// BROKEN inside an orchestrator: re-evaluates on every replay.
[Function(nameof(OrderOrchestrator))]
public static async Task<string> RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<OrderRequest>()!;

    var receivedAt = DateTime.UtcNow;       // different value every replay
    var traceId = Guid.NewGuid();           // a new GUID every replay

    var validated = await context.CallActivityAsync<bool>(
        nameof(ValidateOrderActivity), order);
    // ...
}

Both DateTime.UtcNow and Guid.NewGuid() look harmless. The problem is that the first run records one timestamp and one GUID, then every replay computes fresh ones. If receivedAt or traceId ever feeds a branch, a comparison, or an activity input, the replayed run disagrees with the recorded history and the workflow corrupts. The failure is intermittent (replay only happens after a yield, and only some values flow into decisions), which is exactly what makes it hard to catch in testing and ugly in production.

The fix is to take time and identity from the context, which returns replay-stable values:

// CORRECT: context helpers return the same value on every replay.
var receivedAt = context.CurrentUtcDateTime;   // same instant every replay
var traceId = context.NewGuid();               // same GUID every replay

context.CurrentUtcDateTime records the current UTC time on the first execution and replays that exact instant afterward. context.NewGuid() produces a replay-safe GUID the same way. (One naming gotcha: the property is CurrentUtcDateTime. Some docs prose mis-spells it CurrentDateTimeUtc, which does not exist and will not compile.)

The same reasoning rules out a few more things. Don't call new Random() in an orchestrator; if you need randomness, return it from an activity, where the result is saved to history and replayed like any other activity output. Don't read environment variables or configuration directly, since those can change between the first run and a replay hours later; pass config in as orchestrator input or fetch it from an activity. And don't do real I/O (database, file, HTTP) in the orchestrator: it would fire again on every replay, and a network call is never replay-stable anyway. Push all of it into activities.

Delays have their own replay-safe form. A Task.Delay or Thread.Sleep in an orchestrator both blocks a thread and re-evaluates on replay; the durable equivalent is context.CreateTimer, which records the wake-up time in history and releases the worker entirely until then, so a workflow can wait minutes or days without holding any resources.

People trip on this rule for a fair reason: the broken code compiles, passes a quick local test, and looks like ordinary C#. The orchestrator only betrays you under replay, and replay only happens after a yield on a worker that may not be the one that started the run. The mental shortcut that keeps you safe is to read every line of an orchestrator and ask whether it would return the same value if the method ran again right now. If the answer is no, it belongs in an activity or behind a context helper.

When to use Durable Functions vs queues

Durable Functions is not the default answer to "my functions need to talk to each other." A plain storage queue with a table for state is cheaper, simpler to provision, and entirely enough for a large class of problems. If the work is a single hand-off (one function drops a message, another picks it up, does its job, and that's the end of it) a queue is the right tool. Keep those functions stateless and idempotent, carry whatever state you need on the message itself, and you never have to think about replay rules or determinism.

The line to watch for is stateful coordination across steps. The moment you need the output of one step to drive the next, retries that don't redo work that already succeeded, results from parallel branches aggregated back together, a workflow that waits on an external event or a human approval, durable delays measured in hours or days, or a status endpoint that answers "where is order 4815 right now," a bare queue stops being enough. You can build all of that on queues and tables, but you'll be hand-rolling correlation IDs, a status table, poison-message handling, and timeout plumbing across several queues. That hand-rolled state machine is exactly the thing an orchestrator replaces, and it replaces it with code that reads like the workflow it implements.

So the honest recommendation: reach for a queue first. Don't pull in Durable Functions for a single fire-and-forget hand-off; the orchestrator's constraints and the task hub's storage footprint are real overhead that buys you nothing there. The signal to switch is the second or third piece of coordination bookkeeping you find yourself writing by hand. When you're maintaining a correlation ID and a status table and retry logic just to keep a multi-step process straight, you've already built a worse version of what Durable Functions gives you, and that's when the orchestration earns its complexity.

What's next

Chaining is the first of several orchestration patterns, and it's deliberately the simplest: a straight line of activities. The same replay engine powers fan-out/fan-in (running activities in parallel and aggregating their results), waiting on external events for human-in-the-loop approval, and durable entities for stateful objects. Each is a later part of this series, and each rests on the single fact this article was built around: the orchestrator is C# that replays.

Which coordination problem pushed you past a plain queue first: a multi-step sequence that needed to survive restarts, fan-out with result aggregation, or waiting on an external event or approval?

DEV Community: Martin Oehlert

Infrastructure as Code: Terraform for Your New Stack

Four modules, drawn where the lifetime changes

The environment: every interesting argument is a one-way door

The app: one variable is the whole interface

Dapr components: scopes is the security boundary, and both ways of getting it wrong are silent

The subnet you cannot resize

The NSG rules everyone pastes are the Consumption rules

Private DNS: the zone the private endpoint will not create for you

Coming from azurerm v4

The identity chain, and the edge Terraform cannot see

State, and the lease nothing expires

Tags: the provider argument that does not exist

The pipeline: plan on the PR, apply behind an environment gate

Drift detection that is still believed in week three

Conclusion

Azure Front Door, API Management and Container Apps: What Each Hop Trusts

The shape you land on, and the tiers it forces

Front Door: what the edge actually does for you

WAF on a JSON API

One origin, and it is the gateway

The probe tests the gateway

API Management: the only layer that knows the caller

A backend per app, reached through the VNet

One inbound policy, in the order it has to run

Subscription keys, per product

Managed identity to Container Apps, and who else gets a token

Container Apps: the layer that decides who gets in and how many answer

Which ingress settings let the gateway in

The HTTP rule counts a rate, not concurrency

Queue depth for the app nobody calls over HTTP

Revisions: the rollout Front Door no longer does

The trust chain, hop by hop

Sidecar to app: the token you no longer set

App to data, and the network under all four hops

Verify on first deploy

Conclusion

Azure Functions to Container Apps, One Endpoint at a Time

Phase 1: deploy alongside and route nothing

The probe is the first thing that breaks

Phase 2: send the new endpoints and nothing else

A route is a switch, not a dial

The host header decides which app answers

Both runtimes on one Service Bus topic

consumerID is the subscription name, and it defaults to your app ID

The envelope goes one way cleanly and one way silently

How many requests one call actually makes

resiliency.yaml is the right answer and it does not reach your deployment

Phase 3: one endpoint at a time

Your 90/10 split is a 100/0 split

Caching and affinity change what a cutover means

Going private takes the whole environment

What you do when it goes wrong

The runbook is mostly about the writes

The health signal is not symmetric

There is no feature flag in API Management

The cross-runtime trace is two settings away from existing

Phase 4: turning the Function App off

ActiveMessageCount zero does not mean the old consumer is gone

Conclusion

.NET Aspire: Orchestrating Cloud-Native Apps

What the AppHost actually owns

It is not docker-compose, and it does not replace it

One file that starts everything

Three ways to write that reference, one that compiles

The state store you did not write is in memory

ServiceDefaults: what every service inherits

The health endpoints disappear outside Development

One request, seven spans

Restarting the app proves nothing about durability

If you switch to the http profile

Where the model stops

Conclusion

Building Your First Dapr + Web API Service

Two services, two app IDs

One JSON context, wired two different ways

Starting both, with a sidecar each

Calling a service you cannot name

What the handler does, and what it refuses to do

Failures as values

Dapr components: `scopes` is the security boundary, and both ways of getting it wrong are silent