DEV Community

Cover image for APIM-Track and Calculate LLM Token Usage
shailugit
shailugit

Posted on

APIM-Track and Calculate LLM Token Usage

#api #azure #apigateway #llm

The llm-emit-token-metric policy in Azure API Management enables the automatic emission of custom LLM token‑usage metrics to Application Insights . These metrics include prompt tokens, completion tokens, and total tokens, depending on the capabilities of the underlying model provider. By capturing these values at runtime, the policy provides a reliable mechanism for cost tracking, usage monitoring, and anomaly detection across all LLM‑powered APIs.

From an operational perspective, the policy integrates seamlessly into the APIM request pipeline. When a client invokes an APIM‑hosted LLM endpoint, APIM forwards the request to the configured backend—such as Azure OpenAI, OpenAI, Anthropic, or other supported providers. The backend returns its response along with usage metadata that includes token counts. The llm-emit-token-metric policy extracts these token metrics and publishes them to the Application Insights customMetrics table, enriched with any custom dimensions defined in the policy. This creates a structured, queryable telemetry stream that engineering teams can use to analyze consumption patterns, build FinOps dashboards, and enforce governance around LLM usage.

Below is a sample API using three custom dimensions.

Please note that this policy emits the following metrics into Application Insights customMetrics table

  • Prompt Tokens > Input tokens sent to the model

  • Completion Tokens > Output tokens generated

  • Total Tokens > Sum of prompt + completion

To check the Metrics you can leverage the following queries

customMetrics
| where name in ("Prompt Tokens", "Completion Tokens", "Total Tokens")
| take 50

customMetrics
| where name in ("Prompt Tokens", "Completion Tokens", "Total Tokens")
| extend ApiName = tostring(customDimensions.["ApiName"])
| summarize
    TotalRequests = countif(name == "Total Tokens"),
    TotalTokens = sumif(value, name == "Total Tokens"),
    PromptTokens = sumif(value, name == "Prompt Tokens"),
    CompletionTokens = sumif(value, name == "Completion Tokens"),
    AvgTokensPerRequest = round(avgif(value, name == "Total Tokens"), 1)
    by ApiName
| order by TotalTokens desc
Enter fullscreen mode Exit fullscreen mode

Limitaion

Do not use highly variable dimensions (e.g., user-agent, request-id).
Use stable identifiers like: SubscriptionId, ApiName, DeploymentName, ClientIp etc.
Please review the policy limitations here carefully as this is not intended to traces individual calls like calculating cost per request or per user basis as it can easy reach the Azure monitor cardinality limits. You can create your solution eighter sending individual records using log-to-EventHub policy or using send request policy or using trace policy
https://learn.microsoft.com/en-us/azure/api-management/llm-emit-token-metric-policy

Top comments (0)