Fixing an Azure Function Timeout on a Slow Model Call

#ai #debugging #llm #machinelearning

Timeout value of 00:05:00 exceeded by function and a client that gives up after roughly four minutes are two different failures with two different fixes, and the setting that fixes the first has no effect on the second.

The error you are looking at

The host-side timeout appears in your logs in this shape, with the duration formatted as hh:mm:ss:

[Error] Timeout value of 00:05:00 exceeded by function
'Functions.chat' (Id: '32daf701-...'). Initiating cancellation.

In a .NET stack trace the same event surfaces as Microsoft.Azure.WebJobs.Host.FunctionTimeoutException. This is the Functions host enforcing its own limit: Microsoft documents that when an execution exceeds the timeout duration, a timeout error occurs and the language worker process restarts — and for C# apps running in-process, the host process itself restarts. That restart is why a single timed-out invocation can also disturb requests that were running alongside it on the same instance.

If instead your client sees a connection reset or an empty response at a little under four minutes, and the function log shows no timeout at all, you are hitting a different limit. Read on.

Three clocks, not one

The host’s functionTimeout. Set in host.json, it bounds one execution. Microsoft documents defaults of 5 minutes on the Consumption plan with a maximum of 10, and 30 minutes with no enforced maximum on Flex Consumption, Premium, Dedicated and Container Apps. This is the clock the error message above belongs to.
The platform’s HTTP response limit. Microsoft documents that regardless of the function app timeout setting, 230 seconds is the maximum amount of time an HTTP-triggered function can take to respond to a request, because of the default idle timeout of Azure Load Balancer. No host.json value moves it, and no hosting plan removes it. Microsoft, Azure Functions scale and hosting.
Your HTTP client’s timeout. The SDK you use to call the model has a default request timeout and a default retry count, and they compose: two retries of a call with a generous timeout can outlast both clocks above without any single attempt looking slow. Check the value your SDK version actually uses rather than assuming, and set it explicitly.

Diagnosing is a matter of which clock rang. Host timeout: the log line above, with a duration that matches your plan default. Platform limit: roughly 230 seconds, no host-side error, dead connection. Client timeout: an exception from the SDK inside your own handler, which you can catch.

Raising the one you can raise

functionTimeout lives at the root of host.json and takes a duration string:

{
  "version": "2.0",
  "functionTimeout": "00:10:00",
  "extensions": {
    "http": {
      "routePrefix": "api"
    }
  }
}

On the Consumption plan, 00:10:00 is the ceiling and a larger value is not honoured. On Flex Consumption, Premium and Dedicated there is no enforced maximum — but “unbounded” is not “guaranteed”. Microsoft documents a 60-minute grace period for an execution during scale-in on Flex Consumption and Premium, and a 10-minute grace period during platform updates. A three-hour execution is legal and will still occasionally be cut off by a platform update.

Getting past 230 seconds

Because the 230-second limit is an idle timeout on the connection, the two documented ways past it are both about not being idle, or about not being on the connection at all.

Stream the response. Set stream: true on the completion request and write each chunk out as it arrives. Bytes flow continuously, the connection is never idle, and the reader starts at time-to-first-token rather than at the end. This is the right answer for anything a human is watching.
Return 202 and let the caller poll. Microsoft points at the Durable Functions async operation-tracking pattern for exactly this: accept the request, start the work, return an accepted response with a status URL, and let the client poll it. The HTTP request that must finish inside 230 seconds is now the one that just enqueues work.
Bound the generation. Set max_tokens to the smallest value that serves the use case. Output tokens cost time in direct proportion, so an unbounded max_tokens is an unbounded wall-clock time, and on Azure OpenAI it also inflates the estimate used for rate limiting — see the 429 page.

The clock in your own code

Set the SDK timeout to something meaningfully below your function timeout, and set retries deliberately. The failure mode to avoid is a client that retries three times at 120 seconds each inside a function bounded at 300 — the host kills the execution mid-retry, the worker restarts, and the log shows a timeout with no clue that three model calls were paid for and discarded.

const client = new AzureOpenAI({
  endpoint: process.env.AZURE_OPENAI_ENDPOINT,
  apiKey: process.env.AZURE_OPENAI_API_KEY,
  apiVersion: "2024-10-21",
  timeout: 60_000,   // per attempt
  maxRetries: 2,     // worst case ~180s, inside a 230s budget
});

The arithmetic is the point: attempts multiplied by per-attempt timeout must fit inside the smallest of the three clocks, which for an HTTP trigger is always 230 seconds. If it does not fit, the fix is not a larger timeout somewhere. It is a different response shape.

One more consequence of the host-side timeout is worth planning for. The documented behaviour on a timeout is that the language worker process restarts, and for C# apps running in-process the host process itself restarts. Neither of those is scoped to the offending invocation. Every other execution running on that instance goes down with it. So a single pathological request — one enormous prompt, one runaway generation — does not degrade one caller, it takes out everything that instance was working on, and the resulting error pattern looks like an infrastructure fault rather than one bad input. Cap the input size at the edge, not just the output.

The timeout that is really a queue

There is a fourth clock, and it is the one that produces a timeout with no slow model call anywhere in the trace. The HTTP extension throttles concurrency per instance, and the defaults differ by plan: maxConcurrentRequests defaults to 100 on a Consumption plan and to unbounded (-1) on Premium and Dedicated, and maxOutstandingRequests defaults to 200 and unbounded on the same split.

maxConcurrentRequests is the number of HTTP functions executed in parallel. A model proxy is almost entirely wait — the process is doing nothing while a provider generates tokens — so a hundred concurrent invocations is not a hundred busy CPUs, it is a hundred open sockets. On a Consumption plan, request 101 does not fail; it queues. Its total latency is now queue time plus model time, your traces attribute all of it to the outbound call because that is the only span you instrumented, and the model looks like it slowed down when it did not.

maxOutstandingRequests is the ceiling on queued plus in-progress requests, and requests over it are rejected with a 429 “Too Busy” rather than queued. Separately, dynamicThrottlesEnabled — documented as defaulting to true on Consumption and false on Premium and Dedicated — periodically samples system counters for connections, threads, processes, memory and CPU, and rejects requests with the same 429 while any counter sits above a built-in 80% threshold. Both are useful behaviours. Both are also easy to mistake for the model provider throttling you, and the remedy is opposite in each case.

{
  "version": "2.0",
  "functionTimeout": "00:10:00",
  "extensions": {
    "http": {
      "maxConcurrentRequests": 400,
      "maxOutstandingRequests": 800
    }
  }
}

Raising these on a wait-bound workload is usually right, but raise them with the instance’s connection limit in mind. The legacy Consumption plan is documented at 600 active and 1,200 total outbound connections per instance, so a concurrency setting above that converts a queue you could see into connection errors you cannot. The hosting plan page works through where those limits sit.

The 230-second limit, the plan timeout table and the grace periods are Microsoft’s documented values at the time of writing. They have changed as plans were added and retired; check the hosting comparison article before designing around a specific number.