An HTTP-Triggered Azure Function That Calls a Model API

#ai #machinelearning #tutorial #llm

A function that takes an HTTP request, calls a model, and returns the answer is about twenty lines. The interesting part is what the platform does around those twenty lines, and two of its defaults will surprise you on the first day of real traffic.

Scaffold the project

This uses the Node.js v4 programming model, where a function is registered in code rather than described in a function.json file. Azure Functions Core Tools creates the project:

func init model-proxy --worker-runtime node --language javascript --model V4
cd model-proxy
npm install openai

The v4 model puts every function under src/functions/ and discovers them from the main field in package.json. There is no per-function directory and no binding JSON to keep in sync with the code.

The handler

Create src/functions/chat.js. The client is constructed once at module scope, outside the handler, so that its connection pool survives between invocations on a warm instance — a new client per request means a new TLS handshake per request, which on a model call is pure added latency.

const { app } = require("@azure/functions");
const { AzureOpenAI } = require("openai");

const client = new AzureOpenAI({
  endpoint: process.env.AZURE_OPENAI_ENDPOINT,
  apiKey: process.env.AZURE_OPENAI_API_KEY,
  apiVersion: "2024-10-21",
});

app.http("chat", {
  methods: ["POST"],
  authLevel: "function",
  handler: async (request, context) => {
    const body = await request.json();
    if (!body || typeof body.prompt !== "string") {
      return { status: 400, jsonBody: { error: "prompt is required" } };
    }

    const completion = await client.chat.completions.create({
      model: process.env.AZURE_OPENAI_DEPLOYMENT,
      messages: [{ role: "user", content: body.prompt }],
      max_tokens: 512,
    });

    context.log("finish_reason", completion.choices[0].finish_reason);
    return { jsonBody: { text: completion.choices[0].message.content } };
  },
});

Two details in there are Azure-specific rather than OpenAI-specific. The model parameter is not a model name — on Azure it is your deployment name, which you chose when you created the deployment and which may or may not resemble the model it points at. And apiVersion is mandatory; the Azure endpoint rejects a request without one, and the version you pin decides which request fields the service will accept.

Returning jsonBody lets the runtime set the content type and serialise for you. Returning body instead gives you a string response with no JSON content type, which is a common cause of a client that receives the right bytes and refuses to parse them.

Where the key goes

Locally, settings live in local.settings.json, which is not deployed and should not be committed. In Azure they are app settings:

az functionapp config appsettings set \
  --resource-group rg-model \
  --name fn-model-proxy \
  --settings \
    AZURE_OPENAI_ENDPOINT="https://my-aoai.openai.azure.com/" \
    AZURE_OPENAI_DEPLOYMENT="gpt-4o-prod" \
    AZURE_OPENAI_API_KEY="<key>"

An app setting is encrypted at rest, but it is readable by anyone with the right control-plane role and it appears in ARM exports. For anything you would mind seeing in a support ticket, put the value in a vault and reference it — Key Vault references in function app settings keeps the code unchanged. Better still, drop the key entirely and use the function app’s managed identity against the Cognitive Services OpenAI User role.

Deploy it

Create the function app. Microsoft now documents the Consumption plan as legacy and points new serverless apps at the Flex Consumption plan; check az functionapp create --help for the current Flex-specific location flag, because it differs from the Consumption one shown here.
```
az functionapp create \
  --resource-group rg-model \
  --name fn-model-proxy \
  --storage-account stmodelproxy \
  --consumption-plan-location westeurope \
  --runtime node --runtime-version 20 --functions-version 4
```
Publish the code. Core Tools builds a package and pushes it to the app’s deployment endpoint.
```
func azure functionapp publish fn-model-proxy
```

Retrieve the function key and call it. With authLevel: "function" the key goes in a x-functions-key header, or in a code query parameter.

KEY=$(az functionapp function keys list \
  --resource-group rg-model --name fn-model-proxy \
  --function-name chat --query default -o tsv)

curl -X POST "https://fn-model-proxy.azurewebsites.net/api/chat" \
  -H "x-functions-key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Summarise the CAP theorem in two sentences."}'

Two limits that bite immediately

The 230-second wall. Microsoft documents that regardless of the function app timeout setting, 230 seconds is the maximum time an HTTP-triggered function can take to respond, because of the default idle timeout of Azure Load Balancer. Raising functionTimeout in host.json does not move it. A slow model call with a large max_tokens can reach it, and the client sees a connection close rather than a clean error. The documented ways out are to stream the response so bytes keep arriving, or to return 202 immediately and let the caller poll — the timeout page works through both. Microsoft, Azure Functions scale and hosting.

The authLevel default is not what you think. Microsoft documents three values — anonymous (no key), function (a function-specific key) and admin (the master key) — and then documents that when no level is set, C#, Java, PowerShell and Python default to function, the Node v3 model defaults to function, and the Node v4 model defaults to anonymous. Omit authLevel from the snippet above and you have published an unauthenticated proxy to a metered model endpoint on a public hostname. Set it explicitly, always.

The throttles you did not configure

A function that spends most of its life waiting on a model is a high-concurrency, low-CPU workload, which is precisely the shape the HTTP extension’s defaults are not tuned for. Three settings in host.json govern it, and all three have plan-dependent defaults:

{
  "version": "2.0",
  "extensions": {
    "http": {
      "routePrefix": "api",
      "maxConcurrentRequests": 100,
      "maxOutstandingRequests": 200,
      "dynamicThrottlesEnabled": true
    }
  }
}

maxConcurrentRequests — the maximum number of HTTP functions executed in parallel on one instance. Microsoft documents the default as 100 on a Consumption plan and unbounded (-1) on Premium and Dedicated. Request 101 does not fail; it waits, and that wait is latency your traces will attribute to the model.
maxOutstandingRequests — queued plus in-progress. Default 200 on Consumption, unbounded on Premium and Dedicated. Requests over this limit are rejected with a 429 “Too Busy” response.
dynamicThrottlesEnabled — default true on Consumption and false on Premium and Dedicated. When on, the pipeline periodically checks system counters — connections, threads, processes, memory, CPU — and rejects requests with a 429 “Too Busy” while any of them is over a built-in high threshold of 80%.

Put those together and you get the single most misdiagnosed failure in this pattern. Under load, your function proxying Azure OpenAI starts returning 429. The obvious conclusion is that the model deployment is rate limiting you, and the obvious remedy is a quota increase — which changes nothing, because the 429 was generated by your own function host before a request ever reached Azure OpenAI.

The tell is in the body and the headers: a real Azure OpenAI 429 carries x-ratelimit-* headers and a message about rate limits, and a “Too Busy” 429 carries neither. Log which side produced it, or you will spend a day on the wrong 429.

routePrefix is the last of the four and is cosmetic until it is not: it defaults to api, which is why the URL above contains a segment nobody put in the code. Setting it to an empty string removes the prefix entirely, which matters if you are placing this function behind a gateway that already owns the path.

Plan names, runtime versions and CLI flags in this area move; the 230-second figure, the authorization levels and the throttle defaults are the documented values at the time of writing. Confirm against the Azure Functions hosting comparison and the HTTP bindings reference before you design around any of them.