Running Inspect AI Evals on a Custom OpenAI-Compatible API: openai-api/ Model Names, Two Retry Layers and Hidden Errors

Inspect AI is the open-source evaluation framework from the UK AI Security Institute. You write a task (dataset, solver, scorer), run inspect eval, and get a log you can browse. It has first-party providers for the big labs, plus a generic openai-api provider for anything that speaks the OpenAI Chat Completions format: vLLM, a LiteLLM proxy, Ollama's /v1 endpoint or a hosted gateway.

The setup is two environment variables, but a few defaults change your results and your request count in ways that are easy to miss. I tested inspect_ai 0.3.281 with openai 3.35.1 on Python 3.13.5 against a small local mock server. The mock logged the path, headers and JSON body of every request and could return 400, 401, 429 or 503 on demand. Everything below is what went over the wire. The examples use APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions with that in mind. Its base URL is https://apiclaw.biz/v1. Replace YOUR_MODEL_ID with an exact ID from your provider's model list rather than guessing one.

The model string picks your environment variables

pip install inspect-ai openai

export APICLAW_API_KEY="your-key"
export APICLAW_BASE_URL="https://apiclaw.biz/v1"

inspect eval arith.py --model openai-api/apiclaw/YOUR_MODEL_ID

The model string has three parts: openai-api, a service name you choose, and the model ID. Inspect upper-cases the service name and reads <SERVICE>_API_KEY and <SERVICE>_BASE_URL. Hyphens become underscores, so openai-api/my-gw/... reads MY_GW_API_KEY and MY_GW_BASE_URL. Pick a short name and stick with it, because it also appears in your logs and progress display.

If either variable is missing you get a clear message before any request goes out: No APICLAW_API_KEY defined in the environment. (or ..._BASE_URL). If you leave the service out entirely (openai-api/YOUR_MODEL_ID), you get a ValueError: openai-api model names must include a service prefix wrapped in a long TaskGroup traceback, which is easy to misread as a network problem.

Everything after the service name is passed through as the model ID, slashes included. openai-api/apiclaw/deepseek/deepseek-chat sent "model": "deepseek/deepseek-chat", which is what you want for gateways that use vendor-prefixed IDs.

Keep /v1 in the base URL

Inspect hands the base URL to the OpenAI Python SDK unchanged, and the SDK only appends /chat/completions. With APICLAW_BASE_URL=http://host/v1 the mock saw POST /v1/chat/completions. Without /v1 it saw POST /chat/completions, which most servers answer with a 404. The request carried Authorization: Bearer <your key> and the SDK's AsyncOpenAI/Python user agent.

What a default request contains

Here is the task I used:

# arith.py
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate

@task
def arith():
    return Task(
        dataset=[Sample(input=f"What is 2+2? (variant {i})", target="4") for i in range(20)],
        solver=generate(),
        scorer=match(),
    )

With no generation options, the body was just model and messages. No temperature, no max_tokens, no seed, so you get whatever defaults the server applies. If you want reproducible runs, say so on the command line: --temperature 0 --max-tokens 50 --seed 7 arrived as "temperature": 0.0, "max_tokens": 50, "seed": 7.

Streaming is off unless you ask for it. -M stream=true added "stream": true and "stream_options": {"include_usage": true}, and token counts still showed up in the summary because the mock sent a usage chunk at the end. If your server does not send usage when streaming, expect missing token totals.

If the model wraps its reasoning in <think>...</think>, Inspect splits it out. The mock answered <think>let me reason</think>ANSWER: 4, and the stored message had a reasoning block plus a text block, with output.completion equal to ANSWER: 4. The match() scorer saw only the answer.

Concurrency is adaptive by default

Inspect does not send one request at a time. Adaptive concurrency is on by default; the config docstring gives its bounds as a minimum of 10, a start of 20 and a maximum of 100 connections. With 60 samples and a mock that took 300 ms per reply, I saw up to 30 requests in flight at once.

That matters if your provider limits requests per minute or per day, which is how many flat-rate plans work. Set a fixed number instead:

inspect eval arith.py --model openai-api/apiclaw/YOUR_MODEL_ID --max-connections 4

With --max-connections 4 the peak was exactly 4. Also count your requests honestly: every sample is at least one call, and model-graded scorers add more (see below).

Two retry layers multiply

This was the biggest surprise. Inspect creates an AsyncOpenAI client and leaves the SDK's own max_retries at its default of 2, so every Inspect attempt is up to three HTTP requests. Inspect then retries failed attempts itself, and its max_retries defaults to unlimited.

With the mock returning 429 and Retry-After: 1 for one sample:

• --max-retries 3 sent 12 requests over about 37 seconds (4 Inspect attempts times 3 SDK attempts; the x-stainless-retry-count header went 0, 1, 2 four times).
• With no limits set, it had sent 15 requests after 75 seconds and was still going when I killed it. The gaps between Inspect's attempts grew from about 3 to 9, 13 and 26 seconds.
• 503 behaved the same as 429. A 401 or 400 was sent once and not retried.

To get one retry layer you control, turn the SDK's off with a model arg and give Inspect a limit:

inspect eval arith.py --model openai-api/apiclaw/YOUR_MODEL_ID \
  -M max_retries=0 --max-retries 3 --timeout 120

-M arguments go to the OpenAI client constructor, so max_retries=0 applies there. With that and --max-retries 3 the same 429 produced 4 requests, not 12. --timeout caps the whole request including retries; with --timeout 20 the run stopped after 4 requests.

One bad request fails the whole eval, unless you let it hide

I made the mock return a 400 for one sample out of five. By default the eval ended with status error and no scores at all, even though four samples had succeeded.

The usual fix is --fail-on-error 0.25 (tolerate up to 25% failed samples) or --no-fail-on-error. Both gave status success and accuracy 1.000. That number is computed over the four samples that worked; the failed one is left out rather than counted as wrong. If a server rejects your hardest prompts, for example because they exceed the context window, your accuracy goes up. Check the errored sample count in the log before you compare models:

from inspect_ai.log import read_eval_log

log = read_eval_log("logs/your-run.eval")
errored = sum(1 for s in log.samples if s.error)
print(log.status, f"{errored}/{len(log.samples)} samples errored")

The grader is the model you are testing

model_graded_qa() and the other model-graded scorers use the grader model role, and if you do not set it they use the model under evaluation. In my run each sample sent two requests to the same model: the answer, then a grading prompt starting "You are assessing a submitted answer...". Letting a model grade itself is rarely what you want, and it doubles your request count. Point the grader somewhere else:

inspect eval graded.py --model openai-api/apiclaw/YOUR_MODEL_ID \
  --model-role grader=openai-api/apiclaw/YOUR_GRADER_MODEL_ID

With that set, the mock saw the answer request go to one model ID and the grading request to the other. Both use the same APICLAW_* variables because they share a service name.

The same settings from Python

from inspect_ai import eval
from arith import arith

logs = eval(
    arith(),
    model="openai-api/apiclaw/YOUR_MODEL_ID",
    model_base_url="https://apiclaw.biz/v1",
    model_args={"max_retries": 0},
    max_retries=3,
    max_connections=4,
)
print(logs[0].status)

model_base_url replaces the _BASE_URL variable; the key still comes from APICLAW_API_KEY. For one-off calls outside an eval, get_model("openai-api/apiclaw/YOUR_MODEL_ID", base_url=..., api_key=...) and await model.generate("...") worked the same way.

What ends up in the logs

Inspect writes a .eval file for every run, including failed ones, and it stores each request's messages and parameters so you can inspect them in inspect view. I searched all 18 logs from my tests for the API key and for the string Bearer and found neither. Prompts and responses are stored in full, so treat the logs/ folder like any other file with test data in it.

Checklist

• Use openai-api/<service>/<model> and set <SERVICE>_API_KEY and <SERVICE>_BASE_URL, with /v1 in the URL.
• Pass --temperature and --seed explicitly; nothing is sent by default.
• Set --max-connections if your plan limits requests.
• Use -M max_retries=0 with --max-retries N and --timeout, or a 429 can retry for a long time.
• After a run with --fail-on-error or --no-fail-on-error, count errored samples before trusting accuracy.
• Give model-graded scorers their own grader with --model-role grader=....

Disclosure: I build APIClaw, the OpenAI-compatible gateway used in these examples (flat monthly plans; crypto on the site, other payment methods through support). Everything above applies to any compatible endpoint.