Your AI Agent Got a 200. The Tool Call Still Failed.

AI agents rarely fail in the place where the log says they failed.

A request can return HTTP 200, the model can produce a perfectly valid response, and the workflow can still be wrong because a tool call was rejected, timed out, or executed against stale arguments. The application then continues with an empty result and writes a confident answer.

I have seen this happen in two forms:

• the model returns a tool call, but the application never executes it;
• the application executes the tool, gets an error, and sends the next model request without preserving that failure.

Both bugs look like model quality problems. They are usually integration problems.

This guide shows a small failure-aware loop for OpenAI-compatible APIs. It uses ordinary Python, the official OpenAI client, and a deliberately unreliable tool. The point is not to build a framework. The point is to make every transition visible enough to test.

Start with the HTTP boundary

Before adding an agent framework, prove that the API boundary works. Keep the base URL and model ID configurable:

export OPENAI_BASE_URL="https://www.aifast.hk/v1"
export OPENAI_API_KEY="replace-with-your-key"
export OPENAI_MODEL="copy-an-exact-model-id-from-your-provider"

Check model discovery first:

curl -sS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Accept: application/json" | jq -r '.data[].id'

Then send the smallest chat request:

curl -sS "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$OPENAI_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK\"}],\"temperature\":0}"

Do not put the real key in a shell history, issue, or screenshot. The examples use an environment variable so the credential stays outside the request body.

A successful chat request proves very little about tools. Model discovery, chat completions, streaming, structured output, and tool calls can fail independently. Test the feature you plan to use.

The broken loop

Here is the kind of loop that creates silent failures:

response = client.chat.completions.create(
    model=model,
    messages=messages,
    tools=tools,
)

message = response.choices[0].message
messages.append(message)

for call in message.tool_calls or []:
    result = run_tool(call.function.name, call.function.arguments)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": result,
    })

# The next request happens even when run_tool() returned an exception string.

There are three problems here.

First, the code does not distinguish a tool result from a tool failure. Second, it does not validate that the requested tool exists. Third, it assumes that a model response containing tool_calls is enough to prove that the requested action happened.

The model only suggested an action. Your application owns execution.

Make failures part of the message history

A safer version records a structured result for every call. The model can then decide whether to retry, ask for a missing input, or explain that the task cannot be completed.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://www.aifast.hk/v1"),
)

model = os.environ["OPENAI_MODEL"]

TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for one city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"}
            },
            "required": ["city"],
            "additionalProperties": False
        }
    }
}]


def get_weather(city):
    # Replace this with a real provider call.
    if city.lower() == "timeout-city":
        raise TimeoutError("weather provider timed out")
    return {"city": city, "temperature_c": 21, "source": "demo"}


def execute_tool(call):
    name = call.function.name
    try:
        arguments = json.loads(call.function.arguments)
    except json.JSONDecodeError as exc:
        return {
            "ok": False,
            "error": "invalid_arguments",
            "message": str(exc),
        }

    if name != "get_weather":
        return {
            "ok": False,
            "error": "unknown_tool",
            "message": f"Tool {name!r} is not registered.",
        }

    if not isinstance(arguments.get("city"), str) or not arguments["city"].strip():
        return {
            "ok": False,
            "error": "validation_error",
            "message": "city must be a non-empty string",
        }

    try:
        return {"ok": True, "data": get_weather(arguments["city"])}
    except TimeoutError as exc:
        return {
            "ok": False,
            "error": "upstream_timeout",
            "retryable": True,
            "message": str(exc),
        }
    except Exception as exc:
        return {
            "ok": False,
            "error": "tool_error",
            "retryable": False,
            "message": str(exc),
        }

The important detail is not the weather function. It is the shape of the returned object. A caller can now tell the difference between successful data, malformed arguments, an unknown tool, and a retryable upstream error.

Preserve the tool call ID

The next model request must contain the assistant message with its tool call and a tool message carrying the same tool_call_id.

def run_agent(user_text, max_rounds=4):
    messages = [
        {
            "role": "system",
            "content": "Use tools when needed. Never invent tool results. "
                       "If a tool fails, explain the failure or try again only "
                       "when the error is retryable.",
        },
        {"role": "user", "content": user_text},
    ]

    for round_no in range(max_rounds):
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=TOOLS,
            temperature=0,
        )
        assistant = response.choices[0].message
        messages.append(assistant)

        calls = assistant.tool_calls or []
        if not calls:
            return assistant.content or "The model returned no text."

        for call in calls:
            result = execute_tool(call)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result, ensure_ascii=False),
            })

    raise RuntimeError("agent exceeded the tool-call round limit")

Do not silently drop the assistant tool-call message. Do not create a tool message with a made-up ID. Providers may reject the sequence, while other providers or wrappers may hide the mistake and produce confusing output.

Test the failure path, not only the happy path

Most agent tests ask for a successful answer. That misses the bug.

You need at least these cases:

1. the model answers without a tool;
2. the model requests a valid tool and the tool succeeds;
3. the model sends malformed JSON arguments;
4. the model requests a tool that is not registered;
5. the tool returns a retryable timeout;
6. the provider returns an API error before the tool loop starts;
7. the model emits multiple tool calls in one response;
8. the loop reaches its maximum number of rounds.

A minimal test should assert the message sequence, not just the final text:

def test_tool_failure_is_preserved():
    result = execute_tool(type("Call", (), {
        "function": type("Function", (), {
            "name": "get_weather",
            "arguments": '{"city":"timeout-city"}',
        })(),
    })())

    assert result["ok"] is False
    assert result["error"] == "upstream_timeout"
    assert result["retryable"] is True

For an integration test, use a fake model response and inspect the second request. It should include the original tool call ID and a JSON error object. If the second request contains only the user's original question, your application discarded the evidence the model needs.

Retry only what is safe to retry

A timeout from an idempotent read operation can usually be retried with a limit and backoff. A payment, email, database write, or deployment cannot be retried safely just because the network timed out. The server may have completed the action before the client lost the response.

Keep retry policy with the tool, not in a blanket agent wrapper:

RETRYABLE = {"upstream_timeout", "connection_reset", "rate_limited"}


def should_retry(result, attempt):
    return (
        result.get("ok") is False
        and result.get("error") in RETRYABLE
        and attempt < 2
    )

For rate limits, respect the provider's retry headers when present. For 401, fix credentials or configuration; retrying the same request will not repair an invalid key. For 400, inspect the payload and model ID. For 5xx, capture the request ID and response body, then decide whether the operation is safe to repeat.

The status code is a starting point, not a diagnosis.

Log evidence without logging secrets

When a tool loop fails, record:

• request ID, if the provider returns one;
• model ID and API operation;
• tool name and a hash or redacted form of arguments;
• round number and tool-call ID;
• HTTP status and provider error code;
• elapsed time for the model and the tool;
• whether the operation was read-only or mutating.

Do not record API keys, full authorization headers, private prompts, or user data by default. A log that proves the failure but leaks the credential is not a successful debugging setup.

What I check before calling an agent reliable

I run one real request through the same base URL used by the application. I verify the exact model ID from /models. Then I force a tool timeout and confirm that the next model request contains the failure, not an empty success placeholder.

That last test catches more bugs than another happy-path demo.

I also test multiple tool calls in one model response. The executor records each call ID separately and returns one tool message for every call, even when one succeeds and another fails. Collapsing those results into a single message makes it hard to tell which action produced which output.

For production incidents, I keep the provider request ID beside the tool-call ID. They answer different questions: the request ID helps the API provider trace the model request, while the tool-call ID connects the model's proposed action to the result my application returned. If either identifier disappears between log lines, debugging turns into guesswork.

Write operations need one more guard. Before retrying a payment, deployment, email, or database mutation, attach an idempotency key or check the target state. A timeout means the client did not receive a response; it does not prove the server abandoned the operation. Blind retries can turn one uncertain action into two real actions.

An OpenAI-compatible endpoint gives you a familiar request shape. It does not guarantee identical models, tool behavior, limits, or error details. Keep the wire-level test, tool executor, retry policy, and message history visible. When something breaks, you want evidence from the boundary instead of a polished answer that happens to be wrong.

References

• OpenAI function calling guide
• OpenAI error codes
• OpenAI production best practices
• AIFast developer documentation

The examples use an OpenAI-compatible API endpoint and a demo tool. Test the exact model and tool capabilities in your own environment before relying on them in production.