Was the model unsure, or did the API call fail?

In an LLM application, seeing UNCERTAIN doesn't necessarily mean the model was unsure.

Sometimes the provider call failed entirely. If your application treats both cases the same way, you can end up debugging prompts and confidence thresholds when the real problem is an API key, a model name, or a network issue.

Reproduce the difference without an API key

You don't need a provider SDK or real credentials to see the distinction. This small example simulates a failed call and a model response that doesn't pass the confidence check.

Requires boothpy >= 0.5.4.

import booth

def dead(prompt):
    raise RuntimeError("401: invalid API key")

def unsure(prompt):
    return '{"answer": "Paris", "confidence": 0.2}'

a = booth.check(dead, "What is the capital of France?")
b = booth.check(unsure, "What is the capital of France?")

print(a.status, a.answer, a.call_failed, len(a.attempts))
# UNCERTAIN None True 2

print(b.status, b.answer, b.call_failed)
# UNCERTAIN Paris False

Both checks return UNCERTAIN, but for different reasons.

In the first case, the call fails. With the default max_retries=1, BOOTH makes two attempts. Neither reaches a usable response, so the result has no answer and call_failed is True.

In the second case, the call returns a response, but its confidence is too low for the checkpoint to accept it. The answer is available, and call_failed is False.

That distinction matters: retrying a request with an invalid API key costs another call without fixing the credential.

What call_failed actually tells you

BOOTH added result.call_failed in v0.5.4. It's also present in v0.5.6.

One detail matters: result.call_failed is True only when every attempt was a call failure.

Imagine the first attempt returns an uncertain response, and the retry then hits an unavailable provider. The overall result's call_failed can still be False, because not every attempt failed at the call level. To inspect the last attempt specifically, check:

last_attempt = result.attempts[-1]

if last_attempt.call_failed:
    print("The latest call failed:", last_attempt.error)

Log call failures, not every attempt error

BOOTH also provides an on_attempt callback. Be careful about which field you check: attempt.error can be populated for parsing failures too, such as a response where no valid JSON object was found. An error value alone doesn't prove that the provider call failed.

Check attempt.call_failed instead:

def log_attempt(index, attempt):
    if attempt.call_failed:
        print(
            f"Attempt {index + 1}: "
            f"the call itself failed: {attempt.error}"
        )

result = booth.check(
    call_llm,
    "What is the capital of France?",
    on_attempt=log_attempt,
)

For a simpler debugging path, you don't need a callback at all. After the check, inspect the last attempt:

if result.attempts and result.attempts[-1].error:
    print(result.attempts[-1].error)

That gives you the latest attempt's error message when one is available. Remember that it can describe a parsing failure as well as a call failure.

There's another limitation worth knowing: a call failure can still produce result.method == "parse_failure", the same method label that can appear when a model returns unusable output. Don't use method to distinguish provider failures from bad model output. Use the call-failure flags instead.

This came from a real integration issue

I added this distinction after a contributor's Gemini integration run exposed the problem. A retired model name returned a 404, and a rejected API key returned a 401, yet check() surfaced UNCERTAIN without making the underlying provider error obvious.

That report helped motivate call_failed. The integration example is available in Akash Goswamy's Gemini example.

Failure isn't the same as uncertainty

When a checkpoint doesn't accept an LLM result, start by identifying which layer failed.

• Call failure: inspect provider errors, credentials, network issues, and model availability.
• Output rejection: inspect parsing, ambiguity, confidence, and application validation.
• Evidence disagreement: inspect the supplied evidence and the comparison function.

BOOTH can't prevent every failure, and a successful API call doesn't prove an answer is correct. The goal is to make the checkpoint outcome easier for an application to inspect, rather than forcing every failure into one vague category.

I'm building BOOTH, a lightweight, provider-agnostic Python checkpoint layer for LLM outputs. It includes check() and acheck() for output checks, plus check_with_evidence() for comparing an answer against evidence supplied by your application.

• GitHub: https://github.com/Vedantgitbot/booth
• PyPI: https://pypi.org/project/boothpy/
• Troubleshooting guide: https://github.com/Vedantgitbot/booth/blob/main/TROUBLESHOOTING.md

When an LLM check fails in your application, do you log provider errors separately, or treat every failure as bad output?