DEV Community: Vedant Brahmbhatt

Was the model unsure, or did the API call fail?

Vedant Brahmbhatt — Thu, 22 Oct 2026 05:35:10 +0000

In an LLM application, seeing UNCERTAIN doesn't necessarily mean the model was unsure.

Sometimes the provider call failed entirely. If your application treats both cases the same way, you can end up debugging prompts and confidence thresholds when the real problem is an API key, a model name, or a network issue.

Reproduce the difference without an API key

You don't need a provider SDK or real credentials to see the distinction. This small example simulates a failed call and a model response that doesn't pass the confidence check.

Requires boothpy >= 0.5.4.

import booth

def dead(prompt):
    raise RuntimeError("401: invalid API key")

def unsure(prompt):
    return '{"answer": "Paris", "confidence": 0.2}'

a = booth.check(dead, "What is the capital of France?")
b = booth.check(unsure, "What is the capital of France?")

print(a.status, a.answer, a.call_failed, len(a.attempts))
# UNCERTAIN None True 2

print(b.status, b.answer, b.call_failed)
# UNCERTAIN Paris False

Both checks return UNCERTAIN, but for different reasons.

In the first case, the call fails. With the default max_retries=1, BOOTH makes two attempts. Neither reaches a usable response, so the result has no answer and call_failed is True.

In the second case, the call returns a response, but its confidence is too low for the checkpoint to accept it. The answer is available, and call_failed is False.

That distinction matters: retrying a request with an invalid API key costs another call without fixing the credential.

What `call_failed` actually tells you

BOOTH added result.call_failed in v0.5.4. It's also present in v0.5.6.

One detail matters: result.call_failed is True only when every attempt was a call failure.

Imagine the first attempt returns an uncertain response, and the retry then hits an unavailable provider. The overall result's call_failed can still be False, because not every attempt failed at the call level. To inspect the last attempt specifically, check:

last_attempt = result.attempts[-1]

if last_attempt.call_failed:
    print("The latest call failed:", last_attempt.error)

Log call failures, not every attempt error

BOOTH also provides an on_attempt callback. Be careful about which field you check: attempt.error can be populated for parsing failures too, such as a response where no valid JSON object was found. An error value alone doesn't prove that the provider call failed.

Check attempt.call_failed instead:

def log_attempt(index, attempt):
    if attempt.call_failed:
        print(
            f"Attempt {index + 1}: "
            f"the call itself failed: {attempt.error}"
        )

result = booth.check(
    call_llm,
    "What is the capital of France?",
    on_attempt=log_attempt,
)

For a simpler debugging path, you don't need a callback at all. After the check, inspect the last attempt:

if result.attempts and result.attempts[-1].error:
    print(result.attempts[-1].error)

That gives you the latest attempt's error message when one is available. Remember that it can describe a parsing failure as well as a call failure.

There's another limitation worth knowing: a call failure can still produce result.method == "parse_failure", the same method label that can appear when a model returns unusable output. Don't use method to distinguish provider failures from bad model output. Use the call-failure flags instead.

This came from a real integration issue

I added this distinction after a contributor's Gemini integration run exposed the problem. A retired model name returned a 404, and a rejected API key returned a 401, yet check() surfaced UNCERTAIN without making the underlying provider error obvious.

That report helped motivate call_failed. The integration example is available in Akash Goswamy's Gemini example.

Failure isn't the same as uncertainty

When a checkpoint doesn't accept an LLM result, start by identifying which layer failed.

Call failure: inspect provider errors, credentials, network issues, and model availability.
Output rejection: inspect parsing, ambiguity, confidence, and application validation.
Evidence disagreement: inspect the supplied evidence and the comparison function.

BOOTH can't prevent every failure, and a successful API call doesn't prove an answer is correct. The goal is to make the checkpoint outcome easier for an application to inspect, rather than forcing every failure into one vague category.

I'm building BOOTH, a lightweight, provider-agnostic Python checkpoint layer for LLM outputs. It includes check() and acheck() for output checks, plus check_with_evidence() for comparing an answer against evidence supplied by your application.

GitHub: https://github.com/Vedantgitbot/booth
PyPI: https://pypi.org/project/boothpy/
Troubleshooting guide: https://github.com/Vedantgitbot/booth/blob/main/TROUBLESHOOTING.md

When an LLM check fails in your application, do you log provider errors separately, or treat every failure as bad output?

BOOTH Was Featured in Python Weekly: A Small Milestone for My Open-Source Project

Vedant Brahmbhatt — Fri, 16 Oct 2026 19:19:13 +0000

A small milestone for a project I've been building: BOOTH was featured in Python Weekly!

It appears in Python Weekly Issue #767 (October 15, 2026) under Interesting Projects, Tools, and Libraries, described as a lightweight, zero-dependency Python checkpoint layer for structured LLM output reliability.

I started BOOTH only a couple of months ago, and I'm still learning what it takes to turn a small library into something other developers find useful. Seeing it in a curated newsletter is encouraging.

Why BOOTH exists

A few months ago, a person working an airport gate helped me find mine after I'd gotten lost, boarding pass in hand, signs everywhere, still at the wrong gate. She didn't know where I'd flown in from, how the aircraft was built, or anything about my itinerary beyond that one boarding pass. She didn't need to. She knew the airport, and she knew what to check.

That's the idea BOOTH is built around. The industry's default answer to "the model got it wrong" has mostly been to make the model know more: more context, more compute, bigger models, more tools bolted around it. And the problem still happens, because it isn't always a knowledge problem. Sometimes what's missing isn't more information going in. It's something whose only job is checking whether what came out actually meets the bar.

BOOTH doesn't try to know more than your model does. It just knows what to check.

So the question I kept coming back to was:

Before passing an LLM response to the rest of the application, should we check it first?

A response can sound confident and still be ambiguous, fail validation, or disagree with the evidence your application retrieved. That's the problem BOOTH explores.

I wrote the longer version of this idea on Substack: Knowing less, but knowing what matters.

What BOOTH does

BOOTH is a lightweight, provider-agnostic library that sits between an LLM call and your application logic. Instead of raw text, you get a structured result (ACCEPTED, REPAIRED, AMBIGUOUS, UNCERTAIN or BLOCKED) so your app can decide what to do next.

It has three main entry points:

check(): a synchronous checkpoint for ambiguity and confidence, with reconsideration retries.
acheck(): the async counterpart.
check_with_evidence(): checks an answer against evidence your application supplies, such as documents from a RAG pipeline.

import booth

result = booth.check(call_llm, "What is the capital of France?")

if result.ok:
    print(result.answer)
else:
    print(f"BOOTH returned {result.status}")  # AMBIGUOUS / UNCERTAIN

For evidence grounding, imagine your retrieval pipeline returns a policy saying refunds are available within 45 days, but the generated answer says 90 days:

result = booth.check_with_evidence(
    answer=llm_answer,
    evidence=retrieved_docs,
    compare_fn=your_comparison_function,
)

if result.status == booth.BLOCKED:
    print(result.detail)  # the answer disagrees with the retrieved evidence

You provide the comparison function, and BOOTH makes the checking step explicit.

To be clear about the limits: BOOTH doesn't replace your retrieval pipeline, and agreeing with the supplied evidence doesn't prove the evidence is correct or complete.

Why keep it lightweight?

BOOTH is small on purpose. It has zero runtime dependencies, works with any provider through functions you supply, and returns structured results instead of making decisions for you.

I don't want it to become another framework that forces you to rebuild your application around it. You should be able to try it in an existing Python project without changing your stack.

pip install boothpy

What's next

Testing across providers and models has been the most useful part of building BOOTH. Real integrations expose edge cases that are easy to miss in your own setup. The repo now includes runnable examples with real console output for several providers and local models, including the negative results.

I'm continuing to work on error handling, documentation, integrations, and the behavior of the checkpoint APIs. I'd also welcome contributors who want to test BOOTH, find edge cases, or pick up open issues.

The Python Weekly feature is a nice milestone, but what matters most to me is whether developers find BOOTH useful enough to try in their own applications. If you work with Python, LLMs or RAG, I'd genuinely appreciate your feedback.

Links

Thanks to everyone who has explored the project, reported issues, contributed integrations, or shared feedback. I'm excited to keep building BOOTH in the open.

I Open-Sourced My Project for Hacktoberfest. Here's What Happened When Strangers Started Contributing.

Vedant Brahmbhatt — Wed, 07 Oct 2026 17:16:03 +0000

I thought open-sourcing my project would get me stars.

Instead, strangers broke it.

And honestly, that was much more valuable.

Two months ago, I started building BOOTH, a small checkpoint layer that sits between an LLM call and your application and decides whether the response is trustworthy enough to use.

It's intentionally small. Zero dependencies. Just mine.

For Hacktoberfest, I opened one issue:

Test BOOTH against your favorite LLM provider. Document what breaks.

I expected silence.

Instead, people started finding things I hadn't seen.

A 7B model reported confidence: 1.0 on wrong answers

One contributor tested BOOTH with Ollama and a local 7B model.

The surprising part wasn't that the model got things wrong.

It was that the model reported confidence: 1.0 on every wrong answer.

That's not necessarily a bug in the model. It's a limitation of treating model-generated confidence as if it were an objective measurement.

I hadn't documented that behavior.

Now it's part of the project's reality.

Then came the API problems

Another contributor hit a 401 that only appeared with a newer API key format.

Someone else hit a 404 because the model they were testing had quietly been retired.

Neither issue appeared in my local testing.

That's the thing about testing your own project:

You test the environment you have.

Other people test the environments you don't.

Then a first-time contributor found an actual bug

This one mattered more.

When an underlying API call failed, BOOTH was swallowing the exception and returning a result that looked almost identical to the model simply being uncertain.

Those are completely different situations.

One means:

"The model answered, but we shouldn't trust the answer."

The other means:

"The model/API didn't actually give us a usable answer."

BOOTH was making those two states look the same.

A first-time open-source contributor found it.

I shipped the fix within a day.

That bug probably would have survived much longer without someone approaching the project with completely fresh eyes.

Then someone turned the chaos into documentation

After several people had tested different providers, another contributor volunteered to turn the accumulated discoveries into a TROUBLESHOOTING.md.

Not theoretical documentation.

Real-world:

symptom → cause → fix

The kind of documentation you only write after somebody actually hits the wall.

And it's better documentation because of that.

Four people. Four providers. A much better project.

So far, four independent contributors have tested BOOTH against different providers and environments.

OpenAI.

Groq.

Ollama.

Gemini.

Anthropic.

They didn't owe me stars.

They didn't have to tell me the project was great.

They just ran it, broke it, and told me exactly what happened.

That's probably the most valuable thing that has happened to the project so far.

The part nobody tells you about open source

When you open-source a project, it's tempting to measure everything by:

GitHub stars
forks
contributors
downloads
social media likes

Those numbers are nice.

But they're not necessarily the most valuable outcome.

The real value can be much quieter.

Someone you've never met runs your software in an environment you've never used.

They find the edge case you never considered.

They report it.

You fix it.

Now your project is more honest than it was yesterday.

That's what happened with BOOTH.

It's still small.

But it's been tested against multiple LLM providers by people who had no reason to tell me what I wanted to hear.

They found limitations.

They found broken assumptions.

They found bugs.

And the project is better because of it.

If you're building something and wondering whether it's "ready" to open source, I'd argue that you don't necessarily need to wait until it's perfect.

You might learn more by letting someone else break it.

Your first contributors probably won't make your project popular.

They'll make it honest.

Want to try BOOTH?

If you're curious, you can:

📦 Install BOOTH from PyPI: https://pypi.org/project/boothpy/
🐛 Pick up the Hacktoberfest issue: https://github.com/Vedantgitbot/booth/issues/2
⭐ Explore the source code: https://github.com/Vedantgitbot/booth

If you try it with an LLM provider I haven't tested yet, I'd genuinely love to hear what happens.

Break it. Report it. Make it better.

Hacktoberfest update: BOOTH is getting real contributions

Vedant Brahmbhatt — Mon, 05 Oct 2026 17:05:12 +0000

A few days ago, I opened BOOTH for Hacktoberfest contributors.

BOOTH is a lightweight, provider-agnostic Python library that adds a checkpoint before an LLM output reaches your application.

check() / acheck() handle ambiguity and confidence checks, while check_with_evidence() lets you compare an answer against evidence retrieved by your RAG pipeline.

Since Hacktoberfest started, developers have begun contributing integrations across different LLM providers and models.

More contributions and feedback are always welcome. 🙌

🛠️ Open issues: https://github.com/Vedantgitbot/booth/issues
🔗 Repo: https://github.com/Vedantgitbot/booth

Really happy to see BOOTH growing beyond just me.

Hacktoberfest 2026 is here, and BOOTH is open to contributors!

Vedant Brahmbhatt — Thu, 01 Oct 2026 14:15:49 +0000

I’ve been building BOOTH, a lightweight checkpoint layer for LLM outputs.

The idea is simple: don’t automatically trust every LLM response. Check it first.

With check_with_evidence(), BOOTH lets you compare an LLM’s answer against evidence your RAG pipeline has already retrieved.

BOOTH v0.5.2

Zero runtime dependencies
Provider-agnostic
Sync + async APIs
Structured results
300+ tests
MIT licensed

I’m looking for developers interested in LLM reliability, RAG, and Python to contribute this Hacktoberfest.

🔗 Explore BOOTH: https://github.com/Vedantgitbot/booth
🛠️ Find an issue: https://github.com/Vedantgitbot/booth/issues

Whether it’s your first open-source contribution or your hundredth, you’re welcome!

Your LLM gave you an answer. Should your application trust it?

Vedant Brahmbhatt — Wed, 30 Sep 2026 15:21:13 +0000

I built BOOTH, a small checkpoint layer that sits between your app and an
LLM call and returns a structured decision instead of just fluent text.

Example:

Evidence: "Returns are allowed within 45 days."
LLM answer: "Returns are allowed within 90 days."

Confident, fluent, and unsupported. Self-reported confidence won't catch it.
Checking against the evidence will.

BOOTH doesn't decide what's true. You supply the comparison, and BOOTH turns
the outcome into an explicit status you can't accidentally ignore:

result = booth.check_with_evidence(
    answer=llm_answer,
    evidence=retrieved_docs,
    compare_fn=your_comparison_function,  # your logic, your definition of "agrees"
)

if result.status == booth.BLOCKED:
    print(result.detail)  # why it was blocked

Results also carry a reason and a checker_failed flag, so "the answer disagreed"
and "my comparison function crashed" are never mixed up.

There's also check() / acheck() for plain LLM calls: ambiguity detection plus
confidence checks with reconsideration retries.

It keeps your RAG pipeline and provider as they are. Zero runtime dependencies,
provider-agnostic, MIT, Python 3.9+.

pip install boothpy

GitHub: https://github.com/Vedantgitbot/booth

How are you currently deciding whether an LLM output is safe to pass downstream?

Beta · 300+ tests · CI passing · MIT · Python 3.9+