A few months ago, BOOTH was just a small idea I wanted to explore.
I wasn't sure whether it would become a proper Python library, whether anyone would find it useful, or whether other developers would ever contribute to it.
I just had a problem I wanted to work on.
It started with a thought about something unrelated to programming: getting lost in an airport.
I wrote about that experience in my Substack post, Knowing less, but knowing what matters.
The basic idea was that having more information doesn't always mean making better decisions.
I started thinking about how this applies to AI.
An LLM can process enormous amounts of information and still produce an incorrect answer. It can return perfectly valid JSON, sound confident, and still get something important wrong.
Maybe the answer isn't always to make models know more. Maybe we also need better ways to check their outputs before applications rely on them.
That thought eventually became BOOTH.
I started building it, published it on GitHub and PyPI, and opened the repository to contributions.
What happened after that has been one of the most interesting parts of the experience.
First, what is BOOTH?
BOOTH is a lightweight, zero-dependency, provider-agnostic Python checkpoint library for LLM output reliability.
The idea is to give developers a way to check an LLM's output before their application uses it.
It currently has three main APIs:
-
booth.check()– Checks parsing, ambiguity, confidence, and optional application-defined validation. It can also retry through reconsideration. -
booth.acheck()– The asynchronous equivalent. -
booth.check_with_evidence()– Compares an existing answer against evidence supplied by the application.
The third API is something I've been particularly interested in.
Imagine you're building a RAG application. Your application already retrieves documents and gets an answer from an LLM. You now want to compare that answer against the evidence you have.
With booth.check_with_evidence(), you can provide the evidence and your own comparison function.
BOOTH doesn't retrieve the evidence or independently establish whether something is true. The application remains responsible for supplying the evidence and deciding how the comparison should work.
I like this approach because it gives developers control over how answers are checked instead of assuming that the model's own confidence is enough.
You can install BOOTH with:
pip install boothpy
It's still an evolving project, and passing a BOOTH check does not prove that an answer is factually correct. That's an important limitation, and one I want to keep being transparent about.
What people started finding interesting about BOOTH
Once the project became public, developers started exploring different parts of it.
One thing I've found encouraging is the interest in the idea of evidence-based checking.
It's easy to focus on asking an LLM to generate an answer. But in many applications, the more important question is what happens after the answer is generated.
Can the application compare it against information it already has? Can developers define their own validation rules? Can they handle uncertain outputs in a more structured way?
Those are some of the problems BOOTH is trying to make easier to work with.
I've also seen interest in its provider-agnostic approach and the possibility of using it with different models and frameworks.
Contributors have added examples involving Gemini, Anthropic, Ollama, and LangChain, among others. Other developers have explored structured-output parsing and different ways of checking model responses.
It's nice to see people take an idea that started in my head and try to make it useful in their own ways.
But something else happened too.
People didn't just try the features and agree with the original idea. They started questioning it.
And that turned out to be just as valuable.
People started challenging the assumptions behind it
One of the most interesting discussions was about confidence.
BOOTH uses a default confidence threshold of 0.7.
But there's an obvious question:
If an LLM says it's confident, should an application actually trust that confidence?
A contributor proposed investigating whether self-reported confidence was meaningfully related to answer correctness.
The experiment used 300 TriviaQA questions with local models.
The results reported by the contributor were:
| Model | Accuracy | Wrong answers that passed 0.7 |
|---|---|---|
| Qwen 2.5 7B | 51.7% | 139 of 139 |
| Llama 3.2 3B | 50.9% | 135 of 143 |
These are results from a particular experiment, not universal benchmarks or claims about all models. The reproducible write-up and raw outputs should be consulted alongside the numbers.
Still, the results raised an uncomfortable question.
For these models, the confidence threshold did very little to reject incorrect answers.
A model could be wrong and still report confidence high enough to pass.
That matters because confidence can look like a useful signal, but it isn't the same thing as correctness.
I don't see this as a reason to abandon the idea of checking confidence. I see it as a reason to understand its limitations and avoid relying on it alone.
Other contributors have also explored independent-sample agreement, tested local models, and worked on handling malformed outputs.
These experiments and improvements have helped expose cases that deserve more attention.
And that's something I genuinely want from an open-source project: people who are willing to test the assumptions, not just use the implementation.
Opening the repository changed more than the code
When I first published BOOTH, I hoped a few developers might try it, report issues, or suggest improvements.
I didn't expect so many different people to get involved.
At the time of writing, the repository has received 14 merged pull requests.
The contributions include:
- Gemini, Anthropic, and Ollama examples.
- LangChain with FAISS and FLAN-T5.
- Evidence-comparison experiments.
- A troubleshooting guide.
- Python typing support through
py.typed. - Mypy integration into CI.
- Structured-output parsing through
parse_fn. - Ollama experiments with Qwen models.
- Independent-sample agreement experiments.
- Improvements to handling malformed model outputs.
Some contributions expanded the examples. Others improved the developer experience. Some went deeper into the behaviour of the library itself.
For an established project, 14 pull requests might not sound like much.
For a project I started as a small personal experiment, it's something I'm genuinely happy about.
What makes it more meaningful is that the contributions haven't all been about adding features.
People have brought different ideas, tested different approaches, found edge cases, and asked questions I hadn't thought about.
I've also appreciated seeing developers share their experiences with BOOTH independently on LinkedIn. Some have written about their first open-source contributions, while others have discussed the integrations and experiments they worked on.
It's a nice feeling when something you built becomes interesting enough for someone else to spend their time exploring it and sharing what they learned.
I'm grateful to everyone who has contributed code, examples, experiments, feedback, and ideas.
You can see the people behind the contributions here:
Not all feedback has been praise, and that's a good thing
I want to mention this separately because I think it's important.
It's encouraging when people appreciate an idea or find a particular feature useful. The interest in evidence-based checking, provider flexibility, and structured validation has been motivating.
But I don't want BOOTH to become a project where people feel they can only say positive things about it.
If someone finds a limitation, I want to hear about it.
If someone tests a model and discovers that an assumption doesn't hold, that's useful.
If someone thinks an API could be designed better, I'd rather have that conversation than defend the original implementation just because I wrote it.
For example, the confidence experiments have made me think more carefully about what confidence scores can and cannot tell us.
The discussions around evaluation have also raised questions about how we should measure BOOTH's behaviour in the first place.
Positive feedback helps me understand which ideas resonate with developers.
Critical feedback helps me understand where those ideas fall short.
Both are important.
I don't think an open-source project becomes better simply by collecting contributions. It becomes better when people feel comfortable exploring it, questioning it, and sharing what they discover.
A new question: how should we evaluate BOOTH?
Another discussion has been around a possible API:
booth.measure()
The idea is to let developers evaluate BOOTH offline using labelled datasets.
At first, it sounds straightforward. Give it a dataset, run some checks, and produce a report.
But the details matter quite a bit.
Some of the questions raised include:
- How should correctness be determined?
- Should calibration be measured before or after retries?
- How should false acceptance be calculated?
- What happens when a model returns the same confidence score for every answer?
- Should evaluation reports expose raw outputs so results can be independently checked?
These are not just implementation details. They affect what the results actually mean.
booth.measure() is still under consideration. I don't want to introduce an evaluation API just because it sounds useful. I'd rather understand what developers need and make sure the measurements are meaningful.
This is another example of how community discussions have changed the way I think about the project.
Sometimes the most useful contribution isn't a new feature. It's a question that makes you stop and reconsider the design.
BOOTH reached beyond GitHub
There have also been a couple of milestones I didn't expect when I started.
BOOTH was featured in Python Weekly, Issue #767, and later by Python Hub.
For a small independent project, that kind of visibility is encouraging.
It means the work reached developers who might not otherwise have come across the repository.
Of course, being featured doesn't automatically mean people are using the library in production. I don't want to confuse visibility with adoption.
But it's still a milestone worth appreciating.
What matters to me now is what happens next: whether developers try BOOTH, whether they find it useful, what limitations they encounter, and whether the project can become genuinely useful in real applications.
What's next?
BOOTH is still early, and there is plenty of work ahead.
Some things I'd like to explore further include:
- Better evaluation and confidence calibration.
- More provider and framework examples.
- Stronger evidence-comparison experiments.
- Better handling of edge cases and malformed outputs.
- A clearer approach to measuring how well the checkpoint actually performs.
I'm also thinking carefully about the roadmap toward v1.0.
I don't want to rush a release just to have a bigger version number. I'd rather get the fundamentals right first: the API, behaviour, documentation, and testing.
And I'd like the community to keep influencing that direction.
If you'd like to try BOOTH
If you're working with Python, LLMs, RAG pipelines, or AI agents, I'd love for you to take a look.
You don't need to become a regular contributor. Even trying one example and telling me what you think would be useful.
You could:
- Install BOOTH and try the quickstart.
- Test it with your favourite model.
- Try the evidence-based checking API in a small RAG application.
- Find an edge case.
- Challenge one of the design decisions.
- Open an issue or submit a pull request.
And please share the failures too.
If something doesn't work, or you think an approach is wrong, I'd genuinely like to hear about it.
Project links:
- GitHub: BOOTH repository
- PyPI: boothpy
- Issues: Report a problem or suggest an idea
- Examples: BOOTH examples
- Contributors: See everyone who has contributed
One question for the community
If you were building a reliability layer for an AI application, what would you trust most?
- The model's own confidence score.
- An external validator.
- Evidence retrieved from your knowledge base.
- A combination of all three.
I'd genuinely like to hear your thoughts, especially if you've tried approaches that worked differently from what you expected.
A few months ago, BOOTH was just a small Python repository I wanted to experiment with.
Now there are developers contributing code, exploring its features, testing models, questioning assumptions, finding limitations, and sharing what they've learned.
I still have a lot to figure out. BOOTH isn't a finished solution to LLM reliability, and I don't want to pretend it is.
But one thing I've learned is that putting a project in public changes the way you build it.
You start with an idea and an implementation. Then other people bring their own use cases, questions, criticisms, and improvements.
Some of them help you see the value in what you've built. Others show you what you haven't got right yet.
I'm grateful for both. And I'm excited to see what we can make of BOOTH together.
Top comments (0)