<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vedant Brahmbhatt</title>
    <description>The latest articles on DEV Community by Vedant Brahmbhatt (@vedant_brahmbhatt_7be1822).</description>
    <link>https://dev.to/vedant_brahmbhatt_7be1822</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4143765%2F0e98e95d-465d-45a3-a178-1548ad72bdb6.png</url>
      <title>DEV Community: Vedant Brahmbhatt</title>
      <link>https://dev.to/vedant_brahmbhatt_7be1822</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vedant_brahmbhatt_7be1822"/>
    <language>en</language>
    <item>
      <title>What Happened When I Built an Open-Source AI Project in Public</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Mon, 02 Nov 2026 15:07:27 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/what-happened-when-i-built-an-open-source-ai-project-in-public-2g2o</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/what-happened-when-i-built-an-open-source-ai-project-in-public-2g2o</guid>
      <description>&lt;p&gt;A few months ago, BOOTH was just a small idea I wanted to explore.&lt;/p&gt;

&lt;p&gt;I wasn't sure whether it would become a proper Python library, whether anyone would find it useful, or whether other developers would ever contribute to it.&lt;/p&gt;

&lt;p&gt;I just had a problem I wanted to work on.&lt;/p&gt;

&lt;p&gt;It started with a thought about something unrelated to programming: getting lost in an airport.&lt;/p&gt;

&lt;p&gt;I wrote about that experience in my Substack post, &lt;a href="https://vedantbrahmbhatt.substack.com/p/knowing-less-but-knowing-what-matters" rel="noopener noreferrer"&gt;Knowing less, but knowing what matters&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The basic idea was that having more information doesn't always mean making better decisions.&lt;/p&gt;

&lt;p&gt;I started thinking about how this applies to AI.&lt;/p&gt;

&lt;p&gt;An LLM can process enormous amounts of information and still produce an incorrect answer. It can return perfectly valid JSON, sound confident, and still get something important wrong.&lt;/p&gt;

&lt;p&gt;Maybe the answer isn't always to make models know more. Maybe we also need better ways to check their outputs before applications rely on them.&lt;/p&gt;

&lt;p&gt;That thought eventually became BOOTH.&lt;/p&gt;

&lt;p&gt;I started building it, published it on GitHub and PyPI, and opened the repository to contributions.&lt;/p&gt;

&lt;p&gt;What happened after that has been one of the most interesting parts of the experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, what is BOOTH?
&lt;/h2&gt;

&lt;p&gt;BOOTH is a lightweight, zero-dependency, provider-agnostic Python checkpoint library for LLM output reliability.&lt;/p&gt;

&lt;p&gt;The idea is to give developers a way to check an LLM's output before their application uses it.&lt;/p&gt;

&lt;p&gt;It currently has three main APIs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;booth.check()&lt;/code&gt; – Checks parsing, ambiguity, confidence, and optional application-defined validation. It can also retry through reconsideration.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;booth.acheck()&lt;/code&gt; – The asynchronous equivalent.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;booth.check_with_evidence()&lt;/code&gt; – Compares an existing answer against evidence supplied by the application.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The third API is something I've been particularly interested in.&lt;/p&gt;

&lt;p&gt;Imagine you're building a RAG application. Your application already retrieves documents and gets an answer from an LLM. You now want to compare that answer against the evidence you have.&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;booth.check_with_evidence()&lt;/code&gt;, you can provide the evidence and your own comparison function.&lt;/p&gt;

&lt;p&gt;BOOTH doesn't retrieve the evidence or independently establish whether something is true. The application remains responsible for supplying the evidence and deciding how the comparison should work.&lt;/p&gt;

&lt;p&gt;I like this approach because it gives developers control over how answers are checked instead of assuming that the model's own confidence is enough.&lt;/p&gt;

&lt;p&gt;You can install BOOTH with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;boothpy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's still an evolving project, and passing a BOOTH check does not prove that an answer is factually correct. That's an important limitation, and one I want to keep being transparent about.&lt;/p&gt;

&lt;h2&gt;
  
  
  What people started finding interesting about BOOTH
&lt;/h2&gt;

&lt;p&gt;Once the project became public, developers started exploring different parts of it.&lt;/p&gt;

&lt;p&gt;One thing I've found encouraging is the interest in the idea of evidence-based checking.&lt;/p&gt;

&lt;p&gt;It's easy to focus on asking an LLM to generate an answer. But in many applications, the more important question is what happens after the answer is generated.&lt;/p&gt;

&lt;p&gt;Can the application compare it against information it already has? Can developers define their own validation rules? Can they handle uncertain outputs in a more structured way?&lt;/p&gt;

&lt;p&gt;Those are some of the problems BOOTH is trying to make easier to work with.&lt;/p&gt;

&lt;p&gt;I've also seen interest in its provider-agnostic approach and the possibility of using it with different models and frameworks.&lt;/p&gt;

&lt;p&gt;Contributors have added examples involving Gemini, Anthropic, Ollama, and LangChain, among others. Other developers have explored structured-output parsing and different ways of checking model responses.&lt;/p&gt;

&lt;p&gt;It's nice to see people take an idea that started in my head and try to make it useful in their own ways.&lt;/p&gt;

&lt;p&gt;But something else happened too.&lt;/p&gt;

&lt;p&gt;People didn't just try the features and agree with the original idea. They started questioning it.&lt;/p&gt;

&lt;p&gt;And that turned out to be just as valuable.&lt;/p&gt;

&lt;h2&gt;
  
  
  People started challenging the assumptions behind it
&lt;/h2&gt;

&lt;p&gt;One of the most interesting discussions was about confidence.&lt;/p&gt;

&lt;p&gt;BOOTH uses a default confidence threshold of 0.7.&lt;/p&gt;

&lt;p&gt;But there's an obvious question:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If an LLM says it's confident, should an application actually trust that confidence?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A contributor proposed investigating whether self-reported confidence was meaningfully related to answer correctness.&lt;/p&gt;

&lt;p&gt;The experiment used 300 TriviaQA questions with local models.&lt;/p&gt;

&lt;p&gt;The results reported by the contributor were:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Accuracy&lt;/th&gt;
&lt;th&gt;Wrong answers that passed 0.7&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen 2.5 7B&lt;/td&gt;
&lt;td&gt;51.7%&lt;/td&gt;
&lt;td&gt;139 of 139&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.2 3B&lt;/td&gt;
&lt;td&gt;50.9%&lt;/td&gt;
&lt;td&gt;135 of 143&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are results from a particular experiment, not universal benchmarks or claims about all models. The reproducible write-up and raw outputs should be consulted alongside the numbers.&lt;/p&gt;

&lt;p&gt;Still, the results raised an uncomfortable question.&lt;/p&gt;

&lt;p&gt;For these models, the confidence threshold did very little to reject incorrect answers.&lt;/p&gt;

&lt;p&gt;A model could be wrong and still report confidence high enough to pass.&lt;/p&gt;

&lt;p&gt;That matters because confidence can look like a useful signal, but it isn't the same thing as correctness.&lt;/p&gt;

&lt;p&gt;I don't see this as a reason to abandon the idea of checking confidence. I see it as a reason to understand its limitations and avoid relying on it alone.&lt;/p&gt;

&lt;p&gt;Other contributors have also explored independent-sample agreement, tested local models, and worked on handling malformed outputs.&lt;/p&gt;

&lt;p&gt;These experiments and improvements have helped expose cases that deserve more attention.&lt;/p&gt;

&lt;p&gt;And that's something I genuinely want from an open-source project: people who are willing to test the assumptions, not just use the implementation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Opening the repository changed more than the code
&lt;/h2&gt;

&lt;p&gt;When I first published BOOTH, I hoped a few developers might try it, report issues, or suggest improvements.&lt;/p&gt;

&lt;p&gt;I didn't expect so many different people to get involved.&lt;/p&gt;

&lt;p&gt;At the time of writing, the repository has received 14 merged pull requests.&lt;/p&gt;

&lt;p&gt;The contributions include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gemini, Anthropic, and Ollama examples.&lt;/li&gt;
&lt;li&gt;LangChain with FAISS and FLAN-T5.&lt;/li&gt;
&lt;li&gt;Evidence-comparison experiments.&lt;/li&gt;
&lt;li&gt;A troubleshooting guide.&lt;/li&gt;
&lt;li&gt;Python typing support through &lt;code&gt;py.typed&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Mypy integration into CI.&lt;/li&gt;
&lt;li&gt;Structured-output parsing through &lt;code&gt;parse_fn&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Ollama experiments with Qwen models.&lt;/li&gt;
&lt;li&gt;Independent-sample agreement experiments.&lt;/li&gt;
&lt;li&gt;Improvements to handling malformed model outputs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some contributions expanded the examples. Others improved the developer experience. Some went deeper into the behaviour of the library itself.&lt;/p&gt;

&lt;p&gt;For an established project, 14 pull requests might not sound like much.&lt;/p&gt;

&lt;p&gt;For a project I started as a small personal experiment, it's something I'm genuinely happy about.&lt;/p&gt;

&lt;p&gt;What makes it more meaningful is that the contributions haven't all been about adding features.&lt;/p&gt;

&lt;p&gt;People have brought different ideas, tested different approaches, found edge cases, and asked questions I hadn't thought about.&lt;/p&gt;

&lt;p&gt;I've also appreciated seeing developers share their experiences with BOOTH independently on LinkedIn. Some have written about their first open-source contributions, while others have discussed the integrations and experiments they worked on.&lt;/p&gt;

&lt;p&gt;It's a nice feeling when something you built becomes interesting enough for someone else to spend their time exploring it and sharing what they learned.&lt;/p&gt;

&lt;p&gt;I'm grateful to everyone who has contributed code, examples, experiments, feedback, and ideas.&lt;/p&gt;

&lt;p&gt;You can see the people behind the contributions here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Vedantgitbot/booth/graphs/contributors" rel="noopener noreferrer"&gt;BOOTH – GitHub Contributors&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Not all feedback has been praise, and that's a good thing
&lt;/h2&gt;

&lt;p&gt;I want to mention this separately because I think it's important.&lt;/p&gt;

&lt;p&gt;It's encouraging when people appreciate an idea or find a particular feature useful. The interest in evidence-based checking, provider flexibility, and structured validation has been motivating.&lt;/p&gt;

&lt;p&gt;But I don't want BOOTH to become a project where people feel they can only say positive things about it.&lt;/p&gt;

&lt;p&gt;If someone finds a limitation, I want to hear about it.&lt;/p&gt;

&lt;p&gt;If someone tests a model and discovers that an assumption doesn't hold, that's useful.&lt;/p&gt;

&lt;p&gt;If someone thinks an API could be designed better, I'd rather have that conversation than defend the original implementation just because I wrote it.&lt;/p&gt;

&lt;p&gt;For example, the confidence experiments have made me think more carefully about what confidence scores can and cannot tell us.&lt;/p&gt;

&lt;p&gt;The discussions around evaluation have also raised questions about how we should measure BOOTH's behaviour in the first place.&lt;/p&gt;

&lt;p&gt;Positive feedback helps me understand which ideas resonate with developers.&lt;/p&gt;

&lt;p&gt;Critical feedback helps me understand where those ideas fall short.&lt;/p&gt;

&lt;p&gt;Both are important.&lt;/p&gt;

&lt;p&gt;I don't think an open-source project becomes better simply by collecting contributions. It becomes better when people feel comfortable exploring it, questioning it, and sharing what they discover.&lt;/p&gt;

&lt;h2&gt;
  
  
  A new question: how should we evaluate BOOTH?
&lt;/h2&gt;

&lt;p&gt;Another discussion has been around a possible API:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;booth.measure()&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The idea is to let developers evaluate BOOTH offline using labelled datasets.&lt;/p&gt;

&lt;p&gt;At first, it sounds straightforward. Give it a dataset, run some checks, and produce a report.&lt;/p&gt;

&lt;p&gt;But the details matter quite a bit.&lt;/p&gt;

&lt;p&gt;Some of the questions raised include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How should correctness be determined?&lt;/li&gt;
&lt;li&gt;Should calibration be measured before or after retries?&lt;/li&gt;
&lt;li&gt;How should false acceptance be calculated?&lt;/li&gt;
&lt;li&gt;What happens when a model returns the same confidence score for every answer?&lt;/li&gt;
&lt;li&gt;Should evaluation reports expose raw outputs so results can be independently checked?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are not just implementation details. They affect what the results actually mean.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;booth.measure()&lt;/code&gt; is still under consideration. I don't want to introduce an evaluation API just because it sounds useful. I'd rather understand what developers need and make sure the measurements are meaningful.&lt;/p&gt;

&lt;p&gt;This is another example of how community discussions have changed the way I think about the project.&lt;/p&gt;

&lt;p&gt;Sometimes the most useful contribution isn't a new feature. It's a question that makes you stop and reconsider the design.&lt;/p&gt;

&lt;h2&gt;
  
  
  BOOTH reached beyond GitHub
&lt;/h2&gt;

&lt;p&gt;There have also been a couple of milestones I didn't expect when I started.&lt;/p&gt;

&lt;p&gt;BOOTH was featured in &lt;a href="https://www.pythonweekly.com/p/python-weekly-issue-767-october-15-2026" rel="noopener noreferrer"&gt;Python Weekly, Issue #767&lt;/a&gt;, and later by &lt;a href="https://x.com/PythonHub/status/2114752745504686172" rel="noopener noreferrer"&gt;Python Hub&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For a small independent project, that kind of visibility is encouraging.&lt;/p&gt;

&lt;p&gt;It means the work reached developers who might not otherwise have come across the repository.&lt;/p&gt;

&lt;p&gt;Of course, being featured doesn't automatically mean people are using the library in production. I don't want to confuse visibility with adoption.&lt;/p&gt;

&lt;p&gt;But it's still a milestone worth appreciating.&lt;/p&gt;

&lt;p&gt;What matters to me now is what happens next: whether developers try BOOTH, whether they find it useful, what limitations they encounter, and whether the project can become genuinely useful in real applications.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next?
&lt;/h2&gt;

&lt;p&gt;BOOTH is still early, and there is plenty of work ahead.&lt;/p&gt;

&lt;p&gt;Some things I'd like to explore further include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Better evaluation and confidence calibration.&lt;/li&gt;
&lt;li&gt;More provider and framework examples.&lt;/li&gt;
&lt;li&gt;Stronger evidence-comparison experiments.&lt;/li&gt;
&lt;li&gt;Better handling of edge cases and malformed outputs.&lt;/li&gt;
&lt;li&gt;A clearer approach to measuring how well the checkpoint actually performs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'm also thinking carefully about the roadmap toward v1.0.&lt;/p&gt;

&lt;p&gt;I don't want to rush a release just to have a bigger version number. I'd rather get the fundamentals right first: the API, behaviour, documentation, and testing.&lt;/p&gt;

&lt;p&gt;And I'd like the community to keep influencing that direction.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you'd like to try BOOTH
&lt;/h2&gt;

&lt;p&gt;If you're working with Python, LLMs, RAG pipelines, or AI agents, I'd love for you to take a look.&lt;/p&gt;

&lt;p&gt;You don't need to become a regular contributor. Even trying one example and telling me what you think would be useful.&lt;/p&gt;

&lt;p&gt;You could:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Install BOOTH and try the quickstart.&lt;/li&gt;
&lt;li&gt;Test it with your favourite model.&lt;/li&gt;
&lt;li&gt;Try the evidence-based checking API in a small RAG application.&lt;/li&gt;
&lt;li&gt;Find an edge case.&lt;/li&gt;
&lt;li&gt;Challenge one of the design decisions.&lt;/li&gt;
&lt;li&gt;Open an issue or submit a pull request.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And please share the failures too.&lt;/p&gt;

&lt;p&gt;If something doesn't work, or you think an approach is wrong, I'd genuinely like to hear about it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Project links:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GitHub: &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;BOOTH repository&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PyPI: &lt;a href="https://pypi.org/project/boothpy/" rel="noopener noreferrer"&gt;boothpy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Issues: &lt;a href="https://github.com/Vedantgitbot/booth/issues" rel="noopener noreferrer"&gt;Report a problem or suggest an idea&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Examples: &lt;a href="https://github.com/Vedantgitbot/booth/tree/main/examples" rel="noopener noreferrer"&gt;BOOTH examples&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Contributors: &lt;a href="https://github.com/Vedantgitbot/booth/graphs/contributors" rel="noopener noreferrer"&gt;See everyone who has contributed&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  One question for the community
&lt;/h2&gt;

&lt;p&gt;If you were building a reliability layer for an AI application, what would you trust most?&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The model's own confidence score.&lt;/li&gt;
&lt;li&gt;An external validator.&lt;/li&gt;
&lt;li&gt;Evidence retrieved from your knowledge base.&lt;/li&gt;
&lt;li&gt;A combination of all three.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I'd genuinely like to hear your thoughts, especially if you've tried approaches that worked differently from what you expected.&lt;/p&gt;

&lt;p&gt;A few months ago, BOOTH was just a small Python repository I wanted to experiment with.&lt;/p&gt;

&lt;p&gt;Now there are developers contributing code, exploring its features, testing models, questioning assumptions, finding limitations, and sharing what they've learned.&lt;/p&gt;

&lt;p&gt;I still have a lot to figure out. BOOTH isn't a finished solution to LLM reliability, and I don't want to pretend it is.&lt;/p&gt;

&lt;p&gt;But one thing I've learned is that putting a project in public changes the way you build it.&lt;/p&gt;

&lt;p&gt;You start with an idea and an implementation. Then other people bring their own use cases, questions, criticisms, and improvements.&lt;/p&gt;

&lt;p&gt;Some of them help you see the value in what you've built. Others show you what you haven't got right yet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I'm grateful for both. And I'm excited to see what we can make of BOOTH together.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>opensource</category>
      <category>rag</category>
    </item>
    <item>
      <title>Was the model unsure, or did the API call fail?</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Thu, 22 Oct 2026 05:35:10 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/was-the-model-unsure-or-did-the-api-call-fail-5bkj</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/was-the-model-unsure-or-did-the-api-call-fail-5bkj</guid>
      <description>&lt;p&gt;In an LLM application, seeing &lt;code&gt;UNCERTAIN&lt;/code&gt; doesn't necessarily mean the model was unsure.&lt;/p&gt;

&lt;p&gt;Sometimes the provider call failed entirely. If your application treats both cases the same way, you can end up debugging prompts and confidence thresholds when the real problem is an API key, a model name, or a network issue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce the difference without an API key
&lt;/h2&gt;

&lt;p&gt;You don't need a provider SDK or real credentials to see the distinction. This small example simulates a failed call and a model response that doesn't pass the confidence check.&lt;/p&gt;

&lt;p&gt;Requires &lt;code&gt;boothpy &amp;gt;= 0.5.4&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;401: invalid API key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;unsure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Paris&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confidence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;: 0.2}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;

&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is the capital of France?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unsure&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is the capital of France?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_failed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="c1"&gt;# UNCERTAIN None True 2
&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_failed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# UNCERTAIN Paris False
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both checks return &lt;code&gt;UNCERTAIN&lt;/code&gt;, but for different reasons.&lt;/p&gt;

&lt;p&gt;In the first case, the call fails. With the default &lt;code&gt;max_retries=1&lt;/code&gt;, BOOTH makes two attempts. Neither reaches a usable response, so the result has no answer and &lt;code&gt;call_failed&lt;/code&gt; is &lt;code&gt;True&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In the second case, the call returns a response, but its confidence is too low for the checkpoint to accept it. The answer is available, and &lt;code&gt;call_failed&lt;/code&gt; is &lt;code&gt;False&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That distinction matters: retrying a request with an invalid API key costs another call without fixing the credential.&lt;/p&gt;

&lt;h2&gt;
  
  
  What &lt;code&gt;call_failed&lt;/code&gt; actually tells you
&lt;/h2&gt;

&lt;p&gt;BOOTH added &lt;code&gt;result.call_failed&lt;/code&gt; in v0.5.4. It's also present in v0.5.6.&lt;/p&gt;

&lt;p&gt;One detail matters: &lt;strong&gt;&lt;code&gt;result.call_failed&lt;/code&gt; is &lt;code&gt;True&lt;/code&gt; only when every attempt was a call failure.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine the first attempt returns an uncertain response, and the retry then hits an unavailable provider. The overall result's &lt;code&gt;call_failed&lt;/code&gt; can still be &lt;code&gt;False&lt;/code&gt;, because not every attempt failed at the call level. To inspect the last attempt specifically, check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;last_attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;last_attempt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_failed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;The latest call failed:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;last_attempt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Log call failures, not every attempt error
&lt;/h2&gt;

&lt;p&gt;BOOTH also provides an &lt;code&gt;on_attempt&lt;/code&gt; callback. Be careful about which field you check: &lt;code&gt;attempt.error&lt;/code&gt; can be populated for parsing failures too, such as a response where no valid JSON object was found. An error value alone doesn't prove that the provider call failed.&lt;/p&gt;

&lt;p&gt;Check &lt;code&gt;attempt.call_failed&lt;/code&gt; instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;log_attempt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_failed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Attempt &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;the call itself failed: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;call_llm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is the capital of France?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;on_attempt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;log_attempt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a simpler debugging path, you don't need a callback at all. After the check, inspect the last attempt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That gives you the latest attempt's error message when one is available. Remember that it can describe a parsing failure as well as a call failure.&lt;/p&gt;

&lt;p&gt;There's another limitation worth knowing: a call failure can still produce &lt;code&gt;result.method == "parse_failure"&lt;/code&gt;, the same method label that can appear when a model returns unusable output. &lt;strong&gt;Don't use &lt;code&gt;method&lt;/code&gt; to distinguish provider failures from bad model output.&lt;/strong&gt; Use the call-failure flags instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  This came from a real integration issue
&lt;/h2&gt;

&lt;p&gt;I added this distinction after a contributor's Gemini integration run exposed the problem. A retired model name returned a 404, and a rejected API key returned a 401, yet &lt;code&gt;check()&lt;/code&gt; surfaced &lt;code&gt;UNCERTAIN&lt;/code&gt; without making the underlying provider error obvious.&lt;/p&gt;

&lt;p&gt;That report helped motivate &lt;code&gt;call_failed&lt;/code&gt;. The integration example is available in &lt;a href="https://github.com/Vedantgitbot/booth/tree/main/examples/AkashGoswamy_gemini" rel="noopener noreferrer"&gt;Akash Goswamy's Gemini example&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure isn't the same as uncertainty
&lt;/h2&gt;

&lt;p&gt;When a checkpoint doesn't accept an LLM result, start by identifying which layer failed.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Call failure:&lt;/strong&gt; inspect provider errors, credentials, network issues, and model availability.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Output rejection:&lt;/strong&gt; inspect parsing, ambiguity, confidence, and application validation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evidence disagreement:&lt;/strong&gt; inspect the supplied evidence and the comparison function.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;BOOTH can't prevent every failure, and a successful API call doesn't prove an answer is correct. The goal is to make the checkpoint outcome easier for an application to inspect, rather than forcing every failure into one vague category.&lt;/p&gt;

&lt;p&gt;I'm building &lt;strong&gt;BOOTH&lt;/strong&gt;, a lightweight, provider-agnostic Python checkpoint layer for LLM outputs. It includes &lt;code&gt;check()&lt;/code&gt; and &lt;code&gt;acheck()&lt;/code&gt; for output checks, plus &lt;code&gt;check_with_evidence()&lt;/code&gt; for comparing an answer against evidence supplied by your application.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GitHub:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PyPI:&lt;/strong&gt; &lt;a href="https://pypi.org/project/boothpy/" rel="noopener noreferrer"&gt;https://pypi.org/project/boothpy/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Troubleshooting guide:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth/blob/main/TROUBLESHOOTING.md" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth/blob/main/TROUBLESHOOTING.md&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When an LLM check fails in your application, do you log provider errors separately, or treat every failure as bad output?&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>llm</category>
      <category>debugging</category>
    </item>
    <item>
      <title>BOOTH Was Featured in Python Weekly: A Small Milestone for My Open-Source Project</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Fri, 16 Oct 2026 19:19:13 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/booth-was-featured-in-python-weekly-a-small-milestone-for-my-open-source-project-58cj</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/booth-was-featured-in-python-weekly-a-small-milestone-for-my-open-source-project-58cj</guid>
      <description>&lt;p&gt;A small milestone for a project I've been building: &lt;strong&gt;BOOTH was featured in Python Weekly!&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It appears in &lt;a href="https://www.pythonweekly.com/p/python-weekly-issue-767-october-15-2026" rel="noopener noreferrer"&gt;Python Weekly Issue #767 (October 15, 2026)&lt;/a&gt; under &lt;em&gt;Interesting Projects, Tools, and Libraries&lt;/em&gt;, described as a lightweight, zero-dependency Python checkpoint layer for structured LLM output reliability.&lt;/p&gt;

&lt;p&gt;I started BOOTH only a couple of months ago, and I'm still learning what it takes to turn a small library into something other developers find useful. Seeing it in a curated newsletter is encouraging.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why BOOTH exists
&lt;/h2&gt;

&lt;p&gt;A few months ago, a person working an airport gate helped me find mine after I'd gotten lost, boarding pass in hand, signs everywhere, still at the wrong gate. She didn't know where I'd flown in from, how the aircraft was built, or anything about my itinerary beyond that one boarding pass. She didn't need to. She knew the airport, and she knew what to check.&lt;/p&gt;

&lt;p&gt;That's the idea BOOTH is built around. The industry's default answer to "the model got it wrong" has mostly been to make the model know more: more context, more compute, bigger models, more tools bolted around it. And the problem still happens, because it isn't always a knowledge problem. Sometimes what's missing isn't more information going in. It's something whose only job is checking whether what came out actually meets the bar.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;BOOTH doesn't try to know more than your model does. It just knows what to check.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So the question I kept coming back to was:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Before passing an LLM response to the rest of the application, should we check it first?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A response can sound confident and still be ambiguous, fail validation, or disagree with the evidence your application retrieved. That's the problem BOOTH explores.&lt;/p&gt;

&lt;p&gt;I wrote the longer version of this idea on Substack: &lt;a href="https://substack.com/home/post/p-215310975" rel="noopener noreferrer"&gt;&lt;em&gt;Knowing less, but knowing what matters&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What BOOTH does
&lt;/h2&gt;

&lt;p&gt;BOOTH is a lightweight, provider-agnostic library that sits between an LLM call and your application logic. Instead of raw text, you get a structured result (&lt;code&gt;ACCEPTED&lt;/code&gt;, &lt;code&gt;REPAIRED&lt;/code&gt;, &lt;code&gt;AMBIGUOUS&lt;/code&gt;, &lt;code&gt;UNCERTAIN&lt;/code&gt; or &lt;code&gt;BLOCKED&lt;/code&gt;) so your app can decide what to do next.&lt;/p&gt;

&lt;p&gt;It has three main entry points:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;check()&lt;/code&gt;&lt;/strong&gt;: a synchronous checkpoint for ambiguity and confidence, with reconsideration retries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;acheck()&lt;/code&gt;&lt;/strong&gt;: the async counterpart.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;check_with_evidence()&lt;/code&gt;&lt;/strong&gt;: checks an answer against evidence your application supplies, such as documents from a RAG pipeline.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;

&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;call_llm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is the capital of France?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BOOTH returned &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# AMBIGUOUS / UNCERTAIN
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For evidence grounding, imagine your retrieval pipeline returns a policy saying refunds are available within 45 days, but the generated answer says 90 days:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;check_with_evidence&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;llm_answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;evidence&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;retrieved_docs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;compare_fn&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;your_comparison_function&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;booth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;BLOCKED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# the answer disagrees with the retrieved evidence
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You provide the comparison function, and BOOTH makes the checking step explicit.&lt;/p&gt;

&lt;p&gt;To be clear about the limits: BOOTH doesn't replace your retrieval pipeline, and agreeing with the supplied evidence doesn't prove the evidence is correct or complete.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why keep it lightweight?
&lt;/h2&gt;

&lt;p&gt;BOOTH is small on purpose. It has zero runtime dependencies, works with any provider through functions you supply, and returns structured results instead of making decisions for you.&lt;/p&gt;

&lt;p&gt;I don't want it to become another framework that forces you to rebuild your application around it. You should be able to try it in an existing Python project without changing your stack.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;boothpy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;Testing across providers and models has been the most useful part of building BOOTH. Real integrations expose edge cases that are easy to miss in your own setup. The repo now includes runnable examples with real console output for several providers and local models, including the negative results.&lt;/p&gt;

&lt;p&gt;I'm continuing to work on error handling, documentation, integrations, and the behavior of the checkpoint APIs. I'd also welcome contributors who want to test BOOTH, find edge cases, or pick up open issues.&lt;/p&gt;

&lt;p&gt;The Python Weekly feature is a nice milestone, but what matters most to me is whether developers find BOOTH useful enough to try in their own applications. If you work with Python, LLMs or RAG, I'd genuinely appreciate your feedback.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.pythonweekly.com/p/python-weekly-issue-767-october-15-2026" rel="noopener noreferrer"&gt;Python Weekly Issue #767&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;BOOTH on GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pypi.org/project/boothpy/" rel="noopener noreferrer"&gt;BOOTH on PyPI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/Vedantgitbot/booth/issues" rel="noopener noreferrer"&gt;Open issues and contributions&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to everyone who has explored the project, reported issues, contributed integrations, or shared feedback. I'm excited to keep building BOOTH in the open.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>python</category>
      <category>rag</category>
    </item>
    <item>
      <title>I Open-Sourced My Project for Hacktoberfest. Here's What Happened When Strangers Started Contributing.</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Wed, 07 Oct 2026 17:16:03 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/i-open-sourced-my-project-for-hacktoberfest-heres-what-happened-when-strangers-started-2bd7</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/i-open-sourced-my-project-for-hacktoberfest-heres-what-happened-when-strangers-started-2bd7</guid>
      <description>&lt;p&gt;I thought open-sourcing my project would get me stars.&lt;/p&gt;

&lt;p&gt;Instead, strangers broke it.&lt;/p&gt;

&lt;p&gt;And honestly, that was much more valuable.&lt;/p&gt;

&lt;p&gt;Two months ago, I started building &lt;strong&gt;BOOTH&lt;/strong&gt;, a small checkpoint layer that sits between an LLM call and your application and decides whether the response is trustworthy enough to use.&lt;/p&gt;

&lt;p&gt;It's intentionally small. Zero dependencies. Just mine.&lt;/p&gt;

&lt;p&gt;For Hacktoberfest, I opened one issue:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Test BOOTH against your favorite LLM provider. Document what breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;I expected silence.&lt;/p&gt;

&lt;p&gt;Instead, people started finding things I hadn't seen.&lt;/p&gt;

&lt;h2&gt;
  
  
  A 7B model reported confidence: 1.0 on wrong answers
&lt;/h2&gt;

&lt;p&gt;One contributor tested BOOTH with Ollama and a local 7B model.&lt;/p&gt;

&lt;p&gt;The surprising part wasn't that the model got things wrong.&lt;/p&gt;

&lt;p&gt;It was that the model reported &lt;strong&gt;confidence: 1.0&lt;/strong&gt; on every wrong answer.&lt;/p&gt;

&lt;p&gt;That's not necessarily a bug in the model. It's a limitation of treating model-generated confidence as if it were an objective measurement.&lt;/p&gt;

&lt;p&gt;I hadn't documented that behavior.&lt;/p&gt;

&lt;p&gt;Now it's part of the project's reality.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then came the API problems
&lt;/h2&gt;

&lt;p&gt;Another contributor hit a &lt;code&gt;401&lt;/code&gt; that only appeared with a newer API key format.&lt;/p&gt;

&lt;p&gt;Someone else hit a &lt;code&gt;404&lt;/code&gt; because the model they were testing had quietly been retired.&lt;/p&gt;

&lt;p&gt;Neither issue appeared in my local testing.&lt;/p&gt;

&lt;p&gt;That's the thing about testing your own project:&lt;/p&gt;

&lt;p&gt;You test the environment you have.&lt;/p&gt;

&lt;p&gt;Other people test the environments you don't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then a first-time contributor found an actual bug
&lt;/h2&gt;

&lt;p&gt;This one mattered more.&lt;/p&gt;

&lt;p&gt;When an underlying API call failed, BOOTH was swallowing the exception and returning a result that looked almost identical to the model simply being uncertain.&lt;/p&gt;

&lt;p&gt;Those are completely different situations.&lt;/p&gt;

&lt;p&gt;One means:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The model answered, but we shouldn't trust the answer."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The other means:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The model/API didn't actually give us a usable answer."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;BOOTH was making those two states look the same.&lt;/p&gt;

&lt;p&gt;A first-time open-source contributor found it.&lt;/p&gt;

&lt;p&gt;I shipped the fix within a day.&lt;/p&gt;

&lt;p&gt;That bug probably would have survived much longer without someone approaching the project with completely fresh eyes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then someone turned the chaos into documentation
&lt;/h2&gt;

&lt;p&gt;After several people had tested different providers, another contributor volunteered to turn the accumulated discoveries into a &lt;code&gt;TROUBLESHOOTING.md&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Not theoretical documentation.&lt;/p&gt;

&lt;p&gt;Real-world:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;symptom → cause → fix&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The kind of documentation you only write after somebody actually hits the wall.&lt;/p&gt;

&lt;p&gt;And it's better documentation because of that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four people. Four providers. A much better project.
&lt;/h2&gt;

&lt;p&gt;So far, four independent contributors have tested BOOTH against different providers and environments.&lt;/p&gt;

&lt;p&gt;OpenAI.&lt;/p&gt;

&lt;p&gt;Groq.&lt;/p&gt;

&lt;p&gt;Ollama.&lt;/p&gt;

&lt;p&gt;Gemini.&lt;/p&gt;

&lt;p&gt;Anthropic.&lt;/p&gt;

&lt;p&gt;They didn't owe me stars.&lt;/p&gt;

&lt;p&gt;They didn't have to tell me the project was great.&lt;/p&gt;

&lt;p&gt;They just ran it, broke it, and told me exactly what happened.&lt;/p&gt;

&lt;p&gt;That's probably the most valuable thing that has happened to the project so far.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part nobody tells you about open source
&lt;/h2&gt;

&lt;p&gt;When you open-source a project, it's tempting to measure everything by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GitHub stars&lt;/li&gt;
&lt;li&gt;forks&lt;/li&gt;
&lt;li&gt;contributors&lt;/li&gt;
&lt;li&gt;downloads&lt;/li&gt;
&lt;li&gt;social media likes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those numbers are nice.&lt;/p&gt;

&lt;p&gt;But they're not necessarily the most valuable outcome.&lt;/p&gt;

&lt;p&gt;The real value can be much quieter.&lt;/p&gt;

&lt;p&gt;Someone you've never met runs your software in an environment you've never used.&lt;/p&gt;

&lt;p&gt;They find the edge case you never considered.&lt;/p&gt;

&lt;p&gt;They report it.&lt;/p&gt;

&lt;p&gt;You fix it.&lt;/p&gt;

&lt;p&gt;Now your project is more honest than it was yesterday.&lt;/p&gt;

&lt;p&gt;That's what happened with BOOTH.&lt;/p&gt;

&lt;p&gt;It's still small.&lt;/p&gt;

&lt;p&gt;But it's been tested against multiple LLM providers by people who had no reason to tell me what I wanted to hear.&lt;/p&gt;

&lt;p&gt;They found limitations.&lt;/p&gt;

&lt;p&gt;They found broken assumptions.&lt;/p&gt;

&lt;p&gt;They found bugs.&lt;/p&gt;

&lt;p&gt;And the project is better because of it.&lt;/p&gt;

&lt;p&gt;If you're building something and wondering whether it's "ready" to open source, I'd argue that you don't necessarily need to wait until it's perfect.&lt;/p&gt;

&lt;p&gt;You might learn more by letting someone else break it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your first contributors probably won't make your project popular.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;They'll make it honest.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Want to try BOOTH?
&lt;/h2&gt;

&lt;p&gt;If you're curious, you can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;📦 &lt;strong&gt;Install BOOTH from PyPI:&lt;/strong&gt; &lt;a href="https://pypi.org/project/boothpy/" rel="noopener noreferrer"&gt;https://pypi.org/project/boothpy/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🐛 &lt;strong&gt;Pick up the Hacktoberfest issue:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth/issues/2" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth/issues/2&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⭐ &lt;strong&gt;Explore the source code:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you try it with an LLM provider I haven't tested yet, I'd genuinely love to hear what happens.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Break it. Report it. Make it better.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>hf26challenge</category>
      <category>hacktoberfest</category>
      <category>ai</category>
    </item>
    <item>
      <title>Hacktoberfest update: BOOTH is getting real contributions</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Mon, 05 Oct 2026 17:05:12 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/hacktoberfest-update-booth-is-getting-real-contributions-3a12</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/hacktoberfest-update-booth-is-getting-real-contributions-3a12</guid>
      <description>&lt;p&gt;A few days ago, I opened BOOTH for Hacktoberfest contributors.&lt;/p&gt;

&lt;p&gt;BOOTH is a lightweight, provider-agnostic Python library that adds a checkpoint before an LLM output reaches your application.&lt;/p&gt;

&lt;p&gt;check() / acheck() handle ambiguity and confidence checks, while check_with_evidence() lets you compare an answer against evidence retrieved by your RAG pipeline.&lt;/p&gt;

&lt;p&gt;Since Hacktoberfest started, developers have begun contributing integrations across different LLM providers and models.&lt;/p&gt;

&lt;p&gt;More contributions and feedback are always welcome. 🙌&lt;/p&gt;

&lt;p&gt;🛠️ Open issues: &lt;a href="https://github.com/Vedantgitbot/booth/issues" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth/issues&lt;/a&gt;&lt;br&gt;
🔗 Repo: &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Really happy to see BOOTH growing beyond just me.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>hacktoberfest</category>
      <category>opensource</category>
      <category>python</category>
    </item>
    <item>
      <title>Hacktoberfest 2026 is here, and BOOTH is open to contributors!</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Thu, 01 Oct 2026 14:15:49 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/hacktoberfest-2026-is-here-and-booth-is-open-to-contributors-1m6e</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/hacktoberfest-2026-is-here-and-booth-is-open-to-contributors-1m6e</guid>
      <description>&lt;p&gt;I’ve been building &lt;strong&gt;BOOTH&lt;/strong&gt;, a lightweight checkpoint layer for LLM outputs.&lt;/p&gt;

&lt;p&gt;The idea is simple: &lt;strong&gt;don’t automatically trust every LLM response. Check it first.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;check_with_evidence()&lt;/code&gt;, BOOTH lets you compare an LLM’s answer against evidence your RAG pipeline has already retrieved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;BOOTH v0.5.2&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Zero runtime dependencies&lt;/li&gt;
&lt;li&gt;Provider-agnostic&lt;/li&gt;
&lt;li&gt;Sync + async APIs&lt;/li&gt;
&lt;li&gt;Structured results&lt;/li&gt;
&lt;li&gt;300+ tests&lt;/li&gt;
&lt;li&gt;MIT licensed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I’m looking for developers interested in &lt;strong&gt;LLM reliability, RAG, and Python&lt;/strong&gt; to contribute this Hacktoberfest.&lt;/p&gt;

&lt;p&gt;🔗 &lt;strong&gt;Explore BOOTH:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth&lt;/a&gt;&lt;br&gt;
🛠️ &lt;strong&gt;Find an issue:&lt;/strong&gt; &lt;a href="https://github.com/Vedantgitbot/booth/issues" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth/issues&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Whether it’s your first open-source contribution or your hundredth, you’re welcome!&lt;/p&gt;

</description>
      <category>hacktoberfest</category>
      <category>hf26challenge</category>
      <category>ai</category>
      <category>python</category>
    </item>
    <item>
      <title>Your LLM gave you an answer. Should your application trust it?</title>
      <dc:creator>Vedant Brahmbhatt</dc:creator>
      <pubDate>Wed, 30 Sep 2026 15:21:13 +0000</pubDate>
      <link>https://dev.to/vedant_brahmbhatt_7be1822/your-llm-gave-you-an-answer-should-your-application-trust-it-kcc</link>
      <guid>https://dev.to/vedant_brahmbhatt_7be1822/your-llm-gave-you-an-answer-should-your-application-trust-it-kcc</guid>
      <description>&lt;p&gt;I built BOOTH, a small checkpoint layer that sits between your app and an&lt;br&gt;
LLM call and returns a structured decision instead of just fluent text.&lt;/p&gt;

&lt;p&gt;Example:&lt;/p&gt;

&lt;p&gt;Evidence: "Returns are allowed within 45 days."&lt;br&gt;
LLM answer: "Returns are allowed within 90 days."&lt;/p&gt;

&lt;p&gt;Confident, fluent, and unsupported. Self-reported confidence won't catch it.&lt;br&gt;
Checking against the evidence will.&lt;/p&gt;

&lt;p&gt;BOOTH doesn't decide what's true. You supply the comparison, and BOOTH turns&lt;br&gt;
the outcome into an explicit status you can't accidentally ignore:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;result = booth.check_with_evidence(
    answer=llm_answer,
    evidence=retrieved_docs,
    compare_fn=your_comparison_function,  # your logic, your definition of "agrees"
)

if result.status == booth.BLOCKED:
    print(result.detail)  # why it was blocked
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Results also carry a reason and a checker_failed flag, so "the answer disagreed"&lt;br&gt;
and "my comparison function crashed" are never mixed up.&lt;/p&gt;

&lt;p&gt;There's also check() / acheck() for plain LLM calls: ambiguity detection plus&lt;br&gt;
confidence checks with reconsideration retries.&lt;/p&gt;

&lt;p&gt;It keeps your RAG pipeline and provider as they are. Zero runtime dependencies,&lt;br&gt;
provider-agnostic, MIT, Python 3.9+.&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pip install boothpy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;GitHub: &lt;a href="https://github.com/Vedantgitbot/booth" rel="noopener noreferrer"&gt;https://github.com/Vedantgitbot/booth&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;How are you currently deciding whether an LLM output is safe to pass downstream?&lt;/p&gt;

&lt;p&gt;Beta · 300+ tests · CI passing · MIT · Python 3.9+&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>llm</category>
      <category>rag</category>
    </item>
  </channel>
</rss>
