The latest articles on DEV Community by Edwin Lisowski (@edwin_lisowski). https://dev.to/edwin_lisowski https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2463335%2F82ed05dd-b503-4f2c-b955-8c2188fddbd7.jpg https://dev.to/edwin_lisowski en Edwin Lisowski Wed, 27 Nov 2024 08:51:21 +0000 https://dev.to/edwin_lisowski/sontextcheck-llm-rag-evaluation-framework-59a9 https://dev.to/edwin_lisowski/sontextcheck-llm-rag-evaluation-framework-59a9 <p>Hi all! We open-sourced a framework for testing LLMs, RAGs, and chatbots. The tool automates query generation, completion requests, regression detection, penetration testing, and hallucination assessment. Designed for developers, researchers, and businesses. And we are looking for contributors! Feel free to try it out for yourself and share your feedback!</p> <p><a href="https://github.com/Addepto/contextcheck" rel="noopener noreferrer">Repo on Github</a></p> aiops Edwin Lisowski Thu, 21 Nov 2024 10:20:15 +0000 https://dev.to/edwin_lisowski/contextcheck-an-open-source-framework-for-testing-and-evaluating-llms-rags-chatbots-1mpa https://dev.to/edwin_lisowski/contextcheck-an-open-source-framework-for-testing-and-evaluating-llms-rags-chatbots-1mpa <p>Hey devs!</p> <p>We just open-sourced ContextCheck, a framework for testing and evaluating LLMs, RAGs, and chatbots 🚀</p> <p>What it does:</p> <ul> <li>Generates queries and handles completions</li> <li>Detects regressions and hallucinations</li> <li>Runs penetration tests</li> <li>Works in CI pipelines (YAML-configurable)</li> </ul> <p>We built it while developing our AI Knowledge Base Assistant to solve real headaches with testing and validating LLMs. Now it’s out there for you to use, break, and improve.</p> <p>Try it out and let us know what you think! ➡️ <a href="https://github.com/Addepto/contextcheck" rel="noopener noreferrer">Github repo</a></p> ai Edwin Lisowski Thu, 21 Nov 2024 10:04:20 +0000 https://dev.to/edwin_lisowski/contextcheck-an-open-source-framework-for-testing-and-evaluating-llms-rags-chatbots-3hkn https://dev.to/edwin_lisowski/contextcheck-an-open-source-framework-for-testing-and-evaluating-llms-rags-chatbots-3hkn <p>Hey everyone!</p> <p>I’m one of the co-founders of Addepto, and I’m excited to share ContextCheck—a new open-source framework we’ve developed for testing and evaluating LLMs, RAGs, and chatbots.</p> <p>ContextCheck offers tools to:</p> <ul> <li>Automatically generate queries and request completions</li> <li>Detect regressions and assess hallucinations</li> <li>Perform penetration testing</li> <li>Ensure the robustness and reliability of AI systems</li> </ul> <p>It’s fully configurable via YAML and integrates seamlessly into CI pipelines for automated testing.</p> <p>We built ContextCheck during the development of our AI-powered Knowledge Base Assistant to solve the challenges we faced with testing and validating Large Language Models. It’s a tool designed by developers for developers to tackle real-world issues.</p> <p>We’d love for you to try it out, contribute, and share your feedback!</p> <p><a href="https://github.com/Addepto/contextcheck" rel="noopener noreferrer">Github repo</a></p> showdev github opensource llm