DEV Community: Natalie Fagundo

Open sourcing our new Text-to-SQL LLM app starter template

Natalie Fagundo — Thu, 26 Sep 2024 20:18:09 +0000

We are excited to announce the release of Inductor’s latest open source LLM application starter template, for building Text-to-SQL LLM apps (GitHub repo here).This template is designed to make it easier than ever for developers to build and deploy AI apps that can convert natural language into SQL queries, execute them on a database, and return actionable insights. Whether you're looking to create a tool for data analysts, automate reporting, or build an internal knowledge assistant capable of answering complex data-related questions, this starter template provides everything you need to get up and running quickly.

Just like our other LLM app starter templates, the Text-to-SQL app template offers more than just a basic structure for building your app. It integrates a complete end-to-end developer workflow that supports the iterative nature of building production-ready LLM applications. It’s designed for rapid prototyping, robust testing, and ongoing optimization – key requirements for anyone building enterprise-grade solutions.

Key features

Playground for prototyping: Leverage Inductor's playground feature to experiment with different queries and app configurations interactively.
Built-in testing and evaluation: A full Inductor-powered test suite is included to systematically evaluate the performance of the app across various scenarios.
Automated experimentation and optimization: Iterate quickly with hyperparameters (e.g., to optimize prompts and choice of model) using Inductor’s powerful experimentation and optimization tools.
Comprehensive logging: Capture and analyze real-time interactions and SQL query executions with Inductor’s integrated logging and observability tools.
Seamless SQL generation: Convert user queries into SQL that interacts with your database to retrieve information.
Integrated SQL query validation: Ensure the validity of SQL queries before execution, improving app reliability.
Schema-aware interactions: Automatically generate SQL based on your database schema, so that queries are valid and relevant.
Easy customization: This template is built to be easy to apply to your database, and also framework-agnostic, making it easy to integrate into any LLM-powered system.

This Text-to-SQL LLM app starter template is your gateway to creating LLM-powered applications that turn natural language into database queries with ease. Whether you’re just starting or looking to scale your data-driven tools, this template is designed to accelerate your development process.

A systematic developer workflow for delivering production-grade Text-to-SQL LLM apps

This template is pre-configured to leverage Inductor’s developer platform. Inductor provides the tools you need to build and deliver next-gen LLM applications, offering a systematic approach to every phase of development and deployment such as:

Prototyping playground: Share prototypes with your team in a secure environment that integrates with your tests and logs.
Robust test suites: Test for accuracy, consistency, and edge cases with test cases and quality measures.
Comprehensive logging: Monitor live executions to understand real-world user interactions, identify issues, and optimize application performance.
Automated experimentation: Use hyperparameters to experiment with different configurations (e.g., different models or prompts), enabling rapid iteration and optimization.

By building with Inductor’s platform, you ensure that your LLM application isn’t just functional – it’s optimized for reliability, usefulness, and business impact. Let’s dive deeper and see how Inductor integrates into the development lifecycle for this specific Text-to-SQL application:

Auto-generate playgrounds: Experimentation is key when building LLM applications. With a single command (inductor playground app:get_analytics_results), you can create a fully interactive UI for real-time interactive testing that can be shared with domain specialists. This enables developers and stakeholders to experiment with the app’s behavior without writing additional code. Within an Inductor playground you can view past executions, add executions to test suites, see logged values, and experiment with different configurations using developer-controlled hyperparameters.

Systematic testing with Inductor: Inductor makes it easy to have quality control baked into the development process. The template includes a pre-configured test suite that enables developers to run systematic tests on their Text-to-SQL app. These tests ensure that the generated SQL is accurate, valid, and capable of addressing various query types.

Live monitoring and debugging: Understanding how your application performs under real-world conditions is essential. With Inductor’s logging and monitoring features, you can track every interaction in real-time, enabling you to catch and address issues as they arise.

Automated experimentation: Use hyperparameters to experiment with different configurations (e.g., different models or prompts), enabling rapid iteration and optimization.

Unlocking business potential with Text-to-SQL

The Text-to-SQL LLM app starter template is a powerful entry point for businesses looking to leverage AI to interact with data in more intuitive ways. With this template, you can shift from needing to write complex, technical SQL queries to a conversational experience that enables anyone in your organization to pull insights from databases.

But, that’s just the beginning. By customizing and extending this template, you can evolve your LLM-powered application from simply querying data to driving business actions. For example, you can enable non-technical users to generate SQL queries to analyze performance metrics and then follow up with automated actions like sending reports or triggering alerts – all powered by natural language.

Future expectations for enterprise LLM applications

As enterprises continue to build and adopt LLM-powered applications, the iterative process of evolving from simple data interactions to business actions and impact will redefine how businesses operate and serve their customers. Technologies like Text-to-SQL AI apps offer a powerful starting point, transforming data access into an intuitive experience. However, the future lies in building on this foundation – moving from querying data to influencing real-world actions with seamless, AI-driven interactions.

Looking forward, enterprises that embrace the iterative process of developing and delivering AI apps will be best positioned to harness the full potential of LLMs. Those that continuously refine their applications to meet changing user needs and leverage the power of AI across departments will drive meaningful improvements in KPIs, from revenue growth to customer satisfaction. The roadmap for LLM applications is clear: evolve, iterate, and deliver AI-driven outcomes that shape the future of business.

Inductor is essential for accelerating this evolution. As a comprehensive developer platform, Inductor empowers teams to rapidly prototype, test, iterate on, and monitor their LLM applications. With features like robust test suites, automated experimentation, and comprehensive logging, Inductor ensures that developers can seamlessly build, evaluate, improve, and observe their AI-powered applications without the complexity of building and managing the requisite tools and infrastructure. This iterative approach enables enterprises to continuously refine their applications in order to ship rapidly, make them more adaptable to shifting market demands, and improve business outcomes at every stage.

Ready to build your own Text-to-SQL LLM app?

Getting started is easy. Simply visit the GitHub repository, clone the Text-to-SQL LLM app starter template, and follow the instructions to get your application up and running in minutes. With Inductor, you’ll have access to powerful tools for experimenting, testing, and optimizing your app every step of the way.

🚀 Start building your Text-to-SQL app today and explore the potential of integrating natural language interfaces with your enterprise’s data systems!

💻 Want to learn more? Dive into our documentation or book a demo to see how Inductor can help you build AI-powered applications that drive real business value. 🛠️

Open sourcing our new Chat With Your PDFs LLM app starter template

Natalie Fagundo — Thu, 19 Sep 2024 23:41:39 +0000

We are thrilled to announce the release of Inductor’s newest open source LLM application starter template: Chat With Your PDFs (GitHub repo here). This template is designed to empower developers to quickly build and ship a conversational AI bot that can interact with, answer questions about, and extract information from PDF documents. Whether you’re creating a knowledge base search bot, a customer support bot, a document review assistant, or simply want to build AI that can intelligently answer questions about a collection of PDFs, this starter template provides the foundation that you need.

Much like our previous starter template releases, this template goes beyond just providing the basic structure for an LLM application. It includes an end-to-end developer workflow tailored to the iterative nature of building production-ready LLM applications. Key features of this template include:

Robust test suites: Ensure the accuracy and reliability of your PDF chatbot with systematic testing that covers various interaction scenarios.
Automated experimentation and optimization: Quickly iterate on and refine your application by experimenting with different models, prompts, and configurations using Inductor hyperparameters.
Instant prototyping playground: A secure, auto-generated environment that enables you to prototype and share your PDF chatbot with your team, fully integrated with your test and experimentation setups.
Comprehensive logging: Monitor live traffic, understand user interactions, resolve issues, and continuously improve your application with integrated logging and observability.
PDF parsing and embedding: Leverage Unstructured for smart chunking and Sentence-Transformers for embedding generation, transforming static PDFs into dynamic, interactive knowledge sources. Complex documents, including those with images and tables, are processed and stored in a ChromaDB vector database for efficient retrieval based on user queries.

This Chat With Your PDFs template is your fast track to developing an LLM-powered bot that can handle complex document queries with ease. Whether you’re just starting or looking to scale, this template makes it simple to get up and running in minutes.

‍

Beyond consumer AI: why enterprises need custom LLM applications

The rise of AI-powered knowledge assistants highlights the growing demand for intelligent systems that can seamlessly access and retrieve information. For example, products like ChatPDF, Liner, Eightify, and Phind leverage different data sources to enable chat assistants capable of querying and interacting with documents, web pages, videos, and research papers. These products enable users to extract key insights and summaries in real time, demonstrating the value of LLM-powered applications in various industries.

These products exemplify how AI can enhance productivity by transforming static content into dynamic, interactive resources. Yet, while they may work well for general consumer use cases, enterprises often require more customized and complex solutions to meet their unique demands. For example, businesses dealing with legal documents, HR policy documents, or specialized training materials may need to handle deeper domain-specific queries, process multiple data formats, and integrate seamlessly with existing internal systems – demands that off-the-shelf products often don’t meet.

This is where Inductor comes in. Enterprises often need flexible, tailor-made solutions, and Inductor provides a platform built specifically for LLM app development. Our tools enable developers to rapidly create production-ready AI applications that address unique business needs – for example for use cases such as internal knowledge management, legal compliance, or advanced research.

‍

Unlocking the power of PDF parsing and embedding

At the heart of this starter template is its ability to transform static PDFs into dynamic, conversational experiences. By leveraging Unstructured for smart chunking and Sentence-Transformers for embedding generation, the template ensures that even complex documents are easy to interact with.

Once the PDF content is processed, it's stored in a ChromaDB vector database. This allows the application to efficiently retrieve the most relevant sections based on user queries, ensuring that the LLM has the context it needs to generate accurate, meaningful responses. Using retrieval-augmented generation (RAG), this system seamlessly integrates with an LLM, such as GPT-4o, to provide users with answers and deeper insights from the documents.

Whether you're navigating technical manuals, financial reports, or academic papers, this capability turns static PDFs into a rich source of interactive knowledge.

‍

Supercharge your LLM app development with Inductor

A key feature that sets the Chat With Your PDFs LLM starter template apart is its seamless integration with Inductor, a powerful platform built to streamline the full development and delivery lifecycle of your LLM application. From rapid prototyping to systematic testing and real-time monitoring, Inductor equips developers with the tools needed to build, refine, and deliver applications at every stage. Let’s dive into some of the standout features this integration provides.

‍

Instantly generate a playground UI for experimentation

Experimentation is at the heart of LLM application development, and with Inductor, you can spin up a fully interactive playground UI with a single command. This environment enables you to interactively test different queries, iterate on your application’s behavior, and share your Chat With Your PDFs app with teammates and subject matter experts – all without writing additional code. With your playground, you’ll have a flexible, no-code UI to rapidly iterate on and improve your LLM app. Just run:

inductor playground app:chat_with_pdf

‍

Systematically evaluate your app using test suites

The Chat With Your PDFs template includes a built-in set of test suites designed to test your application’s performance across various scenarios, providing actionable feedback so that you can quickly identify and address any issues. Each test suite includes pre-configured test cases, quality measures, and hyperparameters for evaluation of your app. An example of the results of running an included test suite can be found here.

Whether you're validating how your LLM app handles complex queries or testing its response accuracy across different PDFs, Inductor enables you to rapidly do the evaluations that you need to improve your app efficiently.

‍

Rapidly refine your app with Inductor hyperparameters

Inductor hyperparameters enable you to rapidly test and compare different configurations of your LLM application. Hyperparameters are fully integrated in the Inductor platform where they can be leveraged in playgrounds, test suites, and live A/B testing. Using hyperparameters in playgrounds allows for controlled and collaborative interactive experimentation. Using hyperparameters in test suites allows for systematic selection of optimized configurations.

Here are some examples of hyperparameters included in the Chat With Your PDFs template:

query_num_chat_messages: Adjusts the number of previous chat messages used in the query for retrieving relevant information from your vector database.
query_result_num: Controls how many results are retrieved from the vector database for each query.
query_filter_out_program_messages: Determines whether or not chatbot-generated messages are filtered out from the query sent to the vector database.

By leveraging hyperparameters, you can rapidly run in-depth experiments on multiple different configurations and identify the optimal setup for your application.

‍

Monitor live executions in real time

Understanding how your LLM app performs in real-world conditions is crucial. With Inductor’s live execution logging, every detail of your app’s execution is automatically recorded, including inputs, outputs, the specific text snippets retrieved by the RAG system, and more. This enables you to monitor your app’s performance in real time, giving you invaluable insights into user behavior, system efficiency, and areas for improvement.

‍

🚀 What’s Next?

Ready to build your own AI-powered PDF chatbot? Get started in minutes! Simply visit the GitHub repo, clone the Chat With Your PDFs starter template, and follow the easy steps to start developing your LLM application. It’s that simple! 💻✨

Want to dive deeper? Explore more about Inductor in our documentation or book a demo to see how we can help supercharge your AI projects! 🛠️🚀

Developer workflows for building RAG-based LLM apps with systematic evaluation and iteration

Natalie Fagundo — Wed, 04 Sep 2024 19:37:07 +0000

Robust test suites: Ensure the accuracy and reliability of your PDF chatbot with systematic testing that covers various interaction scenarios.
Automated experimentation and optimization: Quickly iterate on and refine your application by experimenting with different models, prompts, and configurations using Inductor hyperparameters.
Instant prototyping playground: A secure, auto-generated environment that enables you to prototype and share your PDF chatbot with your team, fully integrated with your test and experimentation setups.
Comprehensive logging: Monitor live traffic, understand user interactions, resolve issues, and continuously improve your application with integrated logging and observability.
PDF parsing and embedding: Leverage Unstructured for smart chunking and Sentence-Transformers for embedding generation, transforming static PDFs into dynamic, interactive knowledge sources. Complex documents, including those with images and tables, are processed and stored in a ChromaDB vector database for efficient retrieval based on user queries.

Beyond consumer AI: why enterprises need custom LLM applications

Unlocking the power of PDF parsing and embedding

Whether you're navigating technical manuals, financial reports, or academic papers, this capability turns static PDFs into a rich source of interactive knowledge.

Supercharge your LLM app development with Inductor

Instantly generate a playground UI for experimentation

inductor playground app:chat_with_pdf

Systematically evaluate your app using test suites

Rapidly refine your app with Inductor hyperparameters

Here are some examples of hyperparameters included in the Chat With Your PDFs template:

query_num_chat_messages: Adjusts the number of previous chat messages used in the query for retrieving relevant information from your vector database.
query_result_num: Controls how many results are retrieved from the vector database for each query.
query_filter_out_program_messages: Determines whether or not chatbot-generated messages are filtered out from the query sent to the vector database.

By leveraging hyperparameters, you can rapidly run in-depth experiments on multiple different configurations and identify the optimal setup for your application.

Monitor live executions in real time

🚀 What’s Next?

Want to dive deeper? Explore more about Inductor in our documentation or book a demo to see how we can help supercharge your AI projects! 🛠️🚀

Build production-grade LLM apps faster and smarter with Inductor and MongoDB vector search

Natalie Fagundo — Fri, 23 Aug 2024 21:32:55 +0000

We’re thrilled to announce Inductor’s partnership with MongoDB and release our latest open-source LLM application starter template, designed for a documentation Q&A bot leveraging MongoDB vector search. (GitHub repo here). This template is designed to streamline the development process for RAG-based (Retrieval-Augmented Generation based) LLM applications by leveraging the powerful vector search functionality of MongoDB, along with a seamless Inductor integration for rapid prototyping, testing, experimentation, and monitoring.

This starter template not only provides the foundational scaffolding for a RAG-based LLM application but also incorporates an end-to-end developer workflow optimized for rapid iterative development and delivery. With MongoDB vector search and Inductor, you can efficiently implement and optimize data retrieval, and ensure the quality and performance of your LLM application.

Key components of this integrated workflow include:

MongoDB Vector Search: MongoDB vector search enables fast, scalable data retrieval in order to bring your unique data into your LLM application via retrieval-augmented generation (RAG).
Advanced Test Suites: Systematically test your LLM application to ensure quality and reliability.
Hyperparameter Optimization: Automate experimentation to rapidly find the optimal design for your LLM app, considering factors like model choice, prompt configuration, and retrieval augmentation.
Auto-Generated Playground: Instantly and securely share a prototyping environment that integrates with test suites and hyperparameters for collaborative development.
Integrated Logging: Monitor live traffic to understand usage, resolve issues, facilitate A/B testing, and continually improve your application.

This new template for developing an LLM-powered documentation Q&A bot takes minutes to set up. With minimal effort, you can integrate and configure the application to work with your own documentation, and benefit from MongoDB's efficient and scalable vector search capabilities.

‍

Benefits of using MongoDB vector search for RAG-based LLM applications

Integrating MongoDB vector search into your RAG-based (Retrieval-Augmented Generation based) LLM application offers a number of advantages - enabling efficient, scalable, production-grade data retrieval in order to easily and rapidly enable your LLM application to operate on your unique data. Here are the key benefits:

Efficient large-scale data retrieval

MongoDB's vector search capabilities enable the efficient handling of large datasets. By leveraging high-dimensional vectors to represent data, MongoDB allows for fast and accurate retrieval of relevant information, improving the performance of your LLM applications. This is particularly beneficial for RAG systems that require quick access to vast amounts of context to generate accurate responses.

Improved search accuracy

With vector search, MongoDB can perform similarity searches that go beyond traditional keyword-based methods. This means your RAG LLM application can retrieve contextually relevant documents even if the exact keywords aren’t present. This leads to more accurate and meaningful responses, enhancing the overall user experience.

Scalability and flexibility

MongoDB's architecture is designed for scalability, allowing your application to grow seamlessly as your data and user base expand. Whether you're dealing with a few thousand documents or millions, MongoDB can scale to meet your needs without compromising performance. Additionally, its flexible schema supports a variety of data types, making it easier to integrate diverse data sources.

‍

Turbocharge development speed with a seamless integration with Inductor

You will see in this starter template that integrating Inductor ensures that MongoDB’s vector search capabilities are seamlessly combined with the ability to rapidly prototype, test, experiment, and monitor your LLM application. This enables a rapid, streamlined progression from prototype to production, significantly speeding up time to market for your LLM applications.

Enhanced experimentation and optimization

As seen in this starter template, MongoDB vector search can be easily combined with Inductor's hyperparameter optimization tools. This enables you to rapidly and systematically experiment with different retrieval configurations, model parameters, and data representations to find the optimal setup for your application. Such iterative development ensures that you can rapidly and continually improve your LLM application’s accuracy and efficiency.

Inductor Custom Playgrounds enable you to auto-generate a powerful, instantly shareable playground for your LLM app with a single CLI command - and run it within your environment. Playgrounds provide a developer-first approach to prototype and iterate on LLM programs fast, as well as loop collaborators (including less-technical collaborators) into your development process, and get their feedback early and often.

Rigorous and continuous evaluations

An Inductor test suite is included with the documentation Q&A bot application to evaluate its performance and enable you to systematically test and improve. The included test suite consists of a set of test cases, each containing a set of input (i.e., argument) values for the LLM application and an example of an output value that should be considered high-quality or correct. The test suite also includes a set of quality measures specifying how to evaluate the output of the LLM application. Quality measures can be programmatic, human, or LLM-powered. Using Inductor test suites you can:

Rapidly customize quality evaluation for your use case
Auto-generate shareable UIs for human evals, and automate with rigorous LLM-powered evals
Construct, evolve, and share test cases
Automatically orchestrate test suite execution

Robust monitoring and logging

With integrated logging capabilities, you can monitor search queries and retrieval performance in real time. This helps in identifying bottlenecks, understanding user behavior, and resolving issues quickly. The detailed logs also facilitate A/B testing, enabling data-driven decisions to further enhance your application.

Continuous and cost-efficient improvement

By leveraging Inductor’s LLM app development platform and MongoDB vector search within your RAG LLM application, you can achieve a higher level of performance, accuracy, and scalability. This powerful combination ensures that your LLM applications are well-equipped to handle complex queries and provide users with precise, contextually relevant responses.

The documentation Q&A bot application

The LLM-powered documentation Q&A bot leveraging MongoDB vector search (GitHub repo here) is a RAG-based LLM application that answers questions using one or more Markdown documents as a data source to provide context. This starter template is intended for use cases like Q&A on developer documentation that have one or more Markdown documents on which you would like to provide a question-answering (Q&A) capability.

Get started

To get started in minutes, visit the GitHub repo, clone the documentation Q&A starter template leveraging MongoDB vector search, and follow the simple steps provided to start systematically developing your LLM application.

Open-sourcing Inductor LLM app starter templates: An out-of-the-box systematic approach for developing LLM applications

Natalie Fagundo — Wed, 07 Aug 2024 18:13:00 +0000

We are excited to announce that we’ve open-sourced Inductor’s first LLM application starter template (GitHub repo here). This template will make it easy for you to get started with a systematic and iterative development process for building and shipping a RAG-based LLM application. Most templates show you how to get started by providing the simple scaffolding for an LLM application. Beyond that, this template also includes an end-to-end developer workflow optimized for the iterative development required to confidently and efficiently develop a production-grade LLM application. Some key components of the integrated developer workflow are:

Test suites to systematically test the LLM application and ensure its quality.
Hyperparameters to automate experimentation and rapidly find the LLM app design that delivers the results that you need, across choice of model, prompt, retrieval augmentation, and more.
An auto-generated playground that can be instantly and securely shared for prototyping, and that integrates with test suites and hyperparameters.
Integrated logging for monitoring your live traffic in order to understand usage, resolve issues, facilitate A/B testing, and further improve the application.

The first template that we are now releasing is a getting started template for developing an LLM-powered documentation Q&A bot. It takes just minutes to get started, and you can easily integrate and configure the application to work with your own sources of documentation.

Why we’re building LLM application starter templates

As the demand for LLM-powered applications and product features grows, developers and their teams find themselves in need of a comprehensive and streamlined approach to their end-to-end development lifecycle. In the world of traditional application development, there is a well-established development lifecycle and a clear methodology for testing quality. Developers can rely on a structured process that guides them from concept to production, ensuring robust and reliable applications. However, when it comes to developing applications with large language models (LLMs), the path is far less straightforward. LLM application development requires a more experimental and iterative approach, where developers must continually refine and optimize their applications to achieve the desired performance.

This iterative nature presents several challenges, and developers need ways to:

Rapidly prototype with stakeholders
Systematically evaluate their LLM application or feature
Identify and implement improvements
Observe behavior in production and take appropriate action

Without the right tools and workflows, navigating this process is time-consuming and complex.

Our solution: jumpstart systematic LLM app development

Enter Inductor’s LLM application starter templates. Designed to address the unique challenges of LLM application development, each template is open-source and provides an easy path to get started quickly and efficiently. Each template includes the necessary scaffolding to facilitate rapid prototyping as well as streamline the progression from prototype to production.

Here’s what you can expect from each Inductor LLM app starter template:

Application scaffolding: A robust foundation for your LLM application, ensuring you have all the essential components to build upon.
Out-of-the-box UI for rapid prototyping: With a single CLI command, you can start an auto-generated and securely shareable user interface that enables you to quickly prototype and gather feedback from stakeholders, via Inductor playgrounds.
Test suite scaffolding for easy evaluation-driven development: Each template includes an Inductor test suite that can be customized for your particular use case.
Experimentation scaffolding for systematic improvement: Each template includes built-in touchpoints for rapid and automated experimentation, which can be used with Inductor to automate and orchestrate testing of multiple different app variants in order to further improve your app.
Production logging integration for easy observability: Pre-built logging integration to maintain visibility and monitor your application’s performance in a production environment.

The Inductor platform, if used in conjunction with each starter template, provides the tools and systems needed to bring successful production-grade LLM applications to market:

Documentation Q&A bot starter template

The LLM-powered documentation Q&A bot (GitHub repo here) is a RAG-based LLM application that answers questions using one or more Markdown documents as a data source to provide context. This starter template is intended for use cases like Q&A on developer documentation that have one or more Markdown documents on which you would like to provide a question-answering (Q&A) capability.

App architecture

The application is implemented in Python and includes two main components:

An ETL (Extract, Transform, Load) process that parses, chunks, and embeds the relevant Markdown files and populates a vector database. (By default, the starter template uses an included sample Markdown file, and it can easily be configured to instead utilize your own Markdown files.)
The main application entrypoint, which takes a question as input, retrieves relevant content from the vector database, and uses an LLM to generate an answer to the question.

Specifically, the ETL process ingests one or more Markdown files, splits them into chunks by Markdown sections, and converts each section to an embedding using Sentence-Transformers' all-MiniLM-L6-v2 model (the default model for Chroma). The embeddings, along with their associated chunks and metadata, are stored locally in a Chroma vector database. The app can also easily be modified to instead utilize a different vector database.

The main application entrypoint consists of a function that takes a question as input, queries the vector database to retrieve the most relevant Markdown content based on the question, and then uses the OpenAI “gpt-4o” model to generate and return an answer to the question. The app can easily be modified to utilize a different LLM or LLM provider.

Scaffolding for an effective development workflow: Inductor integration

Going from prototype to production with an LLM application for your particular use case requires iterative testing, experimentation, and collaboration, as well as live observability so that you are not flying blind when you ship. To enable doing this rapidly and reliably, Inductor provides a platform for prototyping, evaluating, improving, and observing your LLM app. This starter template includes pre-built scaffolding that leverages Inductor’s capabilities to enable you to iterate quickly, ship reliably, and collaborate effectively.

Key features include:

Test Suites: Easily, rigorously, and continuously test your LLM application with Inductor’s test suites and CLI, to systematically find shortcomings in behavior, accuracy, or cost-efficacy.
Hyperparameters: Dramatically accelerate your experimentation and optimization process with Inductor hyperparameters, to rapidly find the LLM app design that eliminates any shortcomings.
Playgrounds: Quickly prototype and collaborate using a playground that is instantly auto-generated for your LLM app, can be easily and securely shared, and integrates seamlessly with your test suites and hyperparameters.
Logging: Gain deep insights into usage, detect and resolve issues, and continuously improve your application with Inductor’s rich, automated production logging.

With these features, Inductor empowers you to build, refine, and deliver your LLM applications more effectively than ever before.
‍

Test suites

An Inductor test suite is included alongside the documentation Q&A bot application to evaluate its performance and enable you to systematically test and improve. The included test suite consists of a set of test cases, each containing a set of input (i.e., argument) values for your LLM application and an example of an output value that should be considered high-quality or correct. The test suite also includes a set of quality measures specifying how to evaluate the output of your LLM program. Quality measures can be programmatic, human, or LLM-powered. Using Inductor test suites you can:

Rapidly customize quality evaluation for your use case
Auto-generate shareable UIs for human evals, and automate with rigorous LLM-powered evals
Construct, evolve, and share test cases Automatically orchestrate test suite execution

Within the test suite that is part of this starter template, the included set of test cases can be split into the following categories:

Common questions with examples of high quality answers or target outputs
Unanswerable questions
Out of scope questions
Malicious questions

Additionally, test cases can easily be added by modifying the test_cases.yaml file within the starter template.

Along with these test cases are quality measures. This template uses LLM-powered quality measures to assess:

Can the question be answered with the provided context?
Is the target output contained in the answer provided?

Inductor enables easily running the test suite and viewing its results, an example of which are as follows:

‍

Hyperparameters

Improving an LLM application requires iterative experimentation in order to test different variants of your app’s design in order to find the design that yields the desired app behavior and quality. For example, this can include changing the prompt content, adjusting the prompt construction, selecting different models, tweaking model settings (e.g., temperature), and refining retrieval augmentation techniques. Inductor hyperparameters enable you to systematically and rapidly test and evaluate different configurations of your LLM application. This capability helps you assess the quality and cost-effectiveness of various setups, enabling rapid experimentation while maintaining organization and rigor.

In summary, hyperparameters enable you to:

Automate your experimentation in order to rapidly find the LLM app design that delivers the results that you need, across choice of model, prompt, retrieval augmentation, or anything else.
Automatically version and track all experiment results.

This starter template comes pre-built with two key hyperparameters (and you can also easily add more based on your needs):

Rephrasing vector database query: This hyperparameter controls whether to use the original user question to query the vector database or to rephrase the question to generate a more informative and relevant vector database query. Rephrasing can incorporate additional keywords and phrases to improve retrieval accuracy. However, this strategy may introduce trade-offs, such as increased latency and higher costs due to the additional LLM API call required for rephrasing. By using this hyperparameter, you can easily experiment with and evaluate the effectiveness of using the original versus the rephrased question.

Number of contextual results retrieved from the vector database: This hyperparameter sets the number of results to be retrieved from the vector database. Adjusting this setting allows you to control the breadth of information retrieved, which can impact the comprehensiveness and relevance of the responses provided by your documentation Q&A bot.

When you run the test suite included in the starter template, Inductor will automatically run and evaluate your LLM app on all included test cases for all combinations of values of the hyperparameters, so that you can easily and rapidly identify the best hyperparameter configuration for your app (by simply clicking on the “Hparam summary” button seen in the screenshot above). Additionally, as seen in the next section below, you can also interactively experiment with different hyperparameter values via a Customer Playground.

Therefore, by leveraging hyperparameters, you can rapidly evolve your LLM application to achieve the desired balance between performance, accuracy, and cost, ultimately enhancing the user experience of your documentation Q&A bot in a cost-efficient manner.

‍

Playgrounds

In particular, with Custom Playgrounds you can:

Auto-generate a custom playground UI for your LLM app
Run securely in your environment, with your code and data sources
Share instantly
Iteratively develop test suites for systematic evaluation and improvement

All interactive executions of your LLM program in your playground are automatically logged, so that you can easily replay them, and never lose your work.

Inductor enables you to start a playground for your documentation Q&A bot with a single CLI command. An example of such a playground is as follows:

Developers and domain experts can interactively experiment with different combinations of hyperparameters in playgrounds to rapidly prototype and collaborate:

Inductor additionally makes it easy to add active and previous work performed in playgrounds to test suites, enabling you to rapidly transition work from prototyping to systematic evaluation, and ultimately to production, by improving test suites and evaluating changes across essential points of testing and validation.

Live logging and monitoring

When your LLM program is running in production, live monitoring becomes essential for several reasons:

Ensuring intended behavior: Continuous monitoring helps confirm that your application is performing as expected and delivering useful, accurate results.
Issue detection and resolution: By keeping an eye on real-time operations, you can quickly identify and fix any emerging issues before they escalate.
Usage analysis for improvement: Understanding how users actually interact with your application enables you to gather valuable insights and make data-driven enhancements.
Feedback loop to development: Insights and issues identified via live monitoring can be fed back into the development process, enabling continuous improvement.

To facilitate this, the starter template includes the ability (via inclusion of the Inductor decorator) to use Inductor to automatically log multiple elements of the LLM application’s behavior. This automatically enables you to view and analyze many aspects of the LLM app’s behavior, including:

User input: Captures the queries inputted by users.
LLM output: Logs the responses generated by the app.
Latency: Tracks response times to measure and ensure responsiveness.
RAG system elements: Monitors components of the retrieval-augmented generation (RAG) system, such as the set of text snippets retrieved from the vector database in order to respond to any given query.

With this comprehensive live monitoring capability, you can maintain high standards of reliability, performance, and user satisfaction for your LLM applications.

‍

What next?

To get started in minutes, visit the GitHub repo, clone the documentation Q&A starter template, and follow the simple steps provided to start systematically developing your LLM application.

You can also learn more about Inductor by visiting our documentation or booking a demo.

Inductor Custom Playgrounds: A developer-first way to experiment and collaborate on LLM app development

Natalie Fagundo — Tue, 25 Jun 2024 18:48:03 +0000

The only way to build a high-quality LLM application is to get hands-on, iterate, and experiment your way to success, powered by collaboration and data; it is important to then also do rigorous evaluation.

At Inductor, we’re building the tools that developers need to do this, so that you can build and ship production-ready LLM apps far more quickly, easily, and systematically – whether you’re creating an AI chatbot, a documentation assistant, a text-to-SQL feature, or something else powered by LLMs. We’re now rolling out a new capability that we’re super-excited about: Custom Playgrounds.

Auto-generate your custom playground by signing up here!

Inductor Custom Playgrounds enable you to auto-generate a powerful, instantly shareable playground for your LLM app with a single CLI command - and run it within your environment. This makes it easy to loop other (even non-technical) team members into your development process, and also accelerate your own iteration speed.

By leveraging Custom Playgrounds, you can turbocharge your development process, reduce time to market, and create more effective LLM applications and features. Watch our demo video to learn more or read on below!

Why use Custom Playgrounds?

Inductor's Custom Playgrounds are purpose-built for developers’ needs, and offer significant advantages over traditional LLM playgrounds with respect to productivity, usability, and collaboration. Custom Playgrounds

Integrate directly with your code and auto-generate customized UI tailored to your specific LLM application.
Run directly against your environment, facilitating use of private data and internal systems.
Enable robust, secure collaboration – empowering teams to share work, collect feedback, and leverage collective expertise directly within the playground (e.g., for prompt engineering and more).
Accelerate development through features like UI auto-generation, hot-reloading, auto-logging, and integrated test suite management – streamlining the iteration process and enabling rapid prototyping and systematic evaluation.

These enhancements make Custom Playgrounds a more efficient, flexible, and powerful tool for developing and refining LLM applications or features compared to traditional LLM playgrounds and DIY interfaces.

⚠️ Creating DIY capabilities demands greater effort, entails higher risk, and results in increased long-term total cost of ownership (TCO).

Get started with one command

Simply execute a single Inductor CLI command to auto-generate a playground UI for your LLM application. Run securely in your environment, using your data, systems, and programmatic logic.

To auto-generate a playground for your LLM app, just execute the following commands in your terminal:

$ pip install inductor $ inductor playground my.module:my_function

where “my.module:my_function” is the fully qualified name of a Python function that is the entrypoint to your LLM app. (No modifications to your code required!)

If you’re building a multi-turn chat app, you can also add a single inductor.ChatSession type annotation to your LLM app’s entrypoint function before you run the playground CLI command to also get chat-specific capabilities.

See our docs for more information about how to use Custom Playgrounds.

Once generated, your playground enables you to

Instantly share your LLM app with technical and/or non-technical colleagues to collect feedback.
Interact with and evolve your LLM app (with hot-reloading, logging and replay of your interaction history, hyperparameters, and visibility into execution internals).
Easily turn your interactions into repeatable test suites for systematic evaluation.

Get started for free by signing up here or simply running the terminal commands above, to instantly generate a custom playground and turbocharge your LLM app development!

Newsletter - April Edition

Natalie Fagundo — Wed, 03 Apr 2024 21:17:16 +0000

Announcements

We’ve gone live with our new developer tool for evaluating, improving, and observing your LLM applications – both during development and in production. For teams who are serious about shipping production-ready LLM apps (i.e., that are high-quality, trustworthy, and cost-effective), we’ve built Inductor to help you do so much more rapidly, easily, and reliably.

Interested in early access to Inductor? Request access here.

‍

A note from our founder

The developer community has been building web applications for well over 25 years. During that time, we’ve created a set of highly effective practices and tools for building web apps productively, and for reliably delivering great experiences to users. This has been critical to web apps becoming a ubiquitous way in which developers (and businesses) innovate and deliver value to their users.

We as a community are now at the beginning of that journey for LLM-powered applications – a different (and much newer) class of applications that are poised to have at least as much impact as web apps, if not much more. The underlying tech (i.e., large language models) is powerful, but it remains challenging and time-consuming to build and ship LLM apps that reliably deliver high-quality, high-value experiences to users. We’ve seen too many teams struggle to do this (and have experienced this pain ourselves).

At Inductor, we’re working to solve this problem. The work required to build a production-ready LLM app differs in fundamental ways from the work of building other types of applications, such as web apps. In particular, LLM applications require iterative development driven by experimentation and evaluation, as they cannot be a priori written to guarantee desired behavior (e.g., LLMs’ inputs cannot be designed a priori to guarantee desired output behavior). The only way to build a high-quality LLM application is to iterate and experiment your way to success, powered by data and rigorous evaluation; it is essential to then also observe and understand live usage to detect issues and fuel further improvement. Today, the work of doing all of this is too often slow and painful.

We’ve been building Inductor to address this. We’re excited to say that we’ve recently gone live with our new product, and Inductor is already being used by customers to build and deliver production-ready LLM apps. And, we’re working on a whole lot more – we’ll keep you in the loop along the way.

‍

Release Notes

We’ve been busy building – below are some of the exciting new capabilities that we’ve recently added to Inductor (see our demo video for an overview of Inductor’s features beyond the below):

Use Inductor’s application-level Hyperparameters to automatically screen across different versions of your LLM app in order to rapidly determine which is best for your needs. This enables you to super-quickly and systematically test variants of any facet of your LLM app, such as different models, prompts, RAG strategies, or pre- or post-processing approaches.
Add LLM-powered quality measures to your test suites (or live traffic) to automate and scale up human-style evaluations. Inductor automatically determines the degree of alignment between your LLM evals and any corresponding human evals, in order to ensure that your LLM-powered evaluations rigorously reflect your human definitions of quality.
Use Inductor’s rich suite of sharing functionality to securely collaborate with team members or other stakeholders to get feedback and analyze results.
Run quality measures (of any type - function, human, or LLM) on live executions to automatically and continuously assess quality on your live traffic. Filter live executions by quality measures to rapidly diagnose issues and areas for improvement.

Product Demo with Inductor's Founder

Natalie Fagundo — Thu, 14 Mar 2024 18:28:36 +0000

In this quick demo video, we explain how Inductor addresses some of the most critical problems that teams face when working to ship LLM applications: evaluating, monitoring, and systematically improving LLM app quality and cost-effectiveness.

We’ve carefully crafted Inductor for LLM app developers and their teams as a combination of a CLI, API, and web UI to provide a fantastic developer experience for iterating and shipping fast. Inductor is packed with powerful features like:

Test suites
Quality measures
Hyperparameters
Human evals
LLM-powered evals
Live production monitoring
Auto-logging
Collaboration

Turbocharge experimentation with easily defined hyperparameters and inject them into your LLM app to test anything: prompts, models, RAG strategies, or anything else. Easily monitor live traffic with just one line of code – detect and resolve issues, analyze usage, and seamlessly feed back into your development process.

We’ve designed Inductor to make it easy to get started quickly. As soon as you have an account, it takes just a few minutes to create your first test suite and run it on your LLM app – with zero code modifications. To log and monitor your live traffic, simply add one line of code to your app.

If you’re interested in trying Inductor for free, request access here.

Inductor is live on Product Hunt 🚀

Natalie Fagundo — Mon, 04 Mar 2024 17:33:22 +0000

We’re excited to introduce Inductor on Product Hunt! 🎉

Check out our launch to view our new demo video, learn more about Inductor, ask questions, or provide feedback. If you like what you see, please upvote and share so that more LLM app developers can ship production-ready LLM apps far faster and more easily with Inductor.

Inductor is a developer tool for evaluating, monitoring, and improving LLM applications – to enable you to build and ship production-ready LLM applications far faster and more easily.

We’ve carefully crafted Inductor for LLM app developers as a combination of a CLI, API, and web UI to provide a fantastic developer experience for iterating and shipping fast. Inductor is packed with powerful features like:

Test suites
Quality measures
Hyperparameters
Human evals
LLM-powered evals
Live production monitoring
Auto-logging
Collaboration

Rigorously evaluate your LLM app both offline and online by rapidly applying human, programmatic, and LLM-powered quality measures. Turbocharge experimentation with easily defined hyperparameters, and use them in your LLM app to test anything: prompts, models, RAG strategies, or anything else. Easily monitor live traffic with just one line of code – detect and resolve issues, analyze usage, and seamlessly feed back into your development process.

We’ve designed Inductor to make it easy to get started quickly. As soon as you have an account, it takes just a few minutes to create your first test suite and run it on your LLM app using our CLI – with zero code modifications. To log and monitor your live traffic, simply add one line of code to your app.

If you’re interested in trying Inductor for free, request access here.

Follow us to stay in the loop! We'll be launching new features and sharing best practices for systematically shipping production-ready LLM apps.

Introducing Inductor: Ship Production-Ready LLM Apps Dramatically Faster and More Easily

Natalie Fagundo — Fri, 16 Feb 2024 17:43:04 +0000

We've built a new developer tool that makes it far easier to systematically evaluate, monitor, and improve LLM applications – both during development and in production.

We’re excited to introduce the new product that we’ve been building at Inductor! Having seen too many teams experience (and having experienced ourselves) the pain of going from an LLM-powered demo to production-ready LLM-powered application, we’ve built a developer tool that addresses the most critical problems that teams face when working to build and ship LLM applications. Read on to learn more!

Why we built Inductor

LLMs unlock fantastic new possibilities, and make it easy to create compelling demoware. But, going from an idea or demo to an LLM application that is actually production-ready (i.e., high quality, safe, cost-effective) is painful and time-consuming.

Doing so requires grappling with critical questions like

“How good (i.e., high-quality, safe, cost-effective) is our LLM application?”
“How can we improve it? Are we actually improving?”
“Is our LLM app ready to ship? Does it produce high-quality results sufficiently reliably?”
“How is our LLM app behaving on live traffic?”

In order to build and ship a production-ready LLM application, teams need to answer these questions repeatedly as they iteratively develop, as well as deploy and run live. However, today, answering these questions for LLM apps is painful and time-consuming.

Inductor solves this problem by supplying you with the right workflows and tools to answer these questions rapidly, easily, and systematically. In turn, this enables you to deliver high-quality, safe, and cost-effective LLM apps far more rapidly and easily.

With Inductor, you can

Iterate and ship faster – so that you can increase your productivity and get to market faster.

Inductor makes it easy to continuously test and evaluate as you develop, so that you always know your LLM app’s quality and how it is changing. Inductor also enables you to systematically make changes to improve quality and cost-effectiveness by rapidly testing different app variants and actionably analyzing your LLM app’s behavior.

Reliably deliver high-quality results – so that you can ship confidently and safely.

Inductor enables you to rigorously assess your LLM app’s behavior before you deploy, in order to ensure quality and cost-effectiveness when you’re live. You can then use Inductor to easily monitor your live traffic to detect and resolve issues, analyze actual usage in order to improve, and seamlessly feed back into your development process.

Collaborate easily and efficiently – so that you can loop in your team to help build and ship.

Inductor makes it easy for engineering and other roles to collaborate, for example to get critical human feedback from non-engineering stakeholders (e.g., PM, UX, or subject matter experts) to ensure that your LLM app is user-ready. With Inductor, you no longer need to pass around unwieldy spreadsheets that rapidly become outdated.

Carefully crafted for developers and their teams

Teams need to go through three key workflows in order to build and ship production-ready (i.e., high-quality, safe, cost-effective) LLM applications:

Construct, run, and evolve test suites – in order to systematically assess your LLM app’s quality, safety, and cost-effectiveness.

Iteratively experiment and analyze results– in order to actionably understand your LLM app’s behavior, systematically improve, and rigorously decide when you are ready to ship.

Monitor in production and feed back to dev – in order to understand how your users are actually using your LLM app, address any issues, determine where you need to improve, and feed these insights back into your development process.

We’ve carefully crafted Inductor to make it easy and seamless to do these things in your existing development environment, by using Inductor’s CLI, API, and web UI. Inductor is packed with a powerful, easy-to-use set of capabilities purpose-built for the needs of teams working to build and ship LLM apps:

Test suites

Inductor’s test suites make systematic, continuous evaluation easy – both during iterative development, and prior to deployment (to ensure quality and cost-effectiveness when you’re live). Define test cases and quality measures easily via YAML or our Python API. Easily span the full spectrum of automated to human evaluation by using Inductor’s programmatic, human, or LLM-powered quality measures – with a great workflow for ensuring that your LLM-powered quality measures are rigorously calibrated to your human-defined evaluation criteria. Run test suites via the Inductor CLI or API, and actionably analyze the results via our purpose-built web UI.

Hyperparameters

Massively accelerate your iteration by including hyperparameters in your test suites to automatically test different variants of your app. Inductor enables you to easily define and inject hyperparameters into your LLM app to test anything - for example, prompts, models, RAG strategies, or anything else. Inductor then takes care of running every test case against every combination of your hyperparameter values and making the results actionable. You will very soon also be able to include hyperparameters in your live deployments in order to run and analyze live A/B tests.

Live production monitoring

When your LLM app goes live, just add a single line of code to enable logging and monitoring your live traffic using Inductor. Inductor then automatically logs and enables you to richly analyze your live traffic, in order to detect and resolve any issues, analyze live usage in order to improve, and seamlessly feed back into your development process.

Actionable analytics

Whether you are analyzing the results of a test suite run or examining your live traffic, Inductor makes it easy to rapidly extract actionable insights. Use Inductor’s web UI to view summary statistics for your LLM app’s performance as well as examine individual LLM app executions (whether within a test suite or in live traffic), including the details of any specific execution and evaluations of its quality. Use Inductor’s rich filtering capabilities to identify and diagnose areas for improvement. Easily compare the performance of different versions of your LLM app or different app variants parameterized by hyperparameters in order to systematically improve.

Easy, secure collaboration

Collaboration is often critical to building and shipping production-ready LLM apps. For example, it is often essential to get feedback from non-engineering stakeholders (e.g., PM, UX, or subject matter experts), or to collaboratively analyze test suite results or live traffic. Inductor’s secure sharing and permissioning capabilities enable you to easily share, get feedback, and work together using Inductor – with both technical and non-technical team members.

Automatic versioning and auto-logging

Inductor is powered by a data model and platform that is purpose-built for the workflows required to build and ship production-ready LLM applications. Inductor automatically tracks and versions all of your work (e.g., including all test cases, quality measures, and hyperparameters in every test suite), and provides a variety of auto-logging capabilities, so that you can iterate at the speed of thought while staying organized by default. Because Inductor enables you to define your test suites alongside your code (e.g., via YAML or our API), you can also version your test suites within any existing version control system that you use (e.g., via git).

Get started quickly

We’ve designed Inductor to make it easy to get started fast. As soon as you have an account, it takes just a few minutes to create your first test suite and run it on your LLM app – with zero code modifications. To then also use Inductor to log and monitor your live traffic, you only need to add one line of code to your app.

Inductor is built to work seamlessly in your existing development environment by using our CLI and Python API – whether you work in an IDE such as VS Code, use a text editor and terminal, or work in a notebook environment such as Jupyter or Google Colab. Inductor is also designed to work well with your existing tech stack, including any model (whether you are using an LLM API, an open source model, or your own model) and any way of writing LLM apps (from straight-up Python to LangChain and beyond). The Inductor service is architected from the ground up to be runnable in our cloud account or yours; we offer the option to self-host if desired.

Finally, we’ve also designed our pricing to make it easy to get started – we offer straightforward, developer-friendly, per-user-per-month pricing.

Interested in using Inductor?

Request access here, and follow us as we release new features and content!