DEV Community: Will Taner

Evaluating LLMs and Prompts with Electron UI 🤖 💬

Will Taner — Wed, 07 Aug 2024 13:02:07 +0000

What is this UI useful for? 🤨

LLMs are becoming an increasingly prevalent tool across various industries. However, achieving optimal results greatly depends on selecting the correct model and prompts. This process can be extremely time-consuming as it requires extensive trial and error.

This article serves as a tutorial on a tool designed to streamline the process of testing various models and prompts. By using this tool, developers can efficiently identify the most effective combinations.

Big shoutout to Kevin Brisson for creating this Electron UI! You may find the GitHub Repo for his tool here: https://github.com/kbrisso/ai-base

Note: The CLI commands provided in this article are designed for Linux/Mac systems and may not function correctly on Windows machines. If you encounter any issues, please replace the incompatible commands with their Windows equivalents.

Step-by-Step Guide on Getting the Tool up and Running!

(If you would like to watch a video demonstrating the setup, click here)

1. Let's Start by Cloning the Repo 💻

Navigate to a directory of your choosing and run git clone https://github.com/kbrisso/ai-base in your command line.

2. Root Directory Installs 📦

Navigate to the root directory, ai-base, in your command line and run npm install. If you experience an error, run npm audit fix. This will resolve any vulnerabilities in the packages.

3. llmware-wrapper Installs 📦

Navigate to the directory llmware-wrapper in your command line and run the same exact command from above: npm install and npm audit fix if there is an error.

4. Create a Virtual Environment and Install Packages 💾

While in the same directory as above, llmware-wrapper, create a new virtual environment: python3 -m venv venv.

Then enter into the virtual environment by running source venv/bin/activate in the command line.

Then run pip install -r requirements.txt to install the required packages.

Finally, deactivate the virtual environment by running deactivate

5. Copy Local Python Path 🐍

On the file explorer on your IDE, open the file titled llmware-wrapper.properties in the llmware-wrapper directory.

In this file, delete the path that the variable pythonpath is already set to and replace it with the path to your local python interpreter.

If you do not know what your local path is, run which python in the command line then copy and paste the result where you deleted the previous path.

Let's start the UI! 🚀

Navigate to the root directory, ai-base, in your command line and run npm start. After a few seconds a window of the UI should pop up.

Selecting a Model and Prompt 💡

Click on the button that says "Choose a Model". This will display all the available models provided by LLMWare. Once you find a model that you would like to try, click the button that says "Choose" next to the model name.

Click on the button that says "Choose a Prompt". This will display all the available types of prompts you can choose from provided by LLMWare. Additionally, you may find supplemental information about the prompt type, such as a description, to the right of the prompt name. Once you find a prompt type that you would like to try, click the button that says "Choose" next to the prompt name.

Putting it all together... 🧩

After selecting a model and a prompt type, it is time to start querying! Simply add your query to the box labeled "Query" and click the button that says "Run Query". After some time, the response will show up in the box titled "Response".

Note: Depending on the prompt type chosen, an extra box for context will appear. You may use this space to provide relevant details for your query if needed.

Now you can experiment with different models and queries with ease!

Conclusion 🏁

Finding the right combination of models and prompts are crucial to creating a reliable and effective LLM tool. Utilizing this UI, you can find the best combination faster than ever before!

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware

Please be sure to visit our website llmware.ai for more information and updates

The Hardest Problem in RAG... Handling 'NOT FOUND' Answers 🔍🤔

Will Taner — Mon, 24 Jun 2024 17:04:03 +0000

First of All... What is RAG? 🕵️‍♂️

Retrieval-Augmented Generation (RAG) is an approach to natural language processing that references external documents to provide more accurate and contextually relevant answers. Despite its advantages, RAG faces some challenges, one of which is handling 'NOT FOUND' answers. Addressing this issue is crucial for developing an effective and reliable model that everyone can use.

Why 'NOT FOUND' Answers Can Be Concerning ⛔️

Some models respond with "hallucinations" when they cannot find an answer, creating inaccurate responses that may mislead the user. This can undermine the trust users have in the model, making it less reliable and effective.

How Can We Remedy This? 🛠️

For starters, it is better for the model to inform the user that it could not find the answer rather than fabricating one.

Next, we will delve into one way LLMWare handles 'NOT FOUND' cases effectively. By examining these methods, we can gain a better understanding of how to address this issue and enhance the overall performance and reliability of RAG systems.

For the Visual Learners... 📺

Here is a video discussing the same topic as this article. A good idea would be to watch the video, and then work through the steps in this article.

Framework 🖼️

LLMWare
For our new readers, LLMWARE is a comprehensive, open-source framework that provides a unified platform for application patterns based on LLMs, including Retrieval Augmented Generation (RAG).

Please run pip3 install llmware in the command line to download the package.

Import Libraries and Create Context 📚

from llmware.models import ModelCatalog
from llmware.parsers import WikiParser

ModelCatalog: A class within llmware that manages selecting the desired model, loading the model, and configuring the model.

WikiParser: A class within llmware that handles the retrieval and packaging of content from Wikipedia.

text =("BEAVERTON, Ore.--(BUSINESS WIRE)--NIKE, Inc. (NYSE:NKE) today reported fiscal 2024 financial results for its "
      "third quarter ended February 29, 2024.) “We are making the necessary adjustments to drive NIKE’s next chapter "
      "of growth Post this Third quarter revenues were slightly up on both a reported and currency-neutral basis* "
      "at $12.4 billion NIKE Direct revenues were $5.4 billion, slightly up on a reported and currency-neutral basis "
      "NIKE Brand Digital sales decreased 3 percent on a reported basis and 4 percent on a currency-neutral basis "
      "Wholesale revenues were $6.6 billion, up 3 percent on a reported and currency-neutral basis Gross margin "
      "increased 150 basis points to 44.8 percent, including a detriment of 50 basis points due to restructuring charges "
      "Selling and administrative expense increased 7 percent to $4.2 billion, including $340 million of restructuring "
      "charges Diluted earnings per share was $0.77, including $0.21 of restructuring charges. Excluding these "
      "charges, Diluted earnings per share would have been $0.98* “We are making the necessary adjustments to "
      "drive NIKE’s next chapter of growth,” said John Donahoe, President & CEO, NIKE, Inc. “We’re encouraged by "
      "the progress we’ve seen, as we build a multiyear cycle of new innovation, sharpen our brand storytelling and "
      "work with our wholesale partners to elevate and grow the marketplace.")

Here is the initial text for our extraction. It provides details about the popular sports brand, Nike. Feel free to modify this text to suit your needs.

Create Key for Extraction 🔐

extract_key = "company founding date"
dict_key = extract_key.replace(" ", "_")

company_founding_date = ""

Here, we set the company founding date as the target extraction from the text.

Run Initial Extract 🏃

model = ModelCatalog().load_model("slim-extract-tool", temperature=0.0, sample=False)
response = model.function_call(text, function="extract", params=[extract_key])
llm_response = response["llm_response"]

Model: In this snippet, we load LLMWare's slim-extract-tool, which is a 2.8B parameter GGUF model that is fine tuned for general-purpose extraction (GGUF is a quantization method that allows for quicker inference time and decreased model size at the cost of accuracy).

Temperature: This controls the randomness of the output. Valid values range between 0 and 1, where lower values make the model more deterministic, and higher values make the model more random and creative.

Sample: Determines if the output is generated deterministically or probabilistically. False generates deterministic output. True generates probabilistic output.

We then attempt to extract the information from the text using the model and store it in llm_response.

If Answer is Found... ✅

if dict_key in llm_response:

        company_founding_date = llm_response[dict_key]

        if len(company_founding_date) > 0:

            company_founding_date = company_founding_date[0]
            print(f"update: found the {extract_key} value - ", company_founding_date)
            return company_founding_date

If the model successfully finds and extracts the company founding date, we will return the information.

If Answer is Not Found... ❌

else:

    print(f"update: did not find the target value in the text - {company_founding_date}")
    print("update: initiating a secondary process to try to find the information")

    response = model.function_call(text, function="extract", params=["company name"])

If the model does not find the company founding date, we will run a second query to find the company name for future use in gathering more information.

Retrieve Information from Wiki 📖

if "company_name" in response["llm_response"]:
    company_name = response["llm_response"]["company_name"][0]

    if company_name:
        print(f"\nupdate: found the company name - {company_name} - now using to lookup in secondary source")

        output = WikiParser().add_wiki_topic(company_name,target_results=1)

After extracting the company's name from the text, we will then retrieve additional information about the company from Wiki.

Generate a Summary Snippet from Retrieved Article Data ✍️

if output:

    supplemental_text = output["articles"][0]["summary"]

    if len(supplemental_text) > 150:
        supplemental_text_pp = supplemental_text[0:150] + " ... "
    else:
        supplemental_text_pp = supplemental_text

    print(f"update: using lookup - {company_name} - found secondary source article "
                              f"(extract displayed) - ", supplemental_text_pp)

If we have successfully retrieved additional data from the Wiki, we truncate the response if it is over 150 characters and set supplemental_text_pp to

Call Extract Again With New Information 📞

new_response = model.function_call(supplemental_text,params=["company founding date"])

print("\nupdate: reviewed second source article - ", new_response["llm_response"])

Using the new information retrieved from Wiki, we run the same extraction on the model again.

Print Response If Found 🖨️

if "company_founding_date" in new_response["llm_response"]:
    company_founding_date = new_response["llm_response"]["company_founding_date"]
    if company_founding_date:
        print("update: success - found the answer - ", company_founding_date)

If we find the company founding date after incorporating the new information, we print the result.

Fully Integrated Code 🧑‍💻

from llmware.models import ModelCatalog
from llmware.parsers import WikiParser

text =("BEAVERTON, Ore.--(BUSINESS WIRE)--NIKE, Inc. (NYSE:NKE) today reported fiscal 2024 financial results for its "
      "third quarter ended February 29, 2024.) “We are making the necessary adjustments to drive NIKE’s next chapter "
      "of growth Post this Third quarter revenues were slightly up on both a reported and currency-neutral basis* "
      "at $12.4 billion NIKE Direct revenues were $5.4 billion, slightly up on a reported and currency-neutral basis "
      "NIKE Brand Digital sales decreased 3 percent on a reported basis and 4 percent on a currency-neutral basis "
      "Wholesale revenues were $6.6 billion, up 3 percent on a reported and currency-neutral basis Gross margin "
      "increased 150 basis points to 44.8 percent, including a detriment of 50 basis points due to restructuring charges "
      "Selling and administrative expense increased 7 percent to $4.2 billion, including $340 million of restructuring "
      "charges Diluted earnings per share was $0.77, including $0.21 of restructuring charges. Excluding these "
      "charges, Diluted earnings per share would have been $0.98* “We are making the necessary adjustments to "
      "drive NIKE’s next chapter of growth,” said John Donahoe, President & CEO, NIKE, Inc. “We’re encouraged by "
      "the progress we’ve seen, as we build a multiyear cycle of new innovation, sharpen our brand storytelling and "
      "work with our wholesale partners to elevate and grow the marketplace.")


def not_found_then_triage_lookup():

    print("\nNot Found Example - if info not found, then lookup in another source.\n")

    extract_key = "company founding date"
    dict_key = extract_key.replace(" ", "_")

    company_founding_date = ""

    model = ModelCatalog().load_model("slim-extract-tool", temperature=0.0, sample=False)
    response = model.function_call(text, function="extract", params=[extract_key])
    llm_response = response["llm_response"]

    print(f"update: first text reviewed for {extract_key} - llm response: ", llm_response)

    if dict_key in llm_response:

        company_founding_date = llm_response[dict_key]

        if len(company_founding_date) > 0:

            company_founding_date = company_founding_date[0]
            print(f"update: found the {extract_key} value - ", company_founding_date)
            return company_founding_date

        else:

            print(f"update: did not find the target value in the text - {company_founding_date}")
            print("update: initiating a secondary process to try to find the information")

            response = model.function_call(text, function="extract", params=["company name"])

            if "company_name" in response["llm_response"]:
                company_name = response["llm_response"]["company_name"][0]

                if company_name:
                    print(f"\nupdate: found the company name - {company_name} - now using to lookup in secondary source")

                    output = WikiParser().add_wiki_topic(company_name,target_results=1)

                    if output:

                        supplemental_text = output["articles"][0]["summary"]

                        if len(supplemental_text) > 150:
                            supplemental_text_pp = supplemental_text[0:150] + " ... "
                        else:
                            supplemental_text_pp = supplemental_text

                        print(f"update: using lookup - {company_name} - found secondary source article "
                              f"(extract displayed) - ", supplemental_text_pp)

                        new_response = model.function_call(supplemental_text,params=["company founding date"])

                        print("\nupdate: reviewed second source article - ", new_response["llm_response"])

                        if "company_founding_date" in new_response["llm_response"]:
                            company_founding_date = new_response["llm_response"]["company_founding_date"]
                            if company_founding_date:
                                print("update: success - found the answer - ", company_founding_date)

    return company_founding_date


if __name__ == "__main__":

    founding_date = not_found_then_triage_lookup()

You may also find the fully integrated code on our github here

Additionally, the notebook version (ipynb) is available here

Conclusion 🤖

Handling 'NOT FOUND' answers is one of the hardest problems in RAG, but it's a challenge that can be mitigated with thoughtful design. By implementing techniques like broader lookups, LLMWare aims to enhance the overall user experience and reliability of its AI systems.

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware

Please be sure to visit our website llmware.ai for more information and updates.

Sentiment Analysis using CPU-Friendly Small Language Models 😡 😃

Will Taner — Mon, 17 Jun 2024 13:06:59 +0000

What are Small Language Models? 🤔

You have probably heard of Large Language Models, but what about Small Language Models (SLMs)? These compact models excel on specialized tasks, are ideal to use with limited resources, and have faster processing times. Let's explore LLMWare's SLIM models!

Harnessing SLIM Models 💡

In many real-world automated workflows, accomplishing a task requires multiple steps. LLMWare's SLIM models are specifically designed to streamline these processes. These models integrate seamlessly into your programmatic environment, acting as decision points within complex, multistep workflows. They achieve this by producing structured outputs like JSON, SQL, or Python dictionaries, ensuring smooth and efficient workflow automation.

Additionally, these models are designed to be CPU-friendly, making them accessible to many machines. Let's explore how LLMWare's slim-sentiment-tool, equipped with 1.1 billion parameters, can enhance our workflow in sentiment analysis with two practical examples!

For the Visual Learners... 📺

Here is a video discussing the same topic as this article. A good idea would be to watch the video, and then work through the steps in this article.

Framework 🖼️

Please run pip3 install llmware in the command line to download the package.

Importing Libraries and Creating Transcripts 📚

from llmware.agents import LLMfx

LLMfx: A class within llmware that provides an interface to models to interact with text, e.g. to first perform named entity recognition (ner) and then answer a question you want to have answered. It orchestrates multi-model, multi-step processes using SLIM classifier models with centralized journaling, structured work management, and information aggregation.

earnings_transcripts = [
    "This is one of the best quarters we can remember for the industrial sector with significant growth across the "
    "board in new order volume, as well as price increases in excess of inflation.  We continue to see very strong "
    "demand, especially in Asia and Europe. Accordingly, we remain bullish on the tier 1 suppliers and would be "
    "accumulating more stock on any dips. ",

    "Not the worst results, but overall we view as negative signals on the direction of the economy, and the likely "
    "short-term trajectory for the telecom sector, and especially larger market leaders, including AT&T, Comcast, and"
    "Deutsche Telekom.",

    "This quarter was a disaster for Tesla, with falling order volume, increased costs and supply, and negative "
    "guidance for future growth forecasts in 2024 and beyond.",

    "On balance, this was an average result, with earnings in line with expectations and no big surprises to either "
    "the positive or the negative."
    ]

Here, we instantiate the transcripts we will be using for sentiment analysis. Feel free to replace it with the data you would like to analyze.

Example 1: Simple Programmatic Use of Sentiment Classification

Load the Agent 🤖

agent = LLMfx(verbose=True)
agent.load_tool("sentiment")
sentiment = agent.sentiment(text)

This code will create the agent using LLMfx, load the sentiment analysis tool, and utilize the sentiment analysis tool to make an inference on the text's sentiment.

Verbose: This parameter determines if the events written in the journal are also written in stdout. This is defaulted to True.

Print the Output 🖨️

print("sentiment: ", sentiment)
for keys, values in sentiment.items():
    print(f"{keys}-{values}")

This will print the inferred sentiment, the confidence in that sentiment, and the probability of all three of the emotions.

Make a Decision Based on Sentiment ⚖️

sentiment_value = sentiment["llm_response"]["sentiment"]
confidence_level = sentiment["confidence_score"]

if "positive" in sentiment_value:
    print("sentiment is positive .... will take 'positive' analysis path ...", sentiment_value)

if "positive" in sentiment_value and confidence_level > 0.8:
    print("sentiment is positive with high confidence ... ", sentiment_value, confidence_level)

return sentiment

This code extracts and stores the sentiment in the variable sentiment_value. Next, it retrieves the sentiment's confidence score and stores it in the variable confidence_level. The process then goes through two decision points: one if the sentiment is positive, and another if the sentiment is positive with a confidence level greater than 0.8.

Complete Code for First Example ✅

def get_one_sentiment_classification(text):

    agent = LLMfx(verbose=True)
    agent.load_tool("sentiment")
    sentiment = agent.sentiment(text)

    print("sentiment: ", sentiment)
    for keys, values in sentiment.items():
        print(f"{keys}-{values}")

    sentiment_value = sentiment["llm_response"]["sentiment"]
    confidence_level = sentiment["confidence_score"]

    if "positive" in sentiment_value:
        print("sentiment is positive .... will take 'positive' analysis path ...", sentiment_value)

    if "positive" in sentiment_value and confidence_level > 0.8:
        print("sentiment is positive with high confidence ... ", sentiment_value, confidence_level)

    return sentiment

Example 2: Batch Sentiment Analysis with Iteration

Load the Agent and Work 🤖

agent = LLMfx()
agent.load_tool("sentiment")
agent.load_work(earnings_transcripts)

The first two lines are the same as in the first example, load the agent and the sentiment analysis tool. In this example, we will load the LLMfx load_work method. The load_work method is a flexible input mechanism - pass a string, list, dictionary, or combination, and it will 'package' as iterable units of processing work for the agent.

Iterate Through Transcripts and Print Results 🔁

while True:
    output = agent.sentiment()
    # print("update: test - output - ", output)
    if not agent.increment_work_iteration():
        break

This code iterates through the transcripts and performs sentiment analysis on each one. For each iteration, it prints the sentiment, confidence score, and probability distribution of all three emotions. The loop terminates when the increment_work_iteration method of the agent returns False.

Output Response and Clear Work and State 📋

response_output = agent.response_list

agent.clear_work()
agent.clear_state()

return response_output

The response_output variable holds a dictionary containing all the sentiment analysis responses to the transcripts. After gathering these responses, we clear the agent's work queue and key state variables.

Complete Code for Second Example ✅

def review_batch_earning_transcripts():

    agent = LLMfx()
    agent.load_tool("sentiment")

    agent.load_work(earnings_transcripts)

    while True:
        output = agent.sentiment()
        # print("update: test - output - ", output)
        if not agent.increment_work_iteration():
            break

    response_output = agent.response_list

    agent.clear_work()
    agent.clear_state()

    return response_output

Main Block 🔌

if __name__ == "__main__":

    sentiment = get_one_sentiment_classification(earnings_transcripts[0])

    response_output = review_batch_earning_transcripts()

In our main block, we will run both of these examples. In the first run, we will only analyze the sentiment of the first element of earnings_transcripts.

Fully Integrated Code with Example 1 and 2 🧩


from llmware.agents import LLMfx

earnings_transcripts = [
    "This is one of the best quarters we can remember for the industrial sector with significant growth across the "
    "board in new order volume, as well as price increases in excess of inflation.  We continue to see very strong "
    "demand, especially in Asia and Europe. Accordingly, we remain bullish on the tier 1 suppliers and would be "
    "accumulating more stock on any dips. ",

    "Not the worst results, but overall we view as negative signals on the direction of the economy, and the likely "
    "short-term trajectory for the telecom sector, and especially larger market leaders, including AT&T, Comcast, and"
    "Deutsche Telekom.",

    "This quarter was a disaster for Tesla, with falling order volume, increased costs and supply, and negative "
    "guidance for future growth forecasts in 2024 and beyond.",

    "On balance, this was an average result, with earnings in line with expectations and no big surprises to either "
    "the positive or the negative."
    ]


def get_one_sentiment_classification(text):

    agent = LLMfx(verbose=True)
    agent.load_tool("sentiment")
    sentiment = agent.sentiment(text)

    print("sentiment: ", sentiment)
    for keys, values in sentiment.items():
        print(f"{keys}-{values}")

    sentiment_value = sentiment["llm_response"]["sentiment"]
    confidence_level = sentiment["confidence_score"]

    if "positive" in sentiment_value:
        print("sentiment is positive .... will take 'positive' analysis path ...", sentiment_value)

    if "positive" in sentiment_value and confidence_level > 0.8:
        print("sentiment is positive with high confidence ... ", sentiment_value, confidence_level)

    return sentiment


def review_batch_earning_transcripts():

    agent = LLMfx()
    agent.load_tool("sentiment")

    agent.load_work(earnings_transcripts)

    while True:
        output = agent.sentiment()
        # print("update: test - output - ", output)
        if not agent.increment_work_iteration():
            break

    response_output = agent.response_list

    agent.clear_work()
    agent.clear_state()

    return response_output


if __name__ == "__main__":

    sentiment = get_one_sentiment_classification(earnings_transcripts[0])

    response_output = review_batch_earning_transcripts()

You may also find the fully integrated code on our Github repo here

Additionally, the notebook version (ipynb) is available here

Conclusion 🏁

We've shown how LLMWare's SLIM models can be effectively utilized for sentiment analysis. The model's simplicity and powerful capabilities make it an excellent tool for analyzing sentiment and simplifying the multistep process of sentiment analysis.

Thank you for exploring this topic with us. We trust you now have the understanding needed to implement LLMWare's SLIM models for sentiment analysis and leverage its full range of benefits.

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware

Please be sure to visit our website llmware.ai for more information and updates.

Create Phi-3 Chatbot with 20 Lines of Code (Runs Without Wifi) 🚀 🤖

Will Taner — Thu, 06 Jun 2024 16:21:10 +0000

Made in collaboration with Andrew Dang

The Power of Local Chatbots

Chatbots are fun and powerful tools. They excel at organization, summarization, and conversation. The versatility of chatbots makes platforms like ChatGPT popular. However, these platforms train their chatbots on user data, which comes at the cost of user privacy. Hence the appeal of local large language models (LLMs). Users can deploy local LLMs without the need for an internet connection and on consumer hardware. Stick around to learn how to deploy a local chatbot in just 20 lines of code!

Let's get started 🔥

This video will introduce you to the capabilities of the Chatbot and provide an overview of how it is built.

Framework 🖼️

LLMWare

For our new readers, LLMWARE is a comprehensive, open-source framework that provides a unified platform for application patterns based on LLMs, including Retrieval Augmented Generation (RAG).

Streamlit

Streamlit is an open-source Python library that allows developers to create interactive web applications quickly and easily. It's designed for machine learning and data science projects, enabling users to visualize data, run models, and share results in a user-friendly interface.

Please run pip install llmware and pip install streamlit in the command line to download these packages.

Importing Libraries and Configurations 📚

import streamlit as st
from llmware.models import ModelCatalog
from llmware.gguf_configs import GGUFConfigs

GGUFConfigs().set_config("max_output_tokens", 500)

Streamlit(st): Used for creating web applications. It provides functions to display content and handle user interactions.

ModelCatalog: A component of llmware that manages selecting the desired model, loading the model, and configuring the model using the parameters in the Model Loading section below.

GGUFConfigs: A component of llmware that handles global (applies to all currently loaded models) configurations for models, such as output token limits. You may find the full list of global configurations here in the variable _conf_libs under the GGUFConfigs class.

Model Loading 🪫🔋

model = ModelCatalog().load_model(model_name, temperature=0.3, sample=True, max_output=450)

Model Name: The name of the model you want to load from ModelCatalog.

Sample: Determines if the output is generated deterministically or probabilistically. False generates deterministic output. True generates probabilistic output.

Max Output: Specifies the max length of the generated output (ex. if this is set to 100, there will be at most 100 words in the output).

Session State Management 🧮

Manages chat history within the session state of Streamlit, ensuring that previous messages are preserved across reruns of the app.

Ensure there is a list called "messages" in session state

if "messages" not in st.session_state:
   st.session_state.messages = []

Display Chat History

role is either "user" or "assistant" and will retrieve the corresponding message.

for message in st.session_state.messages:
   with st.chat_message(message["role"]):
       st.markdown(message["content"])

Prompt the User and Risplay Responses

If the user enters text, display their text

prompt = st.chat_input("Say something")
if prompt:

   with st.chat_message("user"):
      st.markdown(prompt)

Generate and Display Responses

   with st.chat_message("assistant"):
      bot_response = st.write_stream(model.stream(prompt))

Update Session State

Updates the session state by appending the new user input and the assistant's response to the chat history.

   st.session_state.messages.append({"role": "user", "content": prompt})
   st.session_state.messages.append({"role": "assistant", "content": bot_response})

Main Block ⚙️

The typical Python idiom to ensure that the script runs only when it is executed as the main program, not when imported as a module.

Initializes the application with a predefined list of model names and starts the chat application with the first model in the list.

This script is a straightforward example of integrating AI models into a web application for interactive purposes, showcasing the use of Streamlit for UI and llmware for backend model handling.

if __name__ == "__main__":

   chat_models = ["phi-3-gguf",
               "llama-2-7b-chat-gguf",
               "llama-3-instruct-bartowski-gguf",
               "openhermes-mistral-7b-gguf",
               "zephyr-7b-gguf",
               "tiny-llama-chat-gguf"]

   model_name = chat_models[0]
   simple_chat_ui_app(model_name)

Model List: This list contains the identifiers of various chat models available through the llmware library. These models are pre-trained and configured for generating conversational responses.

Purpose: At runtime, one of these models is selected to power the chat interface. This allows for flexibility in choosing different models based on their capabilities or performance characteristics.

Fully Integrated Code 📄

To run, go to command line ->
streamlit run "path/to/gguf_streaming_chatbot.py"

import streamlit as st
from llmware.models import ModelCatalog
from llmware.gguf_configs import GGUFConfigs

GGUFConfigs().set_config("max_output_tokens", 500)


def simple_chat_ui_app (model_name):

    st.title(f"Simple Chat with {model_name}")

    model = ModelCatalog().load_model(model_name, temperature=0.3, sample=True, max_output=450)

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

    prompt = st.chat_input("Say something")
    if prompt:

        with st.chat_message("user"):
            st.markdown(prompt)

        with st.chat_message("assistant"):

            bot_response = st.write_stream(model.stream(prompt))

        st.session_state.messages.append({"role": "user", "content": prompt})
        st.session_state.messages.append({"role": "assistant", "content": bot_response})

    return 0


if __name__ == "__main__":

    chat_models = ["phi-3-gguf",
                   "llama-2-7b-chat-gguf",
                   "llama-3-instruct-bartowski-gguf",
                   "openhermes-mistral-7b-gguf",
                   "zephyr-7b-gguf",
                   "tiny-llama-chat-gguf"]

    model_name = chat_models[0]

    simple_chat_ui_app(model_name)

You may also find the fully integrated code on our Github repo here

Conclusion 🔒

As we have demonstrated, deploying a local chatbot can be achieved with just 20 lines of code. This simplicity, combined with the enhanced privacy and control, makes local LLMs an attractive option for users seeking powerful and secure chatbot solutions.

Thank you for exploring this topic with us. We hope you are now equipped with the knowledge to deploy your own local chatbot and harness its full potential.

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware

Please be sure to visit our website llmware.ai for more information and updates.