DEV Community: Prashant Iyer

🤖Dueling AIs: Questioning and Answering with Language Models🚀

Prashant Iyer — Sun, 28 Jul 2024 20:36:55 +0000

You've probably asked a question to a language model before and then had it give you an answer. After all, this is what we most commonly use language models for.

But have you ever received a question from a language model? While not as common, this application of AI has diverse use cases in areas like education, where you might want a model to give you practice questions for a test, and in sales enablement, where you question your business's sales team about your products to improve their ability to make sales.

Now, what if we had a face off⚔️ between two different models: one that asked questions about a topic and another that answered them? All without human intervention?

In this article, we're going to look at exactly that. We'll provide a sample passage about OpenAI's AI safety team as context to our models. We'll then let our models duel it out! One model will ask questions based on this passage, and another model will respond!

Our AI Models🤖

Intoducing, slim-q-gen-tiny-tool. This will be our question model, capable of generating 3 different types of questions:

Multiple choice questions
Boolean (true/false) questions
General open-ended questions

Facing off against this will be bling-phi-3-gguf! This will be our answer model, giving appropriate responses to any of the above types of questions.

One important note is that both these models are GGUF quantized. This means that they are smaller and faster versions of their original counterparts. What this means for us is that we can run them on just a CPU, with no need for resources like GPUs!

Step 1: Providing input parameters✏️

This is what our function signature for this example looks like.

def ask_and_answer_game(source_passage, q_model="slim-q-gen-tiny-tool", number_of_tries=10, question_type="question",
                        temperature=0.5):

source_passage is the text input that we will provide our models,
q_model is our questioning model,
number_of_tries is the number of questions we will attempt to generate (more on this later!)
question_type can be either "multiple choice", "boolean" or "question" corresponding to each of the types of questions we saw above,
temperature is a value ranging from 0 to 1 that determines how much variance we will see in our generated questions. Here, the value of 0.5 is relatively high so that we get a good variety of questions with little repetition!

Step 2: Loading in our models🪫🔋

With the inputs taken care of, let's now load in both our models.

q_model = ModelCatalog().load_model(q_model, sample=True, temperature=temperature)

Notice that we have sample=True to increase variety in our model output (the questions generated).

Now, for the answer model.

answer_model = ModelCatalog().load_model("bling-phi-3-gguf")

We won't mess with the sample or temperature options here because we want concise, fact-based answers from this model.

Step 3: Generating our questions🤔💬

We'll try to generate questions number_of_tries times, which in this case is 10. We'll then then update our questions list with only the unique questions, to avoid repetitions.

questions = []

# Loop number_of_tries times
for x in range(0, number_of_tries):
    response = q_model.function_call(source_passage, params=[question_type])
    new_q = response["llm_response"]["question"]

    # Check to see that the question generated is unique
    if new_q and new_q not in questions:
        questions.append(new_q)

An important function here is q_model.function_call(). This is how the llmware library lets you prompt language models with just a single function call. Here, we pass in the source text and question type as arguments.

The function returns response, a dictionary with a lot of information about the call, but we're only interested in the question key, which is located inside the llm_response dictionary.

Step 4: Responding to our questions📝

Now that the questions have been generated, the duel is on! Let's use our answering model to now respond to these questions. We'll loop through our questions list, pass in the source passage as context to the model and ask each question.

# Loop through each question
for i, question in enumerate(questions):
    # Print out the question
    print(f"\nquestion: {i} - {question}")

    # Validate the question list and run inference
    if isinstance(question, list) and len(question) > 0:
        response = answer_model.inference(question[0], add_context=test_passage)

        # Print out the answer
        print(f"response: ", response["llm_response"])

It is important to note that our question model returns each question as a list, with the first element (question[0]) containing the actual string corresponding to the question.

For each question, we then need to perform some validation:

Check to see that the question is of the correct data type (list)
Check to see that the question is not empty.

Then, the answer_model.inference() function will ask our model the question, passing in the test_passage as context.

Finally, we print out the response.

Results!✅

Let's quickly look at our sample passage. This passage was taken from a CNBC news story in May 2024 about OpenAI's work with safety and security.

"OpenAI said Tuesday it has established a new committee to make recommendations to the company’s board about safety and security, weeks after dissolving a team focused on AI safety. In a blog post, OpenAI said the new committee would be led by CEO Sam Altman as well as Bret Taylor, the company’s board chair, and board member Nicole Seligman. The announcement follows the high-profile exit this month of an OpenAI executive focused on safety, Jan Leike. Leike resigned from OpenAI leveling criticisms that the company had under-invested in AI safety work and that tensions with OpenAI’s leadership had reached a breaking point."

Now, let's see what our output looks like!

We can see all the questions that were asked about the passage, as well as concise, fact-based responses given to them!

Note that there are only 9 questions here while we provided number_of_tries=10. This means that one question generated was a duplicate and was ignored.

Conclusion

And with that, we're done with this example! Recall that we used the llmware library to:

Load in a question and answer model
Generate unique questions about a source passage
Respond to each question with accuracy.

And remember that we did all of this on just a CPU! 💻

Check out our YouTube videon on this example!

If you made it this far, thank you for taking the time to go through this topic with us ❤️! For more content like this, make sure to visit our dev.to page.

The source code for many more examples like this one are on our GitHub. Find this example here.

Our repository also contains a notebook for this example that you can run yourself using Google Colab, Jupyter or any other platform that supports .ipynb notebooks.

Join our Discord to interact with a growing community of AI enthusiasts of all levels of experience!

Please be sure to visit our website llmware.ai for more information and updates.

🔉From Sound to Insights: Using AI🤖 for Audio File Transcription and Analysis!🚀

Prashant Iyer — Fri, 28 Jun 2024 20:57:16 +0000

If we were given an audio file, is there any way we could identify the time stamps where specific words were said? Is there any way we could extract all the key words mentioned about a topic?

With AI 🤖, we can do all of this and much more! The key lies in being able to parse audio into text, allowing us to then harness the natural language processing capabilities of language models to perform sophisticated analyses and inferences on our data.

Regardless of who you are, such an approach to audio transcription and analysis will augment how you interact with and extract knowledge from audio files.

Let's see how we can do this with llmware.

AI Tools 🤖

We'll be using two models for this example.

The first is Whisper by OpenAI. This is the model that will allow us to parse the audio files, i.e. convert them from audio to text.

The second is the SLIM (Structured Language Instruction Model) Extract Tool by LLMWare, which we'll be using to ask questions about our audio. This is a GGUF quantized version of a much larger model called slim-extract. All this means is that our model, the SLIM Extract Tool, is a smaller and faster version of the original model. This allows us to run it locally on a CPU, without the need for powerful computational resources like GPUs!

With that out of the way, let's get started with the example.

Step 1: Loading in audio files 🔉🔉

If you have audio files that you want to run the example with, then feel free to use those by setting input_folder appropriately, but if not, the llmware library provides you with several sets of sample audio files!



voice_sample_files = Setup().load_voice_sample_files(small_only=False)
input_folder = os.path.join(voice_sample_files, "greatest_speeches")

Here, we're loading in the greatest_speeches set of audio files.

Step 2: Parsing our audio files 📝

Now that we have our audio files, we can go about parsing them into chunks of text. Recall that we'll be needing the WhisperCPP model to do this. But fortunately, you won't have to directly interact with the model yourself since the Parser class from the llmware library will take care of this for you!



parser_output = Parser(chunk_size=400, max_chunk_size=600).parse_voice(input_folder, write_to_db=False, copy_to_library=False, remove_segment_markers=True, chunk_by_segment=True, real_time_progress=False)

Here, the chunk_size and max_chunk_size indicate how big each chunk of parsed text will be. We're passing in our folder containing the audio files to the parse_voice() function of the Parser class.

The function does accept many more optional arguments about how we'd like the audio to be parsed, but we can ignore them for this example.

Step 3: Text searching 🕵️

Let's now run a text search on our parsed audio. We can try searching for the word "president". What this means is that we want to find all the portions of the audio and corresponding text that have the word "president" in it. We can do this using the fast_search_dicts() function in the Utilies class in the llmware library.



results = Utilities().fast_search_dicts("president", parser_output)

Step 4: Making an AI call on text chunks 🤖

Now that we have a list of text blocks containing the word "president", lets use an AI model to identify which presidents are being mentioned in the selected text blocks.



extract_model = ModelCatalog().load_model("slim-extract-tool", sample=False, temperature=0.0, max_output=200)

Here, we're using the ModelCatalog class to load in our SLIM Extract Tool. Let's now iterate over each text block containing "president".



final_list = []
for i, res in enumerate(results):
    response = extract_model.function_call(res["text"], params=["president name"])

We're making a function_call() for "president name". This is how we ask our Tool to identify the president name in the text block.

Step 5: Analyzing our output 🔍

The function_call() function would have returned a dictionary containing a lot of data about the function call. We specifically want the president_name key in the dictionary.



extracted_name = ""
if "president_name" in response["llm_response"]:
    if len(response["llm_response"]["president_name"]) > 0:
        extracted_name = response["llm_response"]["president_name"][0].lower()
    else:
        print("\nupdate: skipping result - no president name found - ", response["llm_response"], res["text"])

If the value of the president_name key is a non-empty string, then we store its value in extracted_name. Otherwise, no result was found and we print this out.

Now lets see if the president name matched any of the recent American presidents in this list:



various_american_presidents = ["kennedy", "carter", "nixon", "reagan", "clinton", "obama"]

To do this, we'll check if the extracted_name contains any of these American presidents. If we have a match, then we'll add it to our final_list as a dictionary containing some information about the location of the name in the audio as well as the text block it was in.



for president in various_american_presidents:
    if president in extracted_name:
        final_list.append({"key": president, "source": res["file_source"], "time_start": res["coords_x"], "text": res["text"]})

Results! ✅

Let's now output the final_list.



for i, f in enumerate(final_list):
    print("final results: ", i, f)

This is what an one search result in the output would look after running the code.

Here, we have a Python dictionary as output containing:

key: the name of the president identified, which here is "kennedy"
source: the audio file this was found in, which here is "ConcessionStand.wav"
time_start: the time stamp in seconds where the president was mentioned, which here is 339.9 seconds
text: which contains the text chunk the name was found in.

Conclusion

And we're done! To recap, we were able to parse our audio files into text, run a text search on them for the word "president", and then use our SLIM Extract Tool to identify the specific presidents named in our text chunks! And remember that we did all this on just a CPU! 💻

Be sure to check out our YouTube video on this example!

If you made it this far, thank you for taking the time to go through this topic with us ❤️! For more content like this, make sure to visit our dev.to page.

The source code for many more examples like this one are on our GitHub. Find this example here.

Our repository also contains a notebook for this example that you can run yourself using Google Colab, Jupyter or any other platform that supports .ipynb notebooks.

Join our Discord to interact with a growing community of AI enthusiasts of all levels of experience!

Please be sure to visit our website llmware.ai for more information and updates.

🤖AI-Powered Contract Queries: Use Language Models for Effective Analysis!🔥

Prashant Iyer — Fri, 28 Jun 2024 20:48:31 +0000

Imagine you were given a large contract and asked a really specific question about it: "What is the notice for termination for convenience?" It would be an ordeal to locate the answer for this in the contract.

But what if we could use AI 🤖 to analyze the contract and answer this for us?

What we want here is to perform something known as retrieval-augmented generation (RAG). This is the process by which we give a language model some external sources (such as a contract). The external sources are intended to enhance the model's context, giving it a more comprehensive understanding of a topic. The model should then give us more accurate responses to the questions we ask it on the topic.

Now, a general purpose model like Chat-GPT might be able to answer questions about contracts with RAG, but what if we instead used a model that's been trained and fine-tuned specifically on contract data?

Our AI model 🤖

For this example, we'll be using LLMWare's dragon-yi-6b-gguf model. This model is RAG-finetuned for fact-based question-answering on complex business and legal documents.

This means that it is specialized in giving us short and concise responses to questions involving documents like contracts. This makes it perfect for our example!

This is also a GGUF quantized model, meaning that it is a smaller and faster version of the original 6 billion parameter dragon-yi-6b model. Fortunately for us, this means that we can run it on a CPU 💻 without the need for powerful computational resources like GPUs!

Now, let's look at an example of using the llmware library for contract analysis from start to end!

Step 1: Loading in files 📁

Let's start off by loading in our contracts to be analyzed. The llmware library provides sample contracts via the Setup class, but you can also use your own files in this example by replacing the agreements_path below.



local_path = Setup().load_sample_files()
agreements_path = os.path.join(local_path, "AgreementsLarge")

Here, we load in the AgreementsLarge set of files.

Next, we'll create a Library object and add our sample files to this library. An llmware library breaks documents down into text chunks and stores them in a database so that we can access them easily later.



msa_lib = Library().create_new_library("msa_lib503_635")
msa_lib.add_files(agreements_path)

Step 2: Locating MSA files 🔍

Let's say that we want to consider only MSA (master services agreements) files from our sample contracts.

We can first create a Query object containing all our files, and then run a text_search_by_page() to filter only the files that contain "master services agreement" on their front page.



q = Query(msa_lib)
query = "master services agreement"
results = q.text_search_by_page(query, page_num=1, results_only=False)
msa_docs = results["file_source"]

The results from the text search will be a dictionary containing detailed information about the text query. However, we're only interested in the file_source key representing the file names.

Great! We now have our MSA files.

Step 3: Loading our model 🪫🔋

Now, we can load in our model using the Prompt class in the llmware library.



model_name = "llmware/dragon-yi-6b-gguf"
prompter = Prompt().load_model(model_name)

Step 4: Analyzing our files using AI 🧠💡

Let's now iterate over our MSA files, and for each file, we'll:

identify the text chunks containing the word "termination",
add those chunks as a source for our AI call, and
run the AI call "What is the notice for termination for convenience?"

We can start by performing a text query for the word "termination".



for i, docs in enumerate(msa_docs):
    doc_filter = {"file_source": [docs]}
    termination_provisions = q.text_query_with_document_filter("termination", doc_filter)

We'll then add these termination_provisions as a source to our model.



sources = prompter.add_source_query_results(termination_provisions)

And with that done, we can call the LLM and ask it our question.



response = prompter.prompt_with_source("What is the notice for termination for convenience?")

Results! ✅

Let's print out our response and see what the output looks like.



for i, resp in enumerate(response):
    print("update: llm response - ", resp)

Here's what the output of our code looks like:

What we have is a Python dictionary with several keys, notably:

llm_response: giving us the answer to our question, which here is "30 days written notice"
evidence: giving us the text where the model found the answer to the question

The dictionary also contains detailed information about the metadata of the AI call, but these are not relevant to our example and have been omitted from the output above.

Human in the loop! 👤

We're not done just yet! If we wanted to generate a CSV report for a human to review the results of our analysis, we can make use of the HumanInTheLoop class. All we need to do is save the current state of our prompter and call the export_current_interaction_to_csv() function.



prompter.save_state()
csv_output = HumanInTheLoop(prompter).export_current_interaction_to_csv()

Conclusion

And that brings us to the end of our example! To summarize, we used the llmware library to:

Load in sample files
Filter only the MSA files
Use the dragon-yi-6b-gguf model to ask questions about termination provisions.

And remember that we did all of this on just a CPU! 💻

Check out our YouTube video on this example!

If you made it this far, thank you for taking the time to go through this topic with us ❤️! For more content like this, make sure to visit our dev.to page.

The source code for many more examples like this one are on our GitHub. Find this example here.

Our repository also contains a notebook for this example that you can run yourself using Google Colab, Jupyter or any other platform that supports .ipynb notebooks.

Join our Discord to interact with a growing community of AI enthusiasts of all levels of experience!

Please be sure to visit our website llmware.ai for more information and updates.

🤖AI-Powered Data Queries: Ask Questions in Plain English, Get Instant Results!🚀

Prashant Iyer — Tue, 11 Jun 2024 19:17:30 +0000

You've might have heard of SQL. It's a widely used programming language for storing and processing information in relational databases - simply put, relational databases store data in tables, where each row stores an entity and each column stores an attribute for that entity.

Let's say we have a table called customers in a relational database. If I wanted to access the names of all customers (customer_names) that have an annual_spend of at least $1000, I would have to formulate an SQL query like this:



SELECT customer_names FROM customers WHERE annual_spend >= 1000

I would then run this query against the database to access my results.

But what if AI 🤖 could do all this for us?

LLMWare allows us to do just that, making use of small language models such as slim-sql-1b-v0, which is only 1 billion parameters in size.

SLIM SQL Tool

We'll be making use of the SLIM (Structured Language Instruction Model) SQL Tool, which is a GGUF quantized version of the slim-sql-1b-v0 model. This essentially means that our Tool is of a smaller scale than the original model. To our advantage, it doesn't require much computational power to run, so it can run locally on a CPU without an internet connection or a GPU!

This Tool is specialized in small, fast, local prototyping and is effective for SQL operations that involve a single table. The Tool enables us to ask our questions about data entirely in natural language and still get accurate results! Let's look at an example of how to do this from start to finish.

Step 1: Loading our Model 🪫🔋

We'll start off by loading in our SLIM SQL Tool. Here, we check to see if the model is already downloaded locally, if not, we download it using the ModelCatalog class.



sql_tool_repo_path = os.path.join(LLMWareConfig().get_model_repo_path(), "slim-sql-tool")
if not os.path.exists(sql_tool_repo_path):
    ModelCatalog().load_model("llmware/slim-sql-tool")

Step 2: Loading our Data 📊

We then load in the sample customer_table.csv file containing our data. This sample file is provided by the llmware library!



files = os.listdir(sql_tool_repo_path)
csv_file = "customer_table.csv"

Next, we create a new SQL table called customer1 from our CSV file using the SQLTables class provided by the llmware library. This is important because we can run SQL queries on an SQL table, but not on a CSV file!



sql_db = SQLTables(experimental=True)
sql_db.create_new_table_from_csv(sql_tool_repo_path, csv_file, table_name="customer1")
print("update: successfully created new db table")

Setting experimental=True will use a provided testing database to create the table in.

Step 3: Querying with AI 🤖

We're finally getting to the good stuff! Now that we have our model and data, we can begin to ask our questions and get back some results.

We'll first load our agent, which is an instance of the LLMfx class. This class provides us a way to interact with various models through function calls. Our agent will take a natural language input from the user, communicate it to the appropriate model, generate an SQL query, run that query against the database, and then return the results of the query to us. Essentially, this is where the magic happens!



agent = LLMfx()
agent.load_tool("sql", sample=False, get_logits=True, temperature=0.0)

Next, we create a list of natural language questions we're going to be asking given our customer data. Let's see if our agent can answer them!



query_list = ["What is the highest annual spend of any customer?",
                  "Which customer has account number 1234953",
                  "Which customer has the lowest annual spend?"]

We can loop through each of the queries, and let the query_db() function do all the work for us. All our results will be stored in the agent object.



for i, query in enumerate(query_list):
    response = agent.query_db(query, table=table_name)

Results! ✅

Now that we have our results in the agent's research_list, we can print them out.



for x in range(0,len(agent.research_list)):
    print("research: ", x, agent.research_list[x])

For the example we have seen so far, this is what the output would look like for the first question ("What is the highest annual spend of any customer?").

The output is a dictionary containing a lot of detailed information about the steps carried out by the agent, but here are some of the more interesting parts of it:

db_response gives us what we want, the answer to the question! In this case, the response is 93540, meaning that the highest annual spend of any customer was $93540!
sql_query shows us the SQL query that was generated from our natural language question using the SLIM SQL tool. In this case, the query generated was: ```SQL

SELECT MAX(annual_spend) FROM customer1



---

## Conclusion
**And just like that, we've done it!** All we gave the program was a list of natural language questions and a CSV file with data. Behind the scenes, the `llmware` library:
1. created a table in a database with our data,
2. passed our questions into an AI model to get SQL queries,
3. ran the queries against the database, and
4. returned the results of the queries!

And if you're still not impressed, **remember that we can run this example locally on just a CPU 💻**!

Check out our YouTube video on this topic to see us explain the source code and analyze the results!




If you made it this far, thank you for taking the time to go through this topic with us ❤️! For more content like this, make sure to [visit our dev.to page](https://dev.to/llmware).

The source code for many more examples like this one are on [our GitHub](https://github.com/llmware-ai/llmware). Find this example [here](https://github.com/llmware-ai/llmware/blob/main/examples/SLIM-Agents/text2sql-end-to-end-2.py).

Our repository also contains a [notebook for this example](https://github.com/llmware-ai/llmware/blob/main/examples/Notebooks/NoteBook_Examples/text2sql-end-to-end-2-notebook.ipynb) that you can run yourself using Google Colab, Jupyter or any other platform that supports .ipynb notebooks.

[Join our Discord](https://discord.gg/fCztJQeV7J) to interact with a growing community of AI enthusiasts of all levels of experience!

Please be sure to visit our website [llmware.ai](https://llmware.ai/) for more information and updates.