First of All... What is RAG? 🕵️♂️
Retrieval-Augmented Generation (RAG) is an approach to natural language processing that references external documents to provide more accurate and contextually relevant answers. Despite its advantages, RAG faces some challenges, one of which is handling 'NOT FOUND' answers. Addressing this issue is crucial for developing an effective and reliable model that everyone can use.
Why 'NOT FOUND' Answers Can Be Concerning ⛔️
Some models respond with "hallucinations" when they cannot find an answer, creating inaccurate responses that may mislead the user. This can undermine the trust users have in the model, making it less reliable and effective.
How Can We Remedy This? 🛠️
For starters, it is better for the model to inform the user that it could not find the answer rather than fabricating one.
Next, we will delve into one way LLMWare handles 'NOT FOUND' cases effectively. By examining these methods, we can gain a better understanding of how to address this issue and enhance the overall performance and reliability of RAG systems.
For the Visual Learners... 📺
Here is a video discussing the same topic as this article. A good idea would be to watch the video, and then work through the steps in this article.
Framework 🖼️
LLMWare
For our new readers, LLMWARE is a comprehensive, open-source framework that provides a unified platform for application patterns based on LLMs, including Retrieval Augmented Generation (RAG).
Please run pip3 install llmware in the command line to download the package.
Import Libraries and Create Context 📚
from llmware.models import ModelCatalog
from llmware.parsers import WikiParser
ModelCatalog: A class within llmware that manages selecting the desired model, loading the model, and configuring the model.
WikiParser: A class within llmware that handles the retrieval and packaging of content from Wikipedia.
text =("BEAVERTON, Ore.--(BUSINESS WIRE)--NIKE, Inc. (NYSE:NKE) today reported fiscal 2024 financial results for its "
"third quarter ended February 29, 2024.) “We are making the necessary adjustments to drive NIKE’s next chapter "
"of growth Post this Third quarter revenues were slightly up on both a reported and currency-neutral basis* "
"at $12.4 billion NIKE Direct revenues were $5.4 billion, slightly up on a reported and currency-neutral basis "
"NIKE Brand Digital sales decreased 3 percent on a reported basis and 4 percent on a currency-neutral basis "
"Wholesale revenues were $6.6 billion, up 3 percent on a reported and currency-neutral basis Gross margin "
"increased 150 basis points to 44.8 percent, including a detriment of 50 basis points due to restructuring charges "
"Selling and administrative expense increased 7 percent to $4.2 billion, including $340 million of restructuring "
"charges Diluted earnings per share was $0.77, including $0.21 of restructuring charges. Excluding these "
"charges, Diluted earnings per share would have been $0.98* “We are making the necessary adjustments to "
"drive NIKE’s next chapter of growth,” said John Donahoe, President & CEO, NIKE, Inc. “We’re encouraged by "
"the progress we’ve seen, as we build a multiyear cycle of new innovation, sharpen our brand storytelling and "
"work with our wholesale partners to elevate and grow the marketplace.")
Here is the initial text for our extraction. It provides details about the popular sports brand, Nike. Feel free to modify this text to suit your needs.
Create Key for Extraction 🔐
extract_key = "company founding date"
dict_key = extract_key.replace(" ", "_")
company_founding_date = ""
Here, we set the company founding date as the target extraction from the text.
Run Initial Extract 🏃
model = ModelCatalog().load_model("slim-extract-tool", temperature=0.0, sample=False)
response = model.function_call(text, function="extract", params=[extract_key])
llm_response = response["llm_response"]
Model: In this snippet, we load LLMWare's slim-extract-tool, which is a 2.8B parameter GGUF model that is fine tuned for general-purpose extraction (GGUF is a quantization method that allows for quicker inference time and decreased model size at the cost of accuracy).
Temperature: This controls the randomness of the output. Valid values range between 0 and 1, where lower values make the model more deterministic, and higher values make the model more random and creative.
Sample: Determines if the output is generated deterministically or probabilistically. False generates deterministic output. True generates probabilistic output.
We then attempt to extract the information from the text using the model and store it in llm_response.
If Answer is Found... ✅
if dict_key in llm_response:
company_founding_date = llm_response[dict_key]
if len(company_founding_date) > 0:
company_founding_date = company_founding_date[0]
print(f"update: found the {extract_key} value - ", company_founding_date)
return company_founding_date
If the model successfully finds and extracts the company founding date, we will return the information.
If Answer is Not Found... ❌
else:
print(f"update: did not find the target value in the text - {company_founding_date}")
print("update: initiating a secondary process to try to find the information")
response = model.function_call(text, function="extract", params=["company name"])
If the model does not find the company founding date, we will run a second query to find the company name for future use in gathering more information.
Retrieve Information from Wiki 📖
if "company_name" in response["llm_response"]:
company_name = response["llm_response"]["company_name"][0]
if company_name:
print(f"\nupdate: found the company name - {company_name} - now using to lookup in secondary source")
output = WikiParser().add_wiki_topic(company_name,target_results=1)
After extracting the company's name from the text, we will then retrieve additional information about the company from Wiki.
Generate a Summary Snippet from Retrieved Article Data ✍️
if output:
supplemental_text = output["articles"][0]["summary"]
if len(supplemental_text) > 150:
supplemental_text_pp = supplemental_text[0:150] + " ... "
else:
supplemental_text_pp = supplemental_text
print(f"update: using lookup - {company_name} - found secondary source article "
f"(extract displayed) - ", supplemental_text_pp)
If we have successfully retrieved additional data from the Wiki, we truncate the response if it is over 150 characters and set supplemental_text_pp to
Call Extract Again With New Information 📞
new_response = model.function_call(supplemental_text,params=["company founding date"])
print("\nupdate: reviewed second source article - ", new_response["llm_response"])
Using the new information retrieved from Wiki, we run the same extraction on the model again.
Print Response If Found 🖨️
if "company_founding_date" in new_response["llm_response"]:
company_founding_date = new_response["llm_response"]["company_founding_date"]
if company_founding_date:
print("update: success - found the answer - ", company_founding_date)
If we find the company founding date after incorporating the new information, we print the result.
Fully Integrated Code 🧑💻
from llmware.models import ModelCatalog
from llmware.parsers import WikiParser
text =("BEAVERTON, Ore.--(BUSINESS WIRE)--NIKE, Inc. (NYSE:NKE) today reported fiscal 2024 financial results for its "
"third quarter ended February 29, 2024.) “We are making the necessary adjustments to drive NIKE’s next chapter "
"of growth Post this Third quarter revenues were slightly up on both a reported and currency-neutral basis* "
"at $12.4 billion NIKE Direct revenues were $5.4 billion, slightly up on a reported and currency-neutral basis "
"NIKE Brand Digital sales decreased 3 percent on a reported basis and 4 percent on a currency-neutral basis "
"Wholesale revenues were $6.6 billion, up 3 percent on a reported and currency-neutral basis Gross margin "
"increased 150 basis points to 44.8 percent, including a detriment of 50 basis points due to restructuring charges "
"Selling and administrative expense increased 7 percent to $4.2 billion, including $340 million of restructuring "
"charges Diluted earnings per share was $0.77, including $0.21 of restructuring charges. Excluding these "
"charges, Diluted earnings per share would have been $0.98* “We are making the necessary adjustments to "
"drive NIKE’s next chapter of growth,” said John Donahoe, President & CEO, NIKE, Inc. “We’re encouraged by "
"the progress we’ve seen, as we build a multiyear cycle of new innovation, sharpen our brand storytelling and "
"work with our wholesale partners to elevate and grow the marketplace.")
def not_found_then_triage_lookup():
print("\nNot Found Example - if info not found, then lookup in another source.\n")
extract_key = "company founding date"
dict_key = extract_key.replace(" ", "_")
company_founding_date = ""
model = ModelCatalog().load_model("slim-extract-tool", temperature=0.0, sample=False)
response = model.function_call(text, function="extract", params=[extract_key])
llm_response = response["llm_response"]
print(f"update: first text reviewed for {extract_key} - llm response: ", llm_response)
if dict_key in llm_response:
company_founding_date = llm_response[dict_key]
if len(company_founding_date) > 0:
company_founding_date = company_founding_date[0]
print(f"update: found the {extract_key} value - ", company_founding_date)
return company_founding_date
else:
print(f"update: did not find the target value in the text - {company_founding_date}")
print("update: initiating a secondary process to try to find the information")
response = model.function_call(text, function="extract", params=["company name"])
if "company_name" in response["llm_response"]:
company_name = response["llm_response"]["company_name"][0]
if company_name:
print(f"\nupdate: found the company name - {company_name} - now using to lookup in secondary source")
output = WikiParser().add_wiki_topic(company_name,target_results=1)
if output:
supplemental_text = output["articles"][0]["summary"]
if len(supplemental_text) > 150:
supplemental_text_pp = supplemental_text[0:150] + " ... "
else:
supplemental_text_pp = supplemental_text
print(f"update: using lookup - {company_name} - found secondary source article "
f"(extract displayed) - ", supplemental_text_pp)
new_response = model.function_call(supplemental_text,params=["company founding date"])
print("\nupdate: reviewed second source article - ", new_response["llm_response"])
if "company_founding_date" in new_response["llm_response"]:
company_founding_date = new_response["llm_response"]["company_founding_date"]
if company_founding_date:
print("update: success - found the answer - ", company_founding_date)
return company_founding_date
if __name__ == "__main__":
founding_date = not_found_then_triage_lookup()
You may also find the fully integrated code on our github here
Additionally, the notebook version (ipynb) is available here
Conclusion 🤖
Handling 'NOT FOUND' answers is one of the hardest problems in RAG, but it's a challenge that can be mitigated with thoughtful design. By implementing techniques like broader lookups, LLMWare aims to enhance the overall user experience and reliability of its AI systems.
Please check out our Github and leave a star! https://github.com/llmware-ai/llmware
Follow us on Discord here: https://discord.gg/MgRaZz2VAB
Top comments (2)
Great article! Have you dealt with this problem when working with medical data?
Great article!