Extract text from documents

Up to this point, all the examples have been working with sections of text, which have already been split through some other means. What happens if we're working with documents? First we need to get the text out of these documents, then figure out how to index to best support similarity search.

This article shows how documents can have text extracted and segmented to support similarity search.

Install dependencies

Install txtai and all dependencies. Since this article is using optional pipelines, we need to install the pipeline extras package.

pip install txtai[pipeline]

# Get test data
wget -N https://github.com/neuml/txtai/releases/download/v3.5.0/tests.tar.gz
tar -xvzf tests.tar.gz

# Install NLTK
import nltk
nltk.download('punkt')

Create a Textractor instance

The Textractor instance is the main entrypoint for extracting text. This method is backed by Apache Tika, a robust text extraction library written in Java. Apache Tika has support for a large number of file formats: PDF, Word, Excel, HTML and others. The Python Tika package automatically installs Tika and starts a local REST API instance used to read extracted data.

Note: This requires Java to be installed locally.

from txtai.pipeline import Textractor

# Create textractor model
textractor = Textractor()

Extract text

The example below shows how to extract text from a file.

textractor("txtai/article.pdf")

Introducing txtai, an AI-powered search engine built on Transformers Add Natural Language Understanding to any application Search is the base of many applications. Once data starts to pile up, users want to be able to find it. It’s the foundation of the internet and an ever-growing challenge that is never solved or done. The field of Natural Language Processing (NLP) is rapidly evolving with a number of new developments. Large-scale general language models are an exciting new capability allowing us to add amazing functionality quickly with limited compute and people. Innovation continues with new models and advancements coming in at what seems a weekly basis. This article introduces txtai, an AI-powered search engine that enables Natural Language Understanding (NLU) based search in any application. Introducing txtai txtai builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems. txtai also has functionality for zero-shot classification. txtai is open source and available on GitHub. txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below: • paperai — AI-powered literature discovery and review engine for medical/scientific papers • tldrstory — AI-powered understanding of headlines and story text • neuspo — Fact-driven, real-time sports event and news site • codequestion — Ask coding questions directly from the terminal Build an Embeddings index For small lists of texts, the method above works. But for larger repositories of documents, it doesn’t make sense to tokenize and convert all embeddings for each query. txtai supports building pre- computed indices which significantly improves performance. Building on the previous example, the following example runs an index method to build and store the text embeddings. In this case, only the query is converted to an embeddings vector each search. https://github.com/neuml/codequestion https://neuspo.com/ https://github.com/neuml/tldrstory https://github.com/neuml/paperai Introducing txtai, an AI-powered search engine built on Transformers Add Natural Language Understanding to any application Introducing txtai Build an Embeddings index

Note that the text from the article was extracted into a single string. Depending on the articles, this may be acceptable. For long articles, often you'll want to split the content into logical sections to build better downstream vectors.

Extract sentences

textractor = Textractor(sentences=True)
textractor("txtai/article.pdf")

['Introducing txtai, an AI-powered search engine built on Transformers Add Natural Language Understanding to any application Search is the base of many applications.',
 'Once data starts to pile up, users want to be able to find it.',
 'It’s the foundation of the internet and an ever-growing challenge that is never solved or done.',
 'The field of Natural Language Processing (NLP) is rapidly evolving with a number of new developments.',
 'Large-scale general language models are an exciting new capability allowing us to add amazing functionality quickly with limited compute and people.',
 'Innovation continues with new models and advancements coming in at what seems a weekly basis.',
 'This article introduces txtai, an AI-powered search engine that enables Natural Language Understanding (NLU) based search in any application.',
 'Introducing txtai txtai builds an AI-powered index over sections of text.',
 'txtai supports building text indices to perform similarity searches and create extractive question-answering based systems.',
 'txtai also has functionality for zero-shot classification.',
 'txtai is open source and available on GitHub.',
 'txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below: • paperai — AI-powered literature discovery and review engine for medical/scientific papers • tldrstory — AI-powered understanding of headlines and story text • neuspo — Fact-driven, real-time sports event and news site • codequestion — Ask coding questions directly from the terminal Build an Embeddings index For small lists of texts, the method above works.',
 'But for larger repositories of documents, it doesn’t make sense to tokenize and convert all embeddings for each query.',
 'txtai supports building pre- computed indices which significantly improves performance.',
 'Building on the previous example, the following example runs an index method to build and store the text embeddings.',
 'In this case, only the query is converted to an embeddings vector each search.',
 'https://github.com/neuml/codequestion https://neuspo.com/ https://github.com/neuml/tldrstory https://github.com/neuml/paperai Introducing txtai, an AI-powered search engine built on Transformers Add Natural Language Understanding to any application Introducing txtai Build an Embeddings index']

Now the document is split up at the sentence level. These sentences can be feed to a workflow that adds each sentence to an embeddings index. Depending on the task, this may work well. Alternatively, it may be even better to split at the paragraph level.

Extract paragraphs

textractor = Textractor(paragraphs=True)
textractor("txtai/article.pdf")

['Introducing txtai, an AI-powered search engine built on Transformers',
 'Add Natural Language Understanding to any application',
 'Search is the base of many applications. Once data starts to pile up, users want to be able to find it. It’s the foundation of the internet and an ever-growing challenge that is never solved or done.',
 'The field of Natural Language Processing (NLP) is rapidly evolving with a number of new developments. Large-scale general language models are an exciting new capability allowing us to add amazing functionality quickly with limited compute and people. Innovation continues with new models and advancements coming in at what seems a weekly basis.',
 'This article introduces txtai, an AI-powered search engine that enables Natural Language Understanding (NLU) based search in any application.',
 'Introducing txtai txtai builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems. txtai also has functionality for zero-shot classification. txtai is open source and available on GitHub.',
 'txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below:',
 '• paperai — AI-powered literature discovery and review engine for medical/scientific papers • tldrstory — AI-powered understanding of headlines and story text • neuspo — Fact-driven, real-time sports event and news site • codequestion — Ask coding questions directly from the terminal',
 'Build an Embeddings index For small lists of texts, the method above works. But for larger repositories of documents, it doesn’t make sense to tokenize and convert all embeddings for each query. txtai supports building pre- computed indices which significantly improves performance.',
 'Building on the previous example, the following example runs an index method to build and store the text embeddings. In this case, only the query is converted to an embeddings vector each search.',
 'https://github.com/neuml/codequestion https://neuspo.com/ https://github.com/neuml/tldrstory https://github.com/neuml/paperai',
 'Introducing txtai, an AI-powered search engine built on Transformers Add Natural Language Understanding to any application',
 'Introducing txtai Build an Embeddings index']