AI applications are growing rapidly. Developers are building applications using Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), embeddings, and intelligent document processing systems.
But before an AI model can provide useful results, one important step is often overlooked:
Preparing the text properly before sending it to AI systems.
Raw text is rarely ready for AI processing.
It may contain:
- Personal information
- Internal company data
- Large documents that exceed model limits
- Unstructured content
- Unnecessary information that affects retrieval quality
This is where text pre-processing becomes important.
To solve these common challenges, I built AICurt, a lightweight Python toolkit for AI text pre-processing.
PyPI:
https://pypi.org/project/aicurt/
Why Do We Need AICurt?
When building AI applications, developers often need multiple tools for different pre-processing tasks.
A typical AI workflow may look like this:
Raw Documents
↓
Remove or protect sensitive information
↓
Split documents into smaller chunks
↓
Analyse tokens and text size
↓
Send prepared data to LLM / Embedding systems
Each step is important.
If sensitive information is not handled properly, private data may reach AI models.
If documents are not chunked correctly, retrieval quality can decrease.
If token usage is not considered, applications can hit model limits.
AICurt brings these common pre-processing steps into one lightweight toolkit.
What Makes AICurt Different?
There are already many excellent libraries available for AI and text processing.
AICurt is not built to replace them.
The goal is to provide a simple toolkit focused on:
- Lightweight design
- No external APIs
- Local text processing
- Developer-controlled behaviour
- Predictable results
The idea behind AICurt is simple:
Developers should have control over how their data is processed before it reaches AI systems.
Protect Sensitive Data Before Using AI
One of the biggest concerns when working with AI applications is privacy.
Documents may contain:
- Email addresses
- Phone numbers
- Customer information
- Employee information
- Internal identifiers
- Confidential business data
Sending this information directly to an AI model may not always be appropriate.
AICurt provides a custom PII redaction engine that allows developers to define exactly what information should be protected.
Instead of automatically modifying everything, AICurt uses an explicit rule-based approach.
You decide:
- What should be detected
- What should be replaced
- How the replacement should look
Example:
from aicurt.pii import PiiRedactor
redactor = PiiRedactor()
redactor.register_rule(
"EMAIL",
r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
replacement="[EMAIL]",
)
result = redactor.redact(
"Contact alice@example.com"
)
print(result.text)
Output:
Contact [EMAIL]
This gives developers predictable and controlled privacy handling.
Create Custom PII Rules
Every application has different requirements.
A customer support system may need to protect email addresses.
A banking application may need to protect account information.
An internal company tool may need to hide employee identifiers.
AICurt allows custom rules:
redactor.register_rule(
"CUSTOMER_ID",
r"CUST-\d+",
replacement="[CUSTOMER_ID]"
)
You can create rules based on your own data requirements.
Flexible Masking Options
Sometimes replacing information completely is not required.
You may want partial masking instead.
For example:
john@example.com
could become:
j********m
AICurt supports configurable masking:
from aicurt.pii import PiiConfig
config = PiiConfig(
mask_strategy="partial",
mask_char="*",
mask_visible_prefix=1,
mask_visible_suffix=1,
preserve_length=True,
)
You can control:
- Masking style
- Visible characters
- Replacement behaviour
- Length preservation
Improve RAG Applications With Better Chunking
Many AI applications rely on document retrieval.
Examples:
- Chat with your documents
- Enterprise knowledge assistants
- Research assistants
- Internal search systems
Large documents cannot always be sent directly to AI models.
They need to be divided into smaller meaningful sections.
Poor chunking can lead to:
- Missing important context
- Lower retrieval accuracy
- Incorrect AI responses
AICurt provides multiple chunking strategies:
- Word chunking
- Sentence chunking
- Paragraph chunking
- Character chunking
- Token chunking
- Recursive chunking
- Sliding window chunking
Example:
from aicurt.chunking import ChunkStrategy, TextChunker
text = """
Artificial Intelligence is changing software development.
Large Language Models are transforming applications.
"""
chunker = TextChunker()
chunks = chunker.chunk(
text,
strategy=ChunkStrategy.SENTENCE,
chunk_size=100
)
for chunk in chunks:
print(chunk.content)
This helps create consistent input for:
- Embeddings
- Vector databases
- Retrieval pipelines
- LLM applications
Understand Your Text Before Sending It to AI
Token usage matters when working with AI models.
Developers often need to know:
- How many tokens does this document contain?
- How large is my input?
- How much text can I send to a model?
AICurt provides lightweight token analysis utilities.
Example:
from aicurt.tokenizer import SimpleTokenizer
tokenizer = SimpleTokenizer()
text = "Hello AI world"
print(tokenizer.tokenize(text))
print(tokenizer.stats(text))
You can analyse:
- Token count
- Word count
- Character count
Use AICurt From the Command Line
AICurt also provides CLI support.
You can process files directly:
Detect:
aicurt detect sample.txt
Redact:
aicurt redact sample.txt --output clean.txt
Chunk:
aicurt chunk sample.txt --strategy word --chunk-size 50
Generate statistics:
aicurt stats sample.txt
This makes AICurt useful for quick experiments and automation workflows.
Example AI Workflow Using AICurt
A common AI document workflow could look like this:
Customer Documents
↓
AICurt PII Redaction
↓
AICurt Text Chunking
↓
Token Analysis
↓
Embedding Generation
↓
RAG Application
AICurt helps prepare the data before it reaches your AI system.
Built for Developers Who Want Control
The main idea behind AICurt is simple:
AI systems are only as good as the data they receive.
Good pre-processing helps create:
- Safer AI applications
- Better retrieval results
- More predictable workflows
- Cleaner model inputs
AICurt gives developers simple tools to prepare text before using it with AI.
Future Improvements
AICurt has been tested end-to-end during development, but real-world workflows can always reveal new scenarios and edge cases.
I will continue improving the package, fixing reported issues, and adding valuable new features based on feedback and suggestions.
If you use AICurt, I would appreciate your valuable feedback, feature ideas, and suggestions.
Try AICurt
Install:
pip install aicurt
PyPI:
https://pypi.org/project/aicurt/
If you are building LLM applications, RAG systems, embedding pipelines, or AI document workflows, I would love for you to try AICurt and share your experience.
Top comments (0)