While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use.
Excited to announce Swiftide, blazing fast data pipelines for Retrieval Augmented Generation written in Rust. Python bindings soon!
Check it out at https://swiftide.rs
IngestionPipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
.then_chunk(ChunkMarkdown::with_chunk_range(10..512))
.then(MetadataQACode::new(openai_client.clone()))
.then_in_batch(10, Embed::new(openai_client.clone()))
.then_store_with(
Qdrant::try_from_url(qdrant_url)?
.batch_size(50)
.vector_size(1536)
.collection_name("swiftide-examples".to_string())
.build()?,
)
.run()
.await?;
Questions, feedback, complaints and great ideas are more than welcome in the comments <3
Table of Contents
Swiftide
Blazing fast data pipelines for Retrieval Augmented Generation written in Rust
Explore the docs Β»
<a href="https://docs.rs/swiftide/latest/swiftide/" rel="nofollow">API Docs</a>
Β·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=bug&template=bug_report.md">Report Bug</a>
Β·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=enhancement&template=feature_request.md">Request Feature</a>
About The Project
Swiftide is a straightforward, easy-to-use, easy-to-extend asynchronous data ingestion and processing library. It is designed to be used in a RAG (Retrieval Augmented Generation) system. It is built to be fast and efficient, with a focus on parallel processing and asynchronous operations.
While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Ingestion performance went from multiple tens of minutes to a few seconds.
Part of the bosun.ai project. An upcoming platform for autonomous code improvement.
We <3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue.
(backβ¦
Top comments (0)