Announcing Swiftide, blazing fast data pipelines for RAG

While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use.

Excited to announce Swiftide, blazing fast data pipelines for Retrieval Augmented Generation written in Rust. Python bindings soon!

Check it out at https://swiftide.rs

IngestionPipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::with_chunk_range(10..512))
        .then(MetadataQACode::new(openai_client.clone()))
        .then_in_batch(10, Embed::new(openai_client.clone()))
        .then_store_with(
            Qdrant::try_from_url(qdrant_url)?
                .batch_size(50)
                .vector_size(1536)
                .collection_name("swiftide-examples".to_string())
                .build()?,
        )
        .run()
        .await?;

Questions, feedback, complaints and great ideas are more than welcome in the comments <3

bosun-ai / swiftide

Blazing fast data pipelines for Retrieval Augmented Generation written in Rust

Table of Contents

• About The Project
• Example
• Features
• Vision
• Getting Started
  • Prerequisites
  • Installation
• Usage and concepts
• Roadmap
• Contributing
• License

Swiftide

Blazing fast data pipelines for Retrieval Augmented Generation written in Rust


Explore the docs »

<a href="https://docs.rs/swiftide/latest/swiftide/" rel="nofollow">API Docs</a>
·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=bug&amp;template=bug_report.md">Report Bug</a>
·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=enhancement&amp;template=feature_request.md">Request Feature</a>

About The Project

Swiftide is a straightforward, easy-to-use, easy-to-extend asynchronous data ingestion and processing library. It is designed to be used in a RAG (Retrieval Augmented Generation) system. It is built to be fast and efficient, with a focus on parallel processing and asynchronous operations.

While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Ingestion performance went from multiple tens of minutes to a few seconds.

Part of the bosun.ai project. An upcoming platform for autonomous code improvement.

We <3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue.

(back…

View on GitHub