DEV Community

Timon Vonk
Timon Vonk

Posted on

2

Announcing Swiftide, blazing fast data pipelines for RAG

While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use.

Excited to announce Swiftide, blazing fast data pipelines for Retrieval Augmented Generation written in Rust. Python bindings soon!

Check it out at https://swiftide.rs

IngestionPipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::with_chunk_range(10..512))
        .then(MetadataQACode::new(openai_client.clone()))
        .then_in_batch(10, Embed::new(openai_client.clone()))
        .then_store_with(
            Qdrant::try_from_url(qdrant_url)?
                .batch_size(50)
                .vector_size(1536)
                .collection_name("swiftide-examples".to_string())
                .build()?,
        )
        .run()
        .await?;
Enter fullscreen mode Exit fullscreen mode

Questions, feedback, complaints and great ideas are more than welcome in the comments <3

GitHub logo bosun-ai / swiftide

Fast, streaming indexing and query library for AI (RAG) applications, written in Rust

Table of Contents

CI Coverage Status Crate Badge Docs Badge Contributors Stargazers MIT License LinkedIn


Logo

Swiftide


Fast, streaming indexing and query library for AI applications, written in Rust


Read more on swiftide.rs »



<a href="https://docs.rs/swiftide/latest/swiftide/" rel="nofollow">API Docs</a>
·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=bug&amp;template=bug_report.md">Report Bug</a>
·
<a href="https://github.com/bosun-ai/swiftide/issues/new?labels=enhancement&amp;template=feature_request.md">Request Feature</a>
·
<a href="https://discord.gg/3jjXYen9UY" rel="nofollow">Discord</a>
Enter fullscreen mode Exit fullscreen mode

About The Project

Swiftide is a Rust native library for building LLM applications. Large language models are amazing, but need context to solve real problems. Swiftide allows you to ingest, transform and index large amounts of data fast, and then query that data so it it can be injected into prompts. This process is called Retrieval Augmented Generation.

With Swiftide, you can build your AI application from idea to production in a few lines of code.

RAG

While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born…

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

đź‘‹ Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay