swiftide

A quick example

IngestionPipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::with_chunk_range(10..512))
        .then(MetadataQACode::new(openai_client.clone()))
        .then_in_batch(10, Embed::new(openai_client.clone()))
        .then_store_with(
            Qdrant::try_from_url(qdrant_url)?
                .batch_size(50)
                .vector_size(1536)
                .collection_name("swiftide-examples".to_string())
                .build()?,
        )
        .run()
        .await?;

Transform, enrich and persist lots of data

Load data from various sources, transform it, enrich it with metadata, and persist it in a database.

Many existing integrations

Qdrant, OpenAI, Redis, FastEmbed, Spider and many more.

Easy to extend

Write your own loaders, transformers, and storages by extending straight forward traits.

Written in Rust

Fast, safe, and efficient. Built with Rust’s async and streaming features.

Part of Bosun.ai

Part of Bosun.ai and actively used in production.

Reference

Full API documentation available on docs.rs