Fast & streaming LLM applications in Rust

What?

Large language models are amazing, but need context to solve real problems. Ingest, transform, and index large amounts of data. Query it, augment it, and generate answers with Swiftide.

Swiftide aims to be fast, modular, and with minimal abstractions, and can easily be extended by implementing simple traits.

Why?

The AI and LLM landscape is moving fast. The core idea is to have a stable and modular infrastructure, so you can focus on building the actual LLM application.

Written in Rust, Swiftide is fast, safe, and efficient. It is built with Rust’s async and streaming features, and can be used in production.

Some quick examples

Indexing

// Load markdown from the current directory, generate synthetic questions and answers,
// embed it with FastEmbed and store into Qdrant.
indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
        .with_default_llm_client(openai_client)
        .then_chunk(ChunkMarkdown::from_chunk_range(10..512))
        .then(MetadataQAText::default())
        .then(move |mut node: Node| {
          node.metadata.insert("Hello", "Metadata");

          Ok(node)
        })
        .then_in_batch(Embed::new(FastEmbed::default()))
        .then_store_with(
            Qdrant::builder()
                .batch_size(50)
                .vector_size(384)
                .collection_name("swiftide-examples")
                .build()?,
        )
        .run()
        .await?;

Querying

// Query our indexed data by generating subquestions, embedding the transformed query,
// retrieving documents from Qdrant and generating an answer.
query::Pipeline::default()
    .then_transform_query(GenerateSubquestions::from_client(
        openai_client.clone(),
    ))
    .then_transform_query(Embed::from_client(
        openai_client.clone(),
    ))
    .then_retrieve(qdrant.clone())
    .then_answer(Simple::from_client(openai_client.clone()))
    .query("How can I use the query pipeline in Swiftide?")
    .await?;

Agents

#[swiftide_macros::tool(
    description = "Searches code",
    param(name = "code_query", description = "The code query")
)]
async fn search_code(
    context: &dyn AgentContext,
    code_query: &str,
) -> Result<ToolOutput, ToolError> {
    let command_output = context
        .exec_cmd(&Command::shell(format!("rg '{code_query}'")))
        .await?;

    Ok(command_output.into())
}

agents::Agent::builder()
        .llm(&openai)
        .tools(vec![search_code()])
        .build()?
        .query("In what file can I find an example of a swiftide agent?").await?;

Features

Transform, enrich and persist lots of data

Load data from various sources, transform it, enrich it with metadata, and persist it with lazy, asynchronous, parallel pipelines.

Transform code and text

Chunk, transform, and augment code with build in transformers. Swiftide uses tree-sitter to interpret and augment code. Text documents, be it markdown, html or unstructured, is also supported.

Query pipeline

Query your indexed data, transform it, filter it, and generate a response. Purpusely build for Retrieval Augmented Generation.

Agents

Build and run autonomous agents on top of Swiftide. Easily define tools, hook in on important parts of the agent lifecycle, and code something that does the job for you.

Customizable templated prompts

Customize and bring your own prompts, build on Tera, a jinja style templating library.

Many existing integrations

Qdrant, OpenAI, Groq, AWS Bedrock, Redis, FastEmbed, Spider and many more.

Easy to extend

Write your own loaders, transformers, and storages by extending straight forward traits.

Written in Rust

Fast, safe, and efficient. Built with Rust’s async and streaming features.

Part of Bosun.ai

Part of Bosun.ai and actively used in production.

Reference

Full API documentation available on docs.rs