Other providers and stores

Each recipe is a runnable example from the swiftide repository that indexes files into one provider or store; the PgVector, LanceDB and Dashscope recipes also query what they stored. Each section points to the integration page that covers the client options in depth.

Prerequisites

Run an example from the root of the swiftide repository, because the examples read README.md and other files by relative path:

# in the swiftide repository
cargo run -p swiftide-examples --example <name>

To copy a recipe into your own crate, add the swiftide line from its section, plus the runtime and logging crates every example uses:

# in your own crate
cargo add tokio --features full
cargo add tracing-subscriber

Ollama

# in your own crate
cargo add swiftide@0.34 --features ollama,fastembed

# in the swiftide repository
ollama pull llama3.1
cargo run -p swiftide-examples --example index-ollama

indexing::Pipeline::from_loader(FileLoader::new("README.md"))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then(MetadataQAText::new(ollama_client.clone()))
    .then_in_batch(Embed::new(fastembed).with_batch_size(10))
    .then_store_with(memory_store.clone())
    .run()
    .await?;

Source: examples/index_ollama.rs#L39-L45

FastEmbed computes the embeddings locally, so only metadata generation calls Ollama. Ollama::default() talks to http://localhost:11434/v1 and sends no application programming interface (API) key.

Client options and gotchas: OpenAI-compatible providers.

Groq

# in your own crate
cargo add swiftide@0.34 --features groq,fastembed

# in the swiftide repository
cargo run -p swiftide-examples --example index-groq

indexing::Pipeline::from_loader(FileLoader::new("README.md"))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then(MetadataQAText::new(groq_client.clone()))
    .then_in_batch(Embed::new(fastembed).with_batch_size(10))
    .then_store_with(memory_store.clone())
    .run()
    .await?;

Source: examples/index_groq.rs#L39-L45

The client reads GROQ_API_KEY. The example sets the model llama3-8b-8192, which Groq has retired. Change it in the example’s setup to a current model, such as llama-3.1-8b-instant, before you run it.

Client options and gotchas: OpenAI-compatible providers.

Dashscope

# in your own crate
cargo add swiftide@0.34 --features dashscope,lancedb
cargo add temp-dir anyhow

# in the swiftide repository
cargo run -p swiftide-examples --example dashscope

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
    .with_default_llm_client(client.clone())
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then(MetadataQAText::new(client.clone()))
    .then_in_batch(Embed::new(client.clone()).with_batch_size(10))
    .then_store_with(lancedb.clone())
    .run()
    .await?;

Source: examples/dashscope.rs#L38-L45

Dashscope reads QWEN_API_KEY, or DASHSCOPE_API_KEY when that is unset. One client sets both an embed model and a prompt model, so it serves as large language model (LLM) and embedder.

with_default_llm_client gives a client to every transformer added after it that has none of its own, such as MetadataQAText::default(). A transformer built with new(client) keeps its own client.

The loader picks up every markdown file under the current directory. It uses the default filters of the ignore crate, so it skips hidden files and directories, such as .github/. It also skips what .ignore files, the global gitignore and .git/info/exclude exclude, and, inside a git repository, what .gitignore excludes.

let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::GenerateSubquestions::from_client(
        client.clone(),
    ))
    .then_transform_query(query_transformers::Embed::from_client(client.clone()))
    .then_retrieve(lancedb.clone())
    .then_transform_response(response_transformers::Summary::from_client(client.clone()))
    .then_answer(answers::Simple::from_client(client.clone()));

let result = pipeline
    .query("What is swiftide? Please provide an elaborate explanation")
    .await?;

Source: examples/dashscope.rs#L49-L60

The query pipeline generates subquestions, embeds them, retrieves from LanceDB, summarizes the documents with the Summary response transformer, and answers with answers::Simple.

Client options and gotchas: OpenAI-compatible providers and LanceDB.

FastEmbed

# in your own crate
cargo add swiftide@0.34 --features fastembed,qdrant

# in the swiftide repository
docker run -p 6334:6334 qdrant/qdrant
cargo run -p swiftide-examples --example fastembed

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
    .then_in_batch(Embed::new(FastEmbed::builder().batch_size(10).build()?))
    .then_store_with(
        Qdrant::try_from_url(qdrant_url)?
            .batch_size(50)
            .vector_size(384)
            .collection_name("swiftide-examples-fastembed".to_string())
            .build()?,
    )
    .run()
    .await?;

Source: examples/fastembed.rs#L32-L42

The recipe needs Qdrant on QDRANT_URL, which defaults to the gRPC port at http://localhost:6334. It loads, embeds and stores, with no chunker and no metadata.

FastEmbed runs embedding models in your process, with no API key. It downloads the default model from Hugging Face on first use. That model produces 384-dimensional vectors, so the Qdrant collection uses vector_size(384).

batch_size(10) on the FastEmbed builder sets how many texts the model embeds at once (default 256). batch_size(50) on the Qdrant builder sets the nodes per write. Embedding explains how these relate to the pipeline’s batch size.

Client options and gotchas: FastEmbed and Qdrant.

Redis

# in your own crate
cargo add swiftide@0.34 --features redis,tree-sitter
cargo add serde_json

# in the swiftide repository
cargo run -p swiftide-examples --example index-redis

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
    .then_chunk(ChunkCode::try_for_language_and_chunk_size(
        "rust",
        10..2048,
    )?)
    .then_store_with(
        // By default the value is the full node serialized to JSON.
        // We can customize this by providing a custom function.
        Redis::try_build_from_url(&redis_url)?
            .persist_value_fn(|node| Ok(serde_json::to_string(&node.metadata)?))
            .batch_size(50)
            .build()?,
    )
    .run()
    .await?;

Source: examples/index_into_redis.rs#L39-L53

The recipe needs Redis on REDIS_URL, which defaults to redis://localhost:6379. It chunks Rust files with ChunkCode and stores the chunks in Redis, with no LLM or embedder.

Redis is storage only here. It does not implement retrieval, but it also works as a node cache and as a message history backend. Keys, values and the other backends: Storage.

PgVector

# in your own crate
cargo add swiftide@0.34 --features pgvector,ollama,fastembed

# in the swiftide repository
ollama pull llama3.2
cargo run -p swiftide-examples --example index-md-pgvector

The example starts a pgvector/pgvector container through Docker with a test helper, so Docker must be running. In your application, pass your own database URL to db_url.

tracing::info!("Starting indexing pipeline");

indexing::Pipeline::from_loader(FileLoader::new(test_dataset_path).with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then(MetadataQAText::new(llm_client.clone()))
    .then_in_batch(Embed::new(fastembed.clone()).with_batch_size(100))
    .then_store_with(pgv_storage.clone())
    .run()
    .await?;

Source: examples/index_md_into_pgvector.rs#L103-L111

async fn ask_query(
    llm_client: impl SimplePrompt + Clone + 'static,
    embed: FastEmbed,
    vector_store: PgVector,
    questions: Vec<String>,
) -> Result<Vec<String>, Box<dyn std::error::Error>> {
    // By default the search strategy is SimilaritySingleEmbedding
    // which takes the latest query, embeds it, and does a similarity search
    //
    // Pgvector will return an error if multiple embeddings are set
    //
    // The pipeline generates subquestions to increase semantic coverage, embeds these in a single
    // embedding, retrieves the default top_k documents, summarizes them and uses that as context
    // for the final answer.
    let pipeline = query::Pipeline::default()
        .then_transform_query(query_transformers::GenerateSubquestions::from_client(
            llm_client.clone(),
        ))
        .then_transform_query(query_transformers::Embed::from_client(embed))
        .then_retrieve(vector_store.clone())
        .then_transform_response(response_transformers::Summary::from_client(
            llm_client.clone(),
        ))
        .then_answer(answers::Simple::from_client(llm_client.clone()));

    let results: Vec<String> = pipeline
        .query_all(questions)
        .await?
        .iter()
        .map(|result| result.answer().to_string())
        .collect();

    Ok(results)
}

Source: examples/index_md_into_pgvector.rs#L17-L50

query_all sends several questions through one pipeline and returns an answered query for each. The SimilaritySingleEmbedding search strategy needs exactly one vector field; with more, it fails with “Search strategy for multiple vector fields in the schema is not yet implemented”. A CustomStrategy does not have this limit.

Table setup, columns and indexes: PgVector.

LanceDB

# in your own crate
cargo add swiftide@0.34 --features lancedb,openai
cargo add temp-dir

# in the swiftide repository
cargo run -p swiftide-examples --example lancedb

let openai_client = integrations::openai::OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_prompt_model("gpt-4o-mini")
    .build()?;

let tempdir = TempDir::new().unwrap();

// Configure lancedb with a default vector size, a single embedding
// and in addition to embedding the text metadata, also store it in a field
let lancedb = LanceDB::builder()
    .uri(tempdir.child("lancedb").to_str().unwrap())
    .vector_size(1536)
    .with_vector(EmbeddedField::Combined)
    .with_metadata(METADATA_QA_TEXT_NAME)
    .table_name("swiftide_test")
    .build()
    .unwrap();

Source: examples/lancedb.rs#L20-L36

indexing::Pipeline::from_loader(FileLoader::new("README.md"))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then(MetadataQAText::new(openai_client.clone()))
    .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10))
    .then_store_with(lancedb.clone())
    .run()
    .await?;

Source: examples/lancedb.rs#L40-L46

// By default the search strategy is SimilaritySingleEmbedding
// which takes the latest query, embeds it, and does a similarity search
//
// LanceDB will return an error if multiple embeddings are set
//
// The pipeline generates subquestions to increase semantic coverage, embeds these in a single
// embedding, retrieves the default top_k documents, summarizes them and uses that as context
// for the final answer.
let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::GenerateSubquestions::from_client(
        openai_client.clone(),
    ))
    .then_transform_query(query_transformers::Embed::from_client(
        openai_client.clone(),
    ))
    .then_retrieve(lancedb.clone())
    .then_transform_response(response_transformers::Summary::from_client(
        openai_client.clone(),
    ))
    .then_answer(answers::Simple::from_client(openai_client.clone()));

Source: examples/lancedb.rs#L50-L69

LanceDB is embedded: uri points at a local directory, and the table is created on the first run. The example uses a temporary directory, so the data is gone when it exits. The recipe needs OPENAI_API_KEY.

With SimilaritySingleEmbedding, retrieval also needs exactly one vector field, and fails with “Zero or multiple vector fields configured in schema” otherwise. A CustomStrategy does not have this limit.

Remote storage and vector indexes: LanceDB.

Reference

Feature flags