FastEmbed

You embed chunks and queries with local dense and sparse models, pick a model that fits your data, and rerank retrieved documents, all on your CPU without an API key. Turn on the fastembed feature flag:

cargo add swiftide@0.34 --features fastembed

Default models

The hybrid search example builds one dense and one sparse embedder with the default models:

// Ensure all batching is consistent
let batch_size = 64;

let fastembed_sparse = FastEmbed::try_default_sparse().unwrap().to_owned();
let fastembed = FastEmbed::try_default().unwrap().to_owned();

// Set up openai with the mini model, which is great for indexing
let openai = openai::OpenAI::builder()
    .default_prompt_model("gpt-4o-mini")
    .build()
    .unwrap();

// Set up qdrant and use the combined fields (metadata + chunks) for both sparse and dense
// vectors
let qdrant = Qdrant::builder()
    .batch_size(batch_size)
    .vector_size(384)
    .with_vector(EmbeddedField::Combined)
    .with_sparse_vector(EmbeddedField::Combined)
    .collection_name("swiftide-hybrid-example")
    .build()?;

Source: examples/hybrid_search.rs#L23-L43

The excerpt also builds an OpenAI client and Qdrant storage. To compile it as is, add the openai and qdrant feature flags next to fastembed.

FastEmbed::try_default() loads BGE (BAAI General Embedding) small en v1.5, a dense model with 384 dimensions. FastEmbed::try_default_sparse() loads SPLADE++ (Sparse Lexical and Expansion model, prithivida/Splade_PP_en_v1), a sparse model. Both return a Result, because loading the model can fail.

FastEmbed implements EmbeddingModel and SparseEmbeddingModel. You pass the dense instance to the Embed batch transformer in the indexing pipeline and to query_transformers::Embed in the query pipeline. The sparse instance goes to SparseEmbed on both sides. Each instance holds one model. Passing a sparse instance to Embed fails with the permanent error “Expected dense model, got sparse”, and a dense instance in SparseEmbed fails with “Expected sparse model, got dense”.

Your storage’s vector size must match the model: 384 for BGE small.

Model choice

The ask-your-codebase showcase indexes source code, so it picks a model trained on code:

/// A code-aware embedding model that runs on the CPU and needs no API key.
///
/// The model (~600MB) is downloaded on first use.
pub fn local_model() -> Result<FastEmbed> {
    let model = TextEmbedding::try_new(
        InitOptions::new(EmbeddingModel::JinaEmbeddingsV2BaseCode).with_max_length(MAX_TOKENS),
    )?;

    FastEmbed::builder()
        .embedding_model(model)
        .batch_size(BATCH_SIZE)
        .build()
}

Source: examples/ask-your-codebase/src/embedding.rs#L18-L30

jina-embeddings-v2-base-code produces 768-dimensional vectors. fastembed::TextInitOptions::with_max_length raises the token limit to 2048 for each embedded text. In ask-your-codebase that text is a chunk of up to 4096 bytes plus its generated questions and answers. FastEmbed truncates longer texts without an error. The default limit is 512 tokens, or less if the model’s own limit is lower.

Swiftide does not re-export the fastembed crate. To name a model, add it yourself at the major version Swiftide uses:

cargo add fastembed@5 --no-default-features

FastEmbedBuilder::embedding_model accepts a fastembed::TextEmbedding for dense vectors or a fastembed::SparseTextEmbedding for sparse ones. The fastembed documentation lists the dense models, with their dimensions, under fastembed::EmbeddingModel.

The excerpt uses fastembed::InitOptions, which fastembed 5 keeps as a deprecated alias of fastembed::TextInitOptions. Use fastembed::TextInitOptions in new code to avoid the deprecation warning. Its options also set the cache directory (fastembed::TextInitOptions::with_cache_dir) and cap the ONNX (Open Neural Network Exchange) Runtime threads (fastembed::TextInitOptions::with_intra_threads), which default to every core.

Batch size

How the pipeline batch size and the storage batch size interact is covered in Embedding. The FastEmbed side has one cost to keep in mind: FastEmbed pads every text in a batch to the longest one, so long texts in a large batch cost memory.

The docs-qa showcase sets 32 on both embedding steps and leaves FastEmbed at its default of 256. FastEmbed never sees more texts than the pipeline hands over, so its own batch size only matters when it is smaller:

indexing::Pipeline::from_loader(loader)
    .then_chunk(ChunkMarkdown::from_chunk_range(
        MIN_CHUNK_CHARS..MAX_CHUNK_CHARS,
    ))
    // A short title and keywords per chunk; both end up in the embedded text and help the
    // sparse (keyword) side of hybrid search in particular.
    .then(
        MetadataTitle::from_client(llm.clone())
            .prompt_template(TITLE_PROMPT)
            .build()?,
    )
    .then(
        MetadataKeywords::from_client(llm.clone())
            .prompt_template(KEYWORDS_PROMPT)
            .build()?,
    )
    .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_store_with(qdrant.clone())
    .run()
    .await

Source: examples/docs-qa/src/index.rs#L50-L70

ask-your-codebase sets 32 on both the pipeline and the FastEmbed builder, because its embedded texts run up to 2048 tokens. To limit memory, lower the pipeline batch size first.

Reranking

Rerank is a response transformer that scores retrieved documents with a local cross-encoder and keeps the best ones. It implements TransformResponse. docs-qa builds it next to its answer generator:

let reranker = Rerank::builder()
    .top_k(CONTEXT_CHUNKS)
    .document_template(Some(RERANK_TEMPLATE.to_string()))
    .build()?;

let answerer = answers::Simple::builder()
    .client(llm)
    .prompt_template(ANSWER_PROMPT.into())
    .document_template(DOCUMENT_TEMPLATE)
    .build()?;

Source: examples/docs-qa/src/query.rs#L73-L82

The default model is BGE reranker base. To use another one, pass a fastembed::TextRerank wrapped in Arc<tokio::sync::Mutex<_>> to Rerank::builder().model(...). model_batch_size sets the reranker’s batch size, which otherwise uses FastEmbed’s default of 256.

For where Rerank sits in the query pipeline and how to write a rerank template, see Reranking and response transformers.

First run and model cache

The first time a model loads, FastEmbed downloads it from Hugging Face into ./.fastembed_cache in the working directory. Later runs load it from there. Set FASTEMBED_CACHE_DIR to share one cache between projects. If HF_HOME is set, it takes precedence over both, and the models land there. The three docs-qa models (dense, sparse and reranker) take about 2.3 GB. The ask-your-codebase model takes about 600 MB.

The ONNX Runtime library is downloaded once at build time, because the fastembed feature flag turns on fastembed’s ort-download-binaries feature. The first build therefore needs network access, and so does the first run.

Gotchas

A FastEmbed keeps its model behind an Arc<tokio::sync::Mutex<_>>. Clones share the same model, so calls on one instance and its clones run one at a time. A higher with_concurrency on Embed does not speed up embedding.
The model runs inside the async call, not on a blocking thread. A large batch occupies one Tokio worker thread until it finishes.
Indexing and querying must use the same model. See Embedding for what to do when you change it.
Rerank::default(), and Rerank::builder().build() without a model, panic when the default reranker cannot load, for example on a first run without network access. FastEmbed::try_default() returns an error instead.

Reference

Qdrant