# FastEmbed

> Embed chunks and queries with dense and sparse FastEmbed models, and rerank retrieved documents, on your own CPU without an API key.

You embed chunks and queries with local dense and sparse models, pick a model that fits your data, and rerank retrieved documents, all on your CPU without an API key. Turn on the `fastembed` feature flag:

```sh
cargo add swiftide@0.34 --features fastembed
```

## Default models

The hybrid search example builds one dense and one sparse embedder with the default models:

```rust title="examples/hybrid_search.rs"
// Ensure all batching is consistent
let batch_size = 64;

let fastembed_sparse = FastEmbed::try_default_sparse().unwrap().to_owned();
let fastembed = FastEmbed::try_default().unwrap().to_owned();

// Set up openai with the mini model, which is great for indexing
let openai = openai::OpenAI::builder()
    .default_prompt_model("gpt-4o-mini")
    .build()
    .unwrap();

// Set up qdrant and use the combined fields (metadata + chunks) for both sparse and dense
// vectors
let qdrant = Qdrant::builder()
    .batch_size(batch_size)
    .vector_size(384)
    .with_vector(EmbeddedField::Combined)
    .with_sparse_vector(EmbeddedField::Combined)
    .collection_name("swiftide-hybrid-example")
    .build()?;
```

Source: [`examples/hybrid_search.rs#L23-L43`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/hybrid_search.rs#L23-L43)

The excerpt also builds an OpenAI client and Qdrant storage. To compile it as is, add the `openai` and `qdrant` feature flags next to `fastembed`.

`FastEmbed::try_default()` loads BGE (BAAI General Embedding) small en v1.5, a dense model with 384 dimensions. `FastEmbed::try_default_sparse()` loads SPLADE++ (Sparse Lexical and Expansion model, `prithivida/Splade_PP_en_v1`), a sparse model. Both return a `Result`, because loading the model can fail.

`FastEmbed` implements `EmbeddingModel` and `SparseEmbeddingModel`. You pass the dense instance to the `Embed` batch transformer in the indexing pipeline and to `query_transformers::Embed` in the query pipeline. The sparse instance goes to `SparseEmbed` on both sides. Each instance holds one model. Passing a sparse instance to `Embed` fails with the permanent error "Expected dense model, got sparse", and a dense instance in `SparseEmbed` fails with "Expected sparse model, got dense".

Your storage's vector size must match the model: 384 for BGE small.

## Model choice

The ask-your-codebase showcase indexes source code, so it picks a model trained on code:

```rust title="examples/ask-your-codebase/src/embedding.rs"
/// A code-aware embedding model that runs on the CPU and needs no API key.
///
/// The model (~600MB) is downloaded on first use.
pub fn local_model() -> Result<FastEmbed> {
    let model = TextEmbedding::try_new(
        InitOptions::new(EmbeddingModel::JinaEmbeddingsV2BaseCode).with_max_length(MAX_TOKENS),
    )?;

    FastEmbed::builder()
        .embedding_model(model)
        .batch_size(BATCH_SIZE)
        .build()
}
```

Source: [`examples/ask-your-codebase/src/embedding.rs#L18-L30`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/embedding.rs#L18-L30)

`jina-embeddings-v2-base-code` produces 768-dimensional vectors. `fastembed::TextInitOptions::with_max_length` raises the token limit to 2048 for each embedded text. In ask-your-codebase that text is a chunk of up to 4096 bytes plus its generated questions and answers. FastEmbed truncates longer texts without an error. The default limit is 512 tokens, or less if the model's own limit is lower.

Swiftide does not re-export the `fastembed` crate. To name a model, add it yourself at the major version Swiftide uses:

```sh
cargo add fastembed@5 --no-default-features
```

`FastEmbedBuilder::embedding_model` accepts a `fastembed::TextEmbedding` for dense vectors or a `fastembed::SparseTextEmbedding` for sparse ones. The fastembed documentation lists the dense models, with their dimensions, under `fastembed::EmbeddingModel`.

The excerpt uses `fastembed::InitOptions`, which fastembed 5 keeps as a deprecated alias of `fastembed::TextInitOptions`. Use `fastembed::TextInitOptions` in new code to avoid the deprecation warning. Its options also set the cache directory (`fastembed::TextInitOptions::with_cache_dir`) and cap the ONNX (Open Neural Network Exchange) Runtime threads (`fastembed::TextInitOptions::with_intra_threads`), which default to every core.

## Batch size

How the pipeline batch size and the storage batch size interact is covered in [Embedding](/indexing/embedding/#batch-sizes). The FastEmbed side has one cost to keep in mind: FastEmbed pads every text in a batch to the longest one, so long texts in a large batch cost memory.

The docs-qa showcase sets 32 on both embedding steps and leaves FastEmbed at its default of 256. FastEmbed never sees more texts than the pipeline hands over, so its own batch size only matters when it is smaller:

```rust title="examples/docs-qa/src/index.rs"
indexing::Pipeline::from_loader(loader)
    .then_chunk(ChunkMarkdown::from_chunk_range(
        MIN_CHUNK_CHARS..MAX_CHUNK_CHARS,
    ))
    // A short title and keywords per chunk; both end up in the embedded text and help the
    // sparse (keyword) side of hybrid search in particular.
    .then(
        MetadataTitle::from_client(llm.clone())
            .prompt_template(TITLE_PROMPT)
            .build()?,
    )
    .then(
        MetadataKeywords::from_client(llm.clone())
            .prompt_template(KEYWORDS_PROMPT)
            .build()?,
    )
    .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_store_with(qdrant.clone())
    .run()
    .await
```

Source: [`examples/docs-qa/src/index.rs#L50-L70`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa/src/index.rs#L50-L70)

ask-your-codebase sets 32 on both the pipeline and the `FastEmbed` builder, because its embedded texts run up to 2048 tokens. To limit memory, lower the pipeline batch size first.

## Reranking

`Rerank` is a response transformer that scores retrieved documents with a local cross-encoder and keeps the best ones. It implements `TransformResponse`. docs-qa builds it next to its answer generator:

```rust title="examples/docs-qa/src/query.rs"
let reranker = Rerank::builder()
    .top_k(CONTEXT_CHUNKS)
    .document_template(Some(RERANK_TEMPLATE.to_string()))
    .build()?;

let answerer = answers::Simple::builder()
    .client(llm)
    .prompt_template(ANSWER_PROMPT.into())
    .document_template(DOCUMENT_TEMPLATE)
    .build()?;
```

Source: [`examples/docs-qa/src/query.rs#L73-L82`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa/src/query.rs#L73-L82)

The default model is BGE reranker base. To use another one, pass a `fastembed::TextRerank` wrapped in `Arc<tokio::sync::Mutex<_>>` to `Rerank::builder().model(...)`. `model_batch_size` sets the reranker's batch size, which otherwise uses FastEmbed's default of 256.

For where `Rerank` sits in the query pipeline and how to write a rerank template, see [Reranking and response transformers](/querying/reranking-and-response-transformers/).

## First run and model cache

The first time a model loads, FastEmbed downloads it from Hugging Face into `./.fastembed_cache` in the working directory. Later runs load it from there. Set `FASTEMBED_CACHE_DIR` to share one cache between projects. If `HF_HOME` is set, it takes precedence over both, and the models land there. The three docs-qa models (dense, sparse and reranker) take about 2.3 GB. The ask-your-codebase model takes about 600 MB.

The ONNX Runtime library is downloaded once at build time, because the `fastembed` feature flag turns on fastembed's `ort-download-binaries` feature. The first build therefore needs network access, and so does the first run.

## Gotchas

- A `FastEmbed` keeps its model behind an `Arc<tokio::sync::Mutex<_>>`. Clones share the same model, so calls on one instance and its clones run one at a time. A higher `with_concurrency` on `Embed` does not speed up embedding.
- The model runs inside the async call, not on a blocking thread. A large batch occupies one Tokio worker thread until it finishes.
- Indexing and querying must use the same model. See [Embedding](/indexing/embedding/) for what to do when you change it.
- `Rerank::default()`, and `Rerank::builder().build()` without a model, panic when the default reranker cannot load, for example on a first run without network access. `FastEmbed::try_default()` returns an error instead.

## Reference

- [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html)
- [`FastEmbedBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbedBuilder.html)
- [`Rerank`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.Rerank.html)
- [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html)
- [`SparseEmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SparseEmbeddingModel.html)
- [`TransformResponse`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.TransformResponse.html)
- [`transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html)
- [`transformers::SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html)
- [`query_transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.Embed.html)
- [`query_transformers::SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.SparseEmbed.html)
- [fastembed crate documentation](https://docs.rs/fastembed/5)
- [Embedding](/indexing/embedding/)
- [Reranking and response transformers](/querying/reranking-and-response-transformers/)
- [Feature flags](/reference/feature-flags/)

## Next

[Qdrant](/integrations/qdrant/)