Hybrid search and reranking

You build two query pipelines over Qdrant: one that fuses dense and sparse vector search, and one that reranks retrieved documents with a local cross-encoder before the large language model (LLM) answers. Both are runnable examples in the swiftide repository: hybrid-search and reranking.

Setup

To run the examples, clone the repository and start Qdrant with its gRPC port 6334 exposed (Running Qdrant covers the options). Set OPENAI_API_KEY. Run from the repository root, because the examples read swiftide-core/ and README.md by relative path:

git clone https://github.com/bosun-ai/swiftide && cd swiftide
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
cargo run -p swiftide-examples --example hybrid-search
cargo run -p swiftide-examples --example reranking

To use the same code in your own project, add Swiftide with the feature flags the examples use, plus the async runtime and logging they call from main:

cargo add swiftide@0.34 --features openai,qdrant,fastembed,tree-sitter
cargo add tokio --features full
cargo add tracing-subscriber

tree-sitter is only needed for the code chunker in the hybrid example.

Hybrid indexing

// Ensure all batching is consistent
let batch_size = 64;

let fastembed_sparse = FastEmbed::try_default_sparse().unwrap().to_owned();
let fastembed = FastEmbed::try_default().unwrap().to_owned();

// Set up openai with the mini model, which is great for indexing
let openai = openai::OpenAI::builder()
    .default_prompt_model("gpt-4o-mini")
    .build()
    .unwrap();

// Set up qdrant and use the combined fields (metadata + chunks) for both sparse and dense
// vectors
let qdrant = Qdrant::builder()
    .batch_size(batch_size)
    .vector_size(384)
    .with_vector(EmbeddedField::Combined)
    .with_sparse_vector(EmbeddedField::Combined)
    .collection_name("swiftide-hybrid-example")
    .build()?;

Source: examples/hybrid_search.rs#L23-L43

FastEmbed::try_default() loads a local dense embedding model with 384 dimensions, which is why the collection uses vector_size(384). FastEmbed::try_default_sparse() loads SPLADE, a sparse model that weights individual terms. A dense vector matches on meaning. A sparse vector also rewards exact words, such as a type or function name.

with_vector and with_sparse_vector both name EmbeddedField::Combined: the chunk and its metadata embedded as one text. Qdrant stores the two vectors on the same point.

indexing::Pipeline::from_loader(FileLoader::new("swiftide-core/").with_extensions(&["rs"]))
    // Chunk fairly large as the context window is big
    .then_chunk(ChunkCode::try_for_language_and_chunk_size(
        "rust",
        10..2048,
    )?)
    // Generate metadata on the code chunks to increase our chances of finding the right code
    .then(MetadataQACode::from_client(openai.clone()).build().unwrap())
    .then_in_batch(
        transformers::SparseEmbed::new(fastembed_sparse.clone()).with_batch_size(batch_size),
    )
    .then_in_batch(transformers::Embed::new(fastembed.clone()).with_batch_size(batch_size))
    .then_store_with(qdrant.clone())
    .run()
    .await?;

Source: examples/hybrid_search.rs#L47-L61

ChunkCode splits the Rust files in swiftide-core/ with tree-sitter. MetadataQACode asks gpt-4o-mini for five questions and answers per chunk and stores them under the Questions and Answers (code) metadata key. That metadata is part of the combined text, so both vectors include it.

SparseEmbed and Embed are batch transformers. The example uses one batch_size of 64 for both of them and for the Qdrant storage.

Hybrid query

// Use sophisticated model for our query
let openai = openai::OpenAI::builder()
    .default_prompt_model("gpt-4o")
    .build()
    .unwrap();

let query_pipeline = query::Pipeline::from_search_strategy(
    // Return a large amount of documents because we have a large context window
    // By default it uses the Combined fields, no need to configure
    HybridSearch::default().with_top_n(20).with_top_k(20),
)
// Generate subquestions on the initial query to increase our query coverage
.then_transform_query(query_transformers::GenerateSubquestions::from_client(
    openai.clone(),
))
// Generate the same embeddings we used for indexing
.then_transform_query(query_transformers::Embed::from_client(fastembed.clone()))
.then_transform_query(query_transformers::SparseEmbed::from_client(
    fastembed_sparse.clone(),
))
.then_retrieve(qdrant.clone())
// Answer with Simple, which either takes the documents as is (in this case), or any
// transformations applied after querying
.then_answer(answers::Simple::from_client(openai.clone()));

Source: examples/hybrid_search.rs#L65-L88

HybridSearch::default() searches the Combined dense and sparse vectors. with_top_n(20) limits each sub-search to 20 candidates. with_top_k(20) limits the fused result.

The Qdrant retriever needs both query embeddings. query_transformers::Embed sets the dense one and query_transformers::SparseEmbed the sparse one, each with the model used for indexing. If either is missing, retrieval fails with No embedding for query or No sparse embedding for query.

GenerateSubquestions runs first. Its default prompt asks gpt-4o for the original question plus five extra questions. That list becomes the current query, and both embeddings are made from it.

Qdrant fuses the two searches with Reciprocal Rank Fusion (RRF); see Retrieval and search strategies.

let answer = query_pipeline
    .query("What are the different pipelines in Swiftide and how do they work? Provide an elaborate answer with examples.")
    .await
    .unwrap();

println!("{}", answer.answer());

Source: examples/hybrid_search.rs#L92-L97

Stores and filters

Qdrant implements Retrieve for HybridSearch with a Qdrant Filter type, so type inference picks that filter type for HybridSearch::default(). Pass a filter with HybridSearch::from_filter or with_filter, and Qdrant applies it to both sub-searches. Retrieval and search strategies shows a filter.

Duckdb, behind the duckdb feature flag, also implements Retrieve for HybridSearch, without a filter. It combines cosine similarity with a full-text keyword score on the query text, so it needs only the dense query embedding. LanceDB and PgVector do not implement it.

Existing collections

Qdrant creates the collection with its named vectors only when the collection does not exist yet. For an existing collection it logs a warning and keeps the old configuration. A collection created without with_sparse_vector stays without a sparse vector. Use a new collection name, as the example does with swiftide-hybrid-example, or delete the old collection first.

Reranking

let openai_client = integrations::openai::OpenAI::builder()
    .default_prompt_model("gpt-4o")
    .build()?;

let fastembed = fastembed::FastEmbed::builder().batch_size(10).build()?;
let reranker = fastembed::Rerank::builder().top_k(5).build()?;

let qdrant = Qdrant::builder()
    .batch_size(50)
    .vector_size(384)
    .collection_name("swiftide-reranking")
    .build()?;

Source: examples/reranking.rs#L25-L36

Rerank is a response transformer: top_k(5) keeps the five best of the retrieved documents. Reranking and response transformers covers its builder fields and templates.

indexing::Pipeline::from_loader(FileLoader::new("README.md"))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..2048))
    .then_in_batch(Embed::new(fastembed.clone()))
    .then_store_with(qdrant.clone())
    .run()
    .await?;

Source: examples/reranking.rs#L40-L45

The indexing pipeline chunks README.md as markdown, embeds the chunks with FastEmbed and stores them in Qdrant. With one dense vector and no sparse vector, Qdrant stores the embedding as the point’s single unnamed vector.

// By default the search strategy is SimilaritySingleEmbedding
// which takes the latest query, embeds it, and does a similarity search
let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::GenerateSubquestions::from_client(
        openai_client.clone(),
    ))
    .then_transform_query(query_transformers::Embed::from_client(fastembed.clone()))
    .then_retrieve(qdrant.clone())
    .then_transform_response(reranker)
    .then_answer(answers::Simple::from_client(openai_client.clone()));

Source: examples/reranking.rs#L49-L58

query::Pipeline::default() uses SimilaritySingleEmbedding, which retrieves at most 10 documents, its default top_k. then_transform_response(reranker) runs after retrieval and before the answer. It scores the retrieved documents and passes the best 5 to answers::Simple.

let result = pipeline
    .query("What is swiftide? Please provide an elaborate explanation")
    .await?;

println!("{:?}", result.answer());

Source: examples/reranking.rs#L62-L66

Gotchas

Retrieve more documents than you keep. Here retrieval returns up to 10 and Rerank keeps 5. If retrieval returns top_k documents or fewer, Rerank only reorders them.
FastEmbed loads its models when you construct them, in FastEmbed::try_default(), try_default_sparse() and Rerank::builder().build(), and downloads them on the first run. A Rerank built without model loads the default model with expect, so a failed download panics at build() instead of returning an error.
The plain Qdrant retriever returns each document’s content Debug-quoted: wrapped in " with escaped newlines. In both examples, Rerank scores and answers::Simple renders that quoted text. docs-qa works around it with a CleanQdrant wrapper; see Qdrant content workaround.

Combined pipeline

The docs-qa showcase puts a reranker behind a hybrid search. Its strategy fetches 20 candidates per sub-search and keeps 20 after fusion:

query::Pipeline::from_search_strategy(
    HybridSearch::default()
        .with_top_n(CANDIDATES)
        .with_top_k(CANDIDATES),
)

Source: examples/docs-qa/src/query.rs#L108-L112

Its reranker keeps 10 of the 20 candidates and scores each one with the page name and chunk title in front of the content:

let reranker = Rerank::builder()
    .top_k(CONTEXT_CHUNKS)
    .document_template(Some(RERANK_TEMPLATE.to_string()))
    .build()?;

let answerer = answers::Simple::builder()
    .client(llm)
    .prompt_template(ANSWER_PROMPT.into())
    .document_template(DOCUMENT_TEMPLATE)
    .build()?;

Source: examples/docs-qa/src/query.rs#L73-L82

The query pipeline embeds the question twice, retrieves the fused candidates and reranks them before answering:

pipeline
    // Hybrid search needs both a dense and a sparse embedding of the question
    .then_transform_query(query_transformers::Embed::from_client(
        self.embedders.dense.clone(),
    ))
    .then_transform_query(query_transformers::SparseEmbed::from_client(
        self.embedders.sparse.clone(),
    ))
    .then_retrieve(self.qdrant.clone())
    // A local cross-encoder reorders the fused candidates; only the best few remain
    .then_transform_response(self.reranker.clone())
    .then_answer(self.answerer.clone())

Source: examples/docs-qa/src/query.rs#L121-L134

self.qdrant is the CleanQdrant wrapper around Qdrant, so the reranker and the answer see the text as it was indexed. The docs-qa showcase covers the indexing side and shows captured answers.

Reference

Rich metadata and multiple vectors