Reranking and response transformers

Your query pipeline fetches twenty candidate documents, keeps the ten a local reranker scores highest, and answers from those. You can also summarize the documents with a large language model (LLM) or filter them with your own response transformer.

FastEmbed reranker

The docs-qa showcase builds its reranker next to its answer generator:

let reranker = Rerank::builder()
    .top_k(CONTEXT_CHUNKS)
    .document_template(Some(RERANK_TEMPLATE.to_string()))
    .build()?;

let answerer = answers::Simple::builder()
    .client(llm)
    .prompt_template(ANSWER_PROMPT.into())
    .document_template(DOCUMENT_TEMPLATE)
    .build()?;

Source: examples/docs-qa/src/query.rs#L73-L82

It then places the reranker between retrieval and the answer:

pipeline
    // Hybrid search needs both a dense and a sparse embedding of the question
    .then_transform_query(query_transformers::Embed::from_client(
        self.embedders.dense.clone(),
    ))
    .then_transform_query(query_transformers::SparseEmbed::from_client(
        self.embedders.sparse.clone(),
    ))
    .then_retrieve(self.qdrant.clone())
    // A local cross-encoder reorders the fused candidates; only the best few remain
    .then_transform_response(self.reranker.clone())
    .then_answer(self.answerer.clone())

Source: examples/docs-qa/src/query.rs#L121-L134

Rerank is a response transformer. A response transformer runs after then_retrieve and before then_answer. It takes the query in the Retrieved state and returns it in the same state.

Rerank runs a cross-encoder: a model that reads the question and one document together and returns a relevance score. That is slower than comparing embeddings, so you run it only on the candidates the retriever already found.

The builder has four fields:

top_k: how many documents to keep, as a usize. The default is 10.
document_template: an optional Tera template that renders each document before scoring. The setter takes an Option<String>, so pass Some(..).
model: the FastEmbed TextRerank model, wrapped in Arc<tokio::sync::Mutex<_>>. The default is BGE reranker base.
model_batch_size: an optional batch size. The default is FastEmbed’s own.

Rerank scores every document against query.original(), the question as the user asked it. Query transformers such as GenerateSubquestions replace current(), which the retriever searches with and which answers::Simple also sees. The reranker ignores current(). It replaces the query’s documents with the top_k highest-scoring ones, best first.

Rerank lives in swiftide::integrations::fastembed behind the fastembed feature flag:

cargo add swiftide@0.34 --features fastembed

This is a docs-qa answer, written by claude-sonnet-5-5 and reranked locally with BGE reranker base. The sources list the pages and sections of the kept documents, best first:

Swiftide supports OpenAI and Azure OpenAI, Anthropic, Gemini, OpenRouter, AWS Bedrock Converse API, Groq, Ollama, and Dashscope. Integrations are feature-gated, so you enable a provider by adding its Cargo feature to `swiftide`.
...
Sources:
  [1] Swiftide › Swiftide capabilities for agents, typed task graphs, pipelines, and tracing  (README.md)
  [2] Swiftide › Swiftide: Rust framework for LLM agents, task graphs, and RAG  (README.md)
  [3] Swiftide › Swiftide Integrations Overview by Feature-Gated Category  (README.md)
  ...
  [9] Swiftide › Swiftide agent tool defined with the swiftide::tool macro  (README.md)
  [10] Swiftide › Installing Swiftide and Setting Up OpenAI Integration  (README.md)

The openai feature flag in that answer comes from the tenth source, the Quick Start chunk. With a top_k below 10 it would not have reached the model. Check a few answers before you lower top_k.

Rerank templates

// The answer prompt receives the question and the rendered documents; each document is rendered
// with its page title and the chunk title written during indexing.
const ANSWER_PROMPT: &str = "You answer questions about Swiftide, a Rust library for building \
LLM applications, using excerpts from its documentation.

Question: {{ question }}

## Instructions
* Use only the documentation excerpts below. If they do not answer the question, say so plainly.
* Start with the direct answer in one or two sentences, then add supporting details.
* Only when the question asks how to do something, include the feature flags, commands or a \
short Rust snippet from the excerpts. Never invent APIs.
* Format the answer as markdown and keep it under 250 words. Do not refer to \"the excerpts\".

## Documentation excerpts

{{ documents }}
";

const DOCUMENT_TEMPLATE: &str = "### {{ metadata.Page | default(value=\"\") }}: \
{{ metadata.Title | default(value=\"\") }}
{{ content }}";

// The reranker scores the same title-first rendering. A chunk that is mostly code says little
// about what it shows, so its page and the title written during indexing go first.
const RERANK_TEMPLATE: &str = "{{ metadata.Page | default(value=\"\") }}: \
{{ metadata.Title | default(value=\"\") }}
{{ content }}";

Source: examples/docs-qa/src/query.rs#L30-L57

RERANK_TEMPLATE puts the page name and the chunk title, both written during indexing, before the content. A chunk that is mostly code says little about its topic, so the extra context helps it score fairly. The template only changes what the model scores. The documents stay as they were, and the answer generator renders them with its own document_template.

In docs-qa this template fixed one evaluation result. For “What is a task in Swiftide?”, the chunk that lists what a task supports ranked 14th when the reranker scored only chunk text, so it was dropped. The answer missed two features, with answer recall 0.71 and context recall 0.57. With page and title in the template, the chunk made the kept ten and both scores reached 1.00. Evaluating with Ragas explains these scores.

Model download, cache location and load failures are covered in FastEmbed.

Minimal pipeline

The single-file example builds its reranker in its setup:

let openai_client = integrations::openai::OpenAI::builder()
    .default_prompt_model("gpt-4o")
    .build()?;

let fastembed = fastembed::FastEmbed::builder().batch_size(10).build()?;
let reranker = fastembed::Rerank::builder().top_k(5).build()?;

let qdrant = Qdrant::builder()
    .batch_size(50)
    .vector_size(384)
    .collection_name("swiftide-reranking")
    .build()?;

Source: examples/reranking.rs#L25-L36

It then generates subquestions for retrieval, while the reranker still scores against the original question:

// By default the search strategy is SimilaritySingleEmbedding
// which takes the latest query, embeds it, and does a similarity search
let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::GenerateSubquestions::from_client(
        openai_client.clone(),
    ))
    .then_transform_query(query_transformers::Embed::from_client(fastembed.clone()))
    .then_retrieve(qdrant.clone())
    .then_transform_response(reranker)
    .then_answer(answers::Simple::from_client(openai_client.clone()));

Source: examples/reranking.rs#L49-L58

Summaries

// By default the search strategy is SimilaritySingleEmbedding
// which takes the latest query, embeds it, and does a similarity search
let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::GenerateSubquestions::from_client(
        openai_client.clone(),
    ))
    .then_transform_query(query_transformers::Embed::from_client(
        openai_client.clone(),
    ))
    .then_retrieve(qdrant.clone())
    .then_transform_response(response_transformers::Summary::from_client(
        openai_client.clone(),
    ))
    .then_answer(answers::Simple::from_client(openai_client.clone()));

Source: examples/query_pipeline.rs#L39-L52

response_transformers::Summary sends the retrieved documents to an LLM and asks for one summary of them. It stores the reply with transformed_response, which records a transformation event and replaces query.current(). It leaves documents() untouched.

answers::Simple then answers from the summary instead of the rendered documents; see Generating answers. The documents stay on the query, so you can still list sources.

To change the instruction, pass your own prompt_template:

let summary = response_transformers::Summary::builder()
    .client(openai)
    .prompt_template(Prompt::from(
        "Summarize these excerpts as one markdown list. Keep every type name, \
         feature flag and command exactly as written.\n\
         {% for document in documents %}\n\
         ---\n\
         {{ document.metadata.path | default(value=\"unknown\") }}:\n\
         {{ document.content }}\n\
         {% endfor %}",
    ))
    .build()?;

Source: snippets/src/querying/reranking.rs#L75-L86

The template sees only documents, a list of documents with content and metadata. The question is not in its context.

You can chain response transformers. Put Rerank before Summary and the LLM summarizes only the kept documents.

Custom response transformers

/// Keeps the first document of each file, in retrieval order, for at most `max` files.
#[derive(Clone)]
pub struct OnePerFile {
    pub max: usize,
}

#[async_trait]
impl TransformResponse for OnePerFile {
    async fn transform_response(
        &self,
        mut query: Query<states::Retrieved>,
    ) -> Result<Query<states::Retrieved>> {
        let mut seen_paths = HashSet::new();
        let documents = query.documents_mut();

        documents.retain(|document| {
            let path = document
                .metadata()
                .get("path")
                .and_then(|value| value.as_str())
                .unwrap_or_default();
            seen_paths.insert(path.to_string())
        });
        documents.truncate(self.max);

        Ok(query)
    }
}

Source: snippets/src/querying/reranking.rs#L15-L42

TransformResponse has one async method, transform_response. Implement it with #[async_trait], from the async-trait crate or swiftide::reexports::async_trait. The trait requires Clone through DynClone, so derive it.

This transformer edits the list in place through documents_mut. It keeps the first document per path metadata value, in the order the documents arrive, and caps the list at max. Documents without a path share one empty key, so only the first of them is kept. It does not call transformed_response, so answers::Simple still renders the documents one by one. Call transformed_response only when your transformer produces text for the answer to use in place of the documents, as Summary does.

Place it after then_retrieve:

let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::Embed::from_client(openai.clone()))
    .then_retrieve(qdrant)
    .then_transform_response(OnePerFile { max: 8 })
    .then_answer(answers::Simple::from_client(openai));

Source: snippets/src/querying/reranking.rs#L48-L52

For a small change, a closure works too. It must be synchronous, take a Query<states::Retrieved> and return anyhow::Result<Query<states::Retrieved>>. It must also be Clone + Send + Sync, so capture only values that are themselves Clone:

let pipeline = query::Pipeline::default()
    .then_transform_query(query_transformers::Embed::from_client(openai.clone()))
    .then_retrieve(qdrant)
    .then_transform_response(|mut query: Query<states::Retrieved>| {
        query.documents_mut().truncate(5);
        Ok(query)
    })
    .then_answer(answers::Simple::from_client(openai));

Source: snippets/src/querying/reranking.rs#L60-L67

Gotchas

In 0.33 the plain Qdrant retriever returns Debug-quoted content, so Rerank scores the quoted text. See the gotcha in Retrieval and search strategies for the docs-qa workaround.
Rerank changes neither current() nor the transformation history. The Retrieved events in history() hold copies of the documents as the retriever returned them. Comparing them with documents() shows what was dropped only if no wrapper edited the documents after retrieval and the query retrieved once.
The Ragas evaluator records contexts before Rerank runs, so its dataset holds the retrieved candidates, not the kept documents; see The dataset. A custom evaluator can read the reranked documents at its second call, after the answer.
To use another rerank model, build a TextRerank yourself, wrap it in Arc::new(tokio::sync::Mutex::new(..)) and pass it to model. That needs direct fastembed and tokio dependencies, because Swiftide does not re-export fastembed.
Clones of a Rerank share one model behind a Tokio mutex. Scoring is synchronous CPU work, so concurrent queries, such as those from query_all, take turns at the rerank step.
If retrieval returns top_k documents or fewer, Rerank only reorders them. Retrieve more candidates than you keep.

Reference

Rerank
RerankBuilder
response_transformers::Summary
TransformResponse
Query
Document
query::Pipeline
answers::Simple
The docs-qa showcase and the FastEmbed integration

Generating answers