Docs Q&A with evaluation

This tour builds docs-qa, a retrieval-augmented generation (RAG) app with three commands: index, ask and eval. It reads markdown files or an llms-full.txt, and every answer lists the pages and sections it came from. The eval command prints a score table that you can rerun after each retrieval change.

The crate lives in the swiftide repository under examples/docs-qa. Run it with cargo run -p docs-qa at the repository root.

Prerequisites

• Rust 1.97.1, which the swiftide workspace pins in rust-toolchain.toml.
• Qdrant in Docker. Swiftide talks to Qdrant over gRPC, so expose port 6334 as well as 6333. QDRANT_URL (default http://localhost:6334) and QDRANT_API_KEY point the client elsewhere.
• OPENAI_API_KEY for any OpenAI-compatible endpoint. OPENAI_BASE_URL defaults to https://api.openai.com/v1 and OPENAI_MODEL to gpt-5.4-mini. Embeddings and reranking run locally; only model calls go to this endpoint.
• About 2.3GB of disk for the FastEmbed models, downloaded from Hugging Face into .fastembed_cache on first use. Set FASTEMBED_CACHE_DIR to share one cache.
• Optional: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_URL for --langfuse.

The crate turns off default features and enables the feature flags openai, fastembed, qdrant and langfuse. The excerpts derive schemars::JsonSchema, and the loader fetches an llms-full.txt URL with reqwest. To build the same thing in your own crate:

cargo add swiftide@0.34 --no-default-features --features openai,fastembed,qdrant,langfuse
cargo add schemars@1 --features derive
cargo add reqwest@0.13 --no-default-features --features rustls

TLS needs no setup, because swiftide’s HTTP clients bring their own rustls crypto provider (TLS).

Architecture

index writes chunks into one Qdrant collection. ask and eval build the same query pipeline over that collection; eval only adds an evaluator.

Custom loader

examples/docs-qa/src/loader.rs

/// Yields one [`TextNode`] per documentation page, with the page title as metadata.
#[derive(Debug, Clone)]
pub struct DocsLoader {
    sources: Vec<DocsSource>,
}

impl DocsLoader {
    pub fn new(sources: Vec<DocsSource>) -> Self {
        Self { sources }
    }
}

impl Loader for DocsLoader {
    type Output = String;

    fn into_stream(self) -> IndexingStream<String> {
        stream::iter(self.sources)
            .then(DocsSource::pages)
            .map_ok(|pages| stream::iter(pages).map(|page| Ok(page_node(page))))
            .try_flatten()
            .boxed()
            .into()
    }

    // Only needed when the loader is used as a `Box<dyn Loader>`.
    fn into_stream_boxed(self: Box<Self>) -> IndexingStream<String> {
        self.into_stream()
    }
}

fn page_node(page: Page) -> TextNode {
    let mut node = TextNode::new(page.markdown);
    node.path = page.origin.into();
    node.metadata.insert(PAGE_TITLE, page.title.as_str());
    node
}

Source: examples/docs-qa/src/loader.rs#L74-L109

A loader is the start of an indexing pipeline. Loader has one associated type, Output, and into_stream returns an IndexingStream of nodes. Here every page becomes one TextNode, with its origin as node.path and its title under the Page metadata key. A futures stream becomes an IndexingStream with .boxed().into().

into_stream_boxed matters only when you use the loader as a Box<dyn Loader>. The default implementation panics, so implement it on the concrete type.

DocsSource decides how a page is read. MarkdownDir walks a directory with FileLoader and respects .gitignore. MarkdownFile reads one file. LlmsFullFile and LlmsFullUrl split an llms-full.txt into one page per top-level heading.

Hybrid indexing

examples/docs-qa/src/index.rs

indexing::Pipeline::from_loader(loader)
    .then_chunk(ChunkMarkdown::from_chunk_range(
        MIN_CHUNK_CHARS..MAX_CHUNK_CHARS,
    ))
    // A short title and keywords per chunk; both end up in the embedded text and help the
    // sparse (keyword) side of hybrid search in particular.
    .then(
        MetadataTitle::from_client(llm.clone())
            .prompt_template(TITLE_PROMPT)
            .build()?,
    )
    .then(
        MetadataKeywords::from_client(llm.clone())
            .prompt_template(KEYWORDS_PROMPT)
            .build()?,
    )
    .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_store_with(qdrant.clone())
    .run()
    .await

Source: examples/docs-qa/src/index.rs#L50-L70

ChunkMarkdown::from_chunk_range(120..1500) splits each page into chunks of at most 1,499 characters, because the range end is exclusive. It drops chunks whose trimmed text is under 120 bytes, which are mostly badges and stray headings. MetadataTitle stores a title under Title and MetadataKeywords stores keywords under Keywords, so indexing makes two large language model (LLM) calls per chunk. Two batch transformers follow: Embed writes the dense vector and SparseEmbed the sparse one, both from local FastEmbed models. index returns the PipelineStats that run produces. The CLI prints the chunk count of the whole collection instead, which also covers earlier runs.

The metadata transformers use their own Tera templates:

examples/docs-qa/src/index.rs

// Swiftide's default metadata prompts show the expected format inside a code fence, and models
// tend to copy the fence into the metadata. These ask for plain text and use the page title.
// A code-only chunk is titled by the API it shows, because the reranker scores the title too.
const TITLE_PROMPT: &str = "Write a short, descriptive title (at most 10 words) for this \
excerpt from the \"{{ node.metadata.Page }}\" documentation. If the excerpt is mostly code, \
name the API it demonstrates (a macro, trait or builder method), not what the example does. \
Reply with the title only: no quotes, no markdown, no code fences.

{{ node.chunk }}";

const KEYWORDS_PROMPT: &str = "List up to 8 keywords for this excerpt from the \
\"{{ node.metadata.Page }}\" documentation: names of types, crates, features, tools, commands \
and concepts that literally appear in it. Reply with one comma-separated line only: no \
numbering, no markdown, no code fences.

{{ node.chunk }}";

Source: examples/docs-qa/src/index.rs#L24-L39

The default prompts show the expected format inside a code fence, and models tend to copy the fence. These templates ask for plain text and read the page title from node.metadata.Page. A code-only chunk is titled after the API it shows, because the reranker scores the title too.

Store

examples/docs-qa/src/store.rs

let qdrant = Qdrant::builder()
    .batch_size(BATCH_SIZE)
    .vector_size(DENSE_VECTOR_SIZE)
    // One dense and one sparse vector, both over the chunk *and* its metadata
    .with_vector(EmbeddedField::Combined)
    .with_sparse_vector(EmbeddedField::Combined)
    .collection_name(&collection.0)
    .build()?;

Source: examples/docs-qa/src/store.rs#L38-L45

The collection holds one dense and one sparse vector per chunk. Both use EmbeddedField::Combined, which embeds the chunk together with its metadata, so the title and keywords reach the keyword side of the search. The vector size is 384 because the dense model is BGE small, the default of FastEmbed::try_default. The sparse model comes from FastEmbed::try_default_sparse, which loads SPLADE.

Hybrid search and reranking

examples/docs-qa/src/query.rs

query::Pipeline::from_search_strategy(
    HybridSearch::default()
        .with_top_n(CANDIDATES)
        .with_top_k(CANDIDATES),
)

Source: examples/docs-qa/src/query.rs#L108-L112

HybridSearch defaults to 10 for both top_n and top_k. with_top_n and with_top_k take the strategy by value and return it, so they chain onto default() and the result goes straight into the pipeline.

examples/docs-qa/src/query.rs

pipeline
    // Hybrid search needs both a dense and a sparse embedding of the question
    .then_transform_query(query_transformers::Embed::from_client(
        self.embedders.dense.clone(),
    ))
    .then_transform_query(query_transformers::SparseEmbed::from_client(
        self.embedders.sparse.clone(),
    ))
    .then_retrieve(self.qdrant.clone())
    // A local cross-encoder reorders the fused candidates; only the best few remain
    .then_transform_response(self.reranker.clone())
    .then_answer(self.answerer.clone())

Source: examples/docs-qa/src/query.rs#L121-L134

Qdrant’s hybrid retriever needs both a dense and a sparse embedding of the question and fails without either. It runs a dense and a sparse search, each limited to top_n, fuses them with reciprocal rank fusion (RRF), and returns top_k documents: 20 candidates here.

The reranker is a response transformer. It runs between retrieval and the answer generator:

examples/docs-qa/src/query.rs

let reranker = Rerank::builder()
    .top_k(CONTEXT_CHUNKS)
    .document_template(Some(RERANK_TEMPLATE.to_string()))
    .build()?;

let answerer = answers::Simple::builder()
    .client(llm)
    .prompt_template(ANSWER_PROMPT.into())
    .document_template(DOCUMENT_TEMPLATE)
    .build()?;

Source: examples/docs-qa/src/query.rs#L73-L82

Rerank scores each candidate against the original question with a local cross-encoder and keeps top_k of them (default 10). With a document_template, each document is rendered with Tera first, with metadata and content available. answers::Simple renders every kept document with its own document_template and joins them with --- lines into {{ documents }}. Its prompt_template takes a Prompt, hence the .into().

examples/docs-qa/src/query.rs

// The answer prompt receives the question and the rendered documents; each document is rendered
// with its page title and the chunk title written during indexing.
const ANSWER_PROMPT: &str = "You answer questions about Swiftide, a Rust library for building \
LLM applications, using excerpts from its documentation.

Question: {{ question }}

## Instructions
* Use only the documentation excerpts below. If they do not answer the question, say so plainly.
* Start with the direct answer in one or two sentences, then add supporting details.
* Only when the question asks how to do something, include the feature flags, commands or a \
short Rust snippet from the excerpts. Never invent APIs.
* Format the answer as markdown and keep it under 250 words. Do not refer to \"the excerpts\".

## Documentation excerpts

{{ documents }}
";

const DOCUMENT_TEMPLATE: &str = "### {{ metadata.Page | default(value=\"\") }}: \
{{ metadata.Title | default(value=\"\") }}
{{ content }}";

// The reranker scores the same title-first rendering. A chunk that is mostly code says little
// about what it shows, so its page and the title written during indexing go first.
const RERANK_TEMPLATE: &str = "{{ metadata.Page | default(value=\"\") }}: \
{{ metadata.Title | default(value=\"\") }}
{{ content }}";

Source: examples/docs-qa/src/query.rs#L30-L57

Qdrant content workaround

Qdrant retrieval turns the stored content payload into a string with to_string on Qdrant’s own Value type, which formats strings with Debug. Every chunk then comes back wrapped in quotes, with escaped newlines. The example wraps the store in CleanQdrant (src/retrieval.rs), which implements Retrieve for HybridSearch, calls Qdrant, and restores the original text. The reranker, the LLM and the evaluator then see clean markdown.

Evaluation

examples/docs-qa/src/eval.rs

let ragas = Ragas::from_prepared_questions(&questions);

let answered = qa
    .evaluated_pipeline(ragas.clone())
    .query_all(ragas.questions().await)
    .await?;

let ragas_json = ragas.to_json().await;

Source: examples/docs-qa/src/eval.rs#L89-L96

Retrieval-Augmented Generation Assessment (Ragas) is a Python toolkit. Swiftide’s Ragas evaluator only exports data for it and computes no scores.

Ragas::from_prepared_questions takes anything that converts into an EvaluationDataSet. The example implements that conversion for &QuestionSet, which is why the snippet passes &questions. qa.evaluated_pipeline builds the same query pipeline and attaches the evaluator with evaluate_with. query_all runs every question, and to_json exports the dataset with question, answer, contexts and ground_truth.

The evaluator records contexts at retrieval, before any response transformer. ragas.json therefore holds all 20 candidates per question, not the 10 the reranker kept.

LLM judge

To print scores without leaving Rust, a judge splits the ground truth and the answer into claims and checks each one:

examples/docs-qa/src/judge.rs

#[derive(Debug, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
struct Judgement {
    reference_claims: Vec<ReferenceClaim>,
    answer_claims: Vec<AnswerClaim>,
}

#[derive(Debug, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
struct ReferenceClaim {
    // Never read; asking for the claim first makes the model commit to it before judging it.
    #[expect(dead_code)]
    claim: String,
    in_answer: bool,
    in_context: bool,
}

#[derive(Debug, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
struct AnswerClaim {
    #[expect(dead_code)]
    claim: String,
    supported_by_context: bool,
}

Source: examples/docs-qa/src/judge.rs#L91-L114

examples/docs-qa/src/judge.rs

let contexts = answer.contexts().collect::<Vec<_>>().join("\n---\n");
let prompt = Prompt::from(JUDGE_PROMPT)
    .with_context_value("question", answer.question())
    .with_context_value("ground_truth", ground_truth)
    .with_context_value("answer", answer.text())
    .with_context_value("contexts", contexts);

// The JSON schema of `Judgement` is sent along; the reply is parsed into it
let judgement: Judgement =
    retry_transient(|| self.llm.structured_prompt(prompt.clone())).await?;

Source: examples/docs-qa/src/judge.rs#L154-L163

structured_prompt sends the JSON schema of Judgement with the prompt and parses the reply into it. The scores are plain ratios over the claims: faithful (answer claims the reranked context supports), answer recall and context recall. LanguageModelWithBackOff does not wrap StructuredPrompt, so the judge calls the plain OpenAI client and retries transient errors itself, up to five attempts.

Model client

examples/docs-qa/src/models.rs

const DEFAULT_BASE_URL: &str = "https://api.openai.com/v1";

/// A chat model that retries rate limits and other transient errors with exponential backoff.
/// Indexing makes two calls per chunk, concurrently, so it will hit rate limits.
pub type RetryingModel = LanguageModelWithBackOff<OpenAI>;

/// An OpenAI-compatible API: `OpenAI` itself, or any gateway or local server that speaks it.
#[derive(Clone)]
pub struct LlmEndpoint {
    base_url: String,
    api_key: String,
    use_responses_api: bool,
}

impl LlmEndpoint {
    /// Reads `OPENAI_BASE_URL` (optional, defaults to `OpenAI`) and `OPENAI_API_KEY` (required).
    pub fn from_env(use_responses_api: bool) -> Result<Self> {
        let api_key = non_empty_env("OPENAI_API_KEY")
            .context("Set OPENAI_API_KEY to the API key of your OpenAI-compatible endpoint")?;
        let base_url =
            non_empty_env("OPENAI_BASE_URL").unwrap_or_else(|| DEFAULT_BASE_URL.to_string());

        Ok(Self {
            base_url,
            api_key,
            use_responses_api,
        })
    }

    /// A client for `model` on this endpoint.
    pub fn model(&self, model: &str) -> Result<OpenAI> {
        Ok(OpenAI::builder()
            .api_base(&self.base_url)
            .api_key(&self.api_key)
            .default_prompt_model(model)
            .use_responses_api(self.use_responses_api)
            .build()?)
    }

    /// The same client, wrapped in Swiftide's backoff decorator. An unreachable endpoint gives
    /// up after two minutes.
    pub fn retrying_model(&self, model: &str) -> Result<RetryingModel> {
        let backoff = BackoffConfiguration {
            initial_interval_sec: 5,
            max_elapsed_time_sec: 120,
            ..BackoffConfiguration::default()
        };
        Ok(LanguageModelWithBackOff::new(self.model(model)?, backoff))
    }
}

Source: examples/docs-qa/src/models.rs#L30-L79

Answer and metadata calls go through LanguageModelWithBackOff, as in Ask your codebase. Here it retries with a longer backoff than the defaults (1 s start, 60 s cap).

Tracing

examples/docs-qa/src/telemetry.rs

let terminal = tracing_subscriber::fmt::layer()
    .compact()
    .with_writer(std::io::stderr)
    .with_filter(log_filter());

// Reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_URL
let langfuse = send_to_langfuse.then(LangfuseLayer::default);

tracing_subscriber::registry()
    .with(terminal)
    // Langfuse data is emitted at `debug`, so its layer gets its own, wider filter
    .with(
        langfuse
            .clone()
            .map(|layer| layer.with_filter(LevelFilter::DEBUG)),
    )
    .try_init()?;

Source: examples/docs-qa/src/telemetry.rs#L30-L46

The Langfuse layer is set up the same way as in the release notes agent. Langfuse data is emitted at debug level, so its layer gets its own debug-level filter.

Commands

git clone https://github.com/bosun-ai/swiftide && cd swiftide
docker run -d --name docs-qa-qdrant -p 6333:6333 -p 6334:6334 qdrant/qdrant
export OPENAI_API_KEY=sk-...

cargo run -p docs-qa -- index --recreate \
  README.md CONTRIBUTING.md AGENTS.md swiftide-agents-contrib/README.md
cargo run -p docs-qa -- ask "How do I give an agent tools to read and edit files?"
cargo run -p docs-qa -- eval --out examples/docs-qa/ragas.json

Flag	Where	Default	What it does
--model	global	gpt-5.4-mini (OPENAI_MODEL)	Answers questions; default for the other models
--responses-api	global	off	Responses API instead of chat completions
--collection	global	docs-qa	Qdrant collection
--langfuse	global	off	Send traces to Langfuse (LANGFUSE_URL)
--recreate	index	off	Drop the collection first
--metadata-model	index	--model	Writes titles and keywords; pick a cheaper model
--questions	eval	bundled questions.yaml	Questions with ground truths
--out	eval	ragas.json	Where the dataset goes
--judge-model	eval	--model	Grades the answers

Output

The captured run used an OpenAI-compatible gateway in front of AWS Bedrock. claude-haiku-5-5 wrote titles and keywords, and claude-sonnet-5-5 wrote the answers and acted as judge. FastEmbed ran BGE small en v1.5, SPLADE++ and BGE reranker base locally.

Indexed 43 chunks into `docs-qa` in 71.7s

The answer below describes the agents-contrib file tools, which shipped in 0.34. The CLI prints the answer followed by its sources.

Enable the `agents-contrib-file-tools` feature on `swiftide`, then register `ReadFile`, `ExactReplace`, and `ApplyPatch` on your agent with `add_tool`.
...
**Requirement:** the executor must provide a POSIX-like shell with `cat`, `mkdir`, and `rm` commands.

Sources:
  [1] Swiftide › Agent Tools and Hooks: Enabling Contrib File, Planning, and Context Features  (README.md)
  [2] swiftide-agents-contrib › read_file tool: arguments, defaults, and offset/limit behavior  (swiftide-agents-contrib/README.md)
  [3] swiftide-agents-contrib › File Tools: Line-Numbered Reads and Exact-Match Edits  (swiftide-agents-contrib/README.md)
  ...
  [10] Swiftide › Swiftide capabilities for agents, typed task graphs, pipelines, and tracing  (README.md)

#   question                                              ctx faithful   answer  context
                                                                         recall   recall
1   Which crate features and environment variable do I …   10     1.00     1.00     1.00
2   What does the agents-contrib-context feature provid…   10     1.00     1.00     1.00
3   What are the default limits of the read_file tool?     10     1.00     1.00     1.00
4   Which storage and retrieval backends does Swiftide …   10     1.00     1.00     1.00
5   What should I run before opening a pull request?       10     1.00     1.00     1.00
6   What is a task in Swiftide?                            10     1.00     1.00     1.00
7   When does ContextCompression summarize the conversa…   10     1.00     1.00     1.00
8   Which commit message convention does the project fo…   10     1.00     1.00     1.00
9   How do I add a custom tool to an agent with the too…   10     1.00     1.00     1.00
    mean                                                          1.00     1.00     1.00

RAGAS dataset written to examples/docs-qa/ragas.json

ctx is the number of chunks the LLM saw after reranking.

Gotchas

• A chunk that is mostly code ranks poorly when the reranker scores only its text. Render page and chunk title into the reranker’s document_template, and title code-only chunks after the API they show.
• The scores come from one judge over nine questions. Treat them as a regression signal for this corpus, not as a benchmark.
• --responses-api failed against the gateway used for the captured run, which returns a field the async-openai client rejects.
• Ragas exports rows in hash-map order, so eval sorts them back into question order before writing.

Concept pages

• Indexing: Loading, Chunking, Transformers and metadata, Embedding
• Storage: Storage, Qdrant, FastEmbed
• Querying: Retrieval and search strategies, Reranking and response transformers, Answering, Evaluation
• Models and tracing: Structured output, Langfuse
• Recipe: Hybrid search and reranking

Reference

• Loader
• FileLoader
• TextNode
• EmbeddedField
• IndexingStream
• ChunkMarkdown
• MetadataTitle
• MetadataKeywords
• Embed and SparseEmbed (indexing)
• Qdrant
• FastEmbed
• Rerank
• HybridSearch
• Embed and SparseEmbed (query)
• answers::Simple
• Ragas and EvaluationDataSet
• Prompt
• Retrieve
• StructuredPrompt
• LanguageModelWithBackOff
• BackoffConfiguration
• OpenAI
• LangfuseLayer

Next

Agent control recipes