Indexing and querying: streaming indexing pipelines and query pipelines (RAG) # Caching > Make re-indexing incremental with a Redb, Redis or DuckDB node cache that skips files whose path and content have not changed. Add a node cache to your indexing pipeline and a second run skips every file whose path and content have not changed, before any large language model (LLM) call or embedding is made. ## Redb cache The [ask-your-codebase](/showcases/ask-your-codebase/) showcase keeps its node cache in an embedded Redb file next to the code it indexes: examples/ask-your-codebase/src/codebase.rs ```rust /// Remembers which files were fully indexed, so unchanged files are skipped next time. pub fn node_cache(&self) -> Result { std::fs::create_dir_all(&self.data_dir)?; Redb::builder() .database_path(self.data_dir.join("cache.redb")) .table_name("indexed_files") .build() } /// Embedded vector store with one vector per chunk and the source path next to it. pub fn vector_store(&self) -> Result { let uri = self.lancedb_dir(); LanceDB::builder() .uri(uri.to_str().context("Index path is not valid UTF-8")?) .table_name("code_chunks") .vector_size(embedding::DIMENSIONS) .with_vector(EmbeddedField::Combined) .with_metadata(PATH_FIELD) .build() } ``` `Redb::builder()` takes a `database_path` and creates the file if it does not exist. It does not create missing parent directories, so the showcase calls `create_dir_all` first. You can pass an open `redb::Database` with `database` instead of a path. `table_name` defaults to `swiftide`. The second function builds the LanceDB storage, covered on [Storage](/indexing/storage/). The pipeline adds the cache with `filter_cached`: examples/ask-your-codebase/src/index.rs ```rust let stats = indexing::Pipeline::from_loader( FileLoader::new(codebase.root()).with_extensions(&languages.extensions()), ) .filter(test_code::not_test_files(codebase.clone())) // Drop unit tests before the cache, which keys on path and content. Changing the content // after it would make the cache miss on every run. .then(test_code::without_unit_tests) .then(files_found.count()) // Skip files whose path and content are unchanged since the last successful run .filter_cached(codebase.node_cache()?) .then(files_changed.count()) .then(record_relative_path(codebase.clone())) // Split along syntax boundaries (functions, impls, ...) with tree-sitter .then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?) // Ask the LLM which questions each chunk answers; stored as metadata and embedded .then( MetadataQACode::from_client(llm) .prompt_template(QUESTIONS_PROMPT) .num_questions(QUESTIONS_PER_CHUNK) .concurrency(concurrent_requests) .build()?, ) // Embed chunk + metadata locally, in batches .then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE)) .then_store_with(codebase.vector_store()?) .run() .await?; ``` `filter_cached` drops every node the cache knows. Every other node continues to the chunker, the LLM and the embedder. The first run indexes the `swiftide-agents` crate from scratch, with `claude-haiku-5-5` writing the questions and FastEmbed computing the embeddings, as described in [Output](/indexing/overview/#output): ```text $ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5 Indexing Rust files in swiftide-agents ... Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s ``` The second run, with no file changed: ```text $ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5 Indexing Rust files in swiftide-agents ... Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s ``` The cache skips all 14 files, so no prompts are sent and nothing is embedded. ## Cache keys The cache key is a node id: a UUID (universally unique identifier) computed from the node’s path and chunk, not its metadata. The built-in chunkers set `parent_id` on every chunk to the id of the node that entered the pipeline, the first ancestor, even across several chunkers. Caches use `parent_id` when it is set. One key therefore covers a file and all of its chunks. Each backend puts a prefix in front of the id: | Backend | Key format | Prefix setting | | ------- | ----------------- | ------------------------------------------------------------------------------------------------------- | | Redb | `.` | `cache_key_prefix`, empty by default | | Redis | `:` | second argument of `Redis::try_from_url`, or `cache_key_prefix` on `Redis::builder()`, empty by default | | DuckDB | `.` | `cache_key_prefix`, empty by default | The pipeline records the id a node has when it passes `filter_cached`. At the end it marks the `parent_id` of each finished node, or its own id when `parent_id` is not set. The file is cached only when the two ids match. Three things break the match silently. A step after `filter_cached` that rewrites `chunk` or `path` in place changes the node’s id. The file is then never marked and is processed again on every run, with or without a chunker. Keep content-changing steps in front of `filter_cached`, like the showcase does with `test_code::without_unit_tests`. Steps that only add metadata are safe after it. A custom chunker must keep the parent. The built-in chunkers build chunks with `Node::into_chunks`, which sets `parent_id`. A closure or `ChunkerTransformer` that creates fresh nodes with `TextNode::new` or `Node::builder` leaves `parent_id` empty, so the file is never marked. Derive chunks with `into_chunks` or `Node::build_from_other`. The path is part of the id. The same file loaded through `./src` and through an absolute path has two keys. The showcase canonicalizes the codebase root for this reason. ## When nodes are cached Since 0.33, `run` marks a file in the cache only after the stream has completed and every chunk of that file reached the end. A file with a failed chunk is left out and retried on the next run. Two consequences follow: * If `run` returns an error, nothing from that run is cached. The same holds when the process stops before `run` returns. * A source whose chunker produces zero chunks never reaches the end of the stream. It is never cached, and every step up to the chunker runs for it again on each run. To keep progress when some nodes fail, log and drop the errors so that `run` completes. This snippet needs the `redis`, `qdrant` and `openai` feature flags: ```rust // Change the prefix, for example to "docs-v2", to index every file again let cache = Redis::try_from_url(redis_url, "docs-v1")?; Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"])) .filter_cached(cache) .then_chunk(ChunkMarkdown::from_chunk_range(100..1500)) .then(MetadataQAText::new(openai.clone())) // Log and drop failed chunks so the run completes. Their files stay uncached and are // retried next run; every other file is cached. .log_errors() .filter_errors() .then_in_batch(Embed::new(openai).with_batch_size(32)) .then_store_with( Qdrant::builder() .vector_size(1536) .collection_name("docs") .build()?, ) .run() .await?; ``` The file behind a failed chunk stays out of the cache and is retried next run. Every other file is cached. [Errors](/indexing/streams-and-concurrency/#errors) covers where to place `log_errors` and `filter_errors`. ## Backends | Backend | Feature flag | Node type | Notes | | ----------------------------------------------------------------------------------------- | ------------ | -------------- | ------------------------------------------------------- | | [`Redb`](https://docs.rs/swiftide/0.34/swiftide/integrations/redb/struct.Redb.html) | `redb` | `TextNode` | Embedded file, no service to run | | [`Redis`](https://docs.rs/swiftide/0.34/swiftide/integrations/redis/struct.Redis.html) | `redis` | any chunk type | Shared between machines; also a storage | | [`Duckdb`](https://docs.rs/swiftide/0.34/swiftide/integrations/duckdb/struct.Duckdb.html) | `duckdb` | any chunk type | Cache table `swiftide_cache` by default; also a storage | Turn on the feature flag for the backend you pick: ```sh cargo add swiftide@0.34 --features redb ``` `Redis::try_from_url` returns a cache for `String` chunks. For another chunk type, use `Redis::try_build_from_url` or `Redis::builder()`. With Redis, the cache is one line. This example also needs the `qdrant`, `openai` and `tree-sitter` feature flags: examples/index_codebase.rs ```rust indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"])) .filter_cached(Redis::try_from_url(redis_url, "swiftide-examples")?) .then(MetadataQACode::new(openai_client.clone())) .then_chunk(ChunkCode::try_for_language_and_chunk_size( "rust", 10..2048, )?) .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10)) .then_store_with( Qdrant::builder() .batch_size(50) .vector_size(1536) .collection_name("swiftide-examples") .build()?, ) .run() .await?; ``` `MetadataQACode` runs before chunking here. That is safe, because it only adds metadata and the id does not change. No backend fails the pipeline on a cache error. `get` logs the error and reports the node as not cached, so an unreachable Redis means every node is processed again. ## Custom cache To write your own cache, implement [`NodeCache`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.NodeCache.html). This one keeps the ids in memory: ```rust use std::{ collections::HashSet, sync::{Arc, Mutex}, }; use async_trait::async_trait; use swiftide::{indexing::TextNode, traits::NodeCache}; use uuid::Uuid; /// Remembers indexed sources in memory, for tests or one-off runs. #[derive(Debug, Clone, Default)] pub struct InMemoryCache { indexed: Arc>>, } /// The same key the pipeline marks: the source id, shared by all chunks of a file. fn source_id(node: &TextNode) -> Uuid { node.parent_id().unwrap_or_else(|| node.id()) } #[async_trait] impl NodeCache for InMemoryCache { type Input = String; async fn get(&self, node: &TextNode) -> bool { match self.indexed.lock() { Ok(indexed) => indexed.contains(&source_id(node)), Err(error) => { tracing::error!("node cache lock poisoned: {error}"); false } } } async fn set(&self, node: &TextNode) { self.set_by_id(source_id(node)).await; } async fn set_by_id(&self, id: Uuid) { match self.indexed.lock() { Ok(mut indexed) => { indexed.insert(id); } Err(error) => tracing::error!("node cache lock poisoned: {error}"), } } } ``` Implement it with `#[async_trait]`, declare `type Input` as the chunk type, and make the type `Send`, `Sync`, `Debug` and `Clone`. `get` returns `bool`, and `set` and `set_by_id` return nothing, so log errors instead of returning them. Key on `parent_id`, falling back to `id()`, so that `get`, `set` and `set_by_id` agree. The pipeline calls `set_by_id` with the source id when it marks a file. It is a required method since 0.33, so existing custom caches must add it. `clear` is optional, and its default implementation panics. ## Cache invalidation `NodeCache::clear` empties a cache, but each backend does it differently: * Redb deletes its table. * DuckDB drops its cache table and does not recreate it. Every later lookup on the same `Duckdb` logs an error and misses, so build a new `Duckdb` before the next run. * Redis sends `DEL` with the pattern `*`. Redis treats that as a literal key name, so the call removes nothing. With an empty prefix, `clear` returns an error instead. Change the prefix, for example from `docs-v1` to `docs-v2` in the partial-progress snippet. A new prefix works on every backend. Use it to index everything again, for example after you change a prompt or the embedding model. Clearing the cache does not touch the storage. A changed file’s new chunks get new ids and are stored next to the old ones, which stay in LanceDB until you delete the `.ask-your-codebase/` directory. ## Reference * [`NodeCache`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.NodeCache.html) * [`Pipeline::filter_cached`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.filter_cached) * [`Pipeline::log_errors`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.log_errors) * [`Pipeline::filter_errors`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.filter_errors) * [`Node`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html) * [`Redb`](https://docs.rs/swiftide/0.34/swiftide/integrations/redb/struct.Redb.html) * [`Redis`](https://docs.rs/swiftide/0.34/swiftide/integrations/redis/struct.Redis.html) * [`Duckdb`](https://docs.rs/swiftide/0.34/swiftide/integrations/duckdb/struct.Duckdb.html) * [Feature flags](/reference/feature-flags/) * [ask-your-codebase source](https://github.com/bosun-ai/swiftide/tree/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase) ## Next [Storage](/indexing/storage/) # Chunking > Split source code with tree-sitter and markdown or plain text by characters, write a chunker as a closure, and keep path and metadata on every chunk. After this page your indexing pipeline splits code along syntax boundaries, markdown and text by characters, and any other format with a closure, with chunks sized for your embedder. A chunker takes one node and returns many; you add it with `then_chunk`. ## Source code `ChunkCode` needs the `tree-sitter` [feature flag](/reference/feature-flags/): ```sh cargo add swiftide@0.34 --features tree-sitter ``` ```rust // At most 1500 bytes per chunk, the default let python = ChunkCode::try_for_language("python")?; // At most 2000 bytes per chunk; chunks of 200 bytes or less are dropped let rust = ChunkCode::try_for_language_and_chunk_size(SupportedLanguages::Rust, 200..2000)? .with_concurrency(4); Pipeline::from_loader(FileLoader::new("src").with_extensions(&["rs"])) .then_chunk(rust) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `ChunkCode` parses each file with tree-sitter and splits it along syntax boundaries such as functions, `impl` blocks and classes. It adds top-level items to a chunk. It closes the chunk when the next item would bring it to the maximum or past it. An item that is too large on its own is split along its children. The size is a `ChunkSize`, and it counts bytes. A number such as `1500` is a maximum. A range such as `200..2000` adds a minimum: a chunk of 200 bytes or less is dropped. `try_for_language` uses a maximum of 1500 bytes. The language is a `SupportedLanguages` value or a name such as `"python"`; an unknown name returns an error. `with_concurrency` sets how many files are chunked at once. Without it the chunker uses the pipeline’s concurrency. When tree-sitter finds a syntax error, `ChunkCode` logs a warning and returns the whole file as one chunk, whatever its size. ## Several languages `ChunkCode` handles one language. The ask-your-codebase showcase indexes several in one pipeline with its own chunker: examples/ask-your-codebase/src/languages.rs ```rust /// Splits each file with a tree-sitter `ChunkCode` for the file's language. /// /// `ChunkCode` is built for a single language; this chunker picks the right one per file from /// its extension, so one pipeline can index a polyglot codebase. #[derive(Clone)] pub struct ChunkByLanguage { chunkers: Vec<(SupportedLanguages, ChunkCode)>, } impl ChunkByLanguage { /// Chunks are between `chunk_size.start` and `chunk_size.end` bytes; smaller ones are dropped. pub fn try_new(languages: &SourceLanguages, chunk_size: Range) -> Result { let chunkers = languages .0 .iter() .map(|&language| { let chunker = ChunkCode::try_for_language_and_chunk_size(language, chunk_size.clone())?; Ok((language, chunker)) }) .collect::>()?; Ok(Self { chunkers }) } fn chunker_for(&self, path: &Path) -> Option<&ChunkCode> { let extension = path.extension()?.to_str()?; self.chunkers .iter() .find(|(language, _)| language.file_extensions().contains(&extension)) .map(|(_, chunker)| chunker) } } #[async_trait] impl ChunkerTransformer for ChunkByLanguage { type Input = String; type Output = String; async fn transform_node(&self, node: TextNode) -> IndexingStream { match self.chunker_for(&node.path) { Some(chunker) => chunker.transform_node(node).await, None => anyhow!("No chunker for {}", node.path.display()).into(), } } } ``` The showcase builds it with a range of `10..4096` bytes. The [tree-sitter integration](/integrations/tree-sitter/#several-languages-in-one-pipeline) explains how it picks a language when extensions overlap. A file without a matching chunker becomes an error in the stream, and the first error stops `run`. Load only the extensions you have chunkers for, as the showcase does, or add `log_errors()` and `filter_errors()` ([Streams and concurrency](/indexing/streams-and-concurrency/)). ## Markdown and text ```rust // Aims for 100 characters, at most 1499; chunks under 100 bytes after trimming are dropped let markdown = ChunkMarkdown::from_chunk_range(100..1500); // At most 1000 characters; only blank chunks are dropped let text = ChunkText::from_max_characters(1000); ``` `ChunkMarkdown` and `ChunkText` use the text-splitter crate and count characters, not bytes. `ChunkMarkdown` prefers to split at headings, then at paragraphs and other blocks. `ChunkText` splits at blank lines and line breaks, then at sentences and words. Both default to a maximum of 2056 characters. With `from_chunk_range`, text-splitter treats the start of the range as the size to aim for and the end, exclusive, as a hard limit. A chunk grows past the start only when that keeps a larger unit, such as a section, in one piece. Chunks that are empty after trimming, or shorter than the start, are dropped. That check uses the byte length of the trimmed text, which equals the character count only for ASCII. The [docs-qa showcase](/showcases/docs-qa/) splits documentation pages with `ChunkMarkdown::from_chunk_range(120..1500)`, because shorter chunks were mostly badges and stray headings. ## Closures ```rust let paragraphs = |node: TextNode| { let chunks: Vec = node .chunk .split("\n\n") .map(str::trim) .filter(|paragraph| !paragraph.is_empty()) .map(str::to_owned) .collect(); // Each chunk keeps the path, metadata and embed mode of `node` IndexingStream::iter(node.into_chunks(chunks).map(Ok)) }; Pipeline::from_loader(FileLoader::new("notes").with_extensions(&["txt"])) .then_chunk(paragraphs) .then_store_with(MemoryStorage::default()) .run() .await?; ``` A closure `Fn(TextNode) -> IndexingStream` that is `Clone + Send + Sync` is a chunker. It works on text nodes only. `IndexingStream::iter` turns an iterator of `Result` into a stream. `node.into_chunks` builds one node per chunk from the input node. Every chunk keeps the path, metadata, vectors and embed mode of that node. For anything more involved, or for other node types, implement `ChunkerTransformer` on a struct as the showcase does. ## Node expansion ```rust // A JSON Lines file holds one record per line let records = |node: TextNode| { let lines: Vec = node .chunk .lines() .filter(|line| !line.trim().is_empty()) .map(str::to_owned) .collect(); IndexingStream::iter( node.into_chunks(lines) .enumerate() .map(|(index, mut record)| { record.metadata.insert("Record", index); Ok(record) }), ) }; Pipeline::from_loader(FileLoader::new("exports").with_extensions(&["jsonl"])) .then_expand(records) .then_chunk(ChunkText::from_max_characters(1000)) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `then_expand` runs the same `ChunkerTransformer` as `then_chunk`; only its tracing span is named differently. Use it when one node holds several records, as in this JSON Lines file, and use `then_chunk` when you cut content to size. ## Metadata and parents Metadata that a node carries before chunking is copied to every chunk. The showcase stores each file’s relative path as metadata before it chunks, so every chunk can cite its file. The same holds for metadata written by a large language model (LLM). Put a transformer such as `MetadataQACode` before the chunker and every chunk of a file carries the questions for the whole file. Put it after the chunker for metadata per chunk, at one request per chunk. `MetadataQACode` also needs the `tree-sitter` feature flag. Since 0.33 every chunk also carries `parent_id`, the id its source node had when it reached the first chunker. The node cache marks that id when the file’s chunks finish. Two consequences matter while you chunk: * A step between `filter_cached` and the chunker that changes the path or the chunk breaks that match, so the file is never cached. Edit the content before `filter_cached`, as the showcase does. * A file whose chunks all fall under the minimum is never cached and is processed again on every run. [Cache keys](/indexing/caching/#cache-keys) explains the match in full. ## Gotchas * Size chunks for the embedder. With the default embed mode the metadata is embedded together with the chunk, so leave room for it under the model’s input limit. * Chunks come out of `then_chunk` in no particular order, and chunks of different files interleave. `ChunkCode` sets `offset` on each chunk, a running byte count over the chunks before it in the same file. The text chunkers keep the offset of the input node. * Only `ChunkCode` sees syntax. Running `ChunkText` over code cuts through functions at line or sentence boundaries. * `ChunkCode` cannot split a syntax node that has no children. A single token larger than the maximum, such as a long comment or string, is silently left out of the chunks. ## Reference * [`ChunkCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkCode.html) * [`ChunkSize`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/enum.ChunkSize.html) * [`SupportedLanguages`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/enum.SupportedLanguages.html) * [`ChunkMarkdown`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkMarkdown.html) * [`ChunkText`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkText.html) * [`ChunkerTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChunkerTransformer.html) * [`Pipeline::then_chunk`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_chunk) and [`Pipeline::then_expand`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_expand) * [`Pipeline::filter_errors`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.filter_errors) * [`Node::into_chunks`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html#method.into_chunks) * [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html) * The [ask-your-codebase](/showcases/ask-your-codebase/) and [docs-qa](/showcases/docs-qa/) showcases, the [tree-sitter integration](/integrations/tree-sitter/) and [feature flags](/reference/feature-flags/) ## Next [Transformers and metadata](/indexing/transformers-and-metadata/) # Code-aware indexing > Index a codebase with tree-sitter so each code chunk carries a compressed outline of its file, questions about the code, and the symbols it defines and uses. This page builds an indexing pipeline that splits source code along syntax boundaries and gives every chunk the context of its file: an outline trimmed to what the chunk uses, questions the code answers, and the symbols it defines and uses. Tree-sitter is a parser library with one grammar per language. Swiftide wraps it in a chunker and four transformers behind the `tree-sitter` feature flag. The example below also uses OpenAI, a Redis node cache and Qdrant storage: ```sh cargo add swiftide@0.34 --features tree-sitter,openai,redis,qdrant ``` ## Reduced context pipeline This example indexes a Rust codebase. It sets up an OpenAI client, the Redis URL for the node cache and a chunk size. It needs `OPENAI_API_KEY`, plus Redis and Qdrant running locally: ```rust let openai_client = integrations::openai::OpenAI::builder() .default_embed_model("text-embedding-3-small") .default_prompt_model("gpt-3.5-turbo") .build()?; let redis_url = std::env::var("REDIS_URL") .as_deref() .unwrap_or("redis://localhost:6379") .to_owned(); let chunk_size = 2048; ``` The pipeline gives every chunk an outline of its file, trimmed to what that chunk uses: ```rust indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"])) .filter_cached(Redis::try_from_url( redis_url, "swiftide-examples-codebase-reduced-context", )?) .then( indexing::transformers::OutlineCodeTreeSitter::try_for_language( "rust", Some(chunk_size), )?, ) .then(MetadataQACode::new(openai_client.clone())) .then_chunk(ChunkCode::try_for_language_and_chunk_size( "rust", 10..chunk_size, )?) .then(indexing::transformers::CompressCodeOutline::new( openai_client.clone(), )) .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10)) .then_store_with( Qdrant::builder() .batch_size(50) .vector_size(1536) .collection_name("swiftide-examples-codebase-reduced-context") .build()?, ) .run() .await?; ``` The steps run in this order: 1. `filter_cached` skips files Redis has already seen, keyed on path and content as [Cache keys](/indexing/caching/#cache-keys) explains. 2. `OutlineCodeTreeSitter` parses the whole file and stores an outline in the `Outline` metadata field. An outline is the file with the bodies of functions and methods removed. Signatures, types, imports and comments stay. The second argument skips files smaller than that many bytes. Here it equals the chunk size, because a file that fits in one chunk needs no outline. 3. `MetadataQACode` asks a large language model (LLM) for questions and answers about the file. Its default prompt includes the outline when the node has one. 4. `ChunkCode` splits the file into chunks of more than 10 and up to 2048 bytes. 5. `CompressCodeOutline` asks the LLM to keep only the outline lines the chunk needs and writes the result back to `Outline`. 6. `Embed` embeds the chunks in batches of 10, and Qdrant stores them with vectors of 1536 dimensions, the size of `text-embedding-3-small`. See [Caching](/indexing/caching/) and [Storage](/indexing/storage/) for the node cache and the storage. Chunking copies the metadata of a file to every chunk, so each chunk starts with the full outline and the questions of its file. The default embed mode embeds metadata together with the chunk. Compressing the outline keeps that embedded context small and specific. ## Cost per chunk `MetadataQACode` runs before chunking here, so it sends one prompt per file and every chunk shares the answers. Move it after `then_chunk` to get questions per chunk, at one prompt per chunk. `CompressCodeOutline` sends one prompt per chunk of a file larger than the outline threshold. Chunks of smaller files have no outline and cost nothing. On a large codebase that is still many requests. Lower the concurrency or wrap the client in `LanguageModelWithBackOff` to stay within rate limits. ## Question prompt `MetadataQACode` writes its reply to `Questions and Answers (code)`. The default prompt asks generic questions. The [ask-your-codebase](/showcases/ask-your-codebase/) showcase replaces it with a prompt about behaviour, which matches the questions developers ask: examples/ask-your-codebase/src/index.rs ````rust /// Asks for the questions a developer would ask that a chunk answers. /// /// The default `MetadataQACode` prompt asks generic questions ("What does this code do?"). /// Questions about behaviour, phrased the way a developer would ask them, match real questions /// much better. const QUESTIONS_PROMPT: &str = r" # Task List the questions a developer could ask about this codebase that the code below answers, each with a short answer. They are stored with the code to help a search find it. Focus on behaviour: when and in which order things happen, what starts or ends a process, which conditions and configuration change the outcome, what happens on errors, and which extension points (traits, hooks, builders, options) exist and when they apply. # Constraints - Write {{ questions }} questions and answers. - Phrase questions the way a developer would ask them, e.g. 'When does the agent stop?'. - Name the relevant types, functions and variants in the answers. - Only use what can be derived from the code. - Respond only in this format: Q1: A1: # Code ``` {{ node.chunk }} ``` "; ```` The prompt receives `questions` and the node. The default prompt also renders `{{ outline }}`, which `MetadataQACode` sets when the node has an `Outline` field. The showcase prompt leaves it out, because the showcase builds no outline. Add `{{ outline }}` to your own prompt when you use it in the pipeline above, or the outline does not reach the LLM. [Tree-sitter](/integrations/tree-sitter/#questions-per-chunk) shows how to pass a prompt template and the number of questions. ## References and definitions `MetadataRefsDefsCode` adds the symbols each chunk defines and uses, without an LLM: ```rust indexing::Pipeline::from_loader(FileLoader::new("src").with_extensions(&["rs"])) .then_chunk(ChunkCode::try_for_language_and_chunk_size( "rust", 10..2048, )?) .then(MetadataRefsDefsCode::try_from_language("rust")?) .then_store_with(MemoryStorage::default()) .run() .await?; // The same transformer on one node let refs_defs = MetadataRefsDefsCode::try_from_language("rust")?; let node = refs_defs .transform_node(TextNode::new(r#"fn main() { println!("Hello"); }"#)) .await?; assert_eq!(node.metadata.get(NAME_REFERENCES).unwrap(), "println"); assert_eq!(node.metadata.get(NAME_DEFINITIONS).unwrap(), "main"); ``` It stores the symbols the chunk defines in `Definitions (code)` and the symbols it uses but does not define in `References (code)`. Both lists are sorted, without duplicates, and joined with commas. A field is left out when nothing is found. The constants `NAME_DEFINITIONS` and `NAME_REFERENCES` in `swiftide::indexing::transformers::metadata_refs_defs_code` hold the field names. Run it after chunking, so each chunk lists its own symbols. ## Languages `ChunkCode`, `OutlineCodeTreeSitter` and `MetadataRefsDefsCode` take a language. `MetadataQACode` and `CompressCodeOutline` take only an LLM client. The constructor for references and definitions is `try_from_language`; the others use `try_for_language`. [Tree-sitter](/integrations/tree-sitter/#supported-languages) lists which languages each transformer supports. Perl needs the `tree-sitter-perl` feature flag. ## Code splitter outside a pipeline `CodeSplitter` is the splitter inside `ChunkCode`. Use it directly when you need chunks outside an indexing pipeline, for example in a tool: ```rust // Case-insensitive; "c#", "c-sharp", "cpp" and "c++" parse too let language: SupportedLanguages = "python".parse()?; assert_eq!(language.file_extensions(), ["py"]); // Drop chunks of 100 bytes or less, split anything above 1500 bytes let splitter = CodeSplitter::builder() .try_language(language)? .chunk_size(100..1500) .build()?; let chunks = splitter.split(source)?; // Code with a syntax error comes back whole, as one chunk let broken = splitter.split("def main(:\n pass")?; assert_eq!(broken.len(), 1); ``` `SupportedLanguages` parses names without regard to case, and alternative names such as `"c#"`, `"c-sharp"`, `"cpp"` and `"c++"` work. Parsing an unknown name with `parse()` returns a strum parse error that says `Matching variant not found`. The constructors and `try_language` return the error `Treesitter language not supported` instead. The chunk size is a `ChunkSize`. A number is a maximum in bytes; `ChunkSize::default()` is 1500 bytes. A range sets a minimum and a maximum, and a chunk is kept only when it is longer than the minimum. `CodeSplitter::new(language)` uses the default size. ## Gotchas * The splitter drops chunks at or below the minimum and syntax leaves larger than the maximum, and returns a file with a syntax error as one chunk. [Chunking](/indexing/chunking/) covers these cases. * `OutlineCodeTreeSitter` returns an error for a file with a syntax error. The first error aborts `run`, so add `log_errors()` and `filter_errors()` when your codebase contains files that do not parse. See [Streams and concurrency](/indexing/streams-and-concurrency/). * `OutlineCodeTreeSitter` and `MetadataRefsDefsCode` panic on code in a language they do not handle. In a pipeline the panic happens inside a spawned task, so `run` fails with a join error. * Chunk sizes for code are bytes. `ChunkMarkdown` and `ChunkText` count characters, see [Markdown and text](/indexing/chunking/#markdown-and-text). ## Reference * [`ChunkCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkCode.html) * [`OutlineCodeTreeSitter`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.OutlineCodeTreeSitter.html) * [`CompressCodeOutline`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.CompressCodeOutline.html) * [`MetadataQACode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQACode.html) * [`MetadataRefsDefsCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataRefsDefsCode.html) and the [`metadata_refs_defs_code`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/metadata_refs_defs_code/index.html) module with `NAME_DEFINITIONS` and `NAME_REFERENCES` * [`CodeSplitter`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/struct.CodeSplitter.html) * [`ChunkSize`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/enum.ChunkSize.html) * [`SupportedLanguages`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/enum.SupportedLanguages.html) * [`FileLoader`](https://docs.rs/swiftide/0.34/swiftide/indexing/loaders/struct.FileLoader.html) * [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) * [`TextNode`](https://docs.rs/swiftide/0.34/swiftide/indexing/type.TextNode.html) * [`LanguageModelWithBackOff`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.LanguageModelWithBackOff.html) * [Tree-sitter integration](/integrations/tree-sitter/) and [feature flags](/reference/feature-flags/) ## Next [How querying works](/querying/overview/) # Embedding > Add dense and sparse embeddings to nodes with Embed and SparseEmbed, choose the embedded text with EmbedMode, and store each embedded field as a named vector in Qdrant. Here every chunk in your indexing pipeline ends up with a dense and a sparse vector, stored in a Qdrant collection that a retriever can search. You add an embedder as a batch transformer, choose which text it embeds, and configure the storage to keep the vectors you need. Add the `fastembed` and `qdrant` feature flags: ```sh cargo add swiftide@0.34 --features fastembed,qdrant ``` ## Dense and sparse embeddings The docs-qa showcase gives every chunk a dense and a sparse embedding: examples/docs-qa/src/index.rs ```rust indexing::Pipeline::from_loader(loader) .then_chunk(ChunkMarkdown::from_chunk_range( MIN_CHUNK_CHARS..MAX_CHUNK_CHARS, )) // A short title and keywords per chunk; both end up in the embedded text and help the // sparse (keyword) side of hybrid search in particular. .then( MetadataTitle::from_client(llm.clone()) .prompt_template(TITLE_PROMPT) .build()?, ) .then( MetadataKeywords::from_client(llm.clone()) .prompt_template(KEYWORDS_PROMPT) .build()?, ) .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE)) .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE)) .then_store_with(qdrant.clone()) .run() .await ``` `Embed::new` takes any `EmbeddingModel` and fills the node’s `vectors`. `SparseEmbed::new` takes any `SparseEmbeddingModel` and fills `sparse_vectors`. A dense vector captures what the text means. A sparse vector weights the words in it, which helps keyword-like matches in hybrid search. Both are batch transformers, so you add them with `then_in_batch`. Each batch becomes one model call. `with_batch_size` sets the nodes per batch, with a pipeline default of 256. `with_concurrency` sets how many batches run at the same time. Both work on text nodes (`TextNode`) only. A second `Embed` step replaces the dense vectors of the first one instead of adding to them. ## Embed mode A node’s embed mode decides which texts the embedder sees, each under an embedded field: | `EmbedMode` | Embedded fields | | ------------------------------ | ----------------------------------------------------- | | `SingleWithMetadata` (default) | `Combined` | | `PerField` | `Chunk`, plus one `Metadata(name)` per metadata field | | `Both` | all of the above | `Combined` is the metadata as `key: value` lines, a newline, then the chunk. Metadata keys come in sorted order. `Chunk` is the chunk alone. `Metadata(name)` is the value of one metadata field. Values that are not strings are embedded as JSON (JavaScript Object Notation). `Pipeline::with_embed_mode` sets the mode on every node that passes that point. Chunkers that use `Node::into_chunks` copy the mode to each chunk, so you can set it right after the loader. Qdrant names each vector after its field: `Combined`, `Chunk` or `Metadata: Title`. Sparse vectors add `_sparse`, as in `Combined_sparse`. LanceDB prefixes the column with `vector_`, as in `vector_combined`. ## Named vectors With `EmbedMode::PerField`, every node carries several dense vectors. Qdrant stores each one as a named vector: examples/store_multiple_vectors.rs ```rust indexing::Pipeline::from_loader(FileLoader::new("LICENSE")) .with_concurrency(1) .with_embed_mode(EmbedMode::PerField) .then_chunk(ChunkMarkdown::from_chunk_range(20..2048)) .then(MetadataQAText::new(openai_client.clone())) .then(MetadataSummary::new(openai_client.clone())) .then(MetadataTitle::new(openai_client.clone())) .then(MetadataKeywords::new(openai_client.clone())) .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10)) .log_all() .filter_errors() .then_store_with( Qdrant::builder() .batch_size(50) .vector_size(1536) .collection_name("swiftide-multi-vectors") .with_vector(EmbeddedField::Chunk) .with_vector(EmbeddedField::Metadata(metadata_qa_text::NAME.into())) .with_vector(EmbeddedField::Metadata(metadata_summary::NAME.into())) .with_vector( VectorConfig::builder() .embedded_field(EmbeddedField::Metadata(metadata_title::NAME.into())) .distance(Distance::Manhattan) .build()?, ) .with_vector(EmbeddedField::Metadata(metadata_keywords::NAME.into())) .build()?, ) .run() .await?; ``` The example embeds the chunk and four metadata fields that a large language model (LLM) writes. Each `with_vector` call adds a named vector for one embedded field. Each metadata transformer module has a `NAME` constant, such as `metadata_title::NAME`, so the vector names match. `VectorConfig::builder()` overrides the distance or the size of one vector. When a node has several vectors, Qdrant stores only the fields you configured and skips the rest. Without `with_vector`, the collection has one vector for `Combined`. The first `with_vector` call replaces that default. [Similarity search](/integrations/qdrant/#similarity-search) on a collection with named vectors always queries `Combined`, which the example above lacks. To search it, use `EmbedMode::Both` and add `with_vector(EmbeddedField::Combined)`. ### Single vector collections With exactly one dense vector and no sparse vector, Qdrant uses one unnamed vector, both in the collection and in each node that has a single vector. That node skips the field filter. So `with_vector(EmbeddedField::Chunk)` alone breaks in two ways: * `PerField` or `Both` nodes with several vectors arrive as named vectors, and Qdrant rejects the write. * Nodes in the default mode get their `Combined` vector stored without an error. With a single configured vector, configure the field your embed mode produces. ## Hybrid collections Hybrid search queries a dense and a sparse vector. `HybridSearch` uses `Combined` for both by default, and `with_dense_vector_field` and `with_sparse_vector_field` change each one on its own. The hybrid search example loads both models and configures both vectors for `Combined`: examples/hybrid_search.rs ```rust // Ensure all batching is consistent let batch_size = 64; let fastembed_sparse = FastEmbed::try_default_sparse().unwrap().to_owned(); let fastembed = FastEmbed::try_default().unwrap().to_owned(); // Set up openai with the mini model, which is great for indexing let openai = openai::OpenAI::builder() .default_prompt_model("gpt-4o-mini") .build() .unwrap(); // Set up qdrant and use the combined fields (metadata + chunks) for both sparse and dense // vectors let qdrant = Qdrant::builder() .batch_size(batch_size) .vector_size(384) .with_vector(EmbeddedField::Combined) .with_sparse_vector(EmbeddedField::Combined) .collection_name("swiftide-hybrid-example") .build()?; ``` `FastEmbed::try_default()` loads BGE (BAAI General Embedding) small, a dense model with 384 dimensions. `FastEmbed::try_default_sparse()` loads a SPLADE (Sparse Lexical and Expansion) model for sparse vectors. `vector_size` must match the dense model, here 384. Qdrant stores a sparse vector only when its field also has a dense vector configured. So `with_sparse_vector(EmbeddedField::Chunk)` without `with_vector(EmbeddedField::Chunk)` adds a sparse vector that never gets filled. ## Embedding models [FastEmbed](/integrations/fastembed/) runs models on your central processing unit (CPU). It is the only integration that produces sparse embeddings. One instance holds one model, and passing a sparse instance to `Embed` fails with “Expected dense model, got sparse”. [FastEmbed](/integrations/fastembed/#model-choice) shows how to load another model, such as a code-trained one. Provider embeddings come from a client with an embedding model. The named vectors example uses OpenAI, which needs the `openai` feature flag: examples/store_multiple_vectors.rs ```rust let openai_client = integrations::openai::OpenAI::builder() .default_embed_model("text-embedding-3-small") .default_prompt_model("gpt-4o") .build()?; ``` Without `default_embed_model`, every `embed` call fails with “Model not set”. The example sets `vector_size(1536)` to match `text-embedding-3-small`. [Models](/models/overview/) lists which client implements `EmbeddingModel`. Index and query with the same model. A new model needs a new index. ## Batch sizes Batching happens at three levels: * `with_batch_size` on `Embed` sets the nodes per model call, and `batch_size` on `FastEmbed` the texts per model run, as [FastEmbed](/integrations/fastembed/#batch-size) explains. * `batch_size` on the storage sets the nodes per write. Qdrant defaults to 50. [Batching](/indexing/storage/#batching) covers how the storage receives them. With `PerField`, a node with four metadata fields gives five texts, so a batch of 10 nodes sends 50 texts in one call. ## Gotchas * A failed model call turns the whole batch into one error. `filter_errors` then drops every node of that batch, see [Errors before a batch step](/indexing/streams-and-concurrency/#errors-before-a-batch-step). * `LanguageModelWithBackOff` does not retry `embed` calls, as [errors and backoff](/models/errors-backoff-and-usage/) explains. * `Combined` puts the metadata before the chunk. A model that truncates long input cuts the end of the chunk first. FastEmbed truncates at its token limit without an error. `fastembed::TextInitOptions::with_max_length` raises that limit, as ask-your-codebase does. * `PerField` embeds every metadata field, including the ones your storage skips. Each one costs embedding time. * A chunker that builds nodes with `TextNode::new` instead of `Node::into_chunks` resets the embed mode to the default. * Clones of a `FastEmbed` share one model behind a lock, so a higher `with_concurrency` does not speed it up. ## Reference * [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) * [`SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html) * [`EmbedMode`](https://docs.rs/swiftide/0.34/swiftide/indexing/enum.EmbedMode.html) * [`EmbeddedField`](https://docs.rs/swiftide/0.34/swiftide/indexing/enum.EmbeddedField.html) * [`Pipeline::then_in_batch`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_in_batch) * [`Pipeline::with_embed_mode`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.with_embed_mode) * [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html) * [`SparseEmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SparseEmbeddingModel.html) * [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html) * [`Qdrant`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.Qdrant.html) * [`QdrantBuilder::with_vector`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.QdrantBuilder.html#method.with_vector) * [`QdrantBuilder::with_sparse_vector`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.QdrantBuilder.html#method.with_sparse_vector) * [`VectorConfig`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.VectorConfig.html) * [`HybridSearch`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.HybridSearch.html) * [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.OpenAI.html) * [`LanguageModelWithBackOff`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.LanguageModelWithBackOff.html) * [`Node::into_chunks`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html#method.into_chunks) * [Feature flags](/reference/feature-flags/) ## Next [Caching](/indexing/caching/) # Loading data > Start an indexing pipeline from files on disk, nodes you build in code, a tool executor, web pages, Kafka, Fluvio, Parquet or a custom Loader. Every indexing pipeline starts with a stream of nodes. You fill that stream from files, web pages, Kafka, Fluvio, Parquet or your own data, with a loader and `Pipeline::from_loader` or with any stream and `Pipeline::from_stream`. ## Files ```rust let languages = [SupportedLanguages::Rust, SupportedLanguages::Python]; let extensions: Vec<&str> = languages .iter() .flat_map(|language| language.file_extensions().iter().copied()) .collect(); // Extensions without the leading dot; each call adds to the list let loader = FileLoader::new("src") .with_extensions(&extensions) .with_extensions(&["md"]); Pipeline::from_loader(loader) .then_chunk(ChunkText::from_max_characters(2000)) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `FileLoader::new` takes a directory or a single file. Its walk uses the `ignore` crate’s defaults. It skips hidden files and files excluded by `.ignore` files, and by `.gitignore` rules when the root is inside a git repository. `with_extensions` keeps only files with those extensions. Without any call the loader yields every file. `SupportedLanguages::file_extensions` returns the extensions tree-sitter knows for a language. It needs the `tree-sitter` feature flag. `FileLoader` itself needs none. Each file becomes one `TextNode`: * `chunk` holds the file content, with invalid UTF-8 replaced. * `path` is relative when the root is relative. The node id hashes path and chunk, so changing the root from relative to absolute misses the [node cache](/indexing/caching/). * A file that cannot be read becomes an error in the stream. The first error ends `run` unless you filter errors, see [Errors](/indexing/streams-and-concurrency/#errors). ## Nodes in memory ```rust let mut faq = TextNode::new("A loader turns a data source into a stream of nodes."); faq.path = "faq/loading.md".into(); faq.metadata.insert("Source", "faq"); let nodes = vec![ faq, TextNode::new("An indexing pipeline does nothing until you call run."), ]; let storage = MemoryStorage::default(); Pipeline::from_stream(nodes) .then_store_with(storage.clone()) .run() .await?; println!("Stored {} nodes", storage.get_all_values().await.len()); ``` `Pipeline::from_stream` takes anything that converts into an `IndexingStream`. That includes a `Vec`, a boxed `Stream` of results and a Tokio `mpsc` receiver. A channel lets another task produce nodes while the pipeline runs: ```rust let (sender, receiver) = tokio::sync::mpsc::channel(100); tokio::spawn(async move { for line in ["first message", "second message"] { if sender.send(Ok(TextNode::new(line))).await.is_err() { break; } } // Dropping the sender ends the stream, and with it the run }); Pipeline::from_stream(receiver) .then_store_with(MemoryStorage::default()) .run() .await?; ``` The stream ends when every sender is dropped. Until then, `run` keeps waiting for nodes. ## Files from a tool executor ```rust let executor = LocalExecutor::new("."); let files = executor .stream_files(Path::new("src"), Some(vec!["rs".to_string()])) .await?; Pipeline::from_stream(files) .then_chunk(ChunkText::from_max_characters(2000)) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `ToolExecutor::stream_files` returns the files below a path as an `IndexingStream`. `LocalExecutor` uses `path` as given and does not resolve it against the executor’s working directory. `LocalExecutor` needs the `swiftide-agents` feature flag. ## Web pages examples/scraping_index_to_markdown.rs ```rust indexing::Pipeline::from_loader(ScrapingLoader::from_spider( Website::new("https://www.bosun.ai/") .with_limit(1) .to_owned(), )) .then(HtmlToMarkdownTransformer::default()) .then_chunk(ChunkMarkdown::from_chunk_range(20..2048)) .log_all() .then_store_with(MemoryStorage::default()) .run() .await?; ``` `ScrapingLoader` crawls with the `spider` crate and needs the `scraping` feature flag. `ScrapingLoader::from_spider` takes a configured `Website`, here limited to one page. To configure it, add `spider` at the version swiftide uses (2.52), or the `Website` types do not match. Each node carries the raw HTML of one page, with the page URL as its `path`. `HtmlToMarkdownTransformer` converts the HTML to markdown and drops `script` and `style` tags. ### Spider Cloud Since 0.33, `ScrapingLoader` crawls through Spider Cloud when `SPIDER_CLOUD_API_KEY` is set and not empty. It reads these environment variables once per process: | Variable | Effect | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `SPIDER_CLOUD_API_KEY` | Turns Spider Cloud on | | `SPIDER_CLOUD_MODE` | `proxy`, `api`, `unblocker` or `fallback`. Anything else selects smart mode, which uses the unblocker only against bot protection. | | `SPIDER_CLOUD_API_URL` | Optional API endpoint | Spider Cloud cannot reach hosts that only your machine can route to. A start URL on `localhost`, a `.localhost` or `.local` name, a loopback address, or a private or link-local IPv4 address is crawled directly. ## Kafka and Fluvio examples/kafka.rs ```rust static LOADER_TOPIC: &str = "loader"; static STORAGE_TOPIC: &str = "storage"; let mut client_config = ClientConfig::new(); client_config.set("bootstrap.servers", "localhost:9092"); client_config.set("group.id", "group_id"); client_config.set("auto.offset.reset", "earliest"); let loader = Kafka::builder() .client_config(client_config.clone()) .topic(LOADER_TOPIC) .build() .unwrap(); let storage = Kafka::builder() .client_config(client_config) .topic(STORAGE_TOPIC) .create_topic_if_not_exists(true) .batch_size(2usize) .build() .unwrap(); ``` examples/kafka.rs ```rust indexing::Pipeline::from_loader(loader) .then_in_batch(Embed::new(FastEmbed::try_default().unwrap()).with_batch_size(10)) .then_store_with(storage) .run() .await?; ``` The pipelines embed with `FastEmbed`, which needs the `fastembed` feature flag. `Kafka` needs the `kafka` feature flag, which builds librdkafka and needs cmake and a C toolchain. It takes an `rdkafka` `ClientConfig`, re-exported as `integrations::kafka::ClientConfig`. As a loader it subscribes to `topic` and turns each message payload into a node. Messages without a UTF-8 payload are skipped. The node metadata gets `kafka_topic`, `kafka_partition` and `kafka_offset`, plus `kafka_timestamp` and `kafka_key` when the message has them. `Kafka` is also a storage, here writing to a second topic. examples/fluvio.rs ```rust static TOPIC_NAME: &str = "hello-rust"; static PARTITION_NUM: u32 = 0; let loader = Fluvio::builder() .consumer_config_ext( ConsumerConfigExt::builder() .topic(TOPIC_NAME) .partition(PARTITION_NUM) .offset_start(fluvio::Offset::from_end(1)) .build() .unwrap(), ) .build() .unwrap(); ``` `Fluvio` needs the `fluvio` feature flag and is a loader only. It takes a `ConsumerConfigExt` with the topic, partition and start offset. `Offset` comes from the `fluvio` crate, which you add at swiftide’s version (0.50). Each record becomes a node with its key in the `fluvio_key` metadata field, which is null when the record has no key. A Kafka pipeline waits for new messages, so `run` does not return on its own. For both loaders, a receive error ends the run. ## Parquet ```rust let loader = Parquet::builder() .path("data.parquet") .column_name("chunk") .build()?; Pipeline::from_loader(loader) .then_chunk(ChunkText::from_max_characters(2000)) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `Parquet` needs the `parquet` feature flag. `batch_size` sets the rows read per batch and defaults to 1024. The loader reads one column and turns every non-null value into a node. The column must hold strings. A column of another type yields an error, and a missing column panics. Batches are flattened unordered, so node order can differ from row order. Each load prints the column descriptors to stderr. ## Custom loaders When no loader fits, implement `Loader`. The [docs-qa](/showcases/docs-qa/) showcase loads markdown and `llms-full.txt` sources as one node per page: examples/docs-qa/src/loader.rs ```rust /// Yields one [`TextNode`] per documentation page, with the page title as metadata. #[derive(Debug, Clone)] pub struct DocsLoader { sources: Vec, } impl DocsLoader { pub fn new(sources: Vec) -> Self { Self { sources } } } impl Loader for DocsLoader { type Output = String; fn into_stream(self) -> IndexingStream { stream::iter(self.sources) .then(DocsSource::pages) .map_ok(|pages| stream::iter(pages).map(|page| Ok(page_node(page)))) .try_flatten() .boxed() .into() } // Only needed when the loader is used as a `Box`. fn into_stream_boxed(self: Box) -> IndexingStream { self.into_stream() } } fn page_node(page: Page) -> TextNode { let mut node = TextNode::new(page.markdown); node.path = page.origin.into(); node.metadata.insert(PAGE_TITLE, page.title.as_str()); node } ``` * `type Output` is the chunk type of the nodes. Built-in transformers expect `String`. * `into_stream` is synchronous and consumes the loader. Put slow work inside the stream, like the page downloads this loader starts through `DocsSource::pages`, so it runs when the pipeline polls it. * A failing source becomes an `Err` item, not a panic. * `into_stream_boxed` is only needed when you use the loader as a `Box`. Its default implementation panics. * `Loader` requires `Send`, `Sync` and `DynClone`, which any `Clone` type implements. ## Gotchas * `Pipeline::from_loader` calls `into_stream` right away, so work starts when you build the pipeline, not when you call `run`. `Kafka` subscribes to its topic. `Fluvio` connects. `Parquet` opens the file. `ScrapingLoader` starts crawling, and panics outside a Tokio runtime. * `Fluvio` and `Parquet` use `tokio::task::block_in_place`, which panics on a current-thread runtime. `#[tokio::main]` uses a multi-thread runtime by default; `#[tokio::test]` does not, so use `#[tokio::test(flavor = "multi_thread")]`. * `Kafka`, `Fluvio` and `Parquet` panic when they cannot create their consumer or open their file. ## Feature flags Install the feature flags you need, for example: ```sh cargo add swiftide@0.34 --features tree-sitter,scraping,kafka,fluvio,parquet,fastembed ``` [Feature flags](/reference/feature-flags/) lists them all. ## Reference * [`FileLoader`](https://docs.rs/swiftide/0.34/swiftide/indexing/loaders/struct.FileLoader.html) * [`Loader`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Loader.html) * [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html) * [`Pipeline::from_loader`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.from_loader) * [`Pipeline::from_stream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.from_stream) * [`TextNode`](https://docs.rs/swiftide/0.34/swiftide/indexing/type.TextNode.html) * [`SupportedLanguages`](https://docs.rs/swiftide/0.34/swiftide/integrations/treesitter/enum.SupportedLanguages.html) * [`ToolExecutor`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ToolExecutor.html) * [`LocalExecutor`](https://docs.rs/swiftide/0.34/swiftide/agents/tools/local_executor/struct.LocalExecutor.html) * [`ScrapingLoader`](https://docs.rs/swiftide/0.34/swiftide/integrations/scraping/struct.ScrapingLoader.html) * [`HtmlToMarkdownTransformer`](https://docs.rs/swiftide/0.34/swiftide/integrations/scraping/struct.HtmlToMarkdownTransformer.html) * [`Kafka`](https://docs.rs/swiftide/0.34/swiftide/integrations/kafka/struct.Kafka.html) * [`Fluvio`](https://docs.rs/swiftide/0.34/swiftide/integrations/fluvio/struct.Fluvio.html) * [`Parquet`](https://docs.rs/swiftide/0.34/swiftide/integrations/parquet/struct.Parquet.html) ## Next [Chunking](/indexing/chunking/) # How indexing works > An indexing pipeline streams nodes from a loader through filters, transformers, chunkers and embedders into storage, and the pipeline does its work only when you await run(). You can read a Swiftide indexing pipeline line by line and know when each step runs. The example comes from the [ask-your-codebase](/showcases/ask-your-codebase/) showcase. It prepares the source files of a codebase for retrieval-augmented generation (RAG): it splits them into chunks, describes each chunk with a large language model (LLM), embeds it, and stores it in LanceDB. ## The ask-your-codebase pipeline examples/ask-your-codebase/src/index.rs ```rust let stats = indexing::Pipeline::from_loader( FileLoader::new(codebase.root()).with_extensions(&languages.extensions()), ) .filter(test_code::not_test_files(codebase.clone())) // Drop unit tests before the cache, which keys on path and content. Changing the content // after it would make the cache miss on every run. .then(test_code::without_unit_tests) .then(files_found.count()) // Skip files whose path and content are unchanged since the last successful run .filter_cached(codebase.node_cache()?) .then(files_changed.count()) .then(record_relative_path(codebase.clone())) // Split along syntax boundaries (functions, impls, ...) with tree-sitter .then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?) // Ask the LLM which questions each chunk answers; stored as metadata and embedded .then( MetadataQACode::from_client(llm) .prompt_template(QUESTIONS_PROMPT) .num_questions(QUESTIONS_PER_CHUNK) .concurrency(concurrent_requests) .build()?, ) // Embed chunk + metadata locally, in batches .then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE)) .then_store_with(codebase.vector_store()?) .run() .await?; ```  Each node passes through the steps from top to bottom until a step drops it: 1. `FileLoader` walks the codebase root and emits one node per file with a matching extension. 2. `filter` drops test files. The closure sees each `Result`, so errors pass through and still surface. 3. `then(test_code::without_unit_tests)` cuts unit tests out of Rust files. 4. `filter_cached` drops files whose path and content did not change since the last successful run. `codebase.node_cache()` is a showcase helper that builds a `Redb` node cache, see [Caching](/indexing/caching/). 5. `then(record_relative_path(..))` stores the file path, relative to the root, as metadata. 6. `then_chunk` splits each file along syntax boundaries with tree-sitter. `ChunkByLanguage` is the showcase’s own wrapper that picks a Swiftide `ChunkCode` per language. One file becomes several nodes. 7. `MetadataQACode` asks the LLM which questions each chunk answers and stores them as metadata. 8. `then_in_batch` with `Embed` embeds each chunk and its metadata with FastEmbed, on the central processing unit (CPU). `with_batch_size` sets 32 nodes per call. Without it, a batch holds 256 nodes. 9. `then_store_with` writes each node to storage. `codebase.vector_store()` builds a `LanceDB` store, see [Storage](/indexing/storage/). 10. `run().await` drives the stream to the end and returns the run’s statistics as `stats`. ## Nodes A node is a `Node`. It carries one piece of content and everything known about it. | Field | Holds | | --------------------------- | ------------------------------------------------------------------------------- | | `path` | The source path. | | `chunk` | The content, of type `T`. | | `metadata` | Key and value pairs, sorted by key. | | `vectors`, `sparse_vectors` | Embeddings, one per embedded field. `Embed` and `SparseEmbed` fill them. | | `embed_mode` | Whether the chunk and metadata are embedded together, each on its own, or both. | | `offset` | A running position that `ChunkCode` counts from 0 over the chunks it keeps. | | `parent_id` | The `id()` of the source node when the first chunker split it. | Since 0.31 the pipeline is generic. A `Pipeline` streams `Node`. A chunk is any type that implements the `Chunk` marker trait, which requires `Clone + Send + Sync + Debug + AsRef<[u8]> + 'static`. Most built-in steps take `TextNode`, which is `Node`. `id()` returns a universally unique identifier (UUID) computed from the path and the chunk, so it changes when either changes. The node cache keys on this id, see [Cache keys](/indexing/caching/#cache-keys). ## Steps | Step | Method | Trait | | ----------------- | ------------------------------------------------ | --------------------------------- | | loader | `Pipeline::from_loader`, `Pipeline::from_stream` | `Loader` | | filter | `filter` | a closure over `&Result>` | | node cache | `filter_cached` | `NodeCache` | | transformer | `then` | `Transformer`, or a closure | | chunker | `then_chunk`, `then_expand` | `ChunkerTransformer` | | batch transformer | `then_in_batch` | `BatchableTransformer` | | embedder | `then_in_batch(Embed::new(model))` | `EmbeddingModel` | | storage | `then_store_with` | `Persist` | A chunker returns a stream of nodes for each node. `then_expand` behaves like `then_chunk` with a different name in traces. Use it when you split a node for a reason other than chunking. A step can change the node type. `then` with a transformer whose `Output` is `U` turns a `Pipeline` into a `Pipeline`. Nodes keep flowing after storage. ## Execution The pipeline methods only describe the stream, but their arguments run while you build the chain. In the snippet, `codebase.node_cache()?` creates the Redb file and `embedding::local_model()?` loads the FastEmbed model, which downloads it on first use. `run` takes the pipeline by value and does the work: 1. It calls `setup` on every storage backend. LanceDB creates its table here if it is missing. 2. It pulls nodes through the stream until the stream ends. Some steps spawn Tokio tasks, so `run` needs a Tokio runtime. 3. It marks the source nodes in the node cache, but only after the stream ended without an error. 4. It logs `Pipeline completed` at info level and returns the same numbers as `PipelineStats`, see [Statistics](/indexing/streams-and-concurrency/#statistics). `run` returns `Result`, not the nodes. The showcase reads the number of stored chunks from `stats.nodes_stored`. It counts files with pass-through closures such as `.then(files_found.count())`, because the statistics count nodes at the end of the stream, after chunking. `then`, `then_chunk`, `then_in_batch` and storage work on several nodes at once and pass them on as they finish, see [Streams and concurrency](/indexing/streams-and-concurrency/). The showcase passes `.concurrency(concurrent_requests)` to `MetadataQACode` to bound requests to the LLM. ### Errors and the node cache The first error stops the run, and `run` returns it, see [Errors](/indexing/streams-and-concurrency/#errors). It is the error the step failed with, so a `LanguageModelError` can be recovered with `downcast_ref`. A source node is cached only when at least one of its chunks reached the end and none of them failed, and [When nodes are cached](/indexing/caching/#when-nodes-are-cached) covers the consequences. ## Output Two runs on the `swiftide-agents` crate. `claude-haiku-5-5` wrote the questions, through an OpenAI-compatible gateway to Amazon Web Services (AWS) Bedrock. FastEmbed `jina-embeddings-v2-base-code` computed the embeddings. ```text $ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5 Indexing Rust files in swiftide-agents ... Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s ``` The second run, with no file changed: ```text $ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5 Indexing Rust files in swiftide-agents ... Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s ``` The node cache drops all 14 files at step 4, so no prompt is sent. ## Feature flags `Pipeline`, `FileLoader` and `Embed` need no feature flag. The showcase turns off the defaults and turns on `tree-sitter`, `fastembed`, `lancedb`, `redb` and `openai`. To add the same features to your crate: ```sh cargo add swiftide@0.34 --no-default-features --features tree-sitter,fastembed,lancedb,redb,openai ``` TLS needs no setup, because swiftide’s HTTP clients bring their own rustls crypto provider ([TLS](/integrations/overview/#tls)). [Feature flags](/reference/feature-flags/) lists every flag. ## Reference * [`indexing::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html) * [`PipelineStats`](https://docs.rs/swiftide/0.34/swiftide/indexing/statistics/struct.PipelineStats.html) * [`Node`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html) * [`TextNode`](https://docs.rs/swiftide/0.34/swiftide/indexing/type.TextNode.html) * [`EmbedMode`](https://docs.rs/swiftide/0.34/swiftide/indexing/enum.EmbedMode.html) * [`FileLoader`](https://docs.rs/swiftide/0.34/swiftide/indexing/loaders/struct.FileLoader.html) * [`Loader`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Loader.html) * [`Transformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Transformer.html) * [`ChunkerTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChunkerTransformer.html) * [`BatchableTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.BatchableTransformer.html) * [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html) * [`Persist`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Persist.html) * [`NodeCache`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.NodeCache.html) * [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) * [`SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html) * [`ChunkCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkCode.html) * [`MetadataQACode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQACode.html) * [`Redb`](https://docs.rs/swiftide/0.34/swiftide/integrations/redb/struct.Redb.html) * [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html) * [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html) ## Next [Loading data](/indexing/loading/) # Storage > Store indexed nodes with then_store_with in LanceDB, Qdrant, PgVector, DuckDB, Redis, Kafka or MemoryStorage, and write your own storage by implementing the Persist trait. `then_store_with` writes every node to a storage backend. You end up with nodes in LanceDB or Qdrant, a clear picture of what each backend keeps of a node, and a storage of your own built on the `Persist` trait. ## LanceDB The ask-your-codebase showcase keeps its vectors in an embedded LanceDB table, in a directory next to the code. There is no server to run. Enable the `lancedb` feature flag: ```sh cargo add swiftide@0.34 --features lancedb ``` examples/ask-your-codebase/src/codebase.rs ```rust /// Remembers which files were fully indexed, so unchanged files are skipped next time. pub fn node_cache(&self) -> Result { std::fs::create_dir_all(&self.data_dir)?; Redb::builder() .database_path(self.data_dir.join("cache.redb")) .table_name("indexed_files") .build() } /// Embedded vector store with one vector per chunk and the source path next to it. pub fn vector_store(&self) -> Result { let uri = self.lancedb_dir(); LanceDB::builder() .uri(uri.to_str().context("Index path is not valid UTF-8")?) .table_name("code_chunks") .vector_size(embedding::DIMENSIONS) .with_vector(EmbeddedField::Combined) .with_metadata(PATH_FIELD) .build() } ``` `vector_store` builds the storage; `node_cache` belongs to [Caching](/indexing/caching/). The showcase pipeline passes the storage to `then_store_with`, after the embedding step. * `uri` is the directory that holds the table. It is required unless you pass a `connection_pool`. * `table_name` defaults to `swiftide`. * `vector_size` must match the output of your embedder. The showcase uses 768, the size of its FastEmbed code model. * `with_vector(EmbeddedField::Combined)` adds a column for the embedding of the chunk together with its metadata. * `with_metadata(PATH_FIELD)` adds a column for the `path` metadata field. LanceDB stores only what you configure: the node id, the chunk, each configured vector and each metadata field named with `with_metadata`. The node’s own `path` is not a column. The showcase stores the path relative to the codebase root as metadata, so answers can cite their sources. Metadata columns hold text, so a value that is not a JSON string is stored as null. Before the first node arrives, setup creates the table if it does not exist. Nodes are written in batches of 256 with a merge on `id`, so storing the same node twice updates its row. See [LanceDB](/integrations/lancedb/) for retrieval and remote storage. ## Qdrant The docs-qa showcase stores a dense and a sparse vector for every chunk in a Qdrant collection. Enable the `qdrant` feature flag: ```sh cargo add swiftide@0.34 --features qdrant ``` examples/docs-qa/src/store.rs ```rust let qdrant = Qdrant::builder() .batch_size(BATCH_SIZE) .vector_size(DENSE_VECTOR_SIZE) // One dense and one sparse vector, both over the chunk *and* its metadata .with_vector(EmbeddedField::Combined) .with_sparse_vector(EmbeddedField::Combined) .collection_name(&collection.0) .build()?; ``` Qdrant keeps more of the node than LanceDB. Each point’s payload holds every metadata field, plus `path`, `content` (the chunk) and `last_updated_at`. The point id is the node id. Nodes without vectors fail with “Node without vectors”. Setup creates the collection when it is missing. An existing collection is kept as it is, even when its vector configuration differs from the builder. The client reads `QDRANT_URL` (default `http://localhost:6334`, gRPC) and `QDRANT_API_KEY`. The default batch size is 50; docs-qa raises it to 64. A debug build of docs-qa split four markdown pages into 43 chunks and stored them in Qdrant (metadata from `claude-haiku-5-5`, embeddings from local FastEmbed): ```text Indexed 43 chunks into `docs-qa` in 71.7s ``` [Qdrant](/integrations/qdrant/) covers named vectors, hybrid search and filters. ## Backends Every backend sits behind its own feature flag, except `MemoryStorage`: | Storage | Feature flag | What it stores | Default batch size | Retrieval | | --------------- | ------------ | -------------------------------------------------------------- | ------------------ | --------- | | `LanceDB` | `lancedb` | id, chunk, configured vectors, `with_metadata` fields | 256 | yes | | `Qdrant` | `qdrant` | vectors; payload with all metadata, path, chunk | 50 | yes | | `PgVector` | `pgvector` | id, chunk, configured vectors, `with_metadata` fields as JSONB | 50 | yes | | `Duckdb` | `duckdb` | `uuid`, chunk, path, configured vectors; no metadata | none, node by node | yes | | `Redis` | `redis` | the node as JSON under `:` | 10 | no | | `Kafka` | `kafka` | the node as JSON, keyed `:`, on a topic | 32 | no | | `MemoryStorage` | none | the whole node, in memory | none, node by node | no | PgVector fails to store a node that lacks a field named with `with_metadata`; see [PgVector](/integrations/pgvector/). Kafka’s setup fails when the topic does not exist, unless you set `create_topic_if_not_exists`. Kafka is also a loader; see [Kafka and Fluvio](/indexing/loading/#kafka-and-fluvio). DuckDB stores node by node and ignores its builder’s `batch_size`. It rejects a node that lacks a vector for a configured field. When a row with the same `uuid` exists, DuckDB keeps the old row unless you set `upsert_vectors`, which needs DuckDB 1.2.1 or later. Redis works as a key-value store for nodes. `persist_key_fn` changes the key and `persist_value_fn` the value. Both take a plain `fn` pointer, so a closure that captures variables does not compile. The example writes only the metadata and raises the batch size from 10 to 50: examples/index_into_redis.rs ```rust indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"])) .then_chunk(ChunkCode::try_for_language_and_chunk_size( "rust", 10..2048, )?) .then_store_with( // By default the value is the full node serialized to JSON. // We can customize this by providing a custom function. Redis::try_build_from_url(&redis_url)? .persist_value_fn(|node| Ok(serde_json::to_string(&node.metadata)?)) .batch_size(50) .build()?, ) .run() .await?; ``` ### DuckDB No swiftide example uses DuckDB, so this section describes the builder without code. `Duckdb::builder()` returns a `DuckdbBuilder` with these methods: * `connection` takes anything that converts into a `duckdb::Connection`. Swiftide does not re-export the `duckdb` crate, so add it to your crate yourself. * `with_vector(field, size)` adds a vector column of `size` floats for an `EmbeddedField`, such as `EmbeddedField::Combined`. * `table_name` defaults to `swiftide`, and `cache_table` to `swiftide_cache`. `cache_key_prefix` prefixes the node cache keys, which also lets you invalidate the cache by hand. * `schema` replaces the SQL that loads the extensions and creates the table. The upsert statement is built from the table name and vectors, so keep the column names when you change it. `upsert_vectors` is the flag described above. `Duckdb` implements `Persist` for `then_store_with`, `NodeCache` for `filter_cached`, and `Retrieve` for three search strategies: `SimilaritySingleEmbedding`, `HybridSearch` and `CustomStrategy`. The setup installs and loads the `vss` and `fts` extensions before it creates the table. Retrieved documents carry `id` and `path` as metadata, since no other metadata is stored. See [`DuckdbBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/duckdb/struct.DuckdbBuilder.html) for every method. ## MemoryStorage `MemoryStorage` lives in `swiftide::indexing::persist` and needs no feature flag. It suits tests and experiments: ```rust let storage = MemoryStorage::default(); Pipeline::from_stream(vec![ TextNode::new("Swiftide is a Rust library for building LLM applications."), TextNode::new("An indexing pipeline loads, transforms and stores nodes."), ]) .then(|mut node: TextNode| -> anyhow::Result { let words = node.chunk.split_whitespace().count(); node.metadata.insert("Words", words); Ok(node) }) // Clones share the same map, so keep one to read from later .then_store_with(storage.clone()) .run() .await?; for node in storage.get_all_values().await { println!("{:?} words: {}", node.metadata.get("Words"), node.chunk); } ``` Its keys are an incrementing counter, not the node id, so storing the same node twice keeps two copies. `get(key)` reads one node, `get_all_values()` returns all nodes, and `get_all()` returns them with their keys. It has no retrieval for query pipelines. ## Batching A storage chooses how it receives nodes through `batch_size`. When it returns `Some(n)`, the pipeline groups up to `n` nodes and calls `batch_store`. When it returns `None`, the pipeline calls `store` for each node. In both cases, up to the pipeline’s concurrency of these calls run at once. Storage has no concurrency setting of its own, and [Concurrency](/indexing/streams-and-concurrency/#concurrency) explains when the pipeline value is read. For how this differs from the embedder’s batch size, see [Batch sizes](/indexing/embedding/#batch-sizes). Stored nodes flow on, so you can chain a second `then_store_with` to write to two backends. If `setup` fails, for example because Qdrant is unreachable, `run` returns that error. To keep going after a failed write, see [Errors](/indexing/streams-and-concurrency/#errors). ## Custom storage Implement `Persist` to write nodes anywhere else. This one appends each node as a line of JSON: ```rust /// Appends every node as one line of JSON to a file. #[derive(Debug, Clone)] pub struct JsonLines { path: PathBuf, // Batches are stored concurrently; one writer at a time keeps lines whole write_lock: Arc>, } impl JsonLines { pub fn new(path: impl Into) -> Self { Self { path: path.into(), write_lock: Arc::default(), } } async fn append(&self, nodes: &[TextNode]) -> anyhow::Result<()> { let mut lines = String::new(); for node in nodes { lines.push_str(&serde_json::to_string(node)?); lines.push('\n'); } let _guard = self.write_lock.lock().await; let mut file = OpenOptions::new() .create(true) .append(true) .open(&self.path) .await?; file.write_all(lines.as_bytes()).await?; Ok(()) } } #[async_trait] impl Persist for JsonLines { type Input = String; type Output = String; // Runs once, before the first node arrives async fn setup(&self) -> anyhow::Result<()> { if let Some(dir) = self.path.parent() { tokio::fs::create_dir_all(dir).await?; } Ok(()) } // Used when `batch_size` returns `None` async fn store(&self, node: TextNode) -> anyhow::Result { self.append(std::slice::from_ref(&node)).await?; Ok(node) } // Used when `batch_size` returns `Some`; pass the stored nodes on async fn batch_store(&self, nodes: Vec) -> IndexingStream { self.append(&nodes).await.map(|()| nodes).into() } fn batch_size(&self) -> Option { Some(100) } } ``` * `Input` is the chunk type the storage accepts, and `Output` the chunk type it passes on. Both are `String` for text nodes. * `batch_store` returns an `IndexingStream`. Return the stored nodes so later steps receive them, or an error. * `Persist` requires `Debug`, `Clone`, `Send` and `Sync`, so fields like `Rc` or `RefCell` do not compile. Implement it with `#[async_trait]`. Batches are stored concurrently, so guard anything they share. Use the storage like any other: ```rust Pipeline::from_stream(vec![TextNode::new("Stored as a line of JSON.")]) .then_store_with(JsonLines::new("index/nodes.jsonl")) .run() .await?; ``` ## Re-indexing A node’s id is computed from its path and its chunk, as [Cache keys](/indexing/caching/#cache-keys) explains. When you index an unchanged chunk again, LanceDB, Qdrant, PgVector and DuckDB find the same id and keep one row. DuckDB keeps the old row as it is unless `upsert_vectors` is set. When a chunk changes, it gets a new id and a new row, and the row of the old version stays. Swiftide does not delete rows for you. docs-qa’s `--recreate` flag drops the Qdrant collection before indexing. For incremental updates, delete a file’s rows yourself before you index its new version, and skip unchanged files with a [node cache](/indexing/caching/). ## Reference * [`Persist`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Persist.html) * [`Pipeline::then_store_with`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_store_with) * [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html) * [`MemoryStorage`](https://docs.rs/swiftide/0.34/swiftide/indexing/persist/struct.MemoryStorage.html) * [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html) * [`Qdrant`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.Qdrant.html) * [`PgVector`](https://docs.rs/swiftide/0.34/swiftide/integrations/pgvector/struct.PgVector.html) * [`Duckdb`](https://docs.rs/swiftide/0.34/swiftide/integrations/duckdb/struct.Duckdb.html) * [`Redis`](https://docs.rs/swiftide/0.34/swiftide/integrations/redis/struct.Redis.html) * [`Kafka`](https://docs.rs/swiftide/0.34/swiftide/integrations/kafka/struct.Kafka.html) * [`EmbeddedField`](https://docs.rs/swiftide/0.34/swiftide/indexing/enum.EmbeddedField.html) * [ask-your-codebase source](https://github.com/bosun-ai/swiftide/tree/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase) * [docs-qa source](https://github.com/bosun-ai/swiftide/tree/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa) ## Next [Streams and concurrency](/indexing/streams-and-concurrency/) # Streams and concurrency > Set how many nodes an indexing pipeline processes at once, throttle it, split and merge its stream, and drop failed nodes without losing good ones. You set how many nodes each step of an indexing pipeline handles at once, how fast they reach a slow step, which steps they pass through, and what happens to the ones that fail. The pipeline is a stream of nodes, and every step is a stage on that stream. The snippets use `OpenAI`, which needs the `openai` feature flag, and `ChunkCode`, which needs the `tree-sitter` feature flag: ```sh cargo add swiftide@0.34 --features openai,tree-sitter ``` ## Concurrency ```rust indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"])) // Applies to every step added after this call .with_concurrency(8) .then_chunk(ChunkMarkdown::from_chunk_range(100..2048)) // At most one chunk every 50ms enters the next step .throttle(Duration::from_millis(50)) // Its own concurrency wins over the pipeline's 8 .then(MetadataQAText::new(openai.clone()).with_concurrency(2)) .then_in_batch(Embed::new(openai).with_batch_size(64)) .then_store_with(MemoryStorage::default()) .run() .await?; ``` Each transformer, chunker and storage step spawns a Tokio task per node, or per batch for batch transformers and batch storage, and keeps up to its concurrency of them running. Results leave a step as they finish, so the order of nodes is not preserved. `with_concurrency` sets the pipeline value. The default is the number of logical cores of the machine. A step reads the value when you add it, so call `with_concurrency` before the steps it should apply to. A transformer, batch transformer or chunker can bring its own limit through `concurrency()`. When that returns a value, it wins over the pipeline’s. The built-in large language model (LLM) transformers and `Embed` set it with `with_concurrency`, as `MetadataQAText` does above. Storage has no such override and always uses the pipeline value. `throttle` lets at most one node through per duration, at the point where you put it. Here it caps the chunks that reach the LLM at 20 per second, whatever the concurrency. ## Split and merge ```rust let (markdown, code) = indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md", "rs"])) .split_by(|result| { result .as_ref() .is_ok_and(|node| node.path.extension().is_some_and(|ext| ext == "md")) }); let markdown = markdown.then_chunk(ChunkMarkdown::from_chunk_range(100..2048)); let code = code.then_chunk(ChunkCode::try_for_language("rust")?); // One set of statistics for both branches let stats = markdown .merge(code) .then_store_with(MemoryStorage::default()) .run() .await?; ``` `split_by` turns one pipeline into two. The predicate gets each item as a `Result`, errors included. Items for which it returns true go to the first pipeline, the rest to the second. Here, errors go to the code branch. `merge` joins two pipelines with the same node type into one stream. Node cache registrations and recorded failures of both sides are combined, so a `filter_cached` inside either branch still marks its sources and failures in the other branch keep them out of the cache. Each branch of `split_by` gets its own statistics collector, seeded with a copy of what was collected before the split, so a branch you run on its own reports only its own nodes. A collector handle taken before the split sees neither branch. `merge` keeps the collector of the pipeline you call it on, so `run` after the merge returns one `PipelineStats` for both branches. ### Split gotchas * `split_by` is not lazy. It spawns a task that starts reading the stream at once and sends each item into one of two channels that hold 1000 items each. * When one branch is not read, its channel fills up and the split stops, which also stalls the other branch. Run both branches together: merge them, or run them at the same time with `tokio::join!`. Running one to completion before starting the other can hang once the waiting branch has 1000 items. * If a branch is dropped, the split task panics on its next send to it. The panic stays inside that task, and the other branch then ends early without an error. * `merge` keeps the storage setup of the pipeline you call it on and drops that of the pipeline you pass in. Add storage after the merge, as above, or to the branch you call `merge` on. ## Errors ```rust indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"])) .filter(|result| match result { Ok(node) => !node.chunk.trim().is_empty(), // Keep errors, so `log_errors` sees them Err(_) => true, }) .then_chunk(ChunkMarkdown::from_chunk_range(100..2048)) .then(MetadataQAText::new(openai.clone())) // Drop failed nodes before the batch step, so an error does not take its batch with it .log_errors() .filter_errors() .then_in_batch(Embed::new(openai).with_batch_size(64)) .log_nodes() .then_store_with(MemoryStorage::default()) .run() .await?; ``` `run` stops at the first error a step returns and returns that error, and a node cache marks nothing from a failed run. The steps above drop failed nodes so the run continues. The returned error is the one the step failed with, even when a batch step follows it. A `LanguageModelError` from an LLM transformer can be recovered with `downcast_ref` or found in `chain()`, see [Errors, backoff and usage](/models/errors-backoff-and-usage/). * `filter` keeps an item when the closure returns true. It sees nodes and errors, so it can drop specific errors as well as nodes. This one drops empty files and keeps every error. * `log_errors` logs each error at ERROR level and passes it on. * `filter_errors` drops errors without logging them. * `log_nodes` logs each node at DEBUG level. `log_all` is `log_errors` followed by `log_nodes`. Order matters. `filter_errors` before `log_errors` leaves nothing to log. Each pair covers only the steps before it, and errors from steps added after it still abort the run. Add another pair after `then_store_with` if a failed write should not stop the run. ### Errors before a batch step A batch step collects nodes until it has a full batch. Those are `then_in_batch`, and `then_store_with` when the storage has a batch size. When an error reaches it, the step passes the error on and drops the nodes it has collected so far, without a trace. A `filter_errors` after the batch step keeps the run going, but those good nodes are already gone. With a batch size of 64, one failed LLM call can drop up to 63 good chunks. Those dropped nodes are not recorded as failed. When other chunks of the same source complete, the node cache marks that source and the next run skips it. Put a `filter_errors` pair before every batch step, as the snippet does before `then_in_batch`. The source of a node that fails in a transformer stays out of the node cache, so the next run retries it. [Caching](/indexing/caching/) covers when sources are marked. ## Statistics `run` returns the `PipelineStats` of the run, with node counts, token usage and timing, as `let stats = pipeline.run().await?;`. It also logs them as the INFO event `Pipeline completed`. `run` takes the pipeline by value, and `stats()` before `run` returns the empty initial statistics. To watch a run while it is going, clone the handle from `stats_collector()` before you call `run`. [Pipeline statistics](/observability/metrics-and-statistics/#pipeline-statistics) shows both. ## Streams Loaders, chunkers, batch transformers and batch storage return their nodes as an `IndexingStream`. `IndexingStream::iter` takes any iterator of results, `IndexingStream::from_nodes` a `Vec` of nodes, and `IndexingStream::empty` makes an empty stream. Use them to return nodes from your own chunkers and batch transformers. When you build a pipeline with `Pipeline::from_stream`, a `Result>` that is an error and an `anyhow::Error` each become a stream with that one error item. A Tokio channel’s capacity applies backpressure: `send` waits while the pipeline is behind. [Loading](/indexing/loading/#nodes-in-memory) shows `from_stream` with a vector and a channel. ## Reference * [`Pipeline`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html) (`with_concurrency`, `throttle`, `then`, `then_chunk`, `then_in_batch`, `then_store_with`, `split_by`, `merge`, `filter`, `filter_cached`, `filter_errors`, `log_all`, `log_errors`, `log_nodes`, `from_stream`, `run`, `stats`, `stats_collector`) * [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html) * [`Transformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Transformer.html) (`concurrency`) * [`BatchableTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.BatchableTransformer.html) (`concurrency`) * [`ChunkerTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChunkerTransformer.html) (`concurrency`) * [`FileLoader`](https://docs.rs/swiftide/0.34/swiftide/indexing/loaders/struct.FileLoader.html) * [`ChunkMarkdown`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkMarkdown.html) * [`ChunkCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkCode.html) * [`MetadataQAText`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQAText.html) * [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) * [`MemoryStorage`](https://docs.rs/swiftide/0.34/swiftide/indexing/persist/struct.MemoryStorage.html) * [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.OpenAI.html) ## Next [Code-aware indexing](/indexing/code-aware-indexing/) # Transformers and metadata > Add metadata to nodes with closures, the built-in LLM metadata transformers and their prompts, and your own transformers and batch transformers. You add metadata to nodes with closures, with the built-in transformers that ask a large language model (LLM), or with a transformer you write yourself. Metadata travels with each node to the embedder, so a retriever can match on it and an answer can cite it. ## Closures The [ask-your-codebase](/showcases/ask-your-codebase/) showcase stores each file’s path, relative to the codebase root, on every node. examples/ask-your-codebase/src/index.rs ```rust /// Stores the file path, relative to the codebase root, as metadata on the node. /// /// The vector store keeps this field next to each chunk, so answers can cite their sources. fn record_relative_path( codebase: Codebase, ) -> impl Fn(TextNode) -> Result + Clone + Send + Sync + 'static { move |mut node| { let path = codebase .relative_path(&node.path) .to_string_lossy() .into_owned(); node.metadata.insert(PATH_FIELD, path); Ok(node) } } ``` A closure from `TextNode` to `Result` is a transformer if it is `Clone`, `Send`, `Sync` and `'static`, so it cannot borrow local data. You add it with `.then(...)`. The pipeline runs it on many nodes at once and does not keep their order. `node.metadata` is a `Metadata` map. `insert` takes a key that converts into a `String` and a value that converts into a `serde_json::Value`. LanceDB and PgVector store a field only after you declare it with `with_metadata`, which is why the showcase calls `.with_metadata(PATH_FIELD)`. ## Built-in metadata transformers Swiftide ships six transformers that prompt the LLM per node and store the reply as metadata: | Transformer | Metadata field | Feature flag | | --------------------- | ------------------------------ | ------------- | | `MetadataQAText` | `Questions and Answers (text)` | none | | `MetadataQACode` | `Questions and Answers (code)` | `tree-sitter` | | `MetadataSummary` | `Summary` | none | | `MetadataTitle` | `Title` | none | | `MetadataKeywords` | `Keywords` | none | | `CompressCodeOutline` | `Outline` | `tree-sitter` | Each module exports the field name as a `NAME` constant, such as `metadata_title::NAME`. `CompressCodeOutline` skips nodes without an `Outline` field; [Code-aware indexing](/indexing/code-aware-indexing/) shows how to fill it. The next example needs the `openai` and `qdrant` feature flags. examples/index_markdown_lots_of_metadata.rs ```rust indexing::Pipeline::from_loader(FileLoader::new("README.md").with_extensions(&["md"])) .with_concurrency(1) .then_chunk(ChunkMarkdown::from_chunk_range(20..2048)) .then(MetadataQAText::new(openai_client.clone())) .then(MetadataSummary::new(openai_client.clone())) .then(MetadataTitle::new(openai_client.clone())) .then(MetadataKeywords::new(openai_client.clone())) .then_in_batch(Embed::new(openai_client.clone())) .log_all() .filter_errors() .then_store_with( Qdrant::builder() .batch_size(50) .vector_size(1536) .collection_name("swiftide-examples") .build()?, ) .run() .await?; ``` `new(client)` uses the default prompt. `from_client(client)` returns the builder, where you set `prompt_template`, `concurrency` and, for the question transformers, `num_questions` (default 5). `builder()` and `Default` start without a client. After the chunker, each chunk gets its own prompt and reply. Before it, one prompt per file is enough, and chunking copies the reply to every chunk of that file. `concurrency(n)` on the builder, or `with_concurrency(n)`, limits the requests that step sends at once. [Concurrency](/indexing/streams-and-concurrency/#concurrency) explains how this interacts with the pipeline value. By default the embedder sees every metadata field as a `key: value` line in front of the chunk. [Embed mode](/indexing/embedding/#embed-mode) explains how to embed fields separately. ## Prompts The default prompts show the expected answer inside a code fence, and models tend to copy that fence into the metadata. The [docs-qa](/showcases/docs-qa/) showcase replaces the `MetadataTitle` and `MetadataKeywords` prompts: examples/docs-qa/src/index.rs ```rust // Swiftide's default metadata prompts show the expected format inside a code fence, and models // tend to copy the fence into the metadata. These ask for plain text and use the page title. // A code-only chunk is titled by the API it shows, because the reranker scores the title too. const TITLE_PROMPT: &str = "Write a short, descriptive title (at most 10 words) for this \ excerpt from the \"{{ node.metadata.Page }}\" documentation. If the excerpt is mostly code, \ name the API it demonstrates (a macro, trait or builder method), not what the example does. \ Reply with the title only: no quotes, no markdown, no code fences. {{ node.chunk }}"; const KEYWORDS_PROMPT: &str = "List up to 8 keywords for this excerpt from the \ \"{{ node.metadata.Page }}\" documentation: names of types, crates, features, tools, commands \ and concepts that literally appear in it. Reply with one comma-separated line only: no \ numbering, no markdown, no code fences. {{ node.chunk }}"; ``` The pipeline passes them to the builders with `prompt_template`: examples/docs-qa/src/index.rs ```rust indexing::Pipeline::from_loader(loader) .then_chunk(ChunkMarkdown::from_chunk_range( MIN_CHUNK_CHARS..MAX_CHUNK_CHARS, )) // A short title and keywords per chunk; both end up in the embedded text and help the // sparse (keyword) side of hybrid search in particular. .then( MetadataTitle::from_client(llm.clone()) .prompt_template(TITLE_PROMPT) .build()?, ) .then( MetadataKeywords::from_client(llm.clone()) .prompt_template(KEYWORDS_PROMPT) .build()?, ) .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE)) .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE)) .then_store_with(qdrant.clone()) .run() .await ``` `prompt_template` takes anything that converts into a `Prompt`, such as a `&'static str` holding a Tera template. Each transformer adds the node as `node`, so `{{ node.chunk }}` is the chunk and `{{ node.metadata.Page }}` is a field that the docs-qa loader set. The question transformers also add `{{ questions }}`, and `MetadataQACode` adds `{{ outline }}` when the node has an `Outline` field. In the captured docs-qa run, four pages became 43 chunks, and the title and keywords steps made 86 calls to `claude-haiku-5-5` through an OpenAI-compatible gateway to Amazon Web Services (AWS) Bedrock. The whole run, including local embedding and the writes to Qdrant, took: ```text Indexed 43 chunks into `docs-qa` in 71.7s ``` The ask-your-codebase prompt asks for questions a developer would ask: examples/ask-your-codebase/src/index.rs ````rust /// Asks for the questions a developer would ask that a chunk answers. /// /// The default `MetadataQACode` prompt asks generic questions ("What does this code do?"). /// Questions about behaviour, phrased the way a developer would ask them, match real questions /// much better. const QUESTIONS_PROMPT: &str = r" # Task List the questions a developer could ask about this codebase that the code below answers, each with a short answer. They are stored with the code to help a search find it. Focus on behaviour: when and in which order things happen, what starts or ends a process, which conditions and configuration change the outcome, what happens on errors, and which extension points (traits, hooks, builders, options) exist and when they apply. # Constraints - Write {{ questions }} questions and answers. - Phrase questions the way a developer would ask them, e.g. 'When does the agent stop?'. - Name the relevant types, functions and variants in the answers. - Only use what can be derived from the code. - Respond only in this format: Q1: A1: # Code ``` {{ node.chunk }} ``` "; ```` See [Prompts and templates](/models/prompts-and-templates/) for Tera syntax. ## Default LLM client You can set one client on the pipeline instead of on each transformer. This example needs the `openai` feature flag. ```rust Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"])) // Set the client before the steps that use it .with_default_llm_client(openai) .then_chunk(ChunkMarkdown::from_chunk_range(100..1500)) .then(MetadataTitle::default()) .then(MetadataSummary::default()) .then( MetadataQAText::builder() .num_questions(3_usize) .concurrency(4_usize) .build()?, ) .then_store_with(MemoryStorage::default()) .run() .await ``` `then` hands the pipeline defaults to the transformer when you add the step, so call `with_default_llm_client` first. A client on the transformer takes precedence. Without either, each node fails with `No client provided`. ## Custom transformers A transformer is a type that implements `Transformer`. ```rust /// Stores the number of words in a chunk as the `Words` metadata field. #[derive(Clone)] pub struct WordCount; #[async_trait] impl Transformer for WordCount { type Input = String; type Output = String; async fn transform_node(&self, mut node: TextNode) -> anyhow::Result { let words = node.chunk.split_whitespace().count(); node.metadata.insert("Words", words); Ok(node) } } // `then` hands pipeline defaults to every transformer; this one has no use for them. impl WithIndexingDefaults for WordCount {} ``` `transform_node` receives one node and returns it, changed, or an error. The type must be `Clone`, `Send`, `Sync` and `'static`. The trait is an `async_trait`, so add the `async-trait` crate. Override `concurrency` to give the step its own limit. `then` also requires `WithIndexingDefaults`; an empty `impl` is enough. ### Transformer macro The built-in LLM transformers use `#[indexing_transformer]`. This one asks who a chunk is for: ```rust /// The macro adds items next to the struct, so each annotated struct gets its own module. pub mod metadata_audience { use async_trait::async_trait; use swiftide::{indexing::TextNode, indexing_transformer, traits::Transformer}; /// Asks the LLM who a chunk is written for and stores the reply as `Audience`. #[indexing_transformer( metadata_field_name = "Audience", default_prompt_file = "prompts/metadata_audience.prompt.md" )] pub struct MetadataAudience { /// Product name passed to the prompt as `{{ product }}`. #[builder(default = "\"swiftide\".to_string()")] product: String, } #[async_trait] impl Transformer for MetadataAudience { type Input = String; type Output = String; async fn transform_node(&self, mut node: TextNode) -> anyhow::Result { // The macro renders nothing for you: add the node and your own values here. let prompt = self .prompt_template .clone() .with_node(&node) .with_context_value("product", self.product.as_str()); // `prompt` uses this transformer's client, or the pipeline default. let audience = self.prompt(prompt).await?; node.metadata.insert(NAME, audience); Ok(node) } // The macro stores `with_concurrency(n)` but does not apply it. fn concurrency(&self) -> Option { self.concurrency } } } ``` `metadata_field_name` becomes the `NAME` constant. `default_prompt_file` goes to `include_str!`, relative to the source file. The macro adds `client`, `prompt_template` and `concurrency` fields, and generates `builder()`, `new`, `from_client`, `with_concurrency`, `Default` and the `WithIndexingDefaults` impl. `derive(skip_debug, skip_clone, skip_default)` turns off generated derives. You still implement `Transformer`. Build the prompt with `with_node(&node)` before you pass it to the generated `self.prompt(...)`, which uses the transformer’s client or the pipeline default. Override `concurrency()` as the example does, or `with_concurrency` has no effect. The macro adds a `hidden` module, `NAME` and `default_prompt()` next to the struct, so give each annotated struct its own module. `new` and `Default` unwrap the builder, so give each of your own fields `#[builder(default = ...)]`. Your crate needs `swiftide-core`, `anyhow` and `derive_builder` as direct dependencies; [Crates](/reference/crates/) has the `cargo add` line. ## Batch transformers A batch transformer takes a `Vec` of nodes and returns a stream of any length. This one drops repeated chunks: ```rust /// Drops chunks that repeat an earlier chunk in the same batch, such as copied licence headers. #[derive(Clone)] pub struct DropRepeatedChunks { batch_size: usize, } #[async_trait] impl BatchableTransformer for DropRepeatedChunks { type Input = String; type Output = String; async fn batch_transform(&self, nodes: Vec) -> IndexingStream { let mut seen = HashSet::new(); let unique: Vec = nodes .into_iter() .filter(|node| seen.insert(node.chunk.trim().to_owned())) .collect(); unique.into() } fn batch_size(&self) -> Option { Some(self.batch_size) } } impl WithBatchIndexingDefaults for DropRepeatedChunks {} ``` You add it with `then_in_batch`: ```rust Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"])) .then_chunk(ChunkMarkdown::from_chunk_range(100..1500)) .then_in_batch(DropRepeatedChunks { batch_size: 64 }) .then(WordCount) .then_store_with(MemoryStorage::default()) .run() .await ``` The pipeline groups nodes into batches of `batch_size()`, or 256 when it returns `None`. `into()` turns a `Vec` into an `IndexingStream`. `then_in_batch` requires `WithBatchIndexingDefaults`, again with an empty `impl`. A closure from `Vec` to `IndexingStream` is a batch transformer too, with the pipeline batch size. `Embed` and `SparseEmbed` are batch transformers. Batches are formed in arrival order, so the example only catches repeats that land in the same batch. ## Node types `Transformer` and `BatchableTransformer` declare `Input` and `Output` types, both bound by `Chunk`. `then` returns a pipeline of the transformer’s `Output`, so a different `Output` changes the node type for every step after it. Closures and most built-in transformers, chunkers and storage backends take `String` nodes, so add a transformer that changes the node type after every step that needs `String`. ## Reference * [`Pipeline::then`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then) * [`Pipeline::then_in_batch`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_in_batch) * [`Pipeline::with_default_llm_client`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.with_default_llm_client) * [`Pipeline::with_concurrency`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.with_concurrency) * [`Transformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Transformer.html) * [`BatchableTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.BatchableTransformer.html) * [`WithIndexingDefaults`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.WithIndexingDefaults.html) * [`WithBatchIndexingDefaults`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.WithBatchIndexingDefaults.html) * [`Node`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html) and [`TextNode`](https://docs.rs/swiftide/0.34/swiftide/indexing/type.TextNode.html) * [`Chunk`](https://docs.rs/swiftide/0.34/swiftide/indexing/trait.Chunk.html) * [`Metadata`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Metadata.html) * [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html) * [`MetadataQAText`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQAText.html) * [`MetadataQACode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQACode.html) * [`MetadataSummary`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataSummary.html) * [`MetadataTitle`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataTitle.html) * [`MetadataKeywords`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataKeywords.html) * [`CompressCodeOutline`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.CompressCodeOutline.html) * [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) and [`SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html) * [`indexing_transformer`](https://docs.rs/swiftide/0.34/swiftide/attr.indexing_transformer.html) * [`Prompt`](https://docs.rs/swiftide/0.34/swiftide/prompt/struct.Prompt.html) ## Next [Embedding](/indexing/embedding/) # Generating answers > Turn retrieved documents into an answer with answers::Simple, a custom prompt and a document template that lets the answer cite the file each statement comes from. The last step of a query pipeline sends the question and the retrieved documents to a large language model (LLM). Here you set that step up so every answer names the file it came from, the way the [ask-your-codebase](/showcases/ask-your-codebase/) showcase does. `answers::Simple` needs no feature flag. The snippets below use the OpenAI client and Qdrant, which sit behind the `openai` and `qdrant` feature flags: ```sh cargo add swiftide@0.34 --features openai,qdrant ``` examples/ask-your-codebase/src/query.rs ```rust let search = SimilaritySingleEmbedding::default().with_top_k(top_k); let pipeline = query::Pipeline::from_search_strategy(search) // Rephrase the question into several, to widen what the search can match .then_transform_query( GenerateSubquestions::builder() .client(llm.clone()) .prompt_template(SUBQUESTIONS_PROMPT.into()) .build()?, ) // Embed with the same local model that embedded the chunks .then_transform_query(Embed::from_client(embedding::local_model()?)) .then_retrieve(codebase.vector_store()?) // Answer the original question from the retrieved code, each chunk under its file path .then_answer( answers::Simple::builder() .client(llm) .prompt_template(ANSWER_PROMPT.into()) .document_template(CHUNK_TEMPLATE) .build()?, ); let answered = pipeline.query(question).await?; ``` ## The answer generator `then_answer` takes an answer generator and moves the query from `Retrieved` to `Answered`. Swiftide ships one: `answers::Simple`. It renders a prompt, sends it through any client that implements `SimplePrompt`, and stores the reply as the answer. `answers::Simple::from_client(llm)` builds it with the default prompt. The builder lets you replace two templates: * `prompt_template` is the whole prompt. * `document_template` renders each retrieved document before they are joined. It is optional. Without a custom prompt, the model receives this text, shown here with `What is love?` as the question and `My context` as the documents: ```text Answer the following question based on the context provided: What is love? ## Constraints * Do not include any information that is not in the provided context. * If the question cannot be answered by the provided context, state that it cannot be answered. * Answer the question completely and format it as markdown. ## Context --- My context --- ``` The prompt template sees four variables: * `question`: the question the user asked, from `Query::original`. * `original`: the same text as `question`. * `current`: the current query, from `Query::current`. * `documents`: the retrieved documents as one string, or the text of a response transformer that changed `current` (see below). Prompts are Tera templates; [Prompts and templates](/models/prompts-and-templates/) covers the syntax. `original` and `current` need 0.33 or later; see [Upgrading](/reference/upgrading/). ## Source citations The answer can only name a file if the prompt contains the file name. The document template puts it there. examples/ask-your-codebase/src/query.rs ````rust /// Widens the question into the follow-up questions its answer involves. /// /// They are phrased like the questions generated for each chunk while indexing, so the two /// embed close together. const SUBQUESTIONS_PROMPT: &str = r" Your job is to help a search find the code that answers a developer's question about a codebase. Question: {{ question }} Write {{ num_questions }} short follow-up questions that the code answering this question would also answer. Think about the mechanisms involved: the main loop, the types and enum variants, the conditions that are checked, the functions that are called, and the configuration that changes the outcome. ## Constraints * Phrase them the way a developer would ask about this codebase, e.g. 'When does the agent stop?'. * Do not guess names of types or functions. * Respond with the original question and the follow-up questions only, one per line, prefixed with '- '. "; /// Renders a retrieved chunk under its file path, so the answer can cite it. /// /// `metadata.path` is the `PATH_FIELD` metadata; a test keeps the two in step. const CHUNK_TEMPLATE: &str = r"### {{ metadata.path }} ``` {{ content }} ``` "; /// Answers from the retrieved code and cites files. const ANSWER_PROMPT: &str = r" Answer a developer's question about a codebase, using snippets of its source code. Question: {{ question }} ## Constraints * Only use the snippets below. If they do not answer the question, say briefly what is missing. * Be concrete: name the types, functions and variants involved and explain when they apply. * Describe behaviour: what calls what, in which order, and under which conditions. * Cite the file each statement comes from in backticks, e.g. (`src/lib.rs`). * Use concise markdown. Add a short code example only if it shows how to use the API. ## Snippets {{ documents }} "; ```` `CHUNK_TEMPLATE` renders one document. Its template context is the serialized `Document`, so `content` is the chunk text and `metadata` holds the metadata fields your storage kept. ask-your-codebase writes more metadata while indexing, but its LanceDB table keeps only `path`, so `{{ metadata.path }}` becomes a heading above each chunk. `answers::Simple` joins the rendered documents with `---` lines and inserts the result as `{{ documents }}`. `ANSWER_PROMPT` then asks for a citation after each statement. With `claude-sonnet-5-5`, the answers follow it: ```text $ ./target/debug/ask-your-codebase ask --model claude-sonnet-5-5 --path swiftide-agents "How does an agent decide to stop?" # How an agent decides to stop An agent stops when something calls `Agent::stop` (or `stop_with_error`), which moves the state to `State::Stopped(StopReason)`. The main loop in `run_agent` checks that state after each step. `Agent::stop` is a no-op if the agent is already stopped (`src/agent.rs`). ## The stop mechanism - `Agent::stop(reason)` fires the `OnStop` hooks with the reason, then sets `self.state = State::Stopped(reason)`. It only takes effect at the end of the current loop iteration (`src/agent.rs`). ... Sources: - src/agent.rs - src/tools/control.rs - src/lib.rs ... ``` The `Sources:` list does not come from the model. The answered query still holds the documents it was answered from, so the showcase reads their paths back: examples/ask-your-codebase/src/query.rs ```rust impl From> for Answer { /// Keeps the distinct file paths of the retrieved chunks, most relevant first. fn from(query: Query) -> Self { let mut sources: Vec = Vec::new(); for document in query.documents() { let Some(path) = document.metadata().get(PATH_FIELD).and_then(|v| v.as_str()) else { continue; }; if !sources.iter().any(|source| source == path) { sources.push(path.to_owned()); } } Self { text: query.answer().to_owned(), sources, } } } ``` `answer` returns the text and `documents` returns the documents the answer was built from. A response transformer can reorder or trim that list first: the [docs-qa](/showcases/docs-qa/) showcase reranks its documents, so its numbered list of page and section titles comes best first. ## Original and current question A query transformer such as `GenerateSubquestions` rewrites the current query. `question` in the prompt stays the user’s question, so the model answers what was asked, not the list of subquestions. Use `current` when the model should also see what the search looked for: ```rust let answerer = answers::Simple::builder() .client(openai) .prompt_template(Prompt::from( "Answer this question: {{ original }}\n\ The search looked for: {{ current }}\n\ Cite the file of every statement in backticks.\n\n\ {{ documents }}", )) .document_template("File: {{ metadata.path }}\n{{ content }}") .build()?; ``` `answers::Simple` decides what goes into `documents` like this: * If a response transformer changed `current` after retrieval, `documents` is that text and the document template is skipped. `response_transformers::Summary` works this way; see [Reranking and response transformers](/querying/reranking-and-response-transformers/). * Otherwise each document is rendered with the document template, or as its plain content without one. The check compares `current` with the query recorded by the `Retrieved` event, so a rewritten query from before retrieval does not count as a change. A reranker that only reorders documents leaves `current` alone, so the document template still applies. [Transforming queries](/querying/transforming-queries/) explains the transformation history. ## Custom answer generators Implement the `Answer` trait to control the last step yourself. This one skips the LLM call when the retriever found nothing: ```rust /// Answers with the LLM only when the retriever found something. #[derive(Clone)] pub struct AnswerWhenFound { inner: answers::Simple, } #[async_trait] impl Answer for AnswerWhenFound { async fn answer( &self, query: Query, ) -> anyhow::Result> { if query.documents().is_empty() { return Ok(query.answered("No matching documents were found.")); } self.inner.answer(query).await } } ``` `Query::answered` stores the text and moves the query to `Answered`. The trait requires `Clone`, so derive it. Every other case goes to the wrapped `answers::Simple`. The generator goes where `answers::Simple` went: ```rust let pipeline = query::Pipeline::default() .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_answer(AnswerWhenFound { inner: answers::Simple::from_client(openai), }); let answered = pipeline.query("How do I configure retries?").await?; println!("{}", answered.answer()); ``` `then_answer` also accepts a synchronous `Fn(Query) -> Result> + Send + Sync + Clone` closure, for example a fixed answer in a test. A closure that captures a value which is not `Clone` does not compile. ## Gotchas * `prompt_template` takes a `Prompt`, so write `.into()` or `Prompt::from`. `document_template` converts for you, so a `&'static str` works directly. * A document template that reads a missing metadata key fails to render, and the query returns that error. The [docs-qa](/showcases/docs-qa/) templates guard each key with Tera’s `default` filter, as in `{{ metadata.Title | default(value="") }}`. * After answering, `current` holds the answer; see [Transforming queries](/querying/transforming-queries/) for reading the rewritten query from `history`. * The model can only cite files among the retrieved documents. A chunk outside the top results is invisible to it. Raise `top_k` for broad questions; see [Retrieval and search strategies](/querying/retrieval-and-search-strategies/). * `answers::Simple` makes one LLM call and returns its error. Wrap the client in `LanguageModelWithBackOff` to retry transient failures, as docs-qa does; see [Errors, backoff and usage](/models/errors-backoff-and-usage/). ## Reference * [`query::Pipeline::then_answer`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html#method.then_answer) * [`answers::Simple`](https://docs.rs/swiftide/0.34/swiftide/query/answers/struct.Simple.html) * [`Answer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Answer.html) * [`Query`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Query.html) * [`Document`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Document.html) * [`query_transformers::GenerateSubquestions`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.GenerateSubquestions.html) * [`TransformationEvent`](https://docs.rs/swiftide/0.34/swiftide/query/enum.TransformationEvent.html) * [`Prompt`](https://docs.rs/swiftide/0.34/swiftide/prompt/struct.Prompt.html) * [`SimplePrompt`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SimplePrompt.html) * [`response_transformers::Summary`](https://docs.rs/swiftide/0.34/swiftide/query/response_transformers/struct.Summary.html) * [`LanguageModelWithBackOff`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.LanguageModelWithBackOff.html) ## Next [Evaluating with Ragas](/querying/evaluation/) # Evaluating with Ragas > Record each question's retrieved documents and answer with the Ragas evaluator, and export them with ground truths as a JSON dataset for the Python ragas toolkit. You end with a JSON file that holds every question, the answer your query pipeline gave, the documents it retrieved and the answer you expected. You can score that file and compare it between runs. The scores at the end come from the evaluation in the [docs-qa](/showcases/docs-qa/) showcase. ## Evaluation run `Ragas` needs no feature flag. The snippets below use the OpenAI client and Qdrant, which sit behind the `openai` and `qdrant` feature flags: ```sh cargo add swiftide@0.34 --features openai,qdrant ``` ```rust let ragas = Ragas::from_prepared_questions(vec![ ( "What does a retriever return?".to_string(), "A retriever returns documents that match the query.".to_string(), ), ( "Which step turns retrieved documents into an answer?".to_string(), "The answer generator, for example answers::Simple.".to_string(), ), ]); let pipeline = query::Pipeline::default() // Attach the evaluator while the pipeline is still pending .evaluate_with(ragas.clone()) .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)); pipeline.query_all(ragas.questions().await).await?; std::fs::write("ragas.json", ragas.to_json().await)?; ``` `Ragas` is an evaluator: a type that implements `EvaluateQuery`. It holds the dataset, and its clones share that dataset. Pass one clone to the pipeline with `evaluate_with` and keep the other to read the results. `ragas.questions()` returns one pending query per question in the dataset, and `query_all` runs them all. The pipeline calls the evaluator twice per question. Right after `then_retrieve`, `Ragas` stores the content of the retrieved documents as the contexts of that question. After `then_answer`, it stores the answer. `to_json` then writes the dataset. ### Evaluation in docs-qa The [docs-qa](/showcases/docs-qa/) showcase runs the same steps on its own question set: examples/docs-qa/src/eval.rs ```rust let ragas = Ragas::from_prepared_questions(&questions); let answered = qa .evaluated_pipeline(ragas.clone()) .query_all(ragas.questions().await) .await?; let ragas_json = ragas.to_json().await; ``` `evaluated_pipeline` builds the same pipeline as the `ask` command and adds `evaluate_with(ragas)` to it. docs-qa also enables the `fastembed` feature flag for its hybrid search. ## Questions and ground truths `Ragas::from_prepared_questions` takes anything that converts into an `EvaluationDataSet`. An `EvaluationDataSet` converts from: * `Vec` or `&[String]`: questions only, with an empty ground truth. * `Vec<(String, String)>`: pairs of question and ground truth, as in the first snippet. * a JSON string, through `parse`, in the format that `to_json` writes. docs-qa passes `&QuestionSet`, its own YAML-backed type, which implements `From<&QuestionSet> for EvaluationDataSet`. Implement `From` for your own type in the same way. A ground truth is the answer you expect, written in your own words. Scores that compare the answer with the ground truth need it, so write one for every question you can. ### Baseline ground truths Writing ground truths by hand takes time. You can let a first run write them and review them afterwards. ```rust // Runs the pipeline once over every question in the dataset let evaluate = async |ragas: &Ragas| -> anyhow::Result<()> { query::Pipeline::default() .evaluate_with(ragas.clone()) .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant.clone()) .then_answer(answers::Simple::from_client(openai.clone())) .query_all(ragas.questions().await) .await?; Ok(()) }; // First run: questions only, then keep this run's answers as the ground truth let questions = vec![ "What does a retriever return?".to_string(), "Which step turns retrieved documents into an answer?".to_string(), ]; let ragas = Ragas::from_prepared_questions(questions); evaluate(&ragas).await?; ragas.record_answers_as_ground_truth().await; std::fs::write("baseline.json", ragas.to_json().await)?; // Later runs: load the reviewed baseline and evaluate against it let dataset: EvaluationDataSet = std::fs::read_to_string("baseline.json")?.parse()?; let ragas = Ragas::from_prepared_questions(dataset); evaluate(&ragas).await?; ``` The `evaluate` closure builds a pipeline with `evaluate_with(ragas.clone())` and runs `query_all` over `ragas.questions()`, so each call fills the dataset with answers and contexts. `record_answers_as_ground_truth` then copies each recorded answer into its ground truth. Read the file and fix wrong answers before you treat it as the baseline. Parsing an export keeps its ground truths. The next run overwrites the answers and contexts it loaded. ## The dataset The export is a pretty-printed JSON array with one object per question. This is the output that the unit test for `to_json` expects for one question with a ground truth that has not run yet: ```json [ { "answer": "", "contexts": [], "ground_truth": "A programming language", "question": "What is Rust?" } ] ``` The keys are in alphabetical order, unless a crate in your build enables the serde_json feature that preserves insertion order. The rows are in hash map order, which changes from run to run. docs-qa sorts the rows back into the order of its questions file, so the exports of two runs can be diffed. `contexts` holds every retrieved document, recorded before any response transformer runs. docs-qa retrieves 20 candidates and reranks them down to 10, so its `ragas.json` has 20 contexts per question, while the answer generator saw 10. Ragas metrics computed from `contexts` therefore score the retriever’s 20 candidates, not the 10 documents the answer generator saw. ## Scores `Ragas` computes no scores. The JSON is input for the Python [ragas](https://docs.ragas.io) toolkit, which calculates metrics such as faithfulness and context recall. docs-qa shows numbers without Python. Its judge asks a large language model (LLM) to split the ground truth and the answer into claims and to check each one, using [structured output](/models/structured-output/). Its scores are ratios modelled on the ragas metrics of the same names. In this run `claude-sonnet-5-5` both answered and judged: ```sh ./target/debug/docs-qa --model claude-sonnet-5-5 eval --judge-model claude-sonnet-5-5 \ --out examples/docs-qa/ragas.json ``` ```text # question ctx faithful answer context recall recall 1 Which crate features and environment variable do I … 10 1.00 1.00 1.00 2 What does the agents-contrib-context feature provid… 10 1.00 1.00 1.00 3 What are the default limits of the read_file tool? 10 1.00 1.00 1.00 4 Which storage and retrieval backends does Swiftide … 10 1.00 1.00 1.00 5 What should I run before opening a pull request? 10 1.00 1.00 1.00 6 What is a task in Swiftide? 10 1.00 1.00 1.00 7 When does ContextCompression summarize the conversa… 10 1.00 1.00 1.00 8 Which commit message convention does the project fo… 10 1.00 1.00 1.00 9 How do I add a custom tool to an agent with the too… 10 1.00 1.00 1.00 mean 1.00 1.00 1.00 RAGAS dataset written to examples/docs-qa/ragas.json ``` * `ctx` is the number of chunks the LLM saw after reranking. * `faithful` is the share of the answer’s claims that those chunks support. * `answer recall` is the share of the ground truth’s claims that the answer states. * `context recall` is the share of the ground truth’s claims that those chunks support. Before retrieval was tuned, two questions scored lower. One fix added page and title to the rerank template; see [Rerank templates](/querying/reranking-and-response-transformers/#rerank-templates). The other changed the indexing prompt so code-only chunks get titles that name the API they show; see the [docs-qa showcase](/showcases/docs-qa/). ## Custom evaluators Any type that implements `EvaluateQuery` can take the place of `Ragas`. The trait has one async method, `evaluate`, which receives a `QueryEvaluation`: `RetrieveDocuments` with the query after retrieval, or `AnswerQuery` with the answered query. The trait requires `Clone`, so keep shared state behind an `Arc`. This evaluator collects the questions for which the retriever found no documents: ```rust /// Flags questions for which the retriever found nothing. #[derive(Clone, Default)] pub struct EmptyRetrievals { questions: Arc>>, } #[async_trait] impl EvaluateQuery for EmptyRetrievals { async fn evaluate(&self, evaluation: QueryEvaluation) -> anyhow::Result<()> { if let QueryEvaluation::RetrieveDocuments(query) = evaluation && query.documents().is_empty() { self.questions .lock() .unwrap() .push(query.original().to_string()); } Ok(()) } } ``` ```rust let empty = EmptyRetrievals::default(); query::Pipeline::default() .evaluate_with(empty.clone()) .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)) .query("What does a retriever return?") .await?; for question in empty.questions.lock().unwrap().iter() { println!("no documents for: {question}"); } ``` An error from `evaluate` fails the query, so return `Ok` for anything that should only be recorded. ## Gotchas * `evaluate_with` exists only on a pending pipeline. Call it before `then_retrieve`; its position among query transformers does not matter. * `Ragas` matches each query to its row by `original()`. A query whose question is not in the dataset fails with `Question not found`, and so does `query_all`. Query transformers change `current()`, not `original()`, so they do not affect the match. * Rows are keyed by question text, so each question must be unique. docs-qa rejects a questions file with duplicates. * `query_all` returns answers in the order they finish; see [Query methods](/querying/overview/#query-methods). docs-qa matches each answer back to its question with `original()`. * Without the docs-qa workaround in [Retrieval and search strategies](/querying/retrieval-and-search-strategies/), Qdrant contexts in the export arrive `Debug`-quoted in 0.33. * Nine questions and one LLM judge make a regression signal for one corpus, not a benchmark. Keep the question set and the judge model fixed when you compare runs. ## Reference * [`Ragas`](https://docs.rs/swiftide/0.34/swiftide/query/evaluators/ragas/struct.Ragas.html) * [`EvaluationDataSet`](https://docs.rs/swiftide/0.34/swiftide/query/evaluators/ragas/struct.EvaluationDataSet.html) * [`EvaluateQuery`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EvaluateQuery.html) * [`QueryEvaluation`](https://docs.rs/swiftide/0.34/swiftide/query/enum.QueryEvaluation.html) * [`query::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html) * [`Query`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Query.html) * [docs-qa source on GitHub](https://github.com/bosun-ai/swiftide/tree/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa) ## Next [Models](/models/overview/) # How querying works > A query pipeline takes a question from Pending to Retrieved to Answered through query transformers, a retriever, response transformers and an answer generator. You end up with a query pipeline that answers questions about your own code and names the files each answer came from. It is the query side of the [ask-your-codebase](/showcases/ask-your-codebase/) showcase, a retrieval augmented generation (RAG) command line tool. The pipeline reads the `code_chunks` LanceDB table that the showcase’s [indexing pipeline](/indexing/overview/) fills with the same embedding model, so run `ask-your-codebase index` first. The full program needs the `openai`, `fastembed` and `lancedb` feature flags. The helpers and prompt constants live in the showcase. ## The ask-your-codebase pipeline examples/ask-your-codebase/src/query.rs ```rust let search = SimilaritySingleEmbedding::default().with_top_k(top_k); let pipeline = query::Pipeline::from_search_strategy(search) // Rephrase the question into several, to widen what the search can match .then_transform_query( GenerateSubquestions::builder() .client(llm.clone()) .prompt_template(SUBQUESTIONS_PROMPT.into()) .build()?, ) // Embed with the same local model that embedded the chunks .then_transform_query(Embed::from_client(embedding::local_model()?)) .then_retrieve(codebase.vector_store()?) // Answer the original question from the retrieved code, each chunk under its file path .then_answer( answers::Simple::builder() .client(llm) .prompt_template(ANSWER_PROMPT.into()) .document_template(CHUNK_TEMPLATE) .build()?, ); let answered = pipeline.query(question).await?; ``` The pipeline runs four steps for each question: 1. `GenerateSubquestions` asks the large language model (LLM) to rewrite the question into several related questions. Their text becomes the current query. 2. `Embed` embeds the current query with the local FastEmbed model that also embedded the chunks during indexing. 3. `then_retrieve` hands the query to the LanceDB storage, the retriever. It returns the `top_k` most similar chunks as documents. 4. `answers::Simple`, the answer generator, renders the documents into a prompt and asks the LLM for the answer. `SimilaritySingleEmbedding` is the search strategy. [Retrieval and search strategies](/querying/retrieval-and-search-strategies/) explains the `let mut` binding and its type annotation. This is the start of a real run against the `swiftide-agents` crate. The answer came from `claude-sonnet-5-5`, and the embeddings from `jina-embeddings-v2-base-code` running locally: ```text $ ./target/debug/ask-your-codebase ask --model claude-sonnet-5-5 --path swiftide-agents "How does an agent decide to stop?" # How an agent decides to stop An agent stops when something calls `Agent::stop` (or `stop_with_error`), which moves the state to `State::Stopped(StopReason)`. The main loop in `run_agent` checks that state after each step. `Agent::stop` is a no-op if the agent is already stopped (`src/agent.rs`). ... Sources: - src/agent.rs - src/tools/control.rs - src/lib.rs - src/state.rs - src/errors.rs - src/task_agent.rs - src/hooks.rs - src/tasks.rs - src/default_context.rs ``` The sources list comes from the `path` metadata of the documents in the answered query. [Generating answers](/querying/answering/) shows the showcase’s `sources` code and the document template that puts the file path in front of each chunk. ## Query states  A question travels through the pipeline as a `Query`. The state is a type parameter, and it moves from `states::Pending` to `states::Retrieved` to `states::Answered`. A query keeps the original question in `original()` and the text of the latest step in `current()`. A query transformer sets `current()` to the rewritten question. A response transformer such as `Summary` can replace it with a summary, and `answers::Simple` then uses that text in place of the documents. The retriever adds documents, which you read with `documents()`. Once answered, `current()` holds the answer, and `answer()` returns it. The pipeline carries the same state, with different methods per state: | Pipeline state | Methods | | ------------------- | -------------------------------------------------------- | | `states::Pending` | `then_transform_query`, `then_retrieve`, `evaluate_with` | | `states::Retrieved` | `then_transform_response`, `then_answer` | | `states::Answered` | `query`, `query_mut`, `query_all` | The compiler enforces this order. A pipeline without `then_retrieve` has no `then_answer`, and a pipeline without `then_answer` has no `query`. ## Pipeline steps Each step method takes one trait: | Step | Trait | Called | | ------------------------- | ---------------------------------- | ------------------------------------- | | `then_transform_query` | `TransformQuery` | any number of times, before retrieval | | `then_retrieve` | `Retrieve` for the search strategy | once | | `then_transform_response` | `TransformResponse` | any number of times, after retrieval | | `then_answer` | `Answer` | once | Every step accepts a struct that implements the trait, a `Box` or `Arc` of the trait, or a closure. Closures are synchronous and must be `Clone + Send + Sync`. Each step runs every query on its own Tokio task with `tokio::spawn`, so the pipeline needs a Tokio runtime. ### Closure steps ```rust let pipeline = query::Pipeline::default() .then_transform_query(|mut query: Query| { let lowercased = query.current().to_lowercase(); query.transformed_query(lowercased); Ok(query) }) .then_retrieve( |_: &SimilaritySingleEmbedding, query: Query| { let documents = vec![Document::from( "Swiftide is a Rust library for LLM applications.", )]; Ok(query.retrieved_documents(documents)) }, ) .then_transform_response(Ok) .then_answer(|query: Query| { let answer = format!("Found {} document(s)", query.documents().len()); Ok(query.answered(answer)) }); let answered = pipeline.query("What is Swiftide?").await?; println!("{}", answered.answer()); ``` This pipeline needs no model and no storage, which makes it a starting point for tests. `retrieved_documents` and `answered` are the only ways to move a `Query` to the next state, and `transformed_query` records each rewrite in `history()`. ## Search strategy `query::Pipeline::default()` uses `SimilaritySingleEmbedding` with a `top_k` of 10 and no filter. Put an `Embed` query transformer before `then_retrieve`. The retriever searches with the embedding it sets. `query::Pipeline::from_search_strategy` starts a pipeline with any other strategy: `SimilaritySingleEmbedding` with a different `top_k` or a filter, `HybridSearch` or `CustomStrategy`. The retriever must implement `Retrieve` for that strategy. If it does not, the pipeline does not compile. [Retrieval and search strategies](/querying/retrieval-and-search-strategies/) lists which storage supports which strategy. ## Query methods ```rust let mut pipeline = query::Pipeline::default() .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant.clone()) .then_answer(answers::Simple::from_client(openai.clone())); let first = pipeline.query_mut("What is a query pipeline?").await?; let second = pipeline.query_mut("What does a retriever return?").await?; println!("{}\n\n{}", first.answer(), second.answer()); let answered = query::Pipeline::default() .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)) .query_all(vec!["What is a node?", "What is a chunk?"]) .await?; for query in answered { println!("{}: {}", query.original(), query.answer()); } ``` `query` consumes the pipeline. `query_mut` borrows the pipeline mutably, so you can call it again for the next question. `query_all` takes a `Vec` of questions, runs them through the steps concurrently and returns every answered query. `query_all` returns results in the order they finish, not in the order you passed them. Match each result to its question with `original()`. An empty `Vec` never returns, because the pipeline waits for an answer that does not come. The first error ends the call, and the answers that already finished are lost. Every query method accepts anything that implements `AsRef`, such as `&str` or `String`. ## Gotchas * The pipeline steps are lazy. No step runs until `query`, and errors from a model or a storage show up as the error of `query`. Code that builds a step can still do work: in the showcase, `embedding::local_model()` loads the FastEmbed model, and downloads it on first use, before the pipeline exists. * `query` and `query_mut` log `Answered query in N seconds` for every question, and `query_mut` also logs `Sending query`. `query_all` logs `Sending queries` and `Answered all queries in N seconds` once per batch. All are at warn level, so filter the `swiftide_query` target in your tracing subscriber if that is too noisy. * Query transformers, response transformers and answer generators must be owned and `'static`. Pass clients by value and `clone()` the ones you use in more than one step. `then_retrieve` also accepts a reference, such as `&qdrant`, and clones it. * A closure that captures a value without `Clone`, such as a lock guard, does not implement the step trait and fails with a trait-bound error. * Each step runs up to one query per logical central processing unit (CPU) at the same time. The pipeline has no setting to change that. * The source of the query pipeline marks it as being in an early stage. Expect its application programming interface (API) to change between releases, and check [Upgrading](/reference/upgrading/) when you bump Swiftide. * The query pipeline needs no feature flag. The models and retrievers do, for example `openai`, `fastembed`, `lancedb` or `qdrant`. See [Feature flags](/reference/feature-flags/). ## Reference * [`query::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html) * [`Query`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Query.html) * [`states`](https://docs.rs/swiftide/0.34/swiftide/query/states/index.html) * [`Document`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Document.html) * [`TransformQuery`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.TransformQuery.html) * [`Retrieve`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Retrieve.html) * [`TransformResponse`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.TransformResponse.html) * [`Answer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Answer.html) * [`SimilaritySingleEmbedding`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.SimilaritySingleEmbedding.html) * [`GenerateSubquestions`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.GenerateSubquestions.html) * [`query_transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.Embed.html) * [`answers::Simple`](https://docs.rs/swiftide/0.34/swiftide/query/answers/struct.Simple.html) * [`HybridSearch`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.HybridSearch.html) * [`CustomStrategy`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.CustomStrategy.html) * [`EvaluateQuery`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EvaluateQuery.html) * [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/type.OpenAI.html) * [`Qdrant`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.Qdrant.html) * [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html) * [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html) ## Next [Transforming queries](/querying/transforming-queries/) # Reranking and response transformers > Reorder and trim retrieved documents with the FastEmbed Rerank cross-encoder, condense them into an LLM Summary that the answer generator uses instead, or filter them with your own TransformResponse. Your query pipeline fetches twenty candidate documents, keeps the ten a local reranker scores highest, and answers from those. You can also summarize the documents with a large language model (LLM) or filter them with your own response transformer. ## FastEmbed reranker The docs-qa showcase builds its reranker next to its answer generator: examples/docs-qa/src/query.rs ```rust let reranker = Rerank::builder() .top_k(CONTEXT_CHUNKS) .document_template(Some(RERANK_TEMPLATE.to_string())) .build()?; let answerer = answers::Simple::builder() .client(llm) .prompt_template(ANSWER_PROMPT.into()) .document_template(DOCUMENT_TEMPLATE) .build()?; ``` It then places the reranker between retrieval and the answer: examples/docs-qa/src/query.rs ```rust pipeline // Hybrid search needs both a dense and a sparse embedding of the question .then_transform_query(query_transformers::Embed::from_client( self.embedders.dense.clone(), )) .then_transform_query(query_transformers::SparseEmbed::from_client( self.embedders.sparse.clone(), )) .then_retrieve(self.qdrant.clone()) // A local cross-encoder reorders the fused candidates; only the best few remain .then_transform_response(self.reranker.clone()) .then_answer(self.answerer.clone()) ``` `Rerank` is a response transformer. A response transformer runs after `then_retrieve` and before `then_answer`. It takes the query in the `Retrieved` state and returns it in the same state. `Rerank` runs a cross-encoder: a model that reads the question and one document together and returns a relevance score. That is slower than comparing embeddings, so you run it only on the candidates the retriever already found. The builder has four fields: * `top_k`: how many documents to keep, as a `usize`. The default is 10. * `document_template`: an optional Tera template that renders each document before scoring. The setter takes an `Option`, so pass `Some(..)`. * `model`: the FastEmbed `TextRerank` model, wrapped in `Arc>`. The default is BGE reranker base. * `model_batch_size`: an optional batch size. The default is FastEmbed’s own. `Rerank` scores every document against `query.original()`, the question as the user asked it. Query transformers such as `GenerateSubquestions` replace `current()`, which the retriever searches with and which `answers::Simple` also sees. The reranker ignores `current()`. It replaces the query’s documents with the `top_k` highest-scoring ones, best first. `Rerank` lives in `swiftide::integrations::fastembed` behind the `fastembed` feature flag: ```sh cargo add swiftide@0.34 --features fastembed ``` This is a docs-qa answer, written by `claude-sonnet-5-5` and reranked locally with BGE reranker base. The sources list the pages and sections of the kept documents, best first: ```text Swiftide supports OpenAI and Azure OpenAI, Anthropic, Gemini, OpenRouter, AWS Bedrock Converse API, Groq, Ollama, and Dashscope. Integrations are feature-gated, so you enable a provider by adding its Cargo feature to `swiftide`. ... Sources: [1] Swiftide › Swiftide capabilities for agents, typed task graphs, pipelines, and tracing (README.md) [2] Swiftide › Swiftide: Rust framework for LLM agents, task graphs, and RAG (README.md) [3] Swiftide › Swiftide Integrations Overview by Feature-Gated Category (README.md) ... [9] Swiftide › Swiftide agent tool defined with the swiftide::tool macro (README.md) [10] Swiftide › Installing Swiftide and Setting Up OpenAI Integration (README.md) ``` The `openai` feature flag in that answer comes from the tenth source, the Quick Start chunk. With a `top_k` below 10 it would not have reached the model. Check a few answers before you lower `top_k`. ### Rerank templates examples/docs-qa/src/query.rs ```rust // The answer prompt receives the question and the rendered documents; each document is rendered // with its page title and the chunk title written during indexing. const ANSWER_PROMPT: &str = "You answer questions about Swiftide, a Rust library for building \ LLM applications, using excerpts from its documentation. Question: {{ question }} ## Instructions * Use only the documentation excerpts below. If they do not answer the question, say so plainly. * Start with the direct answer in one or two sentences, then add supporting details. * Only when the question asks how to do something, include the feature flags, commands or a \ short Rust snippet from the excerpts. Never invent APIs. * Format the answer as markdown and keep it under 250 words. Do not refer to \"the excerpts\". ## Documentation excerpts {{ documents }} "; const DOCUMENT_TEMPLATE: &str = "### {{ metadata.Page | default(value=\"\") }}: \ {{ metadata.Title | default(value=\"\") }} {{ content }}"; // The reranker scores the same title-first rendering. A chunk that is mostly code says little // about what it shows, so its page and the title written during indexing go first. const RERANK_TEMPLATE: &str = "{{ metadata.Page | default(value=\"\") }}: \ {{ metadata.Title | default(value=\"\") }} {{ content }}"; ``` `RERANK_TEMPLATE` puts the page name and the chunk title, both written during indexing, before the content. A chunk that is mostly code says little about its topic, so the extra context helps it score fairly. The template only changes what the model scores. The documents stay as they were, and the answer generator renders them with its own `document_template`. In docs-qa this template fixed one evaluation result. For “What is a task in Swiftide?”, the chunk that lists what a task supports ranked 14th when the reranker scored only chunk text, so it was dropped. The answer missed two features, with answer recall 0.71 and context recall 0.57. With page and title in the template, the chunk made the kept ten and both scores reached 1.00. [Evaluating with Ragas](/querying/evaluation/#scores) explains these scores. Model download, cache location and load failures are covered in [FastEmbed](/integrations/fastembed/). ### Minimal pipeline The single-file example builds its reranker in its setup: ```rust let openai_client = integrations::openai::OpenAI::builder() .default_prompt_model("gpt-4o") .build()?; let fastembed = fastembed::FastEmbed::builder().batch_size(10).build()?; let reranker = fastembed::Rerank::builder().top_k(5).build()?; let qdrant = Qdrant::builder() .batch_size(50) .vector_size(384) .collection_name("swiftide-reranking") .build()?; ``` It then generates subquestions for retrieval, while the reranker still scores against the original question: ```rust // By default the search strategy is SimilaritySingleEmbedding // which takes the latest query, embeds it, and does a similarity search let pipeline = query::Pipeline::default() .then_transform_query(query_transformers::GenerateSubquestions::from_client( openai_client.clone(), )) .then_transform_query(query_transformers::Embed::from_client(fastembed.clone())) .then_retrieve(qdrant.clone()) .then_transform_response(reranker) .then_answer(answers::Simple::from_client(openai_client.clone())); ``` ## Summaries ```rust // By default the search strategy is SimilaritySingleEmbedding // which takes the latest query, embeds it, and does a similarity search let pipeline = query::Pipeline::default() .then_transform_query(query_transformers::GenerateSubquestions::from_client( openai_client.clone(), )) .then_transform_query(query_transformers::Embed::from_client( openai_client.clone(), )) .then_retrieve(qdrant.clone()) .then_transform_response(response_transformers::Summary::from_client( openai_client.clone(), )) .then_answer(answers::Simple::from_client(openai_client.clone())); ``` `response_transformers::Summary` sends the retrieved documents to an LLM and asks for one summary of them. It stores the reply with `transformed_response`, which records a transformation event and replaces `query.current()`. It leaves `documents()` untouched. `answers::Simple` then answers from the summary instead of the rendered documents; see [Generating answers](/querying/answering/). The documents stay on the query, so you can still list sources. To change the instruction, pass your own `prompt_template`: ```rust let summary = response_transformers::Summary::builder() .client(openai) .prompt_template(Prompt::from( "Summarize these excerpts as one markdown list. Keep every type name, \ feature flag and command exactly as written.\n\ {% for document in documents %}\n\ ---\n\ {{ document.metadata.path | default(value=\"unknown\") }}:\n\ {{ document.content }}\n\ {% endfor %}", )) .build()?; ``` The template sees only `documents`, a list of documents with `content` and `metadata`. The question is not in its context. You can chain response transformers. Put `Rerank` before `Summary` and the LLM summarizes only the kept documents. ## Custom response transformers ```rust /// Keeps the first document of each file, in retrieval order, for at most `max` files. #[derive(Clone)] pub struct OnePerFile { pub max: usize, } #[async_trait] impl TransformResponse for OnePerFile { async fn transform_response( &self, mut query: Query, ) -> Result> { let mut seen_paths = HashSet::new(); let documents = query.documents_mut(); documents.retain(|document| { let path = document .metadata() .get("path") .and_then(|value| value.as_str()) .unwrap_or_default(); seen_paths.insert(path.to_string()) }); documents.truncate(self.max); Ok(query) } } ``` `TransformResponse` has one async method, `transform_response`. Implement it with `#[async_trait]`, from the `async-trait` crate or `swiftide::reexports::async_trait`. The trait requires `Clone` through `DynClone`, so derive it. This transformer edits the list in place through `documents_mut`. It keeps the first document per `path` metadata value, in the order the documents arrive, and caps the list at `max`. Documents without a `path` share one empty key, so only the first of them is kept. It does not call `transformed_response`, so `answers::Simple` still renders the documents one by one. Call `transformed_response` only when your transformer produces text for the answer to use in place of the documents, as `Summary` does. Place it after `then_retrieve`: ```rust let pipeline = query::Pipeline::default() .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_transform_response(OnePerFile { max: 8 }) .then_answer(answers::Simple::from_client(openai)); ``` For a small change, a closure works too. It must be synchronous, take a `Query` and return `anyhow::Result>`. It must also be `Clone + Send + Sync`, so capture only values that are themselves `Clone`: ```rust let pipeline = query::Pipeline::default() .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_transform_response(|mut query: Query| { query.documents_mut().truncate(5); Ok(query) }) .then_answer(answers::Simple::from_client(openai)); ``` ## Gotchas * In 0.33 the plain `Qdrant` retriever returns `Debug`-quoted content, so `Rerank` scores the quoted text. See the gotcha in [Retrieval and search strategies](/querying/retrieval-and-search-strategies/#gotchas) for the docs-qa workaround. * `Rerank` changes neither `current()` nor the transformation history. The `Retrieved` events in `history()` hold copies of the documents as the retriever returned them. Comparing them with `documents()` shows what was dropped only if no wrapper edited the documents after retrieval and the query retrieved once. * The `Ragas` evaluator records contexts before `Rerank` runs, so its dataset holds the retrieved candidates, not the kept documents; see [The dataset](/querying/evaluation/#the-dataset). A custom evaluator can read the reranked documents at its second call, after the answer. * To use another rerank model, build a `TextRerank` yourself, wrap it in `Arc::new(tokio::sync::Mutex::new(..))` and pass it to `model`. That needs direct `fastembed` and `tokio` dependencies, because Swiftide does not re-export `fastembed`. * Clones of a `Rerank` share one model behind a Tokio mutex. Scoring is synchronous CPU work, so concurrent queries, such as those from `query_all`, take turns at the rerank step. * If retrieval returns `top_k` documents or fewer, `Rerank` only reorders them. Retrieve more candidates than you keep. ## Reference * [`Rerank`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.Rerank.html) * [`RerankBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.RerankBuilder.html) * [`response_transformers::Summary`](https://docs.rs/swiftide/0.34/swiftide/query/response_transformers/struct.Summary.html) * [`TransformResponse`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.TransformResponse.html) * [`Query`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Query.html) * [`Document`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Document.html) * [`query::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html) * [`answers::Simple`](https://docs.rs/swiftide/0.34/swiftide/query/answers/struct.Simple.html) * The [docs-qa showcase](/showcases/docs-qa/) and the [FastEmbed integration](/integrations/fastembed/) ## Next [Generating answers](/querying/answering/) # Retrieval and search strategies > Pick a search strategy (similarity, hybrid or custom), set top_k and filters, and pass it to a Qdrant, PgVector, LanceDB or DuckDB storage that acts as the retriever. A search strategy decides how many documents come back and how a retriever finds them. You pick a strategy, set `top_k` and a filter, and hand it to your storage, which acts as the retriever: Qdrant, PgVector, LanceDB or DuckDB. ## Similarity search examples/ask-your-codebase/src/query.rs ```rust let search = SimilaritySingleEmbedding::default().with_top_k(top_k); let pipeline = query::Pipeline::from_search_strategy(search) // Rephrase the question into several, to widen what the search can match .then_transform_query( GenerateSubquestions::builder() .client(llm.clone()) .prompt_template(SUBQUESTIONS_PROMPT.into()) .build()?, ) // Embed with the same local model that embedded the chunks .then_transform_query(Embed::from_client(embedding::local_model()?)) .then_retrieve(codebase.vector_store()?) // Answer the original question from the retrieved code, each chunk under its file path .then_answer( answers::Simple::builder() .client(llm) .prompt_template(ANSWER_PROMPT.into()) .document_template(CHUNK_TEMPLATE) .build()?, ); let answered = pipeline.query(question).await?; ``` `SimilaritySingleEmbedding` searches with the dense embedding that the `Embed` query transformer put on the query. It returns `top_k` documents. The default is 10. ask-your-codebase searches a LanceDB table and sets `top_k` from its `--top-k` flag, which defaults to 20. `with_top_k` takes the strategy by value and returns it, so it chains onto `default()` or `from_filter`. `SimilaritySingleEmbedding::default()` exists only for the unfiltered strategy, so its filter type is inferred as `()`, even on stores that also retrieve with a typed filter. For a filtered search, start from `from_filter` or call `with_filter`, as in [Filters](#filters). ## Hybrid search examples/docs-qa/src/query.rs ```rust query::Pipeline::from_search_strategy( HybridSearch::default() .with_top_n(CANDIDATES) .with_top_k(CANDIDATES), ) ``` `HybridSearch` combines a keyword or sparse search with a dense search. `top_n` caps the candidates of each search. `top_k` caps the documents that come back. Both default to 10. docs-qa fetches 20 candidates and later reranks them down to 10. `with_top_n`, `with_top_k`, `with_dense_vector_field` and `with_sparse_vector_field` take the strategy by value and return it, so they chain without `.to_owned()`. `HybridSearch` implements `Default` for every filter type. When nothing else fixes the filter type, name it, as docs-qa does with `HybridSearch` in its return type, or the compiler stops with E0283. On Qdrant, the query needs a dense and a sparse embedding, so the pipeline runs both query transformers before it retrieves. The example creates two FastEmbed models in its setup: `fastembed` for dense and `fastembed_sparse` for sparse embeddings. ```rust // Use sophisticated model for our query let openai = openai::OpenAI::builder() .default_prompt_model("gpt-4o") .build() .unwrap(); let query_pipeline = query::Pipeline::from_search_strategy( // Return a large amount of documents because we have a large context window // By default it uses the Combined fields, no need to configure HybridSearch::default().with_top_n(20).with_top_k(20), ) // Generate subquestions on the initial query to increase our query coverage .then_transform_query(query_transformers::GenerateSubquestions::from_client( openai.clone(), )) // Generate the same embeddings we used for indexing .then_transform_query(query_transformers::Embed::from_client(fastembed.clone())) .then_transform_query(query_transformers::SparseEmbed::from_client( fastembed_sparse.clone(), )) .then_retrieve(qdrant.clone()) // Answer with Simple, which either takes the documents as is (in this case), or any // transformations applied after querying .then_answer(answers::Simple::from_client(openai.clone())); ``` FastEmbed is the only sparse embedding model in Swiftide, so Qdrant hybrid search needs the `qdrant` and `fastembed` feature flags. The chunks need sparse embeddings too. Run the `SparseEmbed` transformer in the indexing pipeline next to `Embed`: ```rust indexing::Pipeline::from_loader(FileLoader::new("swiftide-core/").with_extensions(&["rs"])) // Chunk fairly large as the context window is big .then_chunk(ChunkCode::try_for_language_and_chunk_size( "rust", 10..2048, )?) // Generate metadata on the code chunks to increase our chances of finding the right code .then(MetadataQACode::from_client(openai.clone()).build().unwrap()) .then_in_batch( transformers::SparseEmbed::new(fastembed_sparse.clone()).with_batch_size(batch_size), ) .then_in_batch(transformers::Embed::new(fastembed.clone()).with_batch_size(batch_size)) .then_store_with(qdrant.clone()) .run() .await?; ``` Qdrant runs a sparse and a dense search, each limited to `top_n`. It merges them with reciprocal rank fusion (RRF) and keeps `top_k`. The collection needs a sparse vector next to the dense one; see [Qdrant](/integrations/qdrant/). DuckDB runs hybrid search without a sparse embedding. It scores the chunk text with BM25 full-text search over `query.current()` and the dense embedding with cosine similarity. Each score is normalized to its maximum, and the final score is 0.8 times the dense score plus 0.2 times the full-text score. After `GenerateSubquestions`, the keyword text is the whole list of generated questions. ## Filters A filter narrows the search to documents with matching metadata. Each storage takes its own filter type. Qdrant takes a `Filter` from its client, which Swiftide re-exports at `swiftide::integrations::qdrant::qdrant_client`: ```rust let only_readme = Filter::must([Condition::matches("path", "README.md".to_string())]); let pipeline = query::Pipeline::from_search_strategy(SimilaritySingleEmbedding::from_filter( only_readme.clone(), )) .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)); // The same filter on a hybrid search applies to the dense and the sparse search let _hybrid = HybridSearch::from_filter(only_readme).with_top_n(20); ``` PgVector and LanceDB take a `String`. PgVector expects `key = value` and compares it with the metadata column `meta_`, so the key must be a metadata field on the table. Swiftide does not put the node path into metadata. To filter on `path`, insert it into each node’s metadata while you index (see [Transformers and metadata](/indexing/transformers-and-metadata/)), as ask-your-codebase does. Without it, PgVector refuses to store the node with “Missing metadata field”. ```rust // `path` must be a metadata field on the table: `PgVector::builder().with_metadata("path")`. // Swiftide does not add it for you; insert `path` into each node's metadata while indexing. let search = SimilaritySingleEmbedding::from_filter("path = README.md".to_string()).with_top_k(5); let pipeline = query::Pipeline::from_search_strategy(search) .then_transform_query(query_transformers::Embed::from_client(openai.clone())) .then_retrieve(pgvector) .then_answer(answers::Simple::from_client(openai)); ``` LanceDB passes the string to its `only_if` method, which takes a SQL predicate on the table’s columns, such as `path = 'README.md'`. That column has a value only if you put `path` in the node metadata and configured it with `with_metadata("path")`. DuckDB has no filter. ## Custom strategies `CustomStrategy` hands the query to your closure, and the closure builds the query the storage runs. `from_query` takes a synchronous closure, `from_async_query` an async one. PgVector runs a `sqlx::QueryBuilder`, LanceDB a LanceDB query, and DuckDB a SQL string. ```rust use sqlx::{Postgres, QueryBuilder}; // The default table is `swiftide_pgv_store`; `meta_path` exists only with `.with_metadata("path")` let table = pgvector.get_table_name().to_owned(); let keyword_search = CustomStrategy::from_query(move |query: &Query| { // The retriever reads `id`, `chunk` and every `meta_` column from each row let mut sql = QueryBuilder::::new(format!("SELECT id, chunk, meta_path FROM {table}")); sql.push(" WHERE to_tsvector('english', chunk) @@ plainto_tsquery('english', ") .push_bind(query.current().to_string()) .push(") LIMIT 10"); Ok(sql) }); let pipeline = query::Pipeline::from_search_strategy(keyword_search) .then_retrieve(pgvector) .then_answer(answers::Simple::from_client(openai)); ``` PgVector maps each row by column name. Select `id`, `chunk` and the `meta_` columns you need. `get_table_name()` returns the table, `swiftide_pgv_store` unless you set another. Swiftide does not re-export sqlx, so add it yourself: ```sh cargo add sqlx@0.9 --no-default-features --features postgres ``` A LanceDB custom strategy returns a query that implements `lancedb::query::ExecutableQuery`. Swiftide does not re-export lancedb either. Enable the `lancedb` feature flag and add the crate at the version Swiftide uses: ```sh cargo add lancedb@0.33 ``` DuckDB reads the first three columns of each row by position: the id, the chunk and the path. ## Retrievers per storage | Storage (feature flag) | Strategies | Filter | Limits | | ----------------------- | --------------------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------- | | `Qdrant` (`qdrant`) | `SimilaritySingleEmbedding`, `HybridSearch` | Qdrant `Filter` | Similarity search uses the combined vector when the collection has more than one vector or a sparse vector | | `PgVector` (`pgvector`) | `SimilaritySingleEmbedding`, `CustomStrategy>` | `String`, `key = value` | One vector field; orders by cosine distance | | `LanceDB` (`lancedb`) | `SimilaritySingleEmbedding`, `CustomStrategy` | `String`, SQL predicate | Exactly one vector field; content comes from the `chunk` column | | `Duckdb` (`duckdb`) | `SimilaritySingleEmbedding`, `HybridSearch`, `CustomStrategy` | none | Uses one vector, picked arbitrarily when several are configured; documents carry `id` and `path` metadata only | Setup for each storage is on [Qdrant](/integrations/qdrant/), [PgVector](/integrations/pgvector/) and [LanceDB](/integrations/lancedb/). DuckDB has no integration page; see the `Duckdb` reference below. ## Custom retrievers A retriever implements `Retrieve` for a search strategy `S`. It receives the strategy and the pending query, and returns `query.retrieved_documents(documents)`. `retrieved_documents` adds the documents and records a retrieval in the query’s transformation history. The trait requires `Clone`. ```rust /// Keeps the documents that share a word with the question. No store, no embeddings. #[derive(Clone)] struct KeywordRetriever { documents: Vec, } #[async_trait] impl Retrieve for KeywordRetriever { async fn retrieve( &self, search_strategy: &SimilaritySingleEmbedding, query: Query, ) -> anyhow::Result> { let words: Vec = query .current() .split_whitespace() .map(str::to_lowercase) .collect(); let matches = self .documents .iter() .filter(|document| { let content = document.content().to_lowercase(); words.iter().any(|word| content.contains(word.as_str())) }) .take(usize::try_from(search_strategy.top_k())?) .cloned() .collect(); Ok(query.retrieved_documents(matches)) } } ``` ```rust let retriever = KeywordRetriever { documents: vec![ Document::from("Swiftide indexes data with an indexing pipeline."), Document::from("A query pipeline answers questions over indexed data."), ], }; let pipeline = query::Pipeline::default() .then_retrieve(retriever) .then_answer(answers::Simple::from_client(openai)); ``` A closure works too: `then_retrieve` accepts any `Fn(&S, Query) -> Result>` that is `Send + Sync + Clone`. The closure is synchronous. ## Gotchas * Retrieval fails when the query has no embedding. Qdrant and LanceDB report “No embedding for query”, PgVector and DuckDB “Missing embedding in query state”. Qdrant hybrid search also needs a sparse embedding (“No sparse embedding for query”). * `HybridSearchBuilder::default()` sets `top_k` and `top_n` to 0. Start from `HybridSearch::default()`, which sets both to 10. * Qdrant only implements hybrid search for `HybridSearch`. Write that type in signatures, as docs-qa does, even without a filter. * Qdrant returns each document’s content `Debug`-quoted: wrapped in `"` with escaped newlines. docs-qa wraps the retriever in `CleanQdrant` ([Qdrant content workaround](/showcases/docs-qa/#qdrant-content-workaround)), which restores the text before reranking and answering. * DuckDB hybrid search does not rank before it cuts. Each search takes the first `top_n` rows of the table, unranked, and only rows in both lists get scored. On a table larger than `top_n`, most chunks are never considered. Set `top_n` to at least the row count, or use `SimilaritySingleEmbedding`. * A small `top_k` hides relevant chunks. In ask-your-codebase, one answer had to infer a detail because the chunk that held it was not among the 20 retrieved. Raise `top_k` for broad questions, or retrieve more and [rerank](/querying/reranking-and-response-transformers/). ## Reference * [`SimilaritySingleEmbedding`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.SimilaritySingleEmbedding.html) * [`HybridSearch`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.HybridSearch.html), [`HybridSearchBuilder`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.HybridSearchBuilder.html) * [`CustomStrategy`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.CustomStrategy.html) * [`Retrieve`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Retrieve.html) * [`SearchStrategy`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SearchStrategy.html) * [`SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html), [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html) * [`query::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html) * [`Qdrant`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.Qdrant.html), [`PgVector`](https://docs.rs/swiftide/0.34/swiftide/integrations/pgvector/struct.PgVector.html), [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html), [`Duckdb`](https://docs.rs/swiftide/0.34/swiftide/integrations/duckdb/struct.Duckdb.html) * Showcases: [ask-your-codebase](/showcases/ask-your-codebase/), [docs-qa](/showcases/docs-qa/) ## Next [Reranking and response transformers](/querying/reranking-and-response-transformers/) # Transforming queries > Rewrite a question into subquestions with GenerateSubquestions, embed it densely and sparsely with the model your chunks were embedded with, or write your own TransformQuery. A query pipeline can rewrite the user’s question before the retriever runs: a large language model (LLM) widens it into related questions, and an embedding model turns the result into the vectors your chunks were stored with. The query transformers need no feature flag. The snippets use the OpenAI client and Qdrant, which sit behind the `openai` and `qdrant` feature flags: ```sh cargo add swiftide@0.34 --features openai,qdrant cargo add async-trait anyhow ``` The second line is for the custom query transformer further down, which uses the `#[async_trait]` attribute and `anyhow::Result`. ## Subquestions ```rust let pipeline = query::Pipeline::default() .then_transform_query( GenerateSubquestions::builder() .client(openai.clone()) .num_questions(3) .build()?, ) // Embed the text that the step above produced, with the model the chunks were embedded with .then_transform_query(Embed::from_client(embedder)) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)); ``` A query transformer runs before retrieval. It takes the pending query and returns a new pending query. You add one with `then_transform_query`, as often as you need, and they run in the order you add them. `GenerateSubquestions` sends the current query to an LLM through `SimplePrompt`. It asks for `num_questions` related questions, 5 by default. The reply replaces the current query, so the next step sees one text: the original question and its subquestions as a list. `GenerateSubquestions::from_client` builds it with the default prompt and count. The default prompt is written for code. It asks the model to think about the dependencies, usage and structure of the code. Replace it when you search other content. ### Subquestion prompts The ask-your-codebase showcase sets its own prompt on the builder: examples/ask-your-codebase/src/query.rs ```rust let search = SimilaritySingleEmbedding::default().with_top_k(top_k); let pipeline = query::Pipeline::from_search_strategy(search) // Rephrase the question into several, to widen what the search can match .then_transform_query( GenerateSubquestions::builder() .client(llm.clone()) .prompt_template(SUBQUESTIONS_PROMPT.into()) .build()?, ) // Embed with the same local model that embedded the chunks .then_transform_query(Embed::from_client(embedding::local_model()?)) .then_retrieve(codebase.vector_store()?) // Answer the original question from the retrieved code, each chunk under its file path .then_answer( answers::Simple::builder() .client(llm) .prompt_template(ANSWER_PROMPT.into()) .document_template(CHUNK_TEMPLATE) .build()?, ); let answered = pipeline.query(question).await?; ``` `SUBQUESTIONS_PROMPT` is a constant next to the pipeline: examples/ask-your-codebase/src/query.rs ````rust /// Widens the question into the follow-up questions its answer involves. /// /// They are phrased like the questions generated for each chunk while indexing, so the two /// embed close together. const SUBQUESTIONS_PROMPT: &str = r" Your job is to help a search find the code that answers a developer's question about a codebase. Question: {{ question }} Write {{ num_questions }} short follow-up questions that the code answering this question would also answer. Think about the mechanisms involved: the main loop, the types and enum variants, the conditions that are checked, the functions that are called, and the configuration that changes the outcome. ## Constraints * Phrase them the way a developer would ask about this codebase, e.g. 'When does the agent stop?'. * Do not guess names of types or functions. * Respond with the original question and the follow-up questions only, one per line, prefixed with '- '. "; /// Renders a retrieved chunk under its file path, so the answer can cite it. /// /// `metadata.path` is the `PATH_FIELD` metadata; a test keeps the two in step. const CHUNK_TEMPLATE: &str = r"### {{ metadata.path }} ``` {{ content }} ``` "; /// Answers from the retrieved code and cites files. const ANSWER_PROMPT: &str = r" Answer a developer's question about a codebase, using snippets of its source code. Question: {{ question }} ## Constraints * Only use the snippets below. If they do not answer the question, say briefly what is missing. * Be concrete: name the types, functions and variants involved and explain when they apply. * Describe behaviour: what calls what, in which order, and under which conditions. * Cite the file each statement comes from in backticks, e.g. (`src/lib.rs`). * Use concise markdown. Add a short code example only if it shows how to use the API. ## Snippets {{ documents }} "; ```` The field takes a `Prompt`, so convert a `&'static str` or `String` with `.into()` or `Prompt::from`. A borrowed `&str` built at runtime needs `.to_string()` first. The template receives two values: `question`, the current query, and `num_questions`. The prompt phrases the subquestions like the questions that its indexing pipeline wrote for each chunk. Similar phrasing puts the query vector close to the chunk vectors. Keep the instruction to repeat the original question: the next step embeds only what the model returns. [Prompts and templates](/models/prompts-and-templates/) covers the template syntax. ## Dense and sparse embeddings examples/docs-qa/src/query.rs ```rust pipeline // Hybrid search needs both a dense and a sparse embedding of the question .then_transform_query(query_transformers::Embed::from_client( self.embedders.dense.clone(), )) .then_transform_query(query_transformers::SparseEmbed::from_client( self.embedders.sparse.clone(), )) .then_retrieve(self.qdrant.clone()) // A local cross-encoder reorders the fused candidates; only the best few remain .then_transform_response(self.reranker.clone()) .then_answer(self.answerer.clone()) ``` `query_transformers::Embed` embeds the current query with an `EmbeddingModel` and stores the vector in `query.embedding`. `query_transformers::SparseEmbed` does the same with a `SparseEmbeddingModel` and fills `query.sparse_embedding`. At this release only `FastEmbed`, behind the `fastembed` feature flag, implements `SparseEmbeddingModel`. Use the models that your [indexing pipeline embedded the chunks with](/indexing/embedding/). Vectors from different models do not compare. The docs-qa showcase creates its dense and sparse `FastEmbed` models together, and loads the same two models for its indexing and its query pipeline. Similarity search needs `embedding`. Hybrid search on Qdrant needs both, and fails with `No sparse embedding for query` when the sparse one is missing. [Retrieval and search strategies](/querying/retrieval-and-search-strategies/) covers both strategies. An embedding step reads the current query at the moment it runs. Put it after every step that rewrites the text. If `Embed` runs before `GenerateSubquestions`, the vector search never sees the subquestions. ## Custom query transformers ```rust /// Puts the instruction that BGE embedding models expect for search queries in front of the /// question. #[derive(Clone)] pub struct BgeQueryInstruction; #[async_trait] impl TransformQuery for BgeQueryInstruction { async fn transform_query( &self, mut query: Query, ) -> anyhow::Result> { let instructed = format!( "Represent this sentence for searching relevant passages: {}", query.current() ); query.transformed_query(instructed); Ok(query) } } ``` `TransformQuery` has one async method, `transform_query`. It takes the `Query` by value and returns it. Implement it with the `#[async_trait]` attribute from the `async-trait` crate. The trait requires `Send`, `Sync` and `DynClone`, so derive `Clone`. The current query has no setter of its own. `transformed_query` replaces it and records a `Transformed` event with the text before and after. The original question never changes. The fields `embedding`, `sparse_embedding` and `documents` are public, so a transformer that computes a vector sets it directly, as `Embed` does. This example prepares a question for an embedding model of the BGE (BAAI General Embedding) family, which suggests an instruction in front of short search queries. The chunks are embedded without it. ```rust let pipeline = query::Pipeline::default() // A closure is enough for a step without state .then_transform_query(|mut query: Query| { let trimmed = query.current().trim().to_string(); query.transformed_query(trimmed); Ok(query) }) .then_transform_query(BgeQueryInstruction) .then_transform_query(Embed::from_client(embedder)) .then_retrieve(qdrant) .then_answer(answers::Simple::from_client(openai)); ``` A closure is a query transformer too. It must be `Fn(Query) -> Result>` and `Clone`, `Send` and `Sync`. Closures are synchronous, so use a struct for a step that awaits. The pipeline logs `Transformed query` at debug level after each step, with the transformer’s `name()` and the new current query. `name()` defaults to the last path segment of the type name, so a closure logs a generic closure name instead of a useful one. Override `name()` on a struct to get a clearer log line. ## Transformed query after retrieval ```rust let answered = pipeline.query("How does an agent decide to stop?").await?; for event in answered.history() { if let Some(documents) = event.documents() { println!( "retrieved {} documents for:\n{}", documents.len(), event.before() ); } else { println!("rewrote:\n{}\ninto:\n{}", event.before(), event.after()); } } ``` Since 0.33 the current query keeps the transformed text after retrieval. The `Retrieved` event records the current query at the moment of retrieval, as both `before()` and `after()`, and the documents it returned. Before 0.33, retrieval cleared the current query. That text is not always what the search used. A vector retriever such as Qdrant searches with `query.embedding`, which `Embed` computed when it ran. A step after `Embed` that rewrites the text changes the event, not the vector. `history()` lists the events in order. Each `transformed_query` call adds a `Transformed` event, and each retrieval adds a `Retrieved` event. `Embed` and `SparseEmbed` add nothing, because they leave the text alone. Once the query is answered, `current()` returns the answer. Read the transformed query from the history instead. [Generating answers](/querying/answering/) shows how the answer generator uses the original and the transformed query. ## Gotchas * Two types are called `Embed`. `query_transformers::Embed` belongs in a query pipeline, `indexing::transformers::Embed` in an indexing pipeline. The same goes for `SparseEmbed`. Import them by module path. * After `GenerateSubquestions`, `Embed` turns the whole list into one vector. The pipeline does not run one search per subquestion. * DuckDB hybrid search, behind the `duckdb` feature flag, uses the subquestion list as its keyword text; see [Retrieval and search strategies](/querying/retrieval-and-search-strategies/). * The FastEmbed [`Rerank` response transformer](/querying/reranking-and-response-transformers/) scores documents against `original()`, and the [`Ragas` evaluator](/querying/evaluation/) matches rows by `original()`. Rewriting the question changes neither. * `Query::new` sets only the original question and leaves the current query empty, so an embedding step would embed an empty string. Pass a `&str` or `String` to `query`, or build the query with `Query::from`, which sets both. ## Reference * [`Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html) (`then_transform_query`) * [`TransformQuery`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.TransformQuery.html) * [`GenerateSubquestions`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.GenerateSubquestions.html) * [`query_transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.Embed.html) * [`query_transformers::SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.SparseEmbed.html) * [`Query`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Query.html) * [`TransformationEvent`](https://docs.rs/swiftide/0.34/swiftide/query/enum.TransformationEvent.html) * [`states::Pending`](https://docs.rs/swiftide/0.34/swiftide/query/states/struct.Pending.html) * [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html) * [`SparseEmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SparseEmbeddingModel.html) * [`SimplePrompt`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SimplePrompt.html) * [`Prompt`](https://docs.rs/swiftide/0.34/swiftide/prompt/struct.Prompt.html) * [`answers::Simple`](https://docs.rs/swiftide/0.34/swiftide/query/answers/struct.Simple.html) * Showcases: [ask-your-codebase](/showcases/ask-your-codebase/) and [docs-qa](/showcases/docs-qa/) ## Next [Retrieval and search strategies](/querying/retrieval-and-search-strategies/)