Transformers and metadata

You add metadata to nodes with closures, with the built-in transformers that ask a large language model (LLM), or with a transformer you write yourself. Metadata travels with each node to the embedder, so a retriever can match on it and an answer can cite it.

Closures

The ask-your-codebase showcase stores each file’s path, relative to the codebase root, on every node.

/// Stores the file path, relative to the codebase root, as metadata on the node.
///
/// The vector store keeps this field next to each chunk, so answers can cite their sources.
fn record_relative_path(
    codebase: Codebase,
) -> impl Fn(TextNode) -> Result<TextNode> + Clone + Send + Sync + 'static {
    move |mut node| {
        let path = codebase
            .relative_path(&node.path)
            .to_string_lossy()
            .into_owned();
        node.metadata.insert(PATH_FIELD, path);
        Ok(node)
    }
}

Source: examples/ask-your-codebase/src/index.rs#L127-L141

A closure from TextNode to Result<TextNode> is a transformer if it is Clone, Send, Sync and 'static, so it cannot borrow local data. You add it with .then(...). The pipeline runs it on many nodes at once and does not keep their order.

node.metadata is a Metadata map. insert takes a key that converts into a String and a value that converts into a serde_json::Value.

LanceDB and PgVector store a field only after you declare it with with_metadata, which is why the showcase calls .with_metadata(PATH_FIELD).

Built-in metadata transformers

Swiftide ships six transformers that prompt the LLM per node and store the reply as metadata:

Transformer	Metadata field	Feature flag
`MetadataQAText`	`Questions and Answers (text)`	none
`MetadataQACode`	`Questions and Answers (code)`	`tree-sitter`
`MetadataSummary`	`Summary`	none
`MetadataTitle`	`Title`	none
`MetadataKeywords`	`Keywords`	none
`CompressCodeOutline`	`Outline`	`tree-sitter`

Each module exports the field name as a NAME constant, such as metadata_title::NAME. CompressCodeOutline skips nodes without an Outline field; Code-aware indexing shows how to fill it. The next example needs the openai and qdrant feature flags.

indexing::Pipeline::from_loader(FileLoader::new("README.md").with_extensions(&["md"]))
    .with_concurrency(1)
    .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
    .then(MetadataQAText::new(openai_client.clone()))
    .then(MetadataSummary::new(openai_client.clone()))
    .then(MetadataTitle::new(openai_client.clone()))
    .then(MetadataKeywords::new(openai_client.clone()))
    .then_in_batch(Embed::new(openai_client.clone()))
    .log_all()
    .filter_errors()
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples")
            .build()?,
    )
    .run()
    .await?;

Source: examples/index_markdown_lots_of_metadata.rs#L39-L57

new(client) uses the default prompt. from_client(client) returns the builder, where you set prompt_template, concurrency and, for the question transformers, num_questions (default 5). builder() and Default start without a client.

After the chunker, each chunk gets its own prompt and reply. Before it, one prompt per file is enough, and chunking copies the reply to every chunk of that file.

concurrency(n) on the builder, or with_concurrency(n), limits the requests that step sends at once. Concurrency explains how this interacts with the pipeline value.

By default the embedder sees every metadata field as a key: value line in front of the chunk. Embed mode explains how to embed fields separately.

Prompts

The default prompts show the expected answer inside a code fence, and models tend to copy that fence into the metadata. The docs-qa showcase replaces the MetadataTitle and MetadataKeywords prompts:

// Swiftide's default metadata prompts show the expected format inside a code fence, and models
// tend to copy the fence into the metadata. These ask for plain text and use the page title.
// A code-only chunk is titled by the API it shows, because the reranker scores the title too.
const TITLE_PROMPT: &str = "Write a short, descriptive title (at most 10 words) for this \
excerpt from the \"{{ node.metadata.Page }}\" documentation. If the excerpt is mostly code, \
name the API it demonstrates (a macro, trait or builder method), not what the example does. \
Reply with the title only: no quotes, no markdown, no code fences.

{{ node.chunk }}";

const KEYWORDS_PROMPT: &str = "List up to 8 keywords for this excerpt from the \
\"{{ node.metadata.Page }}\" documentation: names of types, crates, features, tools, commands \
and concepts that literally appear in it. Reply with one comma-separated line only: no \
numbering, no markdown, no code fences.

{{ node.chunk }}";

Source: examples/docs-qa/src/index.rs#L24-L39

The pipeline passes them to the builders with prompt_template:

indexing::Pipeline::from_loader(loader)
    .then_chunk(ChunkMarkdown::from_chunk_range(
        MIN_CHUNK_CHARS..MAX_CHUNK_CHARS,
    ))
    // A short title and keywords per chunk; both end up in the embedded text and help the
    // sparse (keyword) side of hybrid search in particular.
    .then(
        MetadataTitle::from_client(llm.clone())
            .prompt_template(TITLE_PROMPT)
            .build()?,
    )
    .then(
        MetadataKeywords::from_client(llm.clone())
            .prompt_template(KEYWORDS_PROMPT)
            .build()?,
    )
    .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_store_with(qdrant.clone())
    .run()
    .await

Source: examples/docs-qa/src/index.rs#L50-L70

prompt_template takes anything that converts into a Prompt, such as a &'static str holding a Tera template. Each transformer adds the node as node, so {{ node.chunk }} is the chunk and {{ node.metadata.Page }} is a field that the docs-qa loader set. The question transformers also add {{ questions }}, and MetadataQACode adds {{ outline }} when the node has an Outline field.

In the captured docs-qa run, four pages became 43 chunks, and the title and keywords steps made 86 calls to claude-haiku-5-5 through an OpenAI-compatible gateway to Amazon Web Services (AWS) Bedrock. The whole run, including local embedding and the writes to Qdrant, took:

Indexed 43 chunks into `docs-qa` in 71.7s

The ask-your-codebase prompt asks for questions a developer would ask:

/// Asks for the questions a developer would ask that a chunk answers.
///
/// The default `MetadataQACode` prompt asks generic questions ("What does this code do?").
/// Questions about behaviour, phrased the way a developer would ask them, match real questions
/// much better.
const QUESTIONS_PROMPT: &str = r"
# Task

List the questions a developer could ask about this codebase that the code below answers, each
with a short answer. They are stored with the code to help a search find it.

Focus on behaviour: when and in which order things happen, what starts or ends a process, which
conditions and configuration change the outcome, what happens on errors, and which extension
points (traits, hooks, builders, options) exist and when they apply.

# Constraints

- Write {{ questions }} questions and answers.
- Phrase questions the way a developer would ask them, e.g. 'When does the agent stop?'.
- Name the relevant types, functions and variants in the answers.
- Only use what can be derived from the code.
- Respond only in this format:

Q1: <question>
A1: <answer>

# Code

```
{{ node.chunk }}
```
";

Source: examples/ask-your-codebase/src/index.rs#L39-L70

See Prompts and templates for Tera syntax.

Default LLM client

You can set one client on the pipeline instead of on each transformer. This example needs the openai feature flag.

Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    // Set the client before the steps that use it
    .with_default_llm_client(openai)
    .then_chunk(ChunkMarkdown::from_chunk_range(100..1500))
    .then(MetadataTitle::default())
    .then(MetadataSummary::default())
    .then(
        MetadataQAText::builder()
            .num_questions(3_usize)
            .concurrency(4_usize)
            .build()?,
    )
    .then_store_with(MemoryStorage::default())
    .run()
    .await

Source: snippets/src/indexing/transformers.rs#L21-L35

then hands the pipeline defaults to the transformer when you add the step, so call with_default_llm_client first. A client on the transformer takes precedence. Without either, each node fails with No client provided.

Custom transformers

A transformer is a type that implements Transformer.

/// Stores the number of words in a chunk as the `Words` metadata field.
#[derive(Clone)]
pub struct WordCount;

#[async_trait]
impl Transformer for WordCount {
    type Input = String;
    type Output = String;

    async fn transform_node(&self, mut node: TextNode) -> anyhow::Result<TextNode> {
        let words = node.chunk.split_whitespace().count();
        node.metadata.insert("Words", words);
        Ok(node)
    }
}

// `then` hands pipeline defaults to every transformer; this one has no use for them.
impl WithIndexingDefaults for WordCount {}

Source: snippets/src/indexing/transformers.rs#L40-L57

transform_node receives one node and returns it, changed, or an error. The type must be Clone, Send, Sync and 'static. The trait is an async_trait, so add the async-trait crate. Override concurrency to give the step its own limit.

then also requires WithIndexingDefaults; an empty impl is enough.

Transformer macro

The built-in LLM transformers use #[indexing_transformer]. This one asks who a chunk is for:

/// The macro adds items next to the struct, so each annotated struct gets its own module.
pub mod metadata_audience {
    use async_trait::async_trait;
    use swiftide::{indexing::TextNode, indexing_transformer, traits::Transformer};

    /// Asks the LLM who a chunk is written for and stores the reply as `Audience`.
    #[indexing_transformer(
        metadata_field_name = "Audience",
        default_prompt_file = "prompts/metadata_audience.prompt.md"
    )]
    pub struct MetadataAudience {
        /// Product name passed to the prompt as `{{ product }}`.
        #[builder(default = "\"swiftide\".to_string()")]
        product: String,
    }

    #[async_trait]
    impl Transformer for MetadataAudience {
        type Input = String;
        type Output = String;

        async fn transform_node(&self, mut node: TextNode) -> anyhow::Result<TextNode> {
            // The macro renders nothing for you: add the node and your own values here.
            let prompt = self
                .prompt_template
                .clone()
                .with_node(&node)
                .with_context_value("product", self.product.as_str());

            // `prompt` uses this transformer's client, or the pipeline default.
            let audience = self.prompt(prompt).await?;
            node.metadata.insert(NAME, audience);
            Ok(node)
        }

        // The macro stores `with_concurrency(n)` but does not apply it.
        fn concurrency(&self) -> Option<usize> {
            self.concurrency
        }
    }
}

Source: snippets/src/indexing/transformers.rs#L103-L143

metadata_field_name becomes the NAME constant. default_prompt_file goes to include_str!, relative to the source file. The macro adds client, prompt_template and concurrency fields, and generates builder(), new, from_client, with_concurrency, Default and the WithIndexingDefaults impl. derive(skip_debug, skip_clone, skip_default) turns off generated derives.

You still implement Transformer. Build the prompt with with_node(&node) before you pass it to the generated self.prompt(...), which uses the transformer’s client or the pipeline default. Override concurrency() as the example does, or with_concurrency has no effect.

The macro adds a hidden module, NAME and default_prompt() next to the struct, so give each annotated struct its own module. new and Default unwrap the builder, so give each of your own fields #[builder(default = ...)].

Your crate needs swiftide-core, anyhow and derive_builder as direct dependencies; Crates has the cargo add line.

Batch transformers

A batch transformer takes a Vec of nodes and returns a stream of any length. This one drops repeated chunks:

/// Drops chunks that repeat an earlier chunk in the same batch, such as copied licence headers.
#[derive(Clone)]
pub struct DropRepeatedChunks {
    batch_size: usize,
}

#[async_trait]
impl BatchableTransformer for DropRepeatedChunks {
    type Input = String;
    type Output = String;

    async fn batch_transform(&self, nodes: Vec<TextNode>) -> IndexingStream<String> {
        let mut seen = HashSet::new();
        let unique: Vec<TextNode> = nodes
            .into_iter()
            .filter(|node| seen.insert(node.chunk.trim().to_owned()))
            .collect();
        unique.into()
    }

    fn batch_size(&self) -> Option<usize> {
        Some(self.batch_size)
    }
}

impl WithBatchIndexingDefaults for DropRepeatedChunks {}

Source: snippets/src/indexing/transformers.rs#L61-L86

You add it with then_in_batch:

Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..1500))
    .then_in_batch(DropRepeatedChunks { batch_size: 64 })
    .then(WordCount)
    .then_store_with(MemoryStorage::default())
    .run()
    .await

Source: snippets/src/indexing/transformers.rs#L92-L98

The pipeline groups nodes into batches of batch_size(), or 256 when it returns None. into() turns a Vec<TextNode> into an IndexingStream. then_in_batch requires WithBatchIndexingDefaults, again with an empty impl.

A closure from Vec<TextNode> to IndexingStream<String> is a batch transformer too, with the pipeline batch size. Embed and SparseEmbed are batch transformers.

Batches are formed in arrival order, so the example only catches repeats that land in the same batch.

Node types

Transformer and BatchableTransformer declare Input and Output types, both bound by Chunk. then returns a pipeline of the transformer’s Output, so a different Output changes the node type for every step after it. Closures and most built-in transformers, chunkers and storage backends take String nodes, so add a transformer that changes the node type after every step that needs String.

Reference

Embedding