# Transformers and metadata

> Add metadata to nodes with closures, the built-in LLM metadata transformers and their prompts, and your own transformers and batch transformers.

You add metadata to nodes with closures, with the built-in transformers that ask a large language model (LLM), or with a transformer you write yourself. Metadata travels with each node to the embedder, so a retriever can match on it and an answer can cite it.

## Closures

The [ask-your-codebase](/showcases/ask-your-codebase/) showcase stores each file's path, relative to the codebase root, on every node.

```rust title="examples/ask-your-codebase/src/index.rs"
/// Stores the file path, relative to the codebase root, as metadata on the node.
///
/// The vector store keeps this field next to each chunk, so answers can cite their sources.
fn record_relative_path(
    codebase: Codebase,
) -> impl Fn(TextNode) -> Result<TextNode> + Clone + Send + Sync + 'static {
    move |mut node| {
        let path = codebase
            .relative_path(&node.path)
            .to_string_lossy()
            .into_owned();
        node.metadata.insert(PATH_FIELD, path);
        Ok(node)
    }
}
```

Source: [`examples/ask-your-codebase/src/index.rs#L127-L141`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/index.rs#L127-L141)

A closure from `TextNode` to `Result<TextNode>` is a transformer if it is `Clone`, `Send`, `Sync` and `'static`, so it cannot borrow local data. You add it with `.then(...)`. The pipeline runs it on many nodes at once and does not keep their order.

`node.metadata` is a `Metadata` map. `insert` takes a key that converts into a `String` and a value that converts into a `serde_json::Value`.

LanceDB and PgVector store a field only after you declare it with `with_metadata`, which is why the showcase calls `.with_metadata(PATH_FIELD)`.

## Built-in metadata transformers

Swiftide ships six transformers that prompt the LLM per node and store the reply as metadata:

| Transformer           | Metadata field                 | Feature flag  |
| --------------------- | ------------------------------ | ------------- |
| `MetadataQAText`      | `Questions and Answers (text)` | none          |
| `MetadataQACode`      | `Questions and Answers (code)` | `tree-sitter` |
| `MetadataSummary`     | `Summary`                      | none          |
| `MetadataTitle`       | `Title`                        | none          |
| `MetadataKeywords`    | `Keywords`                     | none          |
| `CompressCodeOutline` | `Outline`                      | `tree-sitter` |

Each module exports the field name as a `NAME` constant, such as `metadata_title::NAME`. `CompressCodeOutline` skips nodes without an `Outline` field; [Code-aware indexing](/indexing/code-aware-indexing/) shows how to fill it. The next example needs the `openai` and `qdrant` feature flags.

```rust title="examples/index_markdown_lots_of_metadata.rs"
indexing::Pipeline::from_loader(FileLoader::new("README.md").with_extensions(&["md"]))
    .with_concurrency(1)
    .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
    .then(MetadataQAText::new(openai_client.clone()))
    .then(MetadataSummary::new(openai_client.clone()))
    .then(MetadataTitle::new(openai_client.clone()))
    .then(MetadataKeywords::new(openai_client.clone()))
    .then_in_batch(Embed::new(openai_client.clone()))
    .log_all()
    .filter_errors()
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples")
            .build()?,
    )
    .run()
    .await?;
```

Source: [`examples/index_markdown_lots_of_metadata.rs#L39-L57`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/index_markdown_lots_of_metadata.rs#L39-L57)

`new(client)` uses the default prompt. `from_client(client)` returns the builder, where you set `prompt_template`, `concurrency` and, for the question transformers, `num_questions` (default 5). `builder()` and `Default` start without a client.

After the chunker, each chunk gets its own prompt and reply. Before it, one prompt per file is enough, and chunking copies the reply to every chunk of that file.

`concurrency(n)` on the builder, or `with_concurrency(n)`, limits the requests that step sends at once. [Concurrency](/indexing/streams-and-concurrency/#concurrency) explains how this interacts with the pipeline value.

By default the embedder sees every metadata field as a `key: value` line in front of the chunk. [Embed mode](/indexing/embedding/#embed-mode) explains how to embed fields separately.

## Prompts

The default prompts show the expected answer inside a code fence, and models tend to copy that fence into the metadata. The [docs-qa](/showcases/docs-qa/) showcase replaces the `MetadataTitle` and `MetadataKeywords` prompts:

```rust title="examples/docs-qa/src/index.rs"
// Swiftide's default metadata prompts show the expected format inside a code fence, and models
// tend to copy the fence into the metadata. These ask for plain text and use the page title.
// A code-only chunk is titled by the API it shows, because the reranker scores the title too.
const TITLE_PROMPT: &str = "Write a short, descriptive title (at most 10 words) for this \
excerpt from the \"{{ node.metadata.Page }}\" documentation. If the excerpt is mostly code, \
name the API it demonstrates (a macro, trait or builder method), not what the example does. \
Reply with the title only: no quotes, no markdown, no code fences.

{{ node.chunk }}";

const KEYWORDS_PROMPT: &str = "List up to 8 keywords for this excerpt from the \
\"{{ node.metadata.Page }}\" documentation: names of types, crates, features, tools, commands \
and concepts that literally appear in it. Reply with one comma-separated line only: no \
numbering, no markdown, no code fences.

{{ node.chunk }}";
```

Source: [`examples/docs-qa/src/index.rs#L24-L39`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa/src/index.rs#L24-L39)

The pipeline passes them to the builders with `prompt_template`:

```rust title="examples/docs-qa/src/index.rs"
indexing::Pipeline::from_loader(loader)
    .then_chunk(ChunkMarkdown::from_chunk_range(
        MIN_CHUNK_CHARS..MAX_CHUNK_CHARS,
    ))
    // A short title and keywords per chunk; both end up in the embedded text and help the
    // sparse (keyword) side of hybrid search in particular.
    .then(
        MetadataTitle::from_client(llm.clone())
            .prompt_template(TITLE_PROMPT)
            .build()?,
    )
    .then(
        MetadataKeywords::from_client(llm.clone())
            .prompt_template(KEYWORDS_PROMPT)
            .build()?,
    )
    .then_in_batch(Embed::new(embedders.dense.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_in_batch(SparseEmbed::new(embedders.sparse.clone()).with_batch_size(EMBED_BATCH_SIZE))
    .then_store_with(qdrant.clone())
    .run()
    .await
```

Source: [`examples/docs-qa/src/index.rs#L50-L70`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/docs-qa/src/index.rs#L50-L70)

`prompt_template` takes anything that converts into a `Prompt`, such as a `&'static str` holding a Tera template. Each transformer adds the node as `node`, so `{{ node.chunk }}` is the chunk and `{{ node.metadata.Page }}` is a field that the docs-qa loader set. The question transformers also add `{{ questions }}`, and `MetadataQACode` adds `{{ outline }}` when the node has an `Outline` field.

In the captured docs-qa run, four pages became 43 chunks, and the title and keywords steps made 86 calls to `claude-haiku-5-5` through an OpenAI-compatible gateway to Amazon Web Services (AWS) Bedrock. The whole run, including local embedding and the writes to Qdrant, took:

```text
Indexed 43 chunks into `docs-qa` in 71.7s
```

The ask-your-codebase prompt asks for questions a developer would ask:

````rust title="examples/ask-your-codebase/src/index.rs"
/// Asks for the questions a developer would ask that a chunk answers.
///
/// The default `MetadataQACode` prompt asks generic questions ("What does this code do?").
/// Questions about behaviour, phrased the way a developer would ask them, match real questions
/// much better.
const QUESTIONS_PROMPT: &str = r"
# Task

List the questions a developer could ask about this codebase that the code below answers, each
with a short answer. They are stored with the code to help a search find it.

Focus on behaviour: when and in which order things happen, what starts or ends a process, which
conditions and configuration change the outcome, what happens on errors, and which extension
points (traits, hooks, builders, options) exist and when they apply.

# Constraints

- Write {{ questions }} questions and answers.
- Phrase questions the way a developer would ask them, e.g. 'When does the agent stop?'.
- Name the relevant types, functions and variants in the answers.
- Only use what can be derived from the code.
- Respond only in this format:

Q1: <question>
A1: <answer>

# Code

```
{{ node.chunk }}
```
";
````

Source: [`examples/ask-your-codebase/src/index.rs#L39-L70`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/index.rs#L39-L70)

See [Prompts and templates](/models/prompts-and-templates/) for Tera syntax.

## Default LLM client

You can set one client on the pipeline instead of on each transformer. This example needs the `openai` feature flag.

```rust
Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    // Set the client before the steps that use it
    .with_default_llm_client(openai)
    .then_chunk(ChunkMarkdown::from_chunk_range(100..1500))
    .then(MetadataTitle::default())
    .then(MetadataSummary::default())
    .then(
        MetadataQAText::builder()
            .num_questions(3_usize)
            .concurrency(4_usize)
            .build()?,
    )
    .then_store_with(MemoryStorage::default())
    .run()
    .await
```

Source: [`snippets/src/indexing/transformers.rs#L21-L35`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/indexing/transformers.rs#L21-L35)

`then` hands the pipeline defaults to the transformer when you add the step, so call `with_default_llm_client` first. A client on the transformer takes precedence. Without either, each node fails with `No client provided`.

## Custom transformers

A transformer is a type that implements `Transformer`.

```rust
/// Stores the number of words in a chunk as the `Words` metadata field.
#[derive(Clone)]
pub struct WordCount;

#[async_trait]
impl Transformer for WordCount {
    type Input = String;
    type Output = String;

    async fn transform_node(&self, mut node: TextNode) -> anyhow::Result<TextNode> {
        let words = node.chunk.split_whitespace().count();
        node.metadata.insert("Words", words);
        Ok(node)
    }
}

// `then` hands pipeline defaults to every transformer; this one has no use for them.
impl WithIndexingDefaults for WordCount {}
```

Source: [`snippets/src/indexing/transformers.rs#L40-L57`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/indexing/transformers.rs#L40-L57)

`transform_node` receives one node and returns it, changed, or an error. The type must be `Clone`, `Send`, `Sync` and `'static`. The trait is an `async_trait`, so add the `async-trait` crate. Override `concurrency` to give the step its own limit.

`then` also requires `WithIndexingDefaults`; an empty `impl` is enough.

### Transformer macro

The built-in LLM transformers use `#[indexing_transformer]`. This one asks who a chunk is for:

```rust
/// The macro adds items next to the struct, so each annotated struct gets its own module.
pub mod metadata_audience {
    use async_trait::async_trait;
    use swiftide::{indexing::TextNode, indexing_transformer, traits::Transformer};

    /// Asks the LLM who a chunk is written for and stores the reply as `Audience`.
    #[indexing_transformer(
        metadata_field_name = "Audience",
        default_prompt_file = "prompts/metadata_audience.prompt.md"
    )]
    pub struct MetadataAudience {
        /// Product name passed to the prompt as `{{ product }}`.
        #[builder(default = "\"swiftide\".to_string()")]
        product: String,
    }

    #[async_trait]
    impl Transformer for MetadataAudience {
        type Input = String;
        type Output = String;

        async fn transform_node(&self, mut node: TextNode) -> anyhow::Result<TextNode> {
            // The macro renders nothing for you: add the node and your own values here.
            let prompt = self
                .prompt_template
                .clone()
                .with_node(&node)
                .with_context_value("product", self.product.as_str());

            // `prompt` uses this transformer's client, or the pipeline default.
            let audience = self.prompt(prompt).await?;
            node.metadata.insert(NAME, audience);
            Ok(node)
        }

        // The macro stores `with_concurrency(n)` but does not apply it.
        fn concurrency(&self) -> Option<usize> {
            self.concurrency
        }
    }
}
```

Source: [`snippets/src/indexing/transformers.rs#L103-L143`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/indexing/transformers.rs#L103-L143)

`metadata_field_name` becomes the `NAME` constant. `default_prompt_file` goes to `include_str!`, relative to the source file. The macro adds `client`, `prompt_template` and `concurrency` fields, and generates `builder()`, `new`, `from_client`, `with_concurrency`, `Default` and the `WithIndexingDefaults` impl. `derive(skip_debug, skip_clone, skip_default)` turns off generated derives.

You still implement `Transformer`. Build the prompt with `with_node(&node)` before you pass it to the generated `self.prompt(...)`, which uses the transformer's client or the pipeline default. Override `concurrency()` as the example does, or `with_concurrency` has no effect.

The macro adds a `hidden` module, `NAME` and `default_prompt()` next to the struct, so give each annotated struct its own module. `new` and `Default` unwrap the builder, so give each of your own fields `#[builder(default = ...)]`.

Your crate needs `swiftide-core`, `anyhow` and `derive_builder` as direct dependencies; [Crates](/reference/crates/) has the `cargo add` line.

## Batch transformers

A batch transformer takes a `Vec` of nodes and returns a stream of any length. This one drops repeated chunks:

```rust
/// Drops chunks that repeat an earlier chunk in the same batch, such as copied licence headers.
#[derive(Clone)]
pub struct DropRepeatedChunks {
    batch_size: usize,
}

#[async_trait]
impl BatchableTransformer for DropRepeatedChunks {
    type Input = String;
    type Output = String;

    async fn batch_transform(&self, nodes: Vec<TextNode>) -> IndexingStream<String> {
        let mut seen = HashSet::new();
        let unique: Vec<TextNode> = nodes
            .into_iter()
            .filter(|node| seen.insert(node.chunk.trim().to_owned()))
            .collect();
        unique.into()
    }

    fn batch_size(&self) -> Option<usize> {
        Some(self.batch_size)
    }
}

impl WithBatchIndexingDefaults for DropRepeatedChunks {}
```

Source: [`snippets/src/indexing/transformers.rs#L61-L86`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/indexing/transformers.rs#L61-L86)

You add it with `then_in_batch`:

```rust
Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..1500))
    .then_in_batch(DropRepeatedChunks { batch_size: 64 })
    .then(WordCount)
    .then_store_with(MemoryStorage::default())
    .run()
    .await
```

Source: [`snippets/src/indexing/transformers.rs#L92-L98`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/indexing/transformers.rs#L92-L98)

The pipeline groups nodes into batches of `batch_size()`, or 256 when it returns `None`. `into()` turns a `Vec<TextNode>` into an `IndexingStream`. `then_in_batch` requires `WithBatchIndexingDefaults`, again with an empty `impl`.

A closure from `Vec<TextNode>` to `IndexingStream<String>` is a batch transformer too, with the pipeline batch size. `Embed` and `SparseEmbed` are batch transformers.

Batches are formed in arrival order, so the example only catches repeats that land in the same batch.

## Node types

`Transformer` and `BatchableTransformer` declare `Input` and `Output` types, both bound by `Chunk`. `then` returns a pipeline of the transformer's `Output`, so a different `Output` changes the node type for every step after it. Closures and most built-in transformers, chunkers and storage backends take `String` nodes, so add a transformer that changes the node type after every step that needs `String`.

## Reference

- [`Pipeline::then`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then)
- [`Pipeline::then_in_batch`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.then_in_batch)
- [`Pipeline::with_default_llm_client`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.with_default_llm_client)
- [`Pipeline::with_concurrency`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html#method.with_concurrency)
- [`Transformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Transformer.html)
- [`BatchableTransformer`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.BatchableTransformer.html)
- [`WithIndexingDefaults`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.WithIndexingDefaults.html)
- [`WithBatchIndexingDefaults`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.WithBatchIndexingDefaults.html)
- [`Node`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Node.html) and [`TextNode`](https://docs.rs/swiftide/0.34/swiftide/indexing/type.TextNode.html)
- [`Chunk`](https://docs.rs/swiftide/0.34/swiftide/indexing/trait.Chunk.html)
- [`Metadata`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Metadata.html)
- [`IndexingStream`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.IndexingStream.html)
- [`MetadataQAText`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQAText.html)
- [`MetadataQACode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQACode.html)
- [`MetadataSummary`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataSummary.html)
- [`MetadataTitle`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataTitle.html)
- [`MetadataKeywords`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataKeywords.html)
- [`CompressCodeOutline`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.CompressCodeOutline.html)
- [`Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html) and [`SparseEmbed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.SparseEmbed.html)
- [`indexing_transformer`](https://docs.rs/swiftide/0.34/swiftide/attr.indexing_transformer.html)
- [`Prompt`](https://docs.rs/swiftide/0.34/swiftide/prompt/struct.Prompt.html)

## Next

[Embedding](/indexing/embedding/)