Rich metadata and multiple vectors

Both recipes fill a Qdrant collection in which every chunk carries four fields that a large language model (LLM) writes: questions and answers, a summary, a title and keywords. One recipe embeds each chunk together with its metadata as a single vector. The other stores the chunk and each metadata field as its own named vector.

Setup

Add Swiftide with the openai and qdrant feature flags, plus the runtime and logging crates the examples use:

cargo add swiftide@0.34 --features openai,qdrant
cargo add tokio --features macros,rt-multi-thread
cargo add tracing-subscriber --features env-filter

Set OPENAI_API_KEY and start Qdrant on localhost:6334, or point QDRANT_URL somewhere else; Running Qdrant has the Docker command. Run the examples from the root of the swiftide repository, because they load README.md and LICENSE by relative path:

cargo run -p swiftide-examples --example index-markdown-metadata
cargo run -p swiftide-examples --example store-multiple-vectors

Metadata transformers

Both recipes use one OpenAI client for prompts and embeddings:

let openai_client = integrations::openai::OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_prompt_model("gpt-4o")
    .build()?;

Source: examples/index_markdown_lots_of_metadata.rs#L32-L35

The client sets gpt-4o for the metadata prompts and text-embedding-3-small for embeddings. That embedding model returns 1536 dimensions, which must match vector_size(1536) on the storage.

indexing::Pipeline::from_loader(FileLoader::new("README.md").with_extensions(&["md"]))
    .with_concurrency(1)
    .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
    .then(MetadataQAText::new(openai_client.clone()))
    .then(MetadataSummary::new(openai_client.clone()))
    .then(MetadataTitle::new(openai_client.clone()))
    .then(MetadataKeywords::new(openai_client.clone()))
    .then_in_batch(Embed::new(openai_client.clone()))
    .log_all()
    .filter_errors()
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples")
            .build()?,
    )
    .run()
    .await?;

Source: examples/index_markdown_lots_of_metadata.rs#L39-L57

ChunkMarkdown::from_chunk_range(20..2048) splits the file into chunks of fewer than 2048 characters. It drops chunks that are under 20 bytes once whitespace is trimmed. Each metadata transformer then sends one prompt per chunk and writes the answer under a fixed metadata key:

Transformer	Metadata key
`MetadataQAText`	`Questions and Answers (text)`
`MetadataSummary`	`Summary`
`MetadataTitle`	`Title`
`MetadataKeywords`	`Keywords`

Each module exposes its key as a NAME constant, for example metadata_qa_text::NAME. The default MetadataQAText prompt asks for at most five questions, so the model may return fewer. The prompts are covered in Transformers and metadata.

Each chunk costs four prompts. with_concurrency(1) sends them one node at a time. Raise the pipeline concurrency when your OpenAI rate limit allows it.

The pipeline keeps the default embed mode. Embed turns each node into one text: the metadata as key: value lines, then the chunk. With only the Combined field configured, Qdrant stores it as the collection’s single unnamed vector. The metadata goes into the point payload next to path and content.

Error handling

log_all logs errors, and logs every node at DEBUG level. Set RUST_LOG=debug to see each chunk with its metadata. filter_errors drops errors instead of stopping the run, but an error can take healthy nodes with it.

then_in_batch(Embed) collects nodes into batches before it embeds them. When a failed prompt reaches that step, the batch returns the error together with every node it had collected so far. filter_errors drops the whole error. One failed prompt can therefore drop up to 255 good nodes in the first recipe, which uses the default batch size of 256, and up to 9 in the second. A failed embedding request also fails its whole batch. Keep the batch size small when your prompts fail often.

Named vectors

The second recipe stores five vectors per chunk instead of one:

indexing::Pipeline::from_loader(FileLoader::new("LICENSE"))
    .with_concurrency(1)
    .with_embed_mode(EmbedMode::PerField)
    .then_chunk(ChunkMarkdown::from_chunk_range(20..2048))
    .then(MetadataQAText::new(openai_client.clone()))
    .then(MetadataSummary::new(openai_client.clone()))
    .then(MetadataTitle::new(openai_client.clone()))
    .then(MetadataKeywords::new(openai_client.clone()))
    .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10))
    .log_all()
    .filter_errors()
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-multi-vectors")
            .with_vector(EmbeddedField::Chunk)
            .with_vector(EmbeddedField::Metadata(metadata_qa_text::NAME.into()))
            .with_vector(EmbeddedField::Metadata(metadata_summary::NAME.into()))
            .with_vector(
                VectorConfig::builder()
                    .embedded_field(EmbeddedField::Metadata(metadata_title::NAME.into()))
                    .distance(Distance::Manhattan)
                    .build()?,
            )
            .with_vector(EmbeddedField::Metadata(metadata_keywords::NAME.into()))
            .build()?,
    )
    .run()
    .await?;

Source: examples/store_multiple_vectors.rs#L45-L74

with_embed_mode(EmbedMode::PerField) right after the loader makes Embed send the chunk and each metadata value as separate texts, so with_batch_size(10) keeps each request at 50 inputs. Each with_vector names one stored field. Qdrant names the vectors Chunk, Metadata: Questions and Answers (text), Metadata: Summary and so on. The title vector uses Distance::Manhattan through VectorConfig::builder(). Embedding explains embed modes and named vectors in full.

Gotchas

Qdrant keeps the vector configuration of an existing collection, as Existing collections explains. That is why the two examples use different collection names. Delete the collection or pick a new name when you change the vectors.

Search on named vectors

A query pipeline with the default search strategy, SimilaritySingleEmbedding, searches the vector named Combined whenever the Qdrant storage is built with more than one with_vector or with any with_sparse_vector. The check uses the Qdrant value in the query pipeline, not the collection. The store-multiple-vectors collection has no Combined vector, so that search fails.

To keep similarity search working, index with EmbedMode::Both and add with_vector(EmbeddedField::Combined). Build the query-side Qdrant with the same with_vector calls. A Qdrant with the default single-vector config sends an unnamed search, which fails against a collection of named vectors.

HybridSearch lets you pick the field: with_dense_vector_field and with_sparse_vector_field both default to EmbeddedField::Combined. Hybrid search also needs a sparse vector for the chosen field, from SparseEmbed and with_sparse_vector. Hybrid search and reranking shows that setup. For other searches on one named vector, use the Qdrant client from Qdrant::client() directly.

Reference

Scraping websites to markdown