Storage

then_store_with writes every node to a storage backend. You end up with nodes in LanceDB or Qdrant, a clear picture of what each backend keeps of a node, and a storage of your own built on the Persist trait.

LanceDB

The ask-your-codebase showcase keeps its vectors in an embedded LanceDB table, in a directory next to the code. There is no server to run. Enable the lancedb feature flag:

cargo add swiftide@0.34 --features lancedb

/// Remembers which files were fully indexed, so unchanged files are skipped next time.
pub fn node_cache(&self) -> Result<Redb> {
    std::fs::create_dir_all(&self.data_dir)?;

    Redb::builder()
        .database_path(self.data_dir.join("cache.redb"))
        .table_name("indexed_files")
        .build()
}

/// Embedded vector store with one vector per chunk and the source path next to it.
pub fn vector_store(&self) -> Result<LanceDB> {
    let uri = self.lancedb_dir();

    LanceDB::builder()
        .uri(uri.to_str().context("Index path is not valid UTF-8")?)
        .table_name("code_chunks")
        .vector_size(embedding::DIMENSIONS)
        .with_vector(EmbeddedField::Combined)
        .with_metadata(PATH_FIELD)
        .build()
}

Source: examples/ask-your-codebase/src/codebase.rs#L56-L77

vector_store builds the storage; node_cache belongs to Caching. The showcase pipeline passes the storage to then_store_with, after the embedding step.

uri is the directory that holds the table. It is required unless you pass a connection_pool.
table_name defaults to swiftide.
vector_size must match the output of your embedder. The showcase uses 768, the size of its FastEmbed code model.
with_vector(EmbeddedField::Combined) adds a column for the embedding of the chunk together with its metadata.
with_metadata(PATH_FIELD) adds a column for the path metadata field.

LanceDB stores only what you configure: the node id, the chunk, each configured vector and each metadata field named with with_metadata. The node’s own path is not a column. The showcase stores the path relative to the codebase root as metadata, so answers can cite their sources. Metadata columns hold text, so a value that is not a JSON string is stored as null.

Before the first node arrives, setup creates the table if it does not exist. Nodes are written in batches of 256 with a merge on id, so storing the same node twice updates its row. See LanceDB for retrieval and remote storage.

Qdrant

The docs-qa showcase stores a dense and a sparse vector for every chunk in a Qdrant collection. Enable the qdrant feature flag:

cargo add swiftide@0.34 --features qdrant

let qdrant = Qdrant::builder()
    .batch_size(BATCH_SIZE)
    .vector_size(DENSE_VECTOR_SIZE)
    // One dense and one sparse vector, both over the chunk *and* its metadata
    .with_vector(EmbeddedField::Combined)
    .with_sparse_vector(EmbeddedField::Combined)
    .collection_name(&collection.0)
    .build()?;

Source: examples/docs-qa/src/store.rs#L38-L45

Qdrant keeps more of the node than LanceDB. Each point’s payload holds every metadata field, plus path, content (the chunk) and last_updated_at. The point id is the node id. Nodes without vectors fail with “Node without vectors”.

Setup creates the collection when it is missing. An existing collection is kept as it is, even when its vector configuration differs from the builder. The client reads QDRANT_URL (default http://localhost:6334, gRPC) and QDRANT_API_KEY. The default batch size is 50; docs-qa raises it to 64.

A debug build of docs-qa split four markdown pages into 43 chunks and stored them in Qdrant (metadata from claude-haiku-5-5, embeddings from local FastEmbed):

Indexed 43 chunks into `docs-qa` in 71.7s

Qdrant covers named vectors, hybrid search and filters.

Backends

Every backend sits behind its own feature flag, except MemoryStorage:

Storage	Feature flag	What it stores	Default batch size	Retrieval
`LanceDB`	`lancedb`	id, chunk, configured vectors, `with_metadata` fields	256	yes
`Qdrant`	`qdrant`	vectors; payload with all metadata, path, chunk	50	yes
`PgVector`	`pgvector`	id, chunk, configured vectors, `with_metadata` fields as JSONB	50	yes
`Duckdb`	`duckdb`	`uuid`, chunk, path, configured vectors; no metadata	none, node by node	yes
`Redis`	`redis`	the node as JSON under `<path>:<node id>`	10	no
`Kafka`	`kafka`	the node as JSON, keyed `<path>:<node id>`, on a topic	32	no
`MemoryStorage`	none	the whole node, in memory	none, node by node	no

PgVector fails to store a node that lacks a field named with with_metadata; see PgVector. Kafka’s setup fails when the topic does not exist, unless you set create_topic_if_not_exists. Kafka is also a loader; see Kafka and Fluvio.

DuckDB stores node by node and ignores its builder’s batch_size. It rejects a node that lacks a vector for a configured field. When a row with the same uuid exists, DuckDB keeps the old row unless you set upsert_vectors, which needs DuckDB 1.2.1 or later.

Redis works as a key-value store for nodes. persist_key_fn changes the key and persist_value_fn the value. Both take a plain fn pointer, so a closure that captures variables does not compile. The example writes only the metadata and raises the batch size from 10 to 50:

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
    .then_chunk(ChunkCode::try_for_language_and_chunk_size(
        "rust",
        10..2048,
    )?)
    .then_store_with(
        // By default the value is the full node serialized to JSON.
        // We can customize this by providing a custom function.
        Redis::try_build_from_url(&redis_url)?
            .persist_value_fn(|node| Ok(serde_json::to_string(&node.metadata)?))
            .batch_size(50)
            .build()?,
    )
    .run()
    .await?;

Source: examples/index_into_redis.rs#L39-L53

DuckDB

No swiftide example uses DuckDB, so this section describes the builder without code. Duckdb::builder() returns a DuckdbBuilder with these methods:

connection takes anything that converts into a duckdb::Connection. Swiftide does not re-export the duckdb crate, so add it to your crate yourself.
with_vector(field, size) adds a vector column of size floats for an EmbeddedField, such as EmbeddedField::Combined.
table_name defaults to swiftide, and cache_table to swiftide_cache. cache_key_prefix prefixes the node cache keys, which also lets you invalidate the cache by hand.
schema replaces the SQL that loads the extensions and creates the table. The upsert statement is built from the table name and vectors, so keep the column names when you change it. upsert_vectors is the flag described above.

Duckdb implements Persist for then_store_with, NodeCache for filter_cached, and Retrieve for three search strategies: SimilaritySingleEmbedding, HybridSearch and CustomStrategy<String>. The setup installs and loads the vss and fts extensions before it creates the table. Retrieved documents carry id and path as metadata, since no other metadata is stored. See DuckdbBuilder for every method.

MemoryStorage

MemoryStorage lives in swiftide::indexing::persist and needs no feature flag. It suits tests and experiments:

let storage = MemoryStorage::default();

Pipeline::from_stream(vec![
    TextNode::new("Swiftide is a Rust library for building LLM applications."),
    TextNode::new("An indexing pipeline loads, transforms and stores nodes."),
])
.then(|mut node: TextNode| -> anyhow::Result<TextNode> {
    let words = node.chunk.split_whitespace().count();
    node.metadata.insert("Words", words);
    Ok(node)
})
// Clones share the same map, so keep one to read from later
.then_store_with(storage.clone())
.run()
.await?;

for node in storage.get_all_values().await {
    println!("{:?} words: {}", node.metadata.get("Words"), node.chunk);
}

Source: snippets/src/indexing/storage.rs#L15-L33

Its keys are an incrementing counter, not the node id, so storing the same node twice keeps two copies. get(key) reads one node, get_all_values() returns all nodes, and get_all() returns them with their keys. It has no retrieval for query pipelines.

Batching

A storage chooses how it receives nodes through batch_size. When it returns Some(n), the pipeline groups up to n nodes and calls batch_store. When it returns None, the pipeline calls store for each node. In both cases, up to the pipeline’s concurrency of these calls run at once. Storage has no concurrency setting of its own, and Concurrency explains when the pipeline value is read. For how this differs from the embedder’s batch size, see Batch sizes.

Stored nodes flow on, so you can chain a second then_store_with to write to two backends. If setup fails, for example because Qdrant is unreachable, run returns that error. To keep going after a failed write, see Errors.

Custom storage

Implement Persist to write nodes anywhere else. This one appends each node as a line of JSON:

/// Appends every node as one line of JSON to a file.
#[derive(Debug, Clone)]
pub struct JsonLines {
    path: PathBuf,
    // Batches are stored concurrently; one writer at a time keeps lines whole
    write_lock: Arc<Mutex<()>>,
}

impl JsonLines {
    pub fn new(path: impl Into<PathBuf>) -> Self {
        Self {
            path: path.into(),
            write_lock: Arc::default(),
        }
    }

    async fn append(&self, nodes: &[TextNode]) -> anyhow::Result<()> {
        let mut lines = String::new();
        for node in nodes {
            lines.push_str(&serde_json::to_string(node)?);
            lines.push('\n');
        }

        let _guard = self.write_lock.lock().await;
        let mut file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(&self.path)
            .await?;
        file.write_all(lines.as_bytes()).await?;
        Ok(())
    }
}

#[async_trait]
impl Persist for JsonLines {
    type Input = String;
    type Output = String;

    // Runs once, before the first node arrives
    async fn setup(&self) -> anyhow::Result<()> {
        if let Some(dir) = self.path.parent() {
            tokio::fs::create_dir_all(dir).await?;
        }
        Ok(())
    }

    // Used when `batch_size` returns `None`
    async fn store(&self, node: TextNode) -> anyhow::Result<TextNode> {
        self.append(std::slice::from_ref(&node)).await?;
        Ok(node)
    }

    // Used when `batch_size` returns `Some`; pass the stored nodes on
    async fn batch_store(&self, nodes: Vec<TextNode>) -> IndexingStream<String> {
        self.append(&nodes).await.map(|()| nodes).into()
    }

    fn batch_size(&self) -> Option<usize> {
        Some(100)
    }
}

Source: snippets/src/indexing/storage.rs#L39-L100

Input is the chunk type the storage accepts, and Output the chunk type it passes on. Both are String for text nodes.
batch_store returns an IndexingStream. Return the stored nodes so later steps receive them, or an error.
Persist requires Debug, Clone, Send and Sync, so fields like Rc or RefCell do not compile. Implement it with #[async_trait].

Batches are stored concurrently, so guard anything they share. Use the storage like any other:

Pipeline::from_stream(vec![TextNode::new("Stored as a line of JSON.")])
    .then_store_with(JsonLines::new("index/nodes.jsonl"))
    .run()
    .await?;

Source: snippets/src/indexing/storage.rs#L106-L109

Re-indexing

A node’s id is computed from its path and its chunk, as Cache keys explains. When you index an unchanged chunk again, LanceDB, Qdrant, PgVector and DuckDB find the same id and keep one row. DuckDB keeps the old row as it is unless upsert_vectors is set. When a chunk changes, it gets a new id and a new row, and the row of the old version stays. Swiftide does not delete rows for you.

docs-qa’s --recreate flag drops the Qdrant collection before indexing. For incremental updates, delete a file’s rows yourself before you index its new version, and skip unchanged files with a node cache.

Reference

Streams and concurrency