How indexing works

You can read a Swiftide indexing pipeline line by line and know when each step runs. The example comes from the ask-your-codebase showcase. It prepares the source files of a codebase for retrieval-augmented generation (RAG): it splits them into chunks, describes each chunk with a large language model (LLM), embeds it, and stores it in LanceDB.

The ask-your-codebase pipeline

let stats = indexing::Pipeline::from_loader(
    FileLoader::new(codebase.root()).with_extensions(&languages.extensions()),
)
.filter(test_code::not_test_files(codebase.clone()))
// Drop unit tests before the cache, which keys on path and content. Changing the content
// after it would make the cache miss on every run.
.then(test_code::without_unit_tests)
.then(files_found.count())
// Skip files whose path and content are unchanged since the last successful run
.filter_cached(codebase.node_cache()?)
.then(files_changed.count())
.then(record_relative_path(codebase.clone()))
// Split along syntax boundaries (functions, impls, ...) with tree-sitter
.then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?)
// Ask the LLM which questions each chunk answers; stored as metadata and embedded
.then(
    MetadataQACode::from_client(llm)
        .prompt_template(QUESTIONS_PROMPT)
        .num_questions(QUESTIONS_PER_CHUNK)
        .concurrency(concurrent_requests)
        .build()?,
)
// Embed chunk + metadata locally, in batches
.then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE))
.then_store_with(codebase.vector_store()?)
.run()
.await?;

Source: examples/ask-your-codebase/src/index.rs#L89-L115

Indexing pipeline

Each node passes through the steps from top to bottom until a step drops it:

FileLoader walks the codebase root and emits one node per file with a matching extension.
filter drops test files. The closure sees each Result, so errors pass through and still surface.
then(test_code::without_unit_tests) cuts unit tests out of Rust files.
filter_cached drops files whose path and content did not change since the last successful run. codebase.node_cache() is a showcase helper that builds a Redb node cache, see Caching.
then(record_relative_path(..)) stores the file path, relative to the root, as metadata.
then_chunk splits each file along syntax boundaries with tree-sitter. ChunkByLanguage is the showcase’s own wrapper that picks a Swiftide ChunkCode per language. One file becomes several nodes.
MetadataQACode asks the LLM which questions each chunk answers and stores them as metadata.
then_in_batch with Embed embeds each chunk and its metadata with FastEmbed, on the central processing unit (CPU). with_batch_size sets 32 nodes per call. Without it, a batch holds 256 nodes.
then_store_with writes each node to storage. codebase.vector_store() builds a LanceDB store, see Storage.
run().await drives the stream to the end and returns the run’s statistics as stats.

Nodes

A node is a Node<T>. It carries one piece of content and everything known about it.

Field	Holds
`path`	The source path.
`chunk`	The content, of type `T`.
`metadata`	Key and value pairs, sorted by key.
`vectors`, `sparse_vectors`	Embeddings, one per embedded field. `Embed` and `SparseEmbed` fill them.
`embed_mode`	Whether the chunk and metadata are embedded together, each on its own, or both.
`offset`	A running position that `ChunkCode` counts from 0 over the chunks it keeps.
`parent_id`	The `id()` of the source node when the first chunker split it.

Since 0.31 the pipeline is generic. A Pipeline<T> streams Node<T>. A chunk is any type that implements the Chunk marker trait, which requires Clone + Send + Sync + Debug + AsRef<[u8]> + 'static. Most built-in steps take TextNode, which is Node<String>.

id() returns a universally unique identifier (UUID) computed from the path and the chunk, so it changes when either changes. The node cache keys on this id, see Cache keys.

Steps

Step	Method	Trait
loader	`Pipeline::from_loader`, `Pipeline::from_stream`	`Loader`
filter	`filter`	a closure over `&Result<Node<T>>`
node cache	`filter_cached`	`NodeCache`
transformer	`then`	`Transformer`, or a closure
chunker	`then_chunk`, `then_expand`	`ChunkerTransformer`
batch transformer	`then_in_batch`	`BatchableTransformer`
embedder	`then_in_batch(Embed::new(model))`	`EmbeddingModel`
storage	`then_store_with`	`Persist`

A chunker returns a stream of nodes for each node. then_expand behaves like then_chunk with a different name in traces. Use it when you split a node for a reason other than chunking.

A step can change the node type. then with a transformer whose Output is U turns a Pipeline<T> into a Pipeline<U>.

Nodes keep flowing after storage.

Execution

The pipeline methods only describe the stream, but their arguments run while you build the chain. In the snippet, codebase.node_cache()? creates the Redb file and embedding::local_model()? loads the FastEmbed model, which downloads it on first use.

run takes the pipeline by value and does the work:

It calls setup on every storage backend. LanceDB creates its table here if it is missing.
It pulls nodes through the stream until the stream ends. Some steps spawn Tokio tasks, so run needs a Tokio runtime.
It marks the source nodes in the node cache, but only after the stream ended without an error.
It logs Pipeline completed at info level and returns the same numbers as PipelineStats, see Statistics.

run returns Result<PipelineStats>, not the nodes. The showcase reads the number of stored chunks from stats.nodes_stored. It counts files with pass-through closures such as .then(files_found.count()), because the statistics count nodes at the end of the stream, after chunking.

then, then_chunk, then_in_batch and storage work on several nodes at once and pass them on as they finish, see Streams and concurrency. The showcase passes .concurrency(concurrent_requests) to MetadataQACode to bound requests to the LLM.

Errors and the node cache

The first error stops the run, and run returns it, see Errors. It is the error the step failed with, so a LanguageModelError can be recovered with downcast_ref. A source node is cached only when at least one of its chunks reached the end and none of them failed, and When nodes are cached covers the consequences.

Output

Two runs on the swiftide-agents crate. claude-haiku-5-5 wrote the questions, through an OpenAI-compatible gateway to Amazon Web Services (AWS) Bedrock. FastEmbed jina-embeddings-v2-base-code computed the embeddings.

$ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5
Indexing Rust files in swiftide-agents ...
Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s

The second run, with no file changed:

$ ./target/debug/ask-your-codebase index swiftide-agents --model claude-haiku-5-5
Indexing Rust files in swiftide-agents ...
Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s

The node cache drops all 14 files at step 4, so no prompt is sent.

Feature flags

Pipeline, FileLoader and Embed need no feature flag. The showcase turns off the defaults and turns on tree-sitter, fastembed, lancedb, redb and openai. To add the same features to your crate:

cargo add swiftide@0.34 --no-default-features --features tree-sitter,fastembed,lancedb,redb,openai

TLS needs no setup, because swiftide’s HTTP clients bring their own rustls crypto provider (TLS). Feature flags lists every flag.

Reference

Loading data