# Your first pipeline

> Index a directory of Rust code into an embedded LanceDB table with local FastEmbed embeddings, then ask it one question and get an answer that cites files.

This page indexes one directory of Rust code into an embedded LanceDB table and answers a question about it, naming the files the answer came from. You run the [ask-your-codebase](/showcases/ask-your-codebase/) showcase and read the parts that make it work. Everything except the LLM (large language model) runs on your machine. Tree-sitter chunks the code, FastEmbed embeds it on the CPU (central processing unit), and LanceDB stores the vectors in a local directory.

## Showcase run

The showcase lives in `examples/ask-your-codebase/` in the swiftide repository. Clone it. To run exactly the code this site shows, check out the commit it is pinned to:

```sh
git clone https://github.com/bosun-ai/swiftide.git
cd swiftide
git checkout e377c4a2e76fe2559629ae0efdae3c6e0fb4d829  # optional: the commit this site documents
export OPENAI_API_KEY=sk-...
cargo run -p ask-your-codebase -- index swiftide-agents
cargo run -p ask-your-codebase -- ask --path swiftide-agents "How does an agent decide to stop?"
```

## Dependencies

To build the same pipelines in your own crate, add the showcase's dependencies:

```sh
cargo add swiftide@0.34 --no-default-features --features tree-sitter,fastembed,lancedb,redb,openai
cargo add fastembed@5 --no-default-features
cargo add tokio --features full
cargo add anyhow async-trait
```

Each integration has its own [feature flag](/reference/feature-flags/). The direct `fastembed` dependency picks a model other than the default. Only the showcase's own chunker needs `async-trait`.

## LLM client

```rust title="examples/ask-your-codebase/src/llm.rs"
impl LlmEndpoint {
    /// Builds a client for this endpoint.
    ///
    /// Rate limits and transient errors are retried with exponential backoff, which matters when
    /// indexing fires many prompts concurrently. After `retry_budget` the error is returned, so
    /// an unreachable or broken endpoint fails instead of hanging.
    pub fn client(&self, retry_budget: Duration) -> Result<Arc<dyn SimplePrompt>> {
        let openai = OpenAI::builder()
            .api_base(&self.base_url)
            .api_key(&self.api_key)
            .default_prompt_model(&self.model)
            .use_responses_api(matches!(self.api, OpenAiApi::Responses))
            .build()?;

        let backoff = BackoffConfiguration {
            max_elapsed_time_sec: retry_budget.as_secs(),
            ..BackoffConfiguration::default()
        };

        Ok(Arc::new(LanguageModelWithBackOff::new(openai, backoff)))
    }
}
```

Source: [`examples/ask-your-codebase/src/llm.rs#L79-L100`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/llm.rs#L79-L100)

The client is swiftide's `OpenAI` integration pointed at a base URL with `api_base` and `api_key`. Indexing sends many prompts at once, so `LanguageModelWithBackOff` wraps it and retries transient errors, such as rate limits, with exponential backoff. A wrong key is a permanent error and fails at once. `max_elapsed_time_sec` in `BackoffConfiguration` caps how long one prompt is [retried](/models/errors-backoff-and-usage/); the showcase allows 60 seconds while indexing and 30 while asking.

## Embedder and storage

```rust title="examples/ask-your-codebase/src/embedding.rs"
/// A code-aware embedding model that runs on the CPU and needs no API key.
///
/// The model (~600MB) is downloaded on first use.
pub fn local_model() -> Result<FastEmbed> {
    let model = TextEmbedding::try_new(
        InitOptions::new(EmbeddingModel::JinaEmbeddingsV2BaseCode).with_max_length(MAX_TOKENS),
    )?;

    FastEmbed::builder()
        .embedding_model(model)
        .batch_size(BATCH_SIZE)
        .build()
}
```

Source: [`examples/ask-your-codebase/src/embedding.rs#L18-L30`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/embedding.rs#L18-L30)

The embedder runs `jina-embeddings-v2-base-code`, a model trained on code, with up to 2048 tokens per text. Its vectors have 768 dimensions, kept in `DIMENSIONS`. `BATCH_SIZE` (32) is the batch size FastEmbed uses inside one embedding call. Keep it equal to the batch size of the pipeline step that calls the embedder.

```rust title="examples/ask-your-codebase/src/codebase.rs"
/// Remembers which files were fully indexed, so unchanged files are skipped next time.
pub fn node_cache(&self) -> Result<Redb> {
    std::fs::create_dir_all(&self.data_dir)?;

    Redb::builder()
        .database_path(self.data_dir.join("cache.redb"))
        .table_name("indexed_files")
        .build()
}

/// Embedded vector store with one vector per chunk and the source path next to it.
pub fn vector_store(&self) -> Result<LanceDB> {
    let uri = self.lancedb_dir();

    LanceDB::builder()
        .uri(uri.to_str().context("Index path is not valid UTF-8")?)
        .table_name("code_chunks")
        .vector_size(embedding::DIMENSIONS)
        .with_vector(EmbeddedField::Combined)
        .with_metadata(PATH_FIELD)
        .build()
}
```

Source: [`examples/ask-your-codebase/src/codebase.rs#L56-L77`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/codebase.rs#L56-L77)

`Redb` is the node cache, a single embedded file. `LanceDB` is the storage: `uri` is a local directory, and `vector_size` must equal the embedder's dimensions. `EmbeddedField::Combined` stores one vector per chunk, computed from the chunk and its metadata. `with_metadata(PATH_FIELD)` keeps the file path in its own column, so the answer can cite it. Both live in `<path>/.ask-your-codebase/`, inside the indexed directory.

## Indexing pipeline

```rust title="examples/ask-your-codebase/src/index.rs"
let stats = indexing::Pipeline::from_loader(
    FileLoader::new(codebase.root()).with_extensions(&languages.extensions()),
)
.filter(test_code::not_test_files(codebase.clone()))
// Drop unit tests before the cache, which keys on path and content. Changing the content
// after it would make the cache miss on every run.
.then(test_code::without_unit_tests)
.then(files_found.count())
// Skip files whose path and content are unchanged since the last successful run
.filter_cached(codebase.node_cache()?)
.then(files_changed.count())
.then(record_relative_path(codebase.clone()))
// Split along syntax boundaries (functions, impls, ...) with tree-sitter
.then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?)
// Ask the LLM which questions each chunk answers; stored as metadata and embedded
.then(
    MetadataQACode::from_client(llm)
        .prompt_template(QUESTIONS_PROMPT)
        .num_questions(QUESTIONS_PER_CHUNK)
        .concurrency(concurrent_requests)
        .build()?,
)
// Embed chunk + metadata locally, in batches
.then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE))
.then_store_with(codebase.vector_store()?)
.run()
.await?;
```

Source: [`examples/ask-your-codebase/src/index.rs#L89-L115`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/index.rs#L89-L115)

The indexing pipeline is a stream of nodes. Nothing runs until `run()` is awaited. `run` returns the pipeline statistics; the showcase reports `nodes_stored` as the chunk count. The marked lines are the core:

- The loader, `FileLoader`, reads the files under the root with the given extensions. It skips hidden files and what `.gitignore` and `.ignore` files exclude. `.gitignore` only applies inside a git repository. Each file becomes one node.
- `filter_cached` drops nodes the node cache has seen before. The cache key is a hash of the path and the content, so an edited file counts as new.
- `then_chunk` runs the chunker. `ChunkByLanguage` is a small showcase type that picks a tree-sitter `ChunkCode` per file extension. It splits along syntax boundaries into chunks of 10 to 4096 bytes.
- `then` runs a transformer on each chunk. `MetadataQACode` asks the LLM, with a custom prompt, for eight questions the chunk answers and stores them as metadata. `concurrency` caps the prompts in flight.
- `then_in_batch` runs a batch transformer. `Embed` sends 32 chunks per call to FastEmbed.
- `then_store_with` hands each chunk to the storage, here LanceDB, through the `Persist` trait.

The other lines are showcase extras that drop tests, record the relative path and count for the report. Leave them out in your own crate, and replace the showcase helpers:

- `FileLoader::new(codebase.root()).with_extensions(&languages.extensions())` becomes `FileLoader::new("./src").with_extensions(&["rs"])`.
- `ChunkByLanguage::try_new(languages, CHUNK_SIZE)?` becomes `ChunkCode::try_for_language_and_chunk_size("rust", 10..4096)?`.
- `codebase.node_cache()?` and `codebase.vector_store()?` are the builders from the storage excerpt.
- `llm` is the client from the LLM client excerpt. Without `prompt_template`, `MetadataQACode` uses its default prompt.

## Query pipeline

```rust title="examples/ask-your-codebase/src/query.rs"
let search = SimilaritySingleEmbedding::default().with_top_k(top_k);

let pipeline = query::Pipeline::from_search_strategy(search)
    // Rephrase the question into several, to widen what the search can match
    .then_transform_query(
        GenerateSubquestions::builder()
            .client(llm.clone())
            .prompt_template(SUBQUESTIONS_PROMPT.into())
            .build()?,
    )
    // Embed with the same local model that embedded the chunks
    .then_transform_query(Embed::from_client(embedding::local_model()?))
    .then_retrieve(codebase.vector_store()?)
    // Answer the original question from the retrieved code, each chunk under its file path
    .then_answer(
        answers::Simple::builder()
            .client(llm)
            .prompt_template(ANSWER_PROMPT.into())
            .document_template(CHUNK_TEMPLATE)
            .build()?,
    );

let answered = pipeline.query(question).await?;
```

Source: [`examples/ask-your-codebase/src/query.rs#L36-L58`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/query.rs#L36-L58)

`GenerateSubquestions` rewrites the question into several related ones, to widen what the search can match. The query transformer `Embed` turns them into one vector with the same local model that embedded the chunks. The search strategy, `SimilaritySingleEmbedding`, runs one vector search with that embedding and returns the `top_k` closest chunks. Its default is 10; the showcase passes 20. `then_retrieve` uses the same `LanceDB` store as the retriever. The answer generator, `answers::Simple`, answers from the retrieved chunks; its document template renders each chunk under its file path. [Answering](/querying/answering/) covers other answer generators.

`query` consumes the pipeline and returns the answered query. To ask more than once, call `query_mut` or `query_all`.

## Output

This output is from the showcase's `OUTPUT.md`. The chunk questions came from claude-haiku-5-5 and the answer from claude-sonnet-5-5, both through an OpenAI-compatible gateway. The embeddings came from FastEmbed on the CPU.

The first run indexes everything:

```text
Indexing Rust files in swiftide-agents ...
Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s
```

A second run finds every file in the node cache and sends no prompts:

```text
Indexing Rust files in swiftide-agents ...
Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s
```

Asking "How does an agent decide to stop?" gives, trimmed:

```text
# How an agent decides to stop

An agent stops when something calls `Agent::stop` (or `stop_with_error`), which moves the state to `State::Stopped(StopReason)`. The main loop in `run_agent` checks that state after each step. `Agent::stop` is a no-op if the agent is already stopped (`src/agent.rs`).

## The stop mechanism

- `Agent::stop(reason)` fires the `OnStop` hooks with the reason, then sets `self.state = State::Stopped(reason)`. It only takes effect at the end of the current loop iteration (`src/agent.rs`).
- `Agent::stop_with_error(&AgentError)` fires `OnStop` with `StopReason::Error` and the error, then sets `Stopped(StopReason::Error)` (`src/agent.rs`).
- You can inspect the result with `stop_reason()`, `is_stopped()` and `state()` (`src/agent.rs`, `src/state.rs`).
...
Sources:
  - src/agent.rs
  - src/tools/control.rs
  - src/lib.rs
  - src/state.rs
  - src/errors.rs
  - src/task_agent.rs
  - src/hooks.rs
  - src/tasks.rs
  - src/default_context.rs
```

## Variations

### OpenAI embeddings

The LanceDB example embeds with OpenAI's `text-embedding-3-small` instead, so its `vector_size` is 1536:

```rust
let openai_client = integrations::openai::OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_prompt_model("gpt-4o-mini")
    .build()?;

let tempdir = TempDir::new().unwrap();

// Configure lancedb with a default vector size, a single embedding
// and in addition to embedding the text metadata, also store it in a field
let lancedb = LanceDB::builder()
    .uri(tempdir.child("lancedb").to_str().unwrap())
    .vector_size(1536)
    .with_vector(EmbeddedField::Combined)
    .with_metadata(METADATA_QA_TEXT_NAME)
    .table_name("swiftide_test")
    .build()
    .unwrap();
```

Source: [`examples/lancedb.rs#L20-L36`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/lancedb.rs#L20-L36)

### Qdrant with the default FastEmbed model

`FastEmbed::builder()` without a model uses the default model, with 384 dimensions. This pipeline stores into Qdrant. It needs the `qdrant` feature flag and a Qdrant server at `QDRANT_URL`, which defaults to `http://localhost:6334`. See [Qdrant](/integrations/qdrant/):

```rust
indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
    .then_in_batch(Embed::new(FastEmbed::builder().batch_size(10).build()?))
    .then_store_with(
        Qdrant::try_from_url(qdrant_url)?
            .batch_size(50)
            .vector_size(384)
            .collection_name("swiftide-examples-fastembed".to_string())
            .build()?,
    )
    .run()
    .await?;
```

Source: [`examples/fastembed.rs#L32-L42`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/fastembed.rs#L32-L42)

For hybrid search and reranking, see the [docs-qa](/showcases/docs-qa/) showcase.

## Gotchas

- Use the same embedding model for indexing and querying, and set `vector_size` to its dimensions. After you change the model, delete `<path>/.ask-your-codebase/` in the indexed directory and index again.
- The first run downloads `jina-embeddings-v2-base-code`, about 600MB, into `.fastembed_cache/` in the current directory.
- When a file changes, its new chunks are added, but the chunks of the old version stay in LanceDB.
- Files enter the node cache only after `run()` finishes without an error. One failed chunk ends the run, and every file of that run is prompted and embedded again next time. The same holds for a run you interrupt.
- A file that produces no chunks is never cached, so it is loaded again on every run.

## Reference

- [`indexing::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html)
- [`NodeCache`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.NodeCache.html)
- [`Persist`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Persist.html)
- [`LanguageModelWithBackOff`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.LanguageModelWithBackOff.html)
- [`BackoffConfiguration`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.BackoffConfiguration.html)
- [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/type.OpenAI.html)
- [`FileLoader`](https://docs.rs/swiftide/0.34/swiftide/indexing/loaders/struct.FileLoader.html)
- [`ChunkCode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.ChunkCode.html)
- [`MetadataQACode`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQACode.html)
- [`transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html)
- [`EmbeddedField`](https://docs.rs/swiftide/0.34/swiftide/indexing/enum.EmbeddedField.html)
- [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html)
- [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html)
- [`Redb`](https://docs.rs/swiftide/0.34/swiftide/integrations/redb/struct.Redb.html)
- [`query::Pipeline`](https://docs.rs/swiftide/0.34/swiftide/query/struct.Pipeline.html)
- [`SimilaritySingleEmbedding`](https://docs.rs/swiftide/0.34/swiftide/query/search_strategies/struct.SimilaritySingleEmbedding.html)
- [`query_transformers::Embed`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.Embed.html)
- [`GenerateSubquestions`](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.GenerateSubquestions.html)
- [`answers::Simple`](https://docs.rs/swiftide/0.34/swiftide/query/answers/struct.Simple.html)

## Next

[Your first task](/start/your-first-task/)