# Metrics and pipeline statistics

> Count token usage per model with the metrics feature flag and a metrics-rs recorder, and report what an indexing run processed from its statistics or from your own counters.

Your metrics backend gets a token counter per model and per label you choose, and each successful indexing run reports how many nodes it handled and how long it took.

## Usage metrics

Add the `metrics` feature flag next to the integration you use, plus the `metrics` crate for the recorder:

```sh
cargo add swiftide@0.34 --features metrics,openai
cargo add metrics@0.24
```

```rust title="examples/usage_metrics.rs"
let metric_metadata = HashMap::from([("example".to_string(), "metadata".to_string())]);
let openai_client = integrations::openai::OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_prompt_model("gpt-4.1-nano")
    // Metadata will be added to every metric
    .metric_metadata(metric_metadata)
    .build()?;
```

Source: [`examples/usage_metrics.rs#L30-L36`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/usage_metrics.rs#L30-L36)

With the `metrics` feature flag, a model client counts the tokens of every call whose response reports usage. `metric_metadata` takes a `HashMap<String, String>`. Each entry becomes a label on every counter this client emits. Swiftide always adds a `model` label with the model of the call. Two clients with different metadata, such as one per tenant or per pipeline, show up as separate series.

### Recorder

```rust title="examples/usage_metrics.rs"
#[derive(Clone, Debug)]
struct PrintHandle(Key);

impl CounterFn for PrintHandle {
    fn increment(&self, value: u64) {
        println!("counter increment for '{}': {}", self.0, value);
    }

    fn absolute(&self, value: u64) {
        println!("counter absolute for '{}': {}", self.0, value);
    }
}

impl GaugeFn for PrintHandle {
    fn increment(&self, value: f64) {
        println!("gauge increment for '{}': {}", self.0, value);
    }

    fn decrement(&self, value: f64) {
        println!("gauge decrement for '{}': {}", self.0, value);
    }

    fn set(&self, value: f64) {
        println!("gauge set for '{}': {}", self.0, value);
    }
}

impl HistogramFn for PrintHandle {
    fn record(&self, value: f64) {
        println!("histogram record for '{}': {}", self.0, value);
    }
}

#[derive(Debug)]
struct PrintRecorder;

impl Recorder for PrintRecorder {
    fn describe_counter(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(counter) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn describe_gauge(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(gauge) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn describe_histogram(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(histogram) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn register_counter(&self, key: &Key, _metadata: &Metadata<'_>) -> Counter {
        Counter::from_arc(Arc::new(PrintHandle(key.clone())))
    }

    fn register_gauge(&self, key: &Key, _metadata: &Metadata<'_>) -> Gauge {
        Gauge::from_arc(Arc::new(PrintHandle(key.clone())))
    }

    fn register_histogram(&self, key: &Key, _metadata: &Metadata<'_>) -> Histogram {
        Histogram::from_arc(Arc::new(PrintHandle(key.clone())))
    }
}

fn init_print_logger() {
    metrics::set_global_recorder(PrintRecorder).unwrap()
}
```

Source: [`examples/usage_metrics.rs#L91-L170`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/usage_metrics.rs#L91-L170)

Swiftide only calls the metrics-rs facade. It never installs a recorder, so your app does that once at startup with `metrics::set_global_recorder`. Without a recorder the counters do nothing. The example's recorder prints every call. An exporter crate such as `metrics-exporter-prometheus` installs a recorder the same way. It then serves the values to your backend.

Install the recorder before the first model call. Swiftide sends the counter descriptions once per process, on the first usage it emits. A recorder installed later never receives them.

### Counters

| Counter                            | Description                    |
| ---------------------------------- | ------------------------------ |
| `swiftide.usage.prompt_tokens`     | token usage for the prompt     |
| `swiftide.usage.completion_tokens` | token usage for the completion |
| `swiftide.usage.total_tokens`      | total token usage              |

These clients emit them:

- OpenAI-compatible clients (`OpenAI`, `Groq`, `Gemini`, `Ollama`, `OpenRouter`, `Dashscope`): chat completions, streaming, `prompt`, structured prompts, embeddings and the Responses API.
- `Anthropic`: prompts, completions and streaming.
- `AwsBedrock`.

Local embedders such as FastEmbed emit nothing. These three counters are all the `metrics` feature flag emits. It adds no pipeline or agent timing metrics.

```rust title="examples/usage_metrics.rs"
indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..512))
    .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10))
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples-metrics")
            .build()?,
    )
    .run()
    .await?;
```

Source: [`examples/usage_metrics.rs#L40-L51`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/usage_metrics.rs#L40-L51)

In this pipeline, the embedder sends one request per batch of 10 chunks, and each request adds to all three counters. Run it from a clone of the Swiftide repository, with Qdrant listening on `localhost:6334` and `OPENAI_API_KEY` set:

```sh
cargo run -p swiftide-examples --example usage-metrics
```

The examples package is not a default workspace member, so `-p swiftide-examples` is required from the workspace root. That package enables many integrations, including LanceDB, Kafka, Fluvio and DuckDB, so the first build takes a while.

The example keeps the output of one run in a comment at the end of `main`. These are its first lines, one per `println!`, for embeddings with `text-embedding-3-small`:

```text
(counter) registered key swiftide.usage.prompt_tokens with unit None and description "token usage for the prompt"
(counter) registered key swiftide.usage.completion_tokens with unit None and description "token usage for the completion"
(counter) registered key swiftide.usage.total_tokens with unit None and description "total token usage"
counter increment for 'Key(swiftide.usage.prompt_tokens, [example = metadata, model = text-embedding-3-small])': 356
counter increment for 'Key(swiftide.usage.completion_tokens, [example = metadata, model = text-embedding-3-small])': 0
counter increment for 'Key(swiftide.usage.total_tokens, [example = metadata, model = text-embedding-3-small])': 356
```

Embeddings have no completion, so `completion_tokens` grows by 0.

### Metrics gotchas

- `AnthropicBuilder::metric_metadata` takes an `Option<HashMap<String, String>>`, so wrap the map in `Some`. The OpenAI-compatible and `AwsBedrock` builders take the map itself.
- The counters only go up. Compute rates and per-period totals in your backend.
- To act on the usage of one call in your own code, set an `on_usage` callback instead. See [Errors, backoff, usage and tokens](/models/errors-backoff-and-usage/).

## Pipeline statistics

Every indexing pipeline carries a `StatsCollector`. `Pipeline::run` returns a `PipelineStats` snapshot of it at the end of the run.

```rust
let pipeline =
    indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
        .then_store_with(MemoryStorage::default());

let stats = pipeline.run().await?;
println!(
    "Stored {} chunks in {:?} ({:.1} chunks/s)",
    stats.nodes_stored,
    stats.duration().unwrap_or_default(),
    stats.nodes_per_second().unwrap_or_default(),
);
```

Source: [`snippets/src/observability/metrics_and_statistics.rs#L40-L51`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/observability/metrics_and_statistics.rs#L40-L51)

`PipelineStats` holds node counts, per-model `token_usage` and timing, with helpers such as `duration` and `nodes_per_second`. `run` also logs one INFO event at the end of a successful run, `Pipeline completed`, with the fields `nodes_processed`, `nodes_stored`, `total_tokens`, `total_requests`, `elapsed_secs` and `nodes_per_sec`. Enable INFO for the `swiftide_indexing` target to see it, for example with `RUST_LOG=swiftide_indexing=info`. [Tracing](/observability/tracing/) shows the subscriber setup.

### Watch a run in progress

`run` takes the pipeline by value. To read the numbers while it runs, clone the collector handle first. `Pipeline::stats_collector` returns the `Arc<StatsCollector>` the pipeline shares with every step.

```rust
let pipeline =
    indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
        .then_store_with(MemoryStorage::default());

// Take the handle before `run` consumes the pipeline
let collector = Arc::clone(pipeline.stats_collector());
let running = tokio::spawn(pipeline.run());

while !running.is_finished() {
    println!("Stored {} chunks so far", collector.get_stats().nodes_stored);
    tokio::time::sleep(std::time::Duration::from_secs(5)).await;
}

let stats = running.await??;
println!("Stored {} chunks", stats.nodes_stored);
```

Source: [`snippets/src/observability/metrics_and_statistics.rs#L59-L74`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/observability/metrics_and_statistics.rs#L59-L74)

`nodes_stored` goes up as nodes reach the end of the pipeline. `nodes_processed` is set once, when the stream is done. `Pipeline::stats` returns the same snapshot as `collector.get_stats()`; before the run every count is 0.

`split_by` gives each branch its own collector, seeded with the counts so far, so a handle taken before the split sees neither branch. `merge` keeps the collector of the pipeline you call it on.

### Statistics gotchas

- `nodes_processed` and `nodes_stored` both count the nodes that reach the end of the pipeline. After a run they are the same number, even when the last step is not storage.
- Nothing in Swiftide records token usage, failed nodes or transformations into the collector during a run. `total_tokens` and `total_requests` are always 0, and so are `nodes_failed` and `transformations_applied`. To record them yourself, call `record_token_usage`, `increment_nodes_failed` or `increment_transformations` on the shared handle.
- The returned stats, the event and the `total_nodes` span field only exist when the run succeeds. If a node error reaches the end of the stream, `run` returns the error instead.
- For token totals per pipeline, give the client of each pipeline its own `metric_metadata` label. The counters add up across all runs of the process, so take per-run numbers as differences in your backend.

## Node counters

The ask-your-codebase showcase takes the chunk count from the stats that `run` returns, and counts files with transformers of its own.

```rust title="examples/ask-your-codebase/src/index.rs"
let stats = indexing::Pipeline::from_loader(
    FileLoader::new(codebase.root()).with_extensions(&languages.extensions()),
)
.filter(test_code::not_test_files(codebase.clone()))
// Drop unit tests before the cache, which keys on path and content. Changing the content
// after it would make the cache miss on every run.
.then(test_code::without_unit_tests)
.then(files_found.count())
// Skip files whose path and content are unchanged since the last successful run
.filter_cached(codebase.node_cache()?)
.then(files_changed.count())
.then(record_relative_path(codebase.clone()))
// Split along syntax boundaries (functions, impls, ...) with tree-sitter
.then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?)
// Ask the LLM which questions each chunk answers; stored as metadata and embedded
.then(
    MetadataQACode::from_client(llm)
        .prompt_template(QUESTIONS_PROMPT)
        .num_questions(QUESTIONS_PER_CHUNK)
        .concurrency(concurrent_requests)
        .build()?,
)
// Embed chunk + metadata locally, in batches
.then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE))
.then_store_with(codebase.vector_store()?)
.run()
.await?;
```

Source: [`examples/ask-your-codebase/src/index.rs#L89-L115`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/ask-your-codebase/src/index.rs#L89-L115)

`files_found.count()` sees every source file that passes the test-file filter. `files_changed.count()` sits after `filter_cached`, so it only sees files the [node cache](/indexing/caching/) did not skip. `stats.nodes_stored` is the number of chunks that reached storage. Each `Tally` wraps an `Arc<AtomicU64>`, and the showcase measures the elapsed time with its own `Instant`, so the time includes loading the models.

The same pattern as a closure [transformer](/indexing/transformers-and-metadata/):

```rust
let chunks_stored = Arc::new(AtomicU64::new(0));
let count_chunk = {
    let chunks_stored = Arc::clone(&chunks_stored);
    move |node: TextNode| -> anyhow::Result<TextNode> {
        chunks_stored.fetch_add(1, Ordering::Relaxed);
        Ok(node)
    }
};

indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then_store_with(MemoryStorage::default())
    .then(count_chunk)
    .run()
    .await?;

let chunks_stored = chunks_stored.load(Ordering::Relaxed);
println!("Stored {chunks_stored} chunks");
```

Source: [`snippets/src/observability/metrics_and_statistics.rs#L15-L32`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/observability/metrics_and_statistics.rs#L15-L32)

A closure from `TextNode` to `anyhow::Result<TextNode>` is a transformer. This one adds 1 to a shared counter and returns the node unchanged. Steps run nodes concurrently, so the counter is an atomic behind an `Arc`, and the closure holds its own clone of it.

The showcase prints these lines from its counters and the run's stats. It indexed the `swiftide-agents` crate with `claude-haiku-5-5` through an OpenAI-compatible gateway, and FastEmbed embeddings on the CPU:

```text
Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s
```

On the second run the Redb node cache skipped every file:

```text
Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s
```

The docs-qa showcase takes another route: after the run, it asks Qdrant how many points the collection holds. That counts the whole collection, not only this run. The number matches the run here because the captured run used `--recreate`, which drops the collection first. With `claude-haiku-5-5` writing chunk titles and keywords, it printed:

```text
Indexed 43 chunks into `docs-qa` in 71.7s
```

## Reference

- [`Pipeline`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.Pipeline.html) (`stats`, `stats_collector`, `run`)
- [`PipelineStats`](https://docs.rs/swiftide/0.34/swiftide/indexing/statistics/struct.PipelineStats.html)
- [`StatsCollector`](https://docs.rs/swiftide/0.34/swiftide/indexing/statistics/struct.StatsCollector.html)
- [`ModelUsage`](https://docs.rs/swiftide/0.34/swiftide/indexing/statistics/struct.ModelUsage.html)
- [`GenericOpenAIBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.GenericOpenAIBuilder.html) (`metric_metadata`)
- [`AnthropicBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/anthropic/struct.AnthropicBuilder.html)
- [`AwsBedrockBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/aws_bedrock_v2/struct.AwsBedrockBuilder.html)
- [Feature flags](/reference/feature-flags/)
- [metrics-rs](https://docs.rs/metrics/0.24)

## Next

[Integrations overview](/integrations/overview/)