Metrics and pipeline statistics

Your metrics backend gets a token counter per model and per label you choose, and each successful indexing run reports how many nodes it handled and how long it took.

Usage metrics

Add the metrics feature flag next to the integration you use, plus the metrics crate for the recorder:

cargo add swiftide@0.34 --features metrics,openai
cargo add metrics@0.24

examples/usage_metrics.rs

let metric_metadata = HashMap::from([("example".to_string(), "metadata".to_string())]);
let openai_client = integrations::openai::OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_prompt_model("gpt-4.1-nano")
    // Metadata will be added to every metric
    .metric_metadata(metric_metadata)
    .build()?;

Source: examples/usage_metrics.rs#L30-L36

With the metrics feature flag, a model client counts the tokens of every call whose response reports usage. metric_metadata takes a HashMap<String, String>. Each entry becomes a label on every counter this client emits. Swiftide always adds a model label with the model of the call. Two clients with different metadata, such as one per tenant or per pipeline, show up as separate series.

Recorder

examples/usage_metrics.rs

#[derive(Clone, Debug)]
struct PrintHandle(Key);

impl CounterFn for PrintHandle {
    fn increment(&self, value: u64) {
        println!("counter increment for '{}': {}", self.0, value);
    }

    fn absolute(&self, value: u64) {
        println!("counter absolute for '{}': {}", self.0, value);
    }
}

19 collapsed lines
impl GaugeFn for PrintHandle {
    fn increment(&self, value: f64) {
        println!("gauge increment for '{}': {}", self.0, value);
    }

    fn decrement(&self, value: f64) {
        println!("gauge decrement for '{}': {}", self.0, value);
    }

    fn set(&self, value: f64) {
        println!("gauge set for '{}': {}", self.0, value);
    }
}

impl HistogramFn for PrintHandle {
    fn record(&self, value: f64) {
        println!("histogram record for '{}': {}", self.0, value);
    }
}

#[derive(Debug)]
struct PrintRecorder;

impl Recorder for PrintRecorder {
    fn describe_counter(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(counter) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn describe_gauge(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(gauge) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn describe_histogram(&self, key_name: KeyName, unit: Option<Unit>, description: SharedString) {
        println!(
            "(histogram) registered key {} with unit {:?} and description {:?}",
            key_name.as_str(),
            unit,
            description
        );
    }

    fn register_counter(&self, key: &Key, _metadata: &Metadata<'_>) -> Counter {
        Counter::from_arc(Arc::new(PrintHandle(key.clone())))
    }

    fn register_gauge(&self, key: &Key, _metadata: &Metadata<'_>) -> Gauge {
        Gauge::from_arc(Arc::new(PrintHandle(key.clone())))
    }

    fn register_histogram(&self, key: &Key, _metadata: &Metadata<'_>) -> Histogram {
        Histogram::from_arc(Arc::new(PrintHandle(key.clone())))
    }
}

fn init_print_logger() {
    metrics::set_global_recorder(PrintRecorder).unwrap()
}

Source: examples/usage_metrics.rs#L91-L170

Swiftide only calls the metrics-rs facade. It never installs a recorder, so your app does that once at startup with metrics::set_global_recorder. Without a recorder the counters do nothing. The example’s recorder prints every call. An exporter crate such as metrics-exporter-prometheus installs a recorder the same way. It then serves the values to your backend.

Install the recorder before the first model call. Swiftide sends the counter descriptions once per process, on the first usage it emits. A recorder installed later never receives them.

Counters

Counter	Description
swiftide.usage.prompt_tokens	token usage for the prompt
swiftide.usage.completion_tokens	token usage for the completion
swiftide.usage.total_tokens	total token usage

These clients emit them:

• OpenAI-compatible clients (OpenAI, Groq, Gemini, Ollama, OpenRouter, Dashscope): chat completions, streaming, prompt, structured prompts, embeddings and the Responses API.
• Anthropic: prompts, completions and streaming.
• AwsBedrock.

Local embedders such as FastEmbed emit nothing. These three counters are all the metrics feature flag emits. It adds no pipeline or agent timing metrics.

examples/usage_metrics.rs

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(10..512))
    .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10))
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples-metrics")
            .build()?,
    )
    .run()
    .await?;

Source: examples/usage_metrics.rs#L40-L51

In this pipeline, the embedder sends one request per batch of 10 chunks, and each request adds to all three counters. Run it from a clone of the Swiftide repository, with Qdrant listening on localhost:6334 and OPENAI_API_KEY set:

cargo run -p swiftide-examples --example usage-metrics

The examples package is not a default workspace member, so -p swiftide-examples is required from the workspace root. That package enables many integrations, including LanceDB, Kafka, Fluvio and DuckDB, so the first build takes a while.

The example keeps the output of one run in a comment at the end of main. These are its first lines, one per println!, for embeddings with text-embedding-3-small:

(counter) registered key swiftide.usage.prompt_tokens with unit None and description "token usage for the prompt"
(counter) registered key swiftide.usage.completion_tokens with unit None and description "token usage for the completion"
(counter) registered key swiftide.usage.total_tokens with unit None and description "total token usage"
counter increment for 'Key(swiftide.usage.prompt_tokens, [example = metadata, model = text-embedding-3-small])': 356
counter increment for 'Key(swiftide.usage.completion_tokens, [example = metadata, model = text-embedding-3-small])': 0
counter increment for 'Key(swiftide.usage.total_tokens, [example = metadata, model = text-embedding-3-small])': 356

Embeddings have no completion, so completion_tokens grows by 0.

Metrics gotchas

• AnthropicBuilder::metric_metadata takes an Option<HashMap<String, String>>, so wrap the map in Some. The OpenAI-compatible and AwsBedrock builders take the map itself.
• The counters only go up. Compute rates and per-period totals in your backend.
• To act on the usage of one call in your own code, set an on_usage callback instead. See Errors, backoff, usage and tokens.

Pipeline statistics

Every indexing pipeline carries a StatsCollector. Pipeline::run returns a PipelineStats snapshot of it at the end of the run.

let pipeline =
    indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
        .then_store_with(MemoryStorage::default());

let stats = pipeline.run().await?;
println!(
    "Stored {} chunks in {:?} ({:.1} chunks/s)",
    stats.nodes_stored,
    stats.duration().unwrap_or_default(),
    stats.nodes_per_second().unwrap_or_default(),
);

Source: snippets/src/observability/metrics_and_statistics.rs#L40-L51

PipelineStats holds node counts, per-model token_usage and timing, with helpers such as duration and nodes_per_second. run also logs one INFO event at the end of a successful run, Pipeline completed, with the fields nodes_processed, nodes_stored, total_tokens, total_requests, elapsed_secs and nodes_per_sec. Enable INFO for the swiftide_indexing target to see it, for example with RUST_LOG=swiftide_indexing=info. Tracing shows the subscriber setup.

Watch a run in progress

run takes the pipeline by value. To read the numbers while it runs, clone the collector handle first. Pipeline::stats_collector returns the Arc<StatsCollector> the pipeline shares with every step.

let pipeline =
    indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
        .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
        .then_store_with(MemoryStorage::default());

// Take the handle before `run` consumes the pipeline
let collector = Arc::clone(pipeline.stats_collector());
let running = tokio::spawn(pipeline.run());

while !running.is_finished() {
    println!("Stored {} chunks so far", collector.get_stats().nodes_stored);
    tokio::time::sleep(std::time::Duration::from_secs(5)).await;
}

let stats = running.await??;
println!("Stored {} chunks", stats.nodes_stored);

Source: snippets/src/observability/metrics_and_statistics.rs#L59-L74

nodes_stored goes up as nodes reach the end of the pipeline. nodes_processed is set once, when the stream is done. Pipeline::stats returns the same snapshot as collector.get_stats(); before the run every count is 0.

split_by gives each branch its own collector, seeded with the counts so far, so a handle taken before the split sees neither branch. merge keeps the collector of the pipeline you call it on.

Statistics gotchas

• nodes_processed and nodes_stored both count the nodes that reach the end of the pipeline. After a run they are the same number, even when the last step is not storage.
• Nothing in Swiftide records token usage, failed nodes or transformations into the collector during a run. total_tokens and total_requests are always 0, and so are nodes_failed and transformations_applied. To record them yourself, call record_token_usage, increment_nodes_failed or increment_transformations on the shared handle.
• The returned stats, the event and the total_nodes span field only exist when the run succeeds. If a node error reaches the end of the stream, run returns the error instead.
• For token totals per pipeline, give the client of each pipeline its own metric_metadata label. The counters add up across all runs of the process, so take per-run numbers as differences in your backend.

Node counters

The ask-your-codebase showcase takes the chunk count from the stats that run returns, and counts files with transformers of its own.

examples/ask-your-codebase/src/index.rs

let stats = indexing::Pipeline::from_loader(
    FileLoader::new(codebase.root()).with_extensions(&languages.extensions()),
)
.filter(test_code::not_test_files(codebase.clone()))
// Drop unit tests before the cache, which keys on path and content. Changing the content
// after it would make the cache miss on every run.
.then(test_code::without_unit_tests)
.then(files_found.count())
// Skip files whose path and content are unchanged since the last successful run
.filter_cached(codebase.node_cache()?)
.then(files_changed.count())
.then(record_relative_path(codebase.clone()))
// Split along syntax boundaries (functions, impls, ...) with tree-sitter
.then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?)
// Ask the LLM which questions each chunk answers; stored as metadata and embedded
.then(
    MetadataQACode::from_client(llm)
        .prompt_template(QUESTIONS_PROMPT)
        .num_questions(QUESTIONS_PER_CHUNK)
        .concurrency(concurrent_requests)
        .build()?,
)
// Embed chunk + metadata locally, in batches
.then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE))
.then_store_with(codebase.vector_store()?)
.run()
.await?;

Source: examples/ask-your-codebase/src/index.rs#L89-L115

files_found.count() sees every source file that passes the test-file filter. files_changed.count() sits after filter_cached, so it only sees files the node cache did not skip. stats.nodes_stored is the number of chunks that reached storage. Each Tally wraps an Arc<AtomicU64>, and the showcase measures the elapsed time with its own Instant, so the time includes loading the models.

The same pattern as a closure transformer:

let chunks_stored = Arc::new(AtomicU64::new(0));
let count_chunk = {
    let chunks_stored = Arc::clone(&chunks_stored);
    move |node: TextNode| -> anyhow::Result<TextNode> {
        chunks_stored.fetch_add(1, Ordering::Relaxed);
        Ok(node)
    }
};

indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then_store_with(MemoryStorage::default())
    .then(count_chunk)
    .run()
    .await?;

let chunks_stored = chunks_stored.load(Ordering::Relaxed);
println!("Stored {chunks_stored} chunks");

Source: snippets/src/observability/metrics_and_statistics.rs#L15-L32

A closure from TextNode to anyhow::Result<TextNode> is a transformer. This one adds 1 to a shared counter and returns the node unchanged. Steps run nodes concurrently, so the counter is an atomic behind an Arc, and the closure holds its own clone of it.

The showcase prints these lines from its counters and the run’s stats. It indexed the swiftide-agents crate with claude-haiku-5-5 through an OpenAI-compatible gateway, and FastEmbed embeddings on the CPU:

Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s

On the second run the Redb node cache skipped every file:

Indexed 0 chunks from 0 new or changed files (14 files found, 14 unchanged) in 0.6s

The docs-qa showcase takes another route: after the run, it asks Qdrant how many points the collection holds. That counts the whole collection, not only this run. The number matches the run here because the captured run used --recreate, which drops the collection first. With claude-haiku-5-5 writing chunk titles and keywords, it printed:

Indexed 43 chunks into `docs-qa` in 71.7s

Reference

• Pipeline (stats, stats_collector, run)
• PipelineStats
• StatsCollector
• ModelUsage
• GenericOpenAIBuilder (metric_metadata)
• AnthropicBuilder
• AwsBedrockBuilder
• Feature flags
• metrics-rs

Next

Integrations overview