# Errors, backoff, usage and tokens

> Tell transient from permanent model errors, retry with LanguageModelWithBackOff, record token usage with on_usage, and estimate tokens with CharEstimator or tiktoken.

By the end of this page your OpenAI-compatible or `AwsBedrock` client retries rate limits on its own, reports the tokens each call used, and you can estimate the size of a prompt before you send it.

```rust
match openai.prompt("Summarise the attached log.".into()).await {
    Ok(answer) => println!("{answer}"),
    Err(LanguageModelError::ContextLengthExceeded(e)) => {
        eprintln!("prompt too long, shorten it: {e}");
    }
    Err(LanguageModelError::TransientError(e)) => {
        eprintln!("rate limit or outage, try again later: {e}");
    }
    Err(error @ LanguageModelError::PermanentError(_)) => return Err(error.into()),
}
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L20-L29`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L20-L29)

A failed model call returns a `LanguageModelError`. Each of its three variants wraps the underlying error:

- `ContextLengthExceeded`: the input is longer than the model accepts. Shorten it before you send it again.
- `TransientError`: the call may succeed if you try again later.
- `PermanentError`: the call fails the same way every time.

The OpenAI-compatible clients sort provider errors into these variants for you:

- The identifier `context_length_exceeded` becomes `ContextLengthExceeded`.
- Network errors and Hypertext Transfer Protocol (HTTP) status 408, 429 and 5xx are transient.
- OpenRouter's rate limit, overload, unavailable, server and timeout error types are transient.
- Quota and billing errors are permanent, even when they arrive with status 429.
- An error inside a successful response is read and sorted the same way.
- A response that fails to parse and holds no such error is transient.
- Invalid arguments and broken streams are permanent.

`AwsBedrock` treats throttling, timeouts, unavailable services and network errors as transient. `Anthropic` treats only network errors on `prompt` as transient. Every other `Anthropic` error is permanent, including status 429.

To build a variant yourself, call `LanguageModelError::transient`, `permanent` or `context_length_exceeded`. An `anyhow::Error` converts into a `PermanentError`.

To return a variant with `?` or `.into()`, bind the whole `LanguageModelError` (`error @ ...`). The `Box<dyn Error + Send + Sync>` inside a variant does not convert into `anyhow::Error`.

### Errors from a pipeline

When a step of an indexing pipeline fails on a model call, `run` returns an `anyhow::Error` with the `LanguageModelError` still inside. So you can tell a rate limit from a broken request.

```rust
let result = indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then(MetadataQAText::new(openai.clone()))
    .then_in_batch(Embed::new(openai))
    .then_store_with(MemoryStorage::default())
    .run()
    .await;

if let Err(error) = result {
    let model_error = error
        .chain()
        .find_map(|cause| cause.downcast_ref::<LanguageModelError>());
    match model_error {
        Some(LanguageModelError::TransientError(_)) => {
            eprintln!("the provider is overloaded, run the pipeline again later: {error}");
        }
        _ => return Err(error),
    }
}
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L37-L55`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L37-L55)

`downcast_ref::<LanguageModelError>()` on the error works too. Searching `chain()` also finds the model error when a step added context around it.

## Backoff

```rust
let openai = LanguageModelWithBackOff::new(
    openai,
    BackoffConfiguration {
        initial_interval_sec: 2,
        multiplier: 2.0,
        randomization_factor: 0.5,
        max_elapsed_time_sec: 120,
    },
);

let answer = openai.prompt("Name one Rust web framework.".into()).await?;
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L63-L73`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L63-L73)

`LanguageModelWithBackOff` wraps a client and retries calls that fail with a `TransientError`. On `prompt` and `complete`, a `ContextLengthExceeded` or `PermanentError` returns at once. The decorated client implements `SimplePrompt`, `ChatCompletion` and `EmbeddingModel` when the inner client does. You can pass it where one of those traits is expected, including `Agent::builder().llm(&openai)`.

It never retries an `Anthropic` rate limit, because `Anthropic` marks it permanent.

`BackoffConfiguration` has four fields:

- `initial_interval_sec`: the wait before the first retry, in seconds. The default is 1.
- `multiplier`: the factor the wait grows by after each attempt. The default is 2.0.
- `randomization_factor`: random jitter on each wait. The default is 0.5, plus or minus half the interval.
- `max_elapsed_time_sec`: the time budget in seconds, including time spent inside calls. The decorator returns the last error as soon as the next wait would pass it, often before the full budget. The default is 60.

`Default::default()` gives you these defaults.

Every failed attempt that the decorator retries logs a WARN event, `Language model request failed, retrying`, with the attempt number, the wait and the error with its causes. When the decorator gives up, it logs the final error once at ERROR, also for a permanent error that it never retried. Only `prompt` and `complete` log these events.

For `complete_stream` the decorator does not send the request again. Your stream still receives every error. After each one, transient or permanent, the decorator waits for the next interval before it reads on. Once `max_elapsed_time_sec` is used up, the stream ends after the last error. The budget restarts after every successful item. OpenAI-compatible streams end after their first error, so the wait only delays that end.

Two kinds of call get no retry. `EmbeddingModel` calls the inner client once. `StructuredPrompt` and methods of the client itself, such as `estimate_tokens`, do not exist on the decorator, so call them on the inner client.

## Usage callbacks

```rust
let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .on_usage(|usage| {
        println!(
            "{} tokens in, {} out",
            usage.prompt_tokens, usage.completion_tokens
        );
        Ok(())
    })
    .build()?;
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L82-L91`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L82-L91)

A usage callback receives the `Usage` of each call: `prompt_tokens`, `completion_tokens` and `total_tokens`. Set it with `on_usage` on the builder of an OpenAI-compatible client (`openai` feature flag), `Anthropic` (`anthropic`) or `AwsBedrock` (`aws-bedrock`). On OpenAI-compatible clients, embedding calls report usage too.

Use `on_usage_async` when the callback needs to await:

```rust
let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .on_usage_async(|usage| {
        Box::pin(async move {
            // Write to a database or a metrics service here.
            println!("{} tokens", usage.total_tokens);
            Ok(())
        })
    })
    .build()?;
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L99-L108`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L99-L108)

The closure gets a borrowed `Usage` and returns a boxed future that may borrow it. Copy the numbers out first only if you hand them to a task that must be `'static`, such as one you spawn.

The providers run the callback differently:

- OpenAI-compatible clients spawn it. An `Err` is logged and the call still succeeds.
- `AwsBedrock`, and `Anthropic` on calls that do not stream, await it. An `Err` from the callback fails the call.
- `Anthropic` streaming calls spawn it and only log an `Err`.

With the `metrics` feature flag, clients also emit usage counters. See [Metrics and statistics](/observability/metrics-and-statistics/).

## Normalized usage

```rust
let request = ChatCompletionRequest::from(vec![ChatMessage::new_user("Hello")]);
let response = openai.complete(&request).await?;

if let Some(usage) = response.usage.as_ref().map(Usage::normalized) {
    let cached = usage
        .details
        .as_ref()
        .and_then(|details| details.input.cached_tokens)
        .unwrap_or(0);
    println!("{} tokens, {cached} from cache", usage.total_tokens);
}
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L116-L126`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L116-L126)

`Usage` has an optional `details` field with the provider's own breakdown. `Usage::normalized` returns a `NormalizedUsage` with one shape for every provider: cached and audio tokens on the input side, and reasoning, audio and prediction tokens on the output side. Its `details` is `None` when the provider reported none of them.

OpenAI fills the details from Responses application programming interface (API) calls and from Chat Completions calls that do not stream. `AwsBedrock` reports cached input tokens. Streamed Chat Completions, embeddings and `Anthropic` report the totals only.

## Token estimation

```rust
let history = vec![
    ChatMessage::new_system("You are a helpful assistant."),
    ChatMessage::new_user("What is a borrow checker?"),
];

let rough = CharEstimator.estimate(history.as_slice()).await?;
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L134-L139`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L134-L139)

`CharEstimator` needs no feature flag. It divides the number of characters by 4, then adds 4 tokens per message and 3 per list of messages.

`estimate` comes from the `EstimateTokens` trait. It accepts any `Estimatable`: a `&str`, a `String`, a `&Prompt` (which it renders first), a `&ChatMessage` or a `&[ChatMessage]`. For a `Vec<ChatMessage>`, pass a slice.

For estimates closer to what OpenAI models see, enable the `tiktoken` feature flag:

```sh
cargo add swiftide@0.34 --features openai,tiktoken
```

```rust
let tokenizer = TikToken::try_from_model("gpt-4o")?;
let prompt = Prompt::from("Explain lifetimes in one paragraph.");
let from_tokenizer = tokenizer.estimate(&prompt).await?;

// The client's tokenizer is gpt-4o unless you set `tiktoken` on its builder.
let from_client = openai.estimate_tokens(&prompt).await?;
```

Source: [`snippets/src/models/errors_backoff_and_usage.rs#L147-L152`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/models/errors_backoff_and_usage.rs#L147-L152)

`TikToken` encodes the text with the Byte Pair Encoding (BPE) tokenizer of the model you name in `try_from_model`. `TikToken::default()` uses the `gpt-4o` tokenizer. With the feature on, an OpenAI client also has `estimate_tokens`, which uses the client's `TikToken`. That is `gpt-4o` whatever the prompt model, unless you set `tiktoken` on the builder.

Both return estimates, not exact counts. They skip what they cannot read as text: image, document, audio and video parts of a message, and the text of reasoning messages. Each message still adds its 4 tokens, so an image-only or reasoning message counts as 4.

## Reference

- [`LanguageModelError`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/errors/enum.LanguageModelError.html)
- [`SimplePrompt`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.SimplePrompt.html)
- [`ChatCompletion`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChatCompletion.html)
- [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html)
- [`StructuredPrompt`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.StructuredPrompt.html)
- [`AgentBuilder`](https://docs.rs/swiftide/0.34/swiftide/agents/struct.AgentBuilder.html) (`llm`)
- [`LanguageModelWithBackOff`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.LanguageModelWithBackOff.html)
- [`BackoffConfiguration`](https://docs.rs/swiftide/0.34/swiftide/indexing/struct.BackoffConfiguration.html)
- [`GenericOpenAIBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.GenericOpenAIBuilder.html) (`on_usage`, `on_usage_async`)
- [`AnthropicBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/anthropic/struct.AnthropicBuilder.html) (`on_usage`, `on_usage_async`)
- [`AwsBedrockBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/aws_bedrock_v2/struct.AwsBedrockBuilder.html) (`on_usage`, `on_usage_async`)
- [`Usage`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.Usage.html)
- [`NormalizedUsage`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.NormalizedUsage.html)
- [`CharEstimator`](https://docs.rs/swiftide/0.34/swiftide/struct.CharEstimator.html)
- [`EstimateTokens`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EstimateTokens.html)
- [`Estimatable`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.Estimatable.html)
- [`ChatMessage`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/enum.ChatMessage.html)
- [`Prompt`](https://docs.rs/swiftide/0.34/swiftide/prompt/struct.Prompt.html)
- [`TikToken`](https://docs.rs/swiftide/0.34/swiftide/integrations/tiktoken/struct.TikToken.html)

## Next

[Tracing](/observability/tracing/)