Errors, backoff, usage and tokens

By the end of this page your OpenAI-compatible or AwsBedrock client retries rate limits on its own, reports the tokens each call used, and you can estimate the size of a prompt before you send it.

match openai.prompt("Summarise the attached log.".into()).await {
    Ok(answer) => println!("{answer}"),
    Err(LanguageModelError::ContextLengthExceeded(e)) => {
        eprintln!("prompt too long, shorten it: {e}");
    }
    Err(LanguageModelError::TransientError(e)) => {
        eprintln!("rate limit or outage, try again later: {e}");
    }
    Err(error @ LanguageModelError::PermanentError(_)) => return Err(error.into()),
}

Source: snippets/src/models/errors_backoff_and_usage.rs#L20-L29

A failed model call returns a LanguageModelError. Each of its three variants wraps the underlying error:

ContextLengthExceeded: the input is longer than the model accepts. Shorten it before you send it again.
TransientError: the call may succeed if you try again later.
PermanentError: the call fails the same way every time.

The OpenAI-compatible clients sort provider errors into these variants for you:

The identifier context_length_exceeded becomes ContextLengthExceeded.
Network errors and Hypertext Transfer Protocol (HTTP) status 408, 429 and 5xx are transient.
OpenRouter’s rate limit, overload, unavailable, server and timeout error types are transient.
Quota and billing errors are permanent, even when they arrive with status 429.
An error inside a successful response is read and sorted the same way.
A response that fails to parse and holds no such error is transient.
Invalid arguments and broken streams are permanent.

AwsBedrock treats throttling, timeouts, unavailable services and network errors as transient. Anthropic treats only network errors on prompt as transient. Every other Anthropic error is permanent, including status 429.

To build a variant yourself, call LanguageModelError::transient, permanent or context_length_exceeded. An anyhow::Error converts into a PermanentError.

To return a variant with ? or .into(), bind the whole LanguageModelError (error @ ...). The Box<dyn Error + Send + Sync> inside a variant does not convert into anyhow::Error.

Errors from a pipeline

When a step of an indexing pipeline fails on a model call, run returns an anyhow::Error with the LanguageModelError still inside. So you can tell a rate limit from a broken request.

let result = indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then(MetadataQAText::new(openai.clone()))
    .then_in_batch(Embed::new(openai))
    .then_store_with(MemoryStorage::default())
    .run()
    .await;

if let Err(error) = result {
    let model_error = error
        .chain()
        .find_map(|cause| cause.downcast_ref::<LanguageModelError>());
    match model_error {
        Some(LanguageModelError::TransientError(_)) => {
            eprintln!("the provider is overloaded, run the pipeline again later: {error}");
        }
        _ => return Err(error),
    }
}

Source: snippets/src/models/errors_backoff_and_usage.rs#L37-L55

downcast_ref::<LanguageModelError>() on the error works too. Searching chain() also finds the model error when a step added context around it.

Backoff

let openai = LanguageModelWithBackOff::new(
    openai,
    BackoffConfiguration {
        initial_interval_sec: 2,
        multiplier: 2.0,
        randomization_factor: 0.5,
        max_elapsed_time_sec: 120,
    },
);

let answer = openai.prompt("Name one Rust web framework.".into()).await?;

Source: snippets/src/models/errors_backoff_and_usage.rs#L63-L73

LanguageModelWithBackOff wraps a client and retries calls that fail with a TransientError. On prompt and complete, a ContextLengthExceeded or PermanentError returns at once. The decorated client implements SimplePrompt, ChatCompletion and EmbeddingModel when the inner client does. You can pass it where one of those traits is expected, including Agent::builder().llm(&openai).

It never retries an Anthropic rate limit, because Anthropic marks it permanent.

BackoffConfiguration has four fields:

initial_interval_sec: the wait before the first retry, in seconds. The default is 1.
multiplier: the factor the wait grows by after each attempt. The default is 2.0.
randomization_factor: random jitter on each wait. The default is 0.5, plus or minus half the interval.
max_elapsed_time_sec: the time budget in seconds, including time spent inside calls. The decorator returns the last error as soon as the next wait would pass it, often before the full budget. The default is 60.

Default::default() gives you these defaults.

Every failed attempt that the decorator retries logs a WARN event, Language model request failed, retrying, with the attempt number, the wait and the error with its causes. When the decorator gives up, it logs the final error once at ERROR, also for a permanent error that it never retried. Only prompt and complete log these events.

For complete_stream the decorator does not send the request again. Your stream still receives every error. After each one, transient or permanent, the decorator waits for the next interval before it reads on. Once max_elapsed_time_sec is used up, the stream ends after the last error. The budget restarts after every successful item. OpenAI-compatible streams end after their first error, so the wait only delays that end.

Two kinds of call get no retry. EmbeddingModel calls the inner client once. StructuredPrompt and methods of the client itself, such as estimate_tokens, do not exist on the decorator, so call them on the inner client.

Usage callbacks

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .on_usage(|usage| {
        println!(
            "{} tokens in, {} out",
            usage.prompt_tokens, usage.completion_tokens
        );
        Ok(())
    })
    .build()?;

Source: snippets/src/models/errors_backoff_and_usage.rs#L82-L91

A usage callback receives the Usage of each call: prompt_tokens, completion_tokens and total_tokens. Set it with on_usage on the builder of an OpenAI-compatible client (openai feature flag), Anthropic (anthropic) or AwsBedrock (aws-bedrock). On OpenAI-compatible clients, embedding calls report usage too.

Use on_usage_async when the callback needs to await:

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .on_usage_async(|usage| {
        Box::pin(async move {
            // Write to a database or a metrics service here.
            println!("{} tokens", usage.total_tokens);
            Ok(())
        })
    })
    .build()?;

Source: snippets/src/models/errors_backoff_and_usage.rs#L99-L108

The closure gets a borrowed Usage and returns a boxed future that may borrow it. Copy the numbers out first only if you hand them to a task that must be 'static, such as one you spawn.

The providers run the callback differently:

OpenAI-compatible clients spawn it. An Err is logged and the call still succeeds.
AwsBedrock, and Anthropic on calls that do not stream, await it. An Err from the callback fails the call.
Anthropic streaming calls spawn it and only log an Err.

With the metrics feature flag, clients also emit usage counters. See Metrics and statistics.

Normalized usage

let request = ChatCompletionRequest::from(vec![ChatMessage::new_user("Hello")]);
let response = openai.complete(&request).await?;

if let Some(usage) = response.usage.as_ref().map(Usage::normalized) {
    let cached = usage
        .details
        .as_ref()
        .and_then(|details| details.input.cached_tokens)
        .unwrap_or(0);
    println!("{} tokens, {cached} from cache", usage.total_tokens);
}

Source: snippets/src/models/errors_backoff_and_usage.rs#L116-L126

Usage has an optional details field with the provider’s own breakdown. Usage::normalized returns a NormalizedUsage with one shape for every provider: cached and audio tokens on the input side, and reasoning, audio and prediction tokens on the output side. Its details is None when the provider reported none of them.

OpenAI fills the details from Responses application programming interface (API) calls and from Chat Completions calls that do not stream. AwsBedrock reports cached input tokens. Streamed Chat Completions, embeddings and Anthropic report the totals only.

Token estimation

let history = vec![
    ChatMessage::new_system("You are a helpful assistant."),
    ChatMessage::new_user("What is a borrow checker?"),
];

let rough = CharEstimator.estimate(history.as_slice()).await?;

Source: snippets/src/models/errors_backoff_and_usage.rs#L134-L139

CharEstimator needs no feature flag. It divides the number of characters by 4, then adds 4 tokens per message and 3 per list of messages.

estimate comes from the EstimateTokens trait. It accepts any Estimatable: a &str, a String, a &Prompt (which it renders first), a &ChatMessage or a &[ChatMessage]. For a Vec<ChatMessage>, pass a slice.

For estimates closer to what OpenAI models see, enable the tiktoken feature flag:

cargo add swiftide@0.34 --features openai,tiktoken

let tokenizer = TikToken::try_from_model("gpt-4o")?;
let prompt = Prompt::from("Explain lifetimes in one paragraph.");
let from_tokenizer = tokenizer.estimate(&prompt).await?;

// The client's tokenizer is gpt-4o unless you set `tiktoken` on its builder.
let from_client = openai.estimate_tokens(&prompt).await?;

Source: snippets/src/models/errors_backoff_and_usage.rs#L147-L152

TikToken encodes the text with the Byte Pair Encoding (BPE) tokenizer of the model you name in try_from_model. TikToken::default() uses the gpt-4o tokenizer. With the feature on, an OpenAI client also has estimate_tokens, which uses the client’s TikToken. That is gpt-4o whatever the prompt model, unless you set tiktoken on the builder.

Both return estimates, not exact counts. They skip what they cannot read as text: image, document, audio and video parts of a message, and the text of reasoning messages. Each message still adds its 4 tokens, so an image-only or reasoning message counts as 4.

Reference

LanguageModelError
SimplePrompt
ChatCompletion
EmbeddingModel
StructuredPrompt
AgentBuilder (llm)
LanguageModelWithBackOff
BackoffConfiguration
GenericOpenAIBuilder (on_usage, on_usage_async)
AnthropicBuilder (on_usage, on_usage_async)
AwsBedrockBuilder (on_usage, on_usage_async)
Usage
NormalizedUsage
CharEstimator
EstimateTokens
Estimatable
ChatMessage
Prompt
TikToken

Tracing