Models

By the end of this page you have a client for your model provider, and you know which trait each part of Swiftide expects from it.

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .default_embed_model("text-embedding-3-small")
    .build()?;

let anthropic = Anthropic::builder()
    .default_prompt_model("claude-sonnet-4-5")
    .build()?;

let ollama = Ollama::builder()
    .default_prompt_model("llama3.2")
    .default_embed_model("nomic-embed-text")
    .build()?;

Source: snippets/src/models/overview.rs#L19-L31

Every provider has a client, and every client has a builder. default_prompt_model sets the model for prompts and chat completions. default_embed_model sets the model for embeddings. OpenAI reads its key from OPENAI_API_KEY. Anthropic reads ANTHROPIC_API_KEY. Ollama runs on your machine and needs no key.

OpenAI and the other OpenAI-compatible clients have no default model. A call without one fails with a permanent error that says Model not set. Anthropic falls back to claude-3-5-sonnet-20241022, a retired model, so always set default_prompt_model.

Each provider sits behind its own feature flag. Turn on the ones you use:

cargo add swiftide@0.34 --features openai,anthropic,ollama

Model traits

Swiftide never calls a provider by name. Each part of it asks for a trait, and you pass any client that implements that trait. The traits live in swiftide::traits. Import a trait to call its method on a client.

Trait	Method	Used by
`SimplePrompt`	`prompt`: one prompt in, text out	Large Language Model (LLM) transformers such as `MetadataQAText`, the `GenerateSubquestions` query transformer, the `answers::Simple` answer generator, and `Pipeline::with_default_llm_client`
`ChatCompletion`	`complete`: messages and tools in, a response out	agents, through `Agent::builder().llm(..)`, and task nodes in `swiftide::tasks`
`EmbeddingModel`	`embed`: a list of texts in, dense vectors out	the `Embed` batch transformer and the `Embed` query transformer
`SparseEmbeddingModel`	`sparse_embed`: a list of texts in, sparse vectors out	the `SparseEmbed` batch transformer and the `SparseEmbed` query transformer
`StructuredPrompt`	`structured_prompt::<T>`: one prompt in, a Rust value out	your own code, for typed responses

Failed calls return a LanguageModelError. complete_stream is the exception: it returns a stream, and errors arrive as items of that stream. See Streaming.

Providers

Client (feature flag)	`SimplePrompt`	`ChatCompletion`	`EmbeddingModel`	`SparseEmbeddingModel`	`StructuredPrompt`
`OpenAI` (`openai`)	yes	yes	yes	no	yes
`Ollama` (`ollama`), `Groq` (`groq`), `Gemini` (`gemini`), `OpenRouter` (`open-router`), `Dashscope` (`dashscope`)	yes	yes	yes	no	yes
`Anthropic` (`anthropic`)	yes	yes	no	no	no
`AwsBedrock` (`aws-bedrock`)	yes	yes	no	no	yes
`FastEmbed` (`fastembed`)	no	no	yes	yes	no

OpenAI is GenericOpenAI<OpenAIConfig>. Ollama, Groq, Gemini, OpenRouter and Dashscope are GenericOpenAI with their own configuration type. So the aliases implement every trait OpenAI does, and you can pass them anywhere a model trait is expected. They are still different types. A field typed OpenAI does not accept an Ollama. The endpoint and the model decide what works at runtime. Not every provider serves embeddings or structured output.

AwsBedrock talks to Amazon Web Services (AWS) Bedrock through the Converse application programming interface (API), in the aws_bedrock_v2 module. FastEmbed runs embedding models locally. Setup for both is on AWS Bedrock and FastEmbed.

Clients as trait objects

let prompt: Arc<dyn SimplePrompt> = Arc::new(openai.clone());
let embed: Arc<dyn EmbeddingModel> = Arc::new(openai.clone());

let answer = prompt.prompt("Name one Rust web framework.".into()).await?;
let vectors = embed.embed(vec![answer]).await?;

let agent = Agent::builder().llm(&openai).build()?;

Source: snippets/src/models/overview.rs#L40-L46

Store a client as Arc<dyn SimplePrompt> or Arc<dyn EmbeddingModel> when you pick the provider at runtime. Cloning a client is cheap. Every built-in client keeps its connection or model behind an Arc, so clones share it.

Agent::builder().llm(&openai) takes a reference. It clones the client into a Box<dyn ChatCompletion>, so you can keep using openai afterwards. Agent needs the swiftide-agents feature flag on top of the provider flags.

The structured_prompt::<T> method is generic, so StructuredPrompt cannot be a trait object. Structured output covers DynStructuredPrompt, the variant that can.

To retry rate limits and server errors, wrap the client in LanguageModelWithBackOff. See Errors, backoff, usage and tokens.

OpenAI-compatible endpoints

A server that speaks the OpenAI API works with the OpenAI client. Point the client at it with api_base and api_key on the builder.

let local = OpenAI::builder()
    .api_base("http://localhost:8000/v1")
    .api_key("not-needed-locally")
    .default_prompt_model("qwen2.5-coder")
    .build()?;

Source: snippets/src/models/overview.rs#L55-L59

With a single endpoint you can skip the code. The default OpenAI client reads OPENAI_BASE_URL and OPENAI_API_KEY from the environment when you build it.

For a custom HTTP (Hypertext Transfer Protocol) client or another configuration type, pass an async_openai::Client to client(..) instead. Azure OpenAI needs that route with GenericOpenAI<AzureConfig>; Azure OpenAI shows it.

The hosted-provider clients have the base URL already set and read their key from the environment:

Client	Key variable
`Ollama`	none
`Groq`	`GROQ_API_KEY`
`Gemini`	`GEMINI_API_KEY`
`OpenRouter`	`OPENROUTER_API_KEY`
`Dashscope`	`QWEN_API_KEY`, then `DASHSCOPE_API_KEY`

OpenRouter without a key does not return an error. It panics on the first request. More on these clients is on OpenAI-compatible providers.

Request settings

Each OpenAI-compatible client holds one Options value. Every request it sends carries those settings:

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .default_embed_model("text-embedding-3-small")
    .default_options(
        Options::builder()
            .max_completion_tokens(2_048)
            .reasoning_effort(ReasoningEffort::Low)
            .seed(42)
            .metadata(serde_json::json!({ "team": "docs" }))
            .dimensions(512),
    )
    .parallel_tool_calls(Some(false))
    .for_end_user("user-1234")
    .build()?;

Source: snippets/src/models/overview.rs#L106-L119

default_options merges into the options the builder already has: a field you set replaces the earlier value. for_end_user and parallel_tool_calls are builder shortcuts for the user and parallel_tool_calls fields.

Apart from reasoning_features, every field starts unset, and Swiftide only sends the fields you set. That includes parallel_tool_calls: set it to Some(false) to stop the model from asking for several tools in one turn. Some reasoning models reject the field, so leave it at None for them.

Field	Sent with
`temperature`, `max_completion_tokens`, `seed`, `presence_penalty`, `metadata`, `reasoning_effort`	`complete`, `prompt` and `structured_prompt`
`user`	`complete`, `prompt`, `structured_prompt` and `embed`; not sent on the Responses API
`dimensions`	`embed` only; text-embedding-3 and later models
`reasoning_features`	the Responses API only, and only when `reasoning_effort` is set

ReasoningEffort is re-exported from async-openai at swiftide::integrations::openai. The Responses API (use_responses_api(true)) ignores seed and presence_penalty and logs a warning. It accepts metadata only as a flat map of string values. reasoning_features defaults to true on OpenAI. It asks for reasoning summaries and encrypted reasoning content. Turn it off with reasoning_features(false).

To change a built client, clone it and edit the copy through options_mut. Each clone has its own Options:

let mut creative = openai.clone();
creative.options_mut().temperature = Some(1.2);
creative.options_mut().presence_penalty = Some(0.5);

Source: snippets/src/models/overview.rs#L128-L130

Extra request fields

Some providers accept request fields that the OpenAI API does not have. Options has extra_body, a JavaScript Object Notation (JSON) map that Swiftide merges into the top level of each request:

let mut extra = serde_json::Map::new();
extra.insert("top_k".into(), serde_json::json!(20));

let ollama = Ollama::builder()
    .default_prompt_model("llama3.2")
    .default_options(Options::builder().extra_body(extra))
    .build()?;

Source: snippets/src/models/overview.rs#L68-L74

Only ChatCompletion requests carry extra_body. prompt, structured_prompt and embed calls do not. So LLM transformers such as MetadataQAText and the answers::Simple answer generator send the request without the extra fields.

AWS Bedrock settings

AwsBedrock has its own Options in swiftide::integrations::aws_bedrock_v2. Set them with AwsBedrockBuilder::default_options, which merges the same way, or change them later with options_mut.

Field	Effect
`max_tokens`, `temperature`, `top_p`, `stop_sequences`	the Converse inference configuration
`tool_strict`	strict schema validation for tool calls; `true` when unset
`anthropic_beta`	Anthropic beta headers, sent in the model request fields
`reasoning_effort`	`output_config.effort` for Claude models: `Low`, `Medium`, `High` or `Max`
`additional_model_request_fields`	any other model request fields, as an `aws_smithy_types::Document`

reasoning_effort here is the Bedrock module’s own ReasoningEffort enum, not the OpenAI one. For a model ID that contains claude-opus-4-5, Swiftide adds the effort-2025-11-24 beta header itself. Behind an inference profile or Amazon Resource Name (ARN), add it to anthropic_beta yourself. With anthropic_beta or reasoning_effort set, additional_model_request_fields must be a JSON object, or the request fails with a permanent error.

Responses API and reasoning

OpenAI uses the Chat Completions API by default. use_responses_api(true) switches complete, complete_stream, prompt and structured_prompt to the OpenAI Responses API. For OpenAI, only that API returns the reasoning of a reasoning model:

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .use_responses_api(true)
    .default_options(Options::builder().reasoning_effort(ReasoningEffort::Medium))
    .build()?;

let mut agent = Agent::builder().llm(&openai).build()?;
agent
    .query("Which crate in this workspace owns the agent loop?")
    .await?;

for message in agent.history().await? {
    if let ChatMessage::Reasoning(item) = message {
        println!("reasoning summary: {:?}", item.summary);
    }
}

Source: snippets/src/models/overview.rs#L83-L98

With reasoning_effort set, reasoning arrives in the reasoning field of ChatCompletionResponse as ReasoningItem values. An agent adds each one to its message history as a ChatMessage::Reasoning, even when the response holds nothing else. On the next request the client sends the items back, so the model continues from its earlier reasoning. These requests set store to false, so the encrypted reasoning lives in your message history, not at OpenAI.

The client sends back only items with encrypted_content. With reasoning_features(false) there is none, and nothing goes back. OpenAI returns summaries only to organizations verified for reasoning access. The Responses API does not send extra_body.

AwsBedrock returns signed Claude reasoning as ReasoningItem values too, and replays them on the next turn. The Anthropic client leaves ChatMessage::Reasoning out of its requests.

Streaming

complete_stream sends a chat completion request and returns a ChatCompletionStream instead of one response. Each item is a Result<ChatCompletionResponse, LanguageModelError>. You read it with StreamExt::next from the futures-util crate. Swiftide does not re-export it:

cargo add futures-util@0.3

let request = ChatCompletionRequest::builder()
    .messages(vec![ChatMessage::new_user(
        "Explain borrowing in Rust in three sentences.",
    )])
    .build()?;

let mut stream: ChatCompletionStream = openai.complete_stream(&request).await;
let mut printed = 0;
let mut last = None;

while let Some(response) = stream.next().await {
    let response = response?;
    let text = response.message().unwrap_or_default();
    print!("{}", &text[printed..]);
    printed = text.len();
    last = Some(response);
}

if let Some(usage) = last.and_then(|response| response.usage) {
    println!("\n{} tokens", usage.total_tokens);
}

Source: snippets/src/models/streaming.rs#L13-L33

Every item is a whole ChatCompletionResponse, not a fragment. message holds all the text so far. tool_calls holds the tool calls rebuilt from the fragments so far. delta holds the latest change: delta.message_chunk is the new text, and delta.tool_calls_chunk maps each tool call index to its fragments so far. The snippet prints the part of message it has not printed yet, which works the same for every client.

The last item is the complete response. Token usage comes in at the end of the stream, so read usage from the last item.

A request that fails before the stream starts gives a stream with a single Err item. A client without its own streaming keeps the default complete_stream. It calls complete and returns a stream with that one result.

An agent that streams calls complete_stream for you. You read its chunks in the on_stream hook. See Streaming.

Deltas only

By default every item clones the full response, which grows with the answer. The OpenAI-compatible clients can send only the delta. Turn off stream_full on the builder:

let openai = OpenAI::builder()
    .default_prompt_model("gpt-5-mini")
    .stream_full(false)
    .build()?;

let mut stream = openai.complete_stream(&request).await;
let mut complete = None;

while let Some(response) = stream.next().await {
    let response = response?;
    if response.message.is_some() {
        complete = Some(response);
        continue;
    }
    if let Some(chunk) = response
        .delta
        .as_ref()
        .and_then(|d| d.message_chunk.as_deref())
    {
        print!("{chunk}");
    }
}

Source: snippets/src/models/streaming.rs#L41-L62

With stream_full(false), every item but the last carries only id and delta. message, tool_calls and usage are None. The last item is still the complete response, with message, tool_calls and usage set. The snippet uses message to tell the last item apart and prints delta.message_chunk for the others.

stream_full defaults to true. It is a setting of GenericOpenAI, so it exists on OpenAI, Ollama, Groq, Gemini, OpenRouter and Dashscope, on the Chat Completions API and the Responses API alike. Anthropic and AwsBedrock have no such setting and always send the accumulated response.

Reference

Prompts and templates