OpenAI and Azure

Here you build an OpenAI client, or the same client for an Azure OpenAI deployment, and pass it to indexing pipelines, query pipelines and agents. The integration sits behind the openai feature flag:

cargo add swiftide@0.34 --features openai

Client

let openai = swiftide::integrations::openai::OpenAI::builder()
    .default_prompt_model("gpt-4o-mini")
    .default_embed_model("text-embedding-3-small")
    .build()?;

Source: examples/agent_can_fail_custom_schema.rs#L64-L67

default_prompt_model sets the model for prompt, structured_prompt, complete and complete_stream. default_embed_model sets the model for embed. The client has no default model. A call without one fails with a permanent error that says Model not set.

Unless you call api_base, api_key or client(..), the builder creates an async_openai::Client with OpenAIConfig::default(). That configuration reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment, and OPENAI_ORG_ID and OPENAI_PROJECT_ID when they are set. The base URL defaults to https://api.openai.com/v1.

OpenAI implements SimplePrompt, ChatCompletion, EmbeddingModel and StructuredPrompt. So one client serves a large language model (LLM) transformer, the Embed batch transformer, an answer generator and Agent::builder().llm(&openai). Agents also need the swiftide-agents feature flag. Cloning the client is cheap: the clones share one HTTP client.

Custom base URL

impl LlmEndpoint {
    /// Builds a client for this endpoint.
    ///
    /// Rate limits and transient errors are retried with exponential backoff, which matters when
    /// indexing fires many prompts concurrently. After `retry_budget` the error is returned, so
    /// an unreachable or broken endpoint fails instead of hanging.
    pub fn client(&self, retry_budget: Duration) -> Result<Arc<dyn SimplePrompt>> {
        let openai = OpenAI::builder()
            .api_base(&self.base_url)
            .api_key(&self.api_key)
            .default_prompt_model(&self.model)
            .use_responses_api(matches!(self.api, OpenAiApi::Responses))
            .build()?;

        let backoff = BackoffConfiguration {
            max_elapsed_time_sec: retry_budget.as_secs(),
            ..BackoffConfiguration::default()
        };

        Ok(Arc::new(LanguageModelWithBackOff::new(openai, backoff)))
    }
}

Source: examples/ask-your-codebase/src/llm.rs#L79-L100

The ask-your-codebase showcase sends its prompts to any OpenAI-compatible endpoint. LlmEndpoint holds the base URL, key and model from the command line. api_base and api_key on the builder override the base URL and key from the environment. Everything else still comes from OpenAIConfig::default(), so OPENAI_ORG_ID and OPENAI_PROJECT_ID are sent as headers when they are set, also to a third-party gateway. The showcase picks the Responses API or Chat Completions from a command-line flag.

The showcase wraps the client in LanguageModelWithBackOff. Transient errors, such as rate limits and outages, are retried with exponential backoff until max_elapsed_time_sec runs out. Permanent errors, including quota and billing errors that arrive as HTTP 429, return at once. LlmEndpoint::client returns Arc<dyn SimplePrompt>, so the rest of the showcase is provider-independent. Errors, backoff, usage and tokens covers the settings.

For anything the two setters do not cover, such as a custom HTTP client or another configuration type, pass a whole async_openai::Client to client(..). It replaces what api_base and api_key set, and whichever you call last wins. Swiftide re-exports OpenAIConfig, AzureConfig and ReasoningEffort from async-openai at swiftide::integrations::openai. It does not re-export async_openai::Client, so add the crate at the version Swiftide uses:

cargo add async-openai@0.41 --no-default-features

Azure OpenAI

Azure OpenAI uses its own URL scheme and authentication header, so it needs AzureConfig instead of OpenAIConfig:

let config = AzureConfig::new()
    .with_api_base("https://my-resource.openai.azure.com")
    .with_api_version("2024-10-21")
    .with_deployment_id("gpt-4o-mini")
    .with_api_key(std::env::var("AZURE_OPENAI_API_KEY")?);

let azure: GenericOpenAI<AzureConfig> = GenericOpenAIBuilder::default()
    .client(async_openai::Client::with_config(config))
    .default_prompt_model("gpt-4o-mini")
    .build()?;

let answer = azure.prompt("Name one Rust web framework.".into()).await?;

Source: snippets/src/integrations/openai.rs#L17-L28

OpenAI is a type alias for GenericOpenAI<OpenAIConfig>. For Azure you build a GenericOpenAI<AzureConfig> through GenericOpenAIBuilder and pass the client with client(..). api_base and api_key exist only on the OpenAI builder. The Azure client implements the same model traits.

AzureConfig::with_api_base takes the resource URL without a path. Requests go to {api_base}/openai/deployments/{deployment_id}/..., with api-version as a query parameter and the key in an api-key header. Without with_api_key, AzureConfig falls back to OPENAI_API_KEY.

A deployment serves one model, so build one client per deployment, for example one for chat and one for embeddings. Set default_prompt_model or default_embed_model anyway: Swiftide refuses to send a request without a model, even though Azure routes by deployment.

GenericOpenAI<AzureConfig> is a different type from OpenAI. A field typed OpenAI does not accept it. Store the client as Arc<dyn SimplePrompt> or another trait object when the provider varies. See Clients as trait objects.

Embeddings

let openai = OpenAI::builder()
    .default_embed_model("text-embedding-3-small")
    .default_options(Options::builder().dimensions(512))
    .build()?;

let vectors = openai
    .embed(vec!["What is a borrow checker?".into()])
    .await?;
println!("{} dimensions", vectors[0].len());

indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then_in_batch(Embed::new(openai.clone()).with_batch_size(64))
    .then_store_with(MemoryStorage::default())
    .run()
    .await?;

Source: snippets/src/integrations/openai.rs#L37-L52

embed sends a list of texts in one request and returns one vector per text. Embedding requests carry two settings from Options: dimensions and user. dimensions works with text-embedding-3 and later models. The vector size you get must match the vector size of your storage.

In an indexing pipeline, the Embed batch transformer calls embed once per batch. with_batch_size(64) sets the batch size. Without it, the pipeline default of 256 applies. Use the same model and dimensions for the Embed query transformer, so queries and chunks land in the same vector space.

Request options

Request settings covers default_options and which call sends which field. Two details are specific to this client.

parallel_tool_calls is unset by default, although the builder method’s documentation says it defaults to true. Once you set it, the client sends it on every chat request, including prompt, structured_prompt and complete calls without tools. OpenAI rejects parallel_tool_calls on a request without tools. Give your agents a separate client when you set it, and keep it off the client that serves transformers and answer generators.

extra_body merges extra fields into the top level of the request body. Only complete and complete_stream send it, on the Chat Completions API and the Responses API alike. prompt, structured_prompt and embed do not.

Responses API

let openai = OpenAI::builder()
    .default_prompt_model("gpt-4.1-mini")
    .default_options(Options::builder().temperature(0.2))
    .use_responses_api(true)
    .build()?;

Source: examples/responses_api.rs#L49-L53

use_responses_api(true) sends complete, complete_stream, prompt and structured_prompt to the OpenAI Responses API instead of the Chat Completions API. embed is unaffected. The flag is off by default.

With reasoning_effort set, the client asks for reasoning summaries and encrypted reasoning content, and sends the encrypted content back on the next turn. Summaries are not sent back. reasoning_features(false) turns that off. The Responses API ignores seed and presence_penalty and logs a warning. Responses API and reasoning shows an agent that keeps its reasoning in the message history.

Streaming and usage

stream_full(false) makes complete_stream send only the delta in every item but the last. See Deltas only.

on_usage and on_usage_async report the token Usage of every request, embeddings included. The on_usage_async callback returns a pinned, boxed future. See Errors, backoff, usage and tokens.

Gotchas

Some OpenAI-compatible gateways cap unset max_completion_tokens at 4096 and truncate silently; see OpenAI-compatible providers.
A gateway’s Responses API has to match OpenAI’s response schema exactly. The captured docs-qa run uses Chat Completions, its default, because its gateway returns "input_tokens_details": null, which async-openai rejects.

Reference

OpenAI-compatible providers