OpenAI-compatible providers

Swiftide talks to Ollama, Groq, Gemini, OpenRouter and Dashscope through the same OpenAI client, each with its own base Uniform Resource Locator (URL) and key. You end up with a working client for each, local embeddings through Ollama, and a client for any other gateway that speaks the OpenAI application programming interface (API).

Client	Feature flag	Base URL	Key variable
`Ollama`	`ollama`	`http://localhost:11434/v1`	none
`Groq`	`groq`	`https://api.groq.com/openai/v1`	`GROQ_API_KEY`
`Gemini`	`gemini`	`https://generativelanguage.googleapis.com/v1beta/openai`	`GEMINI_API_KEY`
`OpenRouter`	`open-router`	`https://openrouter.ai/api/v1`	`OPENROUTER_API_KEY`
`Dashscope`	`dashscope`	`https://dashscope.aliyuncs.com/compatible-mode/v1`	`QWEN_API_KEY`, then `DASHSCOPE_API_KEY`

Each of these feature flags also turns on openai:

cargo add swiftide@0.34 --features ollama

Ollama

let ollama_client = integrations::ollama::Ollama::default()
    .with_default_prompt_model("llama3.1")
    .to_owned();

let fastembed = integrations::fastembed::FastEmbed::try_default()?;
let memory_store = MemoryStorage::default();

Source: examples/index_ollama.rs#L30-L35

Every client on this page is GenericOpenAI with a provider-specific configuration type. It shares the Options and the model traits of OpenAI.

Ollama::default() builds a client without a model. with_default_prompt_model takes &mut self and returns &mut Self, so .to_owned() turns the result back into a client you own. The builder, Ollama::builder(), avoids that step.

The example uses Ollama only for the MetadataQAText transformer and embeds the chunks with FastEmbed, so it also needs the fastembed feature flag.

Local embeddings

Ollama also serves embeddings. Set an embed model and use the same client on both sides:

let ollama = Ollama::builder()
    .default_prompt_model("llama3.2")
    .default_embed_model("nomic-embed-text")
    .build()?;

// nomic-embed-text returns 768 dimensions.
let qdrant = Qdrant::builder()
    .vector_size(768)
    .collection_name("local-docs")
    .build()?;

indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then_in_batch(Embed::new(ollama.clone()).with_batch_size(32))
    .then_store_with(qdrant.clone())
    .run()
    .await?;

let answered = query::Pipeline::default()
    .then_transform_query(query_transformers::Embed::from_client(ollama.clone()))
    .then_retrieve(qdrant)
    .then_answer(answers::Simple::from_client(ollama))
    .query("How do I configure the loader?")
    .await?;

Source: snippets/src/integrations/openai_compatible.rs#L22-L45

The Embed batch transformer embeds the chunks and the Embed query transformer embeds the question. Both must use the same model, or the vectors do not match. The storage vector size must match the model’s output too. nomic-embed-text returns 768 dimensions.

The snippet stores the vectors in Qdrant, so it also needs the qdrant feature flag and a running Qdrant server. Qdrant covers that part.

Pull the models before you run it:

ollama pull nomic-embed-text
ollama pull llama3.2

The Ollama example above uses llama3.1. Pull that one to run it.

Remote Ollama

OllamaConfig is public, so you can point the client at another machine:

let mut config = OllamaConfig::default();
config.with_api_base("http://gpu-box:11434/v1");

let ollama = Ollama::builder()
    .client(async_openai::Client::with_config(config))
    .default_prompt_model("llama3.2")
    .build()?;

let answer = ollama.prompt("Name one Rust web framework.".into()).await?;

Source: snippets/src/integrations/openai_compatible.rs#L54-L62

OllamaConfig::with_api_base takes &mut self, so build the config in a mutable variable first. async_openai::Client is not re-exported. Add the crate at the version Swiftide uses:

cargo add async-openai@0.41 --no-default-features

OllamaConfig sends no Authorization header and reads no environment variable. For an Ollama server behind a proxy that checks a bearer token, use OpenAI with api_base and api_key instead. See Other gateways.

Groq

let groq_client = integrations::groq::Groq::builder()
    .default_prompt_model("llama3-8b-8192")
    .to_owned()
    .build()?;

let fastembed = integrations::fastembed::FastEmbed::try_default()?;
let memory_store = MemoryStorage::default();

Source: examples/index_groq.rs#L29-L35

Groq::builder() gives a builder with the Groq base URL and the key from GROQ_API_KEY. The .to_owned() in the middle clones the builder and changes nothing. Like the Ollama example, this one embeds with FastEmbed and needs the fastembed feature flag.

Groq has retired llama3-8b-8192, the model in the example. Pick a current model from Groq’s model list.

Gemini

let openai = swiftide::integrations::gemini::Gemini::builder()
    .default_embed_model("gemini-embedding-exp-03-07")
    .default_prompt_model("gemini-2.0-flash")
    .build()?;

Source: examples/hello_agents.rs#L109-L112

The variable is named openai, but its type is Gemini. Later in the example it goes to Agent::builder().llm(&openai), which works because Gemini implements ChatCompletion. The agent needs the swiftide-agents feature flag as well. The key comes from GEMINI_API_KEY.

The embed model in the example, gemini-embedding-exp-03-07, is experimental. Check Google’s model list for a current embedding model before you copy it.

OpenRouter

let openrouter = OpenRouter::builder()
    .default_prompt_model("anthropic/claude-sonnet-4.5")
    .build()?;

let config = OpenRouterConfig::builder()
    .api_key(std::env::var("TEAM_OPENROUTER_KEY")?)
    .site_url("https://example.com/my-app")
    .site_name("My App")
    .build()?;

let team_openrouter = OpenRouter::builder()
    .client(async_openai::Client::with_config(config))
    .default_prompt_model("openai/gpt-5-mini")
    .build()?;

Source: snippets/src/integrations/openai_compatible.rs#L71-L84

The first client reads OPENROUTER_API_KEY. The second takes the key from an OpenRouterConfig, built with OpenRouterConfig::builder(). That builder requires api_key, site_url and site_name. Leave one out and build() returns an error at runtime.

site_url sets the HTTP-Referer header, which OpenRouter uses for its app rankings. At this release the X-Title header also takes the site_url value, so site_name is required but not sent. OpenRouterConfig::default() leaves both unset. It then sends https://github.com/bosun-ai/swiftide as HTTP-Referer and Swiftide as X-Title.

Dashscope

let client = DashscopeBuilder::default()
    .default_embed_model("text-embedding-v2")
    .default_prompt_model("qwen-long")
    .build()?;
let tempdir = TempDir::new().unwrap();
let lancedb = LanceDB::builder()
    .uri(tempdir.child("lancedb").to_str().unwrap())
    .vector_size(1536)
    .with_vector(EmbeddedField::Combined)
    .with_metadata(metadata_qa_text::NAME)
    .table_name("swiftide_test")
    .build()
    .unwrap();

Source: examples/dashscope.rs#L22-L34

DashscopeBuilder::default() is the same as Dashscope::builder(). The client prompts with qwen-long and embeds with text-embedding-v2. The example stores the vectors in LanceDB with a vector size of 1536, so it also needs the lancedb feature flag. LanceDB covers that part.

Other gateways

The configuration types of Groq, Gemini and Dashscope are private. To use another key variable or another base URL with them, use OpenAI with api_base and api_key.

let groq = OpenAI::builder()
    .api_base("https://api.groq.com/openai/v1")
    .api_key(std::env::var("TEAM_GROQ_KEY")?)
    .default_prompt_model("llama-3.3-70b-versatile")
    .build()?;

Source: snippets/src/integrations/openai_compatible.rs#L93-L97

For vLLM, LiteLLM or a company gateway, see Custom base URL.

Gotchas

None of the clients has a default model. Set one with default_prompt_model or default_embed_model, or calls fail with Model not set.
A client reads its key when you build it. For Groq, Gemini and Dashscope, a missing key becomes an empty one, and the provider rejects the first request. OpenRouter panics on the first request instead.
Each alias is its own type. To pick the provider at runtime, see Clients as trait objects.
The endpoint decides what works. Every client implements EmbeddingModel and StructuredPrompt, but a provider without embeddings or structured output returns an error at runtime.
Every client uses the Chat Completions API by default. use_responses_api(true) exists on every builder, but only some providers serve the Responses API.
For request fields that only one provider knows, set extra_body in Options. Only complete and complete_stream send it. prompt, structured_prompt and embed do not. See Extra request fields.
Some gateways cap the output at 4096 tokens when max_completion_tokens is unset. Set it in Options, because a cut-off answer looks complete.

Reference

Anthropic