# OpenAI-compatible providers

> Build a client for Ollama, Groq, Gemini, OpenRouter, Dashscope or any other server that speaks the OpenAI API, with the feature flag, base URL and key variable each one uses.

Swiftide talks to Ollama, Groq, Gemini, OpenRouter and Dashscope through the same OpenAI client, each with its own base Uniform Resource Locator (URL) and key. You end up with a working client for each, local embeddings through Ollama, and a client for any other gateway that speaks the OpenAI application programming interface (API).

| Client       | Feature flag  | Base URL                                                  | Key variable                             |
| ------------ | ------------- | --------------------------------------------------------- | ---------------------------------------- |
| `Ollama`     | `ollama`      | `http://localhost:11434/v1`                               | none                                     |
| `Groq`       | `groq`        | `https://api.groq.com/openai/v1`                          | `GROQ_API_KEY`                           |
| `Gemini`     | `gemini`      | `https://generativelanguage.googleapis.com/v1beta/openai` | `GEMINI_API_KEY`                         |
| `OpenRouter` | `open-router` | `https://openrouter.ai/api/v1`                            | `OPENROUTER_API_KEY`                     |
| `Dashscope`  | `dashscope`   | `https://dashscope.aliyuncs.com/compatible-mode/v1`       | `QWEN_API_KEY`, then `DASHSCOPE_API_KEY` |

Each of these feature flags also turns on `openai`:

```sh
cargo add swiftide@0.34 --features ollama
```

## Ollama

```rust title="examples/index_ollama.rs"
let ollama_client = integrations::ollama::Ollama::default()
    .with_default_prompt_model("llama3.1")
    .to_owned();

let fastembed = integrations::fastembed::FastEmbed::try_default()?;
let memory_store = MemoryStorage::default();
```

Source: [`examples/index_ollama.rs#L30-L35`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/index_ollama.rs#L30-L35)

Every client on this page is `GenericOpenAI` with a provider-specific configuration type. It shares the `Options` and the model traits of [`OpenAI`](/integrations/openai/).

`Ollama::default()` builds a client without a model. `with_default_prompt_model` takes `&mut self` and returns `&mut Self`, so `.to_owned()` turns the result back into a client you own. The builder, `Ollama::builder()`, avoids that step.

The example uses Ollama only for the `MetadataQAText` transformer and embeds the chunks with FastEmbed, so it also needs the `fastembed` feature flag.

### Local embeddings

Ollama also serves embeddings. Set an embed model and use the same client on both sides:

```rust
let ollama = Ollama::builder()
    .default_prompt_model("llama3.2")
    .default_embed_model("nomic-embed-text")
    .build()?;

// nomic-embed-text returns 768 dimensions.
let qdrant = Qdrant::builder()
    .vector_size(768)
    .collection_name("local-docs")
    .build()?;

indexing::Pipeline::from_loader(FileLoader::new("docs").with_extensions(&["md"]))
    .then_chunk(ChunkMarkdown::from_chunk_range(100..2048))
    .then_in_batch(Embed::new(ollama.clone()).with_batch_size(32))
    .then_store_with(qdrant.clone())
    .run()
    .await?;

let answered = query::Pipeline::default()
    .then_transform_query(query_transformers::Embed::from_client(ollama.clone()))
    .then_retrieve(qdrant)
    .then_answer(answers::Simple::from_client(ollama))
    .query("How do I configure the loader?")
    .await?;
```

Source: [`snippets/src/integrations/openai_compatible.rs#L22-L45`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/integrations/openai_compatible.rs#L22-L45)

The `Embed` batch transformer embeds the chunks and the `Embed` query transformer embeds the question. Both must use the same model, or the vectors do not match. The storage vector size must match the model's output too. nomic-embed-text returns 768 dimensions.

The snippet stores the vectors in Qdrant, so it also needs the `qdrant` feature flag and a running Qdrant server. [Qdrant](/integrations/qdrant/) covers that part.

Pull the models before you run it:

```sh
ollama pull nomic-embed-text
ollama pull llama3.2
```

The Ollama example above uses `llama3.1`. Pull that one to run it.

### Remote Ollama

`OllamaConfig` is public, so you can point the client at another machine:

```rust
let mut config = OllamaConfig::default();
config.with_api_base("http://gpu-box:11434/v1");

let ollama = Ollama::builder()
    .client(async_openai::Client::with_config(config))
    .default_prompt_model("llama3.2")
    .build()?;

let answer = ollama.prompt("Name one Rust web framework.".into()).await?;
```

Source: [`snippets/src/integrations/openai_compatible.rs#L54-L62`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/integrations/openai_compatible.rs#L54-L62)

`OllamaConfig::with_api_base` takes `&mut self`, so build the config in a mutable variable first. `async_openai::Client` is not re-exported. Add the crate at the version Swiftide uses:

```sh
cargo add async-openai@0.41 --no-default-features
```

`OllamaConfig` sends no `Authorization` header and reads no environment variable. For an Ollama server behind a proxy that checks a bearer token, use `OpenAI` with `api_base` and `api_key` instead. See [Other gateways](#other-gateways).

## Groq

```rust title="examples/index_groq.rs"
let groq_client = integrations::groq::Groq::builder()
    .default_prompt_model("llama3-8b-8192")
    .to_owned()
    .build()?;

let fastembed = integrations::fastembed::FastEmbed::try_default()?;
let memory_store = MemoryStorage::default();
```

Source: [`examples/index_groq.rs#L29-L35`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/index_groq.rs#L29-L35)

`Groq::builder()` gives a builder with the Groq base URL and the key from `GROQ_API_KEY`. The `.to_owned()` in the middle clones the builder and changes nothing. Like the Ollama example, this one embeds with FastEmbed and needs the `fastembed` feature flag.

Groq has retired `llama3-8b-8192`, the model in the example. Pick a current model from Groq's model list.

## Gemini

```rust title="examples/hello_agents.rs"
let openai = swiftide::integrations::gemini::Gemini::builder()
    .default_embed_model("gemini-embedding-exp-03-07")
    .default_prompt_model("gemini-2.0-flash")
    .build()?;
```

Source: [`examples/hello_agents.rs#L109-L112`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/hello_agents.rs#L109-L112)

The variable is named `openai`, but its type is `Gemini`. Later in the example it goes to `Agent::builder().llm(&openai)`, which works because `Gemini` implements `ChatCompletion`. The agent needs the `swiftide-agents` feature flag as well. The key comes from `GEMINI_API_KEY`.

The embed model in the example, `gemini-embedding-exp-03-07`, is experimental. Check Google's model list for a current embedding model before you copy it.

## OpenRouter

```rust
let openrouter = OpenRouter::builder()
    .default_prompt_model("anthropic/claude-sonnet-4.5")
    .build()?;

let config = OpenRouterConfig::builder()
    .api_key(std::env::var("TEAM_OPENROUTER_KEY")?)
    .site_url("https://example.com/my-app")
    .site_name("My App")
    .build()?;

let team_openrouter = OpenRouter::builder()
    .client(async_openai::Client::with_config(config))
    .default_prompt_model("openai/gpt-5-mini")
    .build()?;
```

Source: [`snippets/src/integrations/openai_compatible.rs#L71-L84`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/integrations/openai_compatible.rs#L71-L84)

The first client reads `OPENROUTER_API_KEY`. The second takes the key from an `OpenRouterConfig`, built with `OpenRouterConfig::builder()`. That builder requires `api_key`, `site_url` and `site_name`. Leave one out and `build()` returns an error at runtime.

`site_url` sets the `HTTP-Referer` header, which OpenRouter uses for its app rankings. At this release the `X-Title` header also takes the `site_url` value, so `site_name` is required but not sent. `OpenRouterConfig::default()` leaves both unset. It then sends `https://github.com/bosun-ai/swiftide` as `HTTP-Referer` and `Swiftide` as `X-Title`.

## Dashscope

```rust title="examples/dashscope.rs"
let client = DashscopeBuilder::default()
    .default_embed_model("text-embedding-v2")
    .default_prompt_model("qwen-long")
    .build()?;
let tempdir = TempDir::new().unwrap();
let lancedb = LanceDB::builder()
    .uri(tempdir.child("lancedb").to_str().unwrap())
    .vector_size(1536)
    .with_vector(EmbeddedField::Combined)
    .with_metadata(metadata_qa_text::NAME)
    .table_name("swiftide_test")
    .build()
    .unwrap();
```

Source: [`examples/dashscope.rs#L22-L34`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/dashscope.rs#L22-L34)

`DashscopeBuilder::default()` is the same as `Dashscope::builder()`. The client prompts with `qwen-long` and embeds with `text-embedding-v2`. The example stores the vectors in LanceDB with a vector size of 1536, so it also needs the `lancedb` feature flag. [LanceDB](/integrations/lancedb/) covers that part.

## Other gateways

The configuration types of `Groq`, `Gemini` and `Dashscope` are private. To use another key variable or another base URL with them, use `OpenAI` with `api_base` and `api_key`.

```rust
let groq = OpenAI::builder()
    .api_base("https://api.groq.com/openai/v1")
    .api_key(std::env::var("TEAM_GROQ_KEY")?)
    .default_prompt_model("llama-3.3-70b-versatile")
    .build()?;
```

Source: [`snippets/src/integrations/openai_compatible.rs#L93-L97`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/integrations/openai_compatible.rs#L93-L97)

For vLLM, LiteLLM or a company gateway, see [Custom base URL](/integrations/openai/#custom-base-url).

## Gotchas

- None of the clients has a default model. Set one with `default_prompt_model` or `default_embed_model`, or calls fail with `Model not set`.
- A client reads its key when you build it. For `Groq`, `Gemini` and `Dashscope`, a missing key becomes an empty one, and the provider rejects the first request. `OpenRouter` panics on the first request instead.
- Each alias is its own type. To pick the provider at runtime, see [Clients as trait objects](/models/overview/#clients-as-trait-objects).
- The endpoint decides what works. Every client implements `EmbeddingModel` and `StructuredPrompt`, but a provider without embeddings or structured output returns an error at runtime.
- Every client uses the Chat Completions API by default. `use_responses_api(true)` exists on every builder, but only some providers serve the Responses API.
- For request fields that only one provider knows, set `extra_body` in `Options`. Only `complete` and `complete_stream` send it. `prompt`, `structured_prompt` and `embed` do not. See [Extra request fields](/models/overview/#extra-request-fields).
- Some gateways cap the output at 4096 tokens when `max_completion_tokens` is unset. Set it in `Options`, because a cut-off answer looks complete.

## Reference

- [`GenericOpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.GenericOpenAI.html)
- [`GenericOpenAI::with_default_prompt_model`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.GenericOpenAI.html#method.with_default_prompt_model)
- [`GenericOpenAIBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.GenericOpenAIBuilder.html)
- [`Options`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/struct.Options.html)
- [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/type.OpenAI.html)
- [`Ollama`](https://docs.rs/swiftide/0.34/swiftide/integrations/ollama/type.Ollama.html)
- [`OllamaConfig`](https://docs.rs/swiftide/0.34/swiftide/integrations/ollama/config/struct.OllamaConfig.html)
- [`OllamaConfig::with_api_base`](https://docs.rs/swiftide/0.34/swiftide/integrations/ollama/config/struct.OllamaConfig.html#method.with_api_base)
- [`Groq`](https://docs.rs/swiftide/0.34/swiftide/integrations/groq/type.Groq.html)
- [`Gemini`](https://docs.rs/swiftide/0.34/swiftide/integrations/gemini/type.Gemini.html)
- [`OpenRouter`](https://docs.rs/swiftide/0.34/swiftide/integrations/open_router/type.OpenRouter.html)
- [`OpenRouterConfig`](https://docs.rs/swiftide/0.34/swiftide/integrations/open_router/config/struct.OpenRouterConfig.html)
- [`OpenRouterConfigBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/open_router/config/struct.OpenRouterConfigBuilder.html)
- [`Dashscope`](https://docs.rs/swiftide/0.34/swiftide/integrations/dashscope/type.Dashscope.html)
- [`DashscopeBuilder`](https://docs.rs/swiftide/0.34/swiftide/integrations/dashscope/type.DashscopeBuilder.html)
- [`MetadataQAText`](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.MetadataQAText.html)
- [`Embed` (indexing)](https://docs.rs/swiftide/0.34/swiftide/indexing/transformers/struct.Embed.html)
- [`Embed` (query)](https://docs.rs/swiftide/0.34/swiftide/query/query_transformers/struct.Embed.html)
- [`FastEmbed`](https://docs.rs/swiftide/0.34/swiftide/integrations/fastembed/struct.FastEmbed.html)
- [`Qdrant`](https://docs.rs/swiftide/0.34/swiftide/integrations/qdrant/struct.Qdrant.html)
- [`LanceDB`](https://docs.rs/swiftide/0.34/swiftide/integrations/lancedb/struct.LanceDB.html)
- [`Agent`](https://docs.rs/swiftide/0.34/swiftide/agents/struct.Agent.html)
- [`ChatCompletion`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChatCompletion.html)
- [`EmbeddingModel`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.EmbeddingModel.html)
- [`StructuredPrompt`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.StructuredPrompt.html)

## Next

[Anthropic](/integrations/anthropic/)