Images and multimodal input

By the end of this page you send an image and a question to a chat model in one request and print its answer.

The example uses the OpenAI client, which sits behind the openai feature flag:

cargo add swiftide@0.34 --features openai

let bytes = std::fs::read("diagram.png")?;

let message = ChatMessage::new_user_with_parts(vec![
    ChatMessageContentPart::text("Describe this diagram in one sentence."),
    ChatMessageContentPart::image(ChatMessageContentSource::bytes(bytes, Some("image/png"))),
]);

let request = ChatCompletionRequest::builder()
    .messages(vec![message])
    .build()?;

let response = openai.complete(&request).await?;
println!("{}", response.message().unwrap_or_default());

Source: snippets/src/models/images.rs#L14-L26

Content parts

openai is an OpenAI client built as in Models. The prompt model you set with default_prompt_model must accept images, for example gpt-4o-mini. A client without a model fails with Model not set.

ChatMessage::new_user_with_parts builds a ChatMessage::UserWithParts message. It holds a list of content parts (ChatMessageContentPart) instead of one string. Here the list has a text part and an image part. The client sends them in that order.

Swiftide 0.33 added UserWithParts. It is a breaking change: if you match on ChatMessage, you need an arm for the new variant.

Each media part points at its data through a source (ChatMessageContentSource). ChatMessageContentSource::bytes takes the raw file and an optional media type. The OpenAI client encodes the bytes as base64 and sends them as a data Uniform Resource Locator (URL), data:image/png;base64,.... You do not need a base64 crate yourself.

complete is the method of the ChatCompletion trait, so import swiftide::traits::ChatCompletion. SimplePrompt only takes a text prompt, so it cannot carry images.

Image sources

A source can be a URL, a data URL, bytes, an Amazon Web Services (AWS) Simple Storage Service (S3) object or a provider file id:

let by_url = ChatMessageContentPart::image("https://example.com/chart.png");
let by_data_url = ChatMessageContentPart::image("data:image/png;base64,iVBORw0KGgo=");
let by_file_id =
    ChatMessageContentPart::image(ChatMessageContentSource::file_id("file-abc123"));
let in_s3 = ChatMessageContentPart::image(ChatMessageContentSource::s3(
    "s3://my-bucket/chart.png",
    None::<String>,
));
let raw_bytes = ChatMessageContentPart::image_with_format(vec![0x89, 0x50, 0x4e, 0x47], "png");

Source: snippets/src/models/images.rs#L34-L42

A &str or String becomes a URL source. That covers both https:// URLs and data URLs.
ChatMessageContentSource::file_id refers to a file you uploaded to the provider earlier.
ChatMessageContentSource::s3 points at an object in S3. The second argument is the bucket owner. Without an owner, write None::<String>, because a bare None leaves its type unknown and does not compile.
A Vec<u8> or &[u8] becomes a bytes source without a media type.
ChatMessageContentPart::image_with_format sets the format, such as png, explicitly. AWS Bedrock reads it first and otherwise infers the format from the media type or the file extension. The OpenAI clients ignore it.

Provider support

Every client converts content parts to its provider’s format before it sends the request. A part or source the client cannot convert fails complete with LanguageModelError::PermanentError, before anything reaches the provider. complete_stream returns a stream instead, and the error is its first item.

Client	Image	Document	Audio	Video
OpenAI, Chat Completions (default)	URL, data URL, bytes	no	bytes, `mp3` or `wav`	no
OpenAI, Responses API (`use_responses_api(true)`)	URL, data URL, bytes, file id	URL, bytes, file id	no	no
AWS Bedrock (`aws-bedrock`)	bytes, base64 data URL, S3, `s3://` URL	same sources as image	same sources as image	same sources as image
Anthropic	no	no	no	no

Ollama, Groq, Gemini, OpenRouter and Dashscope are OpenAI-compatible clients and share the OpenAI conversion. Whether the endpoint accepts the image depends on the provider and the model you pick.
On OpenAI Chat Completions, audio must be a bytes source in mp3 or wav. Without a format it is sent as mp3, and any other format fails. The media type does not change this.
AWS Bedrock accepts only base64 data URLs and s3:// URLs. It rejects every other URL, such as https://, http:// or file://, and it rejects file ids.
AWS Bedrock needs a format for images, audio and video. Pass a media type, use a file extension in the URL, or set the format explicitly. Otherwise the call fails with Bedrock image format is required.
An AWS Bedrock document without a resolvable format does not fail. It is sent as txt, so a PDF in bytes without a media type goes out as a text document.
An AWS Bedrock document part needs a non-empty text part in the same message. Bedrock drops empty text parts, so text("") does not count.
The Anthropic client (anthropic feature flag) accepts UserWithParts messages with text parts only. Any other part returns an error.

Gotchas

Token estimates ignore every part except text. See Errors, backoff, usage and tokens.
ChatMessage::as_ref returns an empty string for a UserWithParts message, unless the message has exactly one text part and nothing else. Code that logs or searches message text through as_ref skips these messages.
The Debug output of a source cuts data URLs to their first 32 characters of data, and bytes sources show only their length. Debug logs of the message history stay short.
For image parts without a media type, the OpenAI clients label bytes as application/octet-stream. Pass the real media type to ChatMessageContentSource::bytes. Documents on the Responses API and audio on Chat Completions do not send the media type, so there it has no effect.

Reference

Errors, backoff, usage and tokens