# Describing images

> Send a Portable Network Graphics (PNG) image to an OpenAI model as a base64 data URL in a multi-part user message and print a one-sentence description.

This recipe sends the Swiftide logo to `gpt-4o-mini` and prints the one-sentence description the model writes back. It is the `describe_image.rs` example from the swiftide repository: one user message that carries a text part and an image part, sent with a single chat completion call.

## Setup

To run the example, clone the swiftide repository, set `OPENAI_API_KEY` and run it from the repository root:

```sh
git clone https://github.com/bosun-ai/swiftide && cd swiftide
export OPENAI_API_KEY=sk-...
cargo run -p swiftide-examples --example describe-image
```

The examples crate turns on every integration, so the first build compiles all of them. It needs `cmake` and the Protocol Buffers compiler (`protoc`) on your machine.

To use the code in your own crate, add the OpenAI client, which sits behind the `openai` feature flag, and the `base64` crate to encode the image:

```sh
cargo add swiftide@0.34 --features openai
cargo add base64@0.22 anyhow
cargo add tokio --features macros,rt-multi-thread
```

## Client

```rust
let openai = swiftide::integrations::openai::OpenAI::builder()
    .default_prompt_model("gpt-4o-mini")
    .build()?;
```

Source: [`examples/describe_image.rs#L15-L17`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/describe_image.rs#L15-L17)

The prompt model must accept images. `gpt-4o-mini` does. The client reads the API key from `OPENAI_API_KEY`.

## Image data

```rust
let image_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../images/logo.png");
let image_bytes = std::fs::read(&image_path).with_context(|| format!("Read {image_path:?}"))?;
let encoded = general_purpose::STANDARD.encode(&image_bytes);
let data_url = format!("data:image/png;base64,{encoded}");
```

Source: [`examples/describe_image.rs#L21-L24`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/describe_image.rs#L21-L24)

`env!("CARGO_MANIFEST_DIR")` is the directory of the examples crate, fixed at compile time. The path climbs one level to the repository root and reads `images/logo.png`. The example then encodes the bytes as base64 and wraps them in a data Uniform Resource Locator (URL) of the form `data:image/png;base64,...`.

The snippets leave out the imports. `encode` comes from the `base64::Engine` trait, and `with_context` from `anyhow::Context`, so bring both into scope with `use base64::{Engine as _, engine::general_purpose};` and `use anyhow::Context as _;`. The request types come from `swiftide::chat_completion`, and `complete` from `swiftide::traits::ChatCompletion`.

The media type in the data URL tells the model how to decode the bytes. Use the type of the file you actually send, such as `image/jpeg` for a Joint Photographic Experts Group (JPEG) file.

## Request

```rust
let message = ChatMessage::new_user_with_parts(vec![
    ChatMessageContentPart::text("Describe this image in one sentence."),
    ChatMessageContentPart::image(data_url),
]);

let request = ChatCompletionRequest::builder()
    .messages(vec![message])
    .build()?;
```

Source: [`examples/describe_image.rs#L28-L35`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/describe_image.rs#L28-L35)

The message carries a text part and an image part, in that order. The data URL is a string, so `ChatMessageContentPart::image` takes it as a URL source. [Images and multimodal input](/models/images/#content-parts) explains content parts and sources.

The request has no system message here. Add one with `ChatMessage::new_system` in front of the user message when you want to set the tone or length of the answer.

## Response

```rust
let response = openai.complete(&request).await?;
println!(
    "Image description: {}",
    response.message().unwrap_or("<no response>")
);
```

Source: [`examples/describe_image.rs#L39-L43`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/describe_image.rs#L39-L43)

`complete` sends one request and returns a `ChatCompletionResponse`. `message()` returns the text of the answer as an `Option<&str>`.

## Variations

### Bytes instead of a data URL

You can skip the base64 step. Pass the raw bytes and the media type, as `ChatMessageContentSource::bytes(image_bytes, Some("image/png"))`, and the OpenAI client builds the same data URL for you. A plain `Vec<u8>` also works, but it has no media type, and the OpenAI clients then label the image `application/octet-stream`. [Images and multimodal input](/models/images/) shows this form in full.

### Images on the web

With the OpenAI client, an `https://` URL works as is: `ChatMessageContentPart::image("https://example.com/chart.png")`. The client passes the URL to the provider unchanged.

### Several images

A message can carry more than one image part. Add each one to the list, and refer to them by position in the text part, for example "Compare the first and the second image".

### Other providers

The Anthropic client rejects image parts, and Amazon Web Services (AWS) Bedrock needs an image format and accepts only `data:` and `s3://` URLs. The [provider support table](/models/images/#provider-support) lists what each client accepts.

## Gotchas

- `CARGO_MANIFEST_DIR` is a compile-time path. A binary you copy to another machine does not find the file there. Read the path from configuration or a command-line argument, or embed the file with `include_bytes!`.
- A large image makes a large request. Resize it before you encode it when the model does not need full resolution.

## Reference

- [`ChatMessage`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/enum.ChatMessage.html)
- [`ChatMessageContentPart`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/enum.ChatMessageContentPart.html)
- [`ChatMessageContentSource`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/enum.ChatMessageContentSource.html)
- [`ChatCompletionRequest`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionRequest.html)
- [`ChatCompletionResponse`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionResponse.html)
- [`ChatCompletion`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChatCompletion.html)
- [`OpenAI`](https://docs.rs/swiftide/0.34/swiftide/integrations/openai/type.OpenAI.html)

## Next

[Usage metrics](/cookbook/usage-metrics/)