# Testing agents

> Test Swiftide agents without a provider by scripting the model, calling tools directly, or matching exact requests with MockChatCompletion.

You can test an agent end to end without provider credentials or network access: a scripted model decides which tools to call, and your test asserts on the tool outputs, the requests and the stop reason.

`Agent`, `DefaultContext` and `StopReason` sit behind the `swiftide-agents` feature flag, which is off by default. The tests also need tokio for its test macros, serde_json and async-trait:

```sh
cargo add swiftide@0.34 --features swiftide-agents
cargo add --dev tokio --features macros,rt-multi-thread
cargo add --dev serde_json async-trait anyhow
```

Swiftide re-exports serde_json, async_trait and anyhow under `swiftide::reexports`, so you can use those paths instead of adding the crates.

## Scripted models

```rust
/// A model that answers with scripted responses and records every request it receives.
#[derive(Clone, Default)]
pub struct ScriptedLlm {
    replies: Arc<Mutex<VecDeque<ChatCompletionResponse>>>,
    requests: Arc<Mutex<Vec<Vec<ChatMessage>>>>,
}

impl ScriptedLlm {
    pub fn new(replies: impl IntoIterator<Item = ChatCompletionResponse>) -> Self {
        Self {
            replies: Arc::new(Mutex::new(replies.into_iter().collect())),
            requests: Arc::default(),
        }
    }

    /// The messages of every request, in the order the agent sent them.
    pub fn requests(&self) -> Vec<Vec<ChatMessage>> {
        self.requests.lock().unwrap().clone()
    }
}

#[async_trait]
impl ChatCompletion for ScriptedLlm {
    async fn complete(
        &self,
        request: &ChatCompletionRequest<'_>,
    ) -> Result<ChatCompletionResponse, LanguageModelError> {
        self.requests
            .lock()
            .unwrap()
            .push(request.messages().to_vec());
        self.replies
            .lock()
            .unwrap()
            .pop_front()
            .ok_or_else(|| LanguageModelError::permanent("the script has no more replies"))
    }
}

/// A scripted reply that calls one tool with JSON arguments.
pub fn reply_calling(
    name: &str,
    args: serde_json::Value,
) -> anyhow::Result<ChatCompletionResponse> {
    let call = ToolCall::builder()
        .id(format!("call_{name}"))
        .name(name)
        .args(args.to_string())
        .build()?;
    ChatCompletionResponse::builder()
        .tool_calls(vec![call])
        .build()
}
```

Source: [`snippets/src/agents/testing.rs#L30-L82`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/agents/testing.rs#L30-L82)

An agent talks to its model through the [`ChatCompletion`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChatCompletion.html) trait. Only `complete` is required. The default `complete_stream` wraps `complete` in a stream of one response, so a scripted model also works for an agent with `on_stream` hooks.

`ScriptedLlm` hands out one prepared [`ChatCompletionResponse`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionResponse.html) per completion and records the messages of every request. When the script runs out, it returns a permanent `LanguageModelError`, and `query` fails with `AgentError::CompletionsFailed`.

The state sits behind `Arc<Mutex<..>>` because `AgentBuilder::llm` takes a reference and stores a boxed clone. Your test keeps the original, and both share the same queue and request log.

`reply_calling` builds a response that calls one tool. The arguments are a JSON (JavaScript Object Notation) string, as a provider would send them.

## Agent tests

The tool under test is a plain function tool:

```rust
/// Counts the words in a piece of text.
///
/// # Arguments
/// - `text`: The text to count the words of.
#[swiftide::tool]
pub async fn count_words(_context: &dyn AgentContext, text: &str) -> Result<ToolOutput, ToolError> {
    Ok(text.split_whitespace().count().to_string().into())
}
```

Source: [`snippets/src/agents/testing.rs#L19-L26`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/agents/testing.rs#L19-L26)

The test scripts two turns: the model calls `count_words`, then the `stop` tool.

```rust
#[tokio::test]
async fn counts_the_words_then_stops() -> anyhow::Result<()> {
    let llm = ScriptedLlm::new([
        reply_calling("count_words", json!({ "text": "one two three" }))?,
        reply_calling("stop", json!({}))?,
    ]);
    let mut agent = Agent::builder().llm(&llm).tools([count_words()]).build()?;

    agent
        .query("How many words are in `one two three`?")
        .await?;

    let history = agent.history().await?;
    let counted = history.iter().find_map(|message| match message {
        ChatMessage::ToolOutput(call, output) if call.name() == "count_words" => {
            output.as_text()
        }
        _ => None,
    });
    assert_eq!(counted, Some("3"));
    assert!(matches!(
        agent.stop_reason(),
        Some(StopReason::RequestedByTool(call, None)) if call.name() == "stop"
    ));

    let requests = llm.requests();
    assert_eq!(requests.len(), 2);
    assert!(matches!(
        requests[1].last(),
        Some(ChatMessage::ToolOutput(..))
    ));
    Ok(())
}
```

Source: [`snippets/src/agents/testing.rs#L100-L132`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/agents/testing.rs#L100-L132)

The agent runs the real tool, records its output in the message history and sends the history back to the model. `stop` ends the run with `StopReason::RequestedByTool` and no payload. The test then checks three things:

- the `ChatMessage::ToolOutput` for `count_words` holds `3`
- the stop reason names the `stop` tool
- the second request ended with the tool output, so the model saw the result

Because the agent has no system prompt by default, the first request holds only the user message. To test a system prompt, set one on the builder and assert on the first `ChatMessage::System` in `llm.requests()`.

A reply without tool calls, such as `ChatCompletionResponse::builder().message("Done.").build()`, scripts a turn where the model only answers in text. The default context stops the agent after such a turn, with `StopReason::NoNewMessages`.

## Direct tool invocation

Most tool logic needs no agent at all. Call `invoke` on the tool with a context and a [`ToolCall`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ToolCall.html):

```rust
#[tokio::test]
async fn count_words_rejects_a_misspelled_argument() -> anyhow::Result<()> {
    let tool = count_words();
    let context = DefaultContext::default();

    let call = ToolCall::builder()
        .id("call_1")
        .name("count_words")
        .args(json!({ "text": "one two" }).to_string())
        .build()?;
    let output = tool.invoke(&context, &call).await?;
    assert_eq!(output.as_text(), Some("2"));

    let typo = ToolCall::builder()
        .id("call_2")
        .name("count_words")
        .args(json!({ "txt": "one two" }).to_string())
        .build()?;
    let error = tool.invoke(&context, &typo).await.unwrap_err();
    assert!(matches!(error, ToolError::WrongArguments(_)));
    Ok(())
}
```

Source: [`snippets/src/agents/testing.rs#L136-L157`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/agents/testing.rs#L136-L157)

[`DefaultContext::default()`](https://docs.rs/swiftide/0.34/swiftide/agents/struct.DefaultContext.html) runs commands through a `LocalExecutor` in the current directory, so tools that call `context.executor()` work too. Point it at a temporary directory with `DefaultContext::from_executor` when the tool touches files.

Arguments that do not deserialize into the generated arguments struct return `ToolError::WrongArguments`. In the test above, the misspelled `txt` key leaves the required `text` argument missing. A call without arguments returns `ToolError::MissingArguments`. Extra keys are ignored: `deny_unknown_fields` only applies to the JSON schema the model sees. A misspelled key for an `Option` argument becomes `None` without an error. Inside an agent, both go back to the model as a failed tool output and count toward `tool_retry_limit`.

## MockChatCompletion

For tests that pin down the exact requests, swiftide-core ships `MockChatCompletion` behind its `test-utils` feature flag. The swiftide crate has a `test-utils` feature too, but it does not re-export the module, so add swiftide-core as a dev dependency and import `swiftide_core::test_utils::MockChatCompletion`:

```sh
cargo add --dev swiftide-core@0.34 --features test-utils
```

```rust
#[tokio::test]
async fn sends_the_tool_output_back_to_the_model() -> anyhow::Result<()> {
    let llm = MockChatCompletion::new();
    let question = ChatMessage::new_user("How many words are in `one two`?");
    let count = ToolCall::builder()
        .id("call_1")
        .name("count_words")
        .args(json!({ "text": "one two" }).to_string())
        .build()?;
    let stop = ToolCall::builder().id("call_2").name("stop").build()?;

    llm.expect_complete(
        ChatCompletionRequest::builder()
            .messages(vec![question.clone()])
            .tools(Agent::default_tools())
            .tool(count_words())
            .build()?,
        Ok(ChatCompletionResponse::builder()
            .tool_calls(vec![count.clone()])
            .build()?),
    );
    llm.expect_complete(
        ChatCompletionRequest::builder()
            .messages(vec![
                question,
                ChatMessage::new_assistant(None::<String>, Some(vec![count.clone()])),
                ChatMessage::new_tool_output(count, ToolOutput::text("2")),
            ])
            .tools(Agent::default_tools())
            .tool(count_words())
            .build()?,
        Ok(ChatCompletionResponse::builder()
            .tool_calls(vec![stop])
            .build()?),
    );

    let mut agent = Agent::builder().llm(&llm).tools([count_words()]).build()?;
    agent.query("How many words are in `one two`?").await?;

    assert!(agent.is_stopped());
    Ok(())
}
```

Source: [`snippets/src/agents/testing.rs#L161-L202`](https://github.com/bosun-ai/swiftide-website/blob/master/snippets/src/agents/testing.rs#L161-L202)

Each `expect_complete` queues an expected [`ChatCompletionRequest`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionRequest.html) and the response to return. Expectations are consumed in the order you add them. A request that differs from the next expectation fails the test with both requests printed. An `Err` response reaches the agent as a permanent `LanguageModelError`.

Request equality covers the messages and the full set of tool specs. The agent sends the spec of every tool it has, including the default `stop` tool, so the expected request lists `Agent::default_tools()` next to your own tools.

When the last clone of the mock is dropped with expectations left over, it panics. A test that stops early fails even when every request it made matched.

## Showcase tests

The release notes showcase has 27 tests. From the swiftide repository root:

```sh
cargo test -p release-notes-agent
```

The session tests exercise the approval loop in `session.rs`:

```rust title="examples/release-notes-agent/src/session.rs"
/// Asks the agent for release notes and loops through review rounds until they are written or
/// rejected.
pub async fn draft_release_notes(
    agent: &mut Agent,
    task: Prompt,
    reviewer: &mut impl Reviewer,
) -> Result<Outcome> {
    agent.query(task).await?;
    let mut written = false;
    let mut nudged = false;

    loop {
        // `write_changelog` is wrapped in `ApprovalRequired`: the agent stops and waits.
        let tool_call = match agent.stop_reason() {
            Some(StopReason::FeedbackRequired { tool_call, .. }) => tool_call.clone(),
            // The custom `stop` tool carries the structured release notes.
            Some(StopReason::RequestedByTool(_, Some(notes))) if written => {
                let notes = ReleaseNotes::deserialize(notes)
                    .context("the agent stopped with malformed release notes")?;
                return Ok(Outcome::Written { notes: Some(notes) });
            }
            _ if written => return Ok(Outcome::Written { notes: None }),
            // A turn without a tool call, or the step limit.
            Some(StopReason::NoNewMessages) => {
                nudge_once(agent, &mut nudged).await?;
                continue;
            }
            other => bail!("the agent stopped without writing the notes: {other:?}"),
        };

        let proposal: WriteChangelogArgs =
            serde_json::from_str(tool_call.args().unwrap_or_default())?;

        match reviewer.review(&proposal)? {
            Verdict::Approve => {
                agent
                    .context()
                    .feedback_received(&tool_call, &ToolFeedback::approved())
                    .await?;
                // Resumes the pending tool call, which now runs, and continues the loop.
                agent.run().await?;
                // The write itself can still fail; the agent then hears why and carries on.
                written |= succeeded(agent, &tool_call).await?;
            }
            Verdict::Revise(feedback) => {
                agent
                    .context()
                    .feedback_received(&tool_call, &ToolFeedback::refused())
                    .await?;
                // The refusal is recorded as the tool's output, followed by the feedback.
                agent.query(feedback).await?;
            }
            Verdict::Reject => return Ok(Outcome::Rejected),
        }
    }
}
```

Source: [`examples/release-notes-agent/src/session.rs#L41-L96`](https://github.com/bosun-ai/swiftide/blob/e377c4a2e76fe2559629ae0efdae3c6e0fb4d829/examples/release-notes-agent/src/session.rs#L41-L96)

Five of the six session tests drive this loop with a scripted model like the one above. The showcase's own `ScriptedLlm`, a simpler variant of the one above, takes `(tool name, arguments)` pairs, and an empty name scripts a turn without a tool call. A `ScriptedReviewer` plays the person who approves or asks for a revision, and those tests run against a temporary repository. They cover:

- a revision round, then approval and the write
- stopping without a write, which is an error
- one nudge after a turn without a tool call, and an error after the second one
- a draft that lists a pull request twice, which goes back to the model before review

The duplicate check also ran against a live model. The other paths ran only in these tests.

The sixth session test is a synchronous unit test. It builds a history by hand and checks that reaching the step limit is not mistaken for a turn without a tool call, although both report the same stop reason.

The remaining tests are plain unit tests in the other modules. They cover system prompt rendering, decoding a streamed draft however it is chunked, and restarting the draft preview for a new tool call. They also cover finding a pull request listed in two entries, the stop tool schema, rendering the notes, tool names, clipping tool output, shell quoting, GitHub remote parsing, and rejecting unsafe revisions and paths.

## Gotchas

- `swiftide_agents::test_utils` (`MockTool`, `MockHook`, `chat_request!`) is compiled only in test and debug builds, and its macros expand to `swiftide_core` paths. It serves swiftide's own tests; do not build on it.
- `MockChatCompletion::complete_stream` splits the response text on whitespace inside a spawned task. On a response without text that task panics, the stream closes empty, and `query` returns `AgentError::EmptyStream`. A streamed reply that only calls tools never delivers its tool calls. Script a message on every reply when the agent streams.
- The agent rewrites JSON object arguments before it stores the assistant message, to drop duplicate keys. Build expected arguments with `serde_json`, as above, rather than as hand-typed strings with their own spacing.
- The unit context `()` implements `AgentContext` for tools that never run commands. Its `executor()` panics, and it approves every feedback request.
- A tool call that names a tool the agent does not have is skipped with a warning and gets no output. With a typo in a scripted tool name, the run ends with `StopReason::NoNewMessages` instead of the scripted stop. The next `query` or `run` on that agent fails with `AgentError::UnfulfilledToolCalls`.

## Reference

- [`ChatCompletion`](https://docs.rs/swiftide/0.34/swiftide/traits/trait.ChatCompletion.html)
- [`ChatCompletionRequest`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionRequest.html)
- [`ChatCompletionResponse`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ChatCompletionResponse.html)
- [`ChatMessage`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/enum.ChatMessage.html)
- [`ToolCall`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/struct.ToolCall.html)
- [`ToolError`](https://docs.rs/swiftide/0.34/swiftide/chat_completion/errors/enum.ToolError.html)
- [`Agent`](https://docs.rs/swiftide/0.34/swiftide/agents/struct.Agent.html)
- [`DefaultContext`](https://docs.rs/swiftide/0.34/swiftide/agents/struct.DefaultContext.html)
- [`StopReason`](https://docs.rs/swiftide/0.34/swiftide/agents/enum.StopReason.html)
- [`MockChatCompletion`](https://docs.rs/swiftide-core/0.34/swiftide_core/test_utils/struct.MockChatCompletion.html)
- [`AgentError`](https://docs.rs/swiftide/0.34/swiftide/agents/errors/enum.AgentError.html)

## Next

[How tasks work](/tasks/overview/)