Testing agents

You can test an agent end to end without provider credentials or network access: a scripted model decides which tools to call, and your test asserts on the tool outputs, the requests and the stop reason.

Agent, DefaultContext and StopReason sit behind the swiftide-agents feature flag, which is off by default. The tests also need tokio for its test macros, serde_json and async-trait:

cargo add swiftide@0.34 --features swiftide-agents
cargo add --dev tokio --features macros,rt-multi-thread
cargo add --dev serde_json async-trait anyhow

Swiftide re-exports serde_json, async_trait and anyhow under swiftide::reexports, so you can use those paths instead of adding the crates.

Scripted models

/// A model that answers with scripted responses and records every request it receives.
#[derive(Clone, Default)]
pub struct ScriptedLlm {
    replies: Arc<Mutex<VecDeque<ChatCompletionResponse>>>,
    requests: Arc<Mutex<Vec<Vec<ChatMessage>>>>,
}

impl ScriptedLlm {
    pub fn new(replies: impl IntoIterator<Item = ChatCompletionResponse>) -> Self {
        Self {
            replies: Arc::new(Mutex::new(replies.into_iter().collect())),
            requests: Arc::default(),
        }
    }

    /// The messages of every request, in the order the agent sent them.
    pub fn requests(&self) -> Vec<Vec<ChatMessage>> {
        self.requests.lock().unwrap().clone()
    }
}

#[async_trait]
impl ChatCompletion for ScriptedLlm {
    async fn complete(
        &self,
        request: &ChatCompletionRequest<'_>,
    ) -> Result<ChatCompletionResponse, LanguageModelError> {
        self.requests
            .lock()
            .unwrap()
            .push(request.messages().to_vec());
        self.replies
            .lock()
            .unwrap()
            .pop_front()
            .ok_or_else(|| LanguageModelError::permanent("the script has no more replies"))
    }
}

/// A scripted reply that calls one tool with JSON arguments.
pub fn reply_calling(
    name: &str,
    args: serde_json::Value,
) -> anyhow::Result<ChatCompletionResponse> {
    let call = ToolCall::builder()
        .id(format!("call_{name}"))
        .name(name)
        .args(args.to_string())
        .build()?;
    ChatCompletionResponse::builder()
        .tool_calls(vec![call])
        .build()
}

Source: snippets/src/agents/testing.rs#L30-L82

An agent talks to its model through the ChatCompletion trait. Only complete is required. The default complete_stream wraps complete in a stream of one response, so a scripted model also works for an agent with on_stream hooks.

ScriptedLlm hands out one prepared ChatCompletionResponse per completion and records the messages of every request. When the script runs out, it returns a permanent LanguageModelError, and query fails with AgentError::CompletionsFailed.

The state sits behind Arc<Mutex<..>> because AgentBuilder::llm takes a reference and stores a boxed clone. Your test keeps the original, and both share the same queue and request log.

reply_calling builds a response that calls one tool. The arguments are a JSON (JavaScript Object Notation) string, as a provider would send them.

Agent tests

The tool under test is a plain function tool:

/// Counts the words in a piece of text.
///
/// # Arguments
/// - `text`: The text to count the words of.
#[swiftide::tool]
pub async fn count_words(_context: &dyn AgentContext, text: &str) -> Result<ToolOutput, ToolError> {
    Ok(text.split_whitespace().count().to_string().into())
}

Source: snippets/src/agents/testing.rs#L19-L26

The test scripts two turns: the model calls count_words, then the stop tool.

#[tokio::test]
async fn counts_the_words_then_stops() -> anyhow::Result<()> {
    let llm = ScriptedLlm::new([
        reply_calling("count_words", json!({ "text": "one two three" }))?,
        reply_calling("stop", json!({}))?,
    ]);
    let mut agent = Agent::builder().llm(&llm).tools([count_words()]).build()?;

    agent
        .query("How many words are in `one two three`?")
        .await?;

    let history = agent.history().await?;
    let counted = history.iter().find_map(|message| match message {
        ChatMessage::ToolOutput(call, output) if call.name() == "count_words" => {
            output.as_text()
        }
        _ => None,
    });
    assert_eq!(counted, Some("3"));
    assert!(matches!(
        agent.stop_reason(),
        Some(StopReason::RequestedByTool(call, None)) if call.name() == "stop"
    ));

    let requests = llm.requests();
    assert_eq!(requests.len(), 2);
    assert!(matches!(
        requests[1].last(),
        Some(ChatMessage::ToolOutput(..))
    ));
    Ok(())
}

Source: snippets/src/agents/testing.rs#L100-L132

The agent runs the real tool, records its output in the message history and sends the history back to the model. stop ends the run with StopReason::RequestedByTool and no payload. The test then checks three things:

• the ChatMessage::ToolOutput for count_words holds 3
• the stop reason names the stop tool
• the second request ended with the tool output, so the model saw the result

Because the agent has no system prompt by default, the first request holds only the user message. To test a system prompt, set one on the builder and assert on the first ChatMessage::System in llm.requests().

A reply without tool calls, such as ChatCompletionResponse::builder().message("Done.").build(), scripts a turn where the model only answers in text. The default context stops the agent after such a turn, with StopReason::NoNewMessages.

Direct tool invocation

Most tool logic needs no agent at all. Call invoke on the tool with a context and a ToolCall:

#[tokio::test]
async fn count_words_rejects_a_misspelled_argument() -> anyhow::Result<()> {
    let tool = count_words();
    let context = DefaultContext::default();

    let call = ToolCall::builder()
        .id("call_1")
        .name("count_words")
        .args(json!({ "text": "one two" }).to_string())
        .build()?;
    let output = tool.invoke(&context, &call).await?;
    assert_eq!(output.as_text(), Some("2"));

    let typo = ToolCall::builder()
        .id("call_2")
        .name("count_words")
        .args(json!({ "txt": "one two" }).to_string())
        .build()?;
    let error = tool.invoke(&context, &typo).await.unwrap_err();
    assert!(matches!(error, ToolError::WrongArguments(_)));
    Ok(())
}

Source: snippets/src/agents/testing.rs#L136-L157

DefaultContext::default() runs commands through a LocalExecutor in the current directory, so tools that call context.executor() work too. Point it at a temporary directory with DefaultContext::from_executor when the tool touches files.

Arguments that do not deserialize into the generated arguments struct return ToolError::WrongArguments. In the test above, the misspelled txt key leaves the required text argument missing. A call without arguments returns ToolError::MissingArguments. Extra keys are ignored: deny_unknown_fields only applies to the JSON schema the model sees. A misspelled key for an Option argument becomes None without an error. Inside an agent, both go back to the model as a failed tool output and count toward tool_retry_limit.

MockChatCompletion

For tests that pin down the exact requests, swiftide-core ships MockChatCompletion behind its test-utils feature flag. The swiftide crate has a test-utils feature too, but it does not re-export the module, so add swiftide-core as a dev dependency and import swiftide_core::test_utils::MockChatCompletion:

cargo add --dev swiftide-core@0.34 --features test-utils

#[tokio::test]
async fn sends_the_tool_output_back_to_the_model() -> anyhow::Result<()> {
    let llm = MockChatCompletion::new();
    let question = ChatMessage::new_user("How many words are in `one two`?");
    let count = ToolCall::builder()
        .id("call_1")
        .name("count_words")
        .args(json!({ "text": "one two" }).to_string())
        .build()?;
    let stop = ToolCall::builder().id("call_2").name("stop").build()?;

    llm.expect_complete(
        ChatCompletionRequest::builder()
            .messages(vec![question.clone()])
            .tools(Agent::default_tools())
            .tool(count_words())
            .build()?,
        Ok(ChatCompletionResponse::builder()
            .tool_calls(vec![count.clone()])
            .build()?),
    );
    llm.expect_complete(
        ChatCompletionRequest::builder()
            .messages(vec![
                question,
                ChatMessage::new_assistant(None::<String>, Some(vec![count.clone()])),
                ChatMessage::new_tool_output(count, ToolOutput::text("2")),
            ])
            .tools(Agent::default_tools())
            .tool(count_words())
            .build()?,
        Ok(ChatCompletionResponse::builder()
            .tool_calls(vec![stop])
            .build()?),
    );

    let mut agent = Agent::builder().llm(&llm).tools([count_words()]).build()?;
    agent.query("How many words are in `one two`?").await?;

    assert!(agent.is_stopped());
    Ok(())
}

Source: snippets/src/agents/testing.rs#L161-L202

Each expect_complete queues an expected ChatCompletionRequest and the response to return. Expectations are consumed in the order you add them. A request that differs from the next expectation fails the test with both requests printed. An Err response reaches the agent as a permanent LanguageModelError.

Request equality covers the messages and the full set of tool specs. The agent sends the spec of every tool it has, including the default stop tool, so the expected request lists Agent::default_tools() next to your own tools.

When the last clone of the mock is dropped with expectations left over, it panics. A test that stops early fails even when every request it made matched.

Showcase tests

The release notes showcase has 27 tests. From the swiftide repository root:

cargo test -p release-notes-agent

The session tests exercise the approval loop in session.rs:

examples/release-notes-agent/src/session.rs

/// Asks the agent for release notes and loops through review rounds until they are written or
/// rejected.
pub async fn draft_release_notes(
    agent: &mut Agent,
    task: Prompt,
    reviewer: &mut impl Reviewer,
) -> Result<Outcome> {
    agent.query(task).await?;
    let mut written = false;
    let mut nudged = false;

    loop {
        // `write_changelog` is wrapped in `ApprovalRequired`: the agent stops and waits.
        let tool_call = match agent.stop_reason() {
            Some(StopReason::FeedbackRequired { tool_call, .. }) => tool_call.clone(),
            // The custom `stop` tool carries the structured release notes.
            Some(StopReason::RequestedByTool(_, Some(notes))) if written => {
                let notes = ReleaseNotes::deserialize(notes)
                    .context("the agent stopped with malformed release notes")?;
                return Ok(Outcome::Written { notes: Some(notes) });
            }
            _ if written => return Ok(Outcome::Written { notes: None }),
            // A turn without a tool call, or the step limit.
            Some(StopReason::NoNewMessages) => {
                nudge_once(agent, &mut nudged).await?;
                continue;
            }
            other => bail!("the agent stopped without writing the notes: {other:?}"),
        };

        let proposal: WriteChangelogArgs =
            serde_json::from_str(tool_call.args().unwrap_or_default())?;

        match reviewer.review(&proposal)? {
            Verdict::Approve => {
                agent
                    .context()
                    .feedback_received(&tool_call, &ToolFeedback::approved())
                    .await?;
                // Resumes the pending tool call, which now runs, and continues the loop.
                agent.run().await?;
                // The write itself can still fail; the agent then hears why and carries on.
                written |= succeeded(agent, &tool_call).await?;
            }
            Verdict::Revise(feedback) => {
                agent
                    .context()
                    .feedback_received(&tool_call, &ToolFeedback::refused())
                    .await?;
                // The refusal is recorded as the tool's output, followed by the feedback.
                agent.query(feedback).await?;
            }
            Verdict::Reject => return Ok(Outcome::Rejected),
        }
    }
}

Source: examples/release-notes-agent/src/session.rs#L41-L96

Five of the six session tests drive this loop with a scripted model like the one above. The showcase’s own ScriptedLlm, a simpler variant of the one above, takes (tool name, arguments) pairs, and an empty name scripts a turn without a tool call. A ScriptedReviewer plays the person who approves or asks for a revision, and those tests run against a temporary repository. They cover:

• a revision round, then approval and the write
• stopping without a write, which is an error
• one nudge after a turn without a tool call, and an error after the second one
• a draft that lists a pull request twice, which goes back to the model before review

The duplicate check also ran against a live model. The other paths ran only in these tests.

The sixth session test is a synchronous unit test. It builds a history by hand and checks that reaching the step limit is not mistaken for a turn without a tool call, although both report the same stop reason.

The remaining tests are plain unit tests in the other modules. They cover system prompt rendering, decoding a streamed draft however it is chunked, and restarting the draft preview for a new tool call. They also cover finding a pull request listed in two entries, the stop tool schema, rendering the notes, tool names, clipping tool output, shell quoting, GitHub remote parsing, and rejecting unsafe revisions and paths.

Gotchas

• swiftide_agents::test_utils (MockTool, MockHook, chat_request!) is compiled only in test and debug builds, and its macros expand to swiftide_core paths. It serves swiftide’s own tests; do not build on it.
• MockChatCompletion::complete_stream splits the response text on whitespace inside a spawned task. On a response without text that task panics, the stream closes empty, and query returns AgentError::EmptyStream. A streamed reply that only calls tools never delivers its tool calls. Script a message on every reply when the agent streams.
• The agent rewrites JSON object arguments before it stores the assistant message, to drop duplicate keys. Build expected arguments with serde_json, as above, rather than as hand-typed strings with their own spacing.
• The unit context () implements AgentContext for tools that never run commands. Its executor() panics, and it approves every feedback request.
• A tool call that names a tool the agent does not have is skipped with a warning and gets no output. With a typo in a scripted tool name, the run ends with StopReason::NoNewMessages instead of the scripted stop. The next query or run on that agent fails with AgentError::UnfulfilledToolCalls.

Reference

• ChatCompletion
• ChatCompletionRequest
• ChatCompletionResponse
• ChatMessage
• ToolCall
• ToolError
• Agent
• DefaultContext
• StopReason
• MockChatCompletion
• AgentError

Next

How tasks work