Evaluating with Ragas

You end with a JSON file that holds every question, the answer your query pipeline gave, the documents it retrieved and the answer you expected. You can score that file and compare it between runs. The scores at the end come from the evaluation in the docs-qa showcase.

Evaluation run

Ragas needs no feature flag. The snippets below use the OpenAI client and Qdrant, which sit behind the openai and qdrant feature flags:

cargo add swiftide@0.34 --features openai,qdrant

let ragas = Ragas::from_prepared_questions(vec![
    (
        "What does a retriever return?".to_string(),
        "A retriever returns documents that match the query.".to_string(),
    ),
    (
        "Which step turns retrieved documents into an answer?".to_string(),
        "The answer generator, for example answers::Simple.".to_string(),
    ),
]);

let pipeline = query::Pipeline::default()
    // Attach the evaluator while the pipeline is still pending
    .evaluate_with(ragas.clone())
    .then_transform_query(query_transformers::Embed::from_client(openai.clone()))
    .then_retrieve(qdrant)
    .then_answer(answers::Simple::from_client(openai));

pipeline.query_all(ragas.questions().await).await?;

std::fs::write("ragas.json", ragas.to_json().await)?;

Source: snippets/src/querying/evaluation.rs#L19-L39

Ragas is an evaluator: a type that implements EvaluateQuery. It holds the dataset, and its clones share that dataset. Pass one clone to the pipeline with evaluate_with and keep the other to read the results.

ragas.questions() returns one pending query per question in the dataset, and query_all runs them all. The pipeline calls the evaluator twice per question. Right after then_retrieve, Ragas stores the content of the retrieved documents as the contexts of that question. After then_answer, it stores the answer. to_json then writes the dataset.

Evaluation in docs-qa

The docs-qa showcase runs the same steps on its own question set:

let ragas = Ragas::from_prepared_questions(&questions);

let answered = qa
    .evaluated_pipeline(ragas.clone())
    .query_all(ragas.questions().await)
    .await?;

let ragas_json = ragas.to_json().await;

Source: examples/docs-qa/src/eval.rs#L89-L96

evaluated_pipeline builds the same pipeline as the ask command and adds evaluate_with(ragas) to it. docs-qa also enables the fastembed feature flag for its hybrid search.

Questions and ground truths

Ragas::from_prepared_questions takes anything that converts into an EvaluationDataSet. An EvaluationDataSet converts from:

Vec<String> or &[String]: questions only, with an empty ground truth.
Vec<(String, String)>: pairs of question and ground truth, as in the first snippet.
a JSON string, through parse, in the format that to_json writes.

docs-qa passes &QuestionSet, its own YAML-backed type, which implements From<&QuestionSet> for EvaluationDataSet. Implement From for your own type in the same way.

A ground truth is the answer you expect, written in your own words. Scores that compare the answer with the ground truth need it, so write one for every question you can.

Baseline ground truths

Writing ground truths by hand takes time. You can let a first run write them and review them afterwards.

// Runs the pipeline once over every question in the dataset
let evaluate = async |ragas: &Ragas| -> anyhow::Result<()> {
    query::Pipeline::default()
        .evaluate_with(ragas.clone())
        .then_transform_query(query_transformers::Embed::from_client(openai.clone()))
        .then_retrieve(qdrant.clone())
        .then_answer(answers::Simple::from_client(openai.clone()))
        .query_all(ragas.questions().await)
        .await?;
    Ok(())
};

// First run: questions only, then keep this run's answers as the ground truth
let questions = vec![
    "What does a retriever return?".to_string(),
    "Which step turns retrieved documents into an answer?".to_string(),
];
let ragas = Ragas::from_prepared_questions(questions);
evaluate(&ragas).await?;
ragas.record_answers_as_ground_truth().await;
std::fs::write("baseline.json", ragas.to_json().await)?;

// Later runs: load the reviewed baseline and evaluate against it
let dataset: EvaluationDataSet = std::fs::read_to_string("baseline.json")?.parse()?;
let ragas = Ragas::from_prepared_questions(dataset);
evaluate(&ragas).await?;

Source: snippets/src/querying/evaluation.rs#L47-L72

The evaluate closure builds a pipeline with evaluate_with(ragas.clone()) and runs query_all over ragas.questions(), so each call fills the dataset with answers and contexts. record_answers_as_ground_truth then copies each recorded answer into its ground truth. Read the file and fix wrong answers before you treat it as the baseline. Parsing an export keeps its ground truths. The next run overwrites the answers and contexts it loaded.

The dataset

The export is a pretty-printed JSON array with one object per question. This is the output that the unit test for to_json expects for one question with a ground truth that has not run yet:

[
  {
    "answer": "",
    "contexts": [],
    "ground_truth": "A programming language",
    "question": "What is Rust?"
  }
]

The keys are in alphabetical order, unless a crate in your build enables the serde_json feature that preserves insertion order. The rows are in hash map order, which changes from run to run. docs-qa sorts the rows back into the order of its questions file, so the exports of two runs can be diffed.

contexts holds every retrieved document, recorded before any response transformer runs. docs-qa retrieves 20 candidates and reranks them down to 10, so its ragas.json has 20 contexts per question, while the answer generator saw 10. Ragas metrics computed from contexts therefore score the retriever’s 20 candidates, not the 10 documents the answer generator saw.

Scores

Ragas computes no scores. The JSON is input for the Python ragas toolkit, which calculates metrics such as faithfulness and context recall.

docs-qa shows numbers without Python. Its judge asks a large language model (LLM) to split the ground truth and the answer into claims and to check each one, using structured output. Its scores are ratios modelled on the ragas metrics of the same names. In this run claude-sonnet-5-5 both answered and judged:

./target/debug/docs-qa --model claude-sonnet-5-5 eval --judge-model claude-sonnet-5-5 \
  --out examples/docs-qa/ragas.json

#   question                                              ctx faithful   answer  context
                                                                         recall   recall
1   Which crate features and environment variable do I …   10     1.00     1.00     1.00
2   What does the agents-contrib-context feature provid…   10     1.00     1.00     1.00
3   What are the default limits of the read_file tool?     10     1.00     1.00     1.00
4   Which storage and retrieval backends does Swiftide …   10     1.00     1.00     1.00
5   What should I run before opening a pull request?       10     1.00     1.00     1.00
6   What is a task in Swiftide?                            10     1.00     1.00     1.00
7   When does ContextCompression summarize the conversa…   10     1.00     1.00     1.00
8   Which commit message convention does the project fo…   10     1.00     1.00     1.00
9   How do I add a custom tool to an agent with the too…   10     1.00     1.00     1.00
    mean                                                          1.00     1.00     1.00

RAGAS dataset written to examples/docs-qa/ragas.json

ctx is the number of chunks the LLM saw after reranking.
faithful is the share of the answer’s claims that those chunks support.
answer recall is the share of the ground truth’s claims that the answer states.
context recall is the share of the ground truth’s claims that those chunks support.

Before retrieval was tuned, two questions scored lower. One fix added page and title to the rerank template; see Rerank templates. The other changed the indexing prompt so code-only chunks get titles that name the API they show; see the docs-qa showcase.

Custom evaluators

Any type that implements EvaluateQuery can take the place of Ragas. The trait has one async method, evaluate, which receives a QueryEvaluation: RetrieveDocuments with the query after retrieval, or AnswerQuery with the answered query. The trait requires Clone, so keep shared state behind an Arc. This evaluator collects the questions for which the retriever found no documents:

/// Flags questions for which the retriever found nothing.
#[derive(Clone, Default)]
pub struct EmptyRetrievals {
    questions: Arc<Mutex<Vec<String>>>,
}

#[async_trait]
impl EvaluateQuery for EmptyRetrievals {
    async fn evaluate(&self, evaluation: QueryEvaluation) -> anyhow::Result<()> {
        if let QueryEvaluation::RetrieveDocuments(query) = evaluation
            && query.documents().is_empty()
        {
            self.questions
                .lock()
                .unwrap()
                .push(query.original().to_string());
        }
        Ok(())
    }
}

Source: snippets/src/querying/evaluation.rs#L80-L99

let empty = EmptyRetrievals::default();

query::Pipeline::default()
    .evaluate_with(empty.clone())
    .then_transform_query(query_transformers::Embed::from_client(openai.clone()))
    .then_retrieve(qdrant)
    .then_answer(answers::Simple::from_client(openai))
    .query("What does a retriever return?")
    .await?;

for question in empty.questions.lock().unwrap().iter() {
    println!("no documents for: {question}");
}

Source: snippets/src/querying/evaluation.rs#L105-L117

An error from evaluate fails the query, so return Ok for anything that should only be recorded.

Gotchas

evaluate_with exists only on a pending pipeline. Call it before then_retrieve; its position among query transformers does not matter.
Ragas matches each query to its row by original(). A query whose question is not in the dataset fails with Question not found, and so does query_all. Query transformers change current(), not original(), so they do not affect the match.
Rows are keyed by question text, so each question must be unique. docs-qa rejects a questions file with duplicates.
query_all returns answers in the order they finish; see Query methods. docs-qa matches each answer back to its question with original().
Without the docs-qa workaround in Retrieval and search strategies, Qdrant contexts in the export arrive Debug-quoted in 0.33.
Nine questions and one LLM judge make a regression signal for one corpus, not a benchmark. Keep the question set and the judge model fixed when you compare runs.

Reference

Models