Tracing and Evaluating a Haystack RAG Application with Phoenix

Phoenix is a tool for tracing and evaluating LLM applications. In this tutorial, we will trace and evaluate a Haystack RAG pipeline. We'll evaluate using three different types of evaluations:

1. Relevance: Whether the retrieved documents are relevant to the question.
2. Q&A Correctness: Whether the answer to the question is correct.
3. Hallucination: Whether the answer contains hallucinations.

ℹ️ This notebook requires an OpenAI API key.

Set API Keys

Launch Phoenix and Enable Haystack Tracing

If you don't have a Phoenix API key, you can get one for free at phoenix.arize.com. Arize Phoenix also provides self-hosting options if you'd prefer to run the application yourself instead.

The command below connects Phoenix to your Haystack application and instruments the Haystack library. Any calls to Haystack pipelines from this point forward will be traced and logged to the Phoenix UI.

Set up your Haystack app

For a step-by-step guide to create a RAG pipeline with Haystack, follow the Creating Your First QA Pipeline with Retrieval-Augmentation tutorial.

Run the pipeline with a query. It will automatically create a trace on Phoenix.

Evaluating Retrieved Docs

Now that we've traced our pipeline, let's start by evaluating the retrieved documents.

All evaluations in Phoenix use the same general process:

1. Query and download trace data from Phoenix
2. Add evaluation labels to the trace data. This can be done using the Phoenix library, using Haystack evaluators, or using your own evaluators.
3. Log the evaluation labels to Phoenix
4. View evaluations

We'll use the get_retrieved_documents function to get the trace data for the retrieved documents.

Next we'll use Phoenix's RelevanceEvaluator to evaluate the relevance of the retrieved documents. This evaluator uses a LLM to determine if the retrieved documents contain the answer to the question.

Finally, we'll log the evaluation labels to Phoenix.

If you now click on your document retrieval span in Phoenix, you should see the evaluation labels.

Evaluate Response

With HallucinationEvaluator and QAEvaluator, we can detect the correctness and hallucination score of the generated response.

You should now see the Q&A correctness and hallucination evaluations in Phoenix.