Notebooks
A
Arize AI Arize Ax Tracing Tutorial

Arize Ax Tracing Tutorial

arize-tutorialscookbooksPython
alph-notebooks/arize-tutorials / arize_ax_tracing_tutorial.ipynb

Arize AX Tracing Tutorial: Building a Customer Support Agent

In this tutorial, you'll learn how to instrument an LLM application with complete observability using Arize AX. We'll build a customer support chatbot called SupportBot that handles order status queries and FAQ questions.

By the end of this tutorial, you'll be able to:

  • Capture complete execution traces for LLM calls, tool invocations, and RAG pipelines
  • Collect and log user feedback
  • Run automated LLM-as-Judge evaluations
  • Track multi-turn conversations with session management

Prerequisites

Before starting, make sure you have:

  • An Arize AX account (sign up for free at app.arize.com)
  • Your Space ID and API Key (found in Settings → API Keys)
  • OpenAI API key (or another supported LLM provider)

Part 1: Setup and Installation

First, let's install the required dependencies for tracing with Arize AX.

[1]
Note: you may need to restart the kernel to use updated packages.

Part 2: Configure Tracing

Next, we'll set up the connection to Arize AX and instrument OpenAI to automatically capture traces.

[ ]
[ ]
🔭 OpenTelemetry Tracing Details 🔭
|  Arize Project: my-support-bot
|  Span Processor: BatchSpanProcessor
|  Collector Endpoint: otlp.arize.com
|  Transport: gRPC
|  Transport Headers: {'authorization': '****', 'api_key': '****', 'arize-space-id': '****', 'space_id': '****', 'arize-interface': '****'}
|  
|  Using a default SpanProcessor. `add_span_processor` will overwrite this default.
|  
|  `register` has set this TracerProvider as the global OpenTelemetry default.
|  To disable this behavior, call `register` with `set_global_tracer_provider=False`.

That's it! With just these few lines, OpenInference will automatically capture traces for all your OpenAI calls. The instrumentation handles all the OpenTelemetry boilerplate for you.

Part 3: Your First Trace - Query Classification

Let's start by creating a simple query classifier that determines if a user message is about order status or a general FAQ.

[4]
[5]
Classification: ORDER_STATUS

Head to your Arize AX dashboard, and you'll see this trace appear in real-time! You can see:

  • The parent "classify-query" span
  • The automatically captured OpenAI call details
  • Input and output values

Part 4: Tool Call Tracing

Now let's add the ability to look up order status using OpenAI's function calling.

[8]
[9]
The status of your order number 12345 is "In Transit". The estimated delivery date is 15th March, 2024.

In Arize AX, you'll now see a complete trace tree showing:

  • The parent "handle-order-query" span
  • The LLM call that decided to use a tool
  • The "execute-tool" span with parameters and results
  • The final LLM call that formulated the response

Part 5: RAG Tracing

For FAQ queries, we'll use a simple RAG (Retrieval-Augmented Generation) pipeline. Let's trace both the retrieval and generation steps.

[12]
[13]
Our shipping policy is that we offer free shipping on orders over $50.

Perfect! Now you can see exactly:

  • What query was sent to retrieval
  • Which documents were retrieved
  • What context was passed to the LLM
  • What answer was generated

Part 6: Complete SupportBot

Let's put it all together into a complete SupportBot with full tracing.

[14]
[15]

User: Where is my order #12345?
Bot: Your order #12345 is currently in transit. The estimated delivery date is March 15, 2024.

User: What's your return policy?
Bot: Our return policy allows for returns within 30 days of purchase.

User: Can I track my order?
Bot: Of course, I'd be happy to help you with that. Could you please provide me with your order ID?

Annotations & Evaluations

Part 7: Automated Evaluations

Manual feedback doesn't scale. Let's build automated evaluators using LLM-as-Judge to evaluate thousands of traces.

[16]
[20]
  arize._exporter.client | INFO | Fetching data...

  exporting 40 rows: 100%|█████████████████████████| 40/40 [00:00, 436.27 row/s]
Evaluated 3 tool calls
[21]
  arize.utils.arrow | INFO | ✅ Success! Check out your data at https://app.arize.com/organizations/QWNjb3VudE9yZ2FuaXphdGlvbjoxMzYwMDpZbVlu/spaces/U3BhY2U6MTQyNTM6a3pZWA==/models/modelName/my-support-bot?selectedTab=llmTracing

Part 9: Session Tracking for Multi-Turn Conversations

Real conversations involve multiple turns. Let's add session tracking to group related traces.

[22]
[23]
=== Conversation Started ===
User: Where is my order?
Bot: To help you with your request, could you kindly provide the order ID of your package?

User: It's order 12345
Bot: Your order is currently in transit. The estimated delivery is on the 15th of March, 2024. If you have any more questions or need further assistance, feel free to ask.

User: When will it arrive?
Bot: Your order is still in transit and is estimated to arrive on March 15th, 2024.

In Arize AX, all these traces will be grouped together under the same session ID, allowing you to view the complete conversation thread.