Azure AI Foundry and Arize for Agent Observability and Evaluation

Reference: Azure AI Foundry - LangChain Integration

This notebook demonstrates how to:

1. Build a LangChain multi-chain agent on Azure AI Foundry while tracing all operations to Arize for observability
2. Leverage Azure AI Evaluators to evaluate LLM behavior
3. Log evaluation results to Arize for visibility

Prerequisites:

1. Arize AX account (Sign up for free)
2. Azure AI foundry account and project created (Sign up here)

1. Setup

2. Configure Arize Tracing

Set up OpenTelemetry instrumentation to send traces to Arize for observability.

3. Create Azure AI Agent - Poem Generator

A multi-chain agent: producer (generates content) and a verifier (validates content).

4. Run Agent on sample topics

Traces will be generated and sent to Arize on each agent run

Optional: Test evaluator call

5. Evaluate traces

Export traces from Arize and run the hate and unfairness evaluator on all rows

6. Log evaluation results back to Arize

Traces will have evaluation label, score and explanation attached

7. View Traces and Evals in Arize!

Some things to do next:

• Curate datasets to drive prompt optimization or fine tuning jobs
• Send regressions to labeling queues for human annotators to curate golden datasets
• Create custom metrics, monitors from evaluation labels