Tracing and Evaluating Gemini Audio

This notebook is adapted from Google's "Gemini API: Audio Quickstart Notebook" and provides an example of how to prompt Gemini Flash using an audio file.

In this case, you'll use a sound recording of President John F. Kennedy’s 1961 State of the Union address.

This notebook performs the following tasks:

1. Prompt Gemini to generate a transcript of the audio recording.
2. Trace Gemini API calls and send the traces to the Arize platform with links to audio file for playback.
3. Evaluate the transcription output from Gemini for sentiment analysis using Phoenix Evals and Gemini LLM (LLM as a Judge).

Install dependencies

Configure your Gemini API key

To run the following cell, your API key must be stored it in a Colab Secret named GEMINI_API_KEY. If you don't already have an API key, or you're not sure how to create a Colab Secret, see Authentication for an example.

Load an audio file sample and set the URL

Tracing setup

You'll need to set Arize AX variables (Space id, API key and Developer Key) below to send traces to the Arize AX Platform. Sign up for free here.

Configure prompt

Call Gemini

Evaluate Gemini's output transcript for sentiment analysis

First, export spans from Arize that contain transcript output from Arize

Evaluation Template

Evaluate transcriptions using Gemini as a LLM as a Judge

Send evaluations to Arize

Next Steps

Useful API references:

More details about Gemini API's vision capabilities in the documentation.

If you want to know about the File API, check its API reference or the File API quickstart.

Related examples

Check this example using the audio files to give you more ideas on what the gemini API can do with them:

• Share Voice memos with Gemini API and brainstorm ideas

Continue your discovery of the Gemini API

Have a look at the Audio quickstart to learn about another type of media file, then learn more about prompting with media files in the docs, including the supported formats and maximum length for audio files. .