Tracing Llama 3.2 with the OpenAI API

This guide demonstrates how to use trace open-source models like Llama 3.2, utilizing the OpenAI API.

To instrument an open-source Llama model, Ollama has built-in compatibility with the OpenAI Chat Completions API, making it possible to use more tooling and applications with open-source models locally.

Installing Ollama

Download and execute the installation script from the Ollama website. The script will handle the installation process automatically, including downloading and installing necessary dependencies.

Launching Xterm

Launch the xterm terminal within the Colab.

Launch Terminal & Start the Ollama Server

Once Ollama is installed and the terminal is running, we can start the server using the following command. Be sure to run this in the xterm terminal below!

ollama serve &

The & at the end runs the command in the background, allowing you to continue using your terminal.

Import Libraries

Download Llama 3.2

Using the ollama library , we can request the llama3.2:1b model to run in Colab.

Register OTEL

Create OpenAI Client

Run Queries

Run queries against llama3.2:1b, using the OpenAI API