Text To SQL
Text to SQL on multi-tables database

In this cookbook we will show you how to :
- Use the function calling capabilities of Mistral models
- Build a text2SQL architecture that scales more efficiently than a naive approach where all schemas are integrally injected in the system prompt
- Evaluate your system with Mistral models and leveraging the DeepEval framework
Imports
Collecting mistralai Downloading mistralai-1.3.1-py3-none-any.whl.metadata (28 kB) Requirement already satisfied: langchain in /usr/local/lib/python3.11/dist-packages (0.3.14) Collecting deepeval Downloading deepeval-2.1.9-py3-none-any.whl.metadata (13 kB) Requirement already satisfied: eval-type-backport>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (0.2.2) Requirement already satisfied: httpx>=0.27.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (0.28.1) Collecting jsonpath-python>=1.0.6 (from mistralai) Downloading jsonpath_python-1.0.6-py3-none-any.whl.metadata (12 kB) Requirement already satisfied: pydantic>=2.9.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (2.10.5) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from mistralai) (2.8.2) Collecting typing-inspect>=0.9.0 (from mistralai) Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (6.0.2) Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.11/dist-packages (from langchain) (2.0.37) Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (3.11.11) Requirement already satisfied: langchain-core<0.4.0,>=0.3.29 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.3.29) Requirement already satisfied: langchain-text-splitters<0.4.0,>=0.3.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.3.5) Requirement already satisfied: langsmith<0.3,>=0.1.17 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.2.10) Requirement already satisfied: numpy<2,>=1.22.4 in /usr/local/lib/python3.11/dist-packages (from langchain) (1.26.4) Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.11/dist-packages (from langchain) (2.32.3) Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in /usr/local/lib/python3.11/dist-packages (from langchain) (9.0.0) Requirement already satisfied: tqdm in /usr/local/lib/python3.11/dist-packages (from deepeval) (4.67.1) Requirement already satisfied: pytest in /usr/local/lib/python3.11/dist-packages (from deepeval) (8.3.4) Requirement already satisfied: tabulate in /usr/local/lib/python3.11/dist-packages (from deepeval) (0.9.0) Requirement already satisfied: typer in /usr/local/lib/python3.11/dist-packages (from deepeval) (0.15.1) Requirement already satisfied: rich in /usr/local/lib/python3.11/dist-packages (from deepeval) (13.9.4) Requirement already satisfied: protobuf in /usr/local/lib/python3.11/dist-packages (from deepeval) (4.25.5) Requirement already satisfied: sentry-sdk in /usr/local/lib/python3.11/dist-packages (from deepeval) (2.19.2) Collecting pytest-repeat (from deepeval) Downloading pytest_repeat-0.9.3-py3-none-any.whl.metadata (4.9 kB) Collecting pytest-xdist (from deepeval) Downloading pytest_xdist-3.6.1-py3-none-any.whl.metadata (4.3 kB) Collecting portalocker (from deepeval) Downloading portalocker-3.1.1-py3-none-any.whl.metadata (8.6 kB) Collecting langchain_openai (from deepeval) Downloading langchain_openai-0.3.1-py3-none-any.whl.metadata (2.7 kB) Collecting langchain-community (from deepeval) Downloading langchain_community-0.3.15-py3-none-any.whl.metadata (2.9 kB) Collecting docx2txt~=0.8 (from deepeval) Downloading docx2txt-0.8.tar.gz (2.8 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: importlib-metadata>=6.0.2 in /usr/local/lib/python3.11/dist-packages (from deepeval) (8.5.0) Requirement already satisfied: opentelemetry-api<2.0.0,>=1.24.0 in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.29.0) Requirement already satisfied: opentelemetry-sdk<2.0.0,>=1.24.0 in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.29.0) Collecting opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0 (from deepeval) Downloading opentelemetry_exporter_otlp_proto_grpc-1.29.0-py3-none-any.whl.metadata (2.2 kB) Collecting grpcio==1.67.1 (from deepeval) Downloading grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB) Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.6.0) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (2.4.4) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.2) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (24.3.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.5.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.1.0) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (0.2.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.18.3) Requirement already satisfied: anyio in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (3.7.1) Requirement already satisfied: certifi in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (2024.12.14) Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (1.0.7) Requirement already satisfied: idna in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (3.10) Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.11/dist-packages (from httpcore==1.*->httpx>=0.27.0->mistralai) (0.14.0) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.11/dist-packages (from importlib-metadata>=6.0.2->deepeval) (3.21.0) Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (1.33) Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (24.2) Requirement already satisfied: typing-extensions>=4.7 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (4.12.2) Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.11/dist-packages (from langsmith<0.3,>=0.1.17->langchain) (3.10.14) Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from langsmith<0.3,>=0.1.17->langchain) (1.0.0) Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-api<2.0.0,>=1.24.0->deepeval) (1.2.15) Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval) (1.66.0) Collecting opentelemetry-exporter-otlp-proto-common==1.29.0 (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval) Downloading opentelemetry_exporter_otlp_proto_common-1.29.0-py3-none-any.whl.metadata (1.8 kB) Collecting opentelemetry-proto==1.29.0 (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval) Downloading opentelemetry_proto-1.29.0-py3-none-any.whl.metadata (2.3 kB) Collecting protobuf (from deepeval) Downloading protobuf-5.29.3-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes) Requirement already satisfied: opentelemetry-semantic-conventions==0.50b0 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-sdk<2.0.0,>=1.24.0->deepeval) (0.50b0) Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/dist-packages (from pydantic>=2.9.0->mistralai) (0.7.0) Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.11/dist-packages (from pydantic>=2.9.0->mistralai) (2.27.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->mistralai) (1.17.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2->langchain) (3.4.1) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2->langchain) (2.3.0) Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.11/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.1.1) Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.9.0->mistralai) Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB) Collecting dataclasses-json<0.7,>=0.5.7 (from langchain-community->deepeval) Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB) Collecting httpx-sse<0.5.0,>=0.4.0 (from langchain-community->deepeval) Downloading httpx_sse-0.4.0-py3-none-any.whl.metadata (9.0 kB) Collecting langchain Downloading langchain-0.3.15-py3-none-any.whl.metadata (7.1 kB) Collecting langchain-core<0.4.0,>=0.3.29 (from langchain) Downloading langchain_core-0.3.31-py3-none-any.whl.metadata (6.3 kB) Collecting pydantic-settings<3.0.0,>=2.4.0 (from langchain-community->deepeval) Downloading pydantic_settings-2.7.1-py3-none-any.whl.metadata (3.5 kB) Requirement already satisfied: openai<2.0.0,>=1.58.1 in /usr/local/lib/python3.11/dist-packages (from langchain_openai->deepeval) (1.59.6) Collecting tiktoken<1,>=0.7 (from langchain_openai->deepeval) Downloading tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Requirement already satisfied: iniconfig in /usr/local/lib/python3.11/dist-packages (from pytest->deepeval) (2.0.0) Requirement already satisfied: pluggy<2,>=1.5 in /usr/local/lib/python3.11/dist-packages (from pytest->deepeval) (1.5.0) Collecting execnet>=2.1 (from pytest-xdist->deepeval) Downloading execnet-2.1.1-py3-none-any.whl.metadata (2.9 kB) Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/dist-packages (from rich->deepeval) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from rich->deepeval) (2.18.0) Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.11/dist-packages (from typer->deepeval) (8.1.8) Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.11/dist-packages (from typer->deepeval) (1.5.4) Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community->deepeval) Downloading marshmallow-3.25.1-py3-none-any.whl.metadata (7.3 kB) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.11/dist-packages (from deprecated>=1.2.6->opentelemetry-api<2.0.0,>=1.24.0->deepeval) (1.17.0) Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.11/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.29->langchain) (3.0.0) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/dist-packages (from markdown-it-py>=2.2.0->rich->deepeval) (0.1.2) Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (1.9.0) Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (0.8.2) Requirement already satisfied: sniffio in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (1.3.1) Collecting python-dotenv>=0.21.0 (from pydantic-settings<3.0.0,>=2.4.0->langchain-community->deepeval) Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB) Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.11/dist-packages (from tiktoken<1,>=0.7->langchain_openai->deepeval) (2024.11.6) Downloading mistralai-1.3.1-py3-none-any.whl (261 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 261.3/261.3 kB 6.7 MB/s eta 0:00:00 Downloading deepeval-2.1.9-py3-none-any.whl (588 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 588.2/588.2 kB 16.6 MB/s eta 0:00:00 Downloading grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 38.7 MB/s eta 0:00:00 Downloading jsonpath_python-1.0.6-py3-none-any.whl (7.6 kB) Downloading opentelemetry_exporter_otlp_proto_grpc-1.29.0-py3-none-any.whl (18 kB) Downloading opentelemetry_exporter_otlp_proto_common-1.29.0-py3-none-any.whl (18 kB) Downloading opentelemetry_proto-1.29.0-py3-none-any.whl (55 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 2.9 MB/s eta 0:00:00 Downloading protobuf-5.29.3-cp38-abi3-manylinux2014_x86_64.whl (319 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 319.7/319.7 kB 18.0 MB/s eta 0:00:00 Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) Downloading langchain_community-0.3.15-py3-none-any.whl (2.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 30.3 MB/s eta 0:00:00 Downloading langchain-0.3.15-py3-none-any.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 31.7 MB/s eta 0:00:00 Downloading langchain_core-0.3.31-py3-none-any.whl (412 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 412.2/412.2 kB 18.7 MB/s eta 0:00:00 Downloading langchain_openai-0.3.1-py3-none-any.whl (54 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 kB 3.5 MB/s eta 0:00:00 Downloading portalocker-3.1.1-py3-none-any.whl (19 kB) Downloading pytest_repeat-0.9.3-py3-none-any.whl (4.2 kB) Downloading pytest_xdist-3.6.1-py3-none-any.whl (46 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.1/46.1 kB 2.2 MB/s eta 0:00:00 Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) Downloading execnet-2.1.1-py3-none-any.whl (40 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.6/40.6 kB 1.8 MB/s eta 0:00:00 Downloading httpx_sse-0.4.0-py3-none-any.whl (7.8 kB) Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB) Downloading pydantic_settings-2.7.1-py3-none-any.whl (29 kB) Downloading tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 29.4 MB/s eta 0:00:00 Downloading marshmallow-3.25.1-py3-none-any.whl (49 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 2.6 MB/s eta 0:00:00 Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB) Building wheels for collected packages: docx2txt Building wheel for docx2txt (setup.py) ... done Created wheel for docx2txt: filename=docx2txt-0.8-py3-none-any.whl size=3960 sha256=7b726a0cbe5588eb27a3c2b8518f75ee7ca8ac8380573c834740aee90d33bdcd Stored in directory: /root/.cache/pip/wheels/0f/0e/7a/3094a4ceefe657bff7e12dd9592a9d5b6487ef4338ace0afa6 Successfully built docx2txt Installing collected packages: docx2txt, python-dotenv, protobuf, portalocker, mypy-extensions, marshmallow, jsonpath-python, httpx-sse, grpcio, execnet, typing-inspect, tiktoken, pytest-xdist, pytest-repeat, opentelemetry-proto, pydantic-settings, opentelemetry-exporter-otlp-proto-common, mistralai, dataclasses-json, langchain-core, opentelemetry-exporter-otlp-proto-grpc, langchain_openai, langchain, langchain-community, deepeval Attempting uninstall: protobuf Found existing installation: protobuf 4.25.5 Uninstalling protobuf-4.25.5: Successfully uninstalled protobuf-4.25.5 Attempting uninstall: grpcio Found existing installation: grpcio 1.69.0 Uninstalling grpcio-1.69.0: Successfully uninstalled grpcio-1.69.0 Attempting uninstall: langchain-core Found existing installation: langchain-core 0.3.29 Uninstalling langchain-core-0.3.29: Successfully uninstalled langchain-core-0.3.29 Attempting uninstall: langchain Found existing installation: langchain 0.3.14 Uninstalling langchain-0.3.14: Successfully uninstalled langchain-0.3.14 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow 2.17.1 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 5.29.3 which is incompatible. Successfully installed dataclasses-json-0.6.7 deepeval-2.1.9 docx2txt-0.8 execnet-2.1.1 grpcio-1.67.1 httpx-sse-0.4.0 jsonpath-python-1.0.6 langchain-0.3.15 langchain-community-0.3.15 langchain-core-0.3.31 langchain_openai-0.3.1 marshmallow-3.25.1 mistralai-1.3.1 mypy-extensions-1.0.0 opentelemetry-exporter-otlp-proto-common-1.29.0 opentelemetry-exporter-otlp-proto-grpc-1.29.0 opentelemetry-proto-1.29.0 portalocker-3.1.1 protobuf-5.29.3 pydantic-settings-2.7.1 pytest-repeat-0.9.3 pytest-xdist-3.6.1 python-dotenv-1.0.1 tiktoken-0.8.0 typing-inspect-0.9.0
Load the Chinook database
"Chinook is a sample database available for SQL Server, Oracle, MySQL, etc. It can be created by running a single SQL script. Chinook database is an alternative to the Northwind database, being ideal for demos and testing ORM tools targeting single and multiple database servers."
To run this notebook you will need to download the Chinook datase. You will find more information about this database by clicking on this github link.
To create the Chinook.db in the same directory as this notebook you have several options :
- You can download and build the database via the command line :
curl -s https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql | sqlite3 Chinook.db
- Another strategy consists in running the following script
https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql.
Firstly save the script to a folder/directory on your computer. Then create a database called Chinook with sqlite3 Chinook.db. Ultimately run the script with the command .read Chinook_Sqlite.sql

Reading package lists... Done Building dependency tree... Done Reading state information... Done Suggested packages: sqlite3-doc The following NEW packages will be installed: sqlite3 0 upgraded, 1 newly installed, 0 to remove and 49 not upgraded. Need to get 768 kB of archives. After this operation, 1,873 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 sqlite3 amd64 3.37.2-2ubuntu0.3 [768 kB] Fetched 768 kB in 1s (799 kB/s) debconf: unable to initialize frontend: Dialog debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 1.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Selecting previously unselected package sqlite3. (Reading database ... 124565 files and directories currently installed.) Preparing to unpack .../sqlite3_3.37.2-2ubuntu0.3_amd64.deb ... Unpacking sqlite3 (3.37.2-2ubuntu0.3) ... Setting up sqlite3 (3.37.2-2ubuntu0.3) ... Processing triggers for man-db (2.10.2-1) ...
SQLite version 3.37.2 2022-01-06 13:25:41 Enter ".help" for usage hints. sqlite> .tables Album Employee InvoiceLine PlaylistTrack Artist Genre MediaType Track Customer Invoice Playlist sqlite> SELECT * FROM Artist LIMIT 5; 1|AC/DC 2|Accept 3|Aerosmith 4|Alanis Morissette 5|Alice In Chains sqlite> .exit
Set up clients
Type your API Key··········
Interract with the Chinook database
We are defining two functions :
- run_sql_query that runs sql code on Chinook
- get_sql_schema_of_table that returns the schema of a table specified as input
"[(1, 'AC/DC'), (2, 'Accept'), (3, 'Aerosmith'), (4, 'Alanis Morissette'), (5, 'Alice In Chains'), (6, 'Antônio Carlos Jobim'), (7, 'Apocalyptica'), (8, 'Audioslave'), (9, 'BackBeat'), (10, 'Billy Cobham')]"
Build agent
Test the agent
Let's test the agent and ask a few random questions of increasing complexity
User: What is the oldest player in the NBA? Assistant: I'm sorry, but I can only answer questions related to the Chinook database, which is a digital media store. I don't have information about the NBA or any other topics outside of the Chinook database. Please ask another question.
User: What are the genre in the store? Tool: Getting SQL schema of table Genre Tool: Running code SELECT Name FROM Genre Assistant: Here are the genres available in the store: 1. Rock 2. Jazz 3. Metal 4. Alternative & Punk 5. Rock And Roll 6. Blues 7. Latin 8. Reggae 9. Pop 10. Soundtrack 11. Bossa Nova 12. Easy Listening 13. Heavy Metal 14. R&B/Soul 15. Electronica/Dance 16. World 17. Hip Hop/Rap 18. Science Fiction 19. TV Shows 20. Sci Fi & Fantasy 21. Drama 22. Comedy 23. Alternative 24. Classical 25. Opera
User: What are the albums of the rock band U2? Tool: Getting SQL schema of table Album Tool: Getting SQL schema of table Artist Tool: Running code SELECT Title FROM Album WHERE ArtistId = (SELECT ArtistId FROM Artist WHERE Name = 'U2'); Assistant: The albums of the rock band U2 are : - Achtung Baby - All That You Can't Leave Behind - B-Sides 1980-1990 - How To Dismantle An Atomic Bomb - Pop - Rattle And Hum - The Best Of 1980-1990 - War - Zooropa - Instant Karma: The Amnesty International Campaign to Save Darfur
User: What is the shortest song that the rock band U2 ever composed? Tool: Getting SQL schema of table Artist Tool: Getting SQL schema of table Album Tool: Getting SQL schema of table Track Tool: Running code SELECT Name, Milliseconds FROM Track WHERE Composer = 'U2' ORDER BY Milliseconds ASC LIMIT 1; Assistant: The shortest song that the rock band U2 ever composed is "Trash, Trampoline And The Party Girl".
User: Which track from U2 is the most sold? Tool: Getting SQL schema of table Track Tool: Getting SQL schema of table Artist Tool: Getting SQL schema of table InvoiceLine Tool: Running code SELECT Track.Name, SUM(InvoiceLine.Quantity) as TotalSold FROM Track JOIN InvoiceLine ON Track.TrackId = InvoiceLine.TrackId JOIN Album ON Track.AlbumId = Album.AlbumId JOIN Artist ON Album.ArtistId = Artist.ArtistId WHERE Artist.Name = 'U2' GROUP BY Track.Name ORDER BY TotalSold DESC LIMIT 1; Assistant: The most sold track from U2 is "Sweetest Thing" with 3 copies sold.
User: Which consumer bought the biggest amound of U2 songs? Tool: Getting SQL schema of table Track Tool: Getting SQL schema of table InvoiceLine Tool: Getting SQL schema of table Invoice Tool: Getting SQL schema of table Customer Tool: Getting SQL schema of table Artist Tool: Running code SELECT Customer.CustomerId, Customer.FirstName, Customer.LastName, SUM(InvoiceLine.Quantity) as total_quantity FROM Customer JOIN Invoice ON Customer.CustomerId = Invoice.CustomerId JOIN InvoiceLine ON Invoice.InvoiceId = InvoiceLine.InvoiceId JOIN Track ON InvoiceLine.TrackId = Track.TrackId JOIN Album ON Track.AlbumId = Album.AlbumId JOIN Artist ON Album.ArtistId = Artist.ArtistId WHERE Artist.Name = 'U2' GROUP BY Customer.CustomerId ORDER BY total_quantity DESC LIMIT 1; Assistant: The consumer who bought the biggest amount of U2 songs is Madalena Sampaio. She bought 9 songs.
User: List all artist that have a color in their name Tool: Getting SQL schema of table Artist Tool: Running code SELECT Name FROM Artist WHERE Name LIKE '%red%' OR Name LIKE '%blue%' OR Name LIKE '%green%' OR Name LIKE '%yellow%' OR Name LIKE '%black%' OR Name LIKE '%white%' OR Name LIKE '%purple%' OR Name LIKE '%pink%' OR Name LIKE '%orange%' OR Name LIKE '%brown%' OR Name LIKE '%grey%' OR Name LIKE '%gray%'; Assistant: Here are the artists that have a color in their name: 1. Black Label Society 2. Black Sabbath 3. Pedro Luís & A Parede 4. Banda Black Rio 5. Green Day 6. Deep Purple 7. James Brown 8. Pink Floyd 9. Red Hot Chili Peppers 10. Simply Red 11. The Black Crowes 12. Whitesnake 13. Jackson Browne 14. Black Eyed Peas 15. Charlie Brown Jr. 16. Pedro Luís E A Parede
User: Who are our top Customers according to Invoices? Tool: Getting SQL schema of table Customer Tool: Getting SQL schema of table Invoice Tool: Running code SELECT Customer.CustomerId, Customer.FirstName, Customer.LastName, SUM(Invoice.Total) as TotalSpent FROM Customer JOIN Invoice ON Customer.CustomerId = Invoice.CustomerId GROUP BY Customer.CustomerId ORDER BY TotalSpent DESC LIMIT 10; Assistant: Here are our top 10 customers according to invoices : 1. Helena Holý : 49.62 2. Richard Cunningham : 47.62 3. Luis Rojas : 46.62 4. Ladislav Kovács : 45.62 5. Hugh O'Reilly : 45.62 6. Julia Barnett : 43.62 7. Frank Ralston : 43.62 8. Fynn Zimmermann : 43.62 9. Astrid Gruber : 42.62 10. Victor Stevens : 42.62
Evaluating
Let's try to evaluate the agent in a more formal way.
We will build a test set based on the questions from this Medium article Chinook question/answers
We will evaluate answers via LLM as a judge through the framework DeepEval from which the image here below is taken.

Event loop is already running. Applying nest_asyncio patch to allow async execution...
Evaluating 7 test case(s) in parallel: |██████████|100% (7/7) [Time Taken: 01:11, 10.16s/test case]
====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 1.0, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and directly answers the question., error: None) For test case: - input: Which Employee has the Highest Total Number of Customers? - actual output: The employee with the highest total number of customers is Jane Peacock, with 21 customers. - expected output: Peacock Jane has the most customers (she has 21 customers) - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies the top customers and their spending amounts, aligning with the facts in the expected output. However, it does not directly answer the question by explicitly stating the top five customers, which is a minor deviation from the expected output., error: None) For test case: - input: Who are our top Customers according to Invoices? - actual output: Here are our top customers according to invoices : | CustomerId | FirstName | LastName | TotalSpent | |------------|-----------|------------|------------| | 6 | Helena | Holý | 49.62 | | 26 | Richard | Cunningham | 47.62 | | 57 | Luis | Rojas | 46.62 | | 45 | Ladislav | Kovács | 45.62 | | 46 | Hugh | O'Reilly | 45.62 | | 28 | Julia | Barnett | 43.62 | | 24 | Frank | Ralston | 43.62 | | 37 | Fynn | Zimmermann | 43.62 | | 7 | Astrid | Gruber | 42.62 | | 25 | Victor | Stevens | 42.62 | - expected output: Helena Holy, Richard Cunningham, Luis Rojas, Ladislav Kovacs, and Hugh O’Reilly are the top five customers who have spent the highest amount of money according to the invoice - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.9, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output directly answers the question and does not contradict the expected output., error: None) For test case: - input: How many Rock music listeners are there? - actual output: There are 59 Rock music listeners. - expected output: We found out that all 59 customers in the database have listened to Rock Music. - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and directly answers the question. However, the format and some details differ from the expected output., error: None) For test case: - input: What artists have written most rock music songs? - actual output: The artists who have written the most rock music songs are: 1. **Led Zeppelin** with 114 songs 2. **U2** with 112 songs 3. **Deep Purple** with 92 songs 4. **Iron Maiden** with 81 songs 5. **Pearl Jam** with 54 songs 6. **Van Halen** with 52 songs 7. **Queen** with 45 songs 8. **The Rolling Stones** with 41 songs 9. **Creedence Clearwater Revival** with 40 songs 10. **Kiss** with 35 songs - expected output: Led Zeppelin tops the list of Artists who have written the most Rock Music with 114 songs followed Closely by U2 with 112 music. - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies Iron Maiden as the artist who earned the most, matching the expected output. However, it uses 'approximately' and 'units of currency' instead of the exact amount and currency symbol '$' as in the expected output., error: None) For test case: - input: Which artist has earned the most according to the Invoice Lines? How much is it? - actual output: The artist who earned the most according to the invoice lines is Iron Maiden, with a total earning of approximately 138.60 units of currency. - expected output: The Artist who has earned the most according to the invoice lines is Iron Maiden with a total of $138.6. - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and answers the question. However, it lacks additional details provided in the expected output, such as the total number of songs and the average music length., error: None) For test case: - input: How many tracks have a song length greater than the average song length? - actual output: There are 494 tracks with a song length greater than the average song length. - expected output: Out of 3503 songs in the database, we found out that 494 of these songs have length more than the average music length of 393,599.21 milliseconds. - context: None - retrieval context: None ====================================================================== Metrics Summary - ✅ Correctness (GEval) (score: 0.5, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies Rock as the most popular genre, matching the expected output. However, it provides additional details about other genres and their sales, which are not present in the expected output. It answers the question, but the additional information is not required., error: None) For test case: - input: What is the most popular genre for Australia? - actual output: The most popular genre in Australia is Rock, with a total sales of approximately $21.78. This is followed by Metal with approximately $7.92 in sales. Other genres such as Heavy Metal, Reggae, Latin, and Blues have lower sales ranging from approximately $3.00 to $1.00. - expected output: Rock is the most popular song for Australia - context: None - retrieval context: None ====================================================================== Overall Metric Pass Rates Correctness (GEval): 100.00% pass rate ======================================================================
EvaluationResult(test_results=[TestResult(name='test_case_0', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=1.0, reason='The actual output does not contradict any facts in the expected output and directly answers the question.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Which Employee has the Highest Total Number of Customers?', actual_output='The employee with the highest total number of customers is Jane Peacock, with 21 customers.', expected_output='Peacock Jane has the most customers (she has 21 customers)', context=None, retrieval_context=None), TestResult(name='test_case_1', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output correctly identifies the top customers and their spending amounts, aligning with the facts in the expected output. However, it does not directly answer the question by explicitly stating the top five customers, which is a minor deviation from the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Who are our top Customers according to Invoices?', actual_output="Here are our top customers according to invoices :\n\n| CustomerId | FirstName | LastName | TotalSpent |\n|------------|-----------|------------|------------|\n| 6 | Helena | Holý | 49.62 |\n| 26 | Richard | Cunningham | 47.62 |\n| 57 | Luis | Rojas | 46.62 |\n| 45 | Ladislav | Kovács | 45.62 |\n| 46 | Hugh | O'Reilly | 45.62 |\n| 28 | Julia | Barnett | 43.62 |\n| 24 | Frank | Ralston | 43.62 |\n| 37 | Fynn | Zimmermann | 43.62 |\n| 7 | Astrid | Gruber | 42.62 |\n| 25 | Victor | Stevens | 42.62 |", expected_output='Helena Holy, Richard Cunningham, Luis Rojas, Ladislav Kovacs, and Hugh O’Reilly are the top five customers who have spent the highest amount of money according to the invoice', context=None, retrieval_context=None), TestResult(name='test_case_2', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.9, reason='The actual output directly answers the question and does not contradict the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='How many Rock music listeners are there?', actual_output='There are 59 Rock music listeners.', expected_output='We found out that all 59 customers in the database have listened to Rock Music.', context=None, retrieval_context=None), TestResult(name='test_case_3', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output does not contradict any facts in the expected output and directly answers the question. However, the format and some details differ from the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='What artists have written most rock music songs?', actual_output='The artists who have written the most rock music songs are:\n\n1. **Led Zeppelin** with 114 songs\n2. **U2** with 112 songs\n3. **Deep Purple** with 92 songs\n4. **Iron Maiden** with 81 songs\n5. **Pearl Jam** with 54 songs\n6. **Van Halen** with 52 songs\n7. **Queen** with 45 songs\n8. **The Rolling Stones** with 41 songs\n9. **Creedence Clearwater Revival** with 40 songs\n10. **Kiss** with 35 songs', expected_output='Led Zeppelin tops the list of Artists who have written the most Rock Music with 114 songs followed Closely by U2 with 112 music.', context=None, retrieval_context=None), TestResult(name='test_case_4', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason="The actual output correctly identifies Iron Maiden as the artist who earned the most, matching the expected output. However, it uses 'approximately' and 'units of currency' instead of the exact amount and currency symbol '$' as in the expected output.", strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Which artist has earned the most according to the Invoice Lines? How much is it?', actual_output='The artist who earned the most according to the invoice lines is Iron Maiden, with a total earning of approximately 138.60 units of currency.', expected_output='The Artist who has earned the most according to the invoice lines is Iron Maiden with a total of $138.6.', context=None, retrieval_context=None), TestResult(name='test_case_5', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output does not contradict any facts in the expected output and answers the question. However, it lacks additional details provided in the expected output, such as the total number of songs and the average music length.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='How many tracks have a song length greater than the average song length?', actual_output='There are 494 tracks with a song length greater than the average song length.', expected_output='Out of 3503 songs in the database, we found out that 494 of these songs have length more than the average music length of 393,599.21 milliseconds.', context=None, retrieval_context=None), TestResult(name='test_case_6', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.5, reason='The actual output correctly identifies Rock as the most popular genre, matching the expected output. However, it provides additional details about other genres and their sales, which are not present in the expected output. It answers the question, but the additional information is not required.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='What is the most popular genre for Australia?', actual_output='The most popular genre in Australia is Rock, with a total sales of approximately $21.78. This is followed by Metal with approximately $7.92 in sales. Other genres such as Heavy Metal, Reggae, Latin, and Blues have lower sales ranging from approximately $3.00 to $1.00.', expected_output='Rock is the most popular song for Australia', context=None, retrieval_context=None)], confident_link=None)