3 Model Evaluation
gpu-accelerationtool-callingretrieval-augmented-generationllm-inferencetensorrtnvidia-generative-ai-exampleslarge-language-modelsmicroservicetriton-inference-serverLLMdata-flywheelragnemo
Export
Part III: Model Evaluation Using NeMo Evaluator
This notebook covers the following:
- Pre-requisites: Configurations and Health Checks
- Establish a baseline accuracy benchmark. This uses the off-the-shelf llama-3.2-1b-instruct model
- Evaluate the LoRA customized model
[1]
Before you proceed, make sure that you completed the previous notebooks on data preparation and model fine-tuning to obtain the assets required to follow along.
Configure NeMo Microservices Endpoints
The following code imports necessary configurations and prints the endpoints for the NeMo Data Store, Entity Store, Customizer, Evaluator, and NIM, as well as the namespace and base model.
[3]
[4]
Data Store endpoint: http://data-store.test Entity Store, Customizer, Evaluator endpoint: http://nemo.test NIM endpoint: http://nim.test Namespace: xlam-tutorial-ns Base Model: meta/llama-3.2-1b-instruct
Check Available Models
Specify the customized model name that you got from the previous notebook to the following variable.
[5]
The following code checks if the NIM endpoint hosts the model properly.
[6]
Verify the Availability of the Datasets
In the previous notebook, we uploaded the test dataset along with the train and validation sets using nemo_client.datasets.create(name=DATASET_NAME, namespace=NMS_NAMESPACE, ...).
The following code performs a sanity check to validate the dataset by retrieving it and printing the URL of the files in the dataset.
[7]
Files URL: hf://datasets/xlam-tutorial-ns/xlam-ft-dataset
1.1: Create an Evaluation Config Object
Create an evaluation configuration object for NeMo Evaluator. For more information on various parameters, refer to the NeMo Evaluator configuration in the NeMo microservices documentation.
- The
tasks.custom-tool-calling.dataset.files_urlis used to indicate which test file to use. Note that it's required to upload this to the NeMo Data Store and register with Entity store before using. - The
tasks.dataset.limitargument below specifies how big a subset of test data to run the evaluation on - The evaluation metric
tasks.metrics.tool-calling-accuracyreportsfunction_name_accuracyandfunction_name_and_args_accuracynumbers, which are as their names imply.
[8]
1.2: Launch Evaluation Job
The following code calls the nemo_client.evaluation.jobs.create() method to launch an evaluation job in the NeMo Evaluator.
It uses the evaluation configuration defined in the previous cell and targets the base model.
[9]
Created evaluation job: eval-6QisJKdfmRquYXdFBNBAD7
EvaluationJob(config=Config(type='custom', id='eval-config-UEphFyZQvSb5rFAkPuRYuu', created_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493465), custom_fields={}, description=None, groups=None, name='eval-config-UEphFyZQvSb5rFAkPuRYuu', namespace='default', ownership=None, params=None, project=None, schema_version='1.0', tasks={'custom-tool-calling': ConfigTasks(type='chat-completion', dataset=ConfigTasksDatasetDatasetEv(files_url='hf://datasets/xlam-tutorial-ns/xlam-ft-dataset/testing/xlam-test-single.jsonl', id='dataset-UAEo4C8C4p6uEqTFTn2v8B', created_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493606), custom_fields={}, description=None, format=None, hf_endpoint=None, limit=50, name='dataset-UAEo4C8C4p6uEqTFTn2v8B', namespace='default', ownership=None, project=None, schema_version='1.0', split=None, type_prefix=None, updated_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493607), version_id='main', version_tags=[]), metrics={'tool-calling-accuracy': ConfigTasksMetrics(type='tool-calling', params={'tool_calls_ground_truth': '{{ item.tool_calls | tojson }}'})}, params={'template': {'messages': '{{ item.messages | tojson}}', 'tools': '{{ item.tools | tojson }}', 'tool_choice': 'auto'}})}, type_prefix='eval-config', updated_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493467)), target=Target(type='model', id='eval-target-N8uf4rSaNEAwJ1Kj7YWNve', cached_outputs=None, created_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493687), custom_fields={}, dataset=None, description=None, model='meta/llama-3.2-1b-instruct', name='eval-target-N8uf4rSaNEAwJ1Kj7YWNve', namespace='default', ownership=None, project=None, rag=None, retriever=None, rows=None, schema_version='1.0', type_prefix='eval-target', updated_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493688)), id='eval-6QisJKdfmRquYXdFBNBAD7', created_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493989), custom_fields={}, description=None, namespace='default', output_files_url=None, ownership=None, project=None, result=None, status='created', status_details={'message': None, 'task_status': {}, 'progress': None}, updated_at=datetime.datetime(2025, 6, 20, 16, 36, 27, 493990)) The following code defines a helper function to poll on job status until it finishes:
[10]
Run the helper function:
[11]
Job status: running after 5.03 seconds. Progress: 0.0% Job status: running after 10.05 seconds. Progress: 0.0% Job status: running after 15.06 seconds. Progress: 0.0% Job status: running after 20.08 seconds. Progress: 0.0% Job status: running after 25.09 seconds. Progress: 8.0% Job status: running after 30.11 seconds. Progress: 12.0% Job status: running after 35.12 seconds. Progress: 16.0% Job status: running after 40.13 seconds. Progress: 26.0% Job status: running after 45.15 seconds. Progress: 26.0% Job status: running after 50.16 seconds. Progress: 26.0% Job status: running after 55.18 seconds. Progress: 26.0% Job status: running after 60.20 seconds. Progress: 26.0% Job status: running after 65.21 seconds. Progress: 26.0% Job status: running after 70.23 seconds. Progress: 32.0% Job status: running after 75.24 seconds. Progress: 32.0% Job status: running after 80.26 seconds. Progress: 38.0% Job status: running after 85.28 seconds. Progress: 38.0% Job status: running after 90.29 seconds. Progress: 38.0% Job status: running after 95.31 seconds. Progress: 40.0% Job status: running after 100.32 seconds. Progress: 48.0% Job status: running after 105.34 seconds. Progress: 50.0% Job status: running after 110.35 seconds. Progress: 60.0% Job status: running after 115.37 seconds. Progress: 60.0% Job status: running after 120.38 seconds. Progress: 60.0% Job status: running after 125.40 seconds. Progress: 60.0% Job status: running after 130.42 seconds. Progress: 60.0% Job status: running after 135.43 seconds. Progress: 62.0% Job status: running after 140.45 seconds. Progress: 62.0% Job status: running after 145.46 seconds. Progress: 62.0% Job status: running after 150.48 seconds. Progress: 62.0% Job status: running after 155.50 seconds. Progress: 62.0% Job status: running after 160.51 seconds. Progress: 64.0% Job status: running after 165.53 seconds. Progress: 64.0% Job status: running after 170.55 seconds. Progress: 68.0% Job status: running after 175.56 seconds. Progress: 72.0% Job status: running after 180.58 seconds. Progress: 82.0% Job status: running after 185.60 seconds. Progress: 84.0% Job status: running after 190.61 seconds. Progress: 84.0% Job status: running after 195.63 seconds. Progress: 84.0% Job status: running after 200.65 seconds. Progress: 88.0% Job status: running after 205.67 seconds. Progress: 88.0% Job status: running after 210.68 seconds. Progress: 92.0% Job status: running after 215.70 seconds. Progress: 92.0% Job status: running after 220.71 seconds. Progress: 92.0% Job status: running after 225.73 seconds. Progress: 94.0% Job status: running after 230.75 seconds. Progress: 100.0% Job status: completed after 235.77 seconds. Progress: 100%
1.3 Review Evaluation Metrics
The following code retrieves the evaluation results for the base evaluation job.
[13]
{
"job": "eval-6QisJKdfmRquYXdFBNBAD7",
"id": "evaluation_result-B8PqbKeE9KrzMUxGRCcpF2",
"created_at": "2025-06-20T16:36:27.527337",
"custom_fields": {},
"groups": {},
"namespace": "default",
"tasks": {
"custom-tool-calling": {
"metrics": {
"tool-calling-accuracy": {
"scores": {
"function_name_accuracy": {
"value": 0.12,
"stats": {
"count": 50,
"mean": 0.16,
"sum": 8.0
}
},
"function_name_and_args_accuracy": {
"value": 0.08,
"stats": {
"count": 50,
"mean": 0.12,
"sum": 6.0
}
}
}
}
}
}
},
"updated_at": "2025-06-20T16:36:27.527338"
}
The following code extracts and prints the accuracy scores for the base model.
[14]
Base model: function_name_accuracy: 0.12 Base model: function_name_and_args_accuracy: 0.08
Without any finetuning, the meta/llama-3.2-1b-instruct model should score in the ballpark of about 12% in function_name_accuracy, and 8% in function_name_and_args_accuracy (note that scores will vary by about +/-4% due to non-determinism of LLMs).
(Optional) 1.4 Download and Inspect Results
To take a deeper look into the model's generated outputs, you can download and review the results.
[32]
[33]
Evaluation results for job eval-6QisJKdfmRquYXdFBNBAD7 downloaded successfully to eval-6QisJKdfmRquYXdFBNBAD7.zip.
You can inspect the downloaded results file to observe places where the base model errors. Without any fine-tuning, some models not only return inaccurate function names and arguments, but they may not adhere to a consistent structured / predictable output schema. This makes it difficult to automatically parse these outputs, deterring integration with external systems.
2.1 Launch Evaluation Job
Run another evaluation job with the same evaluation config but with the customized model.
[34]
Created evaluation job for customized model: eval-QbibneF16tyDiCJWc9oxgZ
EvaluationJob(config=Config(type='custom', id='eval-config-QZNjbAHSWjNr5FvAVhuzD', created_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581764), custom_fields={}, description=None, groups=None, name='eval-config-QZNjbAHSWjNr5FvAVhuzD', namespace='default', ownership=None, params=None, project=None, schema_version='1.0', tasks={'custom-tool-calling': ConfigTasks(type='chat-completion', dataset=ConfigTasksDatasetDatasetEv(files_url='hf://datasets/xlam-tutorial-ns/xlam-ft-dataset/testing/xlam-test-single.jsonl', id='dataset-DRno5c1nwQnfmKJQAd5KQA', created_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581838), custom_fields={}, description=None, format=None, hf_endpoint=None, limit=50, name='dataset-DRno5c1nwQnfmKJQAd5KQA', namespace='default', ownership=None, project=None, schema_version='1.0', split=None, type_prefix=None, updated_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581839), version_id='main', version_tags=[]), metrics={'tool-calling-accuracy': ConfigTasksMetrics(type='tool-calling', params={'tool_calls_ground_truth': '{{ item.tool_calls | tojson }}'})}, params={'template': {'messages': '{{ item.messages | tojson}}', 'tools': '{{ item.tools | tojson }}', 'tool_choice': 'auto'}})}, type_prefix='eval-config', updated_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581766)), target=Target(type='model', id='eval-target-PgDfKhiTDkXtx5BsQVAzTQ', cached_outputs=None, created_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581900), custom_fields={}, dataset=None, description=None, model='xlam-tutorial-ns/llama-3.2-1b-xlam-run1@cust-FarcM8gwhL1XFDXQ57qGLL', name='eval-target-PgDfKhiTDkXtx5BsQVAzTQ', namespace='default', ownership=None, project=None, rag=None, retriever=None, rows=None, schema_version='1.0', type_prefix='eval-target', updated_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 581900)), id='eval-QbibneF16tyDiCJWc9oxgZ', created_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 582286), custom_fields={}, description=None, namespace='default', output_files_url=None, ownership=None, project=None, result=None, status='created', status_details={'message': None, 'task_status': {}, 'progress': None}, updated_at=datetime.datetime(2025, 6, 20, 16, 55, 38, 582287)) [35]
Job status: running after 5.03 seconds. Progress: 0.0% Job status: running after 10.04 seconds. Progress: 0.0% Job status: running after 15.06 seconds. Progress: 0.0% Job status: running after 20.08 seconds. Progress: 0.0% Job status: running after 25.09 seconds. Progress: 8.0% Job status: running after 30.11 seconds. Progress: 12.0% Job status: running after 35.13 seconds. Progress: 18.0% Job status: running after 40.14 seconds. Progress: 26.0% Job status: running after 45.16 seconds. Progress: 26.0% Job status: running after 50.18 seconds. Progress: 26.0% Job status: running after 55.19 seconds. Progress: 26.0% Job status: running after 60.21 seconds. Progress: 26.0% Job status: running after 65.22 seconds. Progress: 28.0% Job status: running after 70.24 seconds. Progress: 32.0% Job status: running after 75.26 seconds. Progress: 32.0% Job status: running after 80.27 seconds. Progress: 38.0% Job status: running after 85.29 seconds. Progress: 38.0% Job status: running after 90.30 seconds. Progress: 38.0% Job status: running after 95.32 seconds. Progress: 40.0% Job status: running after 100.34 seconds. Progress: 50.0% Job status: running after 105.35 seconds. Progress: 50.0% Job status: running after 110.37 seconds. Progress: 60.0% Job status: running after 115.38 seconds. Progress: 60.0% Job status: running after 120.40 seconds. Progress: 60.0% Job status: running after 125.42 seconds. Progress: 60.0% Job status: running after 130.43 seconds. Progress: 60.0% Job status: running after 135.45 seconds. Progress: 62.0% Job status: running after 140.46 seconds. Progress: 62.0% Job status: running after 145.48 seconds. Progress: 62.0% Job status: running after 150.49 seconds. Progress: 62.0% Job status: running after 155.51 seconds. Progress: 62.0% Job status: running after 160.52 seconds. Progress: 64.0% Job status: running after 165.54 seconds. Progress: 64.0% Job status: running after 170.56 seconds. Progress: 68.0% Job status: running after 175.57 seconds. Progress: 74.0% Job status: running after 180.59 seconds. Progress: 84.0% Job status: running after 185.60 seconds. Progress: 84.0% Job status: running after 190.61 seconds. Progress: 84.0% Job status: running after 195.63 seconds. Progress: 84.0% Job status: running after 200.65 seconds. Progress: 88.0% Job status: running after 205.66 seconds. Progress: 88.0% Job status: running after 210.68 seconds. Progress: 92.0% Job status: running after 215.70 seconds. Progress: 92.0% Job status: running after 220.71 seconds. Progress: 92.0% Job status: running after 225.73 seconds. Progress: 94.0% Job status: completed after 230.75 seconds. Progress: 100%
2.2 Review Evaluation Metrics
The following code retrieves the evaluation results for the fine-tuned model evaluation job.
[36]
{
"job": "eval-QbibneF16tyDiCJWc9oxgZ",
"id": "evaluation_result-B2uHJZANeXGz92ME8McJj5",
"created_at": "2025-06-20T16:55:38.609088",
"custom_fields": {},
"description": null,
"files_url": null,
"groups": {},
"namespace": "default",
"ownership": null,
"project": null,
"tasks": {
"custom-tool-calling": {
"metrics": {
"tool-calling-accuracy": {
"scores": {
"function_name_accuracy": {
"value": 0.92,
"stats": {
"count": 50,
"max": null,
"mean": 0.92,
"min": null,
"stddev": null,
"stderr": null,
"sum": 46.0,
"sum_squared": null,
"variance": null
}
},
"function_name_and_args_accuracy": {
"value": 0.72,
"stats": {
"count": 50,
"max": null,
"mean": 0.72,
"min": null,
"stddev": null,
"stderr": null,
"sum": 36.0,
"sum_squared": null,
"variance": null
}
}
}
}
}
}
},
"updated_at": "2025-06-20T16:55:38.609089"
}
[38]
Custom model: function_name_accuracy: 0.92 Custom model: function_name_and_args_accuracy: 0.72
A successfully fine-tuned meta/llama-3.2-1b-instruct results in a significant increase in tool calling accuracy with
In this case you should observe roughly the following improvements -
- function_name_accuracy: 12% to 92%
- function_name_and_args_accuracy: 8% to 72%
Since this evaluation was on a limited number of samples for demonstration purposes, you may choose to increase tasks.dataset.limit in your evaluation config simple_tool_calling_eval_config
(Optional) Next Steps
- You may also run the same evaluation on a base
meta/llama-3.1-70b-instructmodel for comparison. For this, first you will need to deploy the corresponding NIM using instructions here. After your NIM is deployed, set that endpoint as your evaluation target like so -
# Create an evaluation target
NIM_URL = "http://0.0.0.0:8000"
EVAL_TARGET = {
"type": "model",
"model": {
"api_endpoint": {
"url": f"{NIM_URL}/v1/completions",
"model_id": "meta/llama-3.1-70b-instruct",
}
}
}
# Start eval job
eval_job = nemo_client.evaluation.jobs.create(
config=simple_tool_calling_eval_config,
target=EVAL_TARGET
)
Running evaluation using the default config in this notebook, you should observe meta/llama-3.1-70b-instruct performance similar to -
- function_name_accuracy: 98%
- function_name_and_args_accuracy: 66%
Remarkably, a LoRA-tuned meta/llama-3.2-1b-instruct achieves accuracy that is close to a model 70 times its size, even outperforming it in the combined function_name_and_args_accuracy score.
You can now proceed with the same processes to fine-tune other NIM for LLMs and evaluate the accuracies between the base model and the fine-tuned model. By doing so, you can produce more accurate models for your use case.