OpenAI Fine Tuning For Function Calling

Fine Tuning For Function Calling

chatgptopenaigpt-4examplesopenai-apiopenai-cookbook

alph-notebooks/openai-cookbook / Fine_tuning_for_function_calling.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Fine tuning with function-calling

This notebook covers how to fine-tune to increase function calling accuracy and reliability. You can find more information on function calling here, and on fine tuning here

For context, from the function calling notebook above:

tools is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.

Function calling is a very powerful tool when it functions as intended. However, we have seen that as the number of functions increases, and the complexity of the task at hand increases, function calling becomes less accurate (e.g.: more hallucinated invocations, and incorrect invocations).

Before fine tuning for function calling, it's best to begin with:

Improvements to the function definitions. Make them more clear, and more distinct from one another.
Experiment with prompt engineering: often a more detailed prompt can help the model call the correct function.

If the steps above fail to improve function calling to a satisfactory level, then you can try fine tuning for function calling.

Overview

This notebook contains three sections

Assessing baseline function calling performance: Evaluating an out-of-the-box gpt-3.5-turbo model on our given function (let's assume that for latency + cost reasons we cannot use gpt-4o for a drone copilot)
Generating synthetic data: Using gpt-4o to create 'golden' set of prompts and function invocations to use as training data
Fine-tuning: Running the fine tuning job, and evaluating the fine-tuned model

Note: This notebook provides an example of how to create synthetic training data for fine tuning for function calling given just a list of functions. While real-world production test evals are preferable, this method produces strong results and can be used in conjunction with real-world training data.

Getting baseline function calling performance

[173]

[211]

The dotenv extension is already loaded. To reload it, use:
  %reload_ext dotenv

Utilities

Let's define utility functions for making calls to the Chat Completions API, one to get the completion and one to get the function call.

[212]

Baseline testing

Let's build an intelligent drone co-pilot. We want to be able to give the co-pilot commands, and have it either call the function for that command, or deny that request if the command is unfeasible. We can first define a system prompt for the copilot.

[213]

Now let's define functions for all of the actions the copilot can take.

[214]

For starters, let's see how function calling performs with some straight forward feasible prompts, and then couple of obviously impossible requests which call the 'reject_request' function.

[220]

[219]

Number of matches: 10 out of 10 (100.00%)
Average latency per request: 826.81 ms
Average tokens used per request: 796.20

Nice! The model performs quite well with these requests. Now let's try some more difficult requests: requests that are almost feasible and are drone-related, but that the drone cannot actually do, and the pilot should reject.

[221]

[222]

Number of matches: 6 out of 10 (60.00%)
Average latency per request: 610.26 ms
Average tokens used per request: 791.90

Now we run into some problems. The model here should reject all of these requests, as they are impossible/conflicting/ambiguous given the functions, however instead the model calls functions that are somewhat related to the request, but incorrect. For example, the model sets follow_me_mode when asked to initiate following on social media.

In this simple case, more prompt engineering may resolve some of these issues, but for the purpose of this example we will demonstrate how fine tuning can be used to improve performance. Additionally, while this case is relatively straightforward, as the number of and complexity of the functions increases, fine tuning becomes more and more impactful.

Again, our goal here is to improve performance and use less tokens, so fine-tuning allows us to:

Omit function and parameter descriptions: remove the description field from function and parameters
Omit parameters: remove the entire properties field from the parameters object
Omit function entirely: remove the entire function object from the functions array

Generating synthetic data

Helper functions

We want to generate every invocation of every function, so that we have full coverage of all potential invocations to create synthetic data for. Then, we will use gpt-4o to come up with prompts that would call each invocation, and we will use that prompt - function invocation pair as training data.

Generating every invocation for a function with fixed enums is more simple, but for a function such as control_gimbal we need to set the tilt and pan integer values, so to generate those synthetic invocations we will first set a placeholder, and then later use gpt-4o to come up with reasonable values.

[253]

The functions below take in all the functions from the function list, and look at all the potential invocations of those functions given each function's parameters. The functions also account for required parameters, so that all the invocations are actually feasible.

[254]

Let's generate every invocation for every function first

Prompts:

[255]

In the below snippet, we generate the invocation of each function except for the reject_request function.

To perform effective fine-tuning we need correctly labeled data. We could manually come up with examples and label the data,
or we can generate synthetic data with the help of gpt-4o

Empirically, gpt-4o needs a bit more help to get good realistic examples of prompts that would generate the reject_request function, so we'll do that next...

[256]

Now that we have all the invocations, let's use gpt-4o to generate prompts that would result in those invocations

[257]

[258]

[259]

0.0% complete
{'name': 'takeoff_drone', 'arguments': {'altitude': 100}}
1.8% complete
{'name': 'land_drone', 'arguments': {'location': 'current'}}
3.5% complete
{'name': 'land_drone', 'arguments': {'location': 'home_base'}}
5.3% complete
{'name': 'land_drone', 'arguments': {'location': 'custom'}}
7.0% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'forward', 'distance': 100}}
8.8% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'backward', 'distance': 50}}
10.5% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'left', 'distance': 10}}
12.3% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'right', 'distance': 10}}
14.0% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'up', 'distance': 10}}
15.8% complete
{'name': 'control_drone_movement', 'arguments': {'direction': 'down', 'distance': 10}}
17.5% complete
{'name': 'set_drone_speed', 'arguments': {'speed': 10}}
19.3% complete
{'name': 'control_camera', 'arguments': {'mode': 'photo'}}
21.1% complete
{'name': 'control_camera', 'arguments': {'mode': 'photo', 'duration': 10}}
22.8% complete
{'name': 'control_camera', 'arguments': {'mode': 'video'}}
24.6% complete
{'name': 'control_camera', 'arguments': {'mode': 'video', 'duration': 60}}
26.3% complete
{'name': 'control_camera', 'arguments': {'mode': 'panorama'}}
28.1% complete
{'name': 'control_camera', 'arguments': {'mode': 'panorama', 'duration': 60}}
29.8% complete
{'name': 'control_gimbal', 'arguments': {'tilt': 45, 'pan': 90}}
31.6% complete
{'name': 'set_drone_lighting', 'arguments': {'mode': 'on'}}
33.3% complete
{'name': 'set_drone_lighting', 'arguments': {'mode': 'off'}}
35.1% complete
{'name': 'set_drone_lighting', 'arguments': {'mode': 'blink'}}
36.8% complete
{'name': 'set_drone_lighting', 'arguments': {'mode': 'sos'}}
38.6% complete
{'name': 'return_to_home', 'arguments': {}}
40.4% complete
{'name': 'set_battery_saver_mode', 'arguments': {'status': 'on'}}
42.1% complete
{'name': 'set_battery_saver_mode', 'arguments': {'status': 'off'}}
43.9% complete
{'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'on'}}
45.6% complete
{'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'off'}}
47.4% complete
{'name': 'set_follow_me_mode', 'arguments': {'status': 'on'}}
49.1% complete
{'name': 'set_follow_me_mode', 'arguments': {'status': 'off'}}
50.9% complete
{'name': 'calibrate_sensors', 'arguments': {}}
52.6% complete
{'name': 'set_autopilot', 'arguments': {'status': 'on'}}
54.4% complete
{'name': 'set_autopilot', 'arguments': {'status': 'off'}}
56.1% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid'}}
57.9% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'red'}}
59.6% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'blue'}}
61.4% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'green'}}
63.2% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'yellow'}}
64.9% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'white'}}
66.7% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink'}}
68.4% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'red'}}
70.2% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'blue'}}
71.9% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'green'}}
73.7% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'yellow'}}
75.4% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'white'}}
77.2% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse'}}
78.9% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'red'}}
80.7% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'blue'}}
82.5% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'green'}}
84.2% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'yellow'}}
86.0% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'white'}}
87.7% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow'}}
89.5% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'red'}}
91.2% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'blue'}}
93.0% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'green'}}
94.7% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'yellow'}}
96.5% complete
{'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'white'}}
98.2% complete
{'name': 'reject_request', 'arguments': {}}

Now let's format the training examples properly. For more documentation on the proper training data formatting for fine tuning for function calling, see here: https://platform.openai.com/docs/guides/fine-tuning/fine-tuning-examples

[19]

[261]

Let's get the drone in the air, how high should it go?
{'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
Ready for takeoff, how high should the drone fly?
{'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
Can you bring the drone down to where we are?
{'name': 'land_drone', 'arguments': '{"location": "current"}'}
Let's get the drone to land right here
{'name': 'land_drone', 'arguments': '{"location": "current"}'}
Bring the drone back to base for landing
{'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
Can you safely land the drone at home base
{'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
Can you make the drone move to the left by 10 units?
{'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
I need the drone to go left, could you move it 10 steps that way?
{'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
Can you move the drone to the right by 10 feet?
{'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
I need the drone to go 10 feet to the right, can you do that?
{'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
Can you make the drone go upwards by 10 units?
{'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
I need the drone to move up, can you do that for me?
{'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
Can you bring the drone lower by 10 feet please?
{'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
I need the drone to descend 10 units, can you make that happen?
{'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
Can you make the drone go faster?
{'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
I think the drone should speed up a bit, don't you think?
{'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
I want to take a picture, can you switch the camera mode to photo
{'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
Let's capture this moment, switch the camera to photo mode please
{'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
Can you switch the camera to photo mode and take a picture for 10 seconds?
{'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
I need to capture something, can you set the camera to take photos for 10 seconds?
{'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
Can you switch the camera to video mode?
{'name': 'control_camera', 'arguments': '{"mode": "video"}'}
I want to record, can you set the camera to video mode?
{'name': 'control_camera', 'arguments': '{"mode": "video"}'}
Can you start recording a video with the camera for a minute
{'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
I need to film something, can you put the camera in video mode for 60 seconds
{'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
Can you switch the camera to panorama mode?
{'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
I'd like to take a 360-degree photo, can you set the camera to panorama mode?
{'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
Can you set the camera to take a panorama shot for a minute
{'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
I'd like to switch the camera mode to panorama and have it last for a minute
{'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
Can you adjust the camera angle up and to the right?
{'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
I need to tilt the camera up and pan it to the right, can you do that?
{'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
Can you turn on the lights for the drone
{'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
I need some extra light, can you activate it on the drone
{'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
Can you turn off the lights on the drone
{'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
I don't need the drone lights on, can you switch them off
{'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
Can you make the drone lights flash?
{'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
I want the drone lights to blink, can you do that?
{'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
Can you switch the drone lights to the SOS mode, just in case?
{'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
I need the drone lights to flash SOS, can you set that up?
{'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
Can you bring the drone back home now?
{'name': 'return_to_home', 'arguments': '{}'}
Is it time for the drone to return to base?
{'name': 'return_to_home', 'arguments': '{}'}
My phone battery is draining so fast, can you turn on battery saver mode
{'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
I need my laptop battery to last longer, can you switch on battery saver mode
{'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
My phone battery is draining too quickly, can you turn off the battery saver mode
{'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
I feel like my device is slower with battery saver on, can we turn it off?
{'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
I want the car to avoid obstacles, can you turn on that feature?
{'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
Can you activate the obstacle avoidance mode for safety purposes?
{'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
I'd like to turn off obstacle detection, how do I do that?
{'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
Can you disable the obstacle avoidance feature for now?
{'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
Can you activate the follow me mode?
{'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
I want the car to follow me, can you turn on that feature?
{'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
I don't want the drone following me anymore, can you turn that off?
{'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
Can you disable the follow-me mode on the drone?
{'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
The sensors are acting up, can you recalibrate them
{'name': 'calibrate_sensors', 'arguments': '{}'}
My device doesn't seem to be sensing correctly, can you adjust it
{'name': 'calibrate_sensors', 'arguments': '{}'}
I'm too tired to drive, can you turn on the autopilot
{'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
Let the car drive itself, turn on autopilot
{'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
I'm feeling more confident, turn off the autopilot
{'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
I think I can handle it, deactivate the automatic pilot
{'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
Can you set the display to a steady yellow color?
{'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
I'd like the LED display to be a solid yellow, please.
{'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
Can you make the lights flash on and off
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
I want the LED display to blink, can you set that up
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
Can you make the lights flash in red?
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
How do I set the display to blink in red?
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
Can you make the lights flash in yellow?
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
How do I set the display to blink in yellow?
{'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
Can you make the lights blink instead of staying steady
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
I want the LEDs to flash, not stay solid
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
Can you make the LED display pulse in red, please?
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
I'd like the LED display to flash in red, can you set that up?
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
I want the LED lights to flash in blue
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
Can you set the display to pulse with a blue color
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
Can you make the lights flash and change to green
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
Let's set the LEDs to blink and switch to green
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
Can you change the flashy lights to yellow and make them pulse
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
I want the LED display to blink in yellow, can you do that
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
Can you change the colors on the display to red and set it to a rainbow pattern?
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
I want the LED display to show a rainbow pattern in red, can you set that up?
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
Can you change the color and pattern of the lights to blue and rainbow?
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
I'm feeling like some colorful lights, can you set it to blue and rainbow?
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
Can you set the LED display to show a rainbow pattern in green color?
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
I'd like the LED display to cycle through colors, starting with green
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
Can you make the lights do a cool rainbow effect
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
Change the color of the lights to white and make them change like a rainbow
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
I changed my mind, can you cancel that request
{'name': 'reject_request', 'arguments': '{}'}
I don't want to proceed with the request anymore, can you reject it
{'name': 'reject_request', 'arguments': '{}'}

Now, back to the rejection function. Let's generate some prompts that are nearly possible, but should result in the reject_request function being called. To do so, we queried gpt-4o asking for requests that are related to, but not quite possible with, the given list of functions.

[262]

[263]

Now combine all the training examples together

[264]

[265]

Fine tuning

Finally, we can kick off the fine-tuning job

[200]

FileID: file-blg0IytwIivZQzc9mbfnS8Pm
Fine-tuning job created: FineTuningJob(id='ftjob-84PQg97hoIAKf21IPnhiNlU1', created_at=1718580285, error=Error(code=None, message=None, param=None), fine_tuned_model=None, finished_at=None, hyperparameters=Hyperparameters(n_epochs='auto', batch_size='auto', learning_rate_multiplier='auto'), model='gpt-3.5-turbo-0125', object='fine_tuning.job', organization_id='org-lb41cclBdkq5pm6BgDhx8DHP', result_files=[], seed=1513865891, status='validating_files', trained_tokens=None, training_file='file-blg0IytwIivZQzc9mbfnS8Pm', validation_file=None, estimated_finish=None, integrations=[], user_provided_suffix='drone')

In addition to creating a fine-tuning job, you can also list existing jobs, retrieve the status of a job, or cancel a job.

[224]

FineTuningJob(id='ftjob-84PQg97hoIAKf21IPnhiNlU1', created_at=1718580285, error=Error(code=None, message=None, param=None), fine_tuned_model='ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC', finished_at=1718581004, hyperparameters=Hyperparameters(n_epochs=3, batch_size=1, learning_rate_multiplier=2), model='gpt-3.5-turbo-0125', object='fine_tuning.job', organization_id='org-lb41cclBdkq5pm6BgDhx8DHP', result_files=['file-F6XPJFLVG9f3mR04KBmwUI9H'], seed=1513865891, status='succeeded', trained_tokens=145983, training_file='file-blg0IytwIivZQzc9mbfnS8Pm', validation_file=None, estimated_finish=None, integrations=[], user_provided_suffix='drone')

After a fine-tuning job has finished, you can also see metrics around how the training process went by querying a fine-tuning job, extracting a file ID from the result_files, and then retrieving that files content. Each results CSV file has the following columns: step, train_loss, train_accuracy, valid_loss, and valid_mean_token_accuracy. While metrics can he helpful, evaluating samples from the fine-tuned model provides the most relevant sense of model quality.

[227]

step,train_loss,train_accuracy,valid_loss,valid_mean_token_accuracy
1,3.63265,0.5,,
2,2.45992,0.80952,,
3,2.77939,0.80952,,
4,3.53073,0.65,,
5,2.61654,0.8,,
6,2.16,0.85714,,
7,2.73706,0.8,,
8,2.56944,0.625,,
9,2.06096,0.78947,,
10,1.69598,0.8,,
11,1.94268,0.77778,,
12,1.61752,0.86667,,
13,1.2442,0.8,,
14,0.73411,0.875,,
15,0.34285,0.875,,
16,0.22229,0.95238,,
17,0.04635,0.95,,
18,0.00626,1.0,,
19,0.60888,0.90909,,
20,0.00092,1.0,,
21,0.8001,0.95,,
22,0.04982,1.0,,
23,0.35494,0.92857,,
24,0.00023,1.0,,
25,0.00034,1.0,,
26,0.0029,1.0,,
27,0.58017,0.875,,
28,0.13018,0.9375,,
29,0.00109,1.0,,
30,6e-05,1.0,,
31,0.61665,0.95,,
32,3e-05,1.0,,
33,0.23598,0.95,,
34,3e-05,1.0,,
35,0.03566,1.0,,
36,1e-05,1.0,,
37,1e-05,1.0,,
38,2e-05,1.0,,
39,2e-05,1.0,,
40,0.00034,1.0,,
41,0.0,1.0,,
42,0.0,1.0,,
43,0.0,1.0,,
44,0.0,1.0,,
45,0.0,1.0,,
46,0.91896,0.95,,
47,0.0,1.0,,
48,0.12006,0.95,,
49,0.0,1.0,,
50,3.92872,0.75,,
51,0.0,1.0,,
52,0.98277,0.90476,,
53,0.0,1.0,,
54,0.0,1.0,,
55,1e-05,1.0,,
56,0.00401,1.0,,
57,0.07366,1.0,,
58,0.0,1.0,,
59,0.0,1.0,,
60,0.0,1.0,,
61,0.0,1.0,,
62,0.10347,0.875,,
63,0.0,1.0,,
64,0.0,1.0,,
65,1e-05,1.0,,
66,2.97112,0.85714,,
67,1.12396,0.875,,
68,2e-05,1.0,,
69,0.00067,1.0,,
70,0.0,1.0,,
71,0.0,1.0,,
72,0.0,1.0,,
73,0.0,1.0,,
74,0.0,1.0,,
75,0.02064,1.0,,
76,0.5146,0.86667,,
77,0.18756,0.95,,
78,6e-05,1.0,,
79,0.0,1.0,,
80,0.21298,0.93333,,
81,0.0,1.0,,
82,0.0,1.0,,
83,0.0,1.0,,
84,0.00139,1.0,,
85,0.0,1.0,,
86,0.85297,0.875,,
87,0.0,1.0,,
88,0.0,1.0,,
89,1.45164,0.875,,
90,0.0,1.0,,
91,0.05329,0.92857,,
92,0.55506,0.93333,,
93,0.42187,0.92857,,
94,0.0,1.0,,
95,0.0,1.0,,
96,0.0,1.0,,
97,0.0,1.0,,
98,0.0,1.0,,
99,0.0,1.0,,
100,0.0,1.0,,
101,0.0,1.0,,
102,0.0,1.0,,
103,0.09194,0.95455,,
104,0.0,1.0,,
105,0.0,1.0,,
106,0.05531,0.95,,
107,0.0,1.0,,
108,0.39621,0.95238,,
109,0.0,1.0,,
110,0.8449,0.95,,
111,0.01258,1.0,,
112,0.0,1.0,,
113,0.0,1.0,,
114,0.0,1.0,,
115,0.00355,1.0,,
116,0.0,1.0,,
117,0.3954,0.94118,,
118,0.00259,1.0,,
119,0.0,1.0,,
120,0.0,1.0,,
121,0.35876,0.95,,
122,0.0,1.0,,
123,0.0,1.0,,
124,5e-05,1.0,,
125,0.0,1.0,,
126,0.0,1.0,,
127,0.0,1.0,,
128,0.0,1.0,,
129,0.0,1.0,,
130,0.01336,1.0,,
131,0.0,1.0,,
132,0.23362,0.95,,
133,0.00157,1.0,,
134,0.0,1.0,,
135,0.00031,1.0,,
136,0.0,1.0,,
137,0.08313,0.92857,,
138,0.0,1.0,,
139,0.0,1.0,,
140,0.0,1.0,,
141,0.43608,0.95,,
142,0.0,1.0,,
143,0.0,1.0,,
144,0.0,1.0,,
145,2e-05,1.0,,
146,1.20409,0.85714,,
147,0.0,1.0,,
148,0.0,1.0,,
149,0.0,1.0,,
150,0.0,1.0,,
151,0.0,1.0,,
152,0.0,1.0,,
153,0.0,1.0,,
154,0.00063,1.0,,
155,0.0,1.0,,
156,0.0,1.0,,
157,0.0,1.0,,
158,6e-05,1.0,,
159,0.0,1.0,,
160,0.0,1.0,,
161,0.0,1.0,,
162,0.0,1.0,,
163,0.0,1.0,,
164,0.0,1.0,,
165,0.0,1.0,,
166,0.0,1.0,,
167,0.0,1.0,,
168,0.0,1.0,,
169,0.0,1.0,,
170,0.0,1.0,,
171,0.0,1.0,,
172,0.0,1.0,,
173,0.0,1.0,,
174,0.00783,1.0,,
175,0.0,1.0,,
176,0.0,1.0,,
177,0.0,1.0,,
178,0.0,1.0,,
179,0.0,1.0,,
180,0.0,1.0,,
181,0.0,1.0,,
182,0.00028,1.0,,
183,0.0,1.0,,
184,0.0,1.0,,
185,0.0003,1.0,,
186,0.0,1.0,,
187,0.0,1.0,,
188,0.0,1.0,,
189,0.0,1.0,,
190,0.0,1.0,,
191,0.0,1.0,,
192,0.0,1.0,,
193,0.00013,1.0,,
194,0.86198,0.875,,
195,0.0,1.0,,
196,0.0,1.0,,
197,0.0,1.0,,
198,0.0,1.0,,
199,0.0,1.0,,
200,0.0,1.0,,
201,0.0,1.0,,
202,0.0,1.0,,
203,0.0,1.0,,
204,0.09954,0.95455,,
205,0.0,1.0,,
206,0.0,1.0,,
207,0.0,1.0,,
208,1.9616,0.9375,,
209,0.0,1.0,,
210,0.0,1.0,,
211,0.0,1.0,,
212,0.0,1.0,,
213,0.0,1.0,,
214,0.0,1.0,,
215,0.0,1.0,,
216,0.0,1.0,,
217,0.0,1.0,,
218,0.0,1.0,,
219,0.0,1.0,,
220,0.0,1.0,,
221,0.0,1.0,,
222,0.0,1.0,,
223,0.0,1.0,,
224,0.0,1.0,,
225,0.0,1.0,,
226,0.00174,1.0,,
227,0.0,1.0,,
228,2e-05,1.0,,
229,0.0,1.0,,
230,0.0,1.0,,
231,0.0,1.0,,
232,0.0,1.0,,
233,0.0,1.0,,
234,0.61895,0.95,,
235,0.0,1.0,,
236,0.0,1.0,,
237,0.0,1.0,,
238,0.0,1.0,,
239,0.54945,0.95,,
240,0.0,1.0,,
241,0.0,1.0,,
242,1.52953,0.9375,,
243,1.19938,0.85714,,
244,0.0,1.0,,
245,0.0,1.0,,
246,0.0,1.0,,
247,0.0,1.0,,
248,8e-05,1.0,,
249,0.0,1.0,,
250,0.0,1.0,,
251,0.0,1.0,,
252,0.0,1.0,,
253,0.0,1.0,,
254,0.0,1.0,,
255,0.0,1.0,,
256,0.0,1.0,,
257,0.0,1.0,,
258,0.0,1.0,,
259,0.0,1.0,,
260,0.0,1.0,,
261,0.0,1.0,,
262,0.0,1.0,,
263,0.0,1.0,,
264,0.0,1.0,,
265,0.0,1.0,,
266,0.0,1.0,,
267,0.88984,0.95,,
268,0.0,1.0,,
269,0.0,1.0,,
270,0.0,1.0,,
271,0.0,1.0,,
272,0.0,1.0,,
273,0.0,1.0,,
274,0.0,1.0,,
275,0.00013,1.0,,
276,0.0,1.0,,
277,0.89825,0.92857,,
278,0.0,1.0,,
279,0.00017,1.0,,
280,0.0,1.0,,
281,0.0,1.0,,
282,0.0,1.0,,
283,0.65667,0.95,,
284,0.0,1.0,,
285,0.0,1.0,,
286,0.0,1.0,,
287,0.0,1.0,,
288,0.0,1.0,,
289,0.0,1.0,,
290,0.0,1.0,,
291,0.0,1.0,,
292,0.28626,0.95238,,
293,0.0,1.0,,
294,0.0,1.0,,
295,0.0,1.0,,
296,0.0,1.0,,
297,0.0,1.0,,
298,0.0,1.0,,
299,0.0,1.0,,
300,0.0,1.0,,
301,0.0,1.0,,
302,0.0,1.0,,
303,0.0,1.0,,
304,0.0,1.0,,
305,0.0,1.0,,
306,0.0,1.0,,
307,0.0,1.0,,
308,0.0,1.0,,
309,0.0,1.0,,

Evaluations

Great! We trained a fine-tuned model for function calling. Let's see how it does on our evaluation set for prompts that the drone assistant should automatically reject.

[226]


Evaluating fine-tuned model with challenging prompts: ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC

Number of matches: 10 out of 10 (100.00%)
Average latency per request: 3519.17 ms
Average tokens used per request: 457.20

Evaluating base model with challenging prompts: gpt-3.5-turbo

Number of matches: 6 out of 10 (60.00%)
Average latency per request: 647.58 ms
Average tokens used per request: 791.90

Great! While the original model only rejected 60%, the fine tuned model rejected 100% requests and used less tokens to do so.

Conclusion

Congratulations! You are now ready to fine tune your model for function calling. We can't wait to see what you build.