Prompt Caching101
Prompt Caching 101
OpenAI offers discounted prompt caching for prompts exceeding 1024 tokens, resulting in up to an 80% reduction in latency for longer prompts over 10,000 tokens. By caching repetitive information across LLM API requests, you can greatly reduce both latency and costs. Prompt caching is scoped at the organization level, meaning only members of the same organization can access shared caches. Additionally, caching is eligible for zero data retention, as no data is stored during the process.
Prompt caching automatically activates for prompts longer than 1024 tokens-- you don't have to change anything in your completions request. When an API request is made, the system first checks if the beginning portion (prefix) of the prompt has already been cached. If a match is found (cache hit), the cached prompt is used, leading to reduced latency and costs. If there's no match, the system processes the full prompt from scratch and caches the prefix for future use.
With these benefits in mind, some of the key use cases where prompt caching can be especially advantageous are:
- Agents using tools and structured outputs: Cache the extended list of tools and schemas.
- Coding and writing assistants: Insert large sections or summaries of codebases and workspaces directly in prompts.
- Chatbots: Cache static portions of multi-turn conversations to maintain context efficiently over extended dialogues.
In this cookbook, we'll go through a couple examples of caching tools and images. Recall that in general, you'll want to put static content like instructions and examples at the beginning of your prompt, and variable content, such as user-specific information, at the end. This also applies to images and tools, which must be identical even in their ordering between requests. All requests, including those with fewer than 1024 tokens, will display a cached_tokens field of the usage.prompt_tokens_details chat completions object indicating how many of the prompt tokens were a cache hit. For requests under 1024 tokens, cached_tokens will be zero. Caching discounts are based on the actual number of tokens processed, including those used for images, which also count toward your rate limits.
Example 1: Caching tools and multi-turn conversations
In this example, we define tools and interactions for a customer support assistant, capable of handling tasks such as checking delivery dates, canceling orders, and updating payment methods. The assistant processes two separate messages, first responding to an initial query, followed by a delayed response to a follow-up query.
When caching tools, it is important that the tool definitions and their order remain identical for them to be included in the prompt prefix. To cache message histories in a multi-turn conversation, append new elements to the end of the messages array. In the response object and the output below, for the second completion run2, you can see that the cached_tokens value is greater than zero, indicating successful caching.
Run 1:
{
"id": "chatcmpl-ADeOueQSi2DIUMdLXnZIv9caVfnro",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"id": "call_5TnLcdD9tyVMVbzNGdejlJJa",
"function": {
"arguments": "{\"order_id\":\"9876543210\"}",
"name": "get_delivery_date"
},
"type": "function"
}
]
}
}
],
"created": 1727816928,
"model": "gpt-4o-mini-2024-07-18",
"object": "chat.completion",
"system_fingerprint": "fp_f85bea6784",
"usage": {
"completion_tokens": 17,
"prompt_tokens": 1079,
"total_tokens": 1096,
"prompt_tokens_details": {
"cached_tokens": 0
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}
Run 2:
{
"id": "chatcmpl-ADeP2i0frELC4W5RVNNkKz6TQ7hig",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"id": "call_viwwDZPuQh8hJFPf2Co1dYJK",
"function": {
"arguments": "{\"order_id\": \"9876543210\"}",
"name": "get_delivery_date"
},
"type": "function"
},
{
"id": "call_t1FFdAhrfvRc5IgqA6WkPKYj",
"function": {
"arguments": "{\"order_id\": \"9876543210\", \"reason\": \"Decided to purchase locally to get it faster.\"}",
"name": "cancel_order"
},
"type": "function"
}
]
}
}
],
"created": 1727816936,
"model": "gpt-4o-mini-2024-07-18",
"object": "chat.completion",
"system_fingerprint": "fp_f85bea6784",
"usage": {
"completion_tokens": 64,
"prompt_tokens": 1136,
"total_tokens": 1200,
"prompt_tokens_details": {
"cached_tokens": 1024
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}
Example 2: Images
In our second example we include multiple image URLs of grocery items in the messages array, along with a user query, run three times with delays. Images—whether linked or encoded in base64 within user messages—qualify for caching. Make sure the detail parameter remains consistent, as it affects how images are tokenized. Note that GPT-4o-mini adds extra tokens to cover image processing costs, even though it uses a low-cost token model for text. Caching discounts are based on the actual number of tokens processed, including those used for images, which also count toward your rate limits.
The output for this example shows that a cache was hit for the second run, however it was not hit for the third run because of a different first url (eggs_url instead of veggie_url), even though the user query is the same.
{
"id": "chatcmpl-ADeV3IrUqhpjMXEgv29BFHtTQ0Pzt",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "The images show the following types of sauces:\n\n1. **Soy Sauce** - Kikkoman brand.\n2. **Worcester Sauce** - Appel brand, listed as \"Dresdner Art.\"\n3. **Tabasco Sauce** - Original pepper sauce.\n\nThe second image shows various vegetables, not sauces.",
"refusal": null,
"role": "assistant"
}
}
],
"created": 1727817309,
"model": "gpt-4o-2024-08-06",
"object": "chat.completion",
"system_fingerprint": "fp_2f406b9113",
"usage": {
"completion_tokens": 65,
"prompt_tokens": 1548,
"total_tokens": 1613,
"prompt_tokens_details": {
"cached_tokens": 0
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}
{
"id": "chatcmpl-ADeVRSI6zFINkx99k7V6ux1v5iF5f",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "The images show different types of items. In the first image, you'll see bottles of sauces like soy sauce, Worcester sauce, and Tabasco. The second image features various vegetables, including:\n\n1. Napa cabbage\n2. Kale\n3. Carrots\n4. Bok choy\n5. Swiss chard\n6. Leeks\n7. Parsley\n\nThese vegetables are arranged on shelves in a grocery store setting.",
"refusal": null,
"role": "assistant"
}
}
],
"created": 1727817333,
"model": "gpt-4o-2024-08-06",
"object": "chat.completion",
"system_fingerprint": "fp_2f406b9113",
"usage": {
"completion_tokens": 86,
"prompt_tokens": 1548,
"total_tokens": 1634,
"prompt_tokens_details": {
"cached_tokens": 1280
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}
{
"id": "chatcmpl-ADeVphj3VALQVrdnt2efysvSmdnBx",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "The second image shows three types of sauces:\n\n1. Soy Sauce (Kikkoman)\n2. Worcestershire Sauce\n3. Tabasco Sauce",
"refusal": null,
"role": "assistant"
}
}
],
"created": 1727817357,
"model": "gpt-4o-2024-08-06",
"object": "chat.completion",
"system_fingerprint": "fp_2f406b9113",
"usage": {
"completion_tokens": 29,
"prompt_tokens": 1548,
"total_tokens": 1577,
"prompt_tokens_details": {
"cached_tokens": 0
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}
Overall tips
To get the most out of prompt caching, consider following these best practices:
-
Place static or frequently reused content at the beginning of prompts: This helps ensure better cache efficiency by keeping dynamic data towards the end of the prompt.
-
Maintain consistent usage patterns: Prompts that aren't used regularly are automatically removed from the cache. To prevent cache evictions, maintain consistent usage of prompts.
-
Monitor key metrics: Regularly track cache hit rates, latency, and the proportion of cached tokens. Use these insights to fine-tune your caching strategy and maximize performance.
By implementing these practices, you can take full advantage of prompt caching, ensuring that your applications are both responsive and cost-efficient. A well-managed caching strategy will significantly reduce processing times, lower costs, and help maintain smooth user experiences.