How To Count Tokens With Tiktoken
How to count tokens with tiktoken
tiktoken is a fast open-source tokenizer by OpenAI.
Given a text string (e.g., "tiktoken is great!") and an encoding (e.g., "cl100k_base"), a tokenizer can split the text string into a list of tokens (e.g., ["t", "ik", "token", " is", " great", "!"]).
Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).
Encodings
Encodings specify how text is converted into tokens. Different models use different encodings.
tiktoken supports three encodings used by OpenAI models:
| Encoding name | OpenAI models |
|---|---|
o200k_base | gpt-4o, gpt-4o-mini |
cl100k_base | gpt-4-turbo, gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large |
p50k_base | Codex models, text-davinci-002, text-davinci-003 |
r50k_base (or gpt2) | GPT-3 models like davinci |
You can retrieve the encoding for a model using tiktoken.encoding_for_model() as follows:
encoding = tiktoken.encoding_for_model('gpt-4o-mini')
Note that p50k_base overlaps substantially with r50k_base, and for non-code applications, they will usually give the same tokens.
Tokenizer libraries by language
For o200k_base, cl100k_base and p50k_base encodings:
- Python: tiktoken
- .NET / C#: SharpToken, TiktokenSharp
- Java: jtokkit
- Golang: tiktoken-go
- Rust: tiktoken-rs
For r50k_base (gpt2) encodings, tokenizers are available in many languages.
- Python: tiktoken (or alternatively GPT2TokenizerFast)
- JavaScript: gpt-3-encoder
- .NET / C#: GPT Tokenizer
- Java: gpt2-tokenizer-java
- PHP: GPT-3-Encoder-PHP
- Golang: tiktoken-go
- Rust: tiktoken-rs
(OpenAI makes no endorsements or guarantees of third-party libraries.)
How strings are typically tokenized
In English, tokens commonly range in length from one character to one word (e.g., "t" or " great"), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., " is" instead of "is " or " "+"is"). You can quickly check how a string is tokenized at the OpenAI Tokenizer, or the third-party Tiktokenizer webapp.
0. Install tiktoken
If needed, install tiktoken with pip:
[notice] A new release of pip is available: 24.0 -> 24.2 [notice] To update, run: pip install --upgrade pip Note: you may need to restart the kernel to use updated packages. [notice] A new release of pip is available: 24.0 -> 24.2 [notice] To update, run: pip install --upgrade pip Note: you may need to restart the kernel to use updated packages.
1. Import tiktoken
2. Load an encoding
Use tiktoken.get_encoding() to load an encoding by name.
The first time this runs, it will require an internet connection to download. Later runs won't need an internet connection.
Use tiktoken.encoding_for_model() to automatically load the correct encoding for a given model name.
3. Turn text into tokens with encoding.encode()
The .encode() method converts a text string into a list of token integers.
[83, 8251, 2488, 382, 2212, 0]
Count tokens by counting the length of the list returned by .encode().
6
4. Turn tokens into text with encoding.decode()
.decode() converts a list of token integers to a string.
'tiktoken is great!'
Warning: although .decode() can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.
For single tokens, .decode_single_token_bytes() safely converts a single integer token to the bytes it represents.
[b't', b'ikt', b'oken', b' is', b' great', b'!']
(The b in front of the strings indicates that the strings are byte strings.)
5. Comparing encodings
Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.
Example string: "antidisestablishmentarianism" r50k_base: 5 tokens token integers: [415, 29207, 44390, 3699, 1042] token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism'] p50k_base: 5 tokens token integers: [415, 29207, 44390, 3699, 1042] token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism'] cl100k_base: 6 tokens token integers: [519, 85342, 34500, 479, 8997, 2191] token bytes: [b'ant', b'idis', b'establish', b'ment', b'arian', b'ism'] o200k_base: 6 tokens token integers: [493, 129901, 376, 160388, 21203, 2367] token bytes: [b'ant', b'idis', b'est', b'ablishment', b'arian', b'ism']
Example string: "2 + 2 = 4" r50k_base: 5 tokens token integers: [17, 1343, 362, 796, 604] token bytes: [b'2', b' +', b' 2', b' =', b' 4'] p50k_base: 5 tokens token integers: [17, 1343, 362, 796, 604] token bytes: [b'2', b' +', b' 2', b' =', b' 4'] cl100k_base: 7 tokens token integers: [17, 489, 220, 17, 284, 220, 19] token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4'] o200k_base: 7 tokens token integers: [17, 659, 220, 17, 314, 220, 19] token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']
Example string: "お誕生日おめでとう" r50k_base: 14 tokens token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557] token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86'] p50k_base: 14 tokens token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557] token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86'] cl100k_base: 9 tokens token integers: [33334, 45918, 243, 21990, 9080, 33334, 62004, 16556, 78699] token bytes: [b'\xe3\x81\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97\xa5', b'\xe3\x81\x8a', b'\xe3\x82\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8\xe3\x81\x86'] o200k_base: 8 tokens token integers: [8930, 9697, 243, 128225, 8930, 17693, 4344, 48669] token bytes: [b'\xe3\x81\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f\xe6\x97\xa5', b'\xe3\x81\x8a', b'\xe3\x82\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8\xe3\x81\x86']
6. Counting tokens for chat completions API calls
ChatGPT models like gpt-4o-mini and gpt-4 use tokens in the same way as older completions models, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.
Below is an example function for counting tokens for messages passed to gpt-3.5-turbo, gpt-4, gpt-4o and gpt-4o-mini.
Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.
In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.
gpt-3.5-turbo Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125. 129 prompt tokens counted by num_tokens_from_messages(). 129 prompt tokens counted by the OpenAI API. gpt-4-0613 129 prompt tokens counted by num_tokens_from_messages(). 129 prompt tokens counted by the OpenAI API. gpt-4 Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613. 129 prompt tokens counted by num_tokens_from_messages(). 129 prompt tokens counted by the OpenAI API. gpt-4o Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06. 124 prompt tokens counted by num_tokens_from_messages(). 124 prompt tokens counted by the OpenAI API. gpt-4o-mini Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18. 124 prompt tokens counted by num_tokens_from_messages(). 124 prompt tokens counted by the OpenAI API.
7. Counting tokens for chat completions with tool calls
Next, we will look into how to apply this calculations to messages that may contain function calls. This is not immediately trivial, due to the formatting of the tools themselves.
Below is an example function for counting tokens for messages that contain tools, passed to gpt-3.5-turbo, gpt-4, gpt-4o and gpt-4o-mini.
gpt-3.5-turbo Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125. 105 prompt tokens counted by num_tokens_for_tools(). 105 prompt tokens counted by the OpenAI API. gpt-4 Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613. 105 prompt tokens counted by num_tokens_for_tools(). 105 prompt tokens counted by the OpenAI API. gpt-4o Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06. 101 prompt tokens counted by num_tokens_for_tools(). 101 prompt tokens counted by the OpenAI API. gpt-4o-mini Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18. 101 prompt tokens counted by num_tokens_for_tools(). 101 prompt tokens counted by the OpenAI API.