Get number of tokens in a string using OpenAI's tiktoken library

This function uses the num_tokens_from_string function provided in OpenAI's tiktoken python library to get the number of tokens in a string.

Usage

num_tokens(text, encoding = "cl100k_base")

Arguments

text

a character string to count the number of tokens from

encoding

a character string that specifies how text is converted into tokens. The default is "cl100k_base" (for ChatGPT models; see https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb for details)

Value

an integer indicating the number of tokens in the input string

References

https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb