storm bobcat
I tried to calculate the prompt tokens used for my requests, but when i calculate it i get a different number than reported by the API.
The API reports 29 prompt tokens and i calculated 17
[
{
role: 'system',
content: 'You are a useful chatbot called ChatGPT.'
},
{ role: 'user', content: 'write me a poem about cookies' }
]
These are the messages.
The system message has 11 tokens
The user message has 6 tokens
Resulting in 17
when i get the tokens for the stringified object it doesn't match the api either
versed patrol
1 token != 1 word, you can check out the openai tokenizer to see how tokenizing works: https://platform.openai.com/tokenizer
You'll need to use a tokenizing library, such as the GPT2/3 tokenizer from huggingface
storm bobcat
1 token != 1 word, you can check out the openai tokenizer to see how tokenizing ...
i never said that i did that, i use a tokenizer
in fact i use the one you gave me in a different thread
but that doesnt matter i do calculate how many tokens the system message is and it is always lower than the one reported from the api
i dont know what parts of the system instruction gets tokenized and counted towards the used prompt tokens
i calculate the total prompt tokens by adding the amount of tokens of each message
