how can I count prompt tokens? with which tool I can do Tokenizer job?
https://platform.openai.com/tokenizer
#Tokenizer
frail shore
obtuse osprey
They link a nodejs package you can use for that at the bottom of the page. If you are using python you will want to look for a library called tiktoken
frail shore
oh tnx
frail shore
They link a nodejs package you can use for that at the bottom of the page. If yo...
it return this...
obtuse osprey
The length of that array should be the amount of tokens in the text
frail shore
The length of that array should be the amount of tokens in the text
oh gotcha
cyan ember
I'll be interested to find out if you get the same figures as are returned from the API. In my experience, the OpenAI tokenizer page and the API use slightly different algorithms. This is especially evident when you have Unicode characters in your prompt.
