I’m trying to make the max tokens used in the divinci model change dynamically but it’s not working. I’m using max_tokens = 4096 - len(word_tokenize(prompt)) but it’s giving errors
#dynamically changing the max tokens
storm oar
The error is saying I’m trying to request more tokens than I actually am
When I print out the length it says less than what the openai error is saying
unkempt basin
When I print out the length it says less than what the openai error is saying
A token is not a word, not a letter, not a sentence, it's a token. In order to make your token amount change dynamically you need to calculate the amount of tokens in your prompt and then 4096 - PROMT_TOKENS = MAX_TOKENS.
storm oar
A token is not a word, not a letter, not a sentence, it's a token. In order to m...
how do i calculate the number of tokens
unkempt basin
There is a great Stackoverflow thread explaining a solution https://stackoverflow.com/questions/72294775/how-do-i-know-how-much-tokens-a-gpt-3-request-used
safe swan
how do i calculate the number of tokens
Hey I explained this earlier in your other thread, you have to use a pre-existing model (a tokenizer) to count the tokens, and subtract it from the initial prompt, as Perny said
storm oar
i know that but i cant find a way to count the tokens
safe swan
I see that you're already doing that calculation, but your word_tokenize function most be wrong if it's not getting the right amount
The way to count the tokens is by using the tokenizer. Like I've already said and Perny has already said
It may be worth looking into that rust compiler issue that you have
storm oar
i cant import that