Tokens count bug

[haughty pollen]

in GPT 3.5 we can't set max token as 4096

[lament canopy]

If you leave max tokens out of your call, it defaults to inf or maximum. This is different to the previous version. So if you want 4096, just leave out max tokens. Also it seems that the error message is invalid because 4096 is the same as 4096 too.

[haughty pollen]

And thats the reason I told there is a bug that err is came from OpenAi

I just put it in embed

[haughty pollen]

So we dont need to use tokenizer to count messages tokens to subtract it from 4096 to set in max tokens?

[lament canopy]

Correct. It works out the max tokens. You only have to set it if you want it lower than the max

[haughty pollen]

Are u sure?

[lament canopy]

The default for turbo is inf. In Davinci you had to set it because the default was really low

[haughty pollen]

You are right

@marsh nexus check this out

[marsh nexus]

it doesn't apply to me because I always reserve 1024 tokens for the bot response

so i need to count tokens

oh actually I see what you're saying

in my example I could just make sure the number of prompt tokens is below 4096 - 1024