Hi All, Probably a stupid Q but I can't find the answer in the API docs. The pricing is shown as the image attached, different for using 4K and 16K context. Is this determined by the length of the prompt? Or do you set it somewhere? Thank you!
#4K and 16K context
modest wolf
lime yacht
The default is 4k. You set it in the model.
gpt-3.5-turbo
gpt-3.5-turbo-16k
I'll add my own anecdote here, 16k context makes the output suffer. I'd avoid it unless you really need it. It's also twice the cost even if you don't use the full context. If you're able to split up your requests, you'd get better value and output by using the 4k model.
brave sentinel
Agree, here. 16k can hold more but doesn't mean it's able to pay attention to that much in one shot. The shorter 4k allows it to focus better on the actual goal of the prompt.