When using stream=true the API doesn't seem to return the usual "usage" key.
Am I missing something or is this expected but not documented?
#Usage key missing when using stream=true
dense jetty
acoustic yacht
This is expected I believe, the API doesn't know how many tokens it has streamed to you until the end of transmission, so it cannot give you a token count while streaming
clever pumice
I played around with the parameters and found out that... each event gives you a token :D so I think that's it, number of events = number of tokens
@acoustic yacht @dense jetty (I could be wrong)
of course, "how many tokens were used from the input" is still missing
😅 to count input tokens, if you give max_tokens greater than 4097, then it says:
"error": {
"message": "This model's maximum context length is 4097 tokens, however you requested 5015 tokens (15 in your prompt; 5000 for the completion). Please reduce your prompt; or completion length.",
"type": "invalid_request_error",
"param": null,
"code": null
}
But I dunno if that's allowed or not lol
more evidence that each event is one token
dense jetty
in the last message it could and should include the usage metrics.
One solution is to count tokens on our side using something like https://deno.land/x/gpt_2_3_tokenizer@v0.0.1/mod.js
clever pumice
I use something like that but it's not 100% accurate
I agree with your message, but I haven't heard much from OpenAI
I will start a discussion with the help soon about this