#Get completion by id?

25 messages · Page 1 of 1 (latest)

potent field
#

I am using the node sdk to stream gpt-3 results to our frontend. This is a great end user experience but I still need the usage info so that we can attribute usage to our end users.

Each chunk that is returned has the completion id but no usage info and there doesn't seem to be any other way to get the usage. Is there a way to lookup past completions by id so that I can track usage better?

#

Here is what each streaming chunk has:

{
  id: 'cmpl-***',
  object: 'text_completion',
  created: 1673670360,
  choices: [ { text: ' hey', index: 0, logprobs: null, finish_reason: null } ],
  model: 'text-davinci-003'
}
edgy goblet
#

Hi there, the usage information is displayed in the response, it's under ["usage"]["total_tokens"]

#

Then divide the amount of tokens by 1000, then multiply that value by the model price on the OpenAI website

edgy goblet
potent field
#

Yes, I agree it should be there but it is not. This only happens when I am streaming the results back. Otherwise I get the full response back.

edgy goblet
#

Oh, because with streaming it won't know the token count until the end of streaming

#

hmm

#

So it says that when the data is done streaming it responds with a data: done message?

potent field
#

Yeah, it would be great if the last response contained the tokens or if I could lookup the finished completion by id but I can't seem to find anyway to use the completion id even though it is returned.

edgy goblet
# 
Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message.
#

Maybe this means that it sends back a response at the end with token information?

#

but I doubt it

#

I guess the best way for now is to use streaming mode off so it gives you the token info

#

yeah there's no way to lookup the completion by ID, I know that for sure right now

potent field
#

It would be nice if the moderation api returned the tokens, too - or something hah. I have been messing with a lot of options because having the streaming enabled is a much better end user experience.

Anyways, thanks for jumping in. Will update this if I do figure anything out.

edgy goblet
#

The moderation API is free though no? So there is no token usage

potent field
#

But it still tokenizes the text so I can get the # of tokens back from it. I would use it to get a count back on the tokens while I streamed the text to the end user.

It doesn't return the # of tokens anyway though so just another shot in the dark trying to get something working.

gusty nova
#

I do something like this if it may be of help, which gets roughly the right number of tokens :) However, it's not perfect, sometimes it fails to process some strings, that's why I have a hack in the catch 😅

import GPT3Tokenizer from 'gpt3-tokenizer';
import {logMessage} from "../utils/logMessage";

const encoder = new GPT3Tokenizer({
    type: 'gpt3',
});

let printedWarnings = 0;

export const encodeLength = (input: string): number => {
    try {
        return encoder.encode(input).bpe.length;
    } catch (e) {
        if (printedWarnings < 10) {
            printedWarnings++;
            logMessage(`Encoding error: input: 
\`\`\`js
const input = \`${input.replace('\`', '\\`')}\`;
\`\`\``, e);
        }
        return Math.floor((input ?? '').split(' ').length * 2.30);
    }
}
#

@potent field ^

gusty nova
#

I tried various methods in the api but it seems there's no GET handling for completions/id

potent field
#

Thanks that is helpful. That is the direction I ended up going, too. Hopefully we will get better stream support and a GET request for completions by id.

gusty nova
#

something else you can do is

#

don't do streaming via the api