prompt responses getting cut off `finished_reason = length`

[raven tusk]

Hello,

I am using the demo code provided in the documnetation:

    const completion = await openai.createCompletion({
      model: "text-davinci-003",
      prompt: generatePrompt(question),
      temperature: 0.6,
    });

I have updated the prompt to a more elaborate question, but the responses are always cut short.

When logging console.log(completion.data.choices)

I get the following output:

[
  {
    text: '\n\nGrilled Chicken and Cilantro Lime Rice\nIngredients:\n\n-',
    index: 0,
    logprobs: null,
    finish_reason: 'length'
  }
]

What is the reason the responses get cut off?

[zealous bluff]

I think you can try response.data.choice[0].text

also add .trim() to remove space before and after the text output

[inner plinth]

I think this is caused by the tokens. If you allow more tokens used, it should not cut off text.
max_tokens: 2049, - At least that's what fixed it for me.

[zealous bluff]

Ah I might miss the point, good idea

max_tokens: 1680

add to the list of
model: "text-davinci-003",
prompt: generatePrompt(question),
temperature: 0.6,

[inner plinth]

2049 is max for max_tokens by the way.

[zealous bluff]

For text davinci 3 the max token is 4000

[inner plinth]

Oh alright. It just said max was 2049 in the docs 😄

[raven tusk]

Thank you both @inner plinth @zealous bluff , I was able to also find the max_tokens property.

Do you know how applications are handling larger outputs? Or are apps supposed to be designed around short output.

For example with ChatGPT the output can reach a few paragraphs, but with this API that would be impossible. So what are standards?

[zealous bluff]

ChatGPT is a fine-tuned model so it requires less calculation when generating the same amounts of output

[zealous bluff]

Base mode like text davinci are not pretrained so it's max prompt is limted

[zealous bluff]

But actually the max tokens of one prompt of chatGPT is also 4000 tokens like text davinci but when you reach the max length it will automatically cut the prompt

You can try tell chatGPT your birthday and then ask it again after a few conversations, it may forget it

[zealous bluff]

You can just set max to 4000 tokens, it can generate output long enough

[raven tusk]

@zealous bluff As I understand max tokens not only affects the size of the prompt you can provide to davinci but also the size of the response from davinci

Is there a way for that to be separated?

[zealous bluff]

Max tokens don’t affect the tokens of prompt you send to it

only limit the output

[zealous bluff]

I’m afraid there is no way to do this

[zealous bluff]

by the way what are you making, a chatbot?

[raven tusk]

Interesting, cause when I set max tokens and increased my prompt side the API gave me an error because my prompt length exceeded acceptable bounds

[raven tusk]

No trying to make a recipe app

[zealous bluff]

Ah I test it again I think I’m right. You might exceed the maximum token of 2000 or 4000 of a model

[zealous bluff]

Interesting thing

[raven tusk]

Ok so prompt size has no limits only Algo response size.

?

[zealous bluff]

The only limit to the prompt the max token of a model. For example text dacinci 3 is 4000 , ada is 2000