Continue generating after hitting max output length.

[woven temple]

Hello, I see that there is a feature of continue generating if you accidentally cut off the message, but there is also the feature in OpenAI interface that once it hits the token output limit, it can keep on generating to continue the text since in most cases the context length is longer than the output length limit.

Example:
the message is merged into the previous one and allows to continue generating even after cutoff.

[chrome sierra]

This is already implemented. The API will send a finish_reason field in the chat completion response which tells us if it got to this point. Except with LibreChat, you can also hit "continue generating" at any time but it only seems to work well if the previous message was cut off

I'm re-reading your message to make sure I understand

in OpenAI interface that once it hits the token output limit, it can keep on generating to continue the text since in most cases the context length is longer than the output length limit

do you mean it will continue generating by itself?

[woven temple]

Yes I don’t know I think the model hit the output limit of 4K tokens for 1 message and stopped abruptly, there was no continue generating option.

[chrome sierra]

It was likely a failure of the API to give that finish reason field but I can also double check. In short this functionality is already implemented and expected behavior

[woven temple]

Ok thanks. I will check if I have the logs when I get home.

[woven temple]

I went ahead and checked, there seems to be no finish_reason.

[chrome sierra]

which endpoint are you using?

looks like Bing..

[woven temple]

Yes, it is Bing.

[chrome sierra]

Bing doesn't return a finish_reason like OpenAI. It's an unofficial API