Model text-davinci-003 does not support dialog (maximum length of dialog memory is one message)

[sly nest]

My settings:

const completion = await openai.createCompletion({
             model: "text-davinci-003",
             prompt: text, 
             // temperature: 0.4,
             max_tokens: 2048,
             top_p: 1,
               frequency_penalty: 0,
             presence_penalty: 0.6,
});

When the bot success answers my question, I say to him "tell me more" and he gives me an answer completely unrelated to the context of the dialogue

Example:

• user: What is React.js?
• bot: React.js is an open-source JavaScript library for building user interfaces. It allows developers to create interactive UIs by breaking down the user interface into smaller parts called components that can be reused in multiple places. The library also provides built-in features such as server-side rendering, and state management. React is used most often for building complex Single Page Applications (SPAs) and mobile web apps.
• user: Tell me more
• bot: Linear programming is a method of analysis that uses mathematical concepts to solve complex optimization problems. It is used to maximize or minimize some target value or function, depending on the data - constraints or relationships.

[gloomy stratus]

Are you sending the whole conversation up to “tell me more” on the second time?

[sly nest]

The second time I ask "tell me more." In the official chat (https://chat.openai.com/chat), the bot fully supports the dialogue, keeping the context of the dialogue

In the official chat (https://chat.openai.com/chat), used is "text-davinci-002-render" model, but maybe someone could help make/adjust the "text-davinci-003" model as similar to "text-davinci-002-render" as possible

[gloomy stratus]

I see, a little bit of confusion here.

[gloomy stratus]

First and foremost, the model behind ChatGPT is not stateful - that is, it does not have memory of previous interactions. Rather, the ChatGPT app implements this functionality by providing past interactions (to a max of 4000 tokens, which is the same limit in other GPT-3 models).

https://help.openai.com/en/articles/6787051-does-chatgpt-remember-what-happened-earlier-in-the-conversation

That being said, you can make GPT-3 DaVinci 003 seem stateful doing the same trick. For each new prompt, do something like:

1. Approximate the token count of the prompt (prompt_count)
2. Get 4096 - prompt_count - 400 tokens from the previous interactions and prepend this to the new prompt

Notice the 400 I subtracted. IIRC, you need to leave space for the response otherwise it won't work.

[sly nest]

Oh, thank you, it all makes sense now

[wooden swallow]

Hello,

I am trying to understand the technique you mentioned for making the GPT-3 DaVinci 003 model seem stateful. Could you please provide more information or clarification on the following points:

• How does approximating the token count of the prompt?
• What should be included in the new prompt?
• Do I need save only last reply from API, or couple of them?

If you can provide peace of code, it will be very helpful.

Thank you

[gloomy stratus]

For calculating the token count, you can either run your input through the GPT-2 tokenizer (it’s the same as GPT-3) and see how many tokens you end up with or approximate with a 1.4 ratio. This step is optional, just helps not exceeding the 4k limit as conversations get bigger.

[gloomy stratus]

You should save all replies. You must define a conversation window and only include in your new prompt interactions on that window.

So if your window is 8 and this session already has 12 interactions, you should only send the last 8 of the 12 interactions along with your new prompt.

It’s just a FIFO