Anyone experiencing this?...

[hoary pasture]

The Davinci API returning out of context replies to questions.... such as you ask it a simple question, and it renders a paragraph of dictation about something totally unrelated to your question lately....

[hoary pasture]

Still occuring... Anyone also experiencing this and like to share your thought on what's going?

[plain ether]

Davinci is a TEXT model, not a chat model. So wht you should do is smoething like this:

This is a conversation between a user and an AI.
User : YOUR QUESTION HERE
AI :```

[hoary pasture]

Ok I will try that and see if that improves it.

[hoary pasture]

I also notice that it cannot continue a conversation. Each time I ask a follow up question on a topic, it answers it like from a new topic, rather than continuing on the same topic from where we left off. Is this because it's a text model, and if I implement: "This is a conversation between a user and an AI.
User : YOUR QUESTION HERE
AI :", it will be able to converse?

[plain ether]

You can add more user and AI below each time a new message is sent. Yes, this will work

[gloomy atlas]

This is true even for turbo model. OpenAI has done this deliberately to limit token usage. If you want your custom AI to remember previous context to continue a conversation you need to add all the responses to an array and stack it where you: previous messages + new user prompt. That's how you build chatbots with openAI. Keep in mind this will result in more token usage.

[hoary pasture]

Good to know..... by stacking, do you mean the saved conversation topics that appear on the left pane such as on ChatGPT's desktop screen?

[gloomy atlas]

No, I mean sending the previous responses with every api call if you want the AI to remember the last conversation. You could send around last 4-5 assuming many tokens are not used and that will help the AI like so:
User: AI, remember my name is Marie okay?
AI: Okay
(send previous convos with this new prompt: ) User: What's my name?
AI: Your name is Marie.
(without stacking prev messages: ) AI: As an AI model, I don't have access to personal information, etc etc.

[hoary pasture]

This makes sense. So is it possible to apply this stacking to our GPT-3 API so that it behaves as I describe? And if so, can you share how?

[gloomy atlas]

Yeah you can do it with DaVinci model or Turbo. Kinda works the same for both. You just need to store the previous responses and pass that as a variable with the new prompt so everything goes to the OpenAI's api call and it gets the context from previous responses.

[hoary pasture]

For the GPT-4 API, will it be able to remember previous context to continue a conversation, like the ChatGPT version on desktop, or do I also need to implement the stacking to it?

[dim zealot]

Yeah I'm seeing that behavior. I added an item to their bugs list.
Submitted Prompt is the Following:
"{"prompt":"Translate to French: The quick brown fox jumped over the lazy dog. ", "max_tokens": 50}"

Expected Response:
Le renard brun rapide a sauté par-dessus le chien paresseux.

Actual Result
Response is the Following:
{"id":"cmpl-78wP7eGOrFP2jcE4I4vOeX27nfYfW","object":"text_completion","created":1682365485,"model":"davinci","choices":[{"text":"_____________\n\nTranslate to Spanish: Please leave the dishes in the sink. ______________\n\nTranslate to Arabic: Hello, how are you? ___________________\n\nWrite the number 55 in Arabic.

[gloomy atlas]

It won't remember. You need to send all previous responses with each new request

[hoary pasture]

So if Openai raises the token limi to the GPT-4 API, will it solve this problem?

[wary hinge]

It's not related to token limit. That's just how LLMs work. They don't have "short time memory" that they can rely on, so you need to pass the whole context every time.

[hoary pasture]

If that's the case then how come ChatGPT Plus is so much better when it basically uses the same data that are passed onto this API?

[wary hinge]

The chat app sends the whole message history that fits in the token limit(context window). Gpt-4 just has a bigger context window, if that's what you're asking.

[hoary pasture]

So then it basically comes down to the token limit imposed on the API's end right? Because let's say I discuss a topic that consists of multiple questions on ChatGPT Plus, it is able to converse to me smoothly like a human, by remembering all pre-texts to stay on topic, as well as understanding the intentions from my questions. Whereas, on the API version, it does not do this, but rather just generically answer each question without connecting their contexts. Picture asking a question to an an educated adult human, and then asking that same question to a 5 year old human child, and there is almost no comparison in the reply/conversation quality from the adult then from the child.

[wary hinge]

There's no difference between the chat model and the API one. The difference is in the front-end(the chat app), as it includes the message history in the request. In your own app you need to do something similar, if you want similar results. The model doesn't have any internal memory, so it treats every request as a new one.

The token limit would only affect how big of a message history you could include.

If there are important instructions that you want the model to always follow, you could make sure that message is always included in the request input.

[hoary pasture]

Got it.... so basically I need to implement techniques like Context Management or Natural Language Understanding onto my chatbot to give it the ability to answer questions sophisticatedly and being able to carry conversation flows like ChatGPT Plus?