Persistent context for davinci-003 conversations?

[austere linden]

So I asked ChatGPT this but couldn't get a clear answer. When asking ChatGPT a question, you can include references previous messages in that particular conversation. I think of the conversation kind of like a 'session'. Is there a way to create a session with davinci such that it maintains a context of the ongoing conversation, and you can reference your previous messages? ChatGPT told me I could send the conversation history in the prompt, but that doesn't seem practical, since the requests will get more expensive with each question.

[formal juniper]

u included the "memory"/"history" in the prompt when making the API call

thats the only way to keep memory

[sullen chasm]

my davinci model has context, i recently implemented a great method, its still costly though

[formal juniper]

one trick is to summarize your chat history

[formal juniper]

https://beta.openai.com/playground

[edgy birch]

Pardon? Do you mean if you add the text "memory"/"history" to the prompt then the gpt model considers past informations?

[edgy birch]

good joke, it doesn't work, unfortunately

[last shard]

What they mean is that one needs to include the previous information. It doesn't have any memory.

[raven grail]

Yes, can confirm, that's what you have to do. Even the ChatGPT implementation I can guarantee you their backend is doing the same thing. Its just due to the way the ML model works.

[formal juniper]

i worded it poorly but yeah Ronald explained what i was trying to say
there is no true memory you are responsible for keeping track of the conversation and sending all previous context with the new question when making the API call
it creates a illusion that it has past memory.

[tiny breach]

we have two problems there: the conversation has a limit of 4000 tokens (when davinci) and the cost gets too high as the conversation gets longer. There should be a way to maintain this context without having to send it every request.

[formal juniper]

to save tokens u summarize the previous conversation after x length witch you would need to calculate and figure yourself on what would make the most sense

but i agree with the would be nice to have a way of maintain the context without sending but that just means openAI handles and they would need to offset the cost to us like charging extra for context/session saving

[tiny breach]

My backend splits requests texts between contexts and an question array, which when it reaches 3800 tokens it removes the first question and inserts the lasts, and going on

But is an good idea to, when it reaches an soft cap (like 1500 tokens) to ask to the API create an summary with important topics and store it in the conversation context

[tiny breach]

Yea but these costs for context saving could be a lot more affordable

[formal juniper]

yep on my other project i have it make a ghost call after it reaches a certain amount of text and summarize it the annoying part here is that now i have keep two logs for each convo

1. for the real log for people to read
   2.the summarized log for saving token

[formal juniper]

i doubt it
if they offer it they would also consider how much they are losing per request over time and offset it in the cost for saving our context

[tiny breach]

What the ChatGPT api will change instead using the actual text completions api?

[formal juniper]

in the end the people that constantly hit 4k tokens will be worth it
but for the average user would still be more expensive then saving it themself

[formal juniper]

most likely just the endpoint URL

this is me just dreaming but i hope they offer something like
if u purchase ChatGPT PRO u get to save X session/context for free with the API

[tiny breach]

“500 tokens for free in every request” could be amazing

But better than it an “conversation_id” could be SO FAR awesome

[wild junco]

Isnt fine tuning something like what this guy means?

[tiny breach]

You just start an conversation with the context and send text/responses per request

[tiny breach]

I don’t think so since fine tuning is an static information which responses an exact or similar response for what it learned

[edgy birch]

I am not sure about this. I tried to pass chatgpt table data and a lot of rules and asked to perform searches acc. to the rules. We had afterwards a lot of talk in this session. Maybe it can differ and repeatedly read the data and rules text but it didn't forget that.

[last shard]

GPT is many things. A law abiding citizen is not one of them. It follows its own rules