New chat api (gpt turbo 3.5)

[weak rune]

Over the api is there a possibility to add text to a existing conversation like in the web.

[queen bobcat]

You have to use the assistant that's what that is for so you send previous conversation to the assistant so it knows what you have been talking about. To send it with every message and you need to do something to maximize the amount of prompt space you can use so tag prior conversations and only send conversations that are relevant to the question or something like this to increase how much you can remember and work with.

[weak rune]

Doesnt that use tokens all the time

and how is that a chat if it doesn't work like in the web or liek chatgpt

[snow cypress]

I am wondering that too, chatGPT isn't saving by itself each conversation under some unique ID for some time at least?

It's a no brainer to send everything from the conversation with each new question, that gradually increase tokens that are spent

[weak rune]

exactly what I mean when you want to make a application that uses this chat feature in some way you'll always have to compute the whole text thus increasing cost extreamly

[calm pecan]

Im assuming that they will add fine tuning to gpt turbo later

[snow cypress]

Why do you mention fine tuning, if I understood well, that is 'training' of the model for your needs, and I only want to have a chats with it?

[stoic birch]

whats the purpose to have a chat when it doesnt work like a chat

[snow cypress]

How do you mean it doesn't work like a chat? I want it to work like regular chatGPT works - that is regular chat, I guess?

[calm pecan]

Well you can manipulate fine tuning to give chat context. If there's a message that your bot makes that's good, you write a system to add that message to the fine tuning model as a good example and give context to important messages.

The ChatGPT in the web browser defiantly have some sort of fine tuned model that can create context no matter the length of the chat

[weak rune]

yeah but the normal chats works that the ai has access to previous comments. And can give anwsers relative to it. For example taking reference to a bigger text. You would have to pay for every api request

[weak rune]

but you cant access context with the chat mode with the api

[calm pecan]

Yes. because the Quizlet, Snapchat, and Instacart examples wouldn't work with the character limit they have. They must have a fine tuned or context

[weak rune]

but you cannot access the context without sending the whole previous text every time which at some point will cost large amounts of tokens

[calm pecan]

Yes, and eventually you will hit the token limit

I've thought of some work arounds like removing old message from the message list but that sacrifices context. I've thought of a system that i have to manually set chats as important so they don't get erases in the context.

Or fine tuning a model that works for my needs.

[weak rune]

but still when trying to make a competitive system it will just be worse than chatgpt

[calm pecan]

I don't intend to reinvent the wheel, rather implement GPT technology into existing programs. and I think that's their goal with GPT Turbo

[weak rune]

but if you want to creat a webapp or so using the api where you want to have context for better results it just simply isn't possible with gpt Turbo there is literally no benefit over DaVinci except cost

[keen wadi]

Hey all!

all these systems like quizlet, snapchat, etc work either with a fixed context memory that sits under the 4000 token limit, or they use embeddings to create memory permanence

Check this out! https://towardsdatascience.com/generative-question-answering-with-long-term-memory-c280e237b144

[calm pecan]

Thanks @keen wadi

[keen wadi]

No problem! Let me know if you have more questions in here, I've done a lot of work with memory/conversational bots and can help out

[calm pecan]

Epic

[weak rune]

Thanks

[weak rune]

how would this work with gpt 3.5 turbo and does this mean in chat gpt for example not everythin is saved

[keen wadi]

This works the same with GPT 3.5 turbo, it has the same context limit

for ChatGPT, not really sure, but they likely call their own API with their own conversation state system

[weak rune]

but like for example couldn't embedding take aways crucial information?

[sweet dagger]

ChatGPT is a fine tuned GPT-3 model

But yeah you would want to store conversation context in the prompt and with embeddings you can make long term memory

[weak rune]

can you explain that with the embedding please i dont quite understand it. Like yeah you save it but at some point you will have to call it or hwo

[sweet dagger]

Yeah embeddings are used to store information in numerical vector stores which captures stuff like semantic meaning of the text. Then you can query those embeddings using semantic search.

If you're using them for long term memory, every time you ask a question it would find the most relevant parts of your chat history (stored in embeddings) and input that information into the prompt, essentially mimicking long term model memory.

[weak rune]

Lets say i every query should be bou d to a text and should only use the text as source. Wouldnt that mean that embedding doesnt work anymore

[keen wadi]

What do you mean?

You would have to embed that text source, in chunks of text

and then search through those embeddings to find the most relevant bits

[weak rune]

which means the text i would give to the api is shorter and thus costs less?

[keen wadi]

Yes that's correct, but at the loss of lower accuracy since it doesn't have knowledge of the entire text at any given point in time if it's longer than 4096 tokens

[weak rune]

so only do it if the text is very long got it

[keen wadi]

Well, you can do it even if the text is not long

embeddings are an effective way to get relevant bits of information to a query