Chain multiple API requests based on historical context

[simple nymph]

Hi,
In ChatGPT UI, I'm able to send multiple questions which assumes context from previous questions and I'm not sure how to implement that with API. For example if I have a following chain of questions:

1. Sort given list of keywords from highest to lowest relevance to Artificial Intelligence:
   machine learning
   probability
   economy
2. recommend more keywords
   This short question 2 will work in ChatGPT but not via API and I need to be repetitive and add everything from question 1.
   How I can configure API to leverage context from previous questions and answers?

[humble coral]

You have to pass the previous messages in, the model has no memory so it only knows what you give it

[west warren]

Sorry to resurrect this, but what is the recommended way to send the previous context? As system messages?

And if sending the previous messages as system content, would it "consume" tokens?

[wispy meadow]

Yes, it would consume tokens

[wispy meadow]

Have you looked into Langchain? It allows you to chain together LLM calls in a similar way to how you're describing

It lets you use the output of one LLM call as the input to the next LLM call, and so forth

[west warren]

sending the message history works, it's just a bit annoying to "spend" (already burned) tokens

[wispy meadow]

yeah, no other way tho

[west warren]

there's also the issue that one reaches the token limit fairly easily; I've been testing the app and longer threads, besides increasing the token costs with each prompt, also hit the upper limit; one solution I'm testing is to "summarise" the discussion and then use the resulting summary instead of passing the whole thread back and forth

[sly pumice]

I’m thinking a hybrid solution would work best for a chatbot. One approach is to have a kindof ’rolling window’ that passes in the last 300 words (for example). There is also the ’summary’ approach like you said, @west warren. It can probably really get as complex as you like. Have the whole thing constantly summarised, and then the summaries are written to a text file. Relevant passages can be then searched by a script, and passed back in as necessary. On a related note, I’ve been wondering what would happen if you passed in current time as part of the context data.

[sly pumice]

That langchain seems at first glance to be exactly what I need. I just hope it’s not too difficult to understand…

[wispy meadow]

You got this!

[sly pumice]

I’m honestly not sure I do. ’Finest rhizomatic instrument’ is not a familiar term to me. Should I be worried?

And what is a ’vectorstore’?

I have a lot of reading to do 😒

[sly pumice]

Yeah, I have given up on that. Here’s what i have so far: ```import openai
import os

Set up the OpenAI API key

openai.api_key = os.getenv('OPENAI_API_KEY')

Initialize the conversation history

conversation_history = ""

Define a function to generate a response using the GPT-3 API

def generate_response(prompt, conversation_history):
# Combine the conversation history and the new prompt
full_prompt = conversation_history + "\n" + prompt

parameters = {
    "model": "text-davinci-002",
    "prompt": full_prompt,
    "temperature": 0.5,
    "max_tokens": 60,
    "n": 1,
    "stop": "\n"
}
response = openai.Completion.create(**parameters)
response_text = response.choices[0].text.strip()

# Update the conversation history
new_conversation_history = conversation_history + "\n" + prompt + "\n" + response_text

# Limit the conversation history length using a rolling window
max_history_lines = 20
lines = new_conversation_history.split("\n")
if len(lines) > max_history_lines:
    lines = lines[-max_history_lines:]
    new_conversation_history = "\n".join(lines)

return response_text, new_conversation_history

Loop indefinitely

while True:
try:
# Prompt the user for input
user_input = input("> ")

    # Exit the chatbot if the user types 'quit'
    if user_input.lower() == 'quit':
        print("Goodbye!")
        break

    # Generate a response using the previous conversation history
    response, conversation_history = generate_response(user_input, conversation_history)

    # Display the response to the user
    print(response)
except openai.OpenAIError as e:
    # Handle any OpenAI API errors that occur during execution
    print(f"OpenAI API error: {e}")
except Exception as e:
    # Handle any other unexpected errors that occur during execution
    print(f"Error: {e}")

It’s pretty horrible so far. It needs a starting prompt, and should probably use that gpt-3.5-turbo special chat model with ’messages’ from ’the ai’ and ’the human’. It seems to be maybe sortof working though.? It’s at least an example of ’passing history back in’ as per OP’s topic.

Max history lines and max tokens should be increased, and maybe someone will have some luck with a suitable initial prompt.

Ooh, I think the prompt should come before the history, not after…

[sly pumice]

import openai
import os

# Set up the OpenAI API key
openai.api_key = os.getenv('OPENAI_API_KEY')

# Initialize the conversation history
conversation_history = [
    {"role": "system", "content": "You are a friendly AI"},
    {"role": "assistant", "content": "We're in a museum"},
    {"role": "user", "content": "I look around"},
    # Add more history here if needed
]

# generate a response using the GPT-3.5-turbo API

def generate_response(prompt, conversation_history):
    conversation_history.append({"role": "user", "content": prompt})

    parameters = {
        "model": "gpt-3.5-turbo",
        "messages": conversation_history,  # Pass thehistory directly as a list
        "max_tokens": 2000,
        "n": 1,
        "stop": "\n"
    }
    response = openai.ChatCompletion.create(**parameters)
    response_text = response.choices[0]['message']['content'].strip()
    tokens_used = response['usage']['total_tokens']

    # Updatehistory
    conversation_history.append({"role": "assistant", "content": response_text})

    # Limit length
    max_history_messages = 10
    if len(conversation_history) > max_history_messages:
        conversation_history = conversation_history[-max_history_messages:]

    return response_text, tokens_used




# Loop
while True:
    try:
        # Prompt the user for input
        user_input = input("> ")

        # Exit the chatbot if the user types 'quitchat'
        if user_input.lower() == 'quitchat':
            print("Goodbye!")
            break

        # generate
        generated_response, tokens_used = generate_response(user_input, conversation_history)

        # Display
        print(generated_response)
        print(f"Tokens used: {tokens_used}")
    except openai.OpenAIError as e:
        # Handle any OpenAI API errors that occur during execution
        print(f"OpenAI API error: {e}")
    except Exception as e:
        # Handle any other unexpected errors that occur during execution
        print(f"Error: {e}")```

Initial conversation history should probably be way longer, I had to cut it down to fit in a discord message.

Here’s proof that there’s some chatting going on tho. 🙂

[neon nexus]

Hi bunny. May I know why the tokens cost for each request and response on average 280-460tokens? It seems like tokens it took More than expected.

[sly pumice]

My initial ’history’ was set quite large, and the token number increases each time because the ’history’ is increasing. This is also called ’context tokens’ I think. I imagine that each request, when the ’history’ is filled will probably be 2000 tokens each time. Here is the price per 1000 tokens for gpt3.5-turbo, which is the chat opitimised model used on the chatgpt website

…and yes, since learning about ’context tokens’ or ’history’ or whatever, my own pricing estimates have skyrocketed!

[neon nexus]

🥺 following up soon. I just went through hell of documents from openAI. My brain is dying.🙂

You did it using Python?

[sly pumice]

Yes, I got help from chatgpt-4 on the website. I have not myself done anything with python for years

Try out that script. Do you have python set up?

[neon nexus]

I wonder I can do it like you. I have knowledge in C, Java , JavaScript now and mySQL. Just trying to kickstart now.

I don't know Python 🥺

[sly pumice]

I have had moments where I thought ’oh my god I can’t do this’ but after spending a few days first asking for ’hello world’ and then asking chatgpt 4 for help, I think that basic script is a good start for a chatbot with history. You can do it!

[neon nexus]

Some out of topic question from me. How much cost it took from free credits since you started to develop it? Enough for you? I saw about 0.30$ deducted from my free account after I tested someone chatbot using my own API key. That time consumed wasn't even more than 30minutes I think. I just asked few questions to the bot.

[sly pumice]

About the same. $0.40 in one session trying to use the davinci model. I wasn’t counting tokens that time though, and the price is different for the 3.5-turbo chat model so… more tests needed…