solar ginkgo
I'm using the GPT-4-1106-preview model with a custom knowledge base (two PDF files). Normally, I input 150 words and get a 500-word response.
Lately, asking a 4-5 word question has ballooned the input_prompt to over 30,000 tokens, while the output remains the same. It seems the input_prompt is adding thousands of unnecessary tokens.. any ideas?
potent girderBOT
Hi @solar ginkgo. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Hmm, I don't know enough to give you a confident answer yet. The extracts provided do not contain specific information about how the GPT-4-1106-preview model calculates tokens or why a short question would result in a large increase in the number of input tokens.
However, it's worth noting that the GPT-4-1106-preview model has a context window of 128,000 tokens, as mentioned in the OpenAI Models Documentation (https://platform.openai.com/docs/models). This means it can consider up to 128,000 tokens from the input when generating a response.
If you're seeing an unexpected increase in the number of input tokens, it could be related to how the model is processing your custom knowledge base or the specific format of your input.
For more information on how to format inputs to ChatGPT models and count tokens, you can refer to this OpenAI Cookbook guide (https://cookbook.openai.com/examples/how_to_format_inputs_to_chatgpt_models#4-counting-tokens).