I am doing something it could be called "web-scrapping" with a custom GPT for work. Is actually just searching for a few informations on some websites to fill in a database i would like to build. It is very basic stuff, i just need the bot to read the websites once and look for some information that mightbe avaliable and put into a spreadsheet (the adress, contact information, etc)... basic stuff. So I give the bot the Headers i would like it to fill out, the name of the companies for it to search the web, and off it goes to Bing with the web browing tool. The issue i am facing is that after doing the task sucessfully 5 times for sets of 3 or 4 websites at time , it just enters an endless loop of "completing the task" and falling into network error. And after trying to re-do the task sometimes i get the msg "The conversation is too long, please start a new one." ....Anyone has any idea on what is happening and what could be done to work around it?
#Network error and limits
tardy cove
plucky berryBOT
Hi @tardy cove. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Based on the information provided in the extracts, it seems like you might be hitting the token limit of the GPT model you're using. The GPT-4-32k model can handle 32,768 tokens or roughly 50 pages of text, and the GPT-3.5-turbo-16k-0613 can handle 16,384 tokens source (https://community.openai.com/t/308038). If your request is too large, you might need to break it up into smaller pieces.
The "network error" you're encountering could be due to a variety of reasons, and without more specific information, it's hard to diagnose the exact issue. It could be related to the specific websites you're trying to scrape, the volume of requests you're making, or other factors.
As for the "The conversation is too long, please start a new one" message, this is likely due to the conversation exceeding the model's token limit. You might need to start a new conversation after a certain number of exchanges to avoid hitting this limit.
For the web scraping part, OpenAI has a web crawler called GPTBot, which is used to improve future models source (https://platform.openai.com/docs/gptbot/usage). However, it's not clear from the extracts whether this can be used for the kind of web scraping you're doing.