So I was trying to use ChatGPT to read a fandom wiki page for Coral Island, trying to organize some prices and availability in an easier way since the wikipedia page itself was being uncooperative in double sorting. Either way, talking with ChatGPT wanting to make a list filtered by season, sorted by base price, etc etc.. and chatGPT responded it was not allowed to access the page. I asked it why it could not access. It said that due to robots.txt it was not allowed. However, I installed a browser extension to monitor and tell me the robots.txt details of any website, and I don't see any issues. Screenshots shared:
#Robots.txt misunderstandings?
1 messages · Page 1 of 1 (latest)
Please follow the posting guidelines.
For better assistance from the community or an OpenAI team member, please follow the posting guidelines by clicking the button below.
➜ Need to contact OpenAI support? Visit https://help.openai.com
https://coralisland.fandom.com/robots.txt
it didn't called the browser, that reply denying the request was an ahllucination