#Claude Code failing to fetch a webpage

2 messages · Page 1 of 1 (latest)

round orchid
#

I have a llms.txt on a documentation site but when claude code tries to fetch it a 403 happens.
I assume it's because of "Block AI training bots" being set to "Block on all pages".

  1. Am I right?
  2. How to disable this for one page? I want to configure this in "blacklist" mode where I set pages where AI bots should not access it.
round orchid
#

Anyone? 😅