#crawler in playground not extracting the content of all the sub pages

2 messages · Page 1 of 1 (latest)

rough badge
low hatch
#

Hi, @rough badge! Which subpages aren't working?

Firecrawl is likely unable to crawl certain pages because they are restricted in the website's robots.txt file, which prevents crawlers from accessing specific parts of the site. You can check these rules by visiting https://feverup.com/robots.txt