Hey I'm trying to extract the content of all these subpages and idk why is not working https://feverup.com/es/barcelona/when/this-weekend
#crawler in playground not extracting the content of all the sub pages
2 messages · Page 1 of 1 (latest)
Hi, @rough badge! Which subpages aren't working?
Firecrawl is likely unable to crawl certain pages because they are restricted in the website's robots.txt file, which prevents crawlers from accessing specific parts of the site. You can check these rules by visiting https://feverup.com/robots.txt