Is the url pattern, includes, excludes, using regular expression? | Firecrawl | Page 1

stable sapphire Jul 16, 2024, 11:49 PM

#

Hi new to firecrawl, anyone know if the url pattern, includes, excludes, using regular expression?

true ocean Jul 17, 2024, 2:45 AM

#

We don't support regular expressions there at the moment, just glob patterns like /*

#

What do you need regex for in particular?

flint dove Aug 12, 2024, 10:34 AM

#

Hi @true ocean ,

I'm also encountering an issue with another website: https://ellijewelry.com/products/ohrstecker-2er-set-basic-kristall-weiss-925er-sterling-silber?variant=45406085185819. I want to scrape all URLs that include the variant part, but using product/* isn't capturing those URLs. Specifically, variant=* is not working as expected.

Is there a roadmap to include regex in the URL include feature? Or does anyone have any suggestions on how to effectively capture these URLs?

true ocean Aug 12, 2024, 6:42 PM

#

Got it. Is the issue that you're not getting urls with the variant=<id> pattern? Or that you can't set te includesPaths to ONLY get links with that issue

#

This may not be a issue with the filtering, but an issue with getting those urls in the first place

#Is the url pattern, includes, excludes, using regular expression?