Set 'ignoreHTTPSErrors' on a PlaywrightCrawler | Apify & Crawlee | Page 1

gloomy lava Mar 21, 2023, 1:01 PM

#

Hi everyone,

I need to set the ignoreHTTPSErrors flag for a Playwright crawler.
const crawler = new PlaywrightCrawler({ launchContext: { launchOptions: { ignoreHTTPSErrors: true, } }, requestHandler: router });
seems to work, but errors on

Type '{ ignoreHTTPSErrors: true; }' is not assignable to type 'LaunchOptions'.\n Object literal may only specify known properties, and 'ignoreHTTPSErrors' does not exist in type 'LaunchOptions'.",

What's the standard way of setting this?
Thanks in advance

cerulean lakeBOT Mar 21, 2023, 1:01 PM

#

@gloomy lava just advanced to level 1! Thanks for your contributions! 🎉

austere spindle Mar 21, 2023, 8:29 PM

#

The type is correct, this option is not there https://playwright.dev/docs/api/class-browsertype#browser-type-launch

BrowserType | Playwright

BrowserType provides methods to launch a specific browser instance or connect to an existing one. The following is a typical example of using Playwright to drive automation:

#

Can toy send me the site, I will test it

dense valve Mar 22, 2023, 8:57 AM

#

it is here when you create new context https://playwright.dev/docs/api/class-browser#browser-new-context

gloomy lava Mar 22, 2023, 9:22 AM

#

Hi Lukas, Honza, thanks for your answers!
I'm not sure I understand, do you tell me that I need to create a new context to set this flag?
I need this to be set at crawler level, can I define this flag for current browser context and those who will eventually be created?

gloomy lava Mar 30, 2023, 7:29 PM

#

@dense valve @austere spindle can you help me with setting this flag at a crawler level? What am I doing wrong?

dense valve Mar 31, 2023, 10:34 AM

#

I would try to change it in the browser context in preNavigationHooks but I am not sure it will work. If you share what site you want to scrape maybe more people can help you.

gloomy lava Apr 4, 2023, 11:46 AM

#

I can't share any specific site because we wan't to provide a list of URL for this crawler, some of them in 'http'. This is why we wan't to set this flag at the crawler level.

torpid stump Apr 4, 2023, 12:30 PM

#

Could you please try adding

browserPoolOptions: {
            preLaunchHooks: [
                async (_, launchContext) => {
                    launchContext.ignoreHTTPSErrors = true;
                },
            ],
        },

to crawler options? The thing is - launchContext has browser launch options, while ignoreHTTPSErrors is new context option, therefore you have to use the preLaunchHooks. I am not sure however whether it's launchContext.ignoreHTTPSErrors or launchContext.launchOptions.ignoreHTTPSErrors

gloomy lava Apr 7, 2023, 7:53 AM

#

It works, thanks for this Andrey!

#Set 'ignoreHTTPSErrors' on a PlaywrightCrawler