Setting cookies is failing

[minor nexus]

Error:

/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:330
                error: new Errors_js_1.ProtocolError(),
                       ^

ProtocolError: Protocol error (Network.deleteCookies): Invalid parameters Failed to deserialize params.name - BINDINGS: mandatory field missing at position 6418

Code :

const goodCookies = session.getCookies('https://www.mywebsite.com/')
console.log(goodCookies) // I see the cookies and they look fine, a list of dicts.
page.setCookie(goodCookies)

[short wind]

@minor nexus just advanced to level 5! Thanks for your contributions! 🎉

[minor nexus]

@rough nimbus regarding cookies.

[rough nimbus]

@minor nexus Are you using pupeteer? https://pptr.dev/api/puppeteer.page.setcookie
then each of the cookies has to be provided as separate paramter. Try:

await page.setCookie(...goodCookies)

[minor nexus]

Ok, then it also need to be changed in Crawlee docs, will try and come back with an answer

[rough nimbus]

may you provide the link for crawlee docs?

[minor nexus]

https://crawlee.dev/api/core/class/Session#getCookies

same error.
What I'm trying to achieve : do one request, get cookies from there, and use them in all the future requests.

[rough nimbus]

I am not sure do you mean by need to be changed in Crawlee docs . The getCookies is working fine, I was mentioning your usage of setCookies that it require to provide each cookie as separate parameter, which is same approach as in Puppeteer.

I just did PoC:

router.addHandler('detail', async ({ page }) => {
    /// ...
    const cookies = await page.cookies();

    await Actor.setValue('cookies', cookies);
    /// ...
});

const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});

And it seems to be working fine. Once the cookies are stored, they are used in all upcoming requests.

[minor nexus]

The PoC looks nice, but how do I filter the cookies in the preNavigationHooks, just for one URL ?

I have :
url1 , url2 ... url 50.
I want to take cookies from url1 and use them in url2...url50

[minor nexus]

still not working, I've pasted the entire cookies and error in paste bin

https://pastebin.com/98kunwBf @rough nimbus

[rough nimbus]

// main.js
import { Actor } from 'apify';
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

await Actor.init();

const startUrls = ['https://example.com']; // Inital URL where the cookies are taken from

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    maxConcurrency: 1, // make sure we start with only one request
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();

// routes.js
import { createPuppeteerRouter, utils } from 'crawlee';

export const router = createPuppeteerRouter();

router.addDefaultHandler(async ({ page, log, crawler }) => {
    crawler.autoscaledPool.maxConcurrency = 4;

    // Queue next requests, due to prenavigation hook they will be provided with the cookies
    await crawler.addRequests([
        { url: 'https://example.com/1', label: "OTHER" },
        { url: 'https://example.com/2', label: "OTHER"  },
        { url: 'https://example.com/3', label: "OTHER"  },
        { url: 'https://example.com/4', label: "OTHER"  },
        { url: 'https://example.com/5', label: "OTHER"  },
    ]);
});

router.addHandler('OTHER', async ({ page, log, crawler }) => {
    console.log(`My cookies are`, await page.cookies());
});