While using sendRequest causes Reclaiming failed request back to the list or queue. requestHandler timed out after 60 seconds.
Also, using PlaywrightCrawler with Proxy.
I am using following code (not same but similar):
{
nagivationTimeoutSecs: 300,
async requestHandler({ sendRequest }) {
async function getData(page: number = 1) {
const res = await sendRequest({
url: "https://example.com/api/foo/bar",
method: "GET",
})
if (res.next) await getData(page + 1)
}
await getData()
}
}