#Best practice to stop/crash the actor/crawler on high ratio of errors?

1 messages · Page 1 of 1 (latest)

stuck dome
#

Following snippet works well for me, but it smells... sb have a cleaner approach?

// Every 3s, check for the ratio of finished (=success) and failed requests and stop the process if it's too bad
setInterval(() => {
  const { requestsFinished, requestsFailed } = crawler.stats.state
  if (requestsFailed > requestsFinished + 10) { // when failed 10 more than finished, stop trying bro
    console.warn(`💣 Too many failed requests, stopping! (${requestsFailed} failed, ${requestsFinished} finished)`)
    process.exit(1)
  }
}, 3000)
obsidian pine
#

There is now some message on apify which comes I guess from the crawler when there are problems. So maybe you can use that if you find out what is generating that message.

stuck dome
#

This @obsidian pine guy knows stuff 🙏