#Scapping/Datacenter IP

13 messages · Page 1 of 1 (latest)

tribal vale
#

There is no option to select "Datacenters" for a rule but there are 2 filters to play with:

  • "Known Bots", this might capture what you want, you can make an exception for "Verified Bot Category" maybe to prevent Google from scraping for example. Depends on if you want to stop all scraping or just the excessive ones.
  • "AS Num", lookup via WHOIS which AS number the IP belongs to and offer a challenge to those, you can for example challenge AS Num 14061 to challenge all IP's from the cloud provider Digital Ocean.

Keep in mind that a lot of traffic comes from datacenter IPs you might want to still want to allow like webhooks from other services or good bots uptime monitoring and the like. So be careful with sweeping rules that block large parts of the internet!

Good luck!

low hatch
#

?bot-noise

ripe cypressBOT
#

There are always bots lurking around in the web. They knock on every domain, IP address, and port constantly looking for security holes.
Most of them are rudimentary and try all exploits randomly, including ones for applications unrelated to your project, such as Wordpress or PHP.
These bots learn about your domains (including .workers.dev and .pages.dev) near-instantly using Certificate Transparency logs, which cannot be disabled and are always issued (even on non-Cloudflare platforms).

Some options to help reduce the noise from these bots:

low hatch
#

block em by ASN

low hatch
#

click on the link on the bot message

#

you cant jut turn it off

#

but you can redirect to a custom domain

viscid slate
#

You can’t

#

No

#

You can use Workers Assets, which doesn't have this issue

viscid slate
#

Basically Workers with Asset serving. I'd recommend moving to it wherever possible. Also because you can deploy it as a route on your domain, which means that there is no pages.dev/workers.dev to hit

tough scroll
#

Anyone using Cloudflare WARP (VPN), which Apple Private Relay uses as well, uses that asn