Any way to force requests to alternate paths when one worker region is down?
#Helsinki workers down?
warped mango
rain pond
Any way to force requests to alternate paths when one worker region is down?
Workers Regions should never be down, are you talking about Workers AI by chance?
warped mango
no, I have global requests and all of my helsinki requests are failing
I'm not using AI
every other region is not failing
rain pond
no, I have global requests and all of my helsinki requests are failing
What exactly are you seeing, a specific error or something? Does a really simple worker like https://quick-test.workers.tylerobrien.dev/ work for example?
warped mango
I don't currently have ssh access to that server, it's a tad complex
rain pond
Workers run on the same machine as receives the request, it's a pretty uniform setup and every CF Edge Metal is running the workers service. There's no load balancer or anything. Workers being down in a region would be super super critical and be fixed before an alternate path would even be useful
warped mango
I'll try to log the response headers, one sec
rain pond
Cloudflare has internal workers like Zaraz, Challenge System, Waiting Room, Image Resizing/Images, Rocket Loader, every Cf API Request, etc that run on every request using the same worker instance, so they're very much invested in Workers themselves
rain pond
I'll try to log the response headers, one sec
yea, you can also tail the worker if you have access and try to see errors from it https://developers.cloudflare.com/workers/observability/logging/real-time-logs/#view-logs-using-wrangler-tail, either from dashboard (Workers & Pages -> Workers -> Logs or npx wrangler tail).
Something more specific I'd imagine
warped mango
its not reaching the worker
getting logs right now, one moment
I'm getting a Forbidden response
it's from a server though, not a client – why is this getting triggered
looks like cf-mitigated: challenge?
rain pond
Check Security -> Events in the cloudflare website to see what service is causing a challenge
rain pond
it's from a server though, not a client – why is this getting triggered
well that's all up to your security settings, unless you added a specific exception for server requests identified from someway it's all going to run through the normal waf/security settings/etc. CF's default security settings shouldn't cause that though, the security event will show the specific service causing it
warped mango
ah its hitting Bot fight mode but it's certainly not a bot. not sure why that would be triggered
rain pond
it's an automated request, from a server no? That would be a bot
You can disable it under Security -> Bots
warped mango
it's my own server hitting my internal api – it's not a known bot hitting any other network
rain pond
If it was a known bot like Google Crawler for example, those are excluded from Bot Fight Mode
Bot Fight Mode is designed as a high security measure to stop automated requests
Important considerations you need to be aware of before turning on BFM or SBFM
BFM and SBFM are high security features intended to quickly help customers under active attack stop as many bots as possible. Due to the high security threshold, false positives do sometimes happen. BFM has limited control. You cannot bypass or skip BFM using the Skip action in WAF custom rules or using Page Rules. BFM will be disabled if there are any IP Access rules present. If you turned on BFM during an attack, and the attack has subsided, we recommend either disabling the feature using IP Access rules to bypass BFM, or looking at Bot Management for Enterprise, which gives you the ability to precisely customize your security threshold and create exception rules as needed. SBFM can be bypassed with IP Access Allow action rules. You can use the Skip action in WAF custom rules to specify where Super Bot Fight Mode should not run.
I wouldn't even say that's a false positive though, it's designed to only allow Users/real browsers, not any unverified bots.
If you're on Free with Bot Fight Mode, your only option is to turn it off.
If you're on Pro or higher with Super Bot Fight Mode, you could create a Custom Rule to bypass it (or turn it off)
warped mango
that's a bummer. I'm going to move my workers to another domain so it doesnt trigger
rain pond
that also works lol
warped mango
thanks for the insight, I really appreciate it