#Cloudflare Fails to mitigate DDoS Attack, Enabling "Verified Bots" doesn't block deindexing it

113 messages · Page 1 of 1 (latest)

vapid sandal
#

For the past years, I believed Cloudflare was a trustworthy company, and have seen it as an example in the field. Highly ethical, always on the edge of innovation, full of competent people, great market coverage.

I even started slowly racking up stocks on Cloudflare beginning in 2019 due to this. But since then, I've seen more and more articles about how Cloudflare failed for them.

I've shrugged off the employment drama because a company needs to stay profitable and sales in Enterprise B2B is hard and requiring.

I've shrugged off the casino complaining of being dropped off Business plan when they were bypassing ISP limits and affecting the Cloudflare IP Trust and generating issues for other customers.

Now what should I believe when I myself have my whole site deindexed in a critical period for us (we're now leading in sanctioning the Govt. for a corruption case for mismanagement of EU funds, and Google is how people found about us)?

I've got DDoSed (Layer 7) quite heavily, I thought Cloudflare mitigates out of the box, but I was wrong. It doesn't, most of the requests (or at least plenty to overwhelm our backend server) were going through. Not DOS, DDOS. Hundreds, if not thousands of different IPs.

We managed to resolve and mitigate the attack with Super Bot Fight Mode which "detects bad bots". We allowed "Trusted Bots", which seems not to cover Google, Bing, Yandex bots via IPv6.

I haven't complained of extra usage that was billed on Workers / Pages, it was negligible. I didn't complain of extra usaged on D3, it was negligible, ~1$ per hour and we mitigated it fast with Super Bot Fight Mode.

But I am highly skeptical that Google will maintain us on #1 page after we answered 403 on our most important pages for days on end. We know we got deindexed, how google will treat the resolution of the 403s, not sure.

Screw cloudflare. We got Cloudflare for stability and for it's reputation. It seems things have changed since we started using them heavily.

craggy iron
#

Google is a trusted bot

#

So are all the other search engines you listed

#

You either had a custom rule that did not account for trusted bots or the 403 was served by your origin

vapid sandal
#

Bet?

#

Managed rule by cloudflare. Google is a trusted bot on IPv4

#

not on IPv6

#

"manage definite bots"

#

And this seems to be Bign

image.png
#

Any "Trusted Bot" is trusted only if it's not IPv6

#

Because ❤️ Cloudflare

#

And since the internet is slowly transitioning to IPv6, something that Cloudflare was proud to advertise it was also promoting, hey, this happen s, maybe some crawlers will use IPv6 if supported too

#

Tough luck for me

#

Impact of deindexing

image.png
#

And enjoy our main page deindexed due to 403

craggy iron
#

that's a bot that is hosted on google's cloud service

vapid sandal
#

That is Google

craggy iron
#

notice the lack of a user agent?

#

and the fact that is not a google crawler ip address

vapid sandal
#

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

craggy iron
#

same applies to bing, they would not have empty user agents

vapid sandal
#

I've sent you two pictures. The one lacking user agent is Yandex

craggy iron
#

you sent two pictures: google & bing respectively
no trace of yandex

vapid sandal
vapid sandal
craggy iron
#

hope this helps

vapid sandal
#

Ah, gotcha, it blocked our Worker

craggy iron
#

those are not google nor bing nor yandex

vapid sandal
#

It's cloudflare

#

We have a translation proxy

#

Google -> Translation Proxy - > Cloudflare Base-docs -> Tunnel -> Infra

#

So from what it seems, the managed challenge was triggering for Workers requests

#

jesus

craggy iron
#

I mean, those are definitely automated requests

vapid sandal
#

True

#

You'd expect it to bypass for their own infra thoguh

craggy iron
#

create a Skip custom rule that checks the origin of the workers subrequest

vapid sandal
#

Yes, the fix I know

craggy iron
#

well, that's a cross-site workers request

vapid sandal
#

The issue is it occuring

craggy iron
#

otherwise it wouldnt change the IP to the workers ip

vapid sandal
#

It's cross-site but on same zone, *.incorpo.ro

#

Wouldn't it be normal for it to go under verified bots?

craggy iron
#

no, verified bots are verified bots, not your own bots

vapid sandal
#

My expectancy when I get a service that is under the same provider is that it will not ban itself, or cause issues due to poor integration with mitigation

#

The scope of Pages / Workers and the current push of Cloudflare is for it to expand the services that Cloudflare provides

#

They left beta.

#

Ok, it's not "verified bots", make a "Allow workers from own zone" and enable by default as is normal and expected

#

Also, it shows source ASN as being from various, and the IP is from Cloudflare

craggy iron
#

the IP is the worker's ip for subrequests to cloudflare zones

#

it retains the original source ASN for firewall events and such though for some weird reason

vapid sandal
#

And that wierd behaviour triggered the managed bot because it was off

#

(Range wasn't fitting expected value)

proper prawn
# vapid sandal For the past years, I believed Cloudflare was a trustworthy company, and have se...

the ddos protection that cf has by default will mitigate blatant and known attacks, its better to miss 10 attacks than it is to break 1 site due to a system that is too aggresive.

Only you, as the site owner, can define what is and what is not an attack, the default ddos mitigation is a nice-to-have but not a silver bullet. The silver bullet is the powerful WAF that allows you to create incredibly granular rules to target only attack traffic

thorn pier
proper prawn
# vapid sandal My expectancy when I get a service that is under the same provider is that it wi...

this would potentially enable your own worker, or other workers to act as a reflective vector to carry attacks, the system as it is right now is in my opinion good.

With the current tools available, you have the option to authenticate your own workers and let them bypass other waf/rate limit rules while preventing other workers/cf services from external parties bypassing them, use a secret header and verify it with a snippet or a waf rule

vapid sandal
#

I agree, that is an issue. I'll add the flag for it not to have this implementation

proper prawn
vapid sandal
proper prawn
#

whether you use cloudflare or the fanciest enterprise provider, none is able to provide an out of the box flawlesss and automated mitigation, even if sales tries to say otherwise

vapid sandal
#

The issue was that enabling bot protection mode resolved the issue while also breaking Google due to this reflection fetch being blocked as a bad bot

proper prawn
#

yeah the bot protection mode is basically a way to say "screw anything that isn't a browser"

#

I believe they had some heads up about it on the docs, I wonder if those arent visible on the dash

vapid sandal
#

So it's not a page rule (eg: 401 for specific route), it's a a managed rule

vapid sandal
#

What happens is that if you are a verified bot, you access a reverse proxy via Pages / Workers, and the pages forwards that request towards the backend (also Cloudflare proxied via tunnels), it triggers bot detection on your worker

#

So the issue is the reflection mitigation with "global_fetch_strictly_public"

#

Solution I see here is just bypassing that

proper prawn
#

your best option in this case I believe would be to authorize your own workers to bypass all rules yeah

vapid sandal
#

Agreed

proper prawn
#

use a secret header and you should be good to go, as long as a malicious user isnt able to craft requests within the authorized workers it should be a non issue

vapid sandal
#

I personally believe managed rules should not run even with that script, as it creates edge cases, as this one is

#

Google requests URL that is public, but with Bot protection enabled

#

It will be allowed since the IP is in trusted list

#

Workers forwards that. It seems to forward the ASN / Useragent, etc, but not IP

#

On the public endpoint #2 - it detects it with ASN / useragent / etc from original request, but with IP outside of trusted bots list

proper prawn
#

the problem is the forwarding part, it can potentially be spoofed by somebody else, thats why the initial connection is the only one that can be verified against

vapid sandal
#

Is it normal then for the ASN to still show as Google / Hetzner / etc?

proper prawn
#

yeah

#

where im getting at is that somebody could use a worker and spoof the incoming ip addresses, this in principle is blocked but its something that, in theory, could be exploited, thats why the first connection is the only one you can fully trust

#

allowing requests that come from workers but report a valid ip from a bot would open a fair amount of attacks towards your backend/other customers backend

#

you could argue that no matter the situation you don't want your workers to be challenged, but its best to let the site owner make that decision

vapid sandal
#

I agree on the last statement, however what I don't understand is how it could lead to IP spoofing

#

The origin filter bypass is sane

proper prawn
#

somebody managing to set the x-real-ip, x-forwarded ip etc

#

and report anything

vapid sandal
#

Well, at that point I would say it's not Cloudflare's fault. The logic with a worker being able to fetch("") resources (eg: Ai agent with internet access) makes sense

#

And it's good for the original flag to disable such access and to route it back on top of the filter

#

It's highly unlikely for an application to allow setting X-Forwarded-For by end users.

#

Thanks for the help in finding about the flag

#

I'll enable it and hopefully it'll resolve the issue

#

There should be some mention on how it's behaving however. I know origin bindings were implemented a while ago but I wanted to be able to bind some routes to other workers and have my translation proxy handle that

#

I believe documentation needs to mention this on the super-bot-fight-mode, that if youi're using a proxy without origin bindings it will break

proper prawn
#

the rule of thumb is that whatever fetches something within cloudflare will be subject to waf/rate limit, etc, even if the original visitor is a known bot/trusted ip. I can see why this might be confusing when not considered

vapid sandal
proper prawn
#

Yep, any security related product

#

even the ddos protection itself, which would be a major problem because that one you don't have any way to control/filter, hence why its very conservative before blocking any traffic

vapid sandal
#

Being conservative by default is not a problem, it's common practice. You'd rather have higher load than lose customers

proper prawn
#

yup

vapid sandal
#

However we'd expect it to detect abnormal patterns, traffic spiked significantly on a single URL, with multiple consecutive requests on the URL by the same IP, etc

#

Some we are able to filter out (eg: We can add rate limits)

#

But they lack granularity unless you go forward with Business / Enterprise I believe

#

I'll resolve this, if we can resolve the issue with the middleware not filtering itself out

#

We know that we can just enable bot-protection-mode and it'll have minor consequences except for managed challenge which users are used to

#

Appreciate the fast response, I'll try to resolve it and hopefully next time it'll not break