Critical Cloudflare 503's on Express Backend | Railway | Page 1

twilit light Oct 3, 2022, 5:02 PM

#

Two quick questions on outages I've using Cloudflare + Railway

Does railway do any rate-limiting? If so, can this be disabled?
Is there a more lengthy post about issues when enabling cloudflare for backend services? (beyond what is mentioned here https://docs.railway.app/deploy/exposing-your-app)

What we're seeing is that immediately after enabling the Cloudflare orange cloud, the express backend chokes on 100% of requests returning 503 errors. I've had many back-and-forths with the Cloudflare team and the current guess is that the origin server (Railway) is throwing these 503s.

noble tide Oct 3, 2022, 5:04 PM

#

Does railway do any rate-limiting? If so, can this be disabled?
no, it doesn't do any ratelimiting

charred scarab Oct 3, 2022, 5:05 PM

#

Please share your custom domain's url, railway's static url & project id.

abstract olive Oct 3, 2022, 5:10 PM

#

Did you see the Cloudflare section at the bottom?

#

Where you need to have it set to full not flexible?

#

We don’t do rate limiting

twilit light Oct 4, 2022, 4:41 PM

#

Screen_Shot_2022-10-04_at_10.40.18_AM.png

#

I have it set to full not flexible

#

api.earni.fi is where all these issues are happening

#

Project: 46b499ee-c469-4cd0-8d59-e80657e4db3c

Service: 90a80fa7-879c-4463-8675-e3a1aeeafc8c

charred scarab Oct 4, 2022, 4:44 PM

#

Your application isn't responding

#

Open https://api.earni.fi/ in your browser and share a screenshot of that

#

Also, what's the url that railway provided you?

twilit light Oct 4, 2022, 4:45 PM

#

charred scarab Your application isn't responding

Short-term outage, apologies. Upstream API went down that my whole app relies on. Should be fixed soon

#

Screen_Shot_2022-10-04_at_10.46.03_AM.png

charred scarab Oct 4, 2022, 4:46 PM

#

twilit light Short-term outage, apologies. Upstream API went down that my whole app relies on...

cool, lmk whenever it's fixed.

twilit light Oct 4, 2022, 6:27 PM

#

Fixed! Ready for your thoughts 🙏

https://api.earni.fi/v1/health

charred scarab Oct 4, 2022, 6:49 PM

#

twilit light Fixed! Ready for your thoughts 🙏 https://api.earni.fi/v1/health

It returns OK for me

#

Does it return 503 for you?

twilit light Oct 4, 2022, 6:54 PM

#

No, because I've disabled cloudflare

#

The moment I enable cloudflare I get all the 503's included in the chrome inspector screenshot above

abstract olive Oct 4, 2022, 6:55 PM

#

twilit light The moment I enable cloudflare I get all the 503's included in the chrome inspec...

Can you put cf infront of api2?

#

So we can see

twilit light Oct 4, 2022, 6:55 PM

#

abstract olive Can you put cf infront of api2?

I cannot. If I enable cloudflare at all (grey cloud) it will still 503 all of "api.*"

abstract olive Oct 4, 2022, 6:55 PM

#

twilit light I cannot. If I enable cloudflare at all (grey cloud) it will still 503 all of "a...

Right but do it on api2?

#

Or add another route that we can check on?

#

Without repro it’s impossible to check

twilit light Oct 4, 2022, 6:55 PM

#

Here is Cloudflare support's most-recent message

Screen_Shot_2022-10-04_at_12.55.49_PM.png

#

I've been going back-and-forth with them nearly 2-months on this issue

abstract olive Oct 4, 2022, 6:56 PM

#

twilit light Here is Cloudflare support's most-recent message

We have a boatload of people who use orange cloud on Railway

#

So this is quite odd

#

It seems to be a config issue somewhere

#

Because this is very possible

twilit light Oct 4, 2022, 6:57 PM

#

abstract olive Right but do it on api2?

Config within my cloudflare configuration?

#

If cloudflare were doing the 503's, it would be a cloudflare error though

abstract olive Oct 4, 2022, 6:57 PM

#

twilit light Config within my cloudflare configuration?

Likely, since the Railway setup is pretty straightforward

#

But I need to narrow it down

#

Can you add an api3 that just points to api? And turn on orange cloud for it?

twilit light Oct 4, 2022, 6:58 PM

#

This first began with someone DOSing my systems two months ago. Thousands of requests per minute

twilit light Oct 4, 2022, 6:58 PM

#

abstract olive Can you add an api3 that just points to api? And turn on orange cloud for it?

I cannot, because currently cloudflare is "paused" at the global level for the entire domain. All routes are resolved with only the cloudflare nameservers. If I enable any cloudflare DNS, it will 503 the primary "api.*"

#

Everything is set to grey cloud in the above example ^

abstract olive Oct 4, 2022, 6:59 PM

#

But you can do it on a per URL level from the DNS section right?

#

Like what does your DNS page look like

twilit light Oct 4, 2022, 7:00 PM

#

Screen_Shot_2022-10-04_at_1.00.21_PM.png

abstract olive Oct 4, 2022, 7:00 PM

#

Ya don’t do that

twilit light Oct 4, 2022, 7:00 PM

#

Screen_Shot_2022-10-04_at_1.00.41_PM.png

abstract olive Oct 4, 2022, 7:00 PM

#

Go to DNS and toggle it on a route

twilit light Oct 4, 2022, 7:01 PM

#

Turn on orange cloude?

abstract olive Oct 4, 2022, 7:01 PM

#

On just that one route

twilit light Oct 4, 2022, 7:01 PM

#

If cloudflare is disabled it won't send traffic anyways?

#

I'll enable api2 now

abstract olive Oct 4, 2022, 7:01 PM

#

If you need api and api2

#

Add api3

twilit light Oct 4, 2022, 7:01 PM

#

I do not need api2, that was the point of making it

#

Enabled.

abstract olive Oct 4, 2022, 7:01 PM

#

Perf

#

Kk let’s see

twilit light Oct 4, 2022, 7:01 PM

#

Will check back in an hour, going into a meeting 🙏

abstract olive Oct 4, 2022, 7:02 PM

#

twilit light Will check back in an hour, going into a meeting 🙏

Sounds good

#

It looks fine to me BTW

#

https://api2.earni.fi/v1/health

#

CF is pretty instant

twilit light Oct 5, 2022, 1:42 AM

#

And you're verifying that is indeed going through Cloudflare? I don't know how api2.* could be using the cloudflare orange cloud if the global "advanced actions" of disabling cloudflare is still set to Disabled

twilit light Oct 5, 2022, 3:24 PM

#

Hey folks, api2 does not have cloudflare working still. Since my global config has cloudflare disabled

abstract olive Oct 5, 2022, 3:25 PM

#

Ya saw that quite odd :/

#

Do you have another domain you could point there and enable it?

#Critical Cloudflare 503's on Express Backend