Hey Guys👋Getting Request Timeout Issue on Load Test | Railway | Page 1

delicate glacier Dec 5, 2023, 4:11 PM

#

I deployed a node app with express under railway, everything using railway managed services postgres and redis,.

After some time on load test, i started seeing this issue

{"remote_ip":"34.87.74.158","remote_port":443,"url":" https://onebet-dev.up.railway.app/event/bet/place","status":0,"status_text":"","proto":"","headers":{},"cookies":{},"body":null,"timings":{"duration":60000.783,"blocked":0.001,"looking_up":0,"connecting":0,"tls_handshaking":0,"sending":0.125,"waiting":60000.658,"receiving":0},"tls_version":"","tls_cipher_suite":"","ocsp":{"produced_at":0,"this_update":0,"next_update":0,"revoked_at":0,"revocation_reason":"","status":""},"error":"request timeout","error_code":1050,"request":

How to solve this, is this issue due to ratelimiting on railway side or issue on node app side?

shrewd shardBOT Dec 5, 2023, 4:11 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

rigid tideBOT Dec 5, 2023, 4:11 PM

#

Project ID: 2cb48114-4aaa-4153-b45f-277a43de1723

delicate glacier Dec 5, 2023, 4:12 PM

#

2cb48114-4aaa-4153-b45f-277a43de1723

sharp isle Dec 5, 2023, 4:58 PM

#

railway has no hard set rate limits

#

tell us more about how you're load testing, how many connections are you able to I'm currently achieve before you get timeouts?

delicate glacier Dec 5, 2023, 5:38 PM

#

i was trying arounf 100 concurrent connection and total of 2000 requests

#

sometimes at 50 concurrent connections it is happening

sharp isle Dec 5, 2023, 5:38 PM

#

2000 requests per ??

#

can you give me more metrics?

delicate glacier Dec 5, 2023, 5:39 PM

#

2000 req in span of 3-5 mins

#

breaking at 50 or 100 rps

sharp isle Dec 5, 2023, 5:39 PM

#

what kind of data does this endpoint you where testing return?

delicate glacier Dec 5, 2023, 5:40 PM

#

it will return a json, takes input of a payload

#

the processing will take around 2 secs, it has 3 db calls from prisima and then grpc request

#

the load on db and server running the grpc is fine

#

i am not seeing any spikes on railway instance also

#

still it is getting timeout

sharp isle Dec 5, 2023, 5:42 PM

#

are you able to do more than 100 concurrent connections and more than 5000 req/5mins when hosting your app elsewhere?

delicate glacier Dec 5, 2023, 5:42 PM

#

that we havent tried

#

we will try that and update back,

#

but we wanted to understand how railway handle requests for more than 100 rps

#

usually 100-500 rps

hearty ridge Dec 5, 2023, 5:44 PM

#

Given that you’re on the hobby plan, you only have access to 8vCPU and 8GB of RAM. Can you please share your metrics for the period of time you were load testing? it’s possible you have maxed out the CPU

delicate glacier Dec 5, 2023, 5:44 PM

#

do we directly deploy node app and expose with railway, or is it better if we add some load balancer on top

hearty ridge Dec 5, 2023, 5:44 PM

#

Railway can load balance with replicas, that should be a non issue

sharp isle Dec 5, 2023, 5:45 PM

#

have you ran the same test locally and achieved more throughput? are you sure this isn't a bottleneck in your code?

delicate glacier Dec 5, 2023, 5:46 PM

#

hearty ridge Given that you’re on the hobby plan, you only have access to 8vCPU and 8GB of RA...

sure, ill share the metrics, is the project on hobby plan? my account is not creator of project

sharp isle Dec 5, 2023, 5:46 PM

#

what plan is the project owner on?

hearty ridge Dec 5, 2023, 5:47 PM

#

delicate glacier sure, ill share the metrics, is the project on hobby plan? my account is not cre...

I assumed that the project was on your account, your discord account doesn’t have the pro plan role

delicate glacier Dec 5, 2023, 5:47 PM

#

ill check the owners project plan, and also make load test locally and get back on this, thanks for the insights

#

#

these are the metrics in past 6 hrs

sharp isle Dec 5, 2023, 5:49 PM

#

metrics look perfectly normal, looking forward to the local load testing results

hearty ridge Dec 5, 2023, 5:50 PM

#

agreed. Looks like it’s crashing after a spike, but that spike is nowhere near your max

#Hey Guys👋Getting Request Timeout Issue on Load Test