Heavy latency lately? | Railway | Page 1

unique bobcat Jan 28, 2025, 1:48 PM

#

I've been running an app for over a year already in Railway, and suddenly we started experiencing random latency spikes on the app responses. Tried improving and testing through the whole code in case we bugged something out, but there are random moments in the day when response time goes from 0.5-1 second to 16-19 seconds. It's even weirder cause we though it would be a resource usage issue, but the server still has enough resources left, and whenever there is a user spike response time / performance doesn't decrease, so it isn't a concurrent users issue either.

Is there anything going on with the service?

icy terraceBOT Jan 28, 2025, 1:48 PM

#

Project ID: c17693fa-8f81-4f00-ad4d-8f231f687806

unique bobcat Jan 28, 2025, 1:48 PM

#

c17693fa-8f81-4f00-ad4d-8f231f687806

lethal kindle Jan 28, 2025, 1:49 PM

#

Are you using a metal region?

unique bobcat Jan 28, 2025, 1:50 PM

#

us-west1

lethal kindle Jan 28, 2025, 1:52 PM

#

I would tell you to analyze this problem further, the team will probably ask for more data because if it were a general problem there would be more cases here

#

Are you using something like uptime kuma to monitor this latency?

lethal kindle Jan 28, 2025, 1:54 PM

#

unique bobcat us-west1

also, there's a metal us-west and a gcp one

unique bobcat Jan 28, 2025, 1:54 PM

#

I'm measuring the backend response time as stated earlier, that's why it caught my attention

#

users started complaining about random disconnections

#

and after checking the measurements, response time went from less than 1 second to almost 19 seconds per request

#

Some hours ago having 500 users at the same time still kept the 1 second time

#

Now there are just 60 and server is delaying up to 19 seconds per request

#

No logs, no errors, no anything. Tried even the good ol' reset just in case, even with a fresh start there is heavy lag now

lethal kindle Jan 28, 2025, 1:56 PM

#

gotcha, let's wait for a team/conductor to answer this thread

unique bobcat Jan 28, 2025, 2:11 PM

#

Okay, now all out of a sudden it went from 22-25 seconds per response down to 3-7 seconds, still laggy, but a drastic improvement

#

Again, no change at all in the code itself, in fact logged users increased, so it isn't userload either

earnest pollen Jan 28, 2025, 3:41 PM

#

This sounds to me like your database and backend service are in different regions. Can you please send screenshots of both? @unique bobcat

unique bobcat Jan 28, 2025, 3:43 PM

#

both say US WEst Oregon

earnest pollen Jan 28, 2025, 3:43 PM

#

Please send screenshots

unique bobcat Jan 28, 2025, 3:45 PM

#

#

earnest pollen Jan 28, 2025, 3:46 PM

#

Great, that rules out metal/nonmetal

#

If you have any quantifiable data, such as a grafana dashboard, please share that

unique bobcat Jan 28, 2025, 3:50 PM

#

Sadly only what I stated before

#

userload and average response times

honest sierra Jan 28, 2025, 3:50 PM

#

can you please try to switch both services to the v2 runtime

unique bobcat Jan 28, 2025, 3:51 PM

#

honest sierra can you please try to switch both services to the v2 runtime

where is that?}

honest sierra Jan 28, 2025, 3:51 PM

#

within the service settings

unique bobcat Jan 28, 2025, 3:52 PM

#

found it

honest sierra Jan 28, 2025, 3:53 PM

#

if this doesnt change anything we would have to recommend you setup tracing so that you can pinpoint where the "slow" is coming from

unique bobcat Jan 28, 2025, 3:54 PM

#

I'll be testing with the V2 then

#

never knew that was there

#

what does it change?

honest sierra Jan 28, 2025, 3:55 PM

#

moves the workload from docker to podman

unique bobcat Jan 28, 2025, 4:00 PM

#

So far average response time went down to 1.6 seconds, nice

#

will keep updating just in case

honest sierra Jan 28, 2025, 4:03 PM

#

sounds good, I'll leave this thread open, but just know that if you see an increase in latency you will need to add tracing to your app

#

Railway has no observability into what your code is or is not doing.

lethal kindle Jan 28, 2025, 4:16 PM

#

@silence @honest sierra how did u know they weren't running in runtime v2? your admin superpowers?

#

silence fail

honest sierra Jan 28, 2025, 4:26 PM

#

yes, though i didnt look at anything the user couldn't see themselves, if you see a runtime selector set to legacy then you are on legacy, if you do not see a runtime selector at all, then you are on v2, and the selector isnt there because you cannot go back to legacy

unique bobcat Jan 28, 2025, 4:43 PM

#

when did the runtime change?

lethal kindle Jan 28, 2025, 4:44 PM

#

A long time ago, lol

#

As of 2024/06/04 (YYYY/MM/DD)

unique bobcat Jan 28, 2025, 5:13 PM

#

makes sense, I have it running since 2023 almost haha

earnest pollen Jan 28, 2025, 8:17 PM

#

lethal kindle @silence <@539512869780455445> how did u know they weren't running in runtime v2...

It's @silent lol

#

@lethal kindle

lethal kindle Jan 28, 2025, 8:18 PM

#

oh yeah

#

mb

earnest pollen Jan 28, 2025, 8:18 PM

#

s'all good

honest sierra Feb 2, 2025, 6:46 PM

#

!s

#Heavy latency lately?