Can't seem to get the healthcheck to work, works fine locally over FASTapi and Flask | Railway | Page 1

gilded surge Sep 23, 2023, 12:09 PM

#

service ID: 2a262f8f-be17-475a-8463-21e12fafebbf
I really hate opening this ticket guys, i'm sorry in advance but i can't seem to figure it out, i'm sure it's something small i must of missed

im running a pretty big python repository with 7-10 seconds worth of healthchecks being done before returning status 200,
however when i deploy on railway it just keeps timing out

(the API itself worked fine with the current config, it's just the healthcheck endpoint which is acting up)
for context i am running main.py from my procfile and my API is in another python file, both are being initialized though)
Also the API has to run on port 4242 as its interacting with the stripe API via webhooks

tame needleBOT Sep 23, 2023, 12:09 PM

#

Project ID: 2a262f8f-be17-475a-8463-21e12fafebbf

gilded surge Sep 23, 2023, 12:10 PM

#

if anyone would have some sparetime and would maybe be willing to try and help me out i'd greatly appreciate it

gilded surge Sep 23, 2023, 2:56 PM

#

@steep osprey what are my options to run multiple processed concurrently if procfiles are off the board, i dont really wanna subprocess into different py files

#

w popen

steep osprey Sep 23, 2023, 2:57 PM

#

you wanna go over that before we get the health check working?

gilded surge Sep 23, 2023, 2:58 PM

#

that is the root problem of my health

#

as i have a main file

#

and i had a seperate api file to have my endpoints

#

and well i was trying to init both of them seperate via the procfile

#

hence the endpoint not working

#

deploying via 2 services is not really an option as that would defeat the point of the healthcheck

steep osprey Sep 23, 2023, 3:00 PM

#

interesting setup you have

gilded surge Sep 23, 2023, 3:00 PM

#

yes

steep osprey Sep 23, 2023, 3:00 PM

#

what does the main.py file do on its own?

gilded surge Sep 23, 2023, 3:00 PM

#

its the main handler as its a mono repo

#

the repo is like

#

8 or 9

#

k lines

steep osprey Sep 23, 2023, 3:00 PM

#

how many services in this mono repo

gilded surge Sep 23, 2023, 3:01 PM

#

i wire all thru 1

#

or well i did till the healthcheck had other ideas

steep osprey Sep 23, 2023, 3:02 PM

#

how many different ports are in use

gilded surge Sep 23, 2023, 3:02 PM

#

1 pre-set for stripe and 1 that railway assigns itself i believe?

steep osprey Sep 23, 2023, 3:05 PM

#

I see, but unfortunately, per service you can only expose one port publicly, and ideally your app listens on $PORT

gilded surge Sep 23, 2023, 3:05 PM

#

i see, i suppose i can route all over 1 port no? aslong as my endpoints are different

#

its only internal traffic that goes over that api

#

so its not a big deal

steep osprey Sep 23, 2023, 3:07 PM

#

yep the running solution for services that listen on multiple ports is to use endpoints

gilded surge Sep 23, 2023, 3:08 PM

#

its ok i only run stripe over 4242 and the other port would be health

steep osprey Sep 23, 2023, 3:08 PM

#

ideally in the future you would be able to map internal ports to different domains on port 443 externally

gilded surge Sep 23, 2023, 3:08 PM

#

i have 3 assigned domains

#

steep osprey Sep 23, 2023, 3:08 PM

#

but the healthcheck does need to listen to $PORT since that check is made internally

gilded surge Sep 23, 2023, 3:09 PM

#

steep osprey Sep 23, 2023, 3:09 PM

#

gilded surge

yeah there's no native way to map those external domains to internal ports on your service, without running a proxy that does host matching

gilded surge Sep 23, 2023, 3:09 PM

#

ahh

#

okay okay

#

thats a shame

#

endpoints it is

steep osprey Sep 23, 2023, 3:10 PM

#

if you want an example of that, I have one prepared

#

just so you don't need to modify any of your code

gilded surge Sep 23, 2023, 3:10 PM

#

yes sure

#

if you want

steep osprey Sep 23, 2023, 3:11 PM

#

okay one sec, let me find, since I whipped up an example for someone else that wanted to map internal ports to different subdomains on the same service

#

#1154106744306421830 message

#

if you need any clarifications on anything I said in that thread just ask

gilded surge Sep 23, 2023, 3:14 PM

#

quite impressive that you came up w that bro

#

jeesus

steep osprey Sep 23, 2023, 3:15 PM

#

everything I've learnt about railway is from being with the community

gilded surge Sep 23, 2023, 3:19 PM

#

thats amazing, i never been in any coding communities

#

but basicly the crux of what you say

#

i cant have 4242

#

if i want the health path to work

#

without a proxy server

steep osprey Sep 23, 2023, 3:22 PM

#

yeah since your app listens on different ports

gilded surge Sep 23, 2023, 3:22 PM

#

well guys

#

that was 4 hours

#

down the drain

#

💀

#

back 2 github actions i go

#

thanks for the help i appreciate it

steep osprey Sep 23, 2023, 3:23 PM

#

haha I was a sleep 4 hours ago, wish I could have gotten to help you sooner

gilded surge Sep 23, 2023, 3:23 PM

#

imma try 1 sketchy thing

steep osprey Sep 23, 2023, 3:23 PM

#

ouuu what ya got in mind

gilded surge Sep 23, 2023, 3:24 PM

#

imma subprocess the bitch

#

im sick of it

#

via daphne

steep osprey Sep 23, 2023, 3:24 PM

#

does what your going to subprocess need to access the same filesystem as the rest of the monorepo services?

gilded surge Sep 23, 2023, 3:25 PM

#

im subprocessing the endpoint for the healthcheckm

steep osprey Sep 23, 2023, 3:25 PM

#

gilded surge

because you could just run the 3 things separately in 3 different services

gilded surge Sep 23, 2023, 3:25 PM

#

it would defeat the point

steep osprey Sep 23, 2023, 3:25 PM

#

unless your healthcheck actually does more then just return 200

gilded surge Sep 23, 2023, 3:25 PM

#

the whole idea of having the health check halter is to stop any faulty commits coming thru

#

so if i have the same repo on multiple services

#

one would always need to be out of sync w commits

#

of the other

#

thats not really practical

gilded surge Sep 23, 2023, 3:26 PM

#

steep osprey unless your healthcheck actually does more then just return 200

i believe it does like

#

39 healthchecks internally

#

or 37 im not sure

steep osprey Sep 23, 2023, 3:26 PM

#

impressive

gilded surge Sep 23, 2023, 3:26 PM

#

and that just returns 1 int if all pass

gilded surge Sep 23, 2023, 3:27 PM

#

steep osprey impressive

yeah i mean we cant afford our main branch being down for some stupid reason

steep osprey Sep 23, 2023, 3:27 PM

#

that's a whole lot more thorough than return 200 in a /health route

gilded surge Sep 23, 2023, 3:27 PM

#

yes

#

isnt that the point of it though

#

or am i wrong hahaha

#

woops

#

steep osprey Sep 23, 2023, 3:28 PM

#

no you are definitely using a health check properly, though simply returning a fixed status code of 200 is still useful too

#

not for your case, but you know

gilded surge Sep 23, 2023, 3:29 PM

#

yeah but then i could just use the webhooks from railway no?

#

for the deployment status change

#

if thats all i want to query

#

does a healthcheck have to be completed in between the retries?

steep osprey Sep 23, 2023, 3:30 PM

#

for simple code bases it's often plenty to just return 200 so that railway knows when your new deployment is ready to start accepting requests, giving you less of a switch over period

gilded surge Sep 23, 2023, 3:30 PM

#

ahhh

#

i see i see

#

this is our main backend repo so 80% of our code is here

steep osprey Sep 23, 2023, 3:31 PM

#

railway will just retry your healthcheck endpoint for up to 5 minutes

gilded surge Sep 23, 2023, 3:31 PM

#

hence why it big

#

yes i know but it does like

#

20 retries or something

#

is there a time limit how long the check can last in between the retries

#

thats what i wonder

steep osprey Sep 23, 2023, 3:31 PM

#

and if it never gets a 200, it will never switch in your new deployment

gilded surge Sep 23, 2023, 3:32 PM

#

yes i like that

steep osprey Sep 23, 2023, 3:32 PM

#

oh I see what you mean

gilded surge Sep 23, 2023, 3:32 PM

#

as github actions is slow

steep osprey Sep 23, 2023, 3:32 PM

#

what is the timeout of the individual check

gilded surge Sep 23, 2023, 3:32 PM

#

hcheck takes longer than 10 seconds

#

i dont know

#

thats what i wonder

steep osprey Sep 23, 2023, 3:32 PM

#

that's a good question

gilded surge Sep 23, 2023, 3:32 PM

#

i have tried courotoutinng

#

everythin alrdy

#

but i cant get below 7 seconds

steep osprey Sep 23, 2023, 3:33 PM

#

I'm not 100% sure but I can't imagine it wouldn't wait for 10 seconds

gilded surge Sep 23, 2023, 3:33 PM

#

yeah i mean u never know

#

could be something small

steep osprey Sep 23, 2023, 3:37 PM

#

what are your deployment logs looking like during the healthcheck attempts

gilded surge Sep 23, 2023, 3:37 PM

#

attempt # 9392932 failed

#

every few seconds

steep osprey Sep 23, 2023, 3:38 PM

#

what does that correlate to in code? failed database connection or something?

gilded surge Sep 23, 2023, 3:38 PM

#

i dont know its generated by railway

#

it doesnt get past building

#

if it doesnt pass health

steep osprey Sep 23, 2023, 3:38 PM

#

yes it does actually

gilded surge Sep 23, 2023, 3:39 PM

#

#

ah ok

#

for me it doesnt

steep osprey Sep 23, 2023, 3:39 PM

#

your deployment is still ran, there should be deployment logs

#

since your deployment needs to be running for any type of healthcheck to work

gilded surge Sep 23, 2023, 3:39 PM

#

oh i see

#

wtf

#

i didnt see that up until now

#

#

build

steep osprey Sep 23, 2023, 3:40 PM

#

click deploy logs

gilded surge Sep 23, 2023, 3:50 PM

#

#

#

its clearly returning 200

steep osprey Sep 23, 2023, 3:50 PM

#

well isn't that odd

gilded surge Sep 23, 2023, 3:50 PM

#

but railway doesnt think so

#

thats my fastapi get

#

serving a non-200 before its done with the couroutine

#

then serves 200

gilded surge Sep 23, 2023, 3:51 PM

#

gilded surge

as u see here

steep osprey Sep 23, 2023, 3:51 PM

#

you run the healthcheck on every request to that endpoint?

gilded surge Sep 23, 2023, 3:51 PM

#

technically yeah

#

but its awaiting

steep osprey Sep 23, 2023, 3:53 PM

#

okay now throw in an early return 200 just for fun, skip the actual health check, as I understand your situation you are not working on a live site right now so it doesn't matter if stuff crashes?

gilded surge Sep 23, 2023, 3:53 PM

#

hahaha

#

no it is live

#

just not on railway

#

so its ok

steep osprey Sep 23, 2023, 3:54 PM

#

perfect, just as I thought

gilded surge Sep 23, 2023, 3:54 PM

#

the whole idea of railway was to use it like a node

#

with 2 more but it would need to fit in 1 service then

steep osprey Sep 23, 2023, 3:54 PM

#

gotcha

gilded surge Sep 23, 2023, 3:54 PM

#

#

it takes ages to build though

#

give it 4 min

steep osprey Sep 23, 2023, 3:55 PM

#

yep, let me know how that goes

gilded surge Sep 23, 2023, 3:55 PM

#

also i really appreciate you helping me

steep osprey Sep 23, 2023, 3:55 PM

#

is a 4 minute build normal?

gilded surge Sep 23, 2023, 3:55 PM

#

i have been frustrated with this the entire day

#

yeah bro

steep osprey Sep 23, 2023, 3:55 PM

#

are you deploying with a dockerfile?

gilded surge Sep 23, 2023, 3:55 PM

#

yes

#

its dockerizing it automatically

steep osprey Sep 23, 2023, 3:56 PM

#

haha well yeah, I was more so asking if you where bringing your own Dockerfile to the party

gilded surge Sep 23, 2023, 3:56 PM

#

ah no

#

its just a repo with raw .py

steep osprey Sep 23, 2023, 3:56 PM

#

gotcha

gilded surge Sep 23, 2023, 3:56 PM

#

#

it went thru the health now

#

so that means

#

there IS

#

an inidivual dtime limit

#

#

while as u saw my long ass healthcheck DID return 200 eventually

#

so thats not cool

steep osprey Sep 23, 2023, 3:58 PM

#

okay so there is a solution to this

#

how do I explain

gilded surge Sep 23, 2023, 3:59 PM

#

celery all of m?

steep osprey Sep 23, 2023, 3:59 PM

#

haha no, much simpler

gilded surge Sep 23, 2023, 3:59 PM

#

oh

#

WORD

steep osprey Sep 23, 2023, 4:01 PM

#

gilded surge thats my fastapi get

instead of running your healthcheck on every request of that endpoint, only let health.healthcheck ever run once, so that you will see a few failed healthchecks in the deployment logs while your single healthcheck check runs

#

then once your healthcheck finally finished updating the status code that route returns to reflect a failed or successful healthcheck

gilded surge Sep 23, 2023, 4:03 PM

#

so something like this

#

or

#

#

oh woops

#

wait inversed

steep osprey Sep 23, 2023, 4:03 PM

#

haha yeah

gilded surge Sep 23, 2023, 4:03 PM

#

#

i havent slept yet so my bad

#

that looks abour right no?

steep osprey Sep 23, 2023, 4:03 PM

#

let me think this over

gilded surge Sep 23, 2023, 4:04 PM

#

ok bro

steep osprey Sep 23, 2023, 4:05 PM

#

nah I can see this running a health check every request, since the healthcheck value will be zero until the healthcheck returns

gilded surge Sep 23, 2023, 4:05 PM

#

global cache it?

steep osprey Sep 23, 2023, 4:05 PM

#

yeah just make sure you sync access to the global value

gilded surge Sep 23, 2023, 4:06 PM

#

but it can only run once anyways no?

#

#

asyncio sleep it could be feasible too

steep osprey Sep 23, 2023, 4:07 PM

#

you could also have a Boolean flag named healthcheck_in_progress, and on the first request to your endpoint set that true, then run the healthcheck in a thread and update the flag to false and return the correct status

#

that way it's always an instant return of a non successful status code until the moment your app finished the background healthcheck, then the thread updates the status code the route returns and railway switches in your deployment

gilded surge Sep 23, 2023, 4:08 PM

#

i havent worked with threads that much

#

this is awaited / couroutined

#

does that mtter?

#

matter

steep osprey Sep 23, 2023, 4:09 PM

#

nah you could make it work

#

is health.healthcheck non blocking without the await?

gilded surge Sep 23, 2023, 4:10 PM

#

u cant run it without

#

it calls over a 100 functions

#

and theres stripe payments being processed

#

and well

#

thats self explanatory

steep osprey Sep 23, 2023, 4:11 PM

#

fair, then you'd need to await it in a separate thread to turn it in a healthcheck that runs in the background

gilded surge Sep 23, 2023, 4:12 PM

#

coding is so wild sometimes

#

what a great first project to do

steep osprey Sep 23, 2023, 4:15 PM

#

haha so what is this project anyway

#

and if you don't mind me asking (you don't have to answer) where do you currently have it deployed?

gilded surge Sep 23, 2023, 4:15 PM

#

haaaa man

#

you do not want to know

#

so technically this is just the wrapper yes

#

steep osprey Sep 23, 2023, 4:17 PM

#

GPUs

gilded surge Sep 23, 2023, 4:17 PM

#

steep osprey haha so what is this project anyway

yezzor

#

AI

steep osprey Sep 23, 2023, 4:17 PM

#

it needs GPUs?

gilded surge Sep 23, 2023, 4:17 PM

#

yes

#

lots

steep osprey Sep 23, 2023, 4:17 PM

#

are you aware railway doesn't offer GPU compute?

gilded surge Sep 23, 2023, 4:17 PM

#

i have the GPUs myself

#

this is the wrapper

#

on railway

steep osprey Sep 23, 2023, 4:18 PM

#

ohhhhh

gilded surge Sep 23, 2023, 4:18 PM

#

just as a node remember

steep osprey Sep 23, 2023, 4:18 PM

#

I see now

#

very cool

gilded surge Sep 23, 2023, 4:18 PM

#

yes

#

we made 2 trading algoritms last year and they did really well over the year

#

so now this is a step further

#

or well we, back then i had developers and i traded and just instructed

#

but now i learned how to dev myself

steep osprey Sep 23, 2023, 4:23 PM

#

that's awesome!

gilded surge Sep 23, 2023, 4:27 PM

#

brody

#

my friend

#

i owe you something

#

what a brilliant idea

#

#

global_healthcheck = 0

@app.get("/health")
async def health_repo():
    global global_healthcheck
    
    status_code = 300

    if global_healthcheck == 0:    
        _ = await health.healthcheck()
        global_healthcheck += 1

    return {"status_code": status_code}

#

this was the solution

#

thank u so much for ur time man it really means alot

#

man can i buy u a coffee or something hahahahah

#

fucking hell

steep osprey Sep 23, 2023, 4:29 PM

#

looks good!

gilded surge Sep 23, 2023, 4:29 PM

#

steep osprey Sep 23, 2023, 4:29 PM

#

if you want to you can, but you absolutely don't have to

gilded surge Sep 23, 2023, 4:29 PM

#

the commits of a dying man

#

send me ur paypal ill send u a coffee bro

steep osprey Sep 23, 2023, 4:29 PM

#

I hate uvicorn too, you should use hypercorn 🤣

gilded surge Sep 23, 2023, 4:29 PM

#

bro

#

anything with -corn

#

im not touching it anymore

#

holy fuck

steep osprey Sep 23, 2023, 4:30 PM

#

that's why I do golang, no silly things needed to get a web server running in production

steep osprey Sep 23, 2023, 4:30 PM

#

gilded surge send me ur paypal ill send u a coffee bro

I actually have a buymeacoffee, it's in my bio, but seriously you don't need to

gilded surge Sep 23, 2023, 4:31 PM

#

i just saw that yea how coincidental

#

real question though bro why do you NOT work for railway

steep osprey Sep 23, 2023, 4:31 PM

#

not qualified

gilded surge Sep 23, 2023, 4:32 PM

#

in what sense?

#

it seems like you're doing a pretty good job

#

https://www.buymeacoffee.com/brody192/c/6966224

Buy Me a Coffee

Brody is Helping in the Railway Discord server!

I love supporting creators!

#

thanks again

#

fuck me bro

#

i refactored

#

that entire file

#

6 times

#

just for railway to time out in between retries 😭

steep osprey Sep 23, 2023, 4:33 PM

#

haha I'll ask the team about increasing that time limit

steep osprey Sep 23, 2023, 4:33 PM

#

gilded surge in what sense?

in the sense that I have no work experience

gilded surge Sep 23, 2023, 4:33 PM

#

or atleast

#

let them write it down somewhere

#

because the docs werent really helpful

#

im sure i wont be the only one with an almost 1k line healthchecker

gilded surge Sep 23, 2023, 4:34 PM

#

steep osprey in the sense that I have no work experience

i see i see, how old are you?

steep osprey Sep 23, 2023, 4:34 PM

#

good point, I will put it in the docs once I get more information

#

I think 22

gilded surge Sep 23, 2023, 4:34 PM

#

u think

#

im 21

steep osprey Sep 23, 2023, 4:34 PM

#

I stopped counting

gilded surge Sep 23, 2023, 4:34 PM

#

whahwhahaw

#

real

#

me with my bug fix commits today

steep osprey Sep 23, 2023, 4:34 PM

#

🤣

gilded surge Sep 23, 2023, 4:34 PM

#

good lord

steep osprey Sep 23, 2023, 4:35 PM

#

thanks for the train btw, means a lot

gilded surge Sep 23, 2023, 4:35 PM

#

i mean bro you saved me some time thats for sure

steep osprey Sep 23, 2023, 4:36 PM

#

haha maybe

gilded surge Sep 23, 2023, 4:36 PM

#

alright bro i gotta go, imma be more active in this discord cus i learn alot

#

cya

#Can't seem to get the healthcheck to work, works fine locally over FASTapi and Flask