Please take a look at our workers in endpoint h16kk1hi79s3t0 or kn0n8ry69jj1t7
All the workers are stuck at something!!
#Urgent! all our workers not working! Any network issues?
severe hull
quiet raven
Please take a look at our workers in endpoint h16kk1hi79s3t0 or kn0n8ry69jj1t7
A...
Maybe create a support ticket meanwhile i can help you check whats going on by seeing the logs
also whats your template ?
quiet raven
we're using our custom docker image
hmm did it ever worked yet?
quiet raven
how could I create a support ticket?
On the site by the contact button
severe hull
yes, we've running these for months without problem
quiet raven
quiet raven
yes, we've running these for months without problem
Oh can you see the logs of the worker when its stuck?
severe hull
yes, sure, I'll paste it here
quiet raven
nice that would help identify the problem
severe hull
two different worker logs. as far as I can see, I think there's definitely some kind of network problems.
These templates have been running for months without any changes.
for the first screenshot, after our logic is done the worker is just not doing anything.
for the second, we do some requests in our docker logic, and it seems these network requests are all failing
quiet raven
for the first screenshot, after our logic is done the worker is just not doing a...
Its on running state?
severe hull
yes, all stuck in running state
severe hull
I don't know. I'm just guessing there's a network problem in runpod now.
We've been using runpod heaviliy for months and this is quite urgent
These templates have been running without any problem, but since just a few hours ago this problem started happening
quiet raven
two different worker logs. as far as I can see, I think there's definitely some ...
can you copy the last line. the exception (In text form)
severe hull
here's our requests graph.
quiet raven
I don't know. I'm just guessing there's a network problem in runpod now.
We've b...
i c
severe hull
yeah I can paste the last line, but I don't think this will help you. it's just our docker logic.
2024-05-30T02:45:08.562489094Z exception in main_handler in validation check: <class 'requests.exceptions.ConnectionError'>: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
but please look into it asap.. 🙂
quiet raven
Im not really the guy that can access your account's deeply but in technical i can help
What region is this btw?
severe hull
we use all the regions. is this what you mean?
quiet raven
yeah I can paste the last line, but I don't think this will help you. it's just ...
whats this doing?
severe hull
we send a request to amazon s3 to store our image
quiet raven
we use all the regions. is this what you mean?
yeah might be a network outage in one of those regions, or an error in your side to another external service
quiet raven
we send a request to amazon s3 to store our image
Oh but it says validation check
severe hull
yes, but we checked locally to send a request to amazon s3, but that works 😦
severe hull
Oh but it says validation check
oh yeah, not only that, we have other things we do.
validation check means.. as far as I remember, we use Amazon Rekognition service to check for nsfw photos
quiet raven
oh yeah, not only that, we have other things we do.
validation check means.. as ...
can you check that service?
the connection to aws's rekognition can be failing
severe hull
we checked that, it works in my computer
the serious thing is, here when it prints "push_output_image" that means our docker logic is done.
normally after that, it should fetch the next runpod job to start, but it's just stuck here
quiet raven
okay i saw another user just posted this
""
"exception in main_handler: <class 'requests.exceptions.ConnectionError'>: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"
""
seems like there is a problem in runpod's network somewhere
severe hull
I think so too.
Would really appreciate it if you could take a look
quiet raven
I couldn't access into runpod's infra atm im sorry 😦
but im sure there's another internal guys working on this
severe hull
oh no..
quiet raven
For now what you can do is just create a support ticket, and if you have maybe you can send me the ticket id
severe hull
that would take too long.. I'm just DMing RunPod members when we first started using RunPod a year ago.
Thank you
quiet raven
that would take too long.. I'm just DMing RunPod members when we first started u...
Hahaha
severe hull
but they're not responding.. are they all off time?
quiet raven
btw, is your service is a public one?
severe hull
yes
quiet raven
but they're not responding.. are they all off time?
No, they're mainly online in US hours
severe hull
oh, we're in Korea and I guess it's sleeping time in US..
quiet raven
Hmm korea huh
the guy that reported the error seems to be also from korea #💻｜technology message
severe hull
possibly, this is urgent..