#Reporting/blacklisting poorly performing workers

26 messages · Page 1 of 1 (latest)

uneven raft
#

I've noticed that every now and then a bad worker is spawned for my endpoint which takes forever to complete the job when compared to other workers running the same job. Typically my job would take ~40s but there are occassionally workers that have the same gpu but take 70s instead. I want to blacklist these pods from running my endpoint so performance isnt impacted

vast needle
#

Do you have an endpoint id for it?

#

so you're running a specific exact same job that should take the same time but instead on another bad worker it takes longer?

#

i guess its best to create ticket to runpod, so they can check on their side

scarlet glacierBOT
uneven raft
#

Yes endpoint 8ba6bkaiosbww6

#

also the delay times are very inconsistent, some take 20s to start even with flashboot on

#

my regular cold start time is 5s

uneven raft
#

The same job, on the same machine is US-OR-1

3962d3cfdc73d09c9f46a2ead316d862.png
tepid drum
#

same, would love if you could specify in request if a request should not be directed to a worker id

#

I have a retry mechanism when executionTimeout happens, but then most of the times the job goes back to the same worker id : |

uneven raft
#

I've found that US-OR-1 location always has issues, whether its a slower worker, or workers that have broken gpus that wont even start the container. going to be removing it from the allowed locations for the time being

uneven raft
uneven raft
#

hey runpod, is there maybe a way to report these broken servers and have them fixed?

b9a818deb63e02423a68b498839c7f7f-1.png
rancid talon
#

I have the same issue, I really need a api to blacklisting these bad workers

pure flax
#

I am also experiencing this issue

rancid talon
#

I made a ticket on runpod's website, I told them that several of our companies really need to solve this problem. If they reply, I will synchronize it to the group.

uneven raft
#

yea, there should at least be some simplified way for us to report a worker and have somebody investigate&fix it

vast needle
#

Is it that often

uneven raft
vast needle
#

Ah yea of this shouldn't happen that often in good condition

sharp sky
#

Not having the same issue but the request is almost the same. Sometimes a worker fails because of internal errors and misconfiguration, out of space because of memory purging errors and etc (happens in less than 1%). Unfortunately when this worker is in the batch all tasks going to the worker - fails. So it is always a job to be done by hand kicking that worker out of endpoint to stop errors. Definitely need an API to kick unhealthy workers from endpoints!

uneven raft
wicked forge
#

I’ve been using 100s of pods lately and we run in to this issue almost every day. Some pods are just worse.

gusty bobcat
#

I have the ability to delist specific problematic servers while we investigate them.