Serverless endpoint fails with CUDA error

[little mauve]

Hi! I have a WhisperX image, here is the repo that I used for the image: https://github.com/kodxana/whisperx-worker. One of the runpod workers for that image started throwing CUDA failed with the error CUDA capable device is a busy or unavailable error in response to every request. Once this worker was restarted, everything was fixed.

The problem is that it failed 18 requests before we found the error and fixed it. It is the first time this error has happened.

Is there a way to properly set up notifications if the worker fails a lot of requests, or maybe restart the worker if a few requests fail?
And what could cause this error?

[twilit shore]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[wooden mirage]

try to edit your endpoint

then set minimal cuda version to 12.4

scroll down to see this

[little mauve]

Thanks! I set it up. So, as far as I see, it is possible that once a few new CUDA versions are released, I'll have to double-check that there are GPUs that are not available for the selected versions and remove the unhealthy version in case there are?

And what about notifications for such cases? For some reason, the worker wasn't unhealthy; it only started trotting after 1 hour of returning errors, so I'd like to set up a notification if there are a lot of fails and, if possible, without writing a custom microservice for that 😅

[wooden mirage]

Just use a proper handling in your worker's container so that you can send a notification if you Want to

And this, if you don't update your container (not push / change your image tag) , apps inside usually stays that version