My serverless does not deploy the new releases

[atomic schooner]

I was using 1.2.4 version of my image on an endpoint. I started a new release (1.2.5) but workers still works on 1.2.4. To be sure, I released 1.2.6 version and deployed. I have waited 3-4 hours between this version deployments. However, still it uses 1.2.4. What could be the problem?

Thanks.

[dapper gulch]

Do you see Latest and Stale workers on your endpoint page when doing a new release?

[atomic schooner]

Yes, I have 3 stale worker, and 1 extra worker.

When I start a new release, it says initializing.. however still uses the same image

[hasty storm]

Refresh your browser?

Who knows it's not updatign in your browser

[dapper gulch]

Nah its not a browser issue, he said its still using the old image version, must be some serverless bug, I would log a support issue for this on the website.

There should be "Latest Workers" with the new image and there aren't any, so something is broken.

[atomic schooner]

Yes, I agree with you. I deployed 1.2.5 yesterday. However I still have 1.2.4 on workers 😄

[hasty storm]

hmm try creating a new endpoint then?

btw, which region is it and what gpu model did you select?

mine works, i think its should not be a serverless bug

[dapper gulch]

Maybe low availability of the required GPU type?

[hasty storm]

yeah i guess that

[pure flicker]

I have found that with :latest it will not update. Rather if you specify a specific version i.e. :1.01 when you change it to 1.02 (or other) the update will work. You cannot use :latest and assume it will update. It will not. You have to edit the template and provide a tag runpod has never seen.

[atomic schooner]

I dont use latest tag.
I use the cheapiest worker option 16 GB GPU but it says high availability. However, the fact that it was not deployed even after 1 day seemed to be a problem.

[pure flicker]

You updated your template with a new tag? Like this?

[atomic schooner]

I would create a new enpoint right now because my app is currently in development. However, if the same issue occurs on production, It would be problem 😄 Because of that I'm trying to see if there is missing

[hasty storm]

why is the extra workers still initializing anyways

[atomic schooner]

I don't know..

[hasty storm]

well check the logs

[hasty storm]

well, noone has been experiencing this asfar as i know

[hasty storm]

exept because of this

[atomic schooner]

It says "Waiting for logs" for the extra worker

[pure flicker]

Have you tested your image locally?

[atomic schooner]

Nope, I don't have GPU environment on my local 😦, but there is no dramatical change between 1.2.4 and 1.2.5

[hasty storm]

hmm

yeah worth reporting this to runpod i guess

try using the contact

[atomic schooner]

I reported thank you. I will update here if there is a problem

[atomic schooner]

Support suggested me to edit endpoint to set worker count to 0, then increase again. This way fixed the issue. Thank you 🙏

[dapper gulch]

You should not have to do that though

[atomic schooner]

Yes, I think an issue occurred, but I just got rid of the problem

[hasty storm]

same way of deleting and recreating then

[dapper gulch]

Thats a pain, easier just to set max workers to zero and back up again

[hasty storm]

not really for me if once, but not my point there...

[dapper gulch]

What is your point? Its stupid to waste unneccessary time recreating endpoint, getting a different endpoint id, having to change your code that calls the endpoint to use the new endpoint id etc etc, its a stupid point to be honest

[hasty storm]

what point?

[atomic schooner]

Both will create an unavailability anyway 😄 So I think runpod is aware of the problem since they suggested this to reset like this

[fallen pike]

I had this issue occasionally. Just change the maximum or minimum number of worker or any param in the endpoint. It will trigger a new deployment

[hasty storm]

Like from 3 to 5 works?