#Redeployment of minio service is failing. New minio service setup works (tested in staging project)

85 messages · Page 1 of 1 (latest)

cosmic mauve
#

My current minio service volume usage is around 80%. Could that be a reason for the failing redeployments?

also came across the change in stance of open-source by minio. I don't think that should be an issue as I am able to setup a new minio service

desert pagoda
#

Please provide a link to the deployment that is failing

timid socket
#

Could you please share the full logs?

timid socket
#

This is your private deployment, I cant view it

cosmic mauve
#

yes, what can I share?

#

@timid socket does these help?

#

@timid socket can you allocate more volume storage? I am approaching near to 250GB already on this minio service

#

Also the serivce restart command gives a success feedback, but not able to see the reset in memory/cpu usage like it used to until a couple of days back

#

could it be like the running service is not able to stop to yield to the new containers? 🤔

#

this is a production blocker - would appreciate your quick action!

#

I usually do a hourly restart of the minio bucket service to bring down the memory usage and thus my cost. but due to this issue, I am not able to bring down my memory usage and it is resulting into huge cost for me!

#

@echo valley can you help here?

cosmic mauve
#

@ashen elbow ?
Tagging online employees as this is resulting in unwanted cost for us!

ashen elbow
#

You need the volume resized?

#

Above 250GB?

cosmic mauve
#

Yes, there are two issues. Refer to the support threads shared above

#

My service is facing some issue - not able to restart or redeploy either

cosmic mauve
#

@ashen elbow why are my two threads closed? I don't see the issues resolved right?

#

@desert pagoda ?

desert pagoda
#

Please stop tagging team members - #🛂|readme #5

They were closed as they were duplicates.

cosmic mauve
#

Sure. Will not tag further

#

Can you please help me with my issues?

desert pagoda
#

Is your MinIO bucket currently offline?

cosmic mauve
#

No, it is running currently

#

From last 2 weeks

#

But using a lot of memory. I usually have a cron based restart every 1hr

#

Any update?

cosmic mauve
#

Waiting on this!

cosmic mauve
#

Hey, I am not sure how to proceed here, even the threads are closed on the support platform

cosmic mauve
#

Hey @desert pagoda can you please suggest how to proceed here - you have closed both the threads and nobody is replying to this discord thread too! I am totally blocked - Monday would result in peak traffic in production!

cosmic mauve
cosmic mauve
#

how do I pinpoint if this is any issue on my end?

cosmic mauve
#

Hey @desert pagoda

desert pagoda
#

Please stop tagging team members.

cosmic mauve
#

sure, how to proceed here?

desert pagoda
#

Your MinIO is online from my understanding, so I don't believe the urgency is super high here.

cosmic mauve
#

but the cost is up and atleast extend the volume limit above 250GB

cosmic mauve
#

Can the volume be extended now?

desert pagoda
lavish skyBOT
cosmic mauve
#

can someone help extend the 250GB volume limit?

cosmic mauve
#

@desert pagoda I tried two things

  1. Tried to create a duplicate of the Bucket service with a new volume - it works
  2. Tried to disconnect the existing Bucket service volume and attach it to the new Bucket Copy service - it gets stuck in the same deployment step Creating containers
#

does this mean there is some issue with the volume? Can you please extend the limit of my volume?

desert pagoda
#

Please stop pinging team members. Your MinIO service is online, and this thread is within its SLA.

cosmic mauve
#

How can I get my volume extended without your support?

desert pagoda
#

We will extend the volume's space when we are able to come back to this thread.

cosmic mauve
#

what is the SLA for my thread?

#

I am on the PRO plan and it says Priority Support is available

lavish skyBOT
#

New reply sent from Central Station thread:

Replying from the account associated with the plan - please prioritise this thread!

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

The volume has been resized to 500GB.

cosmic mauve
#

Thanks. Would I now be able to initiate redeployment of the service successfully?

desert pagoda
#

I would recommend doing restarts instead of redeploys, since a caveat with volumes is that no two deployments can be online at the same time, so you would experience more downtime with redeploys than you would with instant restarts.

lavish skyBOT
#

New reply sent from Central Station thread:

I generally do restarts every 1 hour to keep the memory usage in check. But redeploy when required

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

When is a redeploy required? A restart would restart Minio and then free the memory it used.

lavish skyBOT
#

New reply sent from Central Station thread:

but restarts are also not happening right now. That is why I was trying redeploy

You're seeing this because this thread has been automatically linked to the Central Station thread.

#

New reply sent from Central Station thread:

can you check?
like I mentioned earlier. I noticed that service deployments get stuck when I attach this particular volume to them.

You're seeing this because this thread has been automatically linked to the Central Station thread.

#

New reply sent from Central Station thread:

what can be done here? This is costing me like atleast 5x of the actual requirement

You're seeing this because this thread has been automatically linked to the Central Station thread.

lavish skyBOT
lavish skyBOT
desert pagoda
#

Please avoid creating duplicate threads.

lavish skyBOT
lavish skyBOT
lavish skyBOT
#

New reply sent from Central Station thread:

I just noticed the service was redeployed automatically (atleast I did not initiate it) and was successful, bringing the memory usage down.

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

Should be fixed now

lavish skyBOT
#

New reply sent from Central Station thread:

Thanks for the update. May I know the issue identified? Also, is there any way to waive of the extra cost incurred?

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

It was a state issue on our end. We have already fixed this issue going forward.

lavish skyBOT
#

New reply sent from Central Station thread:

Would like to understand why was there a delay of 3days for the resolution? What is the support SLA for my account? Is this justified to charge me for your side of mistake?

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

Since there was no downtime reported, the SLA is 3 days from the last user response.

lavish skyBOT
#

New reply sent from Central Station thread:

Just asking - could this has been resolved on the same day I opened thread if my thread had priority?

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

Yes, for enterprise users, the response time is 1 hour.

#

We would love to give all users the same level of support, but unfortunately, that is not feasible given the volume we operate at.

lavish skyBOT
#

New reply sent from Central Station thread:

Thanks for responding.
Would it be possible to share the issue in brief? I am not sure how the cost incurred is justified for me!
Also, I would like to use Railway Storage Buckets (as they are s3 api compatible), but looking forward for the object-versioning feature as Grist service depends on that feature!

You're seeing this because this thread has been automatically linked to the Central Station thread.

#

New reply sent from Central Station thread:

Meanwhile while Storage Buckets lack the object-versioning feature, how much can you extend my existing volume to?

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

I had shared a brief on the issue above; that would be all I am able to say at this time.

At the current spend, 500GB is the ceiling of what we can offer.

lavish skyBOT
#

New reply sent from Central Station thread:

Any idea by when can I expect the object-versioning feature in Storage Buckets?
What should be the spend if I want to extend beyond 500GB?

You're seeing this because this thread has been automatically linked to the Central Station thread.

desert pagoda
#

With only 3 upvotes on that feature request post, we unfortunately would not be able to prioritize it anytime soon.

The spend that would come along with 80% use of a 500GB would likely be sufficient.