#Redeployment of minio service is failing. New minio service setup works (tested in staging project)
85 messages · Page 1 of 1 (latest)
Please provide a link to the deployment that is failing
Could you please share the full logs?
This is your private deployment, I cant view it
yes, what can I share?
@timid socket does these help?
@timid socket can you allocate more volume storage? I am approaching near to 250GB already on this minio service
Also the serivce restart command gives a success feedback, but not able to see the reset in memory/cpu usage like it used to until a couple of days back
could it be like the running service is not able to stop to yield to the new containers? 🤔
this is a production blocker - would appreciate your quick action!
I usually do a hourly restart of the minio bucket service to bring down the memory usage and thus my cost. but due to this issue, I am not able to bring down my memory usage and it is resulting into huge cost for me!
@echo valley can you help here?
@ashen elbow ?
Tagging online employees as this is resulting in unwanted cost for us!
Yes, there are two issues. Refer to the support threads shared above
My service is facing some issue - not able to restart or redeploy either
@ashen elbow why are my two threads closed? I don't see the issues resolved right?
@desert pagoda ?
Please stop tagging team members - #🛂|readme #5
They were closed as they were duplicates.
Is your MinIO bucket currently offline?
No, it is running currently
From last 2 weeks
But using a lot of memory. I usually have a cron based restart every 1hr
Any update?
Waiting on this!
Hey, I am not sure how to proceed here, even the threads are closed on the support platform
Hey @desert pagoda can you please suggest how to proceed here - you have closed both the threads and nobody is replying to this discord thread too! I am totally blocked - Monday would result in peak traffic in production!
hey, duplicates of which issue?
how do I pinpoint if this is any issue on my end?
Hey @desert pagoda
Please stop tagging team members.
sure, how to proceed here?
Your MinIO is online from my understanding, so I don't believe the urgency is super high here.
but the cost is up and atleast extend the volume limit above 250GB
New reply sent from Central Station thread:
https://railway.com/project/d8a9cdda-ca19-44a5-814f-1ecaff088212/volume/474bd174-3c34-4c41-bcc2-05323e718473/metrics?environmentId=35b2e7d0-c9aa-46dd-a20c-2d320f146eb6
volume which need to be extended
https://railway.com/project/d8a9cdda-ca19-44a5-814f-1ecaff088212/service/9d0d3525-a0aa-4f9b-9f49-be6bb3f4cef2/metrics?environmentId=82212507-616a-4d85-a946-87879ae84c13
service which has the redeployment issue
You're seeing this because this thread has been automatically linked to the Central Station thread.
Can the volume be extended now?
New reply sent from Central Station thread:
what does this mean? marking it as in-progress?
You're seeing this because this thread has been automatically linked to the Central Station thread.
can someone help extend the 250GB volume limit?
@desert pagoda I tried two things
- Tried to create a duplicate of the Bucket service with a new volume - it works
- Tried to disconnect the existing Bucket service volume and attach it to the new Bucket Copy service - it gets stuck in the same deployment step
Creating containers
does this mean there is some issue with the volume? Can you please extend the limit of my volume?
Please stop pinging team members. Your MinIO service is online, and this thread is within its SLA.
How can I get my volume extended without your support?
We will extend the volume's space when we are able to come back to this thread.
what is the SLA for my thread?
I am on the PRO plan and it says Priority Support is available
New reply sent from Central Station thread:
Replying from the account associated with the plan - please prioritise this thread!
You're seeing this because this thread has been automatically linked to the Central Station thread.
The volume has been resized to 500GB.
Thanks. Would I now be able to initiate redeployment of the service successfully?
I would recommend doing restarts instead of redeploys, since a caveat with volumes is that no two deployments can be online at the same time, so you would experience more downtime with redeploys than you would with instant restarts.
New reply sent from Central Station thread:
I generally do restarts every 1 hour to keep the memory usage in check. But redeploy when required
You're seeing this because this thread has been automatically linked to the Central Station thread.
When is a redeploy required? A restart would restart Minio and then free the memory it used.
New reply sent from Central Station thread:
but restarts are also not happening right now. That is why I was trying redeploy
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
even now, restart is not working - I don't see the memory dropping
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
can you check?
like I mentioned earlier. I noticed that service deployments get stuck when I attach this particular volume to them.
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
what can be done here? This is costing me like atleast 5x of the actual requirement
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
are you still available? checking on this?
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
should I open another thread for this?
You're seeing this because this thread has been automatically linked to the Central Station thread.
Please avoid creating duplicate threads.
New reply sent from Central Station thread:
understood, what should be my next step to resolve the service restarts?
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
bumping this up for reply
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
bumping this - as the cost is not under control!!
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
I just noticed the service was redeployed automatically (atleast I did not initiate it) and was successful, bringing the memory usage down.
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
was there any issue identified or action taken from the Railway team?
You're seeing this because this thread has been automatically linked to the Central Station thread.
Should be fixed now
New reply sent from Central Station thread:
Thanks for the update. May I know the issue identified? Also, is there any way to waive of the extra cost incurred?
You're seeing this because this thread has been automatically linked to the Central Station thread.
It was a state issue on our end. We have already fixed this issue going forward.
New reply sent from Central Station thread:
Would like to understand why was there a delay of 3days for the resolution? What is the support SLA for my account? Is this justified to charge me for your side of mistake?
You're seeing this because this thread has been automatically linked to the Central Station thread.
Since there was no downtime reported, the SLA is 3 days from the last user response.
New reply sent from Central Station thread:
Just asking - could this has been resolved on the same day I opened thread if my thread had priority?
You're seeing this because this thread has been automatically linked to the Central Station thread.
Yes, for enterprise users, the response time is 1 hour.
We would love to give all users the same level of support, but unfortunately, that is not feasible given the volume we operate at.
New reply sent from Central Station thread:
Thanks for responding.
Would it be possible to share the issue in brief? I am not sure how the cost incurred is justified for me!
Also, I would like to use Railway Storage Buckets (as they are s3 api compatible), but looking forward for the object-versioning feature as Grist service depends on that feature!
You're seeing this because this thread has been automatically linked to the Central Station thread.
New reply sent from Central Station thread:
Meanwhile while Storage Buckets lack the object-versioning feature, how much can you extend my existing volume to?
You're seeing this because this thread has been automatically linked to the Central Station thread.
I had shared a brief on the issue above; that would be all I am able to say at this time.
At the current spend, 500GB is the ceiling of what we can offer.
New reply sent from Central Station thread:
Any idea by when can I expect the object-versioning feature in Storage Buckets?
What should be the spend if I want to extend beyond 500GB?
You're seeing this because this thread has been automatically linked to the Central Station thread.
With only 3 upvotes on that feature request post, we unfortunately would not be able to prioritize it anytime soon.
The spend that would come along with 80% use of a 500GB would likely be sufficient.