Proper way to delete huge amount of object | NetApp | Page 1

fiery field May 2, 2023, 12:03 PM

#

Hey guys!
We want to delete a huge amount of object in our StorageGRID (all in the same bucket and then, delte the bucket). As a Linux guy, my first idea was to use AWS CLI to do that. Is there a more proper way to achieve that? Maybe with ILM policies? All advice are welcome 😁

unkempt rose May 2, 2023, 11:57 PM

#

For bulk changes, using ILM policies is generally recommended. Whether that is moving data around, shifting old data to a different tier or deleting old data.

elfin gorge Jul 2, 2023, 10:25 AM

#

what version are you on for grid? we had problems with delete queue concurrency causing huge imbalance in nodes util.

fiery field Jul 25, 2023, 7:28 PM

#

@elfin gorge We were on StorageGRID version 11.6.0.9 (I'm currently on version 11.7).
To be honest, I used ILM policies to remove 130TiB (200 millions objects) and it tooks a long long time to delete everything (approx 20 days!!!).
And during that period, my GRID was overloaded 😒
I didn't see any imbalance in my nodes.

gritty edge Aug 3, 2023, 1:03 PM

#

@fiery field https://docs.netapp.com/us-en/storagegrid-117/tenant/deleting-s3-bucket-objects.html

This feature has been added in 11.7 😉

Delete objects in bucket

You can use the Tenant Manager to delete the objects in one or more buckets.

fiery field Aug 3, 2023, 1:09 PM

#

Yes, I saw it while I was reading the 11.7 release note. Unfortunately, I was on 11.6 when I deleted my bucket.
I will definitely try this method in the futur if I need to delete another "big" bucket 😇

vale bloom Aug 4, 2023, 3:06 PM

#

Confirm with Support if you can enable async deletes.

#Proper way to delete huge amount of object