quota under quota

[loud oasis]

I know that it is not possible to create another qtree in a folder under a qtree. However, I've been receiving requests from customers asking if it's possible to set quota limits for multi-level folders. Is there any other way?

[frail forge]

You could create the sub-folders as flexvols and junction them into the namespace, with the size of the FlexVol essentially being your quota.

It does add management overhead, however.

Or, and this is just a crazy idea I've done no actual testing on, is create the qtree at the root of the volume like the others and then use a symbolic link from the client-side to give the illusion of it being part of the same folder hierarchy (I did say it was a crazy idea)

[loud oasis]

Thank you!
If I form a FlexVol volume with a junction path, is it possible to trace it like a Windows folder hierarchy?

[frail forge]

@loud oasis I just fired up an ONTAP 9.13.1 lab and tested this, and it works. The first idea.

I did the following via System Manager (you can do the same via CLI):

1. Created a 10TB volume and shared it via CIFS (mount path is /DataVolume)
2. Created a Qtree in the volume and applied a 50MB quota (mount path is /DataVolume/Qtree1)
3. Created a 10TB volume and didn't mount it initially
4. Edit the Volume Mount Path and choose to mount it at /DataVolume/Qtree1/MySubFolder)

I can copy 10TB of data into \\CIFSServer\DataVolume
I can only copy up to 50MB into \\CIFSServer\DataVolume\Qtree1
I can copy 10TB of data in \\CIFSServer\DataVolume\Qtree1\MySubFolder

Hopefully this is what you are after.

Obviously being a seperate FlexVol you'll need to apply a snapshot policy and snapmirror protection as you would normally. That is the caveat, the new flexvol won't inherit those settings from the parent volume.

@loud oasis Here are some screenshots of the mount path, showing it under the Qtree, and the Windows File Explorer showing it just looks link any regular folder path (ignore the test files, that was me testing the quota limitations).

[loud oasis]

thank you!
Can I put a 1TB limit on \CIFSServer\DataVolume\Qtree1\MySubFolder?

[frail forge]

You can, sort of, by limiting the size of that FlexVol that you have mounted at \\CIFSServer\DataVolume\Qtree1\MySubFolder. The FlexVol is the quota.

\\CIFSServer = ONTAP SVM CIFS Server
\DataVolume = Your main FlexVol/FlexGroup with your Qtrees in it
\Qtree = Your Qtree, with your tree quota
\MySubFolder = A 1TB FlexVol you've mounted at this path, so the size of the FlexVol is essentially the quota. The user can't write more than a 1TB (or maybe a little more, depending on storage efficiencies you get). You'd have to do a little maths to work out the optimal FlexVol size to cater for your snapshot reserve.

You can't create another Qtree under the first one, so mounting in a volume is the way I'd do it.

[loud oasis]

Thanks so much Chris! 😆 I'll try it myself!

[river prawn]

Man this is painful to watch. This is going to be a nightmare to manage at scale

[frail forge]

No different to managing hundreds or thousands of flexvols. Which customers do daily.

Also, you get the benefit of being able to use different policies for performance, data protection, and also for fabricpool.

I have a customer with close to 10PiB of data in a single FlexGroup, and over 20PiB total on that single cluster. Several thousand Qtrees. Its very easy to manage, absolutely. Just create a qtree, don't have to worry about snapshots or snapmirror, it's already taken care of.

But, how do manage different performance requirements, and tiering and protection policies for datasets that require something different? But you need to maintain the same namespace for your users? You can't achieve that with capabilities like FlexGroups or Qtrees, that's not what they are designed for.

Not every customer is the same, nor are their requirements. This is something I truly love about ONTAP; you can make it work to fit any requirements. This is what a lot of our design efforts go into, to understand what the customer really needs and how we can get them there. You're not forced into one way or the highway.

[loud oasis]

The verification has been completed and was successful!
　　B (20MB)
　┌ーー┻ーー┐
C(100MB) D(200MB)
I created a folder structure like this and confirmed that I could not save more than the default size for each folder.
The way to do it is to create FlexVol volumes of each size and specify the folder path with the junction path.
If I have to say, I'm a little concerned about it, it's that the folder icon has an arrow mark that looks like a shortcut, as you can see.

[fringe zenith]

The "folder" icons have an arrow like shortcuts, but this is to indicate a reparse point or junction so the client understands that it is a separate file system, with its own attributes like capacity/size.

[kindred belfry]

@frail forge I did this with a huge grid computing environment. I made a bunch of top level 1g flexvols (dev prod test as examples). I then had under each, another set of 15 flexvols representing the item we were working. Under those was a flexvol with the year. On that flexgroup we had qtrees.
At the end if every year, the year volume would marked/mounted as read only and the size was reduced to eliminate any more data.

I had a 4 node cluster (2014) with 2 billion files!

[tough prism]

Seeing this make it’s so painful , one volume gets unmounted by mistake and it will be a nightmare to search , if there are so many volumes mounted in this fashion , if my customer ask this I will deny it , better they create folders and subfolders

[frail forge]

@kindred belfry this is similar to my research customer with the 10PiB in a single FlexGroup (and thats just the data in the flexgroup itself). Under the root of the FlexGroup is a qtree for each research project around the country, and all their working data goes into this (this is the massive data points the reserachers just dump into ONTAP before they sort and analyze and decide what to do with). Under each qtree are a number of flexvols mounted in, for various purposes. When projects conclude and the data "organised" by the research group, the different flexvols are either changed to read-only, like you mentioned, or vol moved to a lower performance storage tier, or tiered with FabricPool. Sometimes all 3.

All of the data is organised and accessed and managed under a common structure, but the volumes mount in from a lot of different places.

At scale this is the best way we, and the customer, found to manage the sheer size of the data and number of files, and still maintain that flexibility of being able to individually target datasets with different protection, security and placement requirements.

The flexibility of ONTAP is a blessing and a curse. There are so many ways you can achieve an outcome. Also ... "there are so many ways you can achieve an outcome" 😄

[kindred belfry]

No kidding! We had only there to level Mount points but there were hundreds of Netapp file systems underneath in the name space

[hardy willow]

does seem a bit chaotic... would work poorly with container-based processing. I guess moving to an administrative UI where the data goes into object storage and is pulled (a copy) for processing would open up for greater inter-project analysis

but these things take time to develop

[frail forge]

You gotta remember, ONTAP works at the very small scale with a tiny FAS2820 or C150 with on a few TB usable, all the way up to the massively large scale, to 75PiB+ in a single cluster, and even up to 60PiB in a single volume (flexgroup with 200 Constituents at 300TB each). You don't have to do things one certain way, you can do what works for you, your business and your customers. You're not tied in to one way that you must present the data. Do what works for you.

For containerisation, something like Astra Trident can manage the storage for you.

[hardy willow]

my point was that you're going to have to do some special magic to administer 10PiB of data so that it is accessible to k8s services... if you're data is stored in S3, the automation becomes easier because locality and permissions are more easily made systematic. You can pull spesific datasets to a pv (also totally dynamic and automated and scaleable) and run analysis there. With the correct tags in S3 and some more complicated sharing api's for the associated meta-data and even result databases, projects can find data outside of their own collections for analysis. You can't beat the access granularity of S3 and pre-signed URL's

datamanagement for the storage administrator is so much easier on S3... basically the data owner does everything himself, total autonomy

[desert linden]

S3 is not intended as a PV though... containers usually expect a POSIX filesystem in their PVs and S3 can only do that with crude hacks

[hardy willow]

never said it was... it's a very well-organized repo

you would still have to pull a copy of the data down to a pv to do the "real work"

[loud oasis]

Hi Chris
Thank you for your great ideas. However, there are some things to consider, so I would like to consult with you.

• What kind of overhead is "It does add management overhead, however." that you mentioned at the beginning?
• Snapshots and AWS backups must be configured on a volume-by-volume basis, and 300 volumes must be configured and managed individually. Is it okay for the server load to back up 300 volumes at the same time?