Maximum IOPS per flexvol?

[unborn crown]

A few years ago, the maximum number of IOPS per flexvol was stated to be about 100K. Is this true today? We used to peg the flexvol before we pegged the aggregate or head so creating flexgroups and adding constituents was the recommended way to get more performance. flexgroups have consequences so we don't do them unless we have to.
Has the per-flexvol
IO headroom increased, and if so, what is the current rule of thumb?

[opaque zephyr]

I don’t see how that can be a thing. More spindles equals more iops. The size of the requests impact iops.
As far as I know its always been based on number of spindles not per volume

[unborn crown]

It was a thing and had to do with how the filer internally handles threads (flashback to Kahuna bottlenecks). You can bottleneck a filer in all sorts of weird and wonderful ways...

[ebon gulch]

I don't think there is a hard and fast recommendation, least of all to a specific range of IOps/Throughput. I come from a customer background and have definitely felt the pain of trying to push too much out of a single volume but that was mainly about saturating a node in an A700. Could you share more about your statement "flexgroups have consequences" @unborn crown ?

[unborn crown]

The rule of thumb I heard was on an A700 but we're A800 now so I was wondering what has changed. I can certainly see the performance improvements over the years...

Flexgroups are certainly a lot better than they used to be but still cause operational challenges. Following best practices - aggregate multiplier of 4 on a 10-node cluster resulting in 40 constituents - I had a nice and fast fg. And then the customer wanted me to snapvault it to our 2-node DR cluster. 20 volumes per head... I don't know how many times I've stumbled across "oh, you can't do that with a flexgroup". From quota support and logical volume reporting/enforcement (now resolved), to recently vol migrate not being supported. Same as autonomous ransomware - flexvols get supported first and then flexgroups in a later release. vol migrate still doesn't support flexgroups. Neither does vol rehost.

[fickle pulsar]

Max throughput would be better to measure.

And flexgroups are getting closer and closer to parity with flexvol. 🙂

[ebon gulch]

Thanks for taking the time to elaborate, Ed, I appreciate it.

[unborn crown]

And flexgroups are getting closer and closer to parity with flexvol. 🙂
But every time you support a new feature, like ARP, you do flexvols first and then flexgroups later. So you get closer but never fully caught up because you keep adding new features (like that's a bad thing :)).

[pastel rapids]

The issue back in the day, was we limited a Flexvol to a single processor domain. So the system might show low CPU and low Aggr demand but the FlexVol would be pegged as one core on the system woul dbe maxed out for that one FlexVol. FlexGroups were the answer. However since then we have do a ton of work to multi thread work from a FlexVol across multiple CPU cores now being able to load across many cores. (9.9.1 had the biggest jump here). So was that the end of FlexGroups? No, FlexGroups can still help performance and in particular help distribute the load across multiple node/agggrs in the cluster. We are getting closer to closing all the feature gaps with FlexGroups but you are right Ed there is often a delay in the feature. I am not aware though of any sort of a ballpark for a per FlexVol IOPs number. It depends of course on the type of IO being done and of course the ONTAP version and hardware platform being used.

[unborn crown]

awesome, thanks @pastel rapids . Does an aggregate multiplier > 1 actually help much these days? I see it would before since you could get a single core going on each constituent but now if you can get multiple cores on the same constituent, that lessens the need for multiple constituents on the same aggregate. Different heads/aggregates, sure, but we're mostly one big SSD aggregate per head in our environment.

[chilly spade]

If you are talking NFS Access ops, I saw a FAS31xx running older ONTAP 7-mode hit 80k IOPS...

@unborn crown what kind of i/o profile do you mean? Usually you can read close to 3-4 GB/s per volume, more or less. It's when you start writing that total performance capacity degrades on a single FlexVol.

[unborn crown]

We were tipping over an A700 with a write load of 1GB/sec to a single flexvol.

[magic bone]

At least back when i use to do a lot of performance stuff there were definitely recommendations on volume to aggr mapping for max performance. It was to do with how threading worked for IO to the aggregate. This was back in the early 9.0 -9.3 days given the threading improvements between 8.3-9.0 i can only imagine its much less of an issue than it was then.

[magic bone]

what's the workload like? is it nice? or something gross like small block random. Would be really hard to tell why its choking without seeing logs

[opaque zephyr]

@unborn crown is that nfs or cifs?
Have you tried nconnect(nfs) or the cifs multi path?

[unborn crown]

NFS. nconnect is not yet an option for us because we still have some rhel7 clients

[chilly spade]

Sounds right. That's still about the same limit.

[pastel rapids]

Version of ONTAP is pretty critcal here as this has been a major area of focus. ONTAP 9.9.1 and higher have much better single volume performance. You are right @unborn crown, multiple Aggrs per controller is less important than it was and my understanding now is that having multiple aggrs provides very little benefit. FlexGroups still help to distribute the load accross multiple volumes and of course multiple controllers in the cluster.

[neat lodge]

I see 100K a lot on a700/a800...

[stuck rune]

Hi Keith, just curious if you know of any official documentation regarding flexvols not pegged to single core anymore?

[pastel rapids]

No, we don't document down to that level.

[chilly spade]

They haven't been pegged since 7-mode days with Kahuna. We have single affinity bottlenecks, but that's it.

[stuck rune]

Thanks, i've found some kb articles which explain this a bit.