We have a A220 that is pretty filled up (80-90% full aggregates). What would be the impact of enabling the higher storage efficiency on existing volumes? What is the typical gains of this? Does it have a negative effect on the snapshots? (we have both snapshot snaplock and snapmirror enabled on the volumes) and finally, I guess you would need to start a "sis" task in order for it to run through the volumes and find any additional efficiency?
#Effects of enabling higher storage efficiency
hazy sun
gentle scroll
normally AFF systems have "efficiency" turned on by default, afaik... have you checked?
running "sis" later is going to balloon your snapshots if there are major savings, so you won't see any savings until the snapshots expire... it will also create larger deltas for snapmirror, etc
hazy sun
This is not the "normal" inline dedupe etc.. that is enabled by default. This is a new option (from ONTA 9.10.1->) which you can add on your volumes.. and this system initially ran 9.8 and we have not enabled this as we upgraded past 9.10.. 🙂
gentle scroll
i have no clue what that is... i'm guess some sort of compression... probably not going to get you much unless you have lots of easily compressed files
wheat lotus
Beginning with ONTAP 9.10.1, you can set the storage efficiency mode when creating a new AFA volume. Using the parameter -storage-efficiency-mode, you can specify whether the volume uses either the efficient mode or the default performance mode. The -storage-efficiency-mode parameter is not supported on non-AFA volumes or on data protection volumes.
some generic commands
set adv
rows 0
volume efficiency inactive-data-compression show
efficiency modify -vserver * -volume * -inline-compression false -compression false
efficiency undo -vserver * -volume * -compression true
vol efficiency undo -vserver * -volume *
volume efficiency modify -vserver * -volume * -compression-type adaptive -compression true
volume efficiency modify -vserver * -volume * -storage-efficiency-mode efficient
to list the state you are in use :
vol efficiency show -fields compression,compression-type ,inline-dedupe,inline-compression,cross-volume-background-dedupe,cross-volume-inline-dedupe,storage-efficiency-mode
half plume
The efficient mode uses Temperature Sensitive Storage Efficiency (TSSE) which uses smaller compression block sizes for hot data and larger compression block sizes for cold data. For the cold data it's using the ZSTD algorithm.
The end result should be better storage efficiency savings while minimizing the perf impact on compressed hot data. But: If you need to actually read from cold data, the impact might be higher than using the "default" efficiency mode.
hazy sun
But if we enable it on a volume with existing data, do you have to run a "scan" command? And does it result in a large snapshot with the data the is potentially compressed differently? Our volumes have a vast ammount of "cold" data.... a lot of powered off VMs that are only started up as a project is needed, some of them not for several months... sadly this is too small to add a StorageGRID setup and migrate off the cold blocks...
wheat lotus
you dont need a storage grid actually for small capacities , any small FAS cand do it FAS2720 or 2820 are good enough with ontaps3
cerulean kernel
you dont need a storage grid actually for small capacities , any small FAS cand ...
this. We have dozens of customers doing exactly that. Just be sure to do proper sizing right at the beginning because the Flexgroup for the S3 bucket has some limitations when scaling (mainly, no rebalance)
wheat lotus
from 9.12 there is manual relbalance on flexgroup vols
there are several tech FAQ's on partnerhub
but by no means this replaces a storage grid ; one good ex is that there is no ILM in ontaps3
even so it is a good solution in case of tiering snapshots not actually the data in active volumes .
cerulean kernel
https://docs.netapp.com/us-en/ontap/flexgroup/manage-flexgroup-rebalance-task.ht...
that doesn't work for S3 bucket flexgroups
since there are no "files" that you could move (the S3 buckets are so well hidden that you can't even see any files or objects on the systemshell, at least not with ls)
half plume
but by no means this replaces a storage grid ; one good ex is that there is no I...
The only SG ILM policies really supported for your FabricPool buckets are replication & erasure-coding. With ONTAP S3 you could solve that simply by using FabricPool mirrors.
dreamy karma
from 9.12 there is manual relbalance on flexgroup vols
I'm leary of recommending rebalance.
For TSSE, it doesn't mean it uses FabricPool to handle the cold data, it just means it is compressed with ZSTD and larger chunks. The downside is reading those may take longer than 1 ms and have higher latency values (not dramatic, but can be noticable when 1 ms is barely acceptable for an application).
It might be a good option.
If you have a lot of stale data and a FAS lying around you could use FabricPool too. Or you could vol move.
last ridge
Resurrecting this old thread, because I have a related question...
Other than explicitly turning on storage-efficiency-mode, what causes this to be flipped from "default" to "efficient"?
The documentation seems to indicate that it is set to "efficient" for any new volumes created on an AFF, but in my testing, that is not the case. New volumes I create are set to "default". But, we have MANY volumes that are set to "efficient", and I have no idea how or why it was enabled. Anyone have any idea? I am thinking that maybe it was set to "efficient" by default on some versions of OnTap, but then it went back to defaulting to "default" on newer versions of OnTap? We are now on 9.13.1P10. Or, maybe it depends on the AFF model is was created on? We have a mixture of AFF-300, AFF-700s, and AFF-400.
From the OnTap documentation:
https://docs.netapp.com/us-en/ontap/volumes/enable-temperature-sensitive-efficiency-concept.html
"The two modes provide a choice between file compression (default), which is the default mode when creating new AFF volumes, or temperature-sensitive storage efficiency (efficient), which enables temperature-sensitive storage efficiency. With ONTAP 9.10.1, temperature-sensitive storage efficiency must be explicitly set to enable auto adaptive compression."
half plume
I think it was only from 9.8 to 9.9.1 that TSSE was enabled by default for any newly volumes (on AFF A-series). Then NetApp noticed "oh damn, for peak IOPS scenarios this leads to higher latency while reading old data which customer might not like". Usually an ONTAP update does not degrade performance but improve it, so that was "bad".
So with 9.10.1 they introduced these two storage efficiency modes (before it was configured with another parameter I don't remember at the moment) and changed the default back. So TSSE is disabled by default again.
Only for C-Series AFF it's enabled by default (and you can't even change that).
And all the newer models (except A20) don't use TSSE anymore. They use larger compression groups for all the data since they can utilize the QAT offloading so there's no performance impact anymore.
If you don't notice any performance impact you could leave it at "efficient" for these vols (or even enable it for other volumes). I mean it helps with compression savings.
Otherwise simply change it back. But you might need to scan old data so that all these cold blocks are recompressed with the faster algorithm.
last ridge
Thanks, @half plume
I just hate that we have some volumes with "default" and some with "efficient". I try to keep to standards, but stuff like this makes it hard. And, we are suspecting that it might be causing some issues setting efficiencies (using Ansible) when we rehost a volume. The efficiency policy ends up being blank, and the only way you can set it is to disable the efficiencies, set it, and then re-enable them. Still trying to re-create that issue though.
dire bane
It all depends on what your data is and how you have it laid out. Here are the items that can affect your space savings without shuttling the data off box:
• Deduplication
• Compression
• Compaction
• Encryption
Since you have an AFF, these can run inline, in that these operations are run in memory before the data is laid down to disk and any write performance penalty is paid immediately. Or they can run post-process via on schedules, where the data is laid down without these features using the full space and full disk speed, then at a time of your choosing, the data is then processed and you reclaim the savings. Typically, post-process is run outside of business hours while they system is at low usage capacity. Post-process is resource intensive because you have to read the data, process it, then write it back, however each method has its benefits and detriments depending on usage and requirements of the system. As a rule of thumb on AFF systems, inline is more efficient than post-process.
A brief primer on what each does:
Deduplication – Each data block (4KB in size) is hashed and the hashes are stored in a hash table. If a new block comes in to the system and needs to be written to disk (inline deduplication), or if a sis operation is running on blocks that were laid down raw (post-process), and a blocks hash matches another hash in the table, that block does not need to be stored again as it is already on disk, and a pointer reference (much smaller than a block) is used to reference the preexisting block rather than writing anther copy of the same block, saving that space.
Compression – Block are grouped up in to compression groups. Adaptive compression groups are 8KB and secondary groups are 32KB. Adaptive is used when random read and high performance is required. Secondary is used when data is sequential written and higher compression savings are required. AFF and flash pools use adaptive by default and HDD uses secondary by default.
Compaction – Since ONTAP writes out all blocks in 4KB sizes, what if you had to store four 1KB files? Well, ONTAP would write each 1KB file in to a 4KB block, wasting 3KB of padded space for each block, so you would use 4x4KB (16KB) of raw storage to write 4x1KB (4KB) of data, wasting 12KB in total. Effectively this is 25% efficient in that you used 16KB to save 4KB of data. Compaction to the rescue, if you have a ton of small files less that 4KB, as compaction would take those 4x1KB files and store them in one 4KB block, thus you would be at 100% efficiency for those four 1KB files. Compaction is an inline only feature and occurs after deduplication and compression. A volume must be thin provision for compaction to be enabled.
Encryption – Encryption can occur on each disk element, via its processor, and this is a Self-Encrypting Disk (SED) and when ONTAP is in charge of the key management of these elements it is referred to as NetApp Storage Encryption (NSE). NSE is transparent to WAFL’s other storage efficiency mechanisms and does not have effects on them. SED is more efficient as ONTAP does not have to use its resources encrypting or decrypting data.
The next option is NetApp Volume Encryption (NVE), where WALF is in charge of encrypting the blocks, typically hardware accelerated via processor extensions (however still a performance impact), however each volume has its own encryption key, thus the same block of data stored to two different volumes, let’s say all zeros, would not have the same output of raw data as the keys are different, resulting in two different ciphertexts for the same input of data, thus WALF would have two store two different blocks. One cannot deduplicated data at an aggregate level when NVE is used. NVE could cause storage inefficacies if you had a large number of volumes each containing similar data, as aggregate deduplication of blocks could not occur.
The last option is NetApp Aggregate Encryption (NAE). NEA gets around the deduplication of data at the aggregate level problem by encrypting all volumes with the same aggregate key. Since all volumes use the same key, in our scenario, if the same block of data needed to be stored on two different volumes, let’s say all zeros again, they would have the same output of raw data as the keys are the same, resulting in the same ciphertext for the same input of data, thus only one raw block would need to be stored on the aggregate.
When talking about depending on what your data is, if it is already compressed data that is static, like media files, it is not going to benefit from Deduplication, Compression, or compaction. If your data are VM’s, that each have the same OS, and you have 100 of them, you are going to benefit greatly.
In a nutshell:
I would make sure that if you are using NVE, that you migrate to NAE if you think you have the same/similar data stored on multiple volumes.
I would make sure that my deduplication is running at the aggregate level. I would make sure deduplication is running inline, and post-process duplication was scheduled for off hours, as inline will pause operations under heavy load.
I would run a deduplication post-process over the weekend (off hours for me) where I would scan old data.
I would volume move data between aggregates to try to put similar data that would benefit from deduplication on the same aggregate, as long as it would not affect the balancing or performance of the system. Effectively each node has its own data aggregate(s) and the performance of that aggregate is tied to the node that owns it; what I am saying is you don’t want all of your data on one node, you want to split it for performance reasons, but you also have to understand that deduplication only works at the aggregate level at most, not the cluster level.
Lastly, if none of these efficiency settings bear fruit, you will need more storage elements, and if you integrate them in to an existing aggregate, you need to rebalance existing data across all raid groups. There are also options to shuttle cold data off to other systems.