#AFF A800 showing high latency and ODX issues

1 messages · Page 1 of 1 (latest)

warped tide
#

Hello folks ,

I had a strange finding today ; an A800 system ontap 9.12p2 is showing strange behavior related to latency in general ; we did an ontap upgrade to 9.13p1 but the upgrade did not fix any of the issues we have seen .

The environment is as follows :

Hosts are hyper-v hosts windows 2019 build1809 HBA drivers Qlogic 9.4.3.21 .
FC Switches MDS 9396t 8.4.2b ; ports are 32gb SW ports all operational . There are 2 MDS’s one per Fabric ( zoning is done one initiator ports to all targets of the FC SVM ) .

MDS BB Credits are showing :
transmit bb credit 64
Received bb credit 32

The system has 18x 7.68 NVME drives FW NA50

ODX is not working at all and when it is enabled the latency on the netapp system on a normal copy or running a iometer can be from 20-50 or even 150 ms ( that is extremely high ) .
When odx is disabled ( via powershell ) latency is a bit better but still too high 2-7 even 8ms with 15ms spikes .

We have done a lot of LUN tests on single host hyper-v and on an esx host with direct RDM LUN .

Some tests show this :
For a workload iometer ; 256K Blocks 50%read sequential we have the following latency :

A800::*> qos statistics volume latency show -volume test_vol2 -vserver FC2_SVM
Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA AVSCAN

-total- - 16.36ms 516.00us 0ms 15.81ms 0ms 0ms 0ms 31.00us 0ms 0ms 0ms 0ms 0ms

Other test using normal file copy ( windows file copy ) on the LUN using 50gb file induces more than 2ms on the system :

There are no error in the log ; the MDS switch does not show out of order packages resent or anything strange / discards etc .

#

The Client opened 2 support tickets one for ODX not working at all either on FC or CIFS and one for High latency .

prisma dawn
#

What are the case numbers @warped tide ?

warped tide
#

there were 2 issues back there one is deemed to be as designed since Single LUN performance on netapp is a bit better than before 9.9.1 but not great in terms of latency

#

the 2nd is ODX on block and there are problems

#

2009681139 – ODX issue for SAN – opened in 02 Aug
2009696751 – Latency issue – opened 11 Aug

prisma dawn
#

Ok.

#

Copy offload is really to help undersized host/network systems to obtain better performance.

#

Is it required? It may be better just to turn it off...

#

What's odd is looking at the 9/4 archives the higher latency is for "other" ops. Also the op size is 256k in the good test vs 4k in the bad test looking at av_testlun_16tb.

warped tide
#

yes it is required unfortunatelly

prisma dawn
#

Any idea why the op size is different from the client end?

#

I'm just taking total throughput in bytes/second divided by ops/second for reads and writes.

warped tide
#

i dont know why the op size is different

#

what i tested in client environment was on a new FC SVM where new zoning was done and i saw pour performance when ODX is enabled ; not only latency but speeds like 100-200mbps sometimes even lower

#

on CIFS tests show way better speeds ;

#

latency on a ODX copy is arround 2 to 6ms

#

i am trying to replicate the issues in my A250 in the lab

#

but i dont have FC yet i still have to get demo MDS's in working condifition to do some direct tests but what i have seen on ISCI luns is that ODX does not work ( since i have network traffic )

#

another strange thing is that on client A800 on cifs when ODX works the os shows copy speed 1.5gbps

#

on my demo a250 i get 2.3gbps when using a cifs share

#

most likely the limitation on their end is the QOS profile that was enabled on the volumes othwerize this cant be explained

prisma dawn
#

Not seeing QoS latency.

#

iSCSI def supports copy offload.

#

That KB itself isn't interesting but just pointing out it specifically is for iSCSI.

#

I've actually done a lab test before with FAS25xx and CIFS and iSCSI and had copy offload work (I can tell because copy throughput was faster than the 1 GbE link for one thing) between protocols.

#

I was surprised by that

valid mortar
#

do you have secondary compression enabled on the volume(s)? at least in the past, using secondary compression disabled ODX (not sure if that's still the case)

warped tide
#

good point to check

#

on lab a250 i know i dont have it but on client side i dont know will have to check

#

Source volumes must be a minimum of 1.25 GB.

Deduplication must be enabled on volumes used with copy offload.

If you use compressed volumes, the compression type must be adaptive and only compression group size 8K is supported.

Secondary compression type is not supported.
#

the above are some of the requirements

#

i have everything on default

#

Aggregate: AFFA250_01_NVME_SSD_1
Node: AFFA250-01

Has Cross Volume Deduplication Savings: true
Cross Volume Background Deduplication: true
Cross Volume Inline Deduplication: true
Has Auto Adaptive Compression Savings true

#

AFFA250::> vol efficiency show -vserver ISCSI_SVM
Vserver Volume State Status Progress Policy

ISCSI_SVM iscsi_ODX_LUN Enabled Idle Idle for 124:30:35 auto
ISCSI_SVM testLUN Enabled Idle Idle for 2362:46:14 auto
ISCSI_SVM testlun2 Enabled Idle Idle for 682:17:52 auto
ISCSI_SVM w2019_test_lun Enabled Idle Idle for 35:30:35 auto
ISCSI_SVM w2019_test_lun_2 Enabled Idle Idle for 35:30:22 auto
ISCSI_SVM w2019_test_lun_3 Enabled Idle Idle for 35:30:22 auto
ISCSI_SVM w2019_test_lun_4 Enabled Idle Idle for 35:30:18 auto
ISCSI_SVM w2019_test_lun_spanned Idle Idle for 35:30:09 auto
Enabled
8 entries were displayed.

#

everything is default

#

[-storage-efficiency-mode {default|efficient}] - Storage Efficiency Mode
Displays information only for those volumes that match the specified storage efficiency mode. The default mode sets 8k adaptive compression on
the volume. The efficient mode sets auto adaptive compression on the volume.

#

maybe this is the thing

#

the ISCSI vols i have are having the policy set with efficient mode

#

the default mode is using 8k that is a requirement for ODX ( that is stated in the documentation )