#Observed behavior in HA NAS with db2 partitioned data

1 messages · Page 1 of 1 (latest)

west tulip
#

We've been hunting performance problems in some DB2 database applications and have fallen into this observation. First...
Lenovo DM5000h, active-active controllers, each with bonded 10gbE to our core switches for NAS load. Controller "A" has one IP address, B another.
OnTap 9.9.1, ESX cluster 7u3, each cluster member has an active/active pair of 10gbE links. Unfortunately, we're using NFS3 so no real multipath... Instead, we're using "poor man's multipathing" where we're mounting storage volumes from the filers in an A,B pattern. (That is, the first of 50+ volumes is mounted from controller A IP, the second from controller B, the third from A, etc.)

Now, the observation....
One of our db2 virtual servers has a tablespace that's partitioned into thirds... 1/3 of the data is on controller B (aggregate B,) one third on controller/aggr A, and the third also on controller/aggr B. This is actual partitioning, where a single table or tablespace is split into thirds in order to spread the I/O load across storage volumes.
We've been referring to this as "B,A,B storage"
The volumes were mounted from the IP addresses closest to the aggregates... volume 1 mounted from IP B, volume 2 from IP A, and volume 3 from IP B, hence "B,A,B network pathing."

By experiment, we discovered that our I/O speed is over 2.25 times faster if we break the "B,A,B" network pathing and mount all three volumes from controller B. ("B,B,B network pathing.")
I believe we're seeing the benefit of having all three volumes sent to the client from a single controller and its deduplication/re-constitution engine.
Is this a valid observation? Is there something I'm overlooking that explains a 2.27x performance increase?

#

As for monitoring, we're using NAbox 3.3 and Harvest 23.08.0-1, and also have Unified Manager. Harvest/Grafana has been a godsend for diagnosing these issues.1000

nimble trout
#

just chiming in from my personal experience.
You can/will see slower performance from one of the NetApp controllers when using both on a single host system.
Our setup was with MS SQL, not DB2, but the same situation.
We had our Windows servers configured for active/active with the MPIO drivers and attached to both controllers using optimized paths to their attached LUN.
However, we still had slower performance from one controller no matter what we did.
In the end we moved everything assigned to a host onto the same controller as we found no other solution

west tulip
#

Thanks! This one is so deep it's confusing to me... We're seeing exactly the opposite behavior!
With two aggregates, one on each controller, we would expect higher performance to use the shortest path to the data. It turns out that addressing volumes on aggregate B A from the IP address of controller A B is much, much higher performing than "A,B" "B,A" addressing.
My feel is that it's the combination of NFS file sharing, partitioned db2 tablespaces, and deduplication but I really don't have whtepaper/testing (beyond my own efforts) that I can point to for confirmation.

waxen apex
#

What tests are you running within DB2? Are you disabling buffer pool cache from the tablespace before running these tests?

west tulip
#

Good question @waxen apex - I have a reasonable grasp of SQL servers and a little performance tuning experience but virtually no DB2 experience.
Our "benchmark" for the impact of storage and pathing changes has been an automated db2backup job, which dumps to disk. (Tiered backups are then effected with disk-to-disk-to-tape with commercial backup software, with much redundancy... inter-site replication, physical tape in three locations, off-sited to fireproof... We've got backup wired tight.)
This db2backup outputs to a dedicated storage volume, also NFS3 mounted. The guest OS boot drive is on yet another volume, and db2 application binaries and supporting folders (the installation folder and everything thereunder) is on one more seperate volume.
So, in total, there's an OS VMDK in one volume, "A" path and "A" storage, the application install folder in a second volume, also "AA" path and storage, the three tablespace stripe volumes "BAB" storage and "BBB" pathing, and lastly the db2backup output folder in it's own "AA" volume.

I do have the ability to do real benchmarking with the "fio" tool, and have done some. When we had BAB storage and pathing there was a 20% discrepancy in performance. "fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1"
With BAB pathing and storage, the three tablespace stripes, 1550, 1239. 1590.
When we switched to BBB pathing and BAB storage, this leveled out nicely 1553, 1593, 1634.