#Seeing high latency on offline volumes.
45 messages · Page 1 of 1 (latest)
Are there any iops reported on these volumes? read/write/other?
What about System Manager?
Okay. Could you share answer to below questions?
- What is your Harvest version?
- What is the current latency value you're observing for this volume?
- What is the latency reported in System Manager for this volume?
- What is the volume_total_ops metric value from your Prometheus/VictoriaMetrics query? (Example below)
Thanks. Could you share what is your Harvest version?
NAbox v4.2.0 (f2056a6)
Harvest 26.05.0
Grafana 12.4.2
Victoria Metrics 2.24.0
Could you share volume_total_ops metric graph as well for this volume?
Please also upload NABox support bundle here
https://upload.nabox.org/saqu-diyu-dywo
Could you also check in System Manager?
uploaded the logs
Thanks. Please share below as well.
- The volume_total_ops panel graph from the Volume Grafana dashboard for this volume, along with the corresponding/equivalent latency panel graph.
- The System Manager–reported values for the same metrics for this volume.
How can i get volume_total_ops panel
You have these IOPS panels in volume dashboard as below.
@analog shale Additionally, could you enable debug logging for the relevant poller affected by this latency issue in NABox? The steps are documented here https://github.com/NetApp/harvest/discussions/4330
After restarting, once the issue occurs, you can upload the latest NABox support bundle here
https://upload.nabox.org/saqu-diyu-dywo
@analog shale
After enabling the debug logs and restarting, If the issue occurred, you can upload the latest support bundle here at same location: https://upload.nabox.org/saqu-diyu-dywo
Thanks for the log files @analog shale they are very helpful in showing the issue. There are five volumes were we see spkes. Each of these volumes has a small number of iops but a large latency delta.
Following these instructions, https://github.com/NetApp/harvest/discussions/4325 can you set latency_io_reqd in your KeyPerf and Restperf collectors to a value of 10 and restart? Let us know if that solves the high latencies you're seeing
As mentioned in #3962, Below are the steps to modify this in Harvest and NABox4. In below example, we'll change latency_io_reqd to 10. Default value is 0 because it matches with PAS and System ...
cd /etc/nabox/harvest/user/zapiperf
-su: cd: /etc/nabox/harvest/user/zapiperf: No such file or directory
can i create the directory
yes
I have applied the above fix and still seeing the same
Can you upload another support bundle to https://upload.nabox.org/saqu-diyu-dywo and we'll take a look
uploaded the logs
Thanks @analog shale .
@analog shale In NABox, this setting isn’t being applied as expected. A fix is needed here in NABox
@eager fog can help.
So, if you directly change the file in active directory, and stop/start the poller (without restarting the container), it should work
So you want me to restart the harvest or poller
just poller, in the UI, not the havrest container
I should have a fixed build for you sometime today
sending you a PM
HI @eager fog ,
i have updgrade the nabox with the version you have provided but still i am seeing the offline volumes on top 5 volume latency
What’s the configuration file in active directory now ?
Yes
Could you add latest NABox support bundle at https://upload.nabox.org/kogi-xyxu-myke ?
HI @trim moss , I have uploaded the logs
@analog shale
Thanks for the logs, Your support bundle shows Nabox version as v4.2.2-27-g5c4e3b6.
We have fix in new Nabox version and we have validated locally. It would be in version v4.2.2-32 (e6c4399), @eager fog will cover the release timeline accordingly.
@eager fog, When will the version will released
Working on it. Hoping today or tomorrow.