#Seeing high latency on offline volumes.

45 messages · Page 1 of 1 (latest)

analog shale
#

There are some volumes which are offline and the grafana is showing high latency on the offline volumes. Is there ny reason,

trim moss
#

Are there any iops reported on these volumes? read/write/other?

#

What about System Manager?

analog shale
#

No

#

iops on the volume and the volume was offline

trim moss
#

Okay. Could you share answer to below questions?

  1. What is your Harvest version?
  2. What is the current latency value you're observing for this volume?
  3. What is the latency reported in System Manager for this volume?
  4. What is the volume_total_ops metric value from your Prometheus/VictoriaMetrics query? (Example below)
analog shale
#

from active iq

trim moss
#

Thanks. Could you share what is your Harvest version?

analog shale
#

NAbox v4.2.0 (f2056a6)

Harvest 26.05.0

Grafana 12.4.2

Victoria Metrics 2.24.0

trim moss
#

Could you share volume_total_ops metric graph as well for this volume?

analog shale
#

uploaded the logs

trim moss
#

Thanks. Please share below as well.

  1. The volume_total_ops panel graph from the Volume Grafana dashboard for this volume, along with the corresponding/equivalent latency panel graph.
  2. The System Manager–reported values for the same metrics for this volume.
analog shale
trim moss
#

You have these IOPS panels in volume dashboard as below.

trim moss
analog shale
arctic spruce
analog shale
#

i have enabled the debug and restarted the harvest

#

uploaded the logs

distant sierra
#

Thanks for the log files @analog shale they are very helpful in showing the issue. There are five volumes were we see spkes. Each of these volumes has a small number of iops but a large latency delta.

Following these instructions, https://github.com/NetApp/harvest/discussions/4325 can you set latency_io_reqd in your KeyPerf and Restperf collectors to a value of 10 and restart? Let us know if that solves the high latencies you're seeing

GitHub

As mentioned in #3962, Below are the steps to modify this in Harvest and NABox4. In below example, we'll change latency_io_reqd to 10. Default value is 0 because it matches with PAS and System ...

analog shale
#

cd /etc/nabox/harvest/user/zapiperf
-su: cd: /etc/nabox/harvest/user/zapiperf: No such file or directory

#

can i create the directory

distant sierra
#

yes

analog shale
#

I have applied the above fix and still seeing the same

distant sierra
analog shale
#

uploaded the logs

trim moss
#

Thanks @analog shale .

#

@analog shale In NABox, this setting isn’t being applied as expected. A fix is needed here in NABox
@eager fog can help.

eager fog
#

So, if you directly change the file in active directory, and stop/start the poller (without restarting the container), it should work

analog shale
#

So you want me to restart the harvest or poller

eager fog
#

just poller, in the UI, not the havrest container

#

I should have a fixed build for you sometime today

eager fog
#

sending you a PM

analog shale
#

HI @eager fog ,

i have updgrade the nabox with the version you have provided but still i am seeing the offline volumes on top 5 volume latency

eager fog
#

What’s the configuration file in active directory now ?

analog shale
#

Yes

trim moss
analog shale
#

HI @trim moss , I have uploaded the logs

arctic spruce
#

@analog shale
Thanks for the logs, Your support bundle shows Nabox version as v4.2.2-27-g5c4e3b6.

We have fix in new Nabox version and we have validated locally. It would be in version v4.2.2-32 (e6c4399), @eager fog will cover the release timeline accordingly.

analog shale
#

@eager fog, When will the version will released

eager fog
#

Working on it. Hoping today or tomorrow.