Viewing QTree Quota data | NetApp | Page 1

muted finch Oct 23, 2024, 9:20 PM

#

Hello,

I have setup Harvest 24.05 with no customizations out of the box. I have validated that data is being collected. However, when I look at some of the dashboards, including the ONTAP:Quota dashboard, I never can see any data. When I drill into it, I think that the fundamental issue is in the definition of the Datacenter and Cluster variables. On this non-working dashboard the Datacenter variable is defined as "label_values(volume_labels{system_type!="7mode"}, datacenter)" and the Cluster variable is defined as "label_values(volume_labels{system_type!="7mode",datacenter=~"$Datacenter"}, cluster)". On a working dashboard (for instance ONTAP:Disk) the Datacenter variable is defined as "label_values(cluster_new_status{system_type!="7mode"},datacenter)" and the Cluster variable is defined as "label_values(disk_labels{system_type!="7mode",datacenter=~"$Datacenter"},cluster)".

Note particularly that the base metric name is "volume_labels" on the non-working dashboard and either "cluster_new_status" or "disk_labels" on the working dashboard.

When I check in the metrics explorer, I cannot find the "volume_labels" metric at all. I do see metrics reporting for both "cluster_new_status" and "disk_labels" however.

I made a copy of the ONTAP:Quota dashboard and edited the values of those variables to match the working ones. Now, I DO see data in that report.

I would have expected that default installation of Harvest would have all of these things already being collected. Do I need to figure out how to programmatically edit all of these dashboards or did I miss something in my installation?

gloomy anchor Oct 24, 2024, 6:12 AM

#

@muted finch Thanks for the feedback. Yes, we should be using similar queries for the cluster and datacenter dropdowns. We'll fix that. Given that volume_labels are not present in your environment, does it mean that there are no volumes present for the clusters? Or could there be an issue while collecting volumes from the cluster? Please upload the Harvest logs here for us to check: https://upload.nabox.org/zeky-vumo-nyxu

More information on how to collect logs can be found here: https://netapp.github.io/harvest/24.08/help/log-collection/

muted finch Oct 24, 2024, 1:30 PM

#

I have confirmed that we do have volumes in our environment with our storage admin. I have uploaded the requested Harvest logs to that URL.

#

Admin:
  # Listen address for Prometheus HTTP service discovery end-point
  # The endpoint is reachable at SCHEMA://<listen>/api/v1/sd
  # Should be one of: localhost:port, :port, or ip:port
  httpsd:
    listen: :8887

    # Basic auth for - /api/v1/sd
    # auth_basic:
    #   username: admin
    #   password: admin

    # create self-signed certs with 'bin/harvest admin tls create server'
    # tls:
    #   cert_file: cert/admin-cert.pem
    #   key_file: cert/admin-key.pem

Tools:
#  grafana_api_token: 'aaa-bbb-ccc-ddd'
#  autosupport_disabled: true

Exporters:
  prometheus:
    exporter: Prometheus
    local_http_addr: 0.0.0.0
    port: 12990
  prometheus1:
    exporter: Prometheus
    port_range: 13000-14000

Defaults:
  collectors:
    - Rest
    - RestPerf
    - Zapi
    - ZapiPerf
    - Ems
  use_insecure_tls: false

Poller_files:
  - /etc/harvest/configs/*.yml

Pollers:
#  # This is an optional poller that provides stats about harvest itself.
#  # To remove it, simply delete the "unix" section
  unix:
    datacenter: local
    addr: localhost
    collectors:
      - Unix
    exporters:
      - prometheus

This is my harvest.yml file

keen beacon Oct 24, 2024, 1:34 PM

#

You can enclose your yaml in three backticks followed by yaml to have it formatted as yaml instead of Markdown. Like this

muted finch Oct 24, 2024, 1:36 PM

#

Thanks!

keen beacon Oct 24, 2024, 1:46 PM

#

The logs have rolled and do not include the ONTAP version. What version of ONTAP are you monitoring? We see that the Rest:Volume collector is working

{"level":"info","Poller":"ontap-aa","collector":"Rest:Volume","metrics":1670,"instances":35,"zBegin":1729758220545,"pollMs":884,"apiMs":726,"parseMs":505,"calcMs":0,"pluginMs":155,"bytesRx":86183,"numCalls":5,"exportMs":0,"instancesExported":1,"metricsExported":1,"bytesRx":86183,"numCalls":5,"caller":"collector/collector.go:593","time":"2024-10-24T08:23:41Z","message":"Collected"}

#

That log line is odd since it says that Harvest collected 35 volumes from ONTAP, but only exported one. I'm going to share some curls that we'd like you to run so that we can replay your response to debug

muted finch Oct 24, 2024, 1:59 PM

#

We are running ONTAP version 9.15.1P1

keen beacon Oct 24, 2024, 2:58 PM

#

Can you give these instructions a try? https://github.com/NetApp/harvest/discussions/3231

muted finch Oct 24, 2024, 3:36 PM

#

Ok, got those curl command outputs uploaded to the same nabox link as before

#

named volumes.gz

keen beacon Oct 24, 2024, 3:37 PM

#

thanks!

muted finch Oct 24, 2024, 6:33 PM

#

What are you looking for in that data? Just curious.

keen beacon Oct 24, 2024, 6:40 PM

#

we're replaying your json through harvest running locally - so instead of talking to a real cluster, we replay the json responses from your cluster to try to recreate the problem

muted finch Oct 24, 2024, 6:44 PM

#

ah, makes sense! Thanks for indulging my curiousity!

keen beacon Oct 24, 2024, 6:49 PM

#

no worries! curiosity is encouraged - ask away.

We've fixed the dashboard issues you raised an added a unit test to ensure that problem does not happen again. https://github.com/NetApp/harvest/pull/3227
We're still digging into why your freshly installed Harvest is not collecting volumes

muted finch Oct 24, 2024, 6:53 PM

#

Very cool! Thanks for doing that!

#

Will that come out in the next release, do you think?

keen beacon Oct 24, 2024, 6:55 PM

#

yes, it is already out in https://github.com/NetApp/harvest/releases/tag/nightly and will also be included in the 24.11 release

muted finch Oct 24, 2024, 6:55 PM

#

Oh, sweet! Thanks for that!

keen beacon Oct 24, 2024, 6:55 PM

#

your welcome

#

can you paste the output of running?
bin/harvest version

muted finch Oct 24, 2024, 8:13 PM

#

bin/harvest version harvest version 24.05.2-1 (commit bae3aad2) (build date 2024-06-13T07:58:11-0400) linux/amd64 checking GitHub for latest... newer version v24.08.0 available at https://github.com/NetApp/harvest/releases/latest

#

Sorry didn't see your message until just now

gloomy anchor Oct 25, 2024, 12:09 PM

#

Thanks @muted finch . We are unable to reproduce this issue in our local environment. To gather more details, I will provide you with a debug build shortly.

gloomy anchor Oct 25, 2024, 12:55 PM

#

Could you please update Harvest to the nightly version from https://github.com/NetApp/harvest/releases/tag/nightly? After updating, navigate to the Harvest installation directory and run the following command. Let it run for a minute, and then share poller_output.log.

Note: Please replace POLLERNAME with the appropriate value from your harvest.yml configuration file.

ENABLE_VOLUME_LOGGING=1 ./bin/poller --poller POLLERNAME --collectors Rest --objects Volume --promPort 12991 2>&1 | tee poller_output.log

muted finch Oct 25, 2024, 9:14 PM

#

Sorry this is taking so long. I'm going to have to install a parallel copy of this and that will take a bit to get done. I'll report back, likely next week, once I have that in place. My plan would be to create a parallel installation using that nightly version, stop my existing Harvest installation, run this briefly to capture the log you want and then start my existing installation again.

gloomy anchor Nov 6, 2024, 7:58 AM

#

@muted finch Have you got a chance to try steps shared earlier? Thanks.

muted finch Nov 8, 2024, 2:28 PM

#

Sorry, got busy with other things. I have run this and have the log file available now. How do you want me to get it to you?

gloomy anchor Nov 8, 2024, 2:29 PM

#

Please upload here https://upload.nabox.org/zeky-vumo-nyxu

muted finch Nov 8, 2024, 2:31 PM

#

Ok, that link says that the share has expired

gloomy anchor Nov 8, 2024, 2:32 PM

#

Okay. Please upload here https://upload.nabox.org/xumo-soyi-rysu

muted finch Nov 8, 2024, 2:32 PM

#

Ok, done

#

And thanks for checking back with me and looking into this!

#

Let me know if you need anything else for troubleshooting. I will make sure to prioritize it more.

gloomy anchor Nov 8, 2024, 4:04 PM

#

Thank you for providing the logs. From the logs, I can see that the volumes are being exported correctly. Could you please upgrade to the this nightly build or use the latest release (v24.11) available at the following link: NetApp Harvest v24.11.0? After upgrading, please check if you are receiving the volume data.

time=2024-11-08T14:26:25.030Z level=INFO source=collector.go:598 msg=Collected Poller=xxxx collector=Rest:Volume apiMs=454 bytesRx=92072 calcMs=0 exportMs=1 instances=35 instancesExported=34 metrics=1686 metricsExported=1424 numCalls=6 parseMs=287 pluginInstances=0 pluginMs=246 pollMs=704 zBegin=1731075984325

#Viewing QTree Quota data