NEW: Harvest 24.02.0 has been released | NetApp | Page 1

hazy crystal Feb 21, 2024, 3:33 PM

#

https://github.com/NetApp/harvest/releases/tag/v24.02.0

📌 Highlights of this major release include:

New Datacenter dashboard which contains health, capacity, performance, storage efficiency, snapshot, power, and temperature details.
Harvest includes SnapMirror active sync EMS events with alert rules. Thanks @Nikhita-13 for reporting.
Harvest monitors FlexCache performance metrics via new FlexCache dashboard. Thanks @vagrant herald for raising.
Harvest detects HA pair down and sensor failures. These are shown in the Health dashboard. Thanks @johnwarlick for raising.
Harvest monitors MetroCluster diagnostics via MetroCluster dashboard. Thanks @wagneradrian92 for reporting.
We improved the performance of all dashboards that include topk queries. Thanks @mamoep for reporting!
We added filter support for the ZapiPerf collector. Thanks @debbrata-netapp for reporting.
A bin/harvest grafana customize command that writes the dashboards to the filesystem so other programs can manage them. Thanks @nicolai-hornung-bl for raising!
We fixed an intermittent latency spike problem that impacted all perf objects. Thanks @summertony15 and @rodenj1 for reporting this critical issue.
⭐ Several of the existing dashboards include new panels in this release:
- Node, Aggregate, SVM dashboard include volume stats panels. Thanks @BrendonA667 for raising
- SnapMirror dashboard includes automated_failover and automated_failover_duplex policies
All EMS alerts include an impact annotation. Thanks @Divya for raising
🌾 Harvest includes new templates to collect:
- NFS rewinds performance metrics. Thanks @shawnahall71 for raising
- NDMP session metrics. Thanks @schumijo for raising
📕 Documentation additions
- How to create templates that use ONTAP's private CLI
- How to validate your harvest.yml file and share a redacted copy with the Harvest team
- Harvest describes high-level concepts Thanks @norespers for raising

GitHub

Release 24.02.0 · NetApp/harvest

24.02.0 / 2024-02-21 Release
📌 Highlights of this major release include:

New Datacenter dashboard which contains node health, capacity, performance, storage efficiency, issues, snapshot, power, a...

lilac bane Feb 22, 2024, 7:30 AM

#

Hi guys, thank you for the new release, looks great 🙂 but what I see that for e.g. in the SVM dashboard I see in the SVM filter all the IP spaces

hazy crystal Feb 22, 2024, 7:32 AM

#

@lilac bane Can you check SVM filter dropdown query in SVM dashboard. It looks okay to me

#

Also, please check if you have reimported the dashboards to sync with the latest release.

lilac bane Feb 22, 2024, 7:42 AM

#

Strange it looks the same. Reset just done and also upgrade to NABox 3.3 done. I will send some printscreens to your support mail adress

hazy crystal Feb 22, 2024, 7:43 AM

#

Sure Thanks. Please share your svm dashboard json as well.

#

@lilac bane Received your email. Can you run ONTAP CLI command vserver show -fields vserver and check if name matches in SVM filter.

lilac bane Feb 22, 2024, 7:45 AM

#

with the CLI command I don't see the IP spaces only the regular SVMs

hazy crystal Feb 22, 2024, 7:47 AM

#

Yes, the intention for the SVM dashboard is to only show regular SVMs in the SVM filter. Perhaps I misunderstood your question?

lilac bane Feb 22, 2024, 8:00 AM

#

I see the difference. The query of the SVM variable in the SVM dashboard is different from the Volume dashboard. When you take the query from volume it shows only SVMs and no IP spaces

hazy crystal Feb 22, 2024, 8:09 AM

#

The Volume dashboard displays SVMs from volume_labels. Therefore, only SVMs with a volume will be visible in the dropdown. Our intention is to show only SVMs. It's possible that you may see a smaller number of SVMs in the Volume dashboard due to the reasons mentioned above. We do not show IP spaces for any of svm dropdowns. They should either be the same as or a subset of the SVMs shown via the vserver show -fields vserver

lilac bane Feb 22, 2024, 8:13 AM

#

but why I see in the SVM drop down IP Spaces? Could it be because ipspace show has a vserver list?

hazy crystal Feb 22, 2024, 8:16 AM

#

I think that IPSpace names are similar to vserver names. If you see extra names to what is shown in vserver show command then that would be an issue.

lilac bane Feb 22, 2024, 8:57 AM

#

so what I see is that with the vserver show -fields vserver command I only see SVMs which is correct, but the query shows a list like I would use ipspace show -fields vservers

hazy crystal Feb 22, 2024, 9:10 AM

#

Yes, because ipspace names are similiar to vservers/svms

lilac bane Feb 22, 2024, 9:23 AM

#

than it is not a issue directly more works as design

hazy crystal Feb 22, 2024, 9:28 AM

#

Yes that is correct

lilac bane Feb 22, 2024, 9:46 AM

#

Ok then I will live with it 🙂

hazy crystal Feb 22, 2024, 9:52 AM

#

Please let us know if you are seeking any changes or additional features in this area.

mystic charm Feb 22, 2024, 1:11 PM

#

We have just setup a new install of 3.4 and Harvest 24.03.. no matter what we try to add a storage system we get the "An Error Occurred" message.. we have also tried to add the system manually, but when we try to start it we get the "An Error Occurred"... this is an ONTAP 9.14.1 system.. we created the user as described and even tried with admin, but no dice 😦 even the "dc logs -f --tail 10 nabox-harvest2" fails with the error that there is no "logs" folder.. ### Update... "su -" did the trick.. and it turned out that we had to allow the nabox in the firewall rules on the NetApp systems...

hazy crystal Feb 22, 2024, 1:16 PM

#

@mystic charm It seems like a DNS problem. Probably try with cluster IP? Similar issues reported earlier with NABox #1176616378011226142 message

mystic charm Feb 22, 2024, 1:17 PM

#

hazy crystal <@743120101942886442> It seems like a DNS problem. Probably try with cluster IP?...

see my update.. it was the ONTAP firewall. It would be nice with a little more details than "an error occured" 😉

hazy crystal Feb 22, 2024, 1:17 PM

#

mystic charm see my update.. it was the ONTAP firewall. It would be nice with a little more ...

@hard notch for NABox changes.

hard notch Feb 22, 2024, 1:19 PM

#

I’ll see what I can do, it’s probably a timeout rejecting with this error

lilac bane Feb 23, 2024, 7:13 AM

#

Hi @hazy crystal I had time for a closer look about this. I see your intention to show all SVMs independent if they have any volume or not. But what confuses me still why I see with the SVM query ip spaces, even if they have a different name to the SVM

hazy crystal Feb 23, 2024, 7:31 AM

#

@lilac bane Let's use the SVM dashboard as an example. As we have previously discussed, the svm_labels metrics populate this SVM filter in the dashboard. The svm_labels metrics are generated from this template: https://github.com/NetApp/harvest/blob/main/conf/zapi/cdot/9.8.0/svm.yaml#L11 when using the Zapi collector. So, if you invoke this Zapi as shown below, it should match with the SVM names displayed in the dropdown for the relevant cluster.

curl --connect-timeout 30 --user USER:PASS --insecure --data-ascii '<?xml version="1.0" encoding="UTF-8"?>
<netapp xmlns="http://www.netapp.com/filer/admin" version="1.130">
  <vserver-get-iter>
    <max-records>20000000</max-records>
    <desired-attributes>
      <vserver-info>
        <vserver-name></vserver-name>
      </vserver-info>
    </desired-attributes>
  </vserver-get-iter>
</netapp>' -H "Content-Type: text/xml" 'https://URL/servlets/netapp.servlets.admin.XMLrequest_filer'

Please replace USER, PASS, and URL with the values applicable to your environment.

fallow creek Feb 23, 2024, 4:18 PM

#

hello,
do you have plan to include the section QoS Adaptative Utilized in the Workload dashboard? I thought this part will be include in this release like you did previosly with the Fixed QoS.
Thanks.

alpine wedge Feb 23, 2024, 4:24 PM

#

Hi @fallow creek That didn't make the cut for 24.02. https://github.com/NetApp/harvest/issues/2497 is the issue tracking it. If you have time to iterate on that feature with us, we should be able to get something in for 24.05

fallow creek Feb 26, 2024, 7:13 AM

#

Hi @alpine wedge
Sure, thanks for opening the issue. I think it will be easy since we have the similar case with Fixed QoS which is impleted and we were working together in achieve that information in a dashboard.

lilac bane Feb 26, 2024, 7:17 AM

#

Hi @hazy crystal thank you for the command. I can confirm that with the ZAPI request there are also IP spaces as SVM listet even if they are not visible with vserver show -fields vserver

hazy crystal Feb 26, 2024, 7:35 AM

#

lilac bane Hi <@1002092615052840980> thank you for the command. I can confirm that with th...

Thanks @lilac bane I understand the problem now. Zapi collector is publishing these IP spaces under svm_labels. As a workaround, Can you update SVM filter query in svm dashboard to below
label_values(svm_labels{cluster=~"$Cluster",type!="system"}, svm)

hazy crystal Feb 26, 2024, 8:59 AM

#

@lilac bane I have opened a issue to track this
https://github.com/NetApp/harvest/issues/2710

lilac bane Feb 26, 2024, 11:37 AM

#

Hi @hazy crystal nice thank you for that 🙂

tired hound Feb 27, 2024, 6:22 PM

#

where do I find NetApp Harvest upgrade file for NAbox?

alpine wedge Feb 27, 2024, 6:24 PM

#

https://github.com/NetApp/harvest/releases/tag/v24.02.0 bottom of the page has

#

https://nabox.org/documentation/upgrade/

#

hazy crystal Feb 28, 2024, 7:34 AM

#

lilac bane Hi <@1002092615052840980> nice thank you for that 🙂

@lilac bane Fix for issue https://github.com/NetApp/harvest/issues/2710 is available in nightly(https://github.com/NetApp/harvest/releases/tag/nightly) builds if you want to try that.

cerulean pumice Jul 11, 2024, 7:26 PM

#

hey folks, I re-enabled the ontapS3SVM template on this release, and i see those ontap s3 metrics show up again in "/metrics" endpoint, but they are all zero value. Do I miss anything else to enable s3 metrics?

#

alpine wedge Jul 11, 2024, 7:41 PM

#

Hi @cerulean pumice I doubt you missed anything. Harvest isn't doing anything special with those metrics. If you check the ONTAP CLI, you should see the same values. Most of those metrics will be >0 when traffic happens between polls. In other words, those metrics are monotonically increasing and Harvest is showing the delta between polls. Harvest is doing that because ONTAP advertises those counters as delta or averages

cerulean pumice Jul 11, 2024, 7:48 PM

#

does this mean there is no traffic happening? but I think we do have s3 server enabled and bucket created

alpine wedge Jul 11, 2024, 7:56 PM

#

right, the metrics being zero is typically because there is no traffic

cerulean pumice Jul 14, 2024, 5:53 AM

#

yeah, I tried to write objects into the bucket, some non-zero metrics are showing up! Do you happen to know is there any official dashboard template for ontaps3 metrics? thank you!

hazy crystal Jul 15, 2024, 5:16 AM

#

@cerulean pumice There is no dashboard for these metrics curently in Harvest. Could you open a feature request for the same here https://github.com/NetApp/harvest/issues?

cerulean pumice Jul 26, 2024, 6:07 PM

#

hey, I am able to see all these ontaps3 perf metrics after enabling the template file, but there is not any capacity metrics. How do i enable ontaps3 capacity related metrics?

#

I see the "ontap_s3.yaml" is set under conf/rest/default.yaml, but i can't see any metrics about it

#

for example, i am not abled to see this metric even though the ontap_s3.yaml template is included in the rest/default.yaml

#

do i miss anything here?

alpine wedge Jul 26, 2024, 7:26 PM

#

Hi @cerulean pumice those metrics come from this endpoint. https://docs.netapp.com/us-en/ontap-restapi-9131/ontap/swagger-ui/index.html#docs-object-store-protocols_s3_buckets

An S3 bucket is a container of objects. Each bucket defines an object namespace. S3 server requests specify objects using a bucket-name and object-name pair. An object consists of data, along with optional metadata and access controls, that is accessible using a name. An object resides within a bucket. There can be more than one bucket in an S3 server. Buckets that are created for the server are associated with an S3 user that is created on the S3 server.

Do you have buckets created?

#

Not sure if you have GitHub notification on or not, but if you have anything to add to https://github.com/NetApp/harvest/issues/3055 that would be appreciated

GitHub

Feature request for Dashboard template for ontaps3 metrics · Issue ...

Is your feature request related to a problem? Please describe. I re-enable the ontap s3 metrics on 24.02.0 release by uncommenting those ontaps3 metrics templates. It'd be good if we have some ...

cerulean pumice Aug 13, 2024, 6:27 PM

#

hey folks, how am i able to see "ontaps3_used_percent" or "ontaps3_logical_used_size" metrics? I am sure we have some traffic going, but still can't see any capacity/usage related metrics for ontaps3. appreciate if you could provide some insight here.

#

also, I do see the "ontap_s3.yaml" is set under conf/rest/default.yaml, but i can't see any metrics about it

alpine wedge Aug 13, 2024, 6:28 PM

#

Hi @cerulean pumice

cerulean pumice Aug 13, 2024, 6:31 PM

#

We previously re-enable the ontap s3 metrics by adding template under zapiperf and restperf

#

then we can see those perfermance related metrics, but how to enable those usage related metrics?

alpine wedge Aug 13, 2024, 6:36 PM

#

Let's focus on ontaps3_used_percent that comes from the conf/rest/9.7.0/ontap_s3.yaml template and is computed here https://github.com/NetApp/harvest/blob/edffb8f444c3f23496fea8465bf9689522e51ca5/conf/rest/9.7.0/ontap_s3.yaml#L27

Let's skip Harvest and query ONTAP directly via REST.
what do you get if you try this, replace ip, user, pass with the appropriate values.
curl -s -k -u $user:$pass 'https://$ip/api/protocols/s3/buckets?fields=logical_used_size,size'

My guess is ONTAP is returning logical_used_size of zero which causes ontaps3_used_percent to be zero since ontaps3_used_percent = logical_used_size/size*100

cerulean pumice Aug 13, 2024, 9:02 PM

#

"uuid": "28c96e1f-599e-11ef-a3a5-00a0b8ca7f10",
"name": "kk-2",
"size": 322122547200,
"logical_used_size": 32768,
"_links": {
"self": {
"href": "/api/protocols/s3/buckets/7020fe28-57cd-11ef-a3a5-00a0b8ca7f10/28c96e1f-599e-11ef-a3a5-00a0b8ca7f10"
}
}

#

we do get the logical_sued_size back

hazy crystal Aug 14, 2024, 6:23 AM

#

@cerulean pumice
Thank you. Could you share the response of the following command via email with us?

curl -s -k -u $user:$pass 'https://$ip/api/protocols/s3/buckets?return_records=true&fields=encryption.enabled,logical_used_size,name,protection_status.destination.is_cloud,protection_status.destination.is_ontap,protection_status.is_protected,qos_policy.name,size,svm.name,uuid,volume.name&ignore_unknown_fields=true'

Also, provide the logs as described in the link below:
https://netapp.github.io/harvest/24.08/help/log-collection/

Please send the information to: ng-harvest-files@netapp.com

cerulean pumice Aug 14, 2024, 8:54 PM

#

{
"svm": {
"uuid": "7020fe28-57cd-11ef-a3a5-00a0b8ca7f10",
"name": "root-admin",
"_links": {
"self": {
"href": "/api/svm/svms/7020fe28-57cd-11ef-a3a5-00a0b8ca7f10"
}
}
},
"uuid": "ff92c8ee-599d-11ef-9d4d-00a0b8cacde9",
"name": "kk-1",
"volume": {
"name": "fg_oss_1723571975"
},
"size": 322122547200,
"logical_used_size": 12288,
"encryption": {
"enabled": false
},
"protection_status": {
"is_protected": false,
"destination": {
"is_ontap": false,
"is_cloud": false
}
},
"_links": {
"self": {
"href": "/api/protocols/s3/buckets/7020fe28-57cd-11ef-a3a5-00a0b8ca7f10/ff92c8ee-599d-11ef-9d4d-00a0b8cacde9"
}
}
}

alpine wedge Aug 15, 2024, 11:36 AM

#

Thanks @cerulean pumice I see your logical_used_size > 0 so something else must be going on. Can you grab your harvest logs and send them to us ng-harvest-files@netapp.com https://netapp.github.io/harvest/24.08/help/log-collection/

cerulean pumice Aug 15, 2024, 6:18 PM

#

hey, sent the log file already, thanks

#NEW: Harvest 24.02.0 has been released