Ontap: Volume shows Metrocluster svms and volumes (*-mc)

[tropic briar]

Nabox Version 4.0.10
Harvest Version 25.05.0

[rotund bear]

We only exclude the volumes listed below from monitoring. Are you suggesting that we should also exclude MetroCluster SVMs and volumes ending with *-mc as well?

      # Ignore transient volumes, e.g. SnapProtect, SnapManager, SnapCenter, CommVault, Clone, and Metadata volumes
      exclude_regex:
        - volume `.+_CVclone`
        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
        - volume `sdw_cl_.+`
        - volume `MDV_CRS_.+`
        - volume `MDV_aud_.+`

[tropic briar]

I think it would be right if i select a specific cluster regardless of whether it is part of a metro cluster or not.
It was like that in our older version 24.05.0-1 if i can recall right 🙂

[rotund bear]

Do you mean there is a filtering issue in dropdowns? Could you pls add more details.

[tropic briar]

Yes. With old Version of Harvest we would get only SVMs and Volumes that belong to the Local Cluster.

I have sent you two screenshots in PVT

[rotund bear]

@tropic briar Could you please check if your dashboard variable queries are similar to the ones below?

For the SVM dropdown, the values are selected from volume_labels for that cluster.

[rotund bear]

The query is correct—it indicates that Harvest is collecting these _mc SVMs.
If the same clusters are involved across NABox3 to NABox4, and as you mentioned these SVMs didn't exist in NABox3, could you confirm whether there was a change in the collector type, such as from ZAPI to REST, between NABox3 and NABox4?

[rotund bear]

Could you please provide the output (volume_data.xml and volume_rest.json) of the following curls commands for relevant cluster for which you see _mc svm records?
Please replace USER, PASSWORD, and CLUSTER_IP with the appropriate values.

curl -s --connect-timeout 30 --user USER:PASSWORD --insecure --data-ascii '<?xml version="1.0" encoding="UTF-8"?>
<netapp xmlns="http://www.netapp.com/filer/admin" version="1.130">
  <volume-get-iter/>
</netapp>' -H "Content-Type: text/xml" 'https://CLUSTER_IP/servlets/netapp.servlets.admin.XMLrequest_filer' 2>&1 | tee volume_data.xml


curl -sk -u USER:PASS 'https://CLUSTER_IP/api/private/cli/volume' > volume_rest.json

You can upload these files @ https://upload.nabox.org/nofu-kagu-syju

[tropic briar]

Nabox3 had Harvest Version 24.05.0-1

[tropic briar]

Will provide tomorrow

[rotund bear]

Thanks. Please share logs from NABox3 and NABox4 as well.

[tropic briar]

which Logs do you need additional to the curl output?

[rotund bear]

Harvest logs

[tropic briar]

Ok i will try

[tropic briar]

@rotund bear have uploaded everything. let me know if you need more 🙂

[shadow vortex]

@tropic briar Thanks for the response and log files
It seems zapi response would be having only 20 records, but rest response have 498.

There are quite number of svms found in rest response and if that is the case, Harvest could export them as-is.

[tropic briar]

We use rest for both nabox and harvest versions

As is said, it would be correct to only query !*-mc SVMs because those svms and volumes arent online until a switchover happens.

[shadow vortex]

Thanks for the info.
Could you please again provide below 2 files after running the below curl commands for relevant cluster?
Please replace USER, PASSWORD, and CLUSTER_IP with the appropriate values.

curl -sk -u USER:PASS 'https://CLUSTER_IP/api/storage/volumes' > volume_public_rest.json

curl -sk -u USER:PASS 'https://CLUSTER_IP/api/svm/svms' > svm_public_rest.json

[shadow vortex]

@tropic briar , As per our investigation regarding those svms with name -mc, doc says that: Vserver names ending with "-mc" are reserved for Vservers created by the system for MetroCluster operations.

If we could ensure that volumes belong to that SVM with the name -mc and if they are offline, then those volumes should not be exported from the template. If they are online due to a switch-over or some equivalent case, then they will be exported as-is.
Does this sound good?

[steel dagger]

I will provide it tomorrow

[steel dagger]

Exactly.
Those SVMs and the respective volumes only come in playce if there is a metrocluster switchover.

[shadow vortex]

Could you confirm that, This behaviour would be expected in SVM object also ?
Meaning, Could you want those svms with -mc ended would not be visible in SVM dashboard?

[tropic briar]

Yes i think it would be right because they deliver No Data as query response anyway.
I would make them visible only if the mc svms are online and delivering storage (e.g. if a switchover is active).

I have uploaded the requested json files

[shadow vortex]

Thanks for the jsons files, I would be checking them and replay with locally.

[shadow vortex]

My bad, I forgot to add the fields as * in the Rest api.
could you please re-run this below api
Please replace USER, PASSWORD, and CLUSTER_IP with the appropriate values.

curl -sk -u USER:PASS 'https://CLUSTER_IP/api/svm/svms?fields=*' > svm_public_updated_rest.json

Btw, The changes for handling svms with -mc are discussed internally and sent out the PR https://github.com/NetApp/harvest/pull/3672 to review. This could handles both volumes and svms with zapi and rest collectors.

[shadow vortex]

@tropic briar The changes are now available in nightly build if you want to try them out. Thanks for raising this use-case.
https://github.com/NetApp/harvest/releases/tag/nightly

[tropic briar]

Thank you very much. We will look into it

[tropic briar]

We cant confirm that its fixed.
We still see all -mc svms and volumes related to them.

Harvest Version Harvest 25.06.02

[muted cairn]

hi @tropic briar can you grab a new support bundle and upload to https://upload.nabox.org/nofu-kagu-syju

[tropic briar]

Hey Chris 🙂
uploaded

[muted cairn]

thanks!

[muted cairn]

The logs look fine. I can see the before you upgraded to nightly the fashap07 poller was exporting 490 volume instances and after the upgrade it was exporting the same number. I would have expected that number to decrease.

The code change for this was pretty straightforward - check if a volume's SVM ends with "-mc" and if the volume has a state of offline. If both of those conditions are true, don't export the volume. Let's make sure that we are getting those values and their values are as expected from your cluster by running the following, replacing $user, $pass, $ip.

curl --silent --insecure -u $user:$pass -H 'Accept: application/json' 'https://$ip/api/storage/volumes?fields=state,svm.name' > volume-state.json

Please upload volume-state.json to the same share.

[tropic briar]

Have uploaded the file with name: volume_rest_2.json because of the old files

[tropic briar]

@muted cairn look at my pm. State is not offline for a mirror volume in a metrocluster. Maybe thats causing it at it would be better to filter state is not online?

Just guessing

[muted cairn]

Thanks @tropic briar it's odd that your curl is not returning the state field. Locally when I run that curl I see the following, but your output has a different shape that is missing state, that's why they are still being shown in the dashboard. Are you sure that you ran the curl I shared with fields=state,svm.name That field should exist on your cluster's REST API

[tropic briar]

The state in a metrocluster for mirror volumes is "-" even in the cli (the output i sent you in pvt)

[muted cairn]

yes I saw that, but still expect the REST response to include something for that field

[tropic briar]

I run you exact command only with our cluster name, user and pw

[muted cairn]

and I guess this cluster has nothing but mirror volumes?

[tropic briar]

Nono the cluster has both

everyhting without -mc is active on the cluster.
If it helps we could have a call and i show you

[muted cairn]

your JSON response is confusing then since none of the volumes included a state

[tropic briar]

Mom

w8 maybe i made a mistake. Will check again and report

Yes sorry..i uploaded the wrong file. Can you check now the newest file uploadedß

[muted cairn]

thanks - the world makes a little bit more sense now 🙂

this data supports your earlier observation - the state field is missing from the volumes with an svm suffixed with -mc. That's not what we expected based on the ONTAP swagger, but we can accommodate that. I'll yell when we have a new nightly with that fix. Thanks a lot for the data

Let's see if SVM's follow a similar pattern of metrocluster svms not having a state. Can you run this and upload, please.
curl --silent --insecure -u $user:$pass -H 'Accept: application/json' 'https://$ip/api/svm/svms?fields=state,name' > svm-state.json

[tropic briar]

Uploaded the requested output.
svm show with state stopped. But look into tha last PM i sent you. I dont know if the api returns this field, but maybe it would be more safe with ith

[muted cairn]

thanks!

[muted cairn]

@tropic briar there is a new nightly build that incorporates the fix discussed above. When you get a chance, let us know how that works for you

[tropic briar]

No. We still can see the -mc svms and mirror Volumes.
I uploaded a new support bundle.

[muted cairn]

I'll put together a nightly build with extra logging enabled to get to the bottom of why you're still seeing them.

Let's also check VictoriaMetrics. If you go to http://$nabox-ip/vm/vmui and type volume_labels{svm=~".*-mc"} in the query box and press Enter, what do you see?

[muted cairn]

Hmm that means there are no volume_labels with an SVM with the mc prefix. That is what we want.

Can you check the volume dashboard again and make sure you weren't seeing cached data after upgrading to nightly? Try pressing the Refresh button on the dashboard and then select a small time range like Last 1 hour so we can be certain that Grafana is not querying for Prometheus metrics before you took the nightly build today