Metric Snapshot Policy Violation

[floral axle]

Hey everyone, I tried upgrading the Nabox4 version for the Harvest to enable the metric for snapshot policy violations (25.11.10).
When I did that, only some of the clusters received the configuration and are working, while others don’t show up at all in the Grafana.

It looks like the plugin does not work for some of my clusters (for the record the Netapp version is 9.14.10)

I can see the clusters connected but the Grafana isn’t displaying any of their data protection metrics.
Do you know how I can fix this?

I’m attaching the file I used to create the new configuration. I added the commands that are written there for the Rest Collector.

https://github.com/NetApp/harvest/discussions/3854

[zinc veldt]

@floral axle Could you upload NABox support bundle @ https://upload.nabox.org/fopo-qymy-gisy

[floral axle]

I can’t unfortunately

I can send you the specific logs you ask for.

[zinc veldt]

Okay. Could you run below command in NABox after ssh into it and then share harvest.log?

docker logs havrest > harvest.log 2>&1

You can email to [e-mail] as well.

[floral axle]

There are two types of errors

[zinc veldt]

Let's focus on snapshot data if there are any errors there. What if you run below command

docker logs havrest 2>&1 | grep -i snapshot

for snapshot errors try below

docker logs havrest 2>&1 | grep -i snapshot | grep -i err

[floral axle]

Can’t run this command

There is no output

Wait

Also, It looks like this in the Nabox

[zinc veldt]

for example in below log, there are 160 plugin instances which points to snapshot policy violation detection.

[floral axle]

Oh ok

[zinc veldt]

second and 3rd clusters may have some errors.

Try below and see if any error for these pollers.. replace pollername in below command

docker logs havrest 2>&1 | grep -i pollername | grep -i err

[floral axle]

For them I cant see any data in the “Harvest-cDOT: Data Protection” dashboard

[zinc veldt]

Yes for atleast 2 clusters, we have no snapshot instances

[floral axle]

These are the errors

[zinc veldt]

These shouldn't cause any problems.

[floral axle]

Wdym

[zinc veldt]

Even volumes are not visible for these clusters?

[floral axle]

No

[zinc veldt]

Okay. That is strange.

[floral axle]

Yes 🙁

[zinc veldt]

Let me give you a command to check related to volume metrics

Try below

docker logs havrest 2>&1 | grep -i rest:volume\ 

and

docker logs havrest 2>&1 | grep -i zapi:volume\ 

[floral axle]

This is for the Rest

[zinc veldt]

Can you also add pollername in filter for which we don’t see volume metric

docker logs havrest 2>&1 | grep pollername | grep -i rest:volume\

[floral axle]

It stucks

[zinc veldt]

Try below

docker logs havrest 2>&1 | grep pollername | grep -i rest:volume

[floral axle]

Doesnt work either

[zinc veldt]

Strange .. could you share what is the issue?

[floral axle]

It just shows ‘>’ for ten minutes

[zinc veldt]

If it is showing like this then command has an issue. It wont run

Works fine for me

docker logs havrest 2>&1 | grep umeng-aff300-01-02 | grep -i rest:volume\ 

Replace umeng-aff300-01-02 with your poller names when you try this

[floral axle]

It stucks 😫

No output

Oh wait!!

[zinc veldt]

Is there any way for you to share support bundle? That will be faster for us to debug.

[floral axle]

I can’t it’s classified 😔

[zinc veldt]

Volume metrics are being collected for this poller

[floral axle]

I can’t see the data protection metrics

All the other metrics work

Mever mind

I cant see nothing 😣

[fierce bay]

Ok, then let's check in VictoriaMetrics UI and see these metrics are available there or not.
You can run this below metrics in query bar,
snapshot_volume_violation_count and
volume_labels

[fierce bay]

This is grafana UI, You can hit VictoriaMetric UI at https://NABOX_IP/vm/vmui/ and then type metrics in query bar

[floral axle]

Unfortunately, I have to go now. I’d really appreciate it if we could talk tomorrow at the same time. I’ll write to you, and I hope we’ll sort this out. Thank you so, so much for all your help so far.
Have a great day!!

[floral axle]

Ill try this first thing tomorrow 🙂

[fierce bay]

Sure, @floral axle , we can continue tomorrow

[floral axle]

Thanks!!