#Inconsistencies in duplicated metrics in KeyPerf x RestPerf

1 messages · Page 1 of 1 (latest)

weary ingot
#

Hey there Harvest Team, congrats on the amazing 25.05 version : )

My team is making some tests with the new KeyPerf collector, but we noticed that some metrics exposed by it are apparently duplicates of those from RestPerf.

Let's take "volume_write_latency" as an example, since it appears in both restperf and keyperf.

I still do not understand if the data points are similar, identical or different.

  • If I take this one, it's identical.
    volume_write_latency{cluster="cluster1",volume="vol13_cluster1"}
  • If I take this one, it only exists in keyPerf and not in restPerf
    volume_write_latency{cluster="cluster1",volume="vol2343_cluster1"} 


So I don't know if we are supposed to disable this metric from keyPerf collector or not. In other words, are  we are supposed to use them both or one of them, or somehow a mix of them and if so, how?

I would appreciate some guidance on this situation.

Thank you in advance,
toxic cypress
#

hi @weary ingot volume_write_latency exists in RestPerf, ZapiPerf, and KeyPerf https://netapp.github.io/harvest/nightly/ontap-metrics/#volume_write_latency

At runtime, Harvest will deduplicate overlapping object/collector pairs based on the order you specify in your harvest.yml. For example: if your poller specifies

        collectors:
        - Rest
        - RestPerf
        - KeyPerf

and you collect the Volume object, your log files will show the Rest:Volume and RestPef:Volume collector running, but not the KeyPerf:Volume collector. That's because RestPerf and KeyPerf are both performance collectors and it doesn't make sense to collect Volume resources twice. Likewise, if there was an object that only existed as a KeyPerf template, it would be started since there is no overlap.

Harvest will also detect at runtime what the cluster supports. For example, ASAr2 does not support RestPerf so any RestPerf collectors for that cluster will be converted to a KeyPerf collector. This only happens for clusters where RestPerf does not exist and is logged like so:

time=2025-06-11T10:33:50.623-04:00 level=INFO source=poller.go:722 msg="upgraded collector" Poller=sti_cluster old=RestPerf new=KeyPerf

weary ingot
#

@toxic cypress Thank you once again for your response. There is an aspect to this discussion which still makes me confused about the lack deduplication job done by Harvest.

For instance, when visualizing the same metric in Grafanavolume_write_latency{cluster="cluster1",volume="cluster1cst01"}

There are two identical metrics (one with harvest_exporter_name="harvest-cluster1-keyperf" and one with harvest_exporter_name="harvest-cluster-restperf"), both with the same value, when in theory there should be no duplication.

volume_write_latency{ [other labels], cluster="cluster1", [...], harvest_exporter_name="harvest-cluster1-keyperf", [...], volume="cluster1cst01"} 0

volume_write_latency{ [other labels], cluster="cluster1", [...], harvest_exporter_name="harvest-cluster1-keyperf", [...], volume="cluster1cst01"} 0

Which brings me to the following question: in this case should we use one of the two collectors? Are they somehow complementary ?

wicked dragon
weary ingot
#

@wicked dragon Done 🙏

#

If there is anything else from our end please let me know

toxic cypress
#

Your harvest.yaml includes four pollers - Is the addr the same for each of the pollers or different?

#

I'm guessing each of those pollers is using the same addr. If so, that would explain why you are seeing duplicates. There should be one poller per cluster. A single poller can include multiple collectors, and as mentioned above, those collectors will be deduplicated.

Try removing all the redundant pollers and changing the one remaining poller's collectors block to

        collectors:
          - Rest
          - RestPerf
          - KeyPerf
          - Ems
weary ingot
wicked dragon
#

Please ensure that you use a single Poller for each address, as previously recommended. For deduplication to function correctly, configure the collectors in the format outlined below.

        collectors:
          - Rest
          - RestPerf
          - KeyPerf
          - Ems
weary ingot
#

I understand. We opted for using multiple pollers per cluster to ease the load per collector that was causing some errors in our architecture, specially when forwarding all the metrics to FluentBit and then Mimir. With multiple collectors per poller, it was causing massive chunks which could not be forwarded in our pipeline

#

in this case I suppose it is a matter of finding another way of easing the load of metrics per collector other than creating multiple pollers per cluster, which is not advisable in this case

wicked dragon
#

That's correct. If you are using only the Rest, RestPerf, KeyPerf, and Ems collectors, deduplication occurs between the RestPerf and KeyPerf collectors. Therefore, you can maintain two pollers with the same address: one poller should include the Rest and Ems collectors, while the other should contain the RestPerf and KeyPerf collectors.

weary ingot
#

@wicked dragon Great. We will attempt to distribute the pollers as such. Thank you and @toxic cypress for the great support

wicked dragon
#

You are welcome.