Custom Rest collector template doesn't seem to merge properly with default template | NetApp | Page 1

wheat sedge Oct 22, 2025, 4:20 PM

#

I am trying to configure the Rest and RestPerf collectors to poll less frequently than the default. I am trying to accomplish this by creating the custom.yaml files in these paths conf/rest/custom.yaml conf/restperf/custom.yaml so that they merge with the default.yamls.
here's what my custom.yamls look like:
Rest:

  custom.yaml: |-
    schedule:
    - counter:  24h
    - data:      5m

RestPerf:

  custom.yaml: |-
    schedule:
    - counter:  24h
    - instance: 720h
    - data:      3m

When I tried it, I was able to get the RestPerf collector successfully polling less frequently (changed from 1 min default to 3 minutes) but the Rest collector just stopped polling at all. Is there something wrong with my Rest custom.yaml?

gentle sentinel Oct 22, 2025, 5:07 PM

#

Hi @zjeffrey, you're close. These work correctly for me.
Try changing your conf/rest/custom.yaml to this

schedule:
  - counter: 24h  # This handles cases such as cluster upgrades or collector cache updates.
  - data:     5m

and your conf/restperf/custom.yaml to:

schedule:
  - counter:  24h
  - instance: 720h   # The instance schedule is only used by the `workload` and `workload_volume` templates
  - data:      3m

these are 5 minutes apart now

time=2025-10-22T12:58:50.351-04:00 level=INFO source=collector.go:601 msg=Collected Poller=sar collector=Rest:Quota apiMs=150 bytesRx=31580 calcMs=0 exportMs=1 instances=49 instancesExported=49 metrics=486 metricsExported=561 numCalls=1 parseMs=1 pluginInstances=0 pluginMs=0 pollMs=151 renderedBytes=131270 zBegin=1761152330198

time=2025-10-22T13:03:50.540-04:00 level=INFO source=collector.go:601 msg=Collected Poller=sar collector=Rest:Quota apiMs=318 bytesRx=31580 calcMs=0 exportMs=1 instances=49 instancesExported=49 metrics=486 metricsExported=561 numCalls=1 parseMs=2 pluginInstances=0 pluginMs=0 pollMs=320 renderedBytes=131270 zBegin=1761152630219

You may also want to increase cache_max_keep since the default value is 5m. You want cache_max_keep to be larger than your collectors polling schedule. https://netapp.github.io/harvest/nightly/prometheus-exporter/#parameters

wheat sedge Oct 22, 2025, 5:39 PM

#

so the indentation is the issue? I tried changing the indentation to match your example but I'm still having the same issue

#

my harvest version is harvest:v24.02.0-gke.2 just fyi

gentle sentinel Oct 22, 2025, 5:42 PM

#

does your yaml contain |-?

wheat sedge Oct 22, 2025, 5:43 PM

#

that's there because I'm storing it as a secret in kubernetes

#

data:
  custom.yaml: c2NoZWR1bGU6CiAgLSBjb3VudGVyOiAgMjRoCiAgLSBkYXRhOiAgICAgIDRt

#

echo c2NoZWR1bGU6CiAgLSBjb3VudGVyOiAgMjRoCiAgLSBkYXRhOiAgICAgIDRt | base64 -d
schedule:
  - counter:  24h
  - data:      4m

#

this is what the logs look like for the rest collector:

kubectl logs netapp-harvest-65456d86c6-7cdwl | grep Rest:Support

2025-10-22T17:29:54Z INF collector/helpers.go:84 > best-fit template Poller=aa-aa-stge01-zone1 collector=Rest:SupportAutoUpdate path=conf/rest/9.12.0/support_auto_update.yaml v=9.16.1
2025-10-22T17:29:57Z INF collector/helpers.go:84 > best-fit template Poller=aa-aa-stge01-zone1 collector=Rest:Support path=conf/rest/9.12.0/support.yaml v=9.16.1

it seems to start the collector but then never collects anything

gentle sentinel Oct 22, 2025, 5:49 PM

#

when I run with debug logs I see the custom.yaml being used. Can you try the same? 24.02 is quite old, let me check there

#

can you upgrade?

wheat sedge Oct 22, 2025, 5:52 PM

#

how do I enable the debug logs? do I have to pass a specific flag?

gentle sentinel Oct 22, 2025, 5:53 PM

#

in the screenshot I shared above, at the end of the line is -l1 which sets loglevel to debug

wheat sedge Oct 22, 2025, 5:53 PM

#

gotcha

#

I do see the logs that say the templates are merged

2025-10-22T17:55:21Z DBG poller/poller.go:221 > set signal handler for [hangup interrupt terminated quit] Poller=aa-aa-stge01-zone1
2025-10-22T17:55:21Z INF poller/poller.go:239 > started in foreground Poller=aa-aa-stge01-zone1 pid=1
2025-10-22T17:55:21Z DBG poller/poller.go:618 > Merged template. Poller=aa-aa-stge01-zone1 template=custom.yaml
2025-10-22T17:55:21Z DBG poller/poller.go:618 > Merged template. Poller=aa-aa-stge01-zone1 template=custom.yaml

gentle sentinel Oct 22, 2025, 5:58 PM

#

ah, I think I see the issue. change your conf/rest/custom.yaml to

schedule:
  - data:     5m

#

In your log files, do you see a message like this?
WRN poller/poller.go:687 > init collector-object error="implementation error => PollCounter"

wheat sedge Oct 22, 2025, 6:01 PM

#

oh yeah I do

#

WRN poller/poller.go:687 > init collector-object error="implementation error => PollCounter" Poller=aa-aa-stge01-zone1 collector=Rest object=Subsystem

gentle sentinel Oct 22, 2025, 6:01 PM

#

version 24.02 of Harvest did not support a poll counter which your schedule has. Later versions of Harvest do support poll counter. So basically you are using a newer version of the template on an older version of Harvest and that doesn't work in this case

wheat sedge Oct 22, 2025, 6:03 PM

#

should I remove the counter field from the restperf schedule too?

gentle sentinel Oct 22, 2025, 6:03 PM

#

no, 24.02 includes that (and needs it)

#

with that change, how are things looking?

wheat sedge Oct 22, 2025, 6:19 PM

#

seems to work now, thanks for helping!

gentle sentinel Oct 22, 2025, 6:20 PM

#

you're welcome, glad it's working!

wheat sedge Oct 27, 2025, 3:17 PM

#

Quick question regarding the cache_max_keep you mentioned, is the problem with the polling interval being greater than cache_max_keep that it would result in gaps in the metric's time series? eg if the cache_max_keep is 7 mins but the polling interval is 10 mins, the time series would show data for 7 mins after the initial scrape, then gets dropped by the cache so there's a gap in data for 3 minutes until the poller scrapes again at 10 mins.

#Custom Rest collector template doesn't seem to merge properly with default template