Architecture Advise for Harvest | NetApp | Page 1

novel oak Mar 17, 2025, 8:28 AM

#

Here are some pain points user are facing:

• Each site (pv with Trident) needs to prepare a Harvest server. Currently, there are a total of 40 primary and standby Harvest servers, and performing version upgrades, configuration changes, or metric adjustments all require a significant amount of time.
-> Maybe using Ansible can help

• Although metric reduction has been applied to Harvest (using conf_path with customconf), both Harvest and AIQUM still consume 6–7% of NetApp’s performance resources (based on the A250 model).
-> Are we able to check the cpu usage of Harvest or any related document?

• For instance, in one of the harvest site, the number of cluster monitors has reached 78. When performance issues arise, clusters need to be split and monitored by two separate Harvest servers.

• The process of joining a cluster monitor to Harvest is manual, which increases the risk of missing monitors.
-> We might can integrate Ansible automation process

• Due to the large number of cluster volumes, more complex PromSQL queries cannot retrieve long-term data effectively.
-> They combine harvest metric with K8s metric and do some calculations. The calculation is complex, and can only show the data within 2 hours, data over the period will be "no data".

Here are some advice for them:

Change the Harvest performance collector interval from 1 minute to 3 minutes.
Disable metrics that are not used (Keep reducing)
Enable jitter

Any other recommendations?

cerulean basin Mar 17, 2025, 7:30 PM

#

Hi @novel oak we'll discuss this in more detail and get back to you.

Some other ideas to explore:

Try VictoriaMetrics and see if it helps with the slow PromQL queries.
Try one of the open-metrics servers that supports downsampling so you can query over longer time ranges
Let us know if you have any suggestions on how we can make the upgrading easier. We've tried to make upgrading straightforward, but maybe there is more we can do in your env. Any specifics on the upgrade pain points, sounds like you are already using conf_path which can help with custom templates?

novel oak Mar 18, 2025, 3:42 AM

#

@cerulean basin I will try to implement the Victoriametrics and integrate with the existing Prometheus.
Yes, they are using conf_path with customconf templates.

novel oak Mar 19, 2025, 5:58 AM

#

@cerulean basin Do we have any automation example for upgrading harvest, change config or add new cluster to harvest.yaml?

uneven cedar Mar 19, 2025, 6:13 AM

#

Hi Tony, what kind of Harvest installation are you using?

novel oak Mar 19, 2025, 6:17 AM

#

@uneven cedar Debian

uneven cedar Mar 19, 2025, 6:44 AM

#

There are no specific examples, but these tasks can be achieved through scripting. For example, upgrading is straightforward using the command listed at NetApp Harvest Upgrade.

Changing the configuration, such as adding or removing a cluster, involves modifying the YAML file via a script. You could accomplish this using languages like Go, Python, etc. NABox performs these tasks. @oak wraith can provide more details.

novel oak Mar 19, 2025, 7:35 AM

#

@cerulean basin From my testing
The performance of victoriametrics might not be better than prometheus
The 8428 is victoriametrics and 9090 is prometheus

oak wraith Mar 20, 2025, 6:49 AM

#

Can you run it on an actually long query ?

novel oak Apr 7, 2025, 6:29 AM

#

We have done the following for harvest and Nabox:

Keep only the templates and objects that are using. (Disable all unused)
Enable jitter to 1m
Change data interval to 3m for performance data

From user testing,
the CPU usage has dropped around 5%.

novel oak Apr 18, 2025, 3:02 AM

#

Seems like on some sites,

Keep only the templates and objects that are being used. (Disable all unused)
Enable jitter to 3m
Change data interval to 3m for performance data

The CPU usage hasn't decreased much.
Do you have any other suggestions on what we can do?

uneven cedar Apr 21, 2025, 6:00 AM

#

@novel oak How much CPU usage we're seeing due to Harvest poll? Is it affecting ONTAP's data layer? Also, have you checked with ONTAP to see if the CPU usage is causing any issues?

novel oak Apr 21, 2025, 8:22 AM

#

@uneven cedar The only way to check the CPU usage of Harvest Poll is to turn off Harvest, and check the CPU usage?

uneven cedar Apr 21, 2025, 8:23 AM

#

Sounds reasonable but pls do check with ONTAP on this.

novel oak Apr 22, 2025, 8:07 AM

#

We have a complex time series panel using multiple group_left and sum functions.
In Grafana, we can only see data within the past 12 hours; if the time range exceeds 12 hours, no data is displayed.
We tried creating a recording rule for this complex metric and set the interval to 1 hour to reduce data points, which worked.
But the pannel only shows data after we setup this recording_rules.
Does anyone know how to integrate historical data with the new metric generated by the recording rule?
Is it possible to use the API /api/v1/query_range?query=XXX&start=2025-03-01T00:00:00Z&end=2025-04-22T00:00:00Z&step=1h
to export the data, convert it into a .prom file, and then import it back into TSDB?

uneven cedar Apr 22, 2025, 8:42 AM

#

Consider downsampling your data to improve query performance without relying on recording rules. If your Harvest poll interval is 3 minutes, you could also increase Prometheus' scrape interval to 3 minutes to prevent duplicate data scrapes.

novel oak Apr 22, 2025, 9:06 AM

#

@uneven cedar Isn't using recording_rules can help with some complex calculation?

uneven cedar Apr 22, 2025, 9:07 AM

#

Both options can be considered depending on the use case.

novel oak Apr 22, 2025, 9:18 AM

#

The scrape interval changed from 2m to 3m.
And is it possible to integrate historical data with the new metric generated by the recording rule?

uneven cedar Apr 22, 2025, 9:19 AM

#

I haven't tried the backfilling via /api/v1/query_range but this should be possible.

#

You should try that in a non prod environment.

novel oak Apr 22, 2025, 9:22 AM

#

Yes, but not sure when I query API can convert to the .prom file like below, what's the next step.
volume_total_ops{aggr="DATA_SSD_N1",cluster="a150",datacenter="tplab",instance="localhost:12991",job="harvest",node="a150-01",style="flexvol",svm="infra",volume="fsa_test"} 5.016298006249962 1740787200000
volume_total_ops{aggr="DATA_SSD_N1",cluster="a150",datacenter="tplab",instance="localhost:12991",job="harvest",node="a150-01",style="flexvol",svm="infra",volume="fsa_test"} 5.812429217691095 1740790800000

Should I write data to /var/lib/prometheus/?

uneven cedar Apr 22, 2025, 9:25 AM

#

I don't think if primetheus supports backfilling.

#

victoria metrics supports that.

#

I doubt if recording rules will run for backfilled data.

novel oak Apr 22, 2025, 9:51 AM

#

Recording rules only apply to data ingested after their creation, which means the panel displays metrics starting from when the rules were set up.
I would like to backfill historical data so it is included in the panel visualization.
While I can successfully query the historical data, I am not sure how to ingest or write this data back into the TSDB with sepcific metric_name

uneven cedar Apr 22, 2025, 10:13 AM

#

https://victoriametrics.com/blog/rules-replay/

#

Victoria metrics may help

novel oak Apr 28, 2025, 8:51 AM

#

@uneven cedar After setting up VictoriaMetrics and VMalert, we can successfully add recording_rules data into VictoriaMetrics DB.

#Architecture Advise for Harvest