Harvest installation K8 deployment with Templates customization

[coral zealot]

I am trying to deploy harvets with K8s
And I am using Templates customization:
conf_path: customconf
in harvest.yml

customconf is a directory copy from conf,
and we make some changes here.

When running
docker run --rm \ --env UID=$(id -u) --env GID=$(id -g) \ --entrypoint "bin/harvest" \ --volume "$(pwd):/opt/temp" \ --volume "$(pwd)/harvest.yml:/opt/harvest/harvest.yml" \ ghcr.io/netapp/harvest \ generate docker full \ --output harvest-compose.yml

I think I should change the default setting from the document, and how to change?
Should I add
--volume "$(pwd)/customconf:/opt/harvest/customconf"\ ?

[slate blaze]

Hi @coral zealot the bin/harvest generate docker command includes a confpath argument to help set the conf path. For example, ``-confpath a/b/c` results in the following docker volumes section being created

volumes:
      - ./cert:/opt/harvest/cert
      - ./cbg/harvest.openlab.yml:/opt/harvest.yml
      - ./secrets/openlab:/opt/harvest/secrets/openlab
      - ./a/b/c:/opt/harvest/a/b/c

To have Harvest us that additional path as a conf_path you will also need to update your poller section in your harvest.yml file. https://netapp.github.io/harvest/nightly/configure-templates/#conf-path

It is better to update your harvest.yml first with the conf_path and then use the generate command you pasted above. If you do that, the generate command will create the additional volume mount for you automatically without requiring the --volume or --confpath arguments. That's because the generate command reads your harvest.yml to help it know what to create

[coral zealot]

From the K8s deployment document,
I only need harvest.yml beforehead.
I cannot run bin/harvest generate docker
I have the conf_path set in harvest.yml.
When I run the command below to generate the compose file:
docker run --rm \ --env UID=$(id -u) --env GID=$(id -g) \ --entrypoint "bin/harvest" \ --volume "$(pwd):/opt/temp" \ --volume "$(pwd)/harvest.yml:/opt/harvest/harvest.yml" \ ghcr.io/netapp/harvest \ generate docker full \ --output harvest-compose.yml
Do I need to add this volume mount?--volume "$(pwd)/customconf:/opt/harvest/customconf"` ?
And should the customconf directory exist beforehead?

Additional, what are the best practices for deploying Harvest on Kubernetes, especially when monitoring many sites and clusters.

[slate blaze]

no, you do not need to add the volume mount. If your harvest.yml includes conf_path, the generate command will create that volume mount line for you.

There are lots of ways to run Harvest in containers and we've documented one way here

[coral zealot]

@slate blaze So just make sure the customconf directory exist before running docker run command?

[slate blaze]

Hi @coral zealot that directory does not need to exist before running docker run

[coral zealot]

@slate blaze
We are currently running NetApp Harvest and Prometheus on physical servers, with 40+ plants, each plant containing several ONTAP clusters. In our current setup, each harvest.yml manages multiple cluster pollers.

We are planning to migrate to Kubernetes, deploying Harvest as individual pods, and use VictoriaMetrics as our TSDB backend (also deployed on K8s).

What are the best practices for this scenario?

1. Should we deploy “one cluster per harvest.yml” (i.e., one Harvest Service/Pod/PV per cluster)?
2. Or is it better to stick to the existing method, with a single harvest.yml covering multiple pollers?

Also, I assume that VictoriaMetrics scrapes the Harvest exporter endpoints periodically, just like Prometheus does.
VictoriaMetrics does not directly replace the Exporter functionality of Harvest; instead, it replaces the Prometheus server’s role of scraping and storing the metrics.
So I do not have to change the exporter setting in harvest.yml
Just keep the prometheus setting?

Any architectural or management suggestions when scaling to many clusters would be greatly appreciated.

[warm oak]

@coral zealot harvest.yml is just a configuration file that Harvest uses to obtain poller information. You can keep the configuration as convenient for your setup. You can also explore this documentation, which explains how to use multiple Harvest configuration files.

That’s right, you can use the Prometheus exporter with VictoriaMetrics.

[coral zealot]

@warm oak @slate blaze
We are deploying NetApp Harvest to monitor multiple ONTAP clusters. We would like to understand:

1. Is there any recommended limit or guideline on how many NetApp clusters a single Harvest instance can manage effectively?
2. Does Harvest provide any built-in High Availability mechanism? If not, what are common practices to achieve HA for Harvest, especially when deployed on Kubernetes?
   Any advice or best practices around scaling Harvest and ensuring its availability would be greatly appreciated.

From this discussion, the recommendation is one poller(cluster) each single harvest container?
https://github.com/NetApp/harvest/discussions/2041

[warm oak]

@coral zealot You can refer to the NABox configuration guide at https://nabox.org/documentation/installation/#compute-resources. This documentation covers components such as VictoriaMetrics, Harvest, and Grafana. In this doc, the term "Controller" refers to the number of clusters.

For Harvest specifically, we recommend allocating a minimum of 100MB of memory per poller.

High availability (HA) is managed by the orchestrator. Harvest itself does not provide built-in HA capabilities. Think of Harvest as a single process per cluster that should remain running.

Our customers typically deploy one container per poller, though some run multiple pollers within a single container. We recommend using one container per poller to avoid creating a monolithic setup and to maintain flexibility and scalability.

[coral zealot]

So, we don't have a limit on pollers per harvest, right?
And the recommended way is to deploy one container per poller,
Which means one cluster per harvest container?

[neat grove]

Yes, There is no specific limit for number of pollers per harvest.
Correct each poller(cluster) would run in their own container.

[coral zealot]

@neat grove So each cluster should have its own harvest.yml?
Or one harvest.yml with multiple pollers will automatically create container for each poller(cluster)?

[neat grove]

It's latter one, one harvest.yml with multiple pollers will automatically create container for each poller(cluster).

[coral zealot]

For what I understand,
I need to add conf_path: customconf to harvest.yaml
And I need a custom_conf directory which contains my conf.
And I need to copy this dir to a PVC(harvest-custom-conf-pvc).
And then alter kub.yaml
Adding
`volumeMounts:

• mountPath: /opt/harvest/custom_conf
  name: harvest-custom-conf-volume
  volumes:
• name: harvest-custom-conf-volume
  persistentVolumeClaim:
  claimName: harvest-custom-conf-pvc`

This is the way I use custom_conf when deployment harvest in K8s.
The kompose convert will not copy the customconf dir to the container.