drowsy tree
Hi, Is there a way to split harvest.yaml to keep different pollers in different configuration files?
Ex.: pollers logically separated by ONTAP clusters locations? (when you have really many clusters...)
Smth like
include_file emea_clusters.yaml
include_file apac_clusters.yaml
grave atlas
@drowsy tree It is not supported currently in Harvest. Which installation method are you using for Harvest? rpm,deb,nabox,container, native?
grave atlas
Okay. Please open a GitHub issue for this at https://github.com/NetApp/harvest/issues. Harvest pollers execute a binary where options can be passed. Running ps -ef | grep poller may display something like below:
bin/poller --poller dc-1 --loglevel 2 --promPort 12993 --config harvest_openlab.yml
For now, you'll need to customize the --config option according to the poller that requires a tailored solution.
drowsy tree
would that be possible then, as a workaround, to have a separate harvest instances an the same host ran with a specific harvest.yaml file which picks differen pollers, right?
something like a bash script executed with different configuration files
grave atlas
Yes it is possible. You can run separate Harvest instances in that case on the same host. If you are using prometheus then just need to make sure that prometheus port configuration in harvest config is unique.
drowsy tree
They have a dedicated Prometheus host, can't multiple harvest instances write to a single exporter?
the configuration: 1 harvest host, 1 Prom+Grafana host
grave atlas
In RHEL, there is a service file that you might need to customize, or you can simply below command to start harvest instance.
bin/harvest start --config harvest_config.yml
grave atlas
They have a dedicated Prometheus host, can't multiple harvest instances write to...
Multiple instances of Harvest will function properly. Each Harvest instance writes metrics to an HTTP endpoint, which Prometheus then scrapes.
See if this helps https://github.com/NetApp/harvest/wiki/FAQ#answer-8
grave atlas
Thanks
drowsy tree
Hi, not sure if this thread is still a valid place to ask more questions - can't make this configuration work. I have 2 separate harvest.yaml files with different pollers and exporters are on different ports. When im starting the second harvest instance - the first one is being killed with "Caught signal [terminating]" and "cleaning up and stopping [pid=xxx]"
i am using --config option to point at different harvest.yaml files
grave atlas
@drowsy tree Could you share both of your harvest.yaml via below commands from the relevant harvest directories.
bin/harvest doctor --print
drowsy tree
Admin:
Tools:
grafana_api_token: '-REDACTED-'
Exporters:
prometheus:
exporter: Prometheus
port: 12992
Defaults:
collectors:
- Zapi
- ZapiPerf
use_insecure_tls: false
Pollers:
barbuda:
datacenter: MUC
addr: -REDACTED-
auth_style: basic_auth
username: -REDACTED-
password: -REDACTED-
use_insecure_tls: true
exporters:
- prometheus
Tools:
grafana_api_token: '-REDACTED-'
Exporters:
prometheus1:
exporter: Prometheus
port: 12990
Defaults:
collectors:
- Zapi
- ZapiPerf
use_insecure_tls: false
Pollers:
jamaica:
datacenter: MUC
addr: -REDACTED-
auth_style: basic_auth
username: -REDACTED-
password: -REDACTED-
use_insecure_tls: true
exporters:
- prometheus1
here it is
grave atlas
Thanks @drowsy tree . You are right. bin/harvest start command will kills pollers which do not exist in config. In your case when you start harvest from second config, it will kill pollers from first config.
We can make it work via container way of harvest. https://netapp.github.io/harvest/23.08/install/containers/
drowsy tree
Yes, that would be an option, just wanted to circumvent the situation my customer to reinstall what is already running. If there is no way to run multiple harvest instances on a single host ENV without them being disturbed then i will take the container configuration, just was looking for the confirmation it's not me misconfigured something but just not possible
grave atlas
we can make it work in rhel also but that would need you to maintain bin/poller command which is managed easily in harvest containers.
drowsy tree
oh. Okay, does it mean that in situation of 400 clusters you woud need to start 400 pollers individually either with bash script or with customizing RHEL service start files?
grave atlas
yes something like that . Container would be a good solution.