high sapphire Oct 28, 2022, 6:58 AM

#

Hi Team,
i can see harvest data under /var/log/harvest file. But I can't see the data on Prometheus web.

  Datacenter |   Poller   |  PID  | PromPort | Status   
-------------+------------+-------+----------+----------
  DC-01      | landcastle | 32898 |    12991 | running  

Can someone help me I am missing. 

I have downloaded the harvester from GitHub and modified the harvest.yml file and run it directly

coarse pollen Oct 28, 2022, 11:55 AM

#

hi @high sapphire few questions. 1) did you add the poller as a scrape target to Prometheus? https://github.com/NetApp/harvest/blob/main/cmd/exporters/prometheus/README.md#static-scrape-targets
2) if you curl the poller and confirm that data is being published? curl 'http://localhost:12991/metrics'

GitHub

harvest/README.md at main · NetApp/harvest

Open-metrics endpoint for ONTAP . Contribute to NetApp/harvest development by creating an account on GitHub.

high sapphire Oct 31, 2022, 4:50 AM

#

coarse pollen hi <@1009417983392759859> few questions. 1) did you add the poller as a scrape t...

HI Chris,
Could you please share any videow how to setup harvester and connect netapp clusters

oak canyon Oct 31, 2022, 7:27 AM

#

@high sapphire What type of installation are you trying? docker, rpm, deb?
Have you added harvest poller scrape point to prometheus configuration as Chris asked above?
Could you share your prometheus configuration?

high sapphire Nov 2, 2022, 10:08 AM

#

@oak canyon
cat /etc/prometheus/prometheus.yml
global:
scrape_interval: 10s

scrape_configs:

job_name: 'prometheus_master'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']

scrape_configs:

job_name: 'harvest'
http_sd_configs:
- url: http://localhost:8887/api/v1/sd

#

vi harvest.yml
Admin:

Listen address for Prometheus HTTP service discovery end-point

The endpoint is reachable at SCHEMA://<listen>/api/v1/sd

Should be one of: localhost:port, :port, or ip:port

httpsd:
listen: :8887
# listen: localhost:8887

# Basic auth for - /api/v1/sd
 auth_basic:
   username: admin
   password: admin

# create self-signed certs with 'bin/harvest admin tls create server'
# tls:
#   cert_file: cert/admin-cert.pem
#   key_file: cert/admin-key.pem

Tools:

grafana_api_token: 'aaa-bbb-ccc-ddd'

autosupport_disabled: true

Exporters:
prometheus:
exporter: Prometheus
local_http_addr: 0.0.0.0
port: 12990
prometheus1:
exporter: Prometheus
port_range: 13000-14000

Defaults:
collectors:
- Zapi
- ZapiPerf
- Ems
use_insecure_tls: false

Pollers:

This is an optional poller that provides stats about harvest itself.

To remove it, simply delete the "unix" section

add your cDOT / 7Mode pollers below, example:

cluster-01:
datacenter: DC-01
addr: 172.23.140.201
auth_style: basic_auth
credentials_file: path/to/credentials.yml # read credentials from the file
username: admin
password: netapp123
use_insecure_tls: true # Disable TLS verification when connecting to ONTAP cluster
exporters:
- prometheus

no need to add collectors, since they're already defined in Defaults

example with certificate authentication

See https://github.com/NetApp/harvest/blob/main/docs/AuthAndPermissions.md#using-certificate-authentication

cluster-01:

datacenter: DC-01

addr: 10.0.0.2

auth_style: certificate_auth

ssl_cert: /path/to/cert.pem

ssl_key: /path/to/key.key

ca_cert: /path/to/server.crt

GitHub

harvest/AuthAndPermissions.md at main · NetApp/harvest

Open-metrics endpoint for ONTAP . Contribute to NetApp/harvest development by creating an account on GitHub.

#

these are the harvest.yml and prometheus configuration

#

i have installed the rpm

oak canyon Nov 2, 2022, 11:12 AM

#

Thanks @high sapphire for sharing the details. It seems that you are using prometheus service discovery end point for scraping. Could you confirm if you have started harvest admin cli as mentioned here https://github.com/NetApp/harvest/blob/main/cmd/exporters/prometheus/README.md#enable-http-service-discovery-in-harvest

high sapphire Nov 2, 2022, 11:16 AM

#

hi Rahul,
using the above harvest.yml config when i start the harvest it was not running. Below is the output

#

#

bin/harvest admin start
4:45PM INF github.com/netapp/harvest/v2/cmd/admin/admin.go:68 > Admin node started BasicAuth=true TLS=false listen=:8887

oak canyon Nov 2, 2022, 11:17 AM

#

okay could you share logs for this poller cluster-01. It should be present under /var/log/harvest

high sapphire Nov 2, 2022, 11:19 AM

#

📎 poller_cluster-01.log

oak canyon Nov 2, 2022, 11:20 AM

#

Ok. In harvest.yaml I see that you are using credentials_file param and also user/pass

high sapphire Nov 2, 2022, 11:20 AM

#

i have installed prometheus using the below link
https://www.fosslinux.com/10398/how-to-install-and-configure-prometheus-on-centos-7.htm

FOSS Linux

How to install and configure Prometheus on CentOS 7

Prometheus collects metrics from monitored targets by scrapping HTTP endpoints on the targets. The following tutorial shows how to install Prometheus on CentOS.

oak canyon Nov 2, 2022, 11:21 AM

#

Issue is there with harvest.yaml. If you are using authentication using user/pass then you can remove credential_file

high sapphire Nov 2, 2022, 11:21 AM

#

oak canyon Ok. In `harvest.yaml` I see that you are using credentials_file param and also ...

okay

#

now they are running

#

#

can you help me how can i integrate with grafana

oak canyon Nov 2, 2022, 11:30 AM

#

okay great

#

Could you confirm if you see these metrics in prometheus first?

high sapphire Nov 2, 2022, 11:31 AM

#

yes with different ports

#

oak canyon Nov 2, 2022, 11:32 AM

#

great that is the harvest metrics end point. Could you check the same in prometheus http://localhost:9090

high sapphire Nov 2, 2022, 12:55 PM

#

prometheus is not starting

coarse pollen Nov 2, 2022, 1:04 PM

#

hi @high sapphire does journalctl -u prometheus or systemctl status prometheus say anything?

high sapphire Nov 2, 2022, 1:08 PM

#

systemctl status prometheus
● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2022-11-02 18:35:47 IST; 1s ago
Process: 46927 ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries (code=exited, status=2)
Main PID: 46927 (code=exited, status=2)

coarse pollen Nov 2, 2022, 1:11 PM

#

thanks @high sapphire perhaps one of those paths is wrong? does journalctl -u prometheus give anymore details?

high sapphire Nov 2, 2022, 1:23 PM

#

📎 new_27.txt

coarse pollen Nov 2, 2022, 1:28 PM

#

thanks. Looks like Prometheus started fine at ts=2022-11-02T09:54:49.954Z did you start it via systemctl then or a different way? And now you are trying to start it via systemctl start prometheus ?

#

you could also try to start directly, skipping your service file, to make sure those commands are correct and to verify it starts that way. That means running this directly /usr/local/bin/prometheus
--config.file /etc/prometheus/prometheus.yml
--storage.tsdb.path /var/lib/prometheus/
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries

high sapphire Nov 2, 2022, 1:32 PM

#

after changing the below config, promentheus is starting but i don't know below config was correct
cat /etc/prometheus/prometheus.yml
global:
scrape_interval: 10s

scrape_configs:

job_name: 'harvest'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']

#scrape_configs:

- job_name: 'harvest'

http_sd_configs:

- url: http://localhost:8887/api/v1/sd

#

can you please check and let me know

coarse pollen Nov 2, 2022, 1:33 PM

#

not sure if you installed promtool but try running promtool check config /etc/prometheus/prometheus.yml

#

when you try to start prometheus, it will also log an error if there are mistakes in the prometheus.yml file

high sapphire Nov 3, 2022, 7:24 AM

#

prometheus is working

oak canyon Nov 3, 2022, 7:33 AM

#

Great. Do you see these metrics now in prometheus?

high sapphire Nov 3, 2022, 8:45 AM

#

📎 new_28.txt

#

i am getting the above

#

@oak canyon my grafana is running on the same server

#

how can i import the data

high sapphire Nov 3, 2022, 9:03 AM

#

i am seeing attached metrics in prometheus

high sapphire Nov 3, 2022, 9:22 AM

#

@oak canyon can you help me how to import the dashboard

oak canyon Nov 3, 2022, 9:46 AM

#

Yes, I see that Harvest metrics aren't visible in prometheus. Could you share the output of curl -s 'http://localhost:8887/api/v1/sd'

high sapphire Nov 3, 2022, 9:58 AM

#

curl -s 'http://localhost:8887/api/v1/sd' not getting any thing

oak canyon Nov 3, 2022, 9:58 AM

#

ok could you check if you have started admin cli?

high sapphire Nov 3, 2022, 9:59 AM

#

bin/harvest admin start
3:29PM INF github.com/netapp/harvest/v2/cmd/admin/admin.go:68 > Admin node started BasicAuth=true TLS=false listen=:8887

oak canyon Nov 3, 2022, 10:00 AM

#

okay looks like you have just started it. This needs to be running if you are using prometheus endpoint discovery. If it goes down then prometheus cannt scrape harvest metrics

#

Let's wait for a minute or 2 and then check again the output of curl -s 'http://localhost:8887/api/v1/sd'

high sapphire Nov 3, 2022, 10:16 AM

#

curl -s 'http://localhost:8887/api/v1/sd'
Unauthorized

oak canyon Nov 3, 2022, 10:19 AM

#

Could you share poller log

high sapphire Nov 3, 2022, 10:21 AM

#

📎 poller_landcastle.log

oak canyon Nov 3, 2022, 10:26 AM

#

Thanks. yeah logs look ok

oak canyon Nov 3, 2022, 10:26 AM

#

high sapphire curl -s 'http://localhost:8887/api/v1/sd' Unauthorized

Strange as this url has no authorization

high sapphire Nov 3, 2022, 10:27 AM

#

https is not working

#

you want me to check

oak canyon Nov 3, 2022, 10:27 AM

#

Hmm this endpoint is on http only and not https

#

Could you share screenshot of page from http://localhost:9090/targets

high sapphire Nov 3, 2022, 10:29 AM

#

#

when i check the prometheus below iam seeing
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"
caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target g...Unauthorized"

oak canyon Nov 3, 2022, 10:31 AM

#

I see that you have basic auth enabled for this in harvest.yaml
auth_basic: username: admin password: admin

#

try this curl
curl -u admin:admin 'http://localhost:8887/api/v1/sd'

high sapphire Nov 3, 2022, 10:32 AM

#

curl -u admin:admin 'http://localhost:8887/api/v1/sd'
[{"targets":["172.23.133.100:12990"],"labels":{"__meta_poller":"landcastle"}},{"targets":[":13003"],"labels":{"__meta_poller":"xhdengcdot1"}},{"targets":[":13001"],"labels":{"__meta_poller":"xhdnstorecdot1"}},{"targets":[":13002"],"labels":{"__meta_poller":"xhditcdot1"}}]

oak canyon Nov 3, 2022, 10:32 AM

#

great!

#

now we just need to modify prometheus.yml with the proper auth to scrape this endpoint

#

Could you share prometheus.yml

high sapphire Nov 3, 2022, 10:33 AM

#

cat /etc/prometheus/prometheus.yml
global:
scrape_interval: 10s

scrape_configs:

job_name: harvest
http_sd_configs:
- url: http://localhost:8887/api/v1/sd

oak canyon Nov 3, 2022, 10:36 AM

#

let's add below in this job

#

as below
- job_name: harvest http_sd_configs: - url: http://localhost:8887/api/v1/sd basic_auth: username: admin password: admin

high sapphire Nov 3, 2022, 2:01 PM

#

now i am getting the data

#

can you let me know how to import the grafana

oak canyon Nov 3, 2022, 2:06 PM

#

Great. We will need to import Grafana dashboards first.
let's add Grafana token at this line in harvest config https://github.com/NetApp/harvest/blob/main/harvest.yml#L20
You can generate one from Grafana http://localhost:3000/org/apikeys

#

steps here https://github.com/NetApp/harvest/blob/main/cmd/tools/grafana/README.md#grafana-api-token

high sapphire Nov 4, 2022, 9:54 AM

#

@oak canyon i have created the token and updated the token in harvest

#

now can you let me know how to import dashboard

oak canyon Nov 4, 2022, 9:54 AM

#

Can you run command bin/grafana import --addr localhost:3000

#

that should be run from harvest dir

high sapphire Nov 4, 2022, 10:03 AM

#

error connect: (403 - 403 Forbidden) You'll need additional permissions to perform this action. Permissions needed: folders:read

oak canyon Nov 4, 2022, 10:10 AM

#

could you share screenshot of page http://localhost:3001/org/apikeys

high sapphire Nov 4, 2022, 10:14 AM

#

oak canyon Nov 4, 2022, 10:16 AM

#

hmm editor api key should work fine. Let's try with an admin key? Could you change the key in harvest.yaml to a new api key with admin permissions

high sapphire Nov 4, 2022, 10:23 AM

#

it was working now

#

thank you rahul !

oak canyon Nov 4, 2022, 10:39 AM

#

great!

#

do you see data in grafana now?

high sapphire Nov 4, 2022, 12:15 PM

#

yes

high sapphire Nov 29, 2022, 6:52 AM

#

@coarse pollen @oak canyon , My prometheus services is stopping continusly

#ISSUE WITH HARVESTER

Listen address for Prometheus HTTP service discovery end-point

The endpoint is reachable at SCHEMA://<listen>/api/v1/sd

Should be one of: localhost:port, :port, or ip:port

grafana_api_token: 'aaa-bbb-ccc-ddd'

autosupport_disabled: true

This is an optional poller that provides stats about harvest itself.

To remove it, simply delete the "unix" section

add your cDOT / 7Mode pollers below, example:

no need to add collectors, since they're already defined in Defaults

example with certificate authentication

See https://github.com/NetApp/harvest/blob/main/docs/AuthAndPermissions.md#using-certificate-authentication

cluster-01:

datacenter: DC-01

addr: 10.0.0.2

auth_style: certificate_auth

ssl_cert: /path/to/cert.pem

ssl_key: /path/to/key.key

ca_cert: /path/to/server.crt

- job_name: 'harvest'

http_sd_configs:

- url: http://localhost:8887/api/v1/sd