#New harvest install, harvest.target errors

1 messages · Page 1 of 1 (latest)

meager star
#

I setup an instance of Harvest to test it out and I cant seem to get the pollers to run. I pretty much took the quickstart(https://netapp.github.io/harvest/24.08/quickstart/) and followed it verbatim but I'm getting the error below::

#systemctl status harvest.target
● harvest.target - "NetApp Harvest Meta Service"
Loaded: loaded (/etc/systemd/system/harvest.target; enabled; vendor preset: disabled)
Active: inactive (dead) since Tue 2024-11-05 13:31:56 MST; 3s ago

Nov 05 13:31:56 infstoharlp0001 systemd[1]: Reached target "NetApp Harvest Meta Service".
Nov 05 13:31:56 infstoharlp0001 systemd[1]: Stopped target "NetApp Harvest Meta Service".
Nov 05 13:31:56 infstoharlp0001 systemd[1]: Stopping "NetApp Harvest Meta Service".
Nov 05 13:31:56 infstoharlp0001 systemd[1]: Dependency failed for "NetApp Harvest Meta Service".
Nov 05 13:31:56 infstoharlp0001 systemd[1]: harvest.target: Job harvest.target/start failed with result 'dependency'.

this is my harvest.target file:
#cat /etc/systemd/system/harvest.target
[Unit]
Description="NetApp Harvest Meta Service"
Requires=poller@jamaica.service

[Install]
WantedBy=multi-user.target

and my harvest.yml file::
#cat /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml
Exporters:
prometheus1:
exporter: Prometheus
port_range: 13000-13100

Defaults:
collectors:
- Zapi
- ZapiPerf
- Ems
- Rest
- RestPerf
use_insecure_tls: true

Pollers:
jamaica:
datacenter: DC-01
addr: 172.x.x.x
auth_style: basic_auth
username: userid
password: password
exporters:
- prometheus1

Prometheus and Grafana are running just no data

tawdry wave
#

@meager star Could you run below command from harvest install directory. Let it run for a minute, and then share poller_output.log

./bin/poller --poller jamaica --promPort 13000 2>&1 | tee poller_output.log

You can upload these logs here https://upload.nabox.org/fyga-zogy-sivy

meager star
#

File uploaded

rotund shuttle
#

thanks for the log file, it looks good. With the poller running, from the CLI, are you able to curl the metrics like so: curl -s http://localhost:13000/metrics

#

The poller is collecting metrics as evidenced by lines like these:
grep 'Collected Poller=jamaica' poller_output.log

meager star
#

'Yes I can see data at http://localhost:13000/metrics, but only when the ./bin/poller --poller jamaica --promPort 13000 2>&1 | tee poller_output.log command is running. I am still getting the "dependency failed" What should I look at next?

#systemctl status harvest.target
● harvest.target - "NetApp Harvest Meta Service"
Loaded: loaded (/etc/systemd/system/harvest.target; enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2024-11-06 10:51:00 MST; 51s ago

Nov 06 10:51:00 infstoharlp0001.xxxxx.org systemd[1]: Reached target "NetApp Harvest Meta Service".
Nov 06 10:51:00 infstoharlp0001.xxxxxx.org systemd[1]: Stopped target "NetApp Harvest Meta Service".
Nov 06 10:51:00 infstoharlp0001.xxxxx.org systemd[1]: Stopping "NetApp Harvest Meta Service".
Nov 06 10:51:00 infstoharlp0001.xxxxxx.org systemd[1]: Dependency failed for "NetApp Harvest Meta Service".
Nov 06 10:51:00 infstoharlp0001.xxxxxxx.org systemd[1]: harvest.target: Job harvest.target/start failed with result 'dependency'.

rotund shuttle
#

what about systemctl status "poller*"

meager star
#

#systemctl status "poller*"
● poller@jamaica.service - "NetApp Harvest Poller instance jamaica"
Loaded: loaded (/etc/systemd/system/poller@.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2024-11-06 11:13:39 MST; 36s ago
Process: 969933 ExecStart=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica (code=exited, status=217/USER)
Main PID: 969933 (code=exited, status=217/USER)

Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Main process exited, code=exited, status=217/USER
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 5.
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Start request repeated too quickly.
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 11:13:39 infstoharlp0001.xxxxxxxx.org systemd[1]: Failed to start "NetApp Harvest Poller instance jamaica".

rotund shuttle
#

thanks that helps code=exited, status=217/USER means that the user does not exist. I bet I know the problem, you used the tarball package to install per step #2 and there is a step missing to create the harvest user that step #7 uses in the poller service. Apologizes, we'll fix that in the documentation. You can change User=harvest to User=root if you want to run everything as root, or you can create a harvest user using something like useradd -m harvest

meager star
#

I added the harvest user now getting the errors below::

`#systemctl status harvest.target
● harvest.target - "NetApp Harvest Meta Service"
Loaded: loaded (/etc/systemd/system/harvest.target; enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2024-11-06 11:45:01 MST; 26s ago

Nov 06 11:45:01 infstoharlp0001.xxxxx.org systemd[1]: Reached target "NetApp Harvest Meta Service".
Nov 06 11:45:01 infstoharlp0001.xxxxx.org systemd[1]: Stopped target "NetApp Harvest Meta Service".
Nov 06 11:45:01 infstoharlp0001.xxxxx.org systemd[1]: Stopping "NetApp Harvest Meta Service".
Nov 06 11:45:01 infstoharlp0001.xxxxx.org systemd[1]: Dependency failed for "NetApp Harvest Meta Service".
Nov 06 11:45:01 infstoharlp0001.xxxxx.org systemd[1]: harvest.target: Job harvest.target/start failed with result 'dependency'.

#systemctl list-dependencies harvest.target
harvest.target
● └─poller@jamaica.service

#

#systemctl status "poller*"
● poller@jamaica.service - "NetApp Harvest Poller instance jamaica"
Loaded: loaded (/etc/systemd/system/poller@.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2024-11-06 11:45:01 MST; 2min 27s ago
Process: 993737 ExecStart=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica (code=exited, status=1/FAILURE)
Main PID: 993737 (code=exited, status=1/FAILURE)

Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 5.
Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Start request repeated too quickly.
Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 11:45:01 infstoharlp0001.xxxxxxx.org systemd[1]: Failed to start "NetApp Harvest Poller instance jamaica".
`

rotund shuttle
#

let's check permissions ls -la /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/

meager star
#

#ls -la /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/
total 488
drwxr-xr-x. 10 root root 242 Nov 6 10:47 .
drwxr-xr-x. 5 root root 220 Nov 4 09:53 ..
drwxr-x---. 3 root root 21 Nov 6 10:47 asup
drwxr-xr-x. 2 root root 57 Aug 12 06:00 autosupport
drwxr-xr-x. 2 root root 69 Nov 5 11:51 bin
drwxr-xr-x. 2 root root 24 Aug 12 06:00 cert
drwxr-xr-x. 11 root root 133 Aug 12 06:00 conf
drwxr-xr-x. 7 root root 95 Aug 12 06:00 container
drwxr-xr-x. 4 root root 43 Aug 12 06:00 grafana
-rw-r--r--. 1 root root 392 Nov 5 13:20 harvest.yml
-rw-r--r--. 1 root root 2100 Nov 5 13:19 harvest.yml.orig
-rw-r--r--. 1 root root 11558 Aug 12 06:00 LICENSE
-rw-r--r--. 1 root root 470690 Nov 6 11:12 poller_output.log
-rw-r--r--. 1 root root 1620 Aug 12 06:00 prom-stack.tmpl
-rw-r--r--. 1 root root 1568 Aug 12 06:00 README.md
drwxr-xr-x. 4 root root 89 Nov 5 10:12 service

rotund shuttle
#

the issue now is the poller service is running as the harvest user while the installed files are owned by root. The harvest user won't be able to read those files. We can fix that by changing ownership of those file

#

Let's create a harvest group first
sudo groupadd harvest
Then add the harvest user to that group
sudo usermod -a -G harvest harvest
then change ownership of the files
sudo chown -R harvest:harvest /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/

meager star
#

it looks like the same error::
#groupadd harvest
groupadd: group 'harvest' already exists
[ 9 241106:12:05 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#usermod -a -G harvest harvest
[ 0 241106:12:06 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#chown -R harvest:harvest /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/
[ 0 241106:12:06 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#systemctl start harvest.target
[ 0 241106:12:07 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#systemctl status harvest.target
● harvest.target - "NetApp Harvest Meta Service"
Loaded: loaded (/etc/systemd/system/harvest.target; enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2024-11-06 12:07:12 MST; 11s ago

Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Reached target "NetApp Harvest Meta Service".
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Stopped target "NetApp Harvest Meta Service".
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Stopping "NetApp Harvest Meta Service".
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Dependency failed for "NetApp Harvest Meta Service".
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: harvest.target: Job harvest.target/start failed with result 'dependency'.

#

#ls -la /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/
total 488
drwxr-xr-x. 10 harvest harvest 242 Nov 6 10:47 .
drwxr-xr-x. 5 root root 220 Nov 4 09:53 ..
drwxr-x---. 3 harvest harvest 21 Nov 6 10:47 asup
drwxr-xr-x. 2 harvest harvest 57 Aug 12 06:00 autosupport
drwxr-xr-x. 2 harvest harvest 69 Nov 5 11:51 bin
drwxr-xr-x. 2 harvest harvest 24 Aug 12 06:00 cert
drwxr-xr-x. 11 harvest harvest 133 Aug 12 06:00 conf
drwxr-xr-x. 7 harvest harvest 95 Aug 12 06:00 container
drwxr-xr-x. 4 harvest harvest 43 Aug 12 06:00 grafana
-rw-r--r--. 1 harvest harvest 392 Nov 5 13:20 harvest.yml
-rw-r--r--. 1 harvest harvest 2100 Nov 5 13:19 harvest.yml.orig
-rw-r--r--. 1 harvest harvest 11558 Aug 12 06:00 LICENSE
-rw-r--r--. 1 harvest harvest 470690 Nov 6 11:12 poller_output.log
-rw-r--r--. 1 harvest harvest 1620 Aug 12 06:00 prom-stack.tmpl
-rw-r--r--. 1 harvest harvest 1568 Aug 12 06:00 README.md
drwxr-xr-x. 4 harvest harvest 89 Nov 5 10:12 service

rotund shuttle
#

yes, the dependency message is unexpected, which OS is this and what does this show systemctl status "poller*"?

meager star
#

Oracle Linux 8
#systemctl status "poller*"
● poller@jamaica.service - "NetApp Harvest Poller instance jamaica"
Loaded: loaded (/etc/systemd/system/poller@.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2024-11-06 12:07:12 MST; 9min ago
Process: 1010825 ExecStart=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica (code=exited, status=1/FAILURE)
Main PID: 1010825 (code=exited, status=1/FAILURE)

Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 5.
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Start request repeated too quickly.
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 12:07:12 infstoharlp0001.xxxxxxx.org systemd[1]: Failed to start "NetApp Harvest Poller instance jamaica".

rotund shuttle
#

does this return anything? ps aux | grep poller

meager star
#

#ps aux | grep poller
root 108 0.0 0.0 0 0 ? I< Nov05 0:00 [edac-poller]
root 1024638 0.0 0.0 12216 1192 pts/0 S+ 12:26 0:00 grep --color=auto poller

rotund shuttle
#

We're close. (code=exited, status=1/FAILURE) is fairly generic. Let's make sure everything is up to date with another sudo systemctl daemon-reload then sudo systemctl start harvest.target and systemctl status "poller*" Also try staring outside same as systemctl but directly via cd /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/ then /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica

meager star
#

#systemctl status "poller*"
● poller@jamaica.service - "NetApp Harvest Poller instance jamaica"
Loaded: loaded (/etc/systemd/system/poller@.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2024-11-06 12:31:34 MST; 10s ago
Process: 1029540 ExecStart=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica (code=exited, status=1/FAILURE)
Main PID: 1029540 (code=exited, status=1/FAILURE)

Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 5.
Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: poller@jamaica.service: Start request repeated too quickly.
Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 12:31:34 infstoharlp0001.xxxxxxxxxx.org systemd[1]: Failed to start "NetApp Harvest Poller instance jamaica".

/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica

runs and populates data in http://infstoharlp0001:13000/metrics

rotund shuttle
#

can you paste systemctl cat poller@jamaica.service

meager star
#

#systemctl cat poller@jamaica.service

/etc/systemd/system/poller@.service

[Unit]
Description="NetApp Harvest Poller instance %I"
PartOf=harvest.target
After=network-online.target
Wants=network-online.target

[Service]
User=harvest
Group=harvest
Type=simple
Restart=on-failure
ExecStart=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f %i

[Install]
WantedBy=harvest.target
[ 0 241106:12:38 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]

rotund shuttle
#

anything useful in journalctl -u poller@jamaica?

meager star
#

Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 4.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: Started "NetApp Harvest Poller instance jamaica".
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org harvest[1040989]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org harvest[1040989]: fork/exec bin/poller: no such file or directory
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Scheduled restart job, restart counter is at 5.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: Stopped "NetApp Harvest Poller instance jamaica".
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Start request repeated too quickly.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: poller@jamaica.service: Failed with result 'exit-code'.
Nov 06 12:46:44 infstoharlp0001.xxxxxxxxx.org systemd[1]: Failed to start "NetApp Harvest Poller instance jamaica".

rotund shuttle
#

ah! fork/exec bin/poller: no such file or directory that's the problem

Your poller service file is missing WorkingDirectory=/opt/netapp/harvest/harvest-24.08.0-1_linux_amd64 in the [Service] section

#

Try adding that, then running systemctl daemon-reload then systemctl restart "poller*"

meager star
#

That seemed to get it running ::
#systemctl start harvest.target
[ 0 241106:13:04 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#systemctl status harvest.target
● harvest.target - "NetApp Harvest Meta Service"
Loaded: loaded (/etc/systemd/system/harvest.target; enabled; vendor preset: disabled)
Active: active since Wed 2024-11-06 13:04:22 MST; 3s ago

Nov 06 13:04:22 infstoharlp0001.xxxxxxx.org systemd[1]: Reached target "NetApp Harvest Meta Service".
[ 0 241106:13:04 root@infstoharlp0001 harvest-24.08.0-1_linux_amd64]
#systemctl status "poller*"
● poller@jamaica.service - "NetApp Harvest Poller instance jamaica"
Loaded: loaded (/etc/systemd/system/poller@.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2024-11-06 13:03:19 MST; 1min 27s ago
Main PID: 1053421 (harvest)
Tasks: 16 (limit: 100060)
Memory: 141.4M
CGroup: /system.slice/system-poller.slice/poller@jamaica.service
├─1053421 /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/bin/harvest --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml start -f jamaica
└─1053555 bin/poller --poller jamaica --loglevel 2 --promPort 13000 --config /opt/netapp/harvest/harvest-24.08.0-1_linux_amd64/harvest.yml

Nov 06 13:04:33 infstoharlp0001.xxxxxxxxx.org harvest[1053555]: 2024-11-06T13:04:33-07:00 INF collector/collector.go:595 > Collected Poller=jamaica apiMs=1236 bytesRx=418871 calcMs=2 collector=ZapiPerf:SMB2 exportMs=37 instances=90 instancesExported=90 metrics=31950 metricsExported>
Nov 06 13:04:33 infstoharlp0001.xxxxxxxxx.org harvest[1053555]: 2024-11-06T13:04:33-07:00 INF collector/collector.go:595 > Collected Poller=jamaica apiMs=575 bytesRx=30191 calcMs=0 collector=ZapiPerf:NFSv3 exportMs=37 instances=6 instancesExported=6 metrics=1032 metricsExported=106>

rotund shuttle
#

Glad we got it figured out - there are two documentation gaps that we'll fix

  1. quick guide needs to mention creating the harvest user/group
  2. the poller service file needs to include the workingdirectory
meager star
#

So the user wasn't created, the directory permissions and the workingdirectory. Thank you Chris for your time and help

rotund shuttle
#

apologizes for the documentation bugs! They will be fixed shortly