Native Harvest: How to change log file location? | NetApp | Page 1

dull escarp Nov 3, 2025, 7:52 AM

#

Hi all,

we used this guide to setup our env: https://netapp.github.io/harvest/25.08/quickstart/
How I can activate harvest-log file? With default setup I don't see any log /var/log/harvest

Thank you and Regards
Luk

#

Harvest is running on a User which don't have access to /var/log
Can I set another log-file location ?

#

Native Harvest: How to enable log file for non-root user?

#

Native Harvest: How to change log file location?

main axle Nov 3, 2025, 8:11 AM

#

Can you try setting an environment variable HARVEST_LOGS to a directory where the harvest user has access? Restart Harvest after that and see if it resolves the issue.

dull escarp Nov 3, 2025, 8:12 AM

#

main axle Can you try setting an environment variable `HARVEST_LOGS` to a directory where ...

in my harvest.yml ?

main axle Nov 3, 2025, 8:13 AM

#

No. Just an env variable

#

at OS level

dull escarp Nov 3, 2025, 8:17 AM

#

set, but no logs are going in

main axle Nov 3, 2025, 8:17 AM

#

hmm ok let me check in code

#

Can you also add an argument for harvest start command --logtofile
You can also add HARVEST_LOGS with custom dir path i like below in service file like below

Environment="HARVEST_LOGS=/opt/harvest/logs"
ExecStart=${HARVEST_INSTALL_PATH}/harvest-${HARVEST_VERSION}-1_linux_amd64/bin/harvest --config ${HARVEST_INSTALL_PATH}/harvest-${HARVEST_VERSION}-1_linux_amd64/harvest.yml start --logtofile -f %i

dull escarp Nov 3, 2025, 8:38 AM

#

Thank you, I will try it and let you now.
Other question, is there also a way to monitor the harvest service itself?

main axle Nov 3, 2025, 8:42 AM

#

Do you mean via systemd? or via some other means?

main axle Nov 3, 2025, 8:59 AM

#

Added these steps via PR
https://github.com/NetApp/harvest/pull/3993/files

solemn raft Nov 3, 2025, 9:15 AM

#

Hi, i'm working with @dull escarp and responsible for Harvest. We tried your approach with HARVEST_LOGS env variable but it does not work. [Unit]
Description="NetApp Harvest Poller instance %I"
PartOf=harvest.target
After=network-online.target
Wants=network-online.target

[Service]
User=rh18
Group=scripts
Type=simple
Restart=on-failure
Environment="HARVEST_LOGS=/services/netapp/harvest/logs"
WorkingDirectory=/services/netapp/harvest/harvest
Environment="HARVEST_LOGS=/services/netapp/harvest/harvest/log"
ExecStart=/services/netapp/harvest/harvest/bin/harvest --config /services/netapp/harvest/harvest/harvest.yml start --logtofile -f %i

[Install]
WantedBy=harvest.target
root@mcosrv2253:/services/netapp/harvest# ll /services/netapp/harvest/logs
total 0

main axle Nov 3, 2025, 9:17 AM

#

Could you share permissions for dir /services/netapp/harvest/logs

solemn raft Nov 3, 2025, 9:19 AM

#

SORRY, i'm a idiot, it runs, just watched in the wrong directory. Thanks for the help

main axle Nov 3, 2025, 9:19 AM

#

I see there are 2 env variables in file
Environment="HARVEST_LOGS=/services/netapp/harvest/logs"

main axle Nov 3, 2025, 9:19 AM

#

solemn raft SORRY, i'm a idiot, it runs, just watched in the wrong directory. Thanks for the...

cool. Pls remove duplicate dir from service file.

dull escarp Nov 3, 2025, 9:20 AM

#

main axle Do you mean via systemd? or via some other means?

to have any log in case of harvest systemd crash

main axle Nov 3, 2025, 9:23 AM

#

You can keep a watch on poller status via systemctl status "poller@*"

solemn raft Nov 3, 2025, 9:30 AM

#

We have the strange situation that everything runs fine for a couple of days and suddenly ALL pollers are gone (we have 19 actually, Ontap and StorageGRID, all REST based), and we don't know why all pollers are gone.

main axle Nov 3, 2025, 9:31 AM

#

What is your Harvest version?

solemn raft Nov 3, 2025, 9:32 AM

#

I suspect in one case it was the upgrade of a StorageGRID Admin Node. But why are then all pollers exiting and not only the one targeting to these admin node?

#

The newest 25.08.1

main axle Nov 3, 2025, 9:34 AM

#

All pollers of harvest are independent. Issue in 1 poller should not impact others. Could you check if there are any OOM erros?

dmesg | grep -i "killed process"
How much memory do you have available? free -h
also disk space?

solemn raft Nov 3, 2025, 9:36 AM

#

root@mcosrv2253:/services/netapp/harvest# free -h
total used free shared buff/cache available
Mem: 62Gi 39Gi 3.2Gi 16Mi 20Gi 22Gi
Swap: 4.0Gi 2.0Mi 4.0Gi

#

Diskspace and memory are not the problem, and there is nothing in dmesg output regarding "killed process".

main axle Nov 3, 2025, 9:38 AM

#

Ok. We'll need to check logs then. See if logs are captured in journal?

journalctl -u "poller*" --no-pager -f

solemn raft Nov 3, 2025, 9:39 AM

#

root@mcosrv2253:/services/netapp/harvest# journalctl -u "poller*" --no-pager -f
-- Logs begin at Mon 2025-07-28 17:02:24 CEST. --
Nov 03 10:21:35 mcosrv2253 harvest[3950105]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950090]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950115]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950100]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950119]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950190]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950116]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950217]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950203]: starting in foreground, enter CTRL+C or close terminal to stop poller
Nov 03 10:21:35 mcosrv2253 harvest[3950207]: starting in foreground, enter CTRL+C or close terminal to stop poller

#

yep, lets wait until i crashes again, then i can check the logs. Thanks so far.

main axle Nov 3, 2025, 9:40 AM

#

sure

main axle Nov 3, 2025, 10:36 AM

#

solemn raft yep, lets wait until i crashes again, then i can check the logs. Thanks so far.

You can see all journalctl logs via command if that gives any pointers on this.

journalctl -u "poller*"

dull escarp Nov 4, 2025, 9:32 AM

#

Poller crashed again, which Log-files you need to take a look on it ?

main axle Nov 4, 2025, 9:35 AM

#

Let's check poller logs and journalctl logs journalctl -u "poller*"

Could you upload them @ https://upload.nabox.org/dece-cafa-fyxa
Also check output of

systemctl status "poller*"

dull escarp Nov 4, 2025, 9:47 AM

#

main axle Let's check poller logs and journalctl logs ```journalctl -u "poller*"``` Could...

thank you, files are uploaded

main axle Nov 4, 2025, 9:48 AM

#

Thanks. Could you share Harvest poller logs as well

dull escarp Nov 4, 2025, 9:49 AM

#

sure, wait

#

are uploaded

main axle Nov 4, 2025, 9:58 AM

#

As per logs, poller process received SIGTERM . It's something outiside pollers which is causing it to shutdown

time=2025-11-03T22:22:31.812+01:00 level=INFO source=poller.go:680 msg="caught signal" Poller=mcistr0100 signal=terminated

dull escarp Nov 4, 2025, 9:59 AM

#

network timeouts possible?

main axle Nov 4, 2025, 9:59 AM

#

Doesn't seem like it.

dull escarp Nov 4, 2025, 10:00 AM

#

ok that`s good. Do you see any chance to figure out what our issue is?

main axle Nov 4, 2025, 10:01 AM

#

Checking. It is possible that systemd only is stopping all pollers. It could be that 1 poller process is not able to run so systemd tries to restart all.

dull escarp Nov 4, 2025, 10:04 AM

#

Thank you very much for your time!

main axle Nov 4, 2025, 10:04 AM

#

looks like poller mspstr0116 is failing. Is it something new which you have added?

#

And since adding that you are noticing crashes?

dull escarp Nov 4, 2025, 10:05 AM

#

we added all our clusters at same time

main axle Nov 4, 2025, 10:06 AM

#

If you check poller_mspstr0116.log
It cann't run due to permission issues. Could you fix its permissions?

#

I see its logs only starting from time=2025-11-03T10:21:35.688+01:00

dull escarp Nov 4, 2025, 10:07 AM

#

which kind of permissions do you mean?

main axle Nov 4, 2025, 10:08 AM

#

time=2025-11-03T10:22:04.241+01:00 level=WARN source=poller.go:861 msg="init collector-object" Poller=mspstr0116 error="StatusCode: 401, Error: auth failed, Message: 401 Unauthorized, API: /api/cluster?fields=%2A&return_records=true" collector=RestPerf object=WorkloadVolume

dull escarp Nov 4, 2025, 10:08 AM

#

so issue with User in ONTAP ?

main axle Nov 4, 2025, 10:08 AM

#

Yes

dull escarp Nov 4, 2025, 10:08 AM

#

ok, I check it

main axle Nov 4, 2025, 10:09 AM

#

https://netapp.github.io/harvest/25.08/prepare-cdot-clusters/#ontap-cli

#

Let me check if we can also fix something in systemd behaviour so that it doesn't restart all pollers when 1 crashes

#

once permissions issues are fixed. That should solve the problem.

#

You can also change harvest.target file at location /etc/systemd/system/harvest.target

#

and change Requires to Wants

#

#

After making this change. Execute below commands

systemctl daemon-reload
systemctl restart harvest.target
systemctl status harvest.target

dull escarp Nov 4, 2025, 10:28 AM

#

Thank you @main axle in fact the harvest user was missing on our mspstr0116 cluster!

main axle Nov 4, 2025, 10:28 AM

#

Makes sense. Let's see how system behaves after above fixes.

dull escarp Nov 4, 2025, 10:29 AM

#

Thank you, I keep my eyes on it and let you know

main axle Nov 4, 2025, 10:29 AM

#

sure

main axle Nov 5, 2025, 6:47 AM

#

main axle and change `Requires` to `Wants`

FYI. This fix will be added to next Harvest release.
https://github.com/NetApp/harvest/pull/4002/files

dull escarp Nov 10, 2025, 9:18 AM

#

@main axle is runs stable now. Thank you for your help!

main axle Nov 10, 2025, 9:29 AM

#

@dull escarp We will do next Harvest release 25.11 probably today/tomorrow. You can wait for that.

#Native Harvest: How to change log file location?