Hey So my Harvest lab instance isn t | NetApp | Page 1

undone peak Sep 26, 2022, 2:36 PM

#

hey @wintry tangle - 9.11 should be fine, we test with 9.11 too

#

can you share the poller's log file?

wintry tangle Sep 26, 2022, 2:36 PM

#

Location?

undone peak Sep 26, 2022, 2:37 PM

#

depends on how you installed, try /var/log/harvest

wintry tangle Sep 26, 2022, 2:38 PM

#

NAbox

#

Not there. 😦

#

@split cradle ???

undone peak Sep 26, 2022, 2:38 PM

#

ah ok - then https://nabox.org/documentation/troubleshooting/

#

from that page, this should be sufficient dc logs nabox-harvest2 > nabox-harvest2.log

wintry tangle Sep 26, 2022, 2:42 PM

#

pstejska-harvest:/# dc logs
dc: can't open 'logs': No such file or directory
pstejska-harvest:/#

#

:\

undone peak Sep 26, 2022, 2:45 PM

#

hmm does dc ps show you the running containers?

#

you are sshed into nabox?

wintry tangle Sep 26, 2022, 2:45 PM

#

Yep

undone peak Sep 26, 2022, 2:45 PM

#

figured or dc wouldn't be found

wintry tangle Sep 26, 2022, 2:46 PM

#

pstejska-harvest:/# dc ps
dc: can't open 'ps': No such file or directory
pstejska-harvest:/#

undone peak Sep 26, 2022, 2:48 PM

#

are you using a recent version of nabox? I don't use it, sounds like @split cradle will need to chime in after all. Most customers I've helped with nabox, the dc commands on the troubleshooting page work fine, so maybe you're on an older version?

#

do you have this directory? /usr/local/nabox/docker-compose/

wintry tangle Sep 26, 2022, 2:50 PM

#

I'm pretty up to date. This is weird. :\

undone peak Sep 26, 2022, 2:51 PM

#

indeed - what if you try docker-compose ps

wintry tangle Sep 26, 2022, 2:53 PM

#

docker logs works. I'm not getting it to go into a text file though.

undone peak Sep 26, 2022, 2:54 PM

#

progress!! try something like docker-compose logs nabox-harvest2 2>&1 | tee nabox-harvest2.log

wintry tangle Sep 26, 2022, 2:55 PM

#

pstejska-harvest:/# docker-compose logs nabox-harvest2 2>&1 | tee nabox-harvest2.log

    Can't find a suitable configuration file in this directory or any
    parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

pstejska-harvest:/# ls
bin dev lib mnt opt run swap usr
boot etc lost+found nabox-api.log proc sbin sys var
data home media nabox-harvest2.log root srv tmp
pstejska-harvest:/# cat nabox-harvest2.log

    Can't find a suitable configuration file in this directory or any
    parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

pstejska-harvest:/#

undone peak Sep 26, 2022, 2:56 PM

#

cd /usr/local/nabox/docker-compose and try again

#

or you could grab everything with docker-compose logs 2>&1 | tee nabox-harvest2.log

#

thanks Paul - looks like metrics are definitely being gathered
2:51PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=74ms calcD=0s collector=ZapiPerf:WorkloadVolume instances=23 metrics=575 parseD=7ms 2:51PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=375ms calcD=1ms collector=ZapiPerf:WorkloadDetail instances=45 metrics=90 parseD=12ms 2:51PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=258ms calcD=0s collector=ZapiPerf:Workload instances=5 metrics=130 parseD=3ms 2:51PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=475ms calcD=0s collector=ZapiPerf:WorkloadDetailVolume instances=207 metrics=414 parseD=45ms 2:54PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=75ms calcD=0s collector=ZapiPerf:WorkloadVolume instances=23 metrics=575 parseD=13ms 2:54PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=226ms calcD=0s collector=ZapiPerf:WorkloadDetail instances=45 metrics=90 parseD=16ms 2:54PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=121ms calcD=0s collector=ZapiPerf:Workload instances=5 metrics=130 parseD=2ms 2:54PM INF zapiperf/zapiperf.go:663 > Collected Poller=pstejska_vsim apiD=312ms calcD=0s collector=ZapiPerf:WorkloadDetailVolume instances=207 metrics=414 parseD=32ms

#

let's check Prometheus

wintry tangle Sep 26, 2022, 3:02 PM

#

Ok. 10.216.33.135/prometheus if you want

undone peak Sep 26, 2022, 3:03 PM

#

looks promising

#

maybe didn't refresh or wait long enough after default.yaml change?

wintry tangle Sep 26, 2022, 3:05 PM

#

Oh it doesn't work if you specify a volume.

#

Weird. I wonder why?

undone peak Sep 26, 2022, 3:06 PM

#

seems to be?

wintry tangle Sep 26, 2022, 3:06 PM

#

Try test2.

#

undone peak Sep 26, 2022, 3:07 PM

#

i don't see that volume in Prometheus 🙂

wintry tangle Sep 26, 2022, 3:07 PM

#

Weird. But it has throughput/iops/latency at the top.

#

pstejska_vsim::> q workload show
(qos workload show)
Workload Wid Policy Group Vserver Volume LUN Qtree File Path

cifs1-wid21774 21774 medium svm0 cifs1 - - -
qtree1-wid34679
34679 name svm0 fg1 - qtree1 -
test2-wid34121 34121 value-fixed svm0 test2 - - -

#

Oh because it has a qos policy assigned to it.

#

pstejska_vsim::> q workload show -instance -workload test2-wid34121
(qos workload show)

      Workload Name: test2-wid34121
     Workload Class: user-defined
        Workload ID: 34121
  Workload Category: -
  Policy Group Name: value-fixed
Read-ahead Tunables: Default
            Vserver: svm0
             Volume: test2
         Qtree Name: -
           LUN Name: -
          File Path: -
 Maximum Throughput: 15000IOPS,468.8MB/s
 Minimum Throughput: -
           Adaptive: false

Is Constituent Volume: false

undone peak Sep 26, 2022, 3:09 PM

#

ONTAP won't send metrics for that volume if they're all zeroes - is is possible nothing is happening with it and zero-suppression is kicking in?

wintry tangle Sep 26, 2022, 3:09 PM

#

No it's active.

undone peak Sep 26, 2022, 3:09 PM

#

hmm

wintry tangle Sep 26, 2022, 3:09 PM

#

pstejska_vsim::> q s v p s -vserver svm0 -volume test2
(qos statistics volume performance show)
Workload ID IOPS Throughput Latency

-total- - 431 26.88MB/s 235.00us
test2-wid34121 34121 431 26.88MB/s 235.00us

#

I think I found a bug!

#

Apparently if you assign an AQoS policy to it then it stops working.

#

I wonder if it's ONTAP doing that?

undone peak Sep 26, 2022, 3:10 PM

#

ah i know what it is

split cradle Sep 26, 2022, 3:10 PM

#

Login as root. Not admin. For the dc thing not working.

undone peak Sep 26, 2022, 3:10 PM

#

this is a bug we fixed recently

wintry tangle Sep 26, 2022, 3:11 PM

#

split cradle Login as root. Not admin. For the dc thing not working.

This is what I get for not having my coffee yet.

undone peak Sep 26, 2022, 3:11 PM

#

🙂 I think it's this https://github.com/NetApp/harvest/issues/1264

GitHub

ONTAP: Volume - E2E QoS Drilldown show user-defined class volumes ·...

Is your feature request related to a problem? Please describe. Currently the E2E QoS Drilldown panels only show root volumes because the class is being collected Describe the solution you'd...

wintry tangle Sep 26, 2022, 3:11 PM

#

Try updating Harvest and see if it works?

undone peak Sep 26, 2022, 3:12 PM

#

previously we were only collecting autovolume - when you assigned a policy it became user-defined, which we now collect

#

yes nightly has this fixed and was published this morning

wintry tangle Sep 26, 2022, 3:12 PM

#

Yeah that makes sense.

#

Hmm, still not showing up after downloading the nightly.

#

#

That works however.

undone peak Sep 26, 2022, 3:24 PM

#

it will take at least 6 minutes after poller restart since these counters are collected every 3m

wintry tangle Sep 26, 2022, 3:25 PM

#

Oh ok

#

Still nothing.

#

It onyl shows up in qos detail resource latency, but not qos detail volume resource latnecy.

undone peak Sep 26, 2022, 3:33 PM

#

checking

undone peak Sep 26, 2022, 4:52 PM

#

for those eagerly following along at home 🙂 once Paul upgraded to nightly, volumes with an adaptive QOS policy are collected and appear in Grafana

#

humble river Sep 30, 2022, 2:59 PM

#

Hello @undone peak I successfully run Harvest 22.08.0 by harvest-22.08.0-1_linux_amd64.tar.gz in RHEL server. I manually created /var/log/harvest directory and the user (non-root) running harvest has read-write permission, I expect to get harvest log there. I found nothing in this directory when harvest is running. After a period of time, I discovered that harvest log writes to /var/log/messages instead of /var/log/harvest by default, it seems it is different from that mentioned in harvest installation. May I know if I am doing something wrong and how I can set harvest log to write to /var/log/harvest directory? Thank you.

undone peak Sep 30, 2022, 3:03 PM

#

hi @humble river I'll double check and it's true that logging depends on the platform and install method. The RPM we create for RHEL is setup to log to /var/log/harvest but let me check where the native install will pick by default

#

by chance are you setting the HARVEST_LOGS environment variable? Probably not, but otherwise this code should default to /var/log/harvest https://github.com/NetApp/harvest/blob/bddc67ca4c8a841cc44ccafd6008bca97a695cb4/pkg/conf/conf.go#L173

#

what version of RHEL?

humble river Sep 30, 2022, 3:10 PM

#

undone peak hi <@1008252114948083827> I'll double check and it's true that logging depends o...

@undone peak Thanks for your reply. I didn't set HARVEST_LOGS environment variable in my case. It is RHEL 8.4.

undone peak Sep 30, 2022, 3:15 PM

#

looking on a RHEL 8.6 system I found

#

here's what I did: download same build on RHEL 8.6 machine. 2) create group and user named harvest 3) untarred and chowned harvest-22.08.0-1_linux_amd64 3) cd into harvest-22.08.0-1_linux_amd64 4) su harvest 5) bin/harvest start unix 6) check /var/log the harvest directory does not exist because the harvest user does not have permission to create. 7) stop harvest 8) mkdir /var/log/harvest 9) chown -R harvest.harvest /var/log/harvest/ 10) start harvest again as the harvest user. 11) ll /var/log/harvest/ -> -rw-r--r-- 1 harvest harvest 886 Sep 30 11:22 poller_unix.log

#

I can try to dig up some strace commands if you want to try to check why the process is not writing to that file. How are you starting the process? bin/harvest, bin/poller, systemd?

humble river Sep 30, 2022, 3:40 PM

#

undone peak I can try to dig up some `strace` commands if you want to try to check why the p...

By systemd

undone peak Sep 30, 2022, 3:40 PM

#

ah ok, can you paste your systemd service file?

humble river Sep 30, 2022, 3:44 PM

#

undone peak ah ok, can you paste your systemd service file?

A service file for Harvest poller instance or HTTP service discovery?

undone peak Sep 30, 2022, 3:45 PM

#

are the log messages you see in var/log/message admin node related or poller related?

humble river Sep 30, 2022, 3:48 PM

#

Here is the logs in /var/log/messages

📎 messages.txt

undone peak Sep 30, 2022, 3:49 PM

#

interesting implies that came from <daemon.info>

humble river Sep 30, 2022, 3:52 PM

#

BTW, this kind of messages write to /var/log/messages in almost each second and make /var filesystem full. That's why I discover this problem.

undone peak Sep 30, 2022, 3:55 PM

#

those log message are from the pollers so back to your question. can you paste the service file for Harvest poller instance. Are you monitoring many clusters - those logs are INFO level and shouldn't be too chatty, but either way, we can fix. After you share the service file, we could also try running manually and making sure that works correctly permission wise

humble river Sep 30, 2022, 4:01 PM

#

📎 servicefile_poller.txt

#

I am monitoring 20 clusters

undone peak Sep 30, 2022, 4:03 PM

#

for example, if you 1) cd to your harvest install directory 2) su to the correct user if needed 3) run bin/harvest start unix this assumes you have the unix poller in your harvest.yml 4) run ps aux | grep poller and ensure it's running 5) check ls -la /var/log/harvest

#

gotcha so you adapted https://github.com/NetApp/harvest/tree/main/service/contrib

#

do you see anything in your /var/log/messages abourt Failed at step EXEC spawning or any errors related to the pollers? I guess not since it sounds like they're running fine just not logging where you want

humble river Sep 30, 2022, 4:16 PM

#

undone peak for example, if you 1) cd to your harvest install directory 2) su to the correct...

I did what you mentioned, I found the log can write to /var/log/harvest now

undone peak Sep 30, 2022, 4:16 PM

#

I've got an idea, one sec

undone peak Sep 30, 2022, 4:35 PM

#

ok I see what's going on. try editting your service file and add the --logtofile argument to the start command like so:
ExecStart=/appl/san/harvest2/harvest-22.08.0/bin/harvest --config /appl/san/harvest2/harvest-22.08.0/harvest.yml start --logtofile -f %i

#

when using -f harvest logs to the foreground, which sysout, which systemd is sending to the journal and messages in your case

humble river Sep 30, 2022, 4:45 PM

#

undone peak ok I see what's going on. try editting your service file and add the `--logtofil...

Yes, adding --logtofile argument to the start command, I can find the poller log writes to /var/log/harvest directory now

#

@undone peak Thank you very much for diagnosis and solve this issue

undone peak Sep 30, 2022, 4:47 PM

#

you bet, we'll work to improve this

humble river Sep 30, 2022, 4:50 PM

#

May I know if you will update it in creating service file for poller instance?

#

https://github.com/NetApp/harvest/tree/main/service/contrib

GitHub

harvest/service/contrib at main · NetApp/harvest

Open-metrics endpoint for ONTAP . Contribute to NetApp/harvest development by creating an account on GitHub.

undone peak Sep 30, 2022, 4:53 PM

#

yes, that's one of the places I'll change. I want to figure out if this is a RHEL only issue. Typically you want to log to stdout so it is logged to the journal. What is surprising is it's also logging to messages. My guess is journald on RHEL8 is setup to do both, but want to double check before I update the example.

humble river Sep 30, 2022, 4:57 PM

#

undone peak yes, that's one of the places I'll change. I want to figure out if this is a RHE...

I see. Again, thanks for helping to solve the problem on this issue. Cheer.

#Hey So my Harvest lab instance isn t