Harvest cannot collect metrics | NetApp | Page 1

lavish steeple Nov 23, 2022, 5:43 AM

#

After this i am seeing some of the metrics are not getting collected, in the poller logs i am seeing a lot of logs stating like this

36AM ERR collector/zapi.go:431 > skipped instance [dummyname-n06.PSU2 Fan2 Fault]: not found in cache Poller=dummyname collector=Zapi:Sensor
5:36AM ERR collector/zapi.go:431 > skipped instance [dummyname-n06.PSU2 Pwr In OK]: not found in cache Poller=dummyname collector=Zapi:Sensor
5:36AM ERR collector/zapi.go:431 > skipped instance [dummyname-n06.PSU2 Pwr Out OK]: not found in cache Poller=dummyname collector=Zapi:Sensor
5:36AM ERR collector/zapi.go:431 > skipped instance [dummyname-n06.PSU2 FAULT]: not found in cache Poller=dummyname collector=Zapi:Sensor

#

it his happening for all of the collectors disk, volumes,qtree

lavish steeple Nov 23, 2022, 8:08 AM

#

I modified the default.yaml file and added client_timeout: 60 but still no luck, i see the same errors

ebon skiff Nov 23, 2022, 1:06 PM

#

hi @lavish steeple can you send the logs from the troubled poller? https://github.com/NetApp/harvest/wiki/FAQ#how-do-i-share-sensitive-log-files-with-netapp

lavish steeple Nov 23, 2022, 4:06 PM

#

@ebon skiff i have sent the logs let me know when you receive

#

the problem is with both the pollers and it came after i did the upgrade today morning

ebon skiff Nov 23, 2022, 5:21 PM

#

hi @lavish steeple can you run docker exec -it poller-svstaas01 bin/harvest version and let's make sure you are running the latest version. Judging by the line numbers in your shared log file, I'm guessing that you are running 22.08 instead of 22.11

#

is it possible that you only executed the first step about "Migrate Prometheus Docker Volume" and did not do the "To upgrade Harvest:" step listed? https://netapp.github.io/harvest/22.11/install/containers/#upgrade-harvest

lavish steeple Nov 23, 2022, 6:05 PM

#

root@unstaas01-dmz-dh01:~/harvest-22.11.0-1_linux_amd64# docker exec -it poller-svstaas01 bin/harvest version
harvest version 22.11.0-1 (commit cd81cbcf) (build date 2022-11-21T15:37:31+0000) linux/amd64
checking GitHub for latest... you have the latest ✓
root@unstaas01-dmz-dh01:~/harvest-22.11.0-1_linux_amd64#

ebon skiff Nov 23, 2022, 6:18 PM

#

thanks, what's odd is the log you shared has 796,895 lines logged at collector/zapi.go:431, which does not match 22.11 https://github.com/NetApp/harvest/blob/cd81cbcf991cc5a8d42b290b34cff692d48bf809/cmd/collectors/zapi/collector/zapi.go#L431 it does exactly match 22.08 though https://github.com/NetApp/harvest/blob/93db10ac9a153450a99e59239e7d6c7b60fe3b11/cmd/collectors/zapi/collector/zapi.go#L431 you did step #3 "Pull new images and restart your containers like so"?

#

checking some other things on my side

#

let's also take a look at docker image ls and docker inspect poller-svstaas01 feel free to email the inspect if it contains sensitive info

#

seems certain that you aren't running 22.11 I've checked some other lines in your log file and compared with Harvest versions. Let's try this again
docker-compose -f prom-stack.yml -f harvest-compose.yml down
docker pull cr.netapp.io/harvest
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans

lavish steeple Nov 24, 2022, 1:35 AM

#

Interesting it seems to work now, only thing i added in above steps were i deleted the old image before i pulled the new one just to make sure there is nothing old left .Its working now🫡

novel edge Nov 24, 2022, 2:05 AM

#

hmm strange. Could you share the commands ran? All we need is below 2 lines to upgrade harvest. Which image are you using from docker hub or netapp repo?

docker pull cr.netapp.io/harvest # or if using Docker Hub: docker pull rahulguptajss/harvest
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans

lavish steeple Nov 24, 2022, 2:12 AM

#

i am using the one from cr.netapp.io/harvest

#Harvest cannot collect metrics