afternoon i ve recently upgraded harvest | NetApp | Page 1

rain garden Dec 12, 2022, 2:06 PM

#

hi @harsh sequoia which dashboard is missing? any errors in the grafana logs?

harsh sequoia Dec 12, 2022, 2:06 PM

#

All the new 22.11 dashboards

#

I only see my old dashboards

rain garden Dec 12, 2022, 2:14 PM

#

in nabox - the maintenance tab, can you paste your system information like so:

harsh sequoia Dec 12, 2022, 2:14 PM

#

hmm I don't see a right lot in grafana logs

rain garden Dec 12, 2022, 2:15 PM

#

yeah me either 😄 I deleted all of my dashboards and clicked the Reset button. Grafana shows two log lines authing to import

harsh sequoia Dec 12, 2022, 2:15 PM

#

rain garden Dec 12, 2022, 2:17 PM

#

@slow moss can you help with dashboard import problem?

harsh sequoia Dec 12, 2022, 2:17 PM

#

I looked for /harvest/bin/import and don't see anything

harsh sequoia Dec 12, 2022, 2:34 PM

#

Noticed grafana-dashboards Hasn't been touched

#

ooo I've just seen some new ones

#

Looks like they're under Harvest cDot

harsh sequoia Dec 12, 2022, 2:56 PM

#

ok now onto some templating issues

#

most of the dashboards are working. cDOT dashboard errors with

#

No available variables to select in drop downs either

rain garden Dec 12, 2022, 3:08 PM

#

i upgraded Harvest to the same build as you and don't see that. We're using different versions of nabox, but the changelog doesn't imply this was fixed. Does anything change if you reload the current page?

harsh sequoia Dec 12, 2022, 3:10 PM

#

your prometheus is a different version

#

Nothing changes

rain garden Dec 12, 2022, 3:13 PM

#

yep i guess nabox version 3.2b bumped Prom, but shouldn't matter for this. If you click on Configuration > Data sources what is your default ?

#

and if you hover over the first panel in the cdot dashboard and press e to edit, what do you see in the Query Datasource?

harsh sequoia Dec 12, 2022, 3:15 PM

#

graphite

rain garden Dec 12, 2022, 3:16 PM

#

ah! that's the problem, change that to Prometheus

harsh sequoia Dec 12, 2022, 3:17 PM

#

hmm the UI wont let me set the detault

#

I might need to modify the container

rain garden Dec 12, 2022, 3:19 PM

#

that should not be required - here's what my nabox grafana shows for data sources

harsh sequoia Dec 12, 2022, 3:20 PM

#

#

#

i'm logged in as admin btw

#

the actual queries are trying to use prometheus

rain garden Dec 12, 2022, 3:22 PM

#

gotcha - i wonder if the issue is you have added other datasources and the nabox code is not handling that well?

#

and you said the other Harvest cdot dashboards are working fine?

harsh sequoia Dec 12, 2022, 3:23 PM

#

most are, I have an issue where alot of templates that require a table need instant changing to off

#

Example:

#

If instant is on the table will not display any data

rain garden Dec 12, 2022, 3:24 PM

#

what datasource do you see for one of the variables in the cdot dashboard, e.g.

harsh sequoia Dec 12, 2022, 3:26 PM

#

trying to use graphite again

#

I guess I can update them

rain garden Dec 12, 2022, 3:27 PM

#

yep but odd you are hitting this - what if you delete them and smash the Reset button again?

harsh sequoia Dec 12, 2022, 3:29 PM

#

the same

rain garden Dec 12, 2022, 3:30 PM

#

let's see what @potent warren has to say. I don't see that behavior

harsh sequoia Dec 12, 2022, 3:31 PM

#

fwiw I updated the harvest templates before upgrading nabox, I was previously running the beta version

slow moss Dec 13, 2022, 9:53 AM

#

Sorry about that, I'm here

#

@harsh sequoia is that a recent version of NAbox ? There was an issue with default DS that was fixed at some point

#

On recent version the default is Prometheus

#

Well not so recent it dates back 3.0 beta releases

harsh sequoia Dec 13, 2022, 10:07 AM

#

its the latest version of NABox on your site @slow moss

#

But I upgraded from the 3.0 beta version

#

#

any thoughts on what I can do, got multiple dashboards not wanting to load data

slow moss Dec 13, 2022, 10:32 AM

#

Go ahead and edit /usr/local/nabox/files/grafana/provisioning/datasources

#

datasources:
- name: Graphite
  type: graphite
  access: proxy
  orgId: 1
  url: http://graphite/graphite
  isDefault: false
  version: 1
  editable: falseapiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Let me know if that is different from this

harsh sequoia Dec 13, 2022, 10:49 AM

#

I have two files in that location

#

graphite.yaml and prometheus.yaml

#

datasources:
- name: Graphite
  type: graphite
  access: proxy
  orgId: 1
  url: http://graphite/graphite
  isDefault: true
  version: 1
  editable: false```

#


datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090```

#

No is default set in prom one

slow moss Dec 13, 2022, 1:15 PM

#

Ok change it manually then

slow moss Dec 14, 2022, 7:33 AM

#

It'll be fixed next release

harsh sequoia Dec 14, 2022, 2:59 PM

#

ok that fixes most stuff. Have you seen issue with the harvest metadata panel?

#

I don't see metrics for poller_status in prometheus at all. Is this meant to be a valid metric?

rain garden Dec 14, 2022, 3:11 PM

#

hi @harsh sequoia listed under known issues - NaBox uses docker compose to run the pollers which means this issue applies

#

The unix poller is responsible for collecting poller stats (poller_status) in this case

harsh sequoia Dec 14, 2022, 3:15 PM

#

Ah I missed this, thanks @rain garden

#

I'm not sure this is the same issue @rain garden pollers are running as root and have wide open permissions

rain garden Dec 14, 2022, 3:23 PM

#

are you running via nabox?

harsh sequoia Dec 14, 2022, 3:23 PM

#

yeah

rain garden Dec 14, 2022, 3:29 PM

#

Are you running the unix poller? I think it's the same issue since nabox is running Harvest pollers as containers and the unix poller is unable to use the /proc filesystem to see the other pollers. Let me double check

harsh sequoia Dec 14, 2022, 3:30 PM

#

 5186  4182 root     S    1768m  22%   2   0% /netapp-harvest/bin/poller -p guea
 5211  4182 root     S    1633m  20%   3   0% /netapp-harvest/bin/poller -p sbg-
 5187  4182 root     S    1432m  17%   0   0% /netapp-harvest/bin/poller -p guea
 5217  4182 root     S    1432m  17%   2   0% /netapp-harvest/bin/poller -p sbg-
 5204  4182 root     S    1432m  17%   0   0% /netapp-harvest/bin/poller -p sbg-```

rain garden Dec 14, 2022, 3:32 PM

#

Are you running the unix poller?

harsh sequoia Dec 14, 2022, 3:32 PM

#

I think so, its what come pre-packaged with nabox

rain garden Dec 14, 2022, 3:33 PM

#

i don't see it in the list you pasted above

harsh sequoia Dec 14, 2022, 3:34 PM

#

So the only pollers we have are netapp pollers

#

But I would expect one for the underlying docker containers for prom/grafana etc? Sorry I am not familiar with what the unix poller is specifically

rain garden Dec 14, 2022, 3:36 PM

#

the unix poller is written by the harvest team to collect poller metrics https://netapp.github.io/harvest/22.11/configure-unix/

#

But I would expect one for the underlying docker containers for prom/grafana etc?
yes, those are there too. dc ps will show them

harsh sequoia Dec 14, 2022, 3:38 PM

#

Ok so I think this isn't baked into nabox ```gueanetappperf01:~# dc ps
Name Command State Ports

container-exporter /bin/container-exporter -- ... Up 9104/tcp
go-carbon /init/run.sh Up (healthy) 2003/tcp, 2003/udp, 2004/tcp, 7002/tcp, 7003/tcp, 7007/tcp, 8080/tcp
grafana /run.sh Up 3000/tcp
graphite /bin/sh -c /run.sh Up 80/tcp
nabox-admin /docker-entrypoint.sh ngin ... Up 80/tcp
nabox-api python api.py Up 5000/tcp
nabox-harvest python3 harvest.py Up 5000/tcp
nabox-harvest2 python3 harvest.py Up 5000/tcp
prometheus /bin/prometheus --config.f ... Up 9090/tcp
traefik /nabox-traefik-entrypoint. ... Up 0.0.0.0:2003->2003/tcp,:::2003->2003/tcp, 0.0.0.0:443->443/tcp,:::443->443/tcp, 0.0.0.0:80->80/tcp,:::80->80/tcp```

rain garden Dec 14, 2022, 3:39 PM

#

@slow moss do you enable the unix poller for nabox?

slow moss Dec 14, 2022, 4:04 PM

#

unfortunately not. as harvest is running in a container it doesn't have access to the system, but there is a basic dashboard for nabox

#

When you say unix poller it's the generic system poller for prom ?

#

Ah it's not

rain garden Dec 14, 2022, 4:06 PM

#

no it is the harvest built one

slow moss Dec 14, 2022, 4:09 PM

#

Let's say I just enable "Unix" poller, even running in a container I should get some useful infos ?

rain garden Dec 14, 2022, 4:10 PM

#

since you are running all pollers in a single container, yes, it should be to mount the /proc fs and report on each poller (haven't tried it)

slow moss Dec 14, 2022, 4:12 PM

#

I might have to present /proc but it's worth a try

rain garden Dec 14, 2022, 4:14 PM

#

this is why i thought it may work

slow moss Dec 14, 2022, 4:14 PM

#

docker already does it, nice

#

Defaults:
  collectors:
    - Zapi
    - ZapiPerf
    - Rest
  exporters:
    - nabox-prometheus
Exporters:
  nabox-prometheus:
    addr: 0.0.0.0
    exporter: Prometheus
    master: true
Pollers:
  unix:
    addr: localhost
    collectors:
      - Unix

That would do, right ?

rain garden Dec 14, 2022, 4:27 PM

#

the unix poller section looks correct - but if it uses the default exporter which port does it use?

slow moss Dec 14, 2022, 4:28 PM

#

that's the catch. does it accepts promPort, like ontap systems ?

rain garden Dec 14, 2022, 4:28 PM

#

yes

slow moss Dec 14, 2022, 4:28 PM

#

\o/

rain garden Dec 14, 2022, 4:28 PM

#

that's a function of the poller instead of the collector

slow moss Dec 14, 2022, 4:28 PM

#

makes sense

slow moss Dec 15, 2022, 2:47 PM

#

I'm just getting 15 lines from the unix poller

#afternoon i ve recently upgraded harvest

Ok so I think this isn't baked into nabox ```gueanetappperf01:~# dc ps Name Command State Ports

Ok so I think this isn't baked into nabox ```gueanetappperf01:~# dc ps
Name Command State Ports