harvest.yml config for collection metrics | NetApp | Page 1

warped fable May 5, 2023, 3:06 PM

#

In an attempt to use nfsmetrics and the new S3 object store dashboards, I've run into a bit fo a snag that is due to my own ignorance.

When the message references harvest.yml needing to be modified to enable rest collectors, specifically... Where is it.

Using the ova, I've located one within /opt/harvest2-conf but upon further digging within it I see that the collectors for the array in question are already listed as

Collectors: 
  - Zapi
  - ZapiPerf
  - Rest 
  - Ems

Am I looking at the wrong one or am I missing the point entirely?

wintry kayak May 5, 2023, 3:11 PM

#

@warped fable That is the correct file. If collectors are enabled, there is no need to make any changes.

warped fable May 5, 2023, 3:11 PM

#

They are, but those dashboards say otherwise 🙂

Does the fact that the Zapi is set higher than rest prioritize that?

stark seal May 5, 2023, 3:12 PM

#

Yes, details here when both are enabled
https://github.com/NetApp/harvest/blob/main/docs/architecture/rest-strategy.md#can-i-use-the-rest-and-zapi-collectors-at-the-same-time

warped fable May 5, 2023, 3:14 PM

#

Note to self: try it first then ask silly questions after the fact.

stark seal May 5, 2023, 3:14 PM

#

it happens

#

hopefully that doc link clears it up too

warped fable May 5, 2023, 3:34 PM

#

Odd.
Rearranging and even commenting out ZAPI does no good.
Reset dashboards and restarted all containers.

Chicken and the egg scenario for me, I'm collecting stats for a case involving performance (extreme CPU utilization) thats impinging the update to 9.12 or I'd just schedule that to avoid this entire Zapi mess lol

wintry kayak May 5, 2023, 3:55 PM

#

okay let's focus on s3 dashboard metrics. Given s3 dashboard/templates are only available for Rest, so changing or commenting out Zapi will not make any difference. Could you share harvest logs to us https://github.com/NetApp/harvest/wiki/FAQ#how-do-i-share-sensitive-log-files-with-netapp. I suspect, you may be hitting permission issues with Rest APIs.

warped fable May 5, 2023, 4:18 PM

#

Permissions, let me check that the harvest user has access to REST before I waste more of your time Rahul.

wintry kayak May 5, 2023, 4:25 PM

#

sure we want to check that user has read access to /api

warped fable May 5, 2023, 5:19 PM

#

Harvest account keeps getting promoted to auth.
Confirmed with admin using admin privs with url https://<host>/API stating API not found, which I presume is fine.

#

But harvest though, I'll recreate a test user I guess and new roles to see if that's fine.

#

Created new user new roles, same issue.
My ansibleuser though which actively uses rest has 0 issues with the same test.

stark seal May 5, 2023, 5:33 PM

#

hi @warped fable can you paste what you're trying and the error?

warped fable May 5, 2023, 5:55 PM

#

Sure.
Then attempting to hit https://<host>/API with the NetApp-harvest user, it just prompts for credentials again.
Not much of an error 🙂

When using the harvest user, harvest-test that I made to test if the role was goobered up, both get waived off.

I do have an Ansible account that I also use rest calls for that does work when prompted for credentials.

#

Box, prompt, etc.

#

Anyway, I'll scrape and send the log.

stark seal May 5, 2023, 6:00 PM

#

thanks. let's check the permissions, something like this vserver services web access show -role $role and security login rest-role show -vserver $admin-vserer -role $role

warped fable May 5, 2023, 6:08 PM

#

It appears to be missing a rest-role.
Create with the read-only role or is there one preferred for harvest?

stark seal May 5, 2023, 6:10 PM

#

read-only role should be fine. Creating with SystemManager works too

warped fable May 5, 2023, 6:41 PM

#

Jockying with roles no joy.
Going to upload logs.

#

Sent.

stark seal May 5, 2023, 6:51 PM

#

received

zenith otter May 5, 2023, 11:00 PM

#

Mmm, zapi is listed before rest in NAbox. is that a problem ?

warped fable May 6, 2023, 1:12 PM

#

Within my end? I made many changes to the config yesterday and eventually lost all reporting when rest was the only collection method 😛

wintry kayak May 8, 2023, 7:52 AM

#

@warped fable Based on the logs, it appears that there are no records available for NFS clients or S3 objects in your clusters. To investigate further, could you run the following commands and check if they provide any records?

curl --location --request GET 'https://IP_ADDRESS/api/protocols/nfs/connected-clients?return_records=true&fields=*' -k --user USER:PASS

curl --location --request GET 'https://IP_ADDRESS/api/protocols/s3/buckets?return_records=true&fields=*' -k --user USER:PASS

warped fable May 8, 2023, 11:52 AM

#

Buckets. I'd agree, investigating playing with them and the eyes the me off.
NFS though during that period I had a vol pushing 40k ops.
I'll check the output from those commands soon.

warped fable May 8, 2023, 1:28 PM

#

wintry kayak <@158942885247451136> Based on the logs, it appears that there are no records a...

User not authorized when using my harvest account, only authorized when demoing with my other account /w rest access (Ansible.)

#

I'm willing to state it's definitely an account issue.
Do you want a scrape of the role and how it's configured?

stark seal May 8, 2023, 1:32 PM

#

yes please

warped fable May 9, 2023, 7:57 PM

#

Apparently my output is blocked by discord

#

Hmm

hardy knoll May 9, 2023, 7:57 PM

#

I'll post it in here for you

warped fable May 9, 2023, 7:58 PM

#

https://pastebin.com/Z5BUt8Kd

Pastebin

CLUSTERVSERVER::> security login role show -vserver h1pdnetapp03 -r...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

#

Pastebin

#

Redacted

hardy knoll May 9, 2023, 7:58 PM

#

           Role          Command/                                      Access
Vserver    Name          Directory                               Query Level
---------- ------------- --------- ----------------------------------- --------
CLUSTERVSERVER harvest2-role DEFAULT                                     readonly
                           cluster                                     readonly
                           lun                                         readonly
                           network interface                           readonly
                           qos workload delete                         readonly
                           qos workload modify                         readonly
                           qos workload show                           readonly
                           security                                    readonly
                           snapmirror                                  readonly
                           statistics                                  readonly
                           storage aggregate                           readonly
                           storage disk                                readonly
                           storage encryption disk                     readonly
                           storage shelf                               readonly
                           system health status show                   readonly
                           system health subsystem modify              readonly
                           system health subsystem show                readonly
                           system node                                 readonly
                           version                                     readonly
                           volume                                      readonly
                           vserver                                     readonly
             netapp-harvest-role DEFAULT                               none
                                 cluster identity modify               readonly
                                 cluster identity show                 readonly
                                 cluster modify                        readonly
                                 cluster show                          readonly
                                 lun create                            readonly
                                 lun modify                            readonly
                                 lun show                              readonly
                                 network interface create              readonly
                                 network interface delete              readonly
                                 network interface modify              readonly
                                 network interface show                readonly
                                 qos workload delete                   readonly
                                 qos workload modify                   readonly
                                 qos workload show                     readonly
                                 statistics                            readonly
                                 system node modify                    readonly
                                 system node show                      readonly
                                 version                               readonly
             netapp-harvest-role1 DEFAULT                              readonly
41 entries were displayed.

warped fable May 9, 2023, 7:59 PM

#

Or drew with the save

warped fable May 9, 2023, 8:11 PM

#

hardy knoll ```CLUSTERVSERVER::*> security login role show -vserver CLUSTERVSERVER -role *ha...

To note, Ive tried all three.

stark seal May 10, 2023, 12:04 PM

#

thanks @hardy knoll and @warped fable , Jesse those roles look fine for zapi, but if I recall correctly, the problem you are having is with REST not ZAPI. If that's correct, you want to use rest-role instead of role. Something like this security login rest-role show -vserver $vserver -role *harvest*

warped fable May 10, 2023, 1:24 PM

#

Yeah the only entry is for the cluster vserver
Name: netapp-harvest-role1
API: /api
Access: readonly

stark seal May 10, 2023, 1:32 PM

#

and does the user in question have an application with http entry associated with that rest role? Something like this

warped fable May 10, 2023, 1:45 PM

#

I had HTTP in there for awhile, I'll add it back and do more testing

warped fable May 10, 2023, 3:41 PM

#

stark seal and does the user in question have an application with http entry associated wit...

That appears to have done the trick, and some adjusting to the priority of api's to talk to for certain information.

stark seal May 10, 2023, 3:41 PM

#

glad to hear!

warped fable May 10, 2023, 3:42 PM

#

I will mention that the nabox website for stand up doesn't include the HTTP application for access 🙂
More than happy to have been a guinea pig

stark seal May 10, 2023, 3:44 PM

#

@zenith otter will update. Thanks for raising

zenith otter May 10, 2023, 3:46 PM

#

Good point !

#

Web site fixed

warped fable May 10, 2023, 4:13 PM

#

partyparrot

#

Thanks guys

urban timber Jul 7, 2023, 10:26 AM

#

hi all, i have the same problem with this dashboard s3, the same eror appear : "1. On Harvest, enable the Rest collector for OntapS3 template in your harvest.yml config for configuration metrics"

#

I followed the whole post, checked that the user created 'harvest2' has access to the rest part via http and enabled ontapi, changed the position of the rest collector in the harvest.yml file.
But I am probably missing something

wintry kayak Jul 7, 2023, 10:28 AM

#

@urban timber Could you please send us the log files for the poller that is using REST to ng-harvest-files@netapp.com via email?

urban timber Jul 7, 2023, 10:43 AM

#

ok, i will send you now. thanks a lot!

wintry kayak Jul 7, 2023, 10:43 AM

#

Thanks

#

@urban timber I see that s3 data is being collected by Rest collector based on the logs for 1 poller. Could you please verify if metrics ontaps3_labels exists in Prometheus?

nabox-harvest2 | 2023-07-07T10:34:50Z INF collector/collector.go:483 > Collected Poller=XXXX apiMs=925 calcMs=0 collector=Rest:OntapS3 instances=27 metrics=162 parseMs=0 pluginMs=0

#

Also I see that you are running 22.11 Harvest version. Is it possible to upgrade to 23.05?

urban timber Jul 7, 2023, 11:05 AM

#

i'll do it

#

the metrics exist

#

in the graph tab i can also see that if i execute it

wintry kayak Jul 7, 2023, 11:11 AM

#

okay we can try debugging panels. For 23.05, graph is like this

#

let's upgrade and import new graphs?

urban timber Jul 7, 2023, 11:12 AM

#

i do it now

#

so, i have this panel now

wintry kayak Jul 7, 2023, 11:12 AM

#

okay good

#

and this is still empty?

urban timber Jul 7, 2023, 11:12 AM

#

the first. bucket drill down works fine

#

the others say me

#

nothing

wintry kayak Jul 7, 2023, 11:13 AM

#

okay so Bucket Used Size is empty? and bucket permissions?

urban timber Jul 7, 2023, 11:13 AM

#

no data

#

size and percent empty

#

permission no data

#

and in the corner to the left i have an i

#

that say

#

You must enable the Rest collector in your harvest.yml config

wintry kayak Jul 7, 2023, 11:15 AM

#

okay let's focus on bucket permissions first. It is a new panel added in 23.05 only and uses a different template
https://github.com/NetApp/harvest/blob/main/conf/rest/9.7.0/ontap_s3_policy.yaml

#

do you see ontaps3_policy_labels in prometheus?

urban timber Jul 7, 2023, 11:16 AM

#

no

wintry kayak Jul 7, 2023, 11:19 AM

#

okay if you run below curl. Replace user, pass and cluster_ip as for poller. Do you see any records?

curl -s -k -u user:pass 'https://cluster_ip/api/private/cli/vserver/object-store-server/bucket/policy'

#

for used size and size %, Could you try below url and see if fields logical_used_size,size are available? What is the ONTAP version?

curl -s -k -u user:pass 'https://cluster_ip/api/protocols/s3/buckets?fields=logical_used_size,size

urban timber Jul 7, 2023, 11:23 AM

#

curl works fine

wintry kayak Jul 7, 2023, 11:24 AM

#

okay could you share latest logs from 23.05 to same email id

urban timber Jul 7, 2023, 11:25 AM

#

version is 9.11.1P8

wintry kayak Jul 7, 2023, 11:25 AM

#

okay that should work

urban timber Jul 7, 2023, 11:25 AM

#

yes, i will share you after uopgrade

#

container is restarting

wintry kayak Jul 7, 2023, 11:25 AM

#

okay yeah let's check once if after upgrade data is visible.

urban timber Jul 7, 2023, 11:26 AM

#

it's been restarting for 5 minutes. does it usually take that long that you know?

wintry kayak Jul 7, 2023, 11:27 AM

#

It should not

#

you may check logs

urban timber Jul 7, 2023, 11:30 AM

#

it seems tha the upgrade via nabox doesn't do well

wintry kayak Jul 7, 2023, 11:31 AM

#

hmm did u use the UI for upgrade?

urban timber Jul 7, 2023, 11:32 AM

#

I'll fix this for a bit and come back

wintry kayak Jul 7, 2023, 11:32 AM

#

okay yeah make sure that you use the correct binary.https://github.com/NetApp/harvest/releases/download/v23.05.0/harvest-23.05.0-1_linux_amd64.tar.gz

urban timber Jul 7, 2023, 11:36 AM

#

ok done

#

so

wintry kayak Jul 7, 2023, 11:36 AM

#

okay great

#

let's check in few mins if dashboard has the metrics

urban timber Jul 7, 2023, 11:38 AM

#

at the moment 'permissions' is populated

wintry kayak Jul 7, 2023, 11:38 AM

#

nice!

urban timber Jul 7, 2023, 11:38 AM

#

we have always the information related to "enable the Rest collector for OntapS3..."

wintry kayak Jul 7, 2023, 11:40 AM

#

Grafana dashboards are static in nature so that will stay unless you collapse that panel or remove it.

urban timber Jul 7, 2023, 11:41 AM

#

ok

#

at the moment size and percent are still empty

wintry kayak Jul 7, 2023, 11:51 AM

#

okay lets try this curl command. Do you see logical_used_size and size returned?

curl -s -k -u user:pass 'https://cluster_ip/api/protocols/s3/buckets?fields=logical_used_size,size

urban timber Jul 7, 2023, 11:52 AM

#

yes

wintry kayak Jul 7, 2023, 11:53 AM

#

okay could u share response of below curl and latest logs over email?

curl -s -k -u user:pass 'https://cluster_ip/api/protocols/s3/buckets?fields=*

#

Do you see ontaps3_logical_used_size and ontaps3_size in prometheus?

urban timber Jul 7, 2023, 11:56 AM

#

yes i can see and execute both

wintry kayak Jul 7, 2023, 11:56 AM

#

okay great

#

then below panel should have the data

#

it uses metric ontaps3_logical_used_size

urban timber Jul 7, 2023, 11:57 AM

#

i will send you the log and the txt with the output of curl

wintry kayak Jul 7, 2023, 11:57 AM

#

not needed currently if prometheus has the data

urban timber Jul 7, 2023, 11:58 AM

#

ok

wintry kayak Jul 7, 2023, 11:58 AM

#

let's debug Buckets used Size panel in grafana

urban timber Jul 7, 2023, 11:58 AM

#

so only the logs?

wintry kayak Jul 7, 2023, 11:58 AM

#

no i mean logs or curl response is not needed currently as you have the data in prometheus

urban timber Jul 7, 2023, 11:58 AM

#

ok

wintry kayak Jul 7, 2023, 11:58 AM

#

We now need to check why grafana is not able to show this in panels

#

may be refresh the dashboard?

urban timber Jul 7, 2023, 11:59 AM

#

i'm tryng to inspect the dashboard

#

bucket used size

wintry kayak Jul 7, 2023, 11:59 AM

#

yes please

urban timber Jul 7, 2023, 11:59 AM

#

data is still in "loading"

wintry kayak Jul 7, 2023, 11:59 AM

#

okay makes sense.

#

let's refresh the browser once?

urban timber Jul 7, 2023, 12:00 PM

#

yes

wintry kayak Jul 7, 2023, 12:04 PM

#

@urban timber How many records do you have in prometheus if you run query count(ontaps3_logical_used_size)

#

I believe that you have lots of records which grafana is taking time to load.

urban timber Jul 7, 2023, 12:06 PM

#

not so many, we have 27 buckets

wintry kayak Jul 7, 2023, 12:07 PM

#

okay that should be fine then

urban timber Jul 7, 2023, 12:07 PM

#

yes fine and fast.. maybe 😅

wintry kayak Jul 7, 2023, 12:08 PM

#

indeed. Do these panels load now?

urban timber Jul 7, 2023, 12:08 PM

#

not yet

wintry kayak Jul 7, 2023, 12:08 PM

#

Strange

#

Do you see this loading issue with other panels in other dashboards? I want to rule out that it's not a network issue.

#

I think you are already inspecting the panel. I would check data panel.

urban timber Jul 7, 2023, 12:16 PM

#

otrher dashboards works fine

#

#

volume for example

#

only s3 dashboard has this issue

#

wintry kayak Jul 7, 2023, 12:24 PM

#

okay we can try creating or cloning these panels a fresh manually in this dashboard?

urban timber Jul 7, 2023, 12:25 PM

#

yes ok

#

how we can do it?

wintry kayak Jul 7, 2023, 12:28 PM

#

Also if you could share screenshot of one of the these panels to see if query is not modified in the panels. Below as per my system

urban timber Jul 7, 2023, 12:30 PM

#

query is empty

wintry kayak Jul 7, 2023, 12:30 PM

#

that is the problem

#

Could you paste ontaps3_logical_used_size{datacenter=~"$Datacenter",cluster=~"$Cluster",svm=~"$SVM",bucket=~"$Bucket"} in query

urban timber Jul 7, 2023, 12:31 PM

#

but if if i do "explore"

#

#

it works fine

wintry kayak Jul 7, 2023, 12:31 PM

#

Yeah let's edit the panel and put the query. Due to mising query you don't see data

#

or you can directly import the dashboard from json

https://raw.githubusercontent.com/NetApp/harvest/release/23.05.0/grafana/dashboards/cmode/s3ObjectStorage.json

stark seal Jul 7, 2023, 12:33 PM

#

@urban timber your screenshot shows that Prometheus has the data, which is great. The Grafana dashboard needs a valid query so it can pull that data from Prometheus and graph it

urban timber Jul 7, 2023, 12:36 PM

#

ok i put the query inside in metrrics browser

#

but it doesn't work anymoe

wintry kayak Jul 7, 2023, 12:38 PM

#

Let's use reset button on NABox UI. That should help

urban timber Jul 7, 2023, 12:39 PM

#

reset?

stark seal Jul 7, 2023, 12:40 PM

#

urban timber Jul 7, 2023, 12:40 PM

#

yes yes i did it 😢

stark seal Jul 7, 2023, 12:41 PM

#

thanks, does that dashboard have a valid query after the reset?

urban timber Jul 7, 2023, 12:42 PM

#

unfortunatly not

stark seal Jul 7, 2023, 12:44 PM

#

can you paste your versions and dashboards, like this

urban timber Jul 7, 2023, 12:45 PM

#

#

wintry kayak Jul 7, 2023, 12:47 PM

#

Could you open ONTAP: S3 Object Storage dashboard under Harvest - cDOT folder and reverify s3 panels?

urban timber Jul 7, 2023, 12:52 PM

#

sorry Rahul, i not understand. this is the dashboard that doesn't work

#

so, i reopend that

#

stark seal Jul 7, 2023, 12:57 PM

#

Are you able to ssh to your nabox instance? If so, can you ssh there and collect the nabox logs and send them to us? It seems like the dashboard import is failing when you press the Reset button in nabox and the logs should help us figure out why. The reason I say the import failed is that panel should have a query and you mentioned earlier that the query was empty. To collect logs https://nabox.org/documentation/troubleshooting#collecting-logs and email them to ng-harvest-files@netapp.com

urban timber Jul 7, 2023, 1:01 PM

#

yes, i'm doing it

#

done

stark seal Jul 7, 2023, 1:13 PM

#

thanks received

urban timber Jul 7, 2023, 1:19 PM

#

stark seal Are you able to ssh to your nabox instance? If so, can you ssh there and collect...

yes, query is empty

stark seal Jul 7, 2023, 1:38 PM

#

thanks! Your logs helped, looks like we have a bug in that dashboard with the datasource. We can unblock you

urban timber Jul 7, 2023, 1:46 PM

#

ok

#

so i say this to the customer and we will wait for the enxt release

#

thanks for your help guys!

wintry kayak Jul 7, 2023, 1:47 PM

#

@urban timber Yes, we'll have the fix ready soon. We can get this fixed in your environment manually if that helps?

urban timber Jul 7, 2023, 1:49 PM

#

ok, i have only 30min from now because i have to go to the airport

#

if we can do it in few minutes ok, otherwise we will do it monday

wintry kayak Jul 7, 2023, 1:50 PM

#

okay yeah it may take time. If not urgent, We can do on Monday.

urban timber Jul 7, 2023, 1:51 PM

#

no, it is not urgent. So we will do it monday

wintry kayak Jul 7, 2023, 1:51 PM

#

sure thanks

urban timber Jul 7, 2023, 1:51 PM

#

so, I wish you a good weekend and thanks again for your availability

stark seal Jul 7, 2023, 1:51 PM

#

you too!

wintry kayak Jul 7, 2023, 1:52 PM

#

Thanks! u as well!

stark seal Jul 7, 2023, 3:04 PM

#

@urban timber we have fixed this in pull request https://github.com/NetApp/harvest/pull/2191 and it will be in tonights nightly. If you get a chance on Monday, please upgrade your nabox Harvest to nightly https://github.com/NetApp/harvest/releases/tag/nightly and confirm if S3 dashboard works for you

urban timber Jul 10, 2023, 7:11 AM

#

hello all! good morning. i will do it by today. thanks!

urban timber Jul 10, 2023, 9:19 AM

#

hi all! i updated harvest, did a dashboards reset

#

et voila'!

#

#

thanks a lot for your helps!

#harvest.yml config for collection metrics