Dashboard "ONTAP: StorageGrid FabricPool" has no "Datacenter" | NetApp | Page 1

olive vessel Sep 13, 2023, 1:07 PM

#

The dashboard "ONTAP: StorageGrid FabricPool" that was introduced in the 23.05 release does not list all the clusters in the list of "ONTAP clusters" on grafana. It only displays a few among several datacentres and clusters defined in the harvest.yml file that every other dashboard is able to correctly display. On trying to investigate this further, it could be noticed that the "Datacenter" tab that is displayed with other dashboards does not get displayed at all.

swift oak Sep 13, 2023, 1:14 PM

#

Hi Madaan - that dashboard is filtering the cluster variable to only those clusters that have a cloud store associated with them. Perhaps the clusters that are missing from the list don't have a cloud store associated with them?

olive vessel Sep 13, 2023, 1:21 PM

#

Hi Chris - I now have several clusters that have object store configured. I have checked this with the commands "storage aggregate object-store show" and "storage aggregate object-store config show"

swift oak Sep 13, 2023, 1:25 PM

#

which version of Harvest are you using and which collector? - after 23.08, mamoep discovered a few bugs with collecting these via ZAPI (they work fine via REST). https://github.com/NetApp/harvest/issues/2344 Those bugs are fixed in nightly if you want to give that a try

olive vessel Sep 13, 2023, 1:29 PM

#

ok thanks Chris. I will give this a try. If it interests you, I noticed that the clusters displayed were the GCP CVOs with a cloud storage bucket, but none of the on-prem clusters with storage grid buckets appeared. I will let you know how this goes after the upgrade.

swift oak Sep 13, 2023, 1:30 PM

#

thanks! Please do

glossy jackal Sep 13, 2023, 1:40 PM

#

23.05 was completely broken with ZAPI collector. 23.08 only has minor issues left, but clusters should be available in the list.

olive vessel Sep 13, 2023, 1:46 PM

#

Thanks Mamoep !

olive vessel Sep 15, 2023, 3:39 PM

#

Hi Chris, I have upgraded the Harvest to the suggested release but the problem still remains !

swift oak Sep 15, 2023, 3:55 PM

#

Hi @olive vessel if you go to Prometheus, do you see the cloud_target_labels metric?

olive vessel Sep 18, 2023, 3:03 PM

#

Hi @swift oak yes I do see "cloud_target_labels" and "cloud_target_used" in metric explorers

swift oak Sep 18, 2023, 3:53 PM

#

excellent, that's encouraging. Did you load the new dashboard included in nightly? If not, can you import it?

olive vessel Sep 18, 2023, 4:02 PM

#

Yes, I have loaded the new dashboard Chris.

#

( if you meant tha Grfana dashboard with the new release )

swift oak Sep 18, 2023, 4:04 PM

#

i mean the dashboard changes in the nightly you took three days ago - if you open the FabricPool dashboard and click on the gear icon, then variables, then select Bucket, what query do you have there?

olive vessel Sep 18, 2023, 4:31 PM

#

I dont have the settings/gear icon - perhaps my limited privileges to the dashboard ?

swift oak Sep 18, 2023, 4:32 PM

#

yes that's possible - are you using nabox or installed some other way?

olive vessel Sep 18, 2023, 4:34 PM

#

nabox - standard installation

swift oak Sep 18, 2023, 4:35 PM

#

and after installing nightly, did you reset harvest dashboards?

olive vessel Sep 18, 2023, 4:43 PM

#

It might be a completely different installation, Chris, but I see no settings options at all. Just dashboards. I or my colleagues do not see - or know about the settings options that you show in the screen above

#

swift oak Sep 18, 2023, 4:45 PM

#

yeah looks like nabox disables that option, not sure if that's intentional or not. @bleak parcel do you know why in nabox dashboards don't include the gear icon?

bleak parcel Sep 18, 2023, 4:46 PM

#

No it should be there, what version of NAbox ?

swift oak Sep 18, 2023, 4:49 PM

#

i can confirm nabox version NAbox 3.2b (2022-12-04) - Alpine Linux 3.14.2 is missing the gear icon, e.g.
What version are you using @olive vessel ?

bleak parcel Sep 18, 2023, 4:50 PM

#

The gear icon would be missing if logged in as guest maybe ?

#

olive vessel Sep 18, 2023, 4:51 PM

#

Yes, that is what we do Yam - we have enabled anonymous access

bleak parcel Sep 18, 2023, 4:52 PM

#

Ok, shouldn't be a concern in your case, but the only explanation I see if you don't have the reset button is you have an old NAbox version, in that case, there is a good chance the dashboard import would fail with recent Harvest versions if I remember correctly. You would see that in nabox-harvest2 logs

olive vessel Sep 18, 2023, 4:54 PM

#

ok, what would you suggest as the next step Yann ? Look at the logs, and then may be consider an upgarde ? What is the logs directory where I can see this information.

bleak parcel Sep 18, 2023, 4:55 PM

#

from ssh, you can run dc restart nabox-harvest2; dc logs -f --tail 100 nabox-harvest2

olive vessel Sep 18, 2023, 5:00 PM

#

cant say why, but the dc commands dont work Yann

#

Each of my clusters is a docker instance and this is what i do to restart them ( which I have done several times today )

#

docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans

bleak parcel Sep 18, 2023, 5:23 PM

#

You're probably not logged in as root

#

oh wait

#

that's not nabox ?

#

I'm confused, you shouldn't be running those commands or your own compose files

swift oak Sep 18, 2023, 5:27 PM

#

yes, looks like you might be using the docker-compose workflow instead of nabox @olive vessel If so, can you confirm that you upgraded your containers via https://netapp.github.io/harvest/23.08/install/containers/#upgrade-harvest

olive vessel Sep 18, 2023, 5:40 PM

#

hi Chris, yes - that is how I have upgraded the containers. I have just done a docker pull and another restart - to be sure.

swift oak Sep 18, 2023, 5:41 PM

#

but still no gear icon in the dashboard?

olive vessel Sep 18, 2023, 5:46 PM

#

So Chris - a very interesting observation. I now have more clusters in the "Ontap Clusters" drop down. I only had the GCP clusters before the upgrade - but I now have all the on-prem clusters where I have enabled FabricPools tiering

#

just that the "Datacentres" drop down is not there - and of course no gear icon, or the other settings option that you had showed me earlier

swift oak Sep 18, 2023, 6:01 PM

#

right, there won't be a datacenter variable on that dashboard. The most recent version should look something like this

Can you try logging into Grafana with the default credentials? https://netapp.github.io/harvest/23.08/install/containers/#grafana

olive vessel Sep 18, 2023, 6:06 PM

#

ah thanks - If we only expected more clusters and not the datacenter variable from the new release - then I think we have that covered now 🙂 I had just been looking for that variable to appear without realizing that the new clusters have been there already

swift oak Sep 18, 2023, 6:06 PM

#

perfect!

olive vessel Sep 18, 2023, 6:06 PM

#

Just logged in with default credentials and I see more options now - let me give you the information from that

olive vessel Sep 18, 2023, 6:33 PM

#

Just in case you still wanted to see this Chris

swift oak Sep 18, 2023, 6:35 PM

#

thanks, and what about the Bucket variable?

#

olive vessel Sep 18, 2023, 10:30 PM

#

no, I dont see the bucket variable.

#

swift oak Sep 19, 2023, 12:05 PM

#

thanks! that means you're using an older version of the image that does not include this recent change. Can you copy/paste one of your poller blocks from your harvest-compose.yml file? Here's one of mine as an example:

olive vessel Sep 19, 2023, 4:17 PM

#

Here it is, Chris

#

swift oak Sep 19, 2023, 6:04 PM

#

thanks Madaan - I see the problem. You are using image: ghcr.io/netapp/harvest:latest instead of nightly (image: ghcr.io/netapp/harvest:nightly)

latest will point to 23.08.
The changes we are discussing were made after 23.08 and only in nightly. If you want to try them you can you follow these upgrade directions, except for step one use docker pull ghcr.io/netapp/harvest:nightly instead of what's listed there https://netapp.github.io/harvest/nightly/install/containers/#upgrade-harvest
at step 3, use this instead of what is listed on the page
docker run --rm \ --entrypoint "bin/harvest" \ --volume "$(pwd):/opt/temp" \ --volume "$(pwd)/harvest.yml:/opt/harvest/harvest.yml" \ ghcr.io/netapp/harvest:nightly \ generate docker full \ --output harvest-compose.yml \ --image ghcr.io/netapp/harvest:nightly

olive vessel Sep 19, 2023, 8:13 PM

#

#

Thanks Chris, got there finally !

olive vessel Sep 20, 2023, 10:58 AM

#

Hi @swift oak , I need a suggestion from you. Now that we have upgraded the on-premise instance, I wanted to upgrade the cloud instance too. We have built the cloud instance to monitor the FSX instances which the on-premise instance could not monitor and still cannot monitor ( I just tested this).

#

Before moving on to upgrade the cloud instance with the nightly 23/08 release, I reminded myself that I had built this instance from a cloudformation template - https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/monitoring-harvest-grafana.html

Monitoring FSx for ONTAP file systems using Harvest and Grafana - F...

You can use standard NetApp monitoring tools to monitor your file system storage usage and performance, with the following Harvest and Grafana solution being one example.

#

and therefore felt that it is best for AWS to release the new template with the new release/patches than update it manually. Would you or NetApp have any recommendation on that ?

#

although I must admit that the template does not indicate what version we are on. I had to assume that it was 23.05 since I installed it 4 months ago.

swift oak Sep 20, 2023, 6:56 PM

#

hi @olive vessel I believe @quiet cosmos was working on some updated documentation to cover upgrade

quiet cosmos Sep 20, 2023, 7:01 PM

#

@olive vessel this hasn't gone through the QA process yet, so please make a backup of your environment before you start! but you should be able to do this:

/usr/local/bin/ansible-playbook /home/ec2-user/harvest_install/manage_harvest.yml --tags upgrade

and then

/usr/local/bin/ansible-playbook /home/ec2-user/harvest_install/manage_harvest.yml --tags dashboard

#

if you're really curious, run docker ps to list your container IDs and then check the version of docker in each container with docker exec $CONTAINER_ID bin/harvest -v

you can do this before & after the steps above to see what changed.

swift oak Sep 20, 2023, 7:06 PM

#

thanks Chris!

quiet cosmos Sep 20, 2023, 7:12 PM

#

thank you Chris 🙂

olive vessel Sep 21, 2023, 11:10 AM

#

Many thanks @swift oak and @quiet cosmos . I have reached a good postition overall with the upgrade. However, on checking this morning, I noticed that while all my on-premise clusters are reporting OK, the GCP clusters are not reporting since the upgrade. These clusters have been reporting well before the upgrade and nothing on them has changed. Do you know of any bug or issues with the reporting of GCP clusters with 23.08-1 ?

quiet cosmos Sep 21, 2023, 11:14 AM

#

I'm not aware of anything, but I have no experience with GCP clusters so I can't help there

swift oak Sep 21, 2023, 11:16 AM

#

@olive vessel can you grab the logs for one of your GCP pollers and email them to ng-harvest-files@netapp.com?

olive vessel Sep 21, 2023, 11:22 AM

#

Because the dockers instances for them are not up, I do not see these clusters reporting/logging on /var/log/harvest. Would there be another place to find those logs @swift oak ?

swift oak Sep 21, 2023, 11:31 AM

#

you can try https://netapp.github.io/harvest/nightly/help/log-collection/#docker-container but if the containers are in a restart loop there may not be a lot there. Worth checking and sharing your harvest-compose.yml for one of these failing pollers too

olive vessel Sep 21, 2023, 12:05 PM

#

just sent the logs, Chris

swift oak Sep 21, 2023, 12:06 PM

#

received, will take a look

olive vessel Sep 21, 2023, 12:38 PM

#

Phew..I reached the resolution Chris. The user and its creadentials were only added to the clusters with the "http" application - but I have now added them with the "ontapi" application as well 🙂

#Dashboard "ONTAP: StorageGrid FabricPool" has no "Datacenter"