Failed to migrate NAbox from 3 to 4 | NetApp | Page 1

tough estuary Jul 23, 2024, 7:26 AM

#

I met an error message when trying to migrate NAbox from 3 to 4
This is the error message:
Restarting Prometheus with admin API enabled...2024/07/23 07:14:12 ERROR Failed to start Prometheus error="exit status 1"

jagged pewter Jul 23, 2024, 7:26 AM

#

@ember widget

ember widget Jul 23, 2024, 7:35 AM

#

What version of NAbox 3 ?

tough estuary Jul 23, 2024, 7:36 AM

#

Another question about NAbox4,
A client they are using both NAbox and self installed harvest with prometheus and Grafana,
They are concerned that if upgrading to NAbox 4 which is using Victoria Metrics rather than Prometheus, will there have any issue?

tough estuary Jul 23, 2024, 7:38 AM

#

ember widget What version of NAbox 3 ?

NAbox 3.4

ember widget Jul 23, 2024, 7:38 AM

#

That’s why. You need 3.5.2. And I need to upgrade the docs 😂

tough estuary Jul 23, 2024, 7:39 AM

#

The minimum requirement for the migration is 3.5.2, is that correct?

ember widget Jul 23, 2024, 7:39 AM

#

Correct

#

Regarding your client, if they are not doing anything fancy, Victoria and prom are compatible. If they are using some things specific to prom then maybe they have to be careful. As far as Harvest goes, it’s the same.

tough estuary Jul 23, 2024, 7:45 AM

#

ember widget Regarding your client, if they are not doing anything fancy, Victoria and prom a...

Noted!
@ember widget
For our client, after disabling all unused metrics in harvest using conf_path, they do see the decline in CPU loading.
And they want to do the same following things in NAbox3:

Disable all unused metrics -> is this the same procedure to harvest?
Increase the file limit for prometheus to 65536 -> where can I do the setting?

ember widget Jul 23, 2024, 7:49 AM

#

Harvest should be customized the same, it’s documented here https://nabox.org/faq/#customize-harvest
File limits I don’t know what has been changed or if that’s necessary but it can be done

tough estuary Jul 23, 2024, 7:56 AM

#

ember widget Harvest should be customized the same, it’s documented here https://nabox.org/fa...

Is the folder structure same in Nabox 3 and 4?
"/etc/nabox/harvest/user'

ember widget Jul 23, 2024, 7:59 AM

#

With NAbox 3 you had to format the yaml file with a trick https://3.nabox.org/faq/#customize-harvest-2
That’s not the case with NAbox 4

tough estuary Jul 23, 2024, 8:21 AM

#

@ember widget If I want to change the conf_path for pollers to disable unused templates and metrics.
Can I just add "customconf" to /opt/harvest2-conf/ and set the conf_path to each pollers in harvest.yml?
And if I upgrade harvest or Nabox, will the change be altered?

ember widget Jul 23, 2024, 8:25 AM

#

@jagged pewter What do you think ? Is there a way to disable templates with customization ?

jagged pewter Jul 23, 2024, 8:30 AM

#

If you are using the default conf_path, you will need to modify and comment out the necessary sections in default.yaml. However, be aware that this file will be overwritten during an upgrade. To avoid this issue, you can change the conf_path to point to your own default.yaml file, which will remain unchanged during upgrades.

tough estuary Jul 23, 2024, 8:35 AM

#

So I should add a new harvest_new.yml and another customconf in "/opt/harvest2-conf/" and then add the conf_path to each pollers which point to the newly created customconf?

jagged pewter Jul 23, 2024, 9:23 AM

#

NABox steps may be different. @ember widget ?

ember widget Jul 23, 2024, 9:32 AM

#

Yes, NAbox does systematically add the harvest distribution conf path I’m afraid.

jagged pewter Jul 23, 2024, 9:33 AM

#

@tough estuary Are you using Rest or Zapi collector where you want to disable templates?

tough estuary Jul 23, 2024, 9:39 AM

#

Using Zapi

jagged pewter Jul 23, 2024, 9:40 AM

#

Okay. I think we need to comment out relevant templates only in default.yaml in NABox. @ember widget Can we do that?

ember widget Jul 23, 2024, 9:42 AM

#

Currently, --confpath is statically assigned

    confPaths := []string{
        "../harvest/conf",
        "active",
    }

jagged pewter Jul 23, 2024, 9:43 AM

#

@tough estuary Your question is for NABox3 or 4?

tough estuary Jul 23, 2024, 9:43 AM

#

NABox3

ember widget Jul 23, 2024, 9:43 AM

#

oh

#

unsupported. deprecated. #imouttahere

#

😄

#

Sorry about that, you said that in the beginning, it was NAbox 3

#

mmmm

#

I'll give you a flag to disable harvest conf in NAbox 4 in exchange for an update 😄

jagged pewter Jul 23, 2024, 9:47 AM

#

@tough estuary Could you also explain the problem statement again? Like why do you want to disable templates? If your concern is to reduce cpu usage in system, Then NABox4 may be a good choice. @ember widget Do we have any resource consumption comparison between nabox3 and 4?

ember widget Jul 23, 2024, 9:48 AM

#

Unfortunately I dont, but there is this https://zetablogs.medium.com/prometheus-vs-victoria-metrics-load-testing-3fa0cc782912

Medium

Prometheus Vs Victoria Metrics Load Testing

Introduction

#

Looks like victoria is harder on CPU

#

Well I take it back, when scaling it's less cpu

jagged pewter Jul 23, 2024, 9:52 AM

#

@tough estuary I recall that there were issues earlier in your setup related to the file limit for the Prometheus process. Additionally, we need to check if there is a high object count in the customer environment that might be driving up CPU usage. Do you know how much CPU is currently being used? If you can share the logs, we can take a closer look.

tough estuary Jul 23, 2024, 9:52 AM

#

jagged pewter <@706068726092398593> Could you also explain the problem statement again? Like w...

Sure, client is using Nabox3, and the utilization is normal and stable on Nabox, so they only want to disable unused metrics on Nabox to decrease the collection size. For the upgrade to Nabox4 , since they have their own-built Prometheus, they have concerned about maintaing two different databases

jagged pewter Jul 23, 2024, 9:54 AM

#

I think something like below may work

ember widget Jul 23, 2024, 9:56 AM

#

If they have their own prom they are doing two collections already, corrcect ?

tough estuary Jul 23, 2024, 9:57 AM

#

ember widget If they have their own prom they are doing two collections already, corrcect ?

Yes, one with their own-built harvest, another was NAbox default prometheus

tough estuary Jul 23, 2024, 9:58 AM

#

jagged pewter I think something like below may work

So the limited.yml will not be erased after upgrading NAbox or harvest?

ember widget Jul 23, 2024, 9:58 AM

#

In your case it could be nice to disable prom/victoria alltogether

tough estuary Jul 23, 2024, 10:00 AM

#

jagged pewter <@706068726092398593> I recall that there were issues earlier in your setup rela...

Yes, We met the file limit issue in prometheus before.
So try to increase that setting in Nabox also.

jagged pewter Jul 23, 2024, 10:05 AM

#

jagged pewter I think something like below may work

This approach is working for me in local tests

@ember widget will limited.yaml persist during upgrades?

#

ember widget Jul 23, 2024, 10:06 AM

#

it should yes

jagged pewter Jul 23, 2024, 10:06 AM

#

@tough estuary You can try this approach and see

ember widget Jul 23, 2024, 10:07 AM

#

collectors: in the yaml will probably be overwritten when you update the configuration

jagged pewter Jul 23, 2024, 10:08 AM

#

yeah managing harvest.yaml will be challenge.. probably have to keep it manual change from shell rather than UI.

ember widget Jul 23, 2024, 10:08 AM

#

yep

tough estuary Jul 23, 2024, 10:12 AM

#

jagged pewter This approach is working for me in local tests <@323825973718155265> will limit...

Is this harvest.yml in /opt/harvest2-conf?
I tried to edit, and it shows read only for harvest.yml

jagged pewter Jul 23, 2024, 10:20 AM

#

Yes. I was able to edit it. I am using NAbox 3.5 (2024-02-26)

ember widget Jul 23, 2024, 10:45 AM

#

Should be editable yes, never login as admin with ssh

tough estuary Jul 25, 2024, 6:27 AM

#

ember widget That’s why. You need 3.5.2. And I need to upgrade the docs 😂

I have upgraded to 3.5.3,
but seeing an error while migrating to nabox4
"Unable to find image 'victoriametrics/vmctl:heads-master-0-g93a29fce4e' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: server misbehaving."

ember widget Jul 25, 2024, 6:28 AM

#

Something went wrong during the upgrade. Can you apply again ?

tough estuary Jul 25, 2024, 6:28 AM

#

ember widget Something went wrong during the upgrade. Can you apply again ?

Sure

tough estuary Jul 25, 2024, 7:25 AM

#

I found Grafana keeps restarting after upgrading to 3.5.3, so Grafana shows unavailable now

grafana grafana/grafana-oss:9.5.14 "/run.sh" grafana 7 minutes ago Restarting (1) 27 seconds ago

ember widget Jul 25, 2024, 7:43 AM

#

what's the error before crash in dc logs grafana ? It's probably about conflicting dashboard and folders

tough estuary Jul 25, 2024, 7:44 AM

#

logger=provisioning.plugins t=2024-07-25T07:43:14.618308117Z level=error msg="Failed to read plugin provisioning files from directory" path=/provisioning-conf/plugins error="open /provisioning-conf/plugins: no such file or directory"
grafana | logger=provisioning.notifiers t=2024-07-25T07:43:14.618397686Z level=error msg="Can't read alert notification provisioning files from directory" path=/provisioning-conf/notifiers error="open /provisioning-conf/notifiers: no such file or directory"

ember widget Jul 25, 2024, 7:44 AM

#

If you rename /usr/local/nabox/files/grafana/dashboards/NAbox/nabox.json to /usr/local/nabox/files/grafana/dashboards/NAbox/nabox.j_ it might be able to start, then rename the file back

ember widget Jul 25, 2024, 7:45 AM

#

tough estuary logger=provisioning.plugins t=2024-07-25T07:43:14.618308117Z level=error msg="Fa...

Those are fine, that's not what's causing the crash

tough estuary Jul 25, 2024, 7:48 AM

#

ember widget Those are fine, that's not what's causing the crash

Error: ✗ alert rules: got invalid response. expected folder, found dashboard

ember widget Jul 25, 2024, 7:48 AM

#

that's the one

#

so what if you :

mv /usr/local/nabox/files/grafana/dashboards/NAbox/nabox.json /usr/local/nabox/files/grafana/dashboards/NAbox/nabox.j_
dc restart grafana

tough estuary Jul 25, 2024, 7:52 AM

#

It shows up in "dc ps" command, but seeing unavailable in GUI

ember widget Jul 25, 2024, 7:53 AM

#

I'm trying to remember the sequence for this issue.

#

it might be a different file :

mv /usr/local/nabox/files/grafana/provisioning/alerting/nabox.yaml /usr/local/nabox/files/grafana/provisioning/alerting/nabox.y_
dc restart grafana

tough estuary Jul 25, 2024, 7:55 AM

#

ember widget it might be a different file : ``` mv /usr/local/nabox/files/grafana/provisionin...

Yes! This works!!!

ember widget Jul 25, 2024, 7:55 AM

#

whooop

#

you should be able to put everything back

tough estuary Jul 25, 2024, 7:56 AM

#

ember widget whooop

rename back and restart?

ember widget Jul 25, 2024, 7:56 AM

#

yes, start with the json file, restart, then the yaml file, restart

#

You don't have a /opt/grafana-dashboards/nabox.json file correct ?

tough estuary Jul 25, 2024, 7:57 AM

#

ember widget You don't have a `/opt/grafana-dashboards/nabox.json` file correct ?

Yes, the folder was empty

ember widget Jul 25, 2024, 8:02 AM

#

also docker images|grep vmctl should return something

tough estuary Jul 25, 2024, 8:02 AM

#

victoriametrics/vmctl heads-master-0-g93a29fce4e f76f053c01c9 4 months ago 42.6MB

ember widget Jul 25, 2024, 8:03 AM

#

good

#

if grafana starts, you're all set

tough estuary Jul 25, 2024, 8:03 AM

#

Thanks @ember widget

ember widget Jul 25, 2024, 8:03 AM

#

Sorry about that !

tough estuary Jul 25, 2024, 9:22 AM

#

@ember widget Is there any prometheus.service in NAbox3 where I can change the limit from1024 to 65536?
[Service]
LimitNOFILE=65536

ember widget Jul 25, 2024, 9:34 AM

#

This is what prometheus container sees :

Max open files            1048576              1048576              files

tough estuary Jul 26, 2024, 3:18 AM

#

ember widget This is what prometheus container sees : ``` Max open files 1048576 ...

Hi Yann,
Is the procedure correct to get the limits?
"ps aux | grep prometheus" to get prometheus pid, I should use the 5366?
And then use "cat /proc/{pid}/limits | grep "Max open files""

tough estuary Jul 26, 2024, 3:40 AM

#

ember widget Jul 26, 2024, 10:05 AM

#

Let me do it

#

nabox-api:~# cat /proc/22002/limits |grep "Max open files"
Max open files            1048576              1048576              files

#

Looking good

tough estuary Jul 29, 2024, 8:06 AM

#

@ember widget In the NAbox3 , we can find harvest conf under /opt/harvest2-conf/conf, where can we find these folders in NAbox 4?

ember widget Jul 29, 2024, 8:06 AM

#

In /etc/nabox/harvest

tough estuary Jul 29, 2024, 8:10 AM

#

But all I see in only
exclude_transient_volumes.yaml in 9.8.0 folder
under /etc/nabox/harvest/active/zapiperf/cdot/9.8.0

ember widget Jul 29, 2024, 10:49 AM

#

Yes, that's a default customization for Harvest in NAbox.
If you're looking for the full harvest distribution, that's in /data/packages/harvest

#

But harvest config file really is /etc/nabox/harvest/harvest.yml

hidden terrace Jul 30, 2024, 8:15 AM

#

Hi @ember widget last migration from 3.5.3 to 4.0.4 failed - ERROR Failed to take Prometheus snapshot error="Failed to parse snapshot response: invalid character 'S' looking for beginning of value"

ember widget Jul 30, 2024, 8:47 AM

#

Are you root ?

tough estuary Sep 26, 2024, 8:56 AM

#

ember widget unsupported. deprecated. #imouttahere

Hi @ember widget , So we cannot change the default conf_path in NAbox3 correct?

ember widget Sep 26, 2024, 2:18 PM

#

Not sure I'm following you !

#Failed to migrate NAbox from 3 to 4