NAbox 3.5b out | NetApp | Page 1

solemn falcon · 2024-02-13T00:08:29.250Z

- Full support for StorageGRID in Web UI - SMTP configuration for Grafana alerting - Added protection against ssh brute force - Improved security of web services - NAbox can now send alerts when running out of capacity (see alerting configuration in Grafana) - Removed deprecated option to set null point mode in Settings - Fix "Collect detailed data" Setting not working - Fix a failure while adding SAML enabled ONTAP system - Fix conditions when NAbox would fail to upgrade https://nabox.org/downloads/

solemn falcon Feb 13, 2024, 12:08 AM

#

Full support for StorageGRID in Web UI
SMTP configuration for Grafana alerting
Added protection against ssh brute force
Improved security of web services
NAbox can now send alerts when running out of capacity (see alerting configuration in Grafana)
Removed deprecated option to set null point mode in Settings
Fix "Collect detailed data" Setting not working
Fix a failure while adding SAML enabled ONTAP system
Fix conditions when NAbox would fail to upgrade
https://nabox.org/downloads/

severe oracle Feb 13, 2024, 1:08 AM

#

Thanks Yann.

I'm seeing the following error after the upgrade:

grafana | Error: ✗ alert rules: got invalid response. expected folder, found dashboard

I saw in another post about renaming an alert file or similar. But I'm new to NABox and not quite suire what this is referring to. Is there a way to workaround this problem? Thanks.

I think this might be related to LDAP .. I'm not sure. I sent an email to help@nabox.org.

tropic fractal Feb 13, 2024, 2:19 AM

#

@solemn falcon Will be checking this out soon. Quick question. If we decide to do a new deployment instead of an upgrade, is there any way to maintain the data from the previous system? I don't think I've seen a backup/restore option anywhere in the system, unless I just am blind

solemn falcon Feb 13, 2024, 7:31 AM

#

Unfortunately there is no UI to backup and restore, but you can use scpto copy the data directory over.

solemn falcon Feb 13, 2024, 7:32 AM

#

severe oracle Thanks Yann. I'm seeing the following error after the upgrade: grafana | Erro...

Thanks, let's try to figure that one out !

tropic fractal Feb 13, 2024, 7:36 AM

#

what is the proper directory to copy over, and do we have to go through and re-add everything again or is there a config file(s) we can move over for that as well?

solemn falcon Feb 13, 2024, 9:50 AM

#

Basically, the /data directory and /opt/harvest2-conf/harvest.yml

tropic fractal Feb 15, 2024, 4:50 AM

#

just pushed the update, seemed to go ok, no errors/etc, but after restarting
Grafana: unavailable
seems as if the container isn't running
dc ps shows
grafana grafana/grafana-oss:9.5.14 "/run.sh" grafana 17 minutes ago Restarting (1) 32 seconds ago

#

not sure if that's because we made changes to some of the yaml files

solemn falcon Feb 15, 2024, 8:06 AM

#

You didn' do a new deployment finally ?

#

please send me dc logs -f grafana
It might not show it on the first try

teal zinc Feb 15, 2024, 9:10 AM

#

I also have a problem that grafana doesn't start anymore after upgrading from NAbox 3.4.1b (2023-12-04) to this new version.. for now I reverted back to a snapshot so it's back online 🙂

Did you already figure out what could be wrong by any chance after the upgrade?

tropic fractal Feb 15, 2024, 3:28 PM

#

this is the last part of the log

grafana | logger=local.finder t=2024-02-15T15:26:00.163525841Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
grafana | logger=query_data t=2024-02-15T15:26:00.166111546Z level=info msg="Query Service initialization"
grafana | logger=live.push_http t=2024-02-15T15:26:00.169747443Z level=info msg="Live Push Gateway initialization"
grafana | logger=ldap t=2024-02-15T15:26:00.169766018Z level=info msg="LDAP enabled, reading config file" file=/var/lib/grafana/ldap.toml
grafana | logger=ngalert t=2024-02-15T15:26:00.173887169Z level=warn msg="Unexpected number of rows updating alert configuration history" rows=0 org=1 hash=8c409350c88d78d2ee938448449e628d
grafana | logger=infra.usagestats.collector t=2024-02-15T15:26:01.359346419Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
grafana | logger=provisioning.plugins t=2024-02-15T15:26:01.408925396Z level=error msg="Failed to read plugin provisioning files from directory" path=/provisioning-conf/plugins error="open /provisioning-conf/plugins: no such file or directory"
grafana | logger=provisioning.notifiers t=2024-02-15T15:26:01.409012118Z level=error msg="Can't read alert notification provisioning files from directory" path=/provisioning-conf/notifiers error="open /provisioning-conf/notifiers: no such file or directory"
grafana | logger=provisioning.alerting t=2024-02-15T15:26:01.409687097Z level=info msg="starting to provision alerting"
grafana | Error: ✗ alert rules: got invalid response. expected folder, found dashboard

severe oracle Feb 15, 2024, 6:56 PM

#

tropic fractal this is the last part of the log grafana | logger=local.finder t=2024-02-15T15...

I had that exact same error in my logs with 3.5b, with Grafana trying to restart every 60 second.

I'm not 100% sure what fixed it, but I renamed /usr/local/nabox/files/grafana/provisioning/alerting/nabox.yaml to .yaml.orig and restarted the Grafana container and it came up.

Odd thing is I noticed that LDAP was then disabled. I re-enabled LDAP and Grafana immediately crashed again. But a stop and start of the container seemed to fix it again. I renamed the alerting yaml back and restarted again, and it was fine. And now LDAP is back working. I cannot explain it. But those were the only two things I had messed with and its all up now.

tropic fractal Feb 15, 2024, 6:58 PM

#

well then. renaming that file did fix the problem. Thanks

solemn falcon Feb 15, 2024, 9:05 PM

#

You can rename it back

severe oracle Feb 15, 2024, 9:10 PM

#

Yeah i renamed mine back after and it still starts. It just seemed to be that initial start after the upgrade

tropic fractal Feb 15, 2024, 10:28 PM

#

weird.

solemn falcon Feb 16, 2024, 7:08 AM

#

@tropic fractal What version were you upgrading from ?

#

@tropic fractal There is an explanation. The NAbox folder replaces the NAbox dashboard, and grafana gets confused. 3.5b has a new alert rule for capacity and it has to be associated with a folder. hence the "NAbox" folder. So I removed the declaration for the NAbox dashboard to make room for folder creation but because alerts gets loaded before dashboards, a first start needs to happen without the alert. This let grafana delete the nabox dashboard and after that all is fine.

Anyways, the updater is supposed to do that but obviouslt something is wrong

tropic fractal Feb 16, 2024, 7:53 AM

#

was going from 3.4b, i think it was a build from Oct

solemn falcon Feb 16, 2024, 9:32 AM

#

I pushed a new 3.5 with an improvement specifically for this

#

if one of you still have a snapshot lying around I'd like to see if it's better !

teal zinc Feb 16, 2024, 10:14 AM

#

did you also update the md5 value for this new 3.5 update file?
since when I download it I get this:

% md5 Downloads/NAbox-3.5b.update
MD5 (Downloads/NAbox-3.5b.update) = 4171cd023eaab770509e553285ea39a5

solemn falcon Feb 16, 2024, 10:15 AM

#

4171cd023eaab770509e553285ea39a5 is correct

teal zinc Feb 16, 2024, 10:15 AM

#

ah ok 🙂

#

let me try it

solemn falcon Feb 16, 2024, 10:16 AM

#

[2024-02-16 09:17:48 CET]

#

that's the download date, you d/l it this morning ?

teal zinc Feb 16, 2024, 10:16 AM

#

I just re-downloaded it

#

but it seems that I already got that file before (in the morning) and then I still needed to rename the file

#

% md5 Downloads/NAbox-3.5b\ (1).update
MD5 (Downloads/NAbox-3.5b (1).update) = 4171cd023eaab770509e553285ea39a5
% rm Downloads/NAbox-3.5b\ (1).update
% md5 Downloads/NAbox-3.5b.update
MD5 (Downloads/NAbox-3.5b.update) = 4171cd023eaab770509e553285ea39a5

solemn falcon Feb 16, 2024, 10:17 AM

#

looking good

teal zinc Feb 16, 2024, 10:18 AM

#

now grafana is still restarting

solemn falcon Feb 16, 2024, 10:19 AM

#

dang it. you have a snapshot ?

teal zinc Feb 16, 2024, 10:20 AM

#

yep but before the fix was as mentioned above renaming that .yaml file

#

since we have 2 nabox instances

#

the first one was already upgraded and when I renamed that file grafana was working again

#

and also renamed it back and restarted grafana and then it was still working

solemn falcon Feb 16, 2024, 10:22 AM

#

if you have a pre-upgrade snapshot, I could use another update round this time running dc logs -f nabox-api before launchin the upgrade

teal zinc Feb 16, 2024, 10:23 AM

#

ok I can revert

#

grafana  | logger=infra.usagestats.collector t=2024-02-16T10:21:32.804587473Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
grafana  | logger=provisioning.plugins t=2024-02-16T10:21:32.880065346Z level=error msg="Failed to read plugin provisioning files from directory" path=/provisioning-conf/plugins error="open /provisioning-conf/plugins: no such file or directory"
grafana  | logger=provisioning.notifiers t=2024-02-16T10:21:32.880116234Z level=error msg="Can't read alert notification provisioning files from directory" path=/provisioning-conf/notifiers error="open /provisioning-conf/notifiers: no such file or directory"
grafana  | logger=provisioning.alerting t=2024-02-16T10:21:32.883229582Z level=info msg="starting to provision alerting"
grafana  | Error: ✗ alert rules: got invalid response. expected folder, found dashboard```

#

is what grafana is showing

solemn falcon Feb 16, 2024, 10:24 AM

#

yes that's exactly the bug I'm failing to fix

teal zinc Feb 16, 2024, 10:24 AM

#

ok I reverted it and powering on now

solemn falcon Feb 16, 2024, 10:25 AM

#

well, that and the web site md5 not updating 🙂

teal zinc Feb 16, 2024, 10:25 AM

#

hehe 🙂

#

I will wait a bit until all clusters are reporting again do you already want the logs from nabox-api now or wait a bit?

#

also currently the nabox-api container is named 0715d5436f34_nabox-api no clue why 🙂

#

4d51478fb24a_nabox-harvest2
67a2494ebf23_go-carbon     
6e7c0922fbeb_nabox-harvest 
85460154d419_cadvisor      
9b892825873d_prometheus    
b49139758267_nabox-admin   
e516d6332c59_graphite      
e6687167f33c_node-exporter 
eb6735119a92_traefik       
grafana ```

#

are the names

#

@solemn falcon the clusters are reporting again how much of the nabox-api you need? 🙂

solemn falcon Feb 16, 2024, 10:32 AM

#

yes it shouldn't have an impact and still something I'm trying to figure out

#

The beginning, where you should see "grafana" on it's own line during the time you launch the update

#

with some curl command results

teal zinc Feb 16, 2024, 10:33 AM

#

so I should tail that log and do the upgrade

solemn falcon Feb 16, 2024, 10:34 AM

#

No if you didn't run dc logs -f then it's too late, no worries I'll do some tests on my side !

teal zinc Feb 16, 2024, 10:34 AM

#

I only reverted now

#

didn't do anything yet

solemn falcon Feb 16, 2024, 10:34 AM

#

ah ok I thought you did apply

teal zinc Feb 16, 2024, 10:34 AM

#

🙂

solemn falcon Feb 16, 2024, 10:34 AM

#

all right let's do it then

#

dc logs -f nabox-api and run the update

teal zinc Feb 16, 2024, 10:35 AM

#

k

#

📎 message.txt

#

and now grafana is restarting again

#

do you require anything else @solemn falcon?

solemn falcon Feb 16, 2024, 10:41 AM

#

Thanks, looks like something isn't happening as it should, I'll keep digging, thanks !

teal zinc Feb 16, 2024, 10:41 AM

#

ok I will just apply the manual fix now then 🙂

solemn falcon Feb 16, 2024, 10:42 AM

#

What is the pre-upgrade version ?

teal zinc Feb 16, 2024, 10:42 AM

#

it was this one: NAbox 3.4.1b (2023-12-04)

solemn falcon Feb 16, 2024, 10:43 AM

#

thank you

teal zinc Feb 16, 2024, 10:44 AM

#

np

#

btw I do see this now when I restart the grafana container / or just run dc ps:

WARN[0000] The "GF_AUTH_ANONYMOUS_ENABLED" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_AUTH_ANONYMOUS_ORG_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_AUTH_ANONYMOUS_ORG_ROLE" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_SMTP_ENABLED" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_SMTP_HOST" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_SMTP_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_SMTP_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "GF_SMTP_FROM_ADDRESS" variable is not set. Defaulting to a blank string.```

solemn falcon Feb 16, 2024, 3:14 PM

#

has your terminal session been established before the update ?

#

dcalias changed with the update and using a different declaration.
Also check you have both .envand .env.custom in /usr/local/nabox/docker-compose/

#

I think I'm on the right track regarding the grafana container, a new build is creating

solemn falcon Feb 16, 2024, 4:22 PM

#

Ok grafana issue seems to be fixed, and no issue with the environment variables in dccommand unless I'm on the ssh session before the update

tropic fractal Feb 21, 2024, 4:06 PM

#

quick question..
Trying to add some rules to IPTABLES for our security scans (specifically for ICMP blocking type 13+14) but the rules get wiped every time the system is restarted. Even trying to do a iptables-save or iptables-apply doesn't make them stick

tropic fractal Feb 22, 2024, 8:04 PM

#

@solemn falcon anything you know that would block this from sticking or anything specific that needs to be changed to keep it after changes are made?

solemn falcon Feb 22, 2024, 8:06 PM

#

tropic fractal quick question.. Trying to add some rules to IPTABLES for our security scans (sp...

Iptables save should work, but you might have to enable the service that goes along I believe.

tropic fractal Feb 22, 2024, 8:07 PM

#

what service is that

solemn falcon Feb 22, 2024, 9:22 PM

#

I would bet on iptables-openrc-1.8.9-r2 x86_64 {iptables} (GPL-2.0-or-later) [installed]

tropic fractal Feb 22, 2024, 9:23 PM

#

iptables is installed by default on this build. just not sure why the changes aren't sticking after a restart.
didn't know if there was something that would change it to some default for NABox because there are several entries for docker stuff in the -L output

#

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.3 tcp dpt:2003
ACCEPT tcp -- anywhere 172.17.0.3 tcp dpt:https
ACCEPT tcp -- anywhere 172.17.0.3 tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

#

so, wasn't sure if there was something that wiped the config and put the defaults in every boot or what

solemn falcon Feb 22, 2024, 9:25 PM

#

so, there is a iptables service you need to enable

#

rc-update add iptables

#

I believe (not tested)

tropic fractal Feb 22, 2024, 9:33 PM

#

where do the entries for docker come from

solemn falcon Feb 23, 2024, 8:47 AM

#

From docker 😄 That's standard / built in docker behaviour you shouldn't have to worry about it

#NAbox 3.5b out