NAbox 4.0.7 is available | NetApp | Page 1

severe ferry · 2024-09-11T23:10:40.091Z

- fix: qcow2 images wouldn't use eth0 by default - fix: enable guest access to Grafana would not work if Grafana organization name had been changed - fix: web site aborts upgrade package upload after 60s - fix: wrong permission in upgrade file would break NAbox during manual upgrades - new: migrate tool for NAbox 3 can be run with -debug for troubleshooting - Grafana 11.2 - Base OS upgrade (Flatcar Container Linux 3975.2.1)

severe ferry Sep 11, 2024, 11:10 PM

#

fix: qcow2 images wouldn't use eth0 by default
fix: enable guest access to Grafana would not work if Grafana organization name had been changed
fix: web site aborts upgrade package upload after 60s
fix: wrong permission in upgrade file would break NAbox during manual upgrades
new: migrate tool for NAbox 3 can be run with -debug for troubleshooting
Grafana 11.2
Base OS upgrade (Flatcar Container Linux 3975.2.1)

jade lion Sep 12, 2024, 1:43 PM

#

Just installed, configured and started a upgrade of a system with 4 clusters and quite some history.
The network startup file was not created. Did that by hand.
Installed Harvest and started migrating. Still running. But a well documented smooth experience so far!

severe ferry Sep 12, 2024, 2:49 PM

#

Thanks Martijn ! I'm worried that the network startup file wasn't created. Is that on VMware ? Did you setup networking in vCenter deployment or did you leave it blank ?
Harvest wasn't installed already ?

jade lion Sep 12, 2024, 3:19 PM

#

severe ferry Thanks Martijn ! I'm worried that the network startup file wasn't created. Is th...

Yes, Harvest was already at 24.08
I set all in VMware 8.0.3, but no start-up file.

severe ferry Sep 12, 2024, 3:20 PM

#

So no /etc/systemd/network/10-nabox.network ?

jade lion Sep 12, 2024, 3:20 PM

#

Migration finished. I had to re-enter the passwords for data collection to start after the migration.

jade lion Sep 12, 2024, 3:20 PM

#

severe ferry So no `/etc/systemd/network/10-nabox.network` ?

Not a single file in the directory

severe ferry Sep 12, 2024, 3:22 PM

#

Ok, would you send a support bundle to https://upload.nabox.org/qyko-zocy-cify ?

jade lion Sep 12, 2024, 3:22 PM

#

Sure!

severe ferry Sep 12, 2024, 3:23 PM

#

Thanks !

jade lion Sep 12, 2024, 3:24 PM

#

It's there!

#

Just another note: the Grafana item on the About screen has a question mark and not the version number

#

Uploaded a screenshot

severe ferry Sep 12, 2024, 3:26 PM

#

Yeah I see that. You changed the default password didn't you ?

jade lion Sep 12, 2024, 3:26 PM

#

Yes, sir 😉 Standard practice

#

But after the migration

severe ferry Sep 12, 2024, 3:27 PM

#

Ok there is some weird race condition with setting up grafana auth I'll try and track that in the logs

#

and I see the issue with network config I think

jade lion Sep 12, 2024, 3:27 PM

#

Did I make a typo?

severe ferry Sep 12, 2024, 3:32 PM

#

No I think it's a bug. just trying to understand why I didn't hit it !

#

How did you configure ip btw ? Originally. Static ip or dhcp in vcenter screen ?

jade lion Sep 12, 2024, 4:06 PM

#

Static!

severe ferry Sep 12, 2024, 5:10 PM

#

With the netmask in the IP address ?

severe ferry Sep 12, 2024, 5:25 PM

#

I think I got it

jade lion Sep 12, 2024, 6:16 PM

#

severe ferry With the netmask in the IP address ?

/23

severe ferry Sep 12, 2024, 7:05 PM

#

for the grafana thing, what you can do is regenerate an api token for the nabox user, in grafana I think it exists, and put it in /etc/nabox/secrets/grafana-secret then systemctl restart naboxd

glass glacier Sep 12, 2024, 7:07 PM

#

If you wanna test on my Harvest to make sure it works, feel free. If you break it you fix it. 😄

severe ferry Sep 12, 2024, 7:07 PM

#

You don't have to tell me twice 😄

#

It'll take me a couple hours to get on VPN then I'm unleashed

#

NAbox 4.0.7 is available

jade lion Sep 12, 2024, 11:43 PM

#

Should I update?

severe ferry Sep 13, 2024, 7:50 AM

#

To be on the safe side but you shouldn't have any problem

jade lion Sep 17, 2024, 7:28 AM

#

Just upgraded, will do the Grafana part today

jade lion Sep 17, 2024, 8:44 AM

#

severe ferry for the grafana thing, what you can do is regenerate an api token for the `nabox...

Do you have some more details? Cannot find an nabox user in grafana, nor service account.

severe ferry Sep 17, 2024, 9:10 AM

#

Yep that’s the problem. You can create a new “nabox” service account and paste the key in the grafana-secret file

jade lion Sep 17, 2024, 10:39 AM

#

I only have three files. One with a typo:
havrest-secret; jwt-secret and a folder called ssl

#

It is even stranger. I do not see any service account in the gui. But when I try to create the nabox service account, it states: Service account already exists

jade lion Sep 17, 2024, 2:33 PM

#

And I cannot log into grafana anymore...

#

Any help would be appreciated

#

I also see that the container name is havrest

severe ferry Sep 17, 2024, 2:37 PM

#

Available for a remote session ?

#

havrest is the REST frontend that is wrapped around Harvest so that's normal

#

In grafana, that's the url where you should be able to see service accounts and there should be one for nabox : /grafana/org/serviceaccounts

#

and in nabox you should have one token created nabox

jade lion Sep 17, 2024, 2:43 PM

#

severe ferry `havrest` is the REST frontend that is wrapped around Harvest so that's normal

On a closed site....

severe ferry Sep 17, 2024, 2:44 PM

#

I keep asking right ? 😄 ok you can send me screeshots of the service accounts page

jade lion Sep 17, 2024, 2:45 PM

#

First problem is: cannot log into grafana anymore. ssh and admin works fine

severe ferry Sep 17, 2024, 2:45 PM

#

what version of NAbox ?

#

sent you a DM btw

jade lion Sep 17, 2024, 2:46 PM

#

4.0.7

severe ferry Sep 17, 2024, 2:46 PM

#

ok cool, so your credentials are denied ?

#

You can reset password in nabox admin page, it should reset grafana password as well

jade lion Sep 17, 2024, 2:48 PM

#

Yeah: invalid username or password. I did a reset, still no access to grafana.

#

Will try again

severe ferry Sep 17, 2024, 2:48 PM

#

watch journalctl -fu naboxd while doing it

jade lion Sep 17, 2024, 2:51 PM

#

Sep 17 1458 naboxd[1454]: time=2024-09-17T1458.928Z level=ERROR source=main.go:131 msg="unable to create Grafana token" error="{"extra":null,"message"
:"Invalid username or password","messageId":"password-auth.failed","statusCode":401,"traceID":""}\n"

#

That is the login error, will now try reset

#

Sep 17 14:51:57 vsa-gra-002 naboxd[3991]: logger=migrator t=2024-09-17T14:51:57.155594775Z level=info msg="Unlocking database"
Sep 17 14:51:57 vsa-gra-002 naboxd[3991]: logger=secrets t=2024-09-17T14:51:57.156037725Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
Sep 17 14:51:57 vsa-gra-002 naboxd[3991]: Admin password changed successfully
Sep 17 14:51:57 vsa-gra-002 chpasswd[4027]: pam_unix(chpasswd:chauthtok): password changed for admin
Sep 17 14:51:57 vsa-gra-002 naboxd[1454]: time=2024-09-17T14:51:57.233Z level=INFO source=password.go:71 msg="password changed" output=""
Sep 17 14:51:57 vsa-gra-002 naboxd[1454]: 2024/09/17 14:51:57 "POST http://vsa-gra-002/api/2.0/system/password HTTP/1.1" from 192.0.0.3:35848 - 200 30B in 324.075501ms
Sep 17 14:51:58 vsa-gra-002 naboxd[1454]: time=2024-09-17T14:51:58.954Z level=ERROR source=main.go:131 msg="unable to create Grafana token" error="{"extra":null,"message"
:"Invalid username or password","messageId":"password-auth.failed","statusCode":401,"traceID":""}\n"
Sep 17 14:51:59 vsa-gra-002 naboxd[1454]: time=2024-09-17T14:51:59.772Z level=ERROR source=auth.go:74 msg="authentication failed" errors="{"errors":["JWTAuthMiddleware: no
Authorization header","BasicAuthMiddleware: unable to read credentials"]}"
Sep 17 14:51:59 vsa-gra-002 naboxd[1454]: 2024/09/17 14:51:59 "GET http://vsa-gra-002/api/2.0/health HTTP/1.1" from 192.0.0.3:35848 - 401 107B in 103.832 s
Sep 17 14:52:08 vsa-gra-002 naboxd[1454]: time=2024-09-17T14:52:08.972Z level=ERROR source=main.go:131 msg="unable to create Grafana token" error="{"extra":null,"message"
:"Invalid username or password","messageId":"password-auth.failed","statusCode":401,"traceID":""}\n"

📎 message.txt

severe ferry Sep 17, 2024, 2:58 PM

#

dc exec grafana grafana cli admin reset-admin-password <new password> ?

jade lion Sep 17, 2024, 3:00 PM

#

yep, that did the trick. Can login again

#

Will look for the token now

severe ferry Sep 17, 2024, 3:00 PM

#

weird, 4.0.7 should do exactly that

#

well, actually Admin password changed successfully comes from grafana

#

what was the output of the successful password change ?

jade lion Sep 17, 2024, 3:02 PM

#

Paste it to the url you sent me.

#

Also a screenshot of the service accounts screen (nu users visible)

jade lion Sep 18, 2024, 3:31 PM

#

@severe ferry Thanks for the amazing support

lusty igloo Sep 24, 2024, 6:51 PM

#

trying to get this going on Azure and it fails to boot
after ~5minutes it just goes into a reboot cycle without actually ever starting

#

went through the same steps/process that I did to get 3.3 working. Converted the disk, uploaded, created new vm using the flatcar image, swapped the OS disk, started the system

severe ferry Sep 24, 2024, 8:06 PM

#

You're probably missing the data disk

#

use the vhdx image, make a new vm and attach a data disk, just had a report today that it works 😄

lusty igloo Sep 24, 2024, 10:24 PM

#

I created the second disk the same way and attached it.
This was with that attached, doing it without being attached yields the same problem.
Did they deploy in Azure? Should I just add a blank data disk instead of the one created by the ova?

severe ferry Sep 25, 2024, 6:20 AM

#

No that was actual Hyper-V. Here is the thing, it seems that Azure image format provided by Flatcar is VHD, not VHDX.
The problem is the VHD is much bigger and I don't have space to uncompress it and pack it. That's why there is no VHD image

lusty igloo Sep 25, 2024, 1:37 PM

#

I use qemu to convert the disk and then the powershell module "Add-AzVhd" to upload it

#

the unpacked image for disk1 is 8gb

#

qemu-img convert -p -f vmdk -O vpc -o subformat=fixed NAbox-disk1.vmdk NAbox-disk1.vhd
Add-AzVhd -LocalFilePath NAbox-disk1.vhd -ResourceGroupName rg-storage -Location eastus2 -DiskName NAbox_4.0.7_disk01 -DiskSKU Standard_LRS -DiskOsType Linux -DiskHyperVGeneration V1

lusty igloo Sep 25, 2024, 4:08 PM

#

hrm. getting a little farther.
Booted the disk on my hyper-v machine, shut it down after it was successfully at the login prompt.
Used hyper-v to convert to vhd, uploaded and it passed the point it was at last time.
Now it's stuck at
hv_netvsc 6045bdb5-ff8f-6045-bdb5-ff8f6045bdb5 eth0: Data path switched from VF: enP21176s1

going to keep poking at it, see what can be done

#

that seems to have gotten it. after it times out and does an emergency shutdown/restart it boots and everything seems to work.
i'll finish the documents and pass them over to you for revision/etc

severe ferry Sep 26, 2024, 8:46 AM

#

Don't use the VMDK as a base for Hyper-V, though, use the .zip

lusty igloo Sep 26, 2024, 6:32 PM

#

aye, it is what i was using for the second round

severe ferry Sep 26, 2024, 7:47 PM

#

I guess I'll need to make some space on my machine and use the actual azure image. Not sure why they don't provide it as vhdx

lusty igloo Sep 26, 2024, 7:59 PM

#

they charge you by gb used, so the bigger the better 😮

lusty igloo Sep 27, 2024, 6:53 PM

#

Is there any way to add another user or group or anything to nabox for the gui?

severe ferry Sep 28, 2024, 6:49 AM

#

Not tested, but if you create a system user it should work to authenticate on the web UI as well. I'm considering extending the LDAP configuration to GUI as well, OAuth could also be an option.

lusty igloo Sep 30, 2024, 4:37 PM

#

where are the ssl allowed ciphers/etc in the new version?
Accepted insecure ciphers: ECDHE-RSA-AES128-SHA256

also, our security team is reporting CSP misconfiguration failures
Port: TCP/443 (80)
Content-Security-Policy header mis-configured Affects:/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 520
Content-Type: text/html charset=utf-8
Date: Sun, 29 Sep 2024 11:14:39 GMT
Strict-Transport-Security: max-age=315360000 includeSubDomains preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1 mode=block
Connection: close

Resolution: Implement or reconfigure the Content Security Policy with source directives.

severe ferry Oct 1, 2024, 6:38 AM

#

You have /usr/share/nabox/traefik/dynamic/ssl.yaml and /usr/share/nabox/traefik/dynamic/hsts.yaml

#

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 is indeed accepted

lusty igloo Oct 1, 2024, 6:54 AM

#

yea, the ssl.yaml file contains that entry, but it's read only filesystem

severe ferry Oct 1, 2024, 7:12 AM

#

indeed it is. You can try this : https://upload.nabox.org/tymi-rase-wixy

lusty igloo Oct 1, 2024, 7:22 AM

#

pushing now. thanks
is this just the ssl changes?

severe ferry Oct 1, 2024, 7:25 AM

#

It's both

lusty igloo Oct 1, 2024, 7:41 AM

#

thanks.
verified the ssl one is fixed, waiting for sec team to rerun the rest of the scans to confirm the other.

severe ferry Oct 1, 2024, 7:47 AM

#

Excellent !

lusty igloo Oct 1, 2024, 9:47 PM

#

another question.
What is this interface and is it in use or can the ip range be changed?

3: br-27627dc19fd4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:95:02🆎49 brd ff:ff:ff:ff:ff:ff
inet 172.19.0.1/16 brd 172.19.255.255 scope global br-27627dc19fd4
valid_lft forever preferred_lft forever

severe ferry Oct 2, 2024, 7:45 AM

#

mmm... That's not supposed to be here.
do you have another one with inet 192.0.0.1/24 ?

#

docker network ls ?

lusty igloo Oct 2, 2024, 7:54 AM

#

docker network ls
NETWORK ID NAME DRIVER SCOPE
27627dc19fd4 build_default bridge local
e9468e9e41a5 host host local
af04ff016e87 nabox_default bridge local
6121605be548 none null local

#

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0d:3a:0d:ff:47 brd ff:ff:ff:ff:ff:ff
inet 172.25.1.38/24 brd 172.25.1.255 scope global eth0
valid_lft forever preferred_lft forever
3: br-27627dc19fd4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:95:02🆎49 brd ff:ff:ff:ff:ff:ff
inet 172.19.0.1/16 brd 172.19.255.255 scope global br-27627dc19fd4
valid_lft forever preferred_lft forever
4: br-af04ff016e87: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:34:3c:76:97 brd ff:ff:ff:ff:ff:ff
inet 192.0.0.1/24 brd 192.0.0.255 scope global br-af04ff016e87
valid_lft forever preferred_lft forever
inet6 fe80::42:34ff:fe3c:7697/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever

severe ferry Oct 2, 2024, 8:08 AM

#

ok you can remove it

#

try docker network prune

#

You might need docker builder prune before

lusty igloo Oct 2, 2024, 8:25 AM

#

ran both, that interface is showing gone. going to see if that fixed my issue.

#

aye, that seems to have done it.
no idea why it was there or where it came from

#

but thank you

#

is there any way to adjust the timeout on the gui

severe ferry Oct 2, 2024, 9:21 AM

#

Some versions were distributed with leftovers from the build process, that's were it came from

#

you want it longer or shorter ?

#

currently it should be 20 minutes

lusty igloo Oct 2, 2024, 2:22 PM

#

would be preferred if it was an option to set it to whatever you want, or disable it
not a huge thing

severe ferry Oct 2, 2024, 3:34 PM

#

And we're talking about NAbox UI timeout or does that cover grafana as well ?

lusty igloo Oct 2, 2024, 3:36 PM

#

nabox is what i was referring to

weary forge Oct 2, 2024, 8:09 PM

#

Hi @severe ferry I just created a new 4.0.7 instance and when I want to open the dashboard I have to many redirect. The URL looks like https://<fqdn>/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/grafana/dashboards

this is from the native URL and also from the links in the admin UI

severe ferry Oct 2, 2024, 8:22 PM

#

Mmmm, it does that if you go on « /grafana » or « /grafana/ » ?

weary forge Oct 3, 2024, 9:17 AM

#

Over the UI it is /grafana/dashboards

severe ferry Oct 3, 2024, 1:53 PM

#

Do you still have the issue ?

weary forge Oct 3, 2024, 5:45 PM

#

severe ferry Do you still have the issue ?

Me yes

severe ferry Oct 3, 2024, 6:12 PM

#

Can you do a curl -v https://ip/grafana/dashboards ?

weary forge Oct 3, 2024, 7:09 PM

#

interesting - <a href="<fqdn>/grafana/grafana/dashboards">Moved Permanently</a>

severe ferry Oct 5, 2024, 9:31 PM

#

Can you check GF_SERVER_ROOT_URL environment in Grafana container ?

weary forge Oct 6, 2024, 7:54 PM

#

can you provide me a short how to? 🙈

severe ferry Oct 7, 2024, 6:41 AM

#

dc exec grafana env|grep ROOT_URL

#

Should work

weary forge Oct 7, 2024, 9:01 PM

#

GF_SERVER_ROOT_URL=https://%(domain)s/grafana/

#

that looks fine

#

#

and URL looks like

severe ferry Oct 7, 2024, 9:05 PM

#

Mmm, did you actually try to delete web site data ?

weary forge Oct 8, 2024, 8:30 AM

#

This was from the private mode but good point, I will delete the data and also use another browser

weary forge Oct 8, 2024, 9:46 AM

#

same issue after deleting the web site data and with firefox

#

the login to the admin UI works it is only the grafana part

lusty igloo Oct 8, 2024, 1:50 PM

#

There doesn't appear to be any way to change the Datacenter/logical group after a system is added to NAbox.

lusty igloo Oct 9, 2024, 5:21 PM

#

Hey Yann, sorry to replay this, but the misconfigured CSP security alert came back on this weeks security scan, so it apparently has not been fixed

#

Content-Security-Policy header mis-configured Affects:/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 520
Content-Security-Policy: self
Content-Type: text/html charset=utf-8
Date: Sat, 05 Oct 2024 13:07:47 GMT
Strict-Transport-Security: max-age=315360000 includeSubDomains preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1 mode=block
Connection: close
Resolution: Implement or reconfigure the Content Security Policy with source directives.

severe ferry Oct 10, 2024, 6:53 AM

#

I might have forgotten the quotes around self.....

lusty igloo Oct 10, 2024, 7:00 AM

#

guess there isn't an easy way to edit that

severe ferry Oct 10, 2024, 7:26 AM

#

https://upload.nabox.org/tymi-rase-wixy

#

Your guess is right 🙂 it's on a read only part of the FS

lusty igloo Oct 10, 2024, 9:00 AM

#

thanks, updated and we'll see what the security scan shows next time it runs.
anything that can be done about the ability to edit the datacenter name after the ontap systems are created?
or do you just have to delete them and create new ones and lose existing data

severe ferry Oct 10, 2024, 1:53 PM

#

Whenever I looked into renaming metrics I think it ends up with reimporting the data somehow.

lusty igloo Oct 14, 2024, 8:29 PM

#

GRR, still there
Evidence for reopened
Port: TCP/443/80
Content-Security-Policy header mis-configured Affects:/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 520
Content-Security-Policy: self
Content-Type: text/html charset=utf-8
Date: Sat, 05 Oct 2024 13:07:47 GMT
Strict-Transport-Security: max-age=315360000 includeSubDomains preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1 mode=block
Connection: close

lusty igloo Oct 15, 2024, 2:23 PM

#

@severe ferry Sorry to keep bugging you about this, but I can't make changes as it's RO

severe ferry Oct 15, 2024, 3:19 PM

#

Mmm, it should be self with single quotes.

severe ferry Oct 15, 2024, 3:38 PM

#

Ok I have no idea about CSP really. Maybe the expeceted syntax is something like default-src 'self';

lusty igloo Oct 15, 2024, 5:41 PM

#

honestly don't know myself, i'll have to look into it to see what the error is actually being reported

lusty igloo Oct 16, 2024, 3:02 PM

#

@severe ferry sent you a DM about the CSP stuff, not sure you get them or not

#NAbox 4.0.7 is available