pulsar timber
We encountered multiple issues after FIP mode got enabled. One of issues was not able to ssh into the cluster console. Then, I was told that they had to go onsite and disable FIP before they can get SSH connection back.
Did we really have to go onsite to disable FIP through local console? Anyway to get ssh back without through local console?
terse pelican
When we ran into this in a lab setting, we used the service-processor to get back in.
pulsar timber
Why would SP work? Isn't use ssh connection to the cluster as well?
old token
SP is serial
terse pelican
At least at the time (would have been 9.11.something on an AFF-A300) the fips setting didn't appear to apply to it. We had other reasons to disable fips, so I can't confirm now.
pulsar timber
I am looking for any docs as to what wouldn’t work after enabled FIPS. I was told nothing would work, any IP connections, SP, SSH etc. They had to hook up local serial port. I doubt what they said. But we changed it back to be disabled. We are on 9.11. Serial port but uses SSH. I was told all ssh wouldn’t work, including SP
distant nova
@pulsar timber All you probably needed was an updated ssh client.
pulsar timber
<@1034216947979665448> All you probably needed was an updated ssh client.
Why is that and what exactly was required to do on the client?
distant nova
I had a scenario where the customer was using an older putty client (0.67 if I recall). It would not work. Updated to the latest (at the time, I think it was 0.71). There were updated key exchanges and ciphers and it worked. So it may just be your ssh client not being up to date
old token
yeah, the ciphers or HMACs can make all the difference. Often the client either doesn't support the required ones, or it has older ones (that are still mandated by the server) disabled by default, things like that. But usually the erroro messages in those cases are pretty clear and point you into the right direction
pulsar timber
After FIPS was enabled, essentially, the cipher suites got restricted. As the result, the previous public/host key on the client may not work, and not able to open new ssh sessions. But.
The ONTAP should generate a new public key and prompt me to add it into known_hosts, in orcder to open new ssh sessions. Correct?
Those existing ssh session should be alive, and won't be terminated. Correct?
old token
I thought you need to reboot all nodes after enabling FIPS? That would solve your (2) ... as for (1), I don't knonw if it will automatically create new host keys if the ciphers don't match anymore or not. Your user's public-keys are probably more at risk of being affected, e.g. if you're using ed25519 keys I think those won't work anymore with FIPS?
distant nova
ONTAP 9.9.1+ no longer requires a reboot for FIPS enablement
pulsar timber
Correct, ONTAP 9.9.1+ is not required to reboot for FIPS enablement.
(1) The cluster did create and prompt me to add a new public-key into known_hosts when I tried to ssh in first time after FIPS was enabled. After I did as instructed, I successfuly ssh'ed in. However, around that time, a co-worker of mine was in the DataCenter, and disabled FIPS via local console. So, I am not certain that the success was before he disabled FIPS or after.
That's why I would like to find out if it makes sense to you that we didn't actually need to go onsite, and all we needed to do was just to update the key on the Linux client?
(2) If I was already in a ssh session, I couldn't think of why it would get terminated. But, just to confirm with you guys?
pulsar timber
I have found a solution to be able to ssh into the cluster afte FIPS gets enabled, and without having to go through local console. The following is KB. Basically, we need to reconfigure SSH public key authentication with supported host key encryption BEFORE enable FIPS:
distant nova
I turn FIPS on every install. The only time I’ve ever had an issue is with a really old ssh client and/or a windows box that hasn’t been updated.
old token
I turn FIPS on every install. The only time I’ve ever had an issue is with a rea...
That's interesting! We have an install base of over 1500 systems with our customers and I can think of only 1 or 2 where FIPS is activated
distant nova
Nearly all my installs are Fed. Nearly mandatory to enable 😁
old token
yeah, makes sense 🙂
pulsar timber
We are not government agency, not required to be FIPS compliant. So, it should not be enabled at beginning with. The manager just feel more secure to use it
distant nova
It’s good to turn on to mitigate the usage of less secure methods over SSL and ssh
old token
FIPS actually disables one of the most secure encryption algorithms, namely Ed25519. Some people say it is because the government was unable to put a backdoor in it so they don't allow it 😉
pulsar timber
It broke up quite some things here, SNMP didn't work, no longer able to use Chrome and Edge, Firefox only(still not fixed after enabled), some our applications still using SnapCreator(and still don't know why)
@old token Where can I found the reference on that FIPS actuall disables ED25519? May be the reason to present my manager not to enable.
fresh tiger
NMAP lookup of supported algorithms pre and post fips enablement in NetApp Release 9.11.1:
== Pre-FIPS enablement: ==
PORT STATE SERVICE
22/tcp open ssh
| ssh2-enum-algos:
| kex_algorithms: (7)
| diffie-hellman-group-exchange-sha256
| diffie-hellman-group-exchange-sha1
| diffie-hellman-group14-sha1
| ecdh-sha2-nistp256
| ecdh-sha2-nistp384
| ecdh-sha2-nistp521
| curve25519-sha256@libssh.org
| server_host_key_algorithms: (4)
| ssh-rsa
| ssh-dss
| ecdsa-sha2-nistp256
| ssh-ed25519
| encryption_algorithms: (9)
| aes256-ctr
| aes192-ctr
| aes128-ctr
| aes256-cbc
| aes192-cbc
| aes128-cbc
| 3des-cbc
| aes128-gcm@openssh.com
| aes256-gcm@openssh.com
| mac_algorithms: (19)
| hmac-sha1
| hmac-sha1-96
| hmac-sha2-256
| hmac-sha2-512
| hmac-sha1-etm@openssh.com
| hmac-sha1-96-etm@openssh.com
| hmac-sha2-256-etm@openssh.com
| hmac-sha2-512-etm@openssh.com
| hmac-md5
| hmac-md5-96
| hmac-ripemd160
| hmac-ripemd160@openssh.com
| umac-64@openssh.com
| umac-128@openssh.com
| hmac-md5-etm@openssh.com
| hmac-md5-96-etm@openssh.com
| hmac-ripemd160-etm@openssh.com
| umac-64-etm@openssh.com
| umac-128-etm@openssh.com
| compression_algorithms: (2)
| none
|_ zlib@openssh.com
== Post FIPS enablement: ==
PORT STATE SERVICE
22/tcp open ssh
| ssh2-enum-algos:
| kex_algorithms: (6)
| diffie-hellman-group-exchange-sha256
| diffie-hellman-group-exchange-sha1
| diffie-hellman-group14-sha1
| ecdh-sha2-nistp256
| ecdh-sha2-nistp384
| ecdh-sha2-nistp521
| server_host_key_algorithms: (1)
| ecdsa-sha2-nistp256
| encryption_algorithms: (9)
| aes256-ctr
| aes192-ctr
| aes128-ctr
| aes256-cbc
| aes192-cbc
| aes128-cbc
| 3des-cbc
| aes128-gcm@openssh.com
| aes256-gcm@openssh.com
| mac_algorithms: (8)
| hmac-sha1
| hmac-sha1-96
| hmac-sha2-256
| hmac-sha2-512
| hmac-sha1-etm@openssh.com
| hmac-sha1-96-etm@openssh.com
| hmac-sha2-256-etm@openssh.com
| hmac-sha2-512-etm@openssh.com
| compression_algorithms: (2)
| none
|_ zlib@openssh.com
fresh tiger
Don't think we document it other than to say, "we follow the FIPS standard."
old token
e.g. here, page 17:
https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2472.pdf
Remove or do not configure Ed25519 host keys as Ed25519 keys are not allowed in FIPS Approved
mode. The crypto office should inform the user to not use Ed25519 keys for user authentication.
pulsar timber
@fresh tiger,
From Insight in "System Manager" GUI, there is a recommendation below. It made people here beleived it'd better to enable FIPS. That's why they said we did so to follow up NetApp's recommendation and to secure the storage better. They thought it was just a simple enablement, without recognizing all the impact. And plus, to rollback is not just a simple disablement. They had to go to DataCenter to disable it because SSH didn't work.
` Global FIPS 140-2 compliance is disabled
Global FIPS 140-2 compliance is disabled on this cluster.
For security reasons, you should ensure ONTAP communicates with external clients or server components outside of ONTAP by using SSL communication that uses FIPS 140-2 compliant cryptography.`
distant nova
FIPS will disable snmpv1 and you are forced to used snmpv2/3. I think it actually displays that before enabling
fresh tiger
<@1008761043843035277>,
From Insight in "System Manager" GUI, there is a recomm...
Ugh! I'll look into getting that verbiage improved.
fresh tiger
FIPS will disable snmpv1 and you are forced to used snmpv2/3. I think it actuall...
cluster1::security config*> modify -interface SSL -is-fips-enabled true
Warning: This command will enable FIPS compliance and can potentially cause some non-compliant components to fail. MetroCluster and Vserver
DR require FIPS to be enabled on both sites in order to be compatible. An SNMP users or SNMP traphosts that are non-compliant to
FIPS will be deleted automatically. An SNMPv1 user, SNMPv2c user or SNMPv3 user (with none or MD5 as authentication protocol or
none or DES as encryption protocol or both) is non-compliant to FIPS. An SNMPv1 traphost or SNMPv3 traphost (configured with an
SNMPv3 user non-compliant to FIPS) is non-compliant to FIPS.
Do you want to continue? {y|n}:
pulsar timber
The one enabled FIPS via the button of "Fix It" in Insight not via CLI, but, I assume he should see the same warning message as you pasted here. He may miss it. We spent a cople of days finally fixed SNMP issue.
One of outstanding issue is although we've already disabled it and reboot all nodes, we still cannot use MS Edge, we could before. Then the following link sounds like we will have to ugrade to 9.12.1. For now, we can only use Firefox.
https://kb.netapp.com/onprem/ontap/dm/System_Manager/ONTAP_System_Manager_9.11.1__shows_"ERR_SSL_KEY_USAGE_INCOMPATIBLE"_due_to_unsupported_cipher_suite
Since you know all details about the issue. Is there anyway we can use Edge before we next upgrade?
fresh tiger
Ah! My expertice runs out when we start talking about browsers. However, I do know someone who is very knowledgeable on them. Might take a day or so. Suspect a solution will either be a tweak to your certificates, or a change in the brower.
pulsar timber
That’d be great, if you could have someone to look into it. Thanks!
One more thing, did you see the same warning if you enabled it on your lab system through Insight (not CLI)?
fickle crag
The one enabled FIPS via the button of "Fix It" in Insight not via CLI, but, I a...
I have some lab systems that are running OnTap 9.8 with FIPs enabled, and I don't have any issue accessing System Manager on them using Chrome or Edge (when you say "Insight", I assume you mean System Manager?) I would agree with Clint that it sounds like it may have something to do with your certificates.
Are you using a signed cert or self-signed?
If you have a signed cert, if you run a "security ssl show", is it pointing to the signed cert for the admin vserver?
pulsar timber
I have some lab systems that are running OnTap 9.8 with FIPs enabled, and I don'...
So, based on the result below, since CN is not fully qualfied name, my understanding is, we are using self-signed, right?
::> security ssl show -vserver eos
Vserver: vservername
Server Certificate Issuing CA: vservername
Server Certificate Serial Number: xxxxxxxxxx
Server Certificate Common Name: vservername
SSL Server Authentication Enabled: true
SSL Client Authentication Enabled: true
Online Certificate Status Protocol Validation Enabled: false
URI of the Default Responder for OCSP Validation:
Force the Use of the Default Responder URI for OCSP Validation: false
Timeout for OCSP Queries: 10s
Maximum Allowable Age for OCSP Responses (secs): unlimited
Maximum Allowable Time Skew for OCSP Response Validation: 5m
Use a NONCE within OCSP Queries: true
fresh tiger
The one enabled FIPS via the button of "Fix It" in Insight not via CLI, but, I a...
That's the Insight message. My role is to improve things that make a customer's job harder than it should be. Think this message falls into that category. I will be working with the SysMgr team to improve this.
Haven't yet been able to catch up with my certificate/browser resource.
fickle crag
So, based on the result below, since CN is not fully qualfied name, my understan...
Correct. If your Issuing CA is the vserver (or, vserver.cert), then it is a self-signed cert.
You really should use a signed cert for security. If you're enabling FIPs, then you should be using a signed cert.
pulsar timber
This message is really helpful.
NP, I will be watching out his/her response here. Thank you!
lament rain
@pulsar timber I was asked to take a look at this. Can you send me the public key for the certificate you are using ?
pulsar timber
<@1034216947979665448> I was asked to take a look at this. Can you send me the p...
Public key for the cert? How could I find out?
lament rain
Do you have it installed in ONTAP ?
pulsar timber
Are you asking for the one with my admin account? I am not sure I understand what you are asking for.
fresh tiger
Yes, it's installed in ONTAP. Do you mean the output from "security certificate print" command Gregg?
lament rain
That output would require the public key to be pasted in, but yes that output is what I need
fresh tiger
Oh. I just got it to work. Hold on while I groom the text and I'll paste an example.
::*> security certificate show-generated
Vserver Serial Number Certificate Name Type
svm1 17ABA804E66ECEF8
svm1_17ABA804E66ECEF8 server
Certificate Authority: svm1
Expiration Date: Sat Jan 18 00:20:22 2025
::*> security certificate print -vserver svm1 -cert-name svm1_17ABA804E66ECEF8
Certificate details:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1705641622889221880 (0x17aba804e66ecef8)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=svm1, C=US
Validity
Not Before: Jan 19 05:20:22 2024 GMT
Not After : Jan 18 05:20:22 2025 GMT
Subject: CN=svm1, C=US
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Snipped off the remainder.
lament rain
That would be the output I need
fresh tiger
@pulsar timber What version of ONTAP are you using? My output is from 9.10.
pulsar timber
If I run "security certificate show-generated", it only shows me two unrelated vservers, not the admin vserver or cluster vserver.
Not sure of how should I run "security certificate print -vserver svm1 -cert-name svm1_17ABA804E66ECEF8" though. Not sure of what cert-name is, but I tried the following, and it asks me to enter public cert which I don't know what it si
::> security certificate print -vserver cluster-vserver-name
Enter the public certificate in PEM format: Press <Enter> when done
By the way , I am using self cert
lament rain
What version of ONTAP are you running ?
fresh tiger
Try "security certificate show-user-installed"
pulsar timber
9.11.1p11
lament rain
@pulsar timber Are you able to join a Zoom session ?
pulsar timber
Yes,
lament rain
Issue has been resolved. Thank you for your patience @pulsar timber !!!
pulsar timber
Thanks a lot for your help!!!
pulsar timber
@lament rain @fresh tiger Would this new installed SSL certificate which was applied onto mgmt IP:por 443 has any impacts on using SnapCreator which is a ZAPI based tool? I am not sure of what Cluster's IP or ports has it used for communications with the cluster. We have some schedule SC jobs running tonight
gilded verge
Ugh! I'll look into getting that verbiage improved.
Clint, I submitted an RFE for that last Friday to have system manager match the cli verbiage. ping me if you can't find it.
lament rain
<@211746462223040512> <@1008761043843035277> Would this new installed SSL certif...
That is up to the client. Some clients may require a CA signed certificate that also has Subject Alternative Name in the x509 extensions. That being said, the cert we whipped up yesterday is certainly better suited than the one you had.
pulsar timber
👍
pulsar timber
Clint, I submitted an RFE for that last Friday to have system manager match the ...
The verbiage needs to be improved, in my opinion, is those from System Manager Insight, as I copied & pasted above. After people read those made them to believe NetApp recommended us to enable FIPS anyway whereas it really should be enabled for Goverment agency's as my understanding now.
gilded verge
The verbiage needs to be improved, in my opinion, is those from System Manager I...
This one? As far as I know FIPS is not just government compliance although that is where it's seen the most. There's no reason why non government adjacent business should not use it. Updating the warning when enabling it should help prevent issues where insecure protocols are disabled.
pulsar timber
I agree with you on that it is not just used by goverment agencies. However, most of organizations (non-goverment agencies) don't use it, based on what I googled, my working experiences, and also as @old token indicated as well. That was what I am trying to say.
I don't know what is the best word to put, but, what I was trying to say is that the word "you should ensure" made people to believe something could be broekn, went wrong if you didn't enable. It sounds encourated people to "Fix It". That was why people here went ahead enabled it without fully examing the risk. It should really just say something like "NetApp is FIPS compliant", and then let customers to decide on what to do next, instead of telling them what to do.
After we enabled it, applications using SnapCreator, all tools using SNMPv1v2 started to fail. Plus, lost all SSH connections. Also, coicidently Edge browser couldn't connect to System Manager right after the change. We are kind of all know causes of these failure after the fact. So, it is not just as easy as just hit the "Fix It" as it sounds like in this Insight screen page.
distant nova
For new systems, I just turn it on. Nothing at all to break. Ssh not working? Update the ssh application. GUI not working? Ssh in and rebuild the ssl cert. as I indicated earlier, SNMP is forced to be more secure. No more v1 and v2/v3 must be used and secured
pulsar timber
I don't believe it was due to old putty version . It looks like the cipher suite got changed by FIPS, and no longer work with the old public key.
distant nova
Very true. More secure. Just need to update…something whether that’s the ssh application or the keys. Sounds like the old key was weak and hence the failing. I generally try to make keys with 2048 bits anymore
pulsar timber
The only thing we still can’t explain is why SnapCreator got broken after enabled
distant nova
Was it using a cert built with weak credentials?
Didn’t that use SSL?
Maybe a weak password?
gilded verge
i'm not in that space, but i consulted on a case where the snapx was somehow stuck using HTTP. that will not work with FIPS of course. it needed to be configured to use HTTPS instead.
old token
Very true. More secure. Just need to update…something whether that’s the ssh app...
"more secure" is debatable I'd say. Just because an algorithm is certified by some US agency doesn't make it "secure". There are lots of crypto algos that have been shown to contain backdoors in the past. Most of our customers actually use Ed25519 because it is more secure... So yeah, debatable at least 😉
also, FIPS certification is slow and/or expensive, so the used cipher versions/libraries are often lagging behind by years (and missing critical fixes, because fixing the code would require a re-certification)
pulsar timber
@old token when you say “missing critical fixes”, can you please show me an example as to what concrete impacts on customers as the result? About “slow”, you didn’t mean performance wise, right?
distant nova
Pretty sure by slow he means the ssh handshake. It does take a couple extra seconds with FIPS mode
old token
The process to get FIPS certification works roughly like this: You take the openssl library, say version 1.0.3, and get it certified. The certification process takes maybe a year or so, then you know that your 1.0.3 version is FIPS certified and that you can use it. In the meantime, the OpenSSL version has had some critical fixes and updates to version 1.0.6 or maybe even 1.1 or 2.0. But you cannot use these versions in FIPS mode as they are not (yet) certified.
That's what I mean by "slow": the openssl version you are running is always lagging behind substantially. Security scanners will notice that and bother you that your OpenSSL version is too old. If there were critical flaws detected in that version 1.0.3 (side-channel attacks etc.) those cannot get fixed until someone invests time and money to get a newer version of the openssl library certified.
pulsar timber
@old token 👍 After read more artcles about FIPS, I more believed in enabling FIPS is a disputable topic.