┊・security

[heavy jackal]

Hello!

Anyone got a good way of parsing the NetApp Security Advisory web page or RSS feed? https://security.netapp.com/advisory/

[celest orchid]

Something separate from the RSS feed it has?

https://security.netapp.com/advisory/index.xml

[heavy jackal]

I think I got it now.

[shut perch]

Anybody have any suggestions for how to securely allow access to ONLY netapp (and a handful or other) discord channels? It seems that if we allow the netapp url, a user can then navigate to all urls or even set up their own discord server. This makes discord horribly insecure compared to slack or reddit and not all enterprise friendly.

[fair perch]

Hi Ed, that’s something you’d need to ask discord support. At a guess, you’ll probably need a layer 8 control.

Ref https://en.m.wikipedia.org/wiki/Layer_8

[celest orchid]

How is Reddit considered secure in this context?

[amber oar]

Assuming because they can restrict access to a specific URL aka a specific subreddit.

[celest orchid]

Not without breaking https url encryption

[amber oar]

True, login process might get borked.

[shut perch]

Yup, we restrict which subreddits we are prepared to allow through (and which slack channels). We have not found a way to restrict discord, so it's simply not allowed. We can't allow somebody to access cloud services and then upload proprietary information. By NetApp migrating to discord, NetApp has essentially removed access for some customers. I get access only through my personal system.

[unreal idol]

@shut perch I can absolutely promise you that all of this feedback has been heard loud and clear, for months now. We've made a decision to center activities and public community presence around Discord. We won't be moving back to Slack. If anything, this space will only grow and grow even more.

[feral dust]

aaah, so NetApp management is also clairvoyant now... neat

[celest orchid]

Yes.

[shut perch]

so they're basically saying they know Discord is not enterprise friendly but decided to do it anyway 🙂

[amber oar]

Yes.

[celest orchid]

If your security policy effectively performs your own mitm, then what happens if you allow this server's URL: https://discord.com/channels/855068651522490400/ *

[shut perch]

Our proxy does indeed perform as a mitm. Every bit of experimentation our security team has done shows that if we allow just one channel, once the users are in discord through that channel, they can then get to other channels (and even set up their own).

[celest orchid]

https://www.theregister.com/2022/09/16/uber_security_incident/

On the topic of security 👀

[tired hill]

https://tenor.com/view/hackers-jonny-lee-miller-diskette-gif-7105732

[trim glade]

yooooo i loved those colored floppies.

https://giphy.com/gifs/14kdiJUblbWBXy

[unreal idol]

I liked the colored clear ones personally

[trim glade]

ooooooooh ya good call

[high aspen]

I actually got some music on a sweet 3 and 1/4 inch a coupla months back. had to break out a usb floppy drive I rescued from being a monitor stabilizing mass. 😄 Cool tunes tho.

[fair perch]

MOD files? 😉

[velvet pumice]

If somebody can please help me on my question about Anti-Rasomware. If I use CryptoSpike or any this soft of software as an example, I could use it to integrate with ONTAP Fpolicy and then to protect NAS shares, as my understanding. My question is, how CryptoSpike can protect files on VM's? Will I have to install the agent on VMs?

[viscid pendant]

It's simple: It won't protect files on VMDKs. It will only protect files directly saved on NAS shares provided by ONTAP.

[arctic hawk]

Like OG1 said, we don't have visibility into the VMDKs. It is a foreign file system.

[velvet pumice]

Then how can we protect files on VM's from Ansomware? Those are going to be a lot of exposed files if it cannot protect. Or, there are some other vendors can do that?

[tired hill]

something that can run on the windows desktop or server. MS Defender

[velvet pumice]

OK. If that's the case, then those sorts of Ani-Rasomware software(CryptoSpike, SnapGuard etc) mainly are for the environment of largely using NAS shares. Right?

[unreal idol]

I can't speak for either of those pieces of software, but what you're describing is why we always kept desktop images so ephemeral, with nothing saved or stored on local drives (via group policy). This way, whatever we ran on the NAS was enough because it housed the user home drive and departmental shares.

If they click on the wrong thing, who cares if the C drive gets nuked, we just reimage the machine in 20 minutes and they're back in business.

[velvet pumice]

Got it, which means we need to set up an infrastructure, or architecture to have NAS shares housed user's and most of business data possible, only then Ransomware protection can be applied well. Thank you all!

[fair perch]

Well, at storage level. Defence in depth suggests you should also look at host side options for VMs

[velvet pumice]

Okay. Initially, I thought those Anti-Rasonmware software I mentioned above can do everything. Ture, they can integrate Fpolicy to scan/protect NAS shares. Now I understand they cannot scan / protect files on VM's, and we have to look at the other options for VMs.

[charred heart]

Has anyone managed to implement 2FA for SSH sessions using IWA (domain-tunnel) authentication?

[west lake]

@charred heart for ONTAP 2FA for SSH only supports local users, so domain-tunnel is not supported.

[charred heart]

Yeah, I'm aware of the local account limitation but thought I'd ask anyway 🙂

[viscid pendant]

Hey Mr. Security Evangelist @west lake, since you're already here... 😉
What's your stance on Vscan? Would you still recommend it to new customers? Is there any ongoing development in that area ONTAP-wise?
For me it seems like all the other cool new security features (ARP, MAV, Cloud Secure, etc.) are getting the focus and it doesn't look like there is much "innovation" on the vendor-side either regarding their storage scanning solutions. My feeling is more and more vendors are deprecating their solutions (like Sophos).

[west lake]

Hi @viscid pendant, I think it still has a place in the layered defense approach to ransomware. For example, you could use it to scan file shares for malware even if that malware is currently not executing. Say you did a file migration through some tool like robocopy and it happened to transfer some malware executables, but no one ever ran it. This would prevent someone from running it in the future where ARP and Cloud Secure may notice and do something about it, but only after it's started to at least encrypt a few files. It's very useful still for completely preventing any malware encryption for signatures it can recognize.

[subtle talon]

We recently upgraded to OnTap 9.8, and we are now getting these messages in AIQUM:

• Ciphers with the suffix CBC are considered insecure.
• To remove the CBC ciphers, run the ONTAP command
security ssh remove -vserver <vserver name> -ciphers aes256-cbc,aes192-cbc,aes128-cbc,3des-cbc

In doing some research, it seems like this is safe to do, as long as your client supports one of the more secure ciphers. But, the most important question is, what is this going to break? Is SSH ever used for anything other than logging into the cluster or SVM to manage them? The one article did say that Snapmirror requires PSK to work, so I'm assuming there are other internal mechanisms that could break.

This also seems to be a SVM level command. Shouldn't the SVMs inherit the ciphers set at the cluster level using the "security config modify -supported-ciphers" command? Or, do the SVMs only inherit these when they are created?

The security hardening guide (TR-4569) recommends using this for the most secure connections. Can this (should this) be used for the "security ssh" settings as well?
security config modify -interface SSL -supported-protocols TLSv1.2 -supportedciphers PSK:DHE:ECDHE:!LOW:!aNULL:!EXP:!eNULL:!3DES:!kDH:!kECDH

[neon pivot]

I'm looking to integrate Cluster Data ONTAP system with Azure Sentinel. Could you please guide me through the steps. Thanks in advance

[arctic hawk]

Dumb question, what is Azure Sentinel used for?

[neon pivot]

they also call it as Microsoft Sentinel which is a cloud native security information and event manager and BTW is not a dumb question

[arctic hawk]

I haven't heard of it before. Is it a FPolicy product?

[west lake]

HI Jim, it's always recommended to test changes like this in your environment first to ensure everything continues to work, but in general SSH is only used for management from the CLI so I can immediately think of something breaking. However, like I said better to test first with some sort of lab environment so you don't have any surpises. Snapmirror requiring PSK as a cipher is related to changes to SSL (TLS) configuration and ciphers and not affected by SSH cipher changes. For SSH ciphers, vservers inherit them from the cluster at vserver creation time, so if you create the vserver after then it will take the cluster settings, but if it was created before the change you'll need to modify the existing vservers as well. Note this is not true of SSL configuration changes as it only applies to SSH. The hardening guide command you mentioned only applies to SSL (TLS) configuration and not SSH.

[subtle talon]

Thank you, Matt. That is the validation I was looking for. We will definitely implement these in the lab first (which we have SP access to, and worst case, I can get physical access to and console into if need be).

[west lake]

You can export file auditing logs from ONTAP in EVTX format and there should be a way to import them into Sentinel. FPolicy doesn't appear to have any native integration with Sentinel.

[pulsar knot]

Is there any intention for Netapp to have more formal support for hardware // software based MFA for the CLI outside of just private keys for local accounts?

[neon pivot]

I don't think it is

[pulsar knot]

https://cloud-netapp-com.cdn.ampproject.org/v/s/cloud.netapp.com/blog/azure-blg-azure-monitoring-best-practices-and-tools?amp_gsa=1&amp_js_v=a9&hs_amp=true&usqp=mq331AQKKAFQArABIIACAw%3D%3D#amp_tf=From %251%24s&aoh=16667324730822&referrer=https%3A%2F%2Fwww.google.com&ampshare=https%3A%2F%2Fcloud.netapp.com%2Fblog%2Fazure-blg-azure-monitoring-best-practices-and-tools

Not much else out there yet it seems

[neon pivot]

Yes Jesse. I don't find any info on it

[fair perch]

Hi Jesse, as a forward looking statement, that would have to be answered by your account team under NDA, sorry! We are aware of customer demand, in particular USA GSA's implementation of Executive Order 14028

Examples of requested MFA options are Yubikey, Duo and TOTP.

[pulsar knot]

Yeaaaah I've badgered them a lot of the last 2.5 years with this, I'll ask again 😉

I've found homebrew-ish solutions that utilize pageant and an open source pgp implementation to middle-man the auth, but where I don't mind utilizing something like that my colleagues and peers are a bit less willing.

[arctic hawk]

Yes please check with your account team as there is obviously movement on this and has to be per the order.

[spare patio]

If I get Ransomware and have on box ML turned on, a snapshot is taken immediately. Does that snapshot also protect storage tiered out to S3 like SG or the cloud?

[arctic hawk]

Coffee isn't working...ML?

[spare patio]

machine learning

[arctic hawk]

Oh the ransomware detection feature. I'd have to look and see. Sometimes snapshots can be deleted, but if the feature locks snaps, it doesn't care if it's warm or cold tiering.

I'd have to check on how that feature locks snapshots.

[spare patio]

Oh cool. Tiers are covered. Excellent.

[arctic hawk]

https://www.netapp.com/media/7334-tr4572.pdf page 8

If abnormal activity is detected, an automatic Snapshot copy is immediately taken, which provides a
restoration point as close as possible to the file infection. Simultaneously, an automatic alert is generated
that allows administrators to see the abnormal file activity so that they can determine whether the activity
is indeed malicious and take appropriate action. Or, if the activity was an expected workload, they can
easily mark it as a false positive; the anti-ransomware ML notes the change in workload and no longer
flags it as a potential attack. In addition, the feature does not disrupt I/O in any way. Instead, it provides
administrators with native analytics, insights, and data recovery capabilities for unprecedented on-box
ransomware detection. The anti-ransomware feature makes it easier than ever to enable automatic
ransomware detection for your NAS workloads in ONTAP.

The only downside is there are ways to delete snaps I believe from things like the previous versions tab of Windows Explorer if you have enough rights, or VSS.

[spare patio]

Glad the snapshot also covers the S3 tier

[arctic hawk]

Ideally you'd want snaplock.

[spare patio]

SnapLock can lock the snapshots on the mirror destination but can't lock snapshots on the source without it affecting production volumes

[arctic hawk]

Right, but you'd have that backup. 🙂

[spare patio]

Yes, so long as the file that was snapshotted on the source were replicated before the attacker deleted the source snapshots

[arctic hawk]

Well it also sends an alert, so in theory you could force a SnapMirror update and lock things down.

[spare patio]

Could have thousands of files encrypted before a human reacts

[placid hound]

On the topic of Anti-ransomware protection, please attend Insight for all the updates! You're going to love what we've been up to!

[verbal wind]

Regarding https://security.netapp.com/advisory/ntap-20220609-0008/ "May 2022 Libcurl Vulnerabilities in NetApp Products"
Does ONTAP actually use libcurl to make external calls? Or is this just listed because curl command exists in underlying OS?

[west lake]

For any vulnerability specific information that is not present in the CVE it's best to contact NetApp support and open a case.

[arctic hawk]

Agreed. This is a fairly new update it appears.

[viscid pendant]

Question regarding the tamperproof snapshots: How does that work with "normal" snapshots created after a SnapLock snapshot? Usually when I restore to a certain snapshot all the snapshots having been created after that snapshot will be gone. But if we have that one tamperproof snapshot it shouldn't allow that... right?

Volume     - 2022-10-06 (current date)
Snap_Daily - 2022-10-05
Snap_Daily - 2022-10-04 (tamper-proof snapshot)
Snap_Daily - 2022-10-03
Snap_Daily - 2022-10-02 --> I will restore to this snapshot
Snap_Daily - 2022-10-01

[west lake]

It sounds like you are talking about a volume snap restore on the primary volume. If it's the latest snapshot you are restoring from, then it will work. If it's an earlier "tamp-proof" aka locked snapshot then the restore fails and you have to restore by either creating a flex clone of the snap you want to restore from or use single file snap restore.

[viscid pendant]

ok, that's what I thought

thanks!

[viscid pendant]

In one of the presentations it was mentioned that MFA for SSH is being enhanced in ONTAP 9.12.1. What's the news here and what's changing from the current way with public keys and passwords?

If I putty to the system what will the new 2nd factor be?

[arctic hawk]

There may be a separate presentation on it.

[viscid pendant]

I will await patiently 🫡

[west lake]

Yubikey hardware tokens (by Yubico) are now supported using either PIV (personal identity verification) or FIDO2 (fast identity online). Also for SAML 2.0 using System Manager Cisco DUO is now supported

[arctic hawk]

This was what I was waiting to say something about.

[west lake]

Cool, I talked about it in my Cyber Resilient session number 1008

[arctic hawk]

Thanks Matt. I missed it. I had other things to do so I missed a few of these sessions

[west lake]

No worries, thanks Paul.

[arctic hawk]

Yw

[viscid pendant]

this sounds awesome!

FIDO2 for the System Manager side would be even more awesomer

[high aspen]

dude cool. I'll have to dig up my old yubikey 🙂

[high aspen]

fully agree, OG1.

[odd bison]

I can't seem to be able to access ONTAP SPI (https://cluster/spi) with a domain account used for managing ONTAP, is that supposed to be working? What permissions am I probably missing?
If it can only be accessed through local accounts; what are the least permissions needed?

[west lake]

That's a strange one as it should be working. Do you get any error at all? Also, are you positive it has the admin role on the cluster level? This will be needed.

[grim whale]

Take a look at these commands
Security login rest-role (verify you have the access)

Security login create -user domain\group -auth domain -app ontapi

Maybe also take a look at the event log. It may show something useful

[odd bison]

HTTP Error 500, and yes the account is absolutely Cluster level admin.
I did some experimenting and found that as long as a local account (-application http) has a role with "services web access" for "spi" it gets access to SPI, it doesn't need admin role

[odd bison]

Good idea! I actually see this there every time I try to login:
query.execution.failed: query="SELECT * FROM asup_cluster_info", error="The user is not authorized for SQL access."

ONTAPI access is not relevant for SPI, my local account doesn't have it and can access SPI

[grim whale]

I misread that. I thought I saw API!

[odd bison]

Any idea why SQL access isn't given to domain accounts (through group) but local accounts without access gets it?

[dusky stump]

Hi

I'm looking for a vulnerability scanner, that includes, linux containers, netapp and vmware.

[west lake]

Thanks, for the details. I did some more looking internally and didn't come up with much very helpful unfortunately. At this point I would recommend opening up a support case on the issue to see if you can get some more dedicated expertise on this problem.

[odd bison]

Alright, I'll do that. Thanks!

[odd bison]

Not sure where to ask this but: I just created an ONTAP S3 user via CLI and didn't get a Secret key for the user. How do I show the Secret key?
Running ONTAP 9.11 and usually doing this through System Manager.
This didn't really help: https://docs.netapp.com/us-en/ontap/s3-config/create-s3-user-task.html

[fair perch]

From that page “S3 users' keys can be displayed with the vserver object-store-server user show command.” .. doesn’t that show it?

[odd bison]

Only the Access key, not the Secret key.
When creating the user in System Manager it shows the Secret key once, and only once.

[small rivet]

Do a set adv and then run the same command it will list both access and secret access key

[odd bison]

Oooh, thanks a lot! That was easy, didn't expect to be able to get the key after creation.

[odd bison]

Thanks for the details I did some more

[hasty zephyr]

We have problem with Ontap and ASUP. Since 29.11.22 we disabled internal anonymous mail relay. in Ontap there is no possability to configure TLS & Auth for smtp ASUP. We need the mails just internal, asup to netapp is https but maybe good to have the possibility.

[fair perch]

@hasty zephyr could you put in a ticket asking if there is anyway to do it? Then give us the number and we can add it to an Request for Enhancement? (I haven’t checked but I assume one for it exists..)

[hasty zephyr]

Sure 🙂 Case-Numer: 2009392506

[fair perch]

Perfect, thanks!

Ok, I’ve asked our security engineering team to add it to an RFE. It may be a long time before anything comes of it, if ever, but it will be tracked. Feel free to ask in about 12 months if you haven’t heard anything

[heavy jackal]

Anyone using file operations auditing at scale yet? Using the built-in command sets ( https://docs.netapp.com/us-en/ontap/nas-audit/create-auditing-config-task.html )
Are you using XML or EVTX?
What are you doing with the logs that it creates?
How are you gathering them and where are you putting them?

Sure wish I could just collect these logs and log-ship them like other logs instead of some custom process. ( https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Event_forwarding_to_a_Syslog_server )

// thinking out loud...
At first thought I am almost thinking of using the windows evtx even though the events will be nfs and smb based, but now after looking at another storage vendors documentation it wouldn't be the same anyhow. If I used xml I am sure I could use some sort of python module to convert and send off to a data lake of some sort, then rotate the log at that point.

[viscid pendant]

You will need third-party tools to process the EVTX audit logs and possibly send them to your SIEM.
Check the following out, but a simple Google-search should get you many more. They all in some sort allow you to ingest the NetApp EVTX audit logs:
Splunk, NXLog, LogRhythm, Lepide Auditor, Change Auditor (Quest), WinCollect (IBM), EventTracker (Netsurion), ADAudit Plus (ManageEngine), LogViewPlus (Clearcove), Logstash (Elastic), ...

You could also try to read the EVTX logs with Timeline Explorer (parsed by EvtxECmd). Never tried it with the NetApp EVTX but at least for Windows event logs this is a nice and easy way to read EVTX.
https://ericzimmerman.github.io/#!index.md

Here's a good explanation for EvtxECmd and Timeline Explorer: https://www.youtube.com/watch?v=YvMg3p7O6ro

[heavy jackal]

I have no problems parsing evtx with Powershell, but thinking it may be easier using python and some xml modules.

[vague flume]

You can increase this, smtp auth is more and more the only way because of companypolicies to send mails - also in our company

I had a user which generated several 100.000 secd.cifsAuth_problem_1 messages using in a script wrongly a local admin instead of his domain user! Is it possible to block users / computers causing so many errors in some way in ontap? I know: best would be stoping the script but this needs often weeks or months

[heavy jackal]

anyone good with using nfs4_setfacl ?

specifically related to setting up auditing

[west lake]

If you know the IP address of the system the script is running on you could create an export rule denying access to that IP address. Typically export rules are for NFS but they can also apply to CIFS.

[feral dust]

A nasty hack is just to add a route from his IP and send it to an IP with no host ... vs route add -vserver your_vserver -destination evil.hacker.ip.addr/32 -gateway some.empty.ip.address. 😈
Effectively a route to /dev/null...

[wispy oriole]

Hello, I have a FAS 2750 on v9.11 ONTAP. I just recently enabled encryption and stored my key safely. I am in the process of encrypting volumes. I would like to do 2 things: 1) not require the key to be entered when starting up the FAS when it has been shut down. 2) Is there a way to test if the key works without affecting the operation of the FAS preventing access to data? Don't want to get caught in a scenario where we're no longer able to access our data.

[wispy oriole]

I figured out number 1 which is the cc-mode-enabled; this command shows if it is enabled or not BR90::*> security key-manager config show
CC-Mode health-monitor-polling-interval
Enabled (in minutes)

---

false -

[unreal idol]

Thanks for updating us with the solution! That way if someone else comes in searching for it, they’ll find it!

Secondly, be aware that NetApp is observing a week-long shutdown this week in observance of the holidays, so responses might be delayed.

[grim whale]

There is a sub command
security key-manager onboard verify-backup

That should do the trick

You will need your passphrase and the output from when you enabled encryption (you know, that begin/end with the stings of characters)

[west lake]

That's right. You can find more information on that command in the ONTAP 9 encryption and security guide starting on page 279: https://docs.netapp.com/us-en/ontap/pdfs/sidebar/Security_and_data_encryption.pdf

Also, if it help, NVE utilizes FIPS 140-2 validated encryption and you can see the certificate from NIST here: https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4144

[feral crater]

does anyone know if there's a way for ONTAP to log which MAC algorithm (e.g. umac-64) is used when a user logs in to the CLI via SSH?

[west lake]

Hey Chris, of course you can see the configured mac algorithms on an SVM using the "security ssh show" command, and you can lock it down to only those you want to be used, but as far as logging the actual mac algorithms used for each SSH session, I don't know if this is logged or not. I'll do some digging internally and see what I can find around this.

[feral crater]

Thanks Matt! I've been able to delete (and then re-add) those algorithms, but I'm concerned that customers may have tools which rely on those particular algorithms and thus don't want to break workflows. Of course, I think this is a very low risk, but still..

[west lake]

Hi Chris. I confirmed with engineering that we do not log the algorithm used. There is a way on most clients to see this, but obviously that will not scale for you and the goal you are trying to achieve. The recommendation at this point would be to have your NetApp account team open a feature enhancement request for this capability.

[feral crater]

Thanks Matt! That's what I thought would happen, but I appreciate you checking for me

[pulsar knot]

Aight.
Unsure if this is the wrong channel or not but I'll find out.

Storage grid//FAS external KMS/KIMP servers, is there something Netapp provides or is there a list of supported platforms?

Or is THALES it?

[feral crater]

can't speak for StorageGRID, but there are definitely other KMS servers than THALES when it comes to FAS

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Systems/FAS_Systems/How_to_determine_which_External_Key_Managers_are_supported_by_ONTAP

[west lake]

I confirmed with the SGRID team today that currently the only externally supported KMIP provider is Thales

[pulsar knot]

So if I've got the requirement to encrypt all Nas data and this SGrid is supporting fabric pools, (organizational politics aside), one would logically surmise SGrid needs done as well.
Even with FAS supporting either an Internal keystore or a few external providers.
Sgrid last I knew (I'm very new to the product, forgive me) supports external only, and that vendor is Thales.

[west lake]

Hi Jesse, i'm not really an SME on the SGrid side, but I know that the E-Series local key manager capability can be leveraged since Santricity is running on the appliances. This link appears to have some more details on that, but I haven't done a deep dive https://docs.netapp.com/us-en/storagegrid-115/admin/reviewing-storagegrid-encryption-methods.html. At the end of the day though it sounds like local key management is possible.

[grim whale]

If the data is encrypted at ONTAP, any blocks tiered off to storage grid (fabric pool) will be efficient (if enabled on fas/aff) and encrypted (if enabled on fas/aff)

(Unless you are using the “all” tiering policy in which case there is no efficiency applied. Not sure if it works be encrypted either,

[west lake]

That's a good point. If you are using software encryption (NVE) then the tierd data on SGRID would also be encrypted already

If it's NSE or just SED though, the encryption is at the disk layer and wouldn't transfer to SGRID when tiered

[pulsar knot]

NAE is likely going to go in place.

[pulsar knot]

This is a good point and appreciated, but as we all know at times compliance officers don't care about logical arguments 🙂

[grim whale]

Remember, with FabricPool...ONTAP only sends the blocks. There is no meta-data stored there. All the Meta stays on the local SSD or HDD. If NVE/NAE is used, the blocks are encrypted and even if someone go a hold of them, what can be done? there is no meta data to provide structure and the blocks are all encrypted.

[pulsar knot]

I'm with you, 100%.
HIPAA people don't care for the technicality of how it works though.

Not trying to be pedantic or argue, in the slightest...

We've been fending this argument off for years, but it's finally inescapable at this point.

[queen egret]

Can I access the cluster and other vservers using an active directory account and still leave CIFS set to workgroup auth? Or when CIFS is configured for domain auth is there anyway to fall back to workgroup auth if DNS is down?

[unreal idol]

Hey @queen egret, welcome! Would you mind posting this in the #1062049169520476220 support forum please?

[queen egret]

Thank you. Sure thing. Sorry about that.

[unreal idol]

No worries!

[west lake]

I share my top of mind thoughts on security this week on LinkedIN if you want to check them out: https://www.linkedin.com/posts/matt-trudewind_linkedin-activity-7024789530838765569-B2xP?utm_source=share&utm_medium=member_desktop

[woven hawk]

Solid post. Just gave you a follow on the tweeter box

[west lake]

Cool thanks much.

[viscid pendant]

Question to the NetApp Security folks: Is ONTAP impacted by CVE-2022-37967?

Just to be clear: I'm not talking about CVE-2022-37966 which is about the default encryption type for Kerberos session keys getting changed from RC4-HMAC to AES: https://support.microsoft.com/en-gb/topic/kb5021131-how-to-manage-the-kerberos-protocol-changes-related-to-cve-2022-37966-fd837ac3-cdec-4e76-a6ec-86e67501407d
There are plenty of NetApp-KBs for that CVE, like this one: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_is_the_impact_of_CVE-2022-37966_to_ONTAP_9

My question is regarding one of the other Kerberos changes (PAC signing) introduced with the Microsoft Nov-2022 patches: https://support.microsoft.com/en-gb/topic/kb5020805-how-to-manage-kerberos-protocol-changes-related-to-cve-2022-37967-997e9acc-67c5-48e1-8d0d-190269bf4efb
I can't find anything about this CVE on the NetApp KB.

I'm asking since in mid 2021 there was another CVE where Microsoft also did some changes with the Kerberos Privilege Attribute Certificate (PAC) which actually impacted CIFS on ONTAP and lead to quite some support cases: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_does_Windows_Server_KB5008380_CVE-2021-42287_in_Enforcement_mode_impact_ONTAP_9_CIFS_Operations

[arctic hawk]

@viscid pendant https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_is_the_impact_of_CVE-2022-37966_to_ONTAP_9

[viscid pendant]

Thanks but as mentioned I'm talking about CVE-2022-37967 and not CVE-2022-37966. The one ending with "7" and not "6". These are different vulnerabilities which are fixed by different patches.

This all happend during the Nov-2022 updates so everyone keeps on mixing them up.

[arctic hawk]

D'oh sorry

This KB lists 37967 at the bottom: https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU527

@viscid pendant

[viscid pendant]

I think that's a typo.

[viscid pendant]

oh nice, I forgot about this

But I'm having a hard time understanding that sentence.... "Security vulnerabilities [...] are tested and fixed in the given supported patches."

Does that mean NetApp will support these CVE-changes with Patches to ONTAP? Or how should I translate that?

I mean there are already NetApp KBs regarding two of the relevant three changes:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_is_the_impact_of_CVE-2022-37966_to_ONTAP_9
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Does_CVE-2022-38023_have_any_impact_to_ONTAP_9

These are great and explain in detail what the impact might be and if ONTAP changes are needed.
I'm basically looking for another KB-article for the third Nov-2022 security-patch: CVE-2022-37967

[arctic hawk]

Leave a feedback on the KB.

The tl;dr is that it's rolled together, but it isn't clear in the documentation. You have to really dig into the IMT.

[viscid pendant]

Sent a mail to '[e-mail]'

[viscid pendant]

Unfortunately I got exactly 0 response with my mail...

Is there another address to send this? @west lake Can you maybe help?

[west lake]

@OG1 Hi, there is a customer support bulletin about the Microsoft CVE's and how they affect ONTAP being targeted to be sent out later this week. However, usually the best way to get updates on things like this is to open a support case. Support can create any KB's if needed and also they can get a direct response on CVE related items from our PSIRT (product security and incident response team) team. I'd recommend opening a case if the support bulletin coming out later this week doesn't address the need.

[arctic hawk]

Oh wait, derp

Nvm.

Usually the KB team is pretty quick to respond so I'm surprised they haven't.

[amber oar]

Yeah that doesn't seem right, normally the KB team do respond on a consistent basis. @celest orchid is there someone we can poke from the KB team to follow up why this was missed?

[celest orchid]

@viscid pendant Have you sent a follow-up email to them yet? I'll DM you my email address if you want to forward your message to me directly so I can check in with the manager of the KB team.

[viscid pendant]

Yea, for CVE-2022-38023 one bulletin just got send out some hours ago: https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU530

What everyone keeps missing: This is about another CVE (CVE-2022-37967) which is not yet mentioned anywhere in the KB (it's linked in two KBs but that must be a typo since this is a totally different issue and only happend to got get released in the same timeframe as the other patches).

Basically customers are getting anxious if this CVE might also have an impact on their production workload... For CVE-2022-37966 and CVE-2022-38023 we now mostly know what to do and what possible impact there might be.
We simply need the same for CVE-2022-37967 (ending with a "7").

[viscid pendant]

sent a follow-up mail now

[west lake]

FYI, I was able to find this support bulletin on CVE-2022-37966: https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU527

[viscid pendant]

Thank you Matt... I'm just gonna repeat myself.... but as mentioned several times now: This is the wrong CVE. 🫤 😭
My customers are asking for guidance regarding CVE-2022-37967 and NOT CVE-2022-37966.

CVE-2022-37966 is about the default encryption type for the Kerberos Session Keys getting changed from RC4-MAC to AES.
This is the KB from Microsoft about this CVE: https://support.microsoft.com/en-us/topic/kb5021131-how-to-manage-the-kerberos-protocol-changes-related-to-cve-2022-37966-fd837ac3-cdec-4e76-a6ec-86e67501407d
And here's the SU from NetApp about it: https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU527
This CVE is handled, everything fine here, everybody knows what to do, nothing to see, walk on.

CVE-2022-37967 is about adding PAC Signatures to Keberos Service Tickets (and validating them).
Here is the KB from Microsoft about this CVE: https://support.microsoft.com/en-us/topic/kb5020805-how-to-manage-kerberos-protocol-changes-related-to-cve-2022-37967-997e9acc-67c5-48e1-8d0d-190269bf4efb
There is NO KB-article or SU anywhere currently from NetApp. --> This is what I am asking for. If that CVE won't impact ONTAP that's also fine. I would be more than happy but I need some statement if that is the case or not...

(Yes, the CVE ending with 7 is linked in the SU527, but that's a typo! Read through the SU-text, there is nothing about PACs, it's all about the Kerberos change.)

[gaunt ruin]

@viscid pendant not sure how helpful this will be to you in the context you're looking for, but i did find that CVE in the December 2022 Samba Vulnerabilities, and ONTAP is marked not affected there. but since i don't see it linked yet, here you go. just in case.
https://security.netapp.com/advisory/ntap-20230110-0003/

[viscid pendant]

Hi Dawn, thanks for checking.
But I think this advisory is for NetApp software which incorporate Samba. ONTAP does not use Samba internally afaik but its own SMB-implementation. That would also explain why it's marked as not affected in this advisory.

[west lake]

@og1 Yeah it was the reference to 37967 in SU527 that threw me off, so apologies on that and thanks for clarifying. I'll continue to see if it can find anything out from our CIFS SME's on Kerberos PAC support. In the mean time, in order to expedite things, I recommend you to open a support case. Support can create any KB's if needed and also they can get a direct response on CVE related items from our PSIRT (product security and incident response team) team.

[west lake]

@viscid pendant I was able to get ahold of someone in NetApp CIFS QA. They fully tested the changes in CVE-2022-37967 and confirmed that "Kerberos authentication works fine with Ontap for CVE-2022-37967 (Kerberos Elevation of Privilege Vulnerability)"

I've requested that a KB be created on this as well

Well that was fast. Here you go: https://kb.netapp.com/onprem/ontap/da/NAS/Does_CVE-2022-37967_impact_ONTAP%3F

[viscid pendant]

You're my hero, Matt! 🙌

thanks alot

[unreal idol]

Heads up…

[celest orchid]

I just got that email too

[feral dust]

ok... when the new NTLMv2 sealing requirements are in force on DC's, does that mean that CIFS client authentication towards a NetApp CIFS SVM will also need to use a sealed authentication request for it to succeed?

(assuming NTLMv2 authentication only since Kerberos is unaffected by these changes)

[viscid pendant]

Every client needs to use RPC sealing once the July updates have been installed on the DCs. So, yes.

[feral dust]

but the client, let's say a java cifs client using NTLMv2 to mount a cifs share doesn't actually authenticate with the DC... the SVM passes the authentication to the DC

[stable vapor]

Hello, Does anyone here have experience using DigiCerts API? I could do with some help.

[stable vapor]

I just looked and worked harder and now i have solved my own issue. I was trying to get our Digicert balance in PS to use on our dashboard. Here is the PS i used

$headers=@{}
$headers.Add("Content-Type", "application/json")
$headers.Add("X-DC-DEVKEY", "YOURAPIKEYHERE")
$response = Invoke-RestMethod -Uri 'https://www.digicert.com/services/v2/finance/balance' -Method GET -Headers $headers
$response

[heavy jackal]

Who's ready for CVE-2022-38023?

[fair perch]

I know one of our customers upgraded 180 nodes on the weekend in preparation, and are doing another 100+ this weekend.

[unreal idol]

https://tenor.com/view/steve-irwin-thumbs-up-excellent-good-nice-gif-4860119

[pulsar knot]

I've got 30 to do in short order.
Going to begin getting the smallest least squeaky wheels done first.
The most recent batches of shelf/disk FW did a bangup job of causing 2 disks to fail on my DR Nas lol

[heavy jackal]

Yup I've got a lot of clusters to update.... Is there a way to detect the user accounts that are not sealing? I see the field for is-signed ..

[viscid pendant]

Event 5838 on your DC: https://support.microsoft.com/en-us/topic/kb5021130-how-to-manage-the-netlogon-protocol-changes-related-to-cve-2022-38023-46ea3067-3989-4d40-963c-680fd9e8ee25#eventerrors5021130
Before you patch your clusters none of them will use RPC sealing for Netlogon.

the field "is-signed" is for SMB signing from CIFS clients to your CIFS server and has nothing to do with Netlogon RPC sealing/signing (which is between your CIFS server and your DC)

[shut perch]

If your clients are using kerberos, this should not affect you, according to the NetApp SU.

[subtle talon]

Please correct me if I'm wrong on this, but if I run:
cifs session show -auth-mechanism !Kerberos -fields auth-mechanism

Anything that shows up there could potentially be affected by CVE-2022-38023 unless you are on a patched version of OnTap with the fix. Is my understanding correct? We have a bunch of Samba and Mac connections that are still using NTLMv2 instead of Kerberos. We have upgraded all our OnTap clusters to 9.8P18.

[viscid pendant]

correct

[soft tendon]

Tamper proof Snapshot: For a volume with tamper-proof snapshots enabled, there’s a limitation on performing a snap restore – say you’re tamper-proof snapping hourly on a volume and noticed at 3pm that the vol has been attacked and noon is the known good restore point. You can’t do an in-place snap restore of the volume to the noon snapshot since the 1pm and 2pm are locked. What do we do?

The only solution I can think of is to flexclone the volume based off the noon snapshot and then use that volume for immediate access. However, at some point, we’ll need to perform a clone-split to get a separate volume – all well and good, but the vols we’re trying to protect are upwards of 40TB in size and we can’t guarantee double the/sufficient space in the aggregate. Same goes for a snapmirror approach of copying the volume elsewhere. Is this a limitation that we’d just have to live with or is there magic filer functionality I’m forgetting or unaware of?

[celest orchid]

This is already posted in its own thread -- please continue the discussion there.
https://discord.com/channels/855068651522490400/1110945266959384628

[sharp plover]

Did you get the answer? Is there a way to active sesions that are not sealed?

[sharp plover]

How do i Idenfity the list of sealed session on a cifs server?? Do i need to set it to seal for session-security-for-ad-ldap as a fix?

[viscid pendant]

The connection from ONTAP to your DCs uses Netlogon to authenticate the client. Netlogon might need to seal its RPC connections, depending on your settings and patch-level of your DC.
This has nothing to with the actual CIFS-sessions from the client to the CIFS server (ONTAP).

Don't confuse "RPC sealing for Netlogon" with "SMB sealing" or even "LDAP sealing". Sealing simply means here that a certain protocol uses encryption for the packets going over the wire.

Before you patched ONTAP: RPC sealing will not be used for Netlogon for any client.
After you patched ONTAP: RPC will be used if the DC supports it.

In other words: If you currently see clients with CIFS sessions using NTLMv1/v2 for authentication, you NEED TO PATCH ONTAP, otherwise these clients will have issues after the July patches have been applied on your DCs. There is no workaround. (Unless you change everything to Kerberos authentication...)

[feral dust]

even if your clients are using NTLMv2 they won't be affected... NTLMv2 causes pass-thru authentication through the filer, so only the filer needs to use sealing... the client doesn't need to "talk" to a DC

which is something that I had hoped would go into the SU530 documentation, but despite my best efforts to get someone to address the larger underlying functions of NTLM v Kerberos with some information, it's still a bit myopic

[viscid pendant]

Clients will be affected though because they won't be able to authenticate without ONTAP getting patched, so users can't access their files. 🤷‍♂️

[feral dust]

yes, the pass-thru authentication from the filer still has to be sealed

[viscid pendant]

Who is TME for ARP (Autonomous Ransomware Protection)? @west lake ?

We have a customer who enabled ARP for over 400 volumes (after 60 days learning phase).
And we're having many many open questions, maybe some bugs, false positive, situations where it's unclear why an EMS got sent for "abnormal activity" event, why none has been sent, why volumes still show up as „Volumes with Abnormal Activity“ even though everything has been cleared, file extensions show up which nobody ever heard of and apparently do not exist, and so on...

The NetApp case (2009599803) has been created on 06/06/2023 and there really is no progress...

The TSE tells us they don't know much about ARP in the Technical Support Center 😐

[amber galleon]

file extensions show up which [...] apparently do not exist
these are probably temporary files that get deleted a few seconds after they were created. This has been a problem even with the old fpolicy based file-extension blocker. Some applications have very strange ideas on what their temporary files should be named

[modest ether]

Does anyone here use Hashicorp Vault KMIP on an SVM level (MCC IP)?

[west lake]

I recommend the case be move to level 2 support if not already there, as ARP has been out more than 1.5 years and is well known to level 2 support engineers. Level 2 can also engage the PM for ARP if needed as well.

[urban needle]

Been upgrading lots of customers over the past month and have several planned. Probably upgraded around a 100 clusters untill now. Nice extra customer contact moment to talk about new innovations and running projects.

[viscid pendant]

This. Ultimately it's good for NetApp (and the customers). Getting so many clusters to current versions means less cases because of known bugs.

[pulsar knot]

I've got a change to finish enabling just about everything needed this coming week, it's been an annoying road due to FUD from apps groups not understanding how protocols work.

Looking forward to it being done and over.

[full oasis]

vSphere Datastores affected by the Microsoft CVE CVE-2022-38023 "require RPC sealing"? Not sure if this is the correct place to ask. If we have NFS backed Datastore on ESX hosts and vCenter 7, that are using NTLM authentication, are they at risk of disconnecting? Or is this only a CIFS issue? And if it's an issue with NFS, does the NetApp patch to allow RPC sealing, also resolve the NFS NTLM authentication (if there is an issue at all?)

[pulsar knot]

The NFS data stores are using NTLM?
It's NTLM authentication driven more than protocol, as I understand it

[grim whale]

That Microsoft fix has zero to do with esxi hosts connecting with nfs.

Esxi uses nfs and in rare cases nfs with Kerberos. Nothing at all to do with ntlm

[fair perch]

Kerberos for ESX datastores is a new one for me.. but it wouldn’t surprise me if you had done it for some clients @grim whale

[grim whale]

I steer people away. Only supported with NFS4.1. If best practices are being followed (isolated vlan, no gateway) I do my best to always use v3. Much less issues

Actually never have done it. Setting up Kerberos interfaces in ONTAP is not pleasant

[high aspen]

Sounds like a lab project for me 🙂

[shut perch]

another day, another security vulnerability. From the author: "The one rated HIGH is probably the worst curl security flaw in a long time." I assume we'll be patching one of more NetApp products for the libcurl bug.
https://github.com/curl/curl/discussions/12026

[unreal idol]

https://www.linkedin.com/feed/update/urn:li:activity:7117742505869082625/

This is cool! @west lake have you seen this?

[odd bison]

I've read the hardening guide for ONTAP, and it's a great tool for what security features are available.
Is there a commonly used standard for security settings in ONTAP that we can follow and point to when asked what security features/settings we have implemented?
I did find something called a STIG (https://www.stigviewer.com/stig/netapp_ontap_dsc_9.x/) but not sure how common practise it is.

[fair perch]

the STIG is for base government level security. It's a good starting point, but won't be suitable for some (too much) or sufficient for others (too little)

[odd bison]

Know of any alternatives I could review? What does your customers usually use?

[fair perch]

it's probably the best starting point

https://www.netapp.com/pdf.html?item=/media/8128-ds-3846.pdf

more of a list than a "what to set"

the STIG is more proscriptive

[odd bison]

That's a new word for me but "proscriptive" and descriptive is what I want. 😄

[unreal idol]

https://discord.com/channels/855068651522490400/1083486771783938099

There's some links in ^this^ thread that might help

I was looking into automating a default Hardened deployment

[dusty shuttle]

I there a guide to setup third party auth. with the SAML auth. on ONTAP? I am a newbie to this, and I think it's more the setup of the idP that I haven't done yet... so if there is a guide of how to set this up with the Microsoft Authenticator or Google ... that would help me a lot 🙂

[viscid pendant]

What IdP do you want to use with System Manager?

[dusty shuttle]

I have tried with a local setup using authentik, but I am willing to try auth0 or similar public setups...

[viscid pendant]

Start here: https://kb.netapp.com/onprem/ontap/dm/System_Manager/System_Manager_SAML_Authentication_Resolution_Guide

Here an example with ADFS: https://kb.netapp.com/onprem/ontap/dm/System_Manager/Configuring_SAML_authentication_in_ONTAP_System_Manager_for_9.8_and_newer

Currently the only qualified/validated solutions are Shibboleth, ADFS and Cisco DUO.
For everything else, this KB (https://kb.netapp.com/onprem/ontap/dm/System_Manager/Is_Azure_AD_supported_as_SAML_IdP_solution_for_System_Manager) states: "Other non-qualified IdPs should work with ONTAP if configured correctly, however NetApp Support does not assist with those IdP configurations"

[gaunt ruin]

We've also seen ping and okta working in addition to azure ad. i've not heard of authentik or auth0 being used. but as og1 says, it should work if it's set up correctly, we just haven't qualified anything except those 3. aside from the claim rules we don't really provide instructions on the IdP side. there are a few tidbits out there, but no step by step on the IdP side. mostly what we provide are things that have tripped up others during setup.

[buoyant mural]

I have a basic question, that I can't seem to wrap my head around the answer. When I create a new NTFS volume and share, the default NTFS permissions (DACL) are ALLOW-Everyone. I want to change this when any new volume is created the everyone is replaced with "DOMAIN\Group". I am trying to figure out how to do this via the cli with vserver security file-directory but I am not understanding the workflow. What would be best is if I could change the default on create, rather than have to run it after the creation. Does anyone have any idea on how to do this? Note, I want to change the NTFS file perms, not the cifs share perms.

[unreal idol]

I would post this in the ONTAP support forum above. You’ll get more direct better answers there than this open broad chat channel

[amber galleon]

you cannot change (as far as I'm aware) ONTAP's default ACLs. Usually customers just connect to the new volume via the C$ share as Domain admin and set the ACLs once manually (or if they're using RoboCopy or whatever to copy data in, this will also set the ACLs correctly). You can use the file-directory command but it's a bit more complicated, you have to create a security descriptor first where you define the ACLs and then connect that to a volume (path) via a policy

i.e.

1. create security descriptor vserver security file-directory ntfs create
2. attach ACLs vserver security file-directory ntfs dacl add
3. create policy vserver security file-directory policy create
4. add policy task to define the path vserver security file-directory policy task add
5. apply vserver security file-directory apply
6. monitor``vserver security file-directory job show`

[grim whale]

The windows way:
Map to c$
Right click on the the share you want to modify. Make your changes

I’ve noticed this behavior with XCP/robocopy where the top level directory has unwanted permissions like everyone/full. Just need to modify from above using c$

[buoyant mural]

thanks, I am trying to avoid the windows way, as I am writing a module to do this via the CLI. @amber galleon Do I have to create the descripter for every volume or just for the vserver? Then apply it after each of the volume is creaeted?

And what happens if it's changed from a windows machine, does the policy reset it?

[amber galleon]

the descriptor is independent of the volume you can attach it to multiple paths. And it doesn't watch or keep track of the permissions it's a one-time job only. so if you change it later it doesn't do anything unless you apply it again (reverting all permissions to what's specified in the policy/descriptors)

[buoyant mural]

thanks, that sounds like exactly what I want to do. I will test it out once I get my SIM back online 😉

[unreal idol]

https://cloud.google.com/blog/products/identity-security/google-researchers-discover-reptar-a-new-cpu-vulnerability

[fluid patio]

Hi

Can ONTAP change the ssl/tls diffie-hellman moduli to 2048 or greater?

[arctic hawk]

If it's not part of ONTAP today it would take a code change. If you really need it please talk to your account team.

[buoyant mural]

@amber galleon I did what I thought was the correct thing, but It added a bunch of additional NTFS perms.

I see they are all added by default when I create the policy. So I can remove them from there.

[feral dust]

Thoughts on ACME adoption? (tls cert management, etc... )

[amber galleon]

most enterprises have their own CA, and you usually don't want to connect your ONTAP system to the internet directly, so I guess it's rather low on the priority list

[feral dust]

newer versions of vault support ACME... so the point is taking some of the manual work out of cert renewel

[arctic hawk]

It would save on having to set up self signed certs at least.

[amber galleon]

Linux will soon assign a CVE number to every single kernel patch (https://lwn.net/Articles/961978/) ... imagine hundreds or thousands of CVEs for every kernel release 🙂

[unreal idol]

https://www.bleepingcomputer.com/news/security/cisco-warns-of-large-scale-brute-force-attacks-against-vpn-services/

Cisco's researchers report that the following services are being actively targeted by this campaign:

Cisco Secure Firewall VPN
Checkpoint VPN
Fortinet VPN
SonicWall VPN
RD Web Services
Miktrotik
Draytek
Ubiquiti

[opal sable]

Hello, is it possible to have some aggregates encrypted using external KMIP and some encrypted without on the same cluster ? Is there any pre-requisite in term of licence on ontap ?

[pallid crown]

if you want to use external key manager it's an additional license

[halcyon sun]

And yes, you choose which aggregates you wish to encrypt, and mix within the same cluster. When you create an aggregate via the CLI add the "-encrypt-with-aggr-key true" parameter. Enabling a key manager does not force encryption on for all aggregates or volumes at all. If you don't want the aggregate encrypted, just don't specify the parameter during creation. The documentation for this is here, https://docs.netapp.com/us-en/ontap/encryption-at-rest/enable-aggregate-level-encryption-nve-license-task.html.

You will receive warnings from ONTAP if trying to perform operations such as volume move between aggregates of differing encryption types; moving a volume on an encrypted aggregate to a non-encrypted aggregate is a good example.

[amber galleon]

I think it depends on whether you use NAE/NVE or NSE. For NSE, all aggregates are encrypted (FIPS compliant). For NSE/NVE you can choose what you want to encrypt

[opal sable]

Thanks for the reply, do you know which one ?

thanks for all the replies

[viscid pendant]

That's not correct afaik. The volume encryption (VE) license which is included in ONTAP One is valid for onboard as well as external key managers.

You might need to buy/license the external key manager but those are 3rd party products. You can check the IMT which ones are currently supported.

[grim whale]

And if you want/need to use secure purge on SSD, you must use an external key manager and you CANNOT use NAE. Only volume encryption is supported with secure purge

[grim whale]

Technically it’s not included with ONTAP One. It’s a separate check box. It’s there in case the system needs to go to a country that’s not allowed to use the encryption methods

When the checkbox is enabled it is included in ONTAP One. If it is not checked, it’s not there.

There was a bug in the writing system sometime last year that disabled that check by default and a few of it customers ended up getting no encrypting capability and we had to rectify it

[viscid pendant]

Last I checked the NoDAR images are only for five countries (Russia, Belarus, Kazakhstan, Armenia and Kyrgyzstan). I think you can very much say it IS included in ONTAP One.

https://docs.netapp.com/us-en/ontap/system-admin/manage-licenses-concept.html#licenses-included-with-ontap-one

Only a very minor percentage of systems would be deployed over there.

[grim whale]

@viscid pendant believe what you want. I do quotes all the time for my company. If you do not check the encryption box you will get the ONTAP One license without the VE key.

So it absolutely included if you check the box.
It is also absolutely NOT included if the box is unchecked

[shut forge]

Has anyone ever encountered a ghost key while using external key-manager. Ghost key showing only this " "

[grim whale]

Just installed a system today. No encryption because the box wasn’t checked. Now we need to do a process to retroactively add it back in

[pallid crown]

Actually if you configure the external key-manager on the cluster SVM it will work and that KMIP server will be used for all SVM's. If you want different KMIP servers per SVM (i.e. multi-tenant), you need the MT_EK_MGMT license. Within the GUI - Licenses - 'Multitenant Encryption Key Management License'.
If you already have an onboard key-manager and want to add an external KMIP, I believe you also need the MT_EK_MGMT license

[viscid pendant]

Yeah I forgot about that one, but MTKM is included in ONTAP One too, at least according to: https://docs.netapp.com/us-en/ontap/system-admin/manage-licenses-concept.html#licenses-included-with-ontap-one

[pallid crown]

I'm not too sure here, MTKM definition below:

Multi-Tenant Key Manager (MTKM) is a key management solution that enables organizations to manage multiple encryption keys securely and efficiently. MTKM is designed to meet the needs of multi-tenant environments, where different customers or departments require their own encryption keys. It offers a central repository for all encryption keys, making it easier to manage and secure data across the enterprise.

Need someone to to verify if this is the same as MT_EK_MGMT license for use on external key-managers ? If so you're probably good to go

[viscid pendant]

Will simple MFA for System Manager ever be a thing? (some customers simply don't want to SAML-all-the-things)
TOTP is already possible for CLI since 9.13.1, so I guess implementing TOTP for System Manager shouldn't be really that complicated. Is there still hope? 😔

[unreal idol]

https://www.linkedin.com/posts/jacob-evan-horne_%3F%3F%3F%3F%3F-%3F%3F%3F%3F%3F-%3F%3F%3F%3F-activity-7212401878318333952-qQBA

Keep an eye on this…

[viscid pendant]

https://tenor.com/view/michael-scott-the-office-steve-carell-explain-like-im-five-eli5-gif-5356727

What's a CMMC

[unreal idol]

https://dodcio.defense.gov/CMMC/About/

In September 2020, the DoD published an interim rule to the DFARS in the Federal Register (DFARS Case 2019-D041), which implemented the DoD’s initial vision for the CMMC program (“CMMC 1.0”) and outlined the basic features of the framework (tiered model, required assessments, and implementation through contracts). The interim rule became effective on November 30, 2020, establishing a five-year phase-in period.

It wasn't suppose to be published until next year, but they're getting it done almost a year early ahead of the election. Read into that what you will.

Hence, Jacob's linkedin post above

The CMMC final rule (32 CFR) should be published in the Federal Register around Sept-Oct this year. Then we get 60 days to freak out before it "goes into effect".

I hope @west lake is on top of this. 🙂

[amber galleon]

@viscid pendant as I understand it this is something like FIPS, i.e. another certification that US suppliers need to be certified in if they want to sell to the US government. So for us in EU it's rather uninteresting 😉

[dusty shuttle]

Has anyone got a guide how to setup a basic SAML Authentication up against Keycloak? We are kinda stuck where in where to put the host metadata inside keycloak... this is mostly new to us, but we have setup other applications to use keycloak but not ONTAP... All guides with NetApp points towards Micro$oft which is not going to happen here 🙂

[amber galleon]

yeah SAML setup is poorly documented by NetApp. I remember working with some NetApp engineers from the UK to get it working with Entra and even they said documentation was lacking ... So while I could tell you how to set it up with Entra by now, I can't help you with KeyCloak sadly. This needs a few more in-depth KB articles or even TRs IMHO

I guess you have seen this KB that details the required attributes...?

there's also a resolution guide if you're looking for a particular error or issue, but other than that the KBs are not very helpful IMHO

[viscid pendant]

There's only this which does not really help: https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/Does_ONTAP_support_configuring_SAML_with_Keycloak

WAIT A DUCKING MINUTE 🤯
https://kb.netapp.com/on-prem/ontap/DM/System_Manager/SM-KBs/What_are_the_pre-requisites_for_enabling_SAML_authentication_in_ONTAP_System_Manager

Did you really silently add Azure AD / Entra ID to your blessed list of "validated IdPs" ?!?
I was waiting for this since I don't know how many years...

Please announce that somewhere, people need to know!!

[amber galleon]

Azure AD was already "supported" before, but it doesn't exist anymore and was replaced by Entra ID, which isn't supported

(or at least wasn't when we wrote our internal guide on how to get it running)

[viscid pendant]

And also update these KBs, they still say "non-qualified" and "not tested by NetApp engineering":

https://kb.netapp.com/on-prem/ontap/DM/System_Manager/SM-KBs/Is_Azure_AD_supported_as_SAML_IdP_solution_for_System_Manager
https://kb.netapp.com/on-prem/ontap/DM/System_Manager/SM-KBs/How_to_set_up_SAML_authentication_in_ONTAP_System_Manager_with_Azure_AD_as_the_IdP

[viscid pendant]

Are you kidding me? Isn't Entra ID only the new (marketing) name as Azure AD and technically the same? So if NetApp supports Azure AD it should also support Entra ID.

[amber galleon]

Apparently (according to our resident Azure AD/Entra expert, Alex) it is different enough... I don't know the specifics though, but you might want to talk to him 😉 The naming and renaming is a mess at Microsoft

[viscid pendant]

"Microsoft Entra ID is the new name for Azure AD. The names Azure Active Directory, Azure AD, and AAD are replaced with Microsoft Entra ID."

don't crush my hopes 😦

[amber galleon]

again, it works just fine, you can use it if you want. Setup guide is in our Wiki

[viscid pendant]

Technically I can, but customers won't let me 🤓 if there's a KB which says it's not supported/qualified/validated. And every Netapp-guy I ask every time links me to that ominous KB with a "wait-for-it" look.
I can azure you (see what I did there?) this KB did not include Azure AD before, I'm checking every few months.

[celest orchid]

Please submit feedback on those articles from in the KB. I'll ask the KB crew to take a look, but to ensure it actually happens, that's the best way to get it done.

[viscid pendant]

Done

And thanks Drew!

[celest orchid]

Maybe @warm flicker will see this later 😎

[warm flicker]

Easiest way is to just leave feedback on the KBs that are inconsistent. We try to turn external feedbacks (including partners) around in 24 - 48 hours)

trying to get an answer / consinstent update to the KB but please also push the feedback button as that helps us to track it especially if you find additional places.

@viscid pendant @amber galleon

[viscid pendant]

Feedback ist raus, danke Andy 😉

[fluid patio]

hi. may i know some file extensions are different from its file type?

after turning on ARP feature

[viscid pendant]

bugs, update to the most current P-release of your version

[fluid patio]

thanks, I got the kb

[plucky moss]

The KB has been updated.

[viscid pendant]

Yeah and now Entra ID is again removed from the "Supported IdP Servers" 😐

I don't get why NetApp does not want to support/validate/approve it... everybody is using Entra ID

[barren sage]

'everyone' no. we are not, many I work with/customers/etc are not
still using their own/our own
But it is odd they don't support it

[viscid pendant]

I know there has been no official EOL announcement of ADFS but Microsoft is pushing everywhere to use Entra ID for federation. And more and more customers are already using it. At least for me many more than the other two "supported IdP servers" Shibboleth and Cisco Duo...

[amber galleon]

Shibboleth is big in the edu world though. Most or all Universities, University hospitals and FhG/MPI in Germany use it (through EduRoam). So from a pure "number of users" point of view, I guess Shibboleth is actually the biggest 🙂

[celest orchid]

https://www.carscoops.com/2024/12/vw-group-data-breach-exposed-location-info-for-800000-evs/

Sometimes your environment is only as secure as you make it 🥴😬

[tired hill]

oof.. sounds like a RGE.

[compact moon]

DHS and your local Highway Patrol will be showing up shortly…

[odd bison]

Is there any reason for ONTAP not to have built-in TOTP (time-based one-time password) for user access?
Just saw the ONTAP 9.16 release notes and WebAuthn can be enabled for System Manager but I guess not for CLI, TOTP just seems like a much simpler approach to MFA.

[viscid pendant]

WebAuthn is for System Manager only. You need the part which is implemented in a web browser to talk with the FIDO2 implementation of your OS. A regular SSH has no understanding of WebAuthn or FIDO2.
To use the FIDO2 implementation for SSH (which got introduced in 9.12.1 already) you need PuTTY CAC or OpenSSH (might need to use the beta version) which are able to call the OS APIs, "regular" PuTTY is not enough, at least in older versions.

Yes, TOTP currently is only for CLI. I agree that TOTP is much simpler to configure but also has many shortcomings security-wise vs FIDO2, especially phishing-resistance. Also touching your Yubikey is much easier than opening your TOTP app and typing down these 6 digits.

[odd bison]

Is there any overlap for MFA between CLI and System Manager?
I mean a single MFA solution for both modes of access to ONTAP.

[viscid pendant]

If you deploy Yubikeys for your storage-admins you can use them as the second factor for both CLI and System Manager.

[unreal idol]

@hoary breach some good feedback here for ya

[barren sage]

Does NetApp/ONTAP use Cesanta/Mongoose?
We just got a series of security alerts coming in about vulns in all of our NetApp systems about this
https://nvd.nist.gov/vuln/detail/CVE-2023-34188

[viscid pendant]

The CVE does not show up here: https://security.netapp.com/advisory/
So I guess you need to ask NetApp support if this CVE is relevant for your systems.

[barren sage]

yea, i looked at those as well, but our security scan just popped them from last nights scan.
And it is on all of our deployments

[raw plover]

ONTAP is not tracked as shipping Mongoose.

[barren sage]

i didn't think so, I can't find anything related to it at all.
Just wondering why it's showing up on our sec scans.

CVE-2023-34188 - Cesanta - Mongoose
Port: TCP/5696 Mongoose httpd/3.7

actually, several CVE for the same app

[raw plover]

Possibly something in between the scanner and target is being flagged. If ONTAP is being flagged then open a support case to get assistance.

[barren sage]

yea, just weird they suddenly popped up

[raw plover]

That CVE has been around awhile now.

[barren sage]

most of these CVE are from 2018/2019 as well

[raw plover]

So something changed to cause it to be flagged.

[grim whale]

If you are scanning ONTAP externally, things will fail gloriously.

ONTAP uses FreeBSD modules. They may be patched internally to correct for cves. The external info is usually not corrected. When scanning externally it appears as not ONTAP and is flagged to fail

As an example, I tell my Nessus customers to create Local read-only admin account. Give the credentials to the scanning team and try again. Nessus has ONTAP missiles and knows how to scan internally

[barren sage]

this is all internal scans

[raw plover]

Nessus had 7-Mode ONTAP awareness - last I checked it did not have the same for ONTAP 9. It would be great if they added it.

[barren sage]

yea. we have so many deviations for ONTAP because there are so many false positives

[ancient oriole]

since when vserver admins can switch privileges ? with set -privilege advanced for example.
As for the rest-role you can't rly filter some advanced commands that's strange

[viscid pendant]

Do any NetApp security guys have more infos about the new vulnerability in SnapCenter? https://security.netapp.com/advisory/ntap-20250324-0001/

I'm mainly interested which SnapCenter Plug-ins are affected. My guess would be that Linux based Plugins are not affected for example. Maybe only SnapCenter Plug-in for Windows?

[celest orchid]

@raw plover ?

[raw plover]

This issue is in the SnapCenter Server, not the plugins.

[viscid pendant]

Ahhhh...hm ok, so it doesn't matter which plugin is being used?

Since it says "allow an authenticated SnapCenter Server user to become an admin user on a remote system where a SnapCenter plug-in has been installed"

The advisory mentions an "authenticated SnapCenter Server user" - I guess we're talking about a local SnapCenter user in the SC DB and not a user in the underlying operation system where SnapCenter Server is installed since that could also be Linux.

So in other words an authenticated user in SnapCenter can basically connect to any host which has a SnapCenter plugin installed and can access the operating system over there with System rights? (or at least with the local admin rights of the SnapCenter plugin user). And it does not matter if only a certain plugin is installed (since the host is for example a MS SQL Server)? The vulnerability is valid for ALL kinds of plugins? (https://docs.netapp.com/us-en/snapcenter/concept/concept_snapcenter_overview.html#snapcenter-plug-ins)

Sorry for all the questions but customers are asking already if there's a need for an emergency change because of the very high CVSS-score of 9.9. Some customers only backup a certain application and only have a certain plugin installed.

[raw plover]

If this only affected only certain plugins it would have been detailed.

[viscid pendant]

ok, thank you!

[hot dune]

Putting on my Pedantic Security Wonk Hat: For FIPS, there's two things.. FIPS Validated and FIPS Certified.. The former applies to the crypto used by a product. For instance, TLS and OpenSSL libraries and binaries would be FIPS Validated. This means FIPS is enabled on those encrypted operations. FIPS Certified is a whole other animal. That's when a partner builds an offering.. e.g. Servers, storage, network, etc.. The submit that whole package for certification. Any change to that configuration could invalidated FIPS Certification. That process takes a VERY long time and is not usually done by vendors like NetApp or VMware. FIPS Validation, however, IS done by vendors as they are shipping the code. /end pendantry.

Hello all.. My name is Mike Foley.. Some of you might recognize me from my time at VMware where, for 7-8 years, I was the face of vSphere Security. I wrote the vSphere Hardening Guide and did a ton of sessions at VMworld and other conferences. I may have even spoken to you at some point! Well, now I'm at NetApp where I'm the tech marketing person for ONTAP Security. I've been here a few weeks and drinking from the fire hose. As I come up to speed I'll be building out additional content to add to the already outstanding stuff that has been done already. If you have ideas or thoughts on what you'd like me to consider covering, please DM me rather than posting here. Thanks and I look forward to helping you all in your storage security journey. mike

[unreal idol]

So happy you found a home here with us, Mike!

[patent peak]

welcome Mike.

I tell people that my being a Technical Marketing Engineer means that 33% of what I say will be full of bunk.

[viscid pendant]

Ohh I think I remember watching some vTPM explainer videos by you! (Or was it Bob Plankers?...)

[hot dune]

I was the shorter version of Bob. 🙂 I brought him in to the vSphere Tech Marketing Group. I did a bunch of vTPM stuff before he got there and he took over when I had some health issues 6 years ago.

[viscid pendant]

Ah nice! Cool you found your way to NetApp

[viscid pendant]

Hi @raw plover I've just noticed that CVE-2025-26512 (https://security.netapp.com/advisory/ntap-20250324-0001/) has been updated and suddenly the words "remote" and "plugin" are completely gone from the description...

Do I understand it correctly that the impact is now solely on the users inside the SnapCenter Server? So by exploiting this vulnerability you are NOT able to gain any new privileges on the hosts added to SnapCenter Server? (the hosts which have the SnapCenter plugins installed)
And additionally, regarding the workaround: If a user in SnapCenter Server is only added to 1x role in SnapCenter Server (not 2x or more) it's also not vulnerable. Right?

Again sorry to be pedantic but we're talking about the backup software here which manages the crown-jewels of the customers: all the databases and its backups.
I don't understand why it's made so difficult to understand the impact of the advisory...

@hot dune If you want to add your two cents

[hot dune]

Looks like the only product affected in SnapCenter. You need a user with 2 or more roles. If you have just one role then it's not exploitable. Best to just install the patch and move on. I suspect that fixes the issue entirely, regardless of whether you have 2 or more roles.

[viscid pendant]

Thanks for the comment! Yes we of course also recommend patching (since that also fixes quite some bugs) but not all customers are that fast. And especially with the new requirements some first need to update PowerShell and/or .NET on server and hosts... which sometimes needs a reboot.

[viscid pendant]

But if only users with 2x roles are vulnerable this really limits the impact. Never actually saw a user in SnapCenter with 2x roles.

[viscid pendant]

@hot dune Can you please add this to the Summary/Impact page? "If you have just one role then it's not exploitable."
Just got the next customer asking about it. With a score of 9.9 (or 8.8 now) the customer would need to do a emergency change - but if it's only relevant when a user has 2x roles then we could plan to update when there is more time.

[odd bison]

The workaround specifically says "Ensure that every low privileged user is configured only with a single role in unfixed versions of SnapCenter.".

[viscid pendant]

Yes, but it does not explicitly state that if you do not have any "low privilged user" with more than "a single role" then this CVE is irrelevant for you.

Customer asks for an official statement. 🤷‍♂️

[amber galleon]

My understanding is that if the remediation for the CVE says "do $foo to be safe", then you are already safe (or unaffected) if you have already done $foo, and that you are at risk if you have not done (or don't want to do) $foo

[viscid pendant]

Yeah but it's still a workaround only. It does not state if it lowers the impact or completely removes it.

[raw plover]

The patch will fix the issue regardless of any other factor. SnapCenter users having more than a single role is considered an edge case so the Workaround was added since it should be feasible for most users. If no users have more than a single role then no other action is necessary to prevent exploit.

[hot dune]

I added my opinion here and at some point in the future I'll probably be able to add this to a Summar/Impact page but I'm not there yet. 1. Been here only a few weeks 2. Don't know the process yet and 3. The owner of the page should be the one to add the clarification I should think.

[hidden sable]

Good morning! Hope everyone is having a good Monday. I’m in the planning phase of implementing SnapMirror to replicate a singular VM to another site to satisfy backup requirements. This VM is the only thing we will be backing up and shouldn’t be any larger than maybe 300gb. Our requirement is that we have data-in-flight encryption enabled for replication traffic. From documentation I can see that Cluster Peering Encryption is the recommended option over making an IPSec tunnel between the source/destination clusters. However our security guy isn’t too thrilled about the PSK aspect of that. How severe would the performance drop be from using IPSec over CPE? I would imagine that after the original transfer of data, subsequent replications would be small and the loss of performance would be negligible. This data is going to be replicated from East coast > west coast if that matters.

[amber galleon]

What models/systems do you have? The new models have IPSec offloading engines which should make the performance impact negligible. When not using offloading, performance might suffer, as you said, but real numbers are hard to come by. It's mainly the throughput that is the limit. Depending on the speed of your network link, that might or might not impact you. As you said, delta updates are usually smaller after the initial transfer, until they aren't (think: Windows updates, some large software installation, database reorg, whatever). There is one KB from 2023 that shows a performance decrease to 50%, but it doesn't say what systems were used in that case, and it's also not a replication scenario but a high-performance local NFS workload.
I would always suggest to do a POC first and see how much a transfer of 300gig impacts the system performance, before implementing it for real

[hidden sable]

Roger that! Unfortunately I don’t think my systems (AFF-A150’s) are new enough hardware to take advantage of that. I appreciate your point on the deltas, updates and the like are easy to forget about. Does replication ever affect actual data being served? The data in question is an NFSv3 link to a datastore, where this VM would have its own datastore.

[amber galleon]

yeah, the A150 doesn't do offloading. As for replication affecting actual data, I don't really know what you mean by that. If you mean if it changes data on the source or destination, then no, replication does a 1:1 replication without any changes. The only thing it requires is additional snapshots on the source (which take a bit of space, depending on how often you update the replication)

[hidden sable]

Sorry, what I meant was would I see performance degradation for the VM while it’s in the process of being replicated?

In regards to the VM’s read/write speed

[amber galleon]

it shouldn't have an impact. However, as IPSec is done in software on the CPU, if your replication traffic bandwidth is too high, it might impact other workloads. You can always throtthe the SnapMirror though

[unreal idol]

I doubt you’d even be able to tell. It’s an asynchronous background process that’s handled separately from the protocol being served to the VM host.

Throttling is key, and as Darkstar said, do a POC to test your uplink throughput. Also, do your SnapMirror over separate/dedicated interfaces if possible (I know you mentioned A150. So not too many free/spares there)

Key thing to remember always is SnapMirror is a pull, not a push, so it will Hoover up as much bandwidth as it can or finds available from the secondary system to your first system. If it’s a big fat pipe, it’s totally possible to saturate smaller physical interfaces

[hidden sable]

Thank you for the great info! You guys are always the best

[lusty mason]

Hello everyone,
We are looking for a reliable VSCAN product. Unfortunately, Trend Micro is in transition and has no experience with the successor product.
Are there any experiences regarding which product is the most widely used? Deep Instinct?

[barren sage]

We've used Trellix for several years and it has been pretty solid

[pseudo garden]

We also use Trellix and works pretty fine but I‘m open for alternatives

[lusty mason]

Gerne! Hier ist die englische Übersetzung:

Thank you very much for your reply. How large is your environment? I have an environment with 35,000 users on FSx ONTAP.

Hello Falcon667
Why your are open for alternatives? Function, pricing... ?

[forest crypt]

How much is impacted by CVE-2025-55182 ? AIQUM? System Manager?

[celest orchid]

The security advisory for that one is here: https://security.netapp.com/advisory/ntap-20251205-0001

AIQUM appears to be not affected, while sysmgr is still being reviewed.

[unreal idol]

https://www.bleepingcomputer.com/news/security/over-900-000-kubernetes-instances-found-exposed-online/

[celest orchid]

ooof

[amber oar]

B r u t a l

Also somewhat not surprising. I would say setting up a cluster historically was not the most ... user friendly.

Or you use one of those platforms that deploys it for you and you are TRUSTING that it was done right.