#strange cifs / smb problem with our citrix servers

1 messages · Page 1 of 1 (latest)

meager acorn
#

today we see in 9.13.1P7 ontap cvo a strange issue with secd.rpc.authrequest.blocked. Message: secd.rpc.authRequest.blocked: Too many CIFS authentication attempts with wrong password from client "xxx.xxx.xxx.xxx" on Vserver "svm_xxxxxxxx".

in parallel we see
Message: secd.cifsAuth.problem: vserver (svm_xxxxxxxxxx) General CIFS authentication problem. Error: User authentication procedure failed CIFS SMB2 Share mapping - Client Ip = xxx.xxx.xxx.xxx
**[ 1] FAILURE: CIFS authentication failed
(what is missing here, is a username and password)

the result was, that on one of our citrix app servers, the user profile share which is stored on ontap cvo on a smb share, from time to time nobody was able to connect to the ontap cvo and user profiles could not be loaded... on lunch we restarted the server and the problem was solved...

another strange thing is in secd.log: 00000070.000931bb 001ccdf0 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097] | [000.000.060] ERR : Client (IP: xxx.xxx.xxx.xxx) blocked due to continuous attempts with wrong password. { in preventBogusAuthRequest() at src/authentication/secd_rpc_auth.cpp:1401 }

i

#

don't remember that there is a possibility to block authentications because of wrong passwords - we do not use an export policy for smb shares... so from which setting comes this block?
furthermore i see: 00000070.000931ba 001ccdf0 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097] | [000.000.057] debug: Entry found. Client IP: xxx.xxx.xxx.xxx Client info: xxx.xxx.xxx.xxx:3, currentTime: 1710399697, creationTime: 1710399638,refreshTime: 1710399639, failCount: 92 { in preventBogusAuthRequest() at src/authentication/secd_rpc_auth.cpp:1395 } ->here i have a afailcount of 92
but then there is another line:

00000070.000931c5 001ccdf1 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097] | [000.000.169] debug: [LOG DUPLICATE DETECTION]: Previously suppressed** 73121** duplicate log attempts for failed RPC secd_rpc_auth_extended on vserver ID 3 and log cache ID '151NBLADE_CIFS337' { in handleRpcResult() at src/diag/secd_log.cpp:838 }
00000070.000931c6 001ccdf1 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097]
00000070.000931c7 001ccdf1 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097] [LOG DUPLICATE DETECTION]: Start suppressing duplicate log attempts for failed RPC secd_rpc_auth_extended on vserver Id 3 and log cache ID '151NBLADE_CIFS337'
00000070.000931c8 001ccdf1 Thu Mar 14 2024 07:01:37 +00:00 [kern_secd:info:11097]

does this mean, 73121 authentications happened with the wrong password - after a server reboot all was fixed.... but what could cause this issue? any idea?

#

(we rebooted the citrix app server for clarification) - we guess, that the reason for the issue was the citrix server itself or something in interacting with ontap - because every other citrix server or user did not have any issue

cunning ledge
#

The way that NTLM authentication happens in that ONTAP will send the username & password, hashed by the challenge/response information to a Domain Controller. The Domain Controller will lock the account if there are too many attempts. If the SMB client is trying to use a local (SVM) username & password, then ONTAP may also lock/block that client/username from further attempts to use SMB. This is a new feature introduced in 9.12.1 in cooperation with ransomware/malicious activity blocking.

meager acorn
#

Looling at kb article T Seems deactivation or configuration is not possible? This is then very Bad because IT looks That Issue Happen When User pw Got expired with an aktive citrix Session. The same can Happen With RDP, and if this Happen on a Terminalserver With 100 User coming from the same IP accessing the storage, how we can Prevent 1-2 users blocking smbaccess for 100 users? There must be any configuration possibility. We Are locking users after 5 wrong auth attempts in the domain, but an expired pw did Not prevent an existing user or a crazy Windows in a Session, which was created before pw Expiration to try again and again to authenticate and IT seems this happens in specific Cases several hundret Times in a Minute. We Are at 9.13.1P7 and this is another Feature which was Not mentioned with one Word in the Release notes, we found also another Not mentioned New Feature. (The crazy waagent for linux in azure cvo, which spamming the logs and console because of the 50mb img jail mount with no inode avail, because the agent dropps thousands of 0 Byte files)

Is a downgrade really the only Option regarding the secd blocks?? Common guys, there must be something to configure at least in systemshell. I cannot imagine, That such a Feature cannot configured?

digital swan
#

@meager acorn please contact support with your issue. There is a way to possibly disable the feature (not recommended) but may be required in your circumstance. Please reference the above KB.

Also, recommend you submit a Feature Enhancement Request via your NetApp Account Team. I am thinking the ability to selectively choose which source IP addresses/hostnames can be excluded from the 1 minute block.

brazen beacon
#

Thank you @meager acorn for bringing this up. We are planning to upgrade to 9.13.1P8 and have many Citrix environments. I’ve just sent the KB to our SAM so he can get more informations about this

meager acorn
#

as long there is no way to deactivate it @brazen beacon i would recommend not to uprade to 9.12 or 9.13.1P7 or P8 - we found currently annoying 2 bugs in 9.13.1P7 (this one and ALERT LINUXAGENT/JAIL/VAR: OUT OF INODES and a third low prio ssue we see in permanently problems with unmounting var_fs on node-reboot and a follow up issue of the spamming on the serial console of this error is, that netapp did not receive console instance vm serial logs on autosupport from connector... and the 5. issues was yesterday, that a user authenticated via powershell with an existing user of a trusted domain with net use and generated then every minute an cifs auth error with his local administrator account... but this could be also an issue on the client side... anyway: the ip blocking causing issues with citrix profiles not getting correctly saved back to storage or with generated temp profiles.... and what i found on investigation in win server smb client eventviewer is thousand of eventid 31010 in a minute:

The SMB client failed to connect to the share.
Error: {Access Denied}
A process has requested access to an object, but has not been granted those access rights, which maybe causing the blocking or is a reason for that - but it happens the whole day from several users... but in smb client is also no user id... so maybe an anonymous session.... - anyway currently with 9.13.1P7 i have too many troubles. updating from 9.11 to 9.13 was definitly a wrong decission... we will stay in the future as long as possible on a release with a high PXX

meager acorn
#

as i opened yesterday the ticket i have now the info, that blocking can deactivated with: system node systemshell -node * -command sudo kenv -p bootarg.secd.block_auth_extended_rpc=false and then diag secd restart -node <node>