ONTAP 9.11.1P11 Cannot recognize "nblade access-cache show" command | NetApp | Page 1

indigo nimbus Oct 3, 2023, 10:18 PM

#

I got access denied error in NFS mount.
1.
We suspect the network ACL may need to be enabled for the client to access the NFS share. Is there anyway via ONTAP or Linux client can we check out if there is such issue between the client and ONTAP? Cannot use "ping" because it is disabled by default here.

When I follow this KB: https://kb.netapp.com/onprem/ontap/da/NAS/Troubleshooting_Access_Denied_or_Mount_Hung_from_NFS_client_for_clustered_Data_ONTAP
Found nblade access-cache doesn't work as shown below. Any idea on why?
*> set diag
Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y
*> nblade access-cache show
Error: "access-cache" is not a recognized command

NetApp Knowledge Base

Troubleshooting Access Denied or Mount Hung from NFS client for clu...

This KB provides troubleshooting procedures to identify the cause of an NFS mount hang or access denied response for ONTAP.

pure perch Oct 4, 2023, 3:18 AM

#

Are you using volumes with Unix or ntfs security? If they are ntfs you may be running into a user mapping issue

Check the export-policy rules. If you are getting access denied then the client is contacting ONTAP but it’s rejecting the mount. The rules are processed in order and maybe the client isn’t being explicitly allowed

indigo nimbus Oct 4, 2023, 11:03 AM

#

Checked all you all said above. Including the following, also "vserver export-policy check-access", the client is able to read-write. "export-policy rule" is fine.

vserver security file-directory show -vserver svm1 -path /vol/volume0/op
Vserver: svm1
File Path: /vol/volume0/op
File Inode Number: 440146
Security Style: unix
Effective Style: unix
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 0
UNIX Group Id: 0
UNIX Mode Bits: 755
UNIX Mode Bits in Text: rwxr-xr-x
ACLs: -

Can you please verify / check if NFS access is blocked due to ACL rule. Also, why "nblade access-cache.." is not working."

pure perch Oct 4, 2023, 11:36 AM

#

Does the nfs client have access to “/“?

Look at
Vol show -vserver svm1 volume root|volume0 -fields policy

Do both volumes have the same policy? If not, check the runes on both!

Export-policy rule show -vserver svm1 -policy <root-policy >|<volume0-policy>

indigo nimbus Oct 4, 2023, 2:52 PM

#

export-policy rules for root and volume0 are different and as shown below. I only listed rules for two clients that we have the issues with (207.10.137.58, 207.10.137.57) or relevant. There is one rule for 207.10.137.0/25 before the two clients, not sure of if this could be the issue.
I also wanted to tell you that the client 207.10.137.25 has no problem to mount these volumes. But, I couldn't find any policy rule can fit in this client.

vserver export-policy rule*> show -vserver svm1 -policyname default
...
Policy Rule Access Client RO
Vserver Name Index Protocol Match Rule

svm1 default 15 cifs, nfs 0.0.0.0/0 any
...

vserver export-policy rule*> show -vserver svm1 -policyname export_policy_volume0
...
svm1 export_policy_volume0 15 nfs 207.10.137.0/25 sys
...
svm1 export_policy_volume0 1054
nfs 207.10.137.58 sys

svm1 export_policy_volume0 1055
nfs 207.10.137.57 sys

pure perch Oct 4, 2023, 5:27 PM

#

0.0.0.0/0 is everybody
207.10.137.0/25 is where your 207.10.137.25 client fits

You may want to share the output of rule 15 with -instance

export-policy rule show -vserver svm1 -policy export_policy_volume0 -rule 15 -instance

Maybe something there will explain it. Maybe include the output for rule 1054 & 1055 side there are similar

static ferry Oct 4, 2023, 5:31 PM

#

note also that rules are processed in order, so if you happen to have a rule further up that matches the client but denies access, that rule will be in effect. It's different from routing, where the longest prefix will be selected.

indigo nimbus Oct 4, 2023, 6:49 PM

#

Okay, so, why 207.10.137.25 client fits in but both 207.10.137.58 and 207.10.137.57 doesn't fit in the rule of "207.10.137.0/25"?

Also, ruleindex 15 is a rule in "default" police which is for root volume, not for export_policy_volume0 though, please double check with two outputs as shown above. Anyway, upon your request, I share the output with instance as below:

vserver export-policy rule*> show -vserver eos-sirius1 -policyname default -instance -ruleindex 15
                                Vserver: svm11
                            Policy Name: default
                             Rule Index: 15
                        Access Protocol: cifs, nfs
List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: 0.0.0.0/0
RO Access Rule: any
RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: 65534
Superuser Security Types: none
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: use_export_policy
Policy ID: 77309411329

#

vserver export-policy rule*> show -vserver svm1 -policyname export_policy_volume0 -ruleindex 1054 -instance

Vserver: svm1
Policy Name: export_policy_volume0
Rule Index: 1054
Access Protocol: nfs
List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: syrsiteststdb01.cc.columbia.edu
RO Access Rule: sys
RW Access Rule: sys
User ID To Which Anonymous Users Are Mapped: 65534
Superuser Security Types: sys
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: use_export_policy
Policy ID: 77309411342

rule 1055 is the same as 1054.

Please let me know! Thank you.

pure perch Oct 4, 2023, 8:12 PM

#

You listed this:

vserver export-policy rule*> show -vserver svm1 -policyname export_policy_volume0
...
svm1 export_policy_volume0 15 nfs 207.10.137.0/25 sys

Which is rule 15 inside the export policy export_policy_volume0.

Let’s see that one. It may be inhibiting access.

You could try this:
Mount svm-ip:/
(Instead of mounting the volume, mount the root namespace and then try to cd into the volume.

I suspect the mount will work and the cd will fail

indigo nimbus Oct 4, 2023, 9:25 PM

#

Yes, you are right. I was confused by that two different rules in two different respective policies but with the same ruleindex number, 15
Please see ruleindex 15 instance below. I am not system admin, so, I don't have to root permission to mount "/" on the Linux client. Can you please explain to me what went wrong here? I need to explain to them before we can perform mounting the "/" root volume.

vserver export-policy rule*> show -vserver svm1 -policyname export_policy_volume0 -ruleindex 15
                                Vserver: svm1
                            Policy Name: export_policy_volume0
                             Rule Index: 15
                        Access Protocol: nfs
List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: 207.10.137.0/25
RO Access Rule: sys
RW Access Rule: sys
User ID To Which Anonymous Users Are Mapped: 65534
Superuser Security Types: any
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: use_export_policy
Policy ID: 77309411342

pure perch Oct 4, 2023, 9:48 PM

#

Mount the "/" would only be a test, not a solution.

#

There may be some other rule you are not listing that is denying the access

#

it might be useful to see the first 15 rules on export_policy_volume0. The 207.10.137.0/25 rule (15) should allow those hosts and it is likely never even getting to rules 1054/1055.

indigo nimbus Oct 4, 2023, 10:03 PM

#

I am sorry, but I need to know reasons why we need to do such test.
I've checked all ruleindex 1-15,and the rest all rules as well. Other than ruleindex 15, and the only one has "207.10.137", there are nothing could be fit in by 207.10.137.57 or 207.10.137.58. all 1-14 has clear and obvious different clients. All these 3 IP's, .25, .57, and .58 should be in the same subnet, and can fit in the same rule 207.10.137.0/25, in my opinion. So, .25 works due to ruleindex 15, .57 and .58 should work due to the same rule.

pure perch Oct 4, 2023, 10:44 PM

#

did not say you need to test. just a suggestion to see if you can at least get to the root of the namespace. If you can't mount /, then other problems are there. If you can mount / but then cannot change directory into your volume, then there is definitely something up with a policy rule preventing access. If you dont want to share...thats fine. I still suspect there may be an errant rule.

#

what about looking at the other parts of the volume:
vserver security file-directory show -vserver svm1 -path /vol/svm_root_volume
vserver security file-directory show -vserver svm1 -path /vol/volume0

#

are you using qtrees? (qtree show) Basically, is volume0/op a normal directory or a qtree? if it is a qtree, it is entirely possible that there is another export-policy attached to the qtree.

indigo nimbus Oct 4, 2023, 11:56 PM

#

*> vserver security file-directory show -vserver svm1 -path /vol/volume0

Vserver: svm1 File Path: /vol/volume0 File Inode Number: 64 Security Style: unix Effective Style: unix DOS Attributes: 10 DOS Attributes in Text: ----D--- Expanded Dos Attributes: - UNIX User Id: 0 UNIX Group Id: 10 UNIX Mode Bits: 755 UNIX Mode Bits in Text: rwxr-xr-x ACLs: -

*> vserver security file-directory show -vserver svm1 -path /

Vserver: svm1 File Path: / File Inode Number: 64 Security Style: unix Effective Style: unix DOS Attributes: 10 DOS Attributes in Text: ----D--- Expanded Dos Attributes: - UNIX User Id: 0 UNIX Group Id: 0 UNIX Mode Bits: 711 UNIX Mode Bits in Text: rwx--x--x ACLs: -

*> qtree show -vserver smv1 -volume /vol/volume0 -qtree op

Vserver Name: svm1 Volume Name: volume0 Qtree Name: op Actual (Non-Junction) Qtree Path: /vol/volume0/op Security Style: unix Oplock Mode: enable User ID: root Group ID: root Unix Permissions: ---rwxr-xr-x Qtree Id: 8 Qtree Status: normal Export Policy: export_policy_volume0 Is Export Policy Inherited: true QoS policy group: -

#

Please find the outputs above for your requests:
vserver security file-directory show -vserver svm1 -path /vol/svm_root_volume
vserver security file-directory show -vserver svm1 -path /vol/volume0

/vol/volume0/op is a qtree, but has the same export-policy as volume0, export_policy_volume0

pure perch Oct 5, 2023, 12:34 AM

#

Silly question, what is the exact path that is being tried on the clients that are failing? Possibly any typo?

indigo nimbus Oct 5, 2023, 2:03 AM

#

No, not possible due to the reason I already said at beginning. Why don’t you think it could be networking ACL between the clients and NetApp?

pure perch Oct 5, 2023, 9:54 AM

#

Because the reason I said. You seeing the access denied message. If it were a network acl it would likely just block and you would see nothing.

By exact path I mean the mount command that you’re trying.

At this point you really need to get someone to test things and see what works and troubleshoot. If you are unwilling to troubleshoot can’t be much more help

indigo nimbus Oct 5, 2023, 11:23 AM

#

The mount path we are using is: "/vol/volume0/op" which is qtree and has the export policy of "export_policy_volume0". Again, not just this one, there are a lot of other volumes have the same issues on this client. The same volumes have been mounted using the same way by ~1000 other clients without issues.

The only thing I didn't do what you asked for is to mount "/", which is something my manager wouldn't permit, and System admin doesn't want to do, because they believe there are no rules before 15, 1-14 would stop the client 207.10.137.57 to meet 207.10.137.0/25

indigo nimbus Oct 5, 2023, 6:37 PM

#

As we suspected, the issue was with ACL, once we enable it, we are able to mount volumes.
So, it looks "access denied" error is not necessarily meaning the networking can go through.

#ONTAP 9.11.1P11 Cannot recognize "nblade access-cache show" command