permission denied errors

[feral thunder]

here is my environment
OS RHEL 9.4
Rancher RKE2 v1.30.6
Trident version 24.10.0
Did helm install

• helm install trident netapp-trident/trident-operator --set iscsiSelfHealingInterval=3m0s --set iscsiSelfHealingWaitTime=6m0s -n trident
  Created backend.json (using iscsi)
  Created storage class
  --- now fun begins
  deploying monitoring app (grafana/prometheus) and getting permission denied
  volume is created on netapp side when an app is deployed
  but getting permission denied when the app tries to create files...
  this is exact error (for monitoring app)
  caller=query_logger.go:93 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"

---

I thought maybe some issue with the app, so deployed different app and have same issue with permission denied

I saw a doc where it mentioned it could possibly selinux, so to test it out; rebuilt the cluster after disabling selinux (this is on my sandbox and not prod)
but I have exact same errors.

Here is my attempt to deploy cockroachdb; once again, I See volume get created but the pod fails to start and in log I see:
creating data directory: mkdir /cockroach/cockroach-data/auxiliary: permission denied

[tender hedge]

It sounds like an export policy issue. What nfs version have you specified in the config?

[feral thunder]

oh - sorry, this is using iSCSI

this is my backend.json (with IPs and password removed)

{
"debug":true,
"managementLIF":"<LIF_IP>",
"svm":"<svm_name>",
"backendName": "<backend_name>",
"username":"vsadmin",
"password":"<password_removed>",
"storageDriverName":"ontap-san",
"storagePrefix":"rke2test_",
"version":1
}

I followed the steps mentioned here (to repare worker nodes) https://docs.netapp.com/us-en/trident/trident-use/worker-node-prep.html#install-the-iscsi-tools

[tender hedge]

What does it show in the output of - kubectl get pvc

[feral thunder]

I do see PVC got created (and also on netapp side)

same with (I see PVC and backend UUID)- tridentctl -n trident get volume

[tender hedge]

Though I have not encountered this error but I was reading some similar issues - check this out https://stackoverflow.com/questions/68079774/write-permissions-on-volume-mount-with-security-context-fsgroup-option

See if this helps else let's wait for someone else to provide more insight into this

[feral thunder]

I do have securitycontext set on both apps

[feral thunder]

so this was interesting; fix was to add fstype: ext4 for ontap-san storage class, and now no more permission denied issue