vernal lodge
Hi guys, I have an old version of Trident 21.10.0 and k8s 1.23.10, but it was working well up until yesterday when it stopped 🙂
We have a cluster with Trident and OnTap NetApp used as NFS storage.
The problem is that after restart of trident-csi pods of daemonset, it takes minutes or hours for them to fully start.
The pods can't register nodes in the controller (which I've tried to restart as well).
error="could not log into the Trident CSI Controller: error communicating with Trident CSI Controller; Put \"https://172.17.51.66:34571/trident/v1/node/xxxx\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)```
The debug logs of the trident-main container show that it is often takes 2 min to complete request :
time="2023-10-11T12:07:03Z" level=error msg="context canceled" handler=AddOrUpdateNode node=xxx requestID=xxx requestSource=REST
time="2023-10-11T12:07:03Z" level=debug msg="REST API call complete." duration=2m28.227231345s method=PUT requestID=xxx requestSource=REST route=AddOrUpdateNode status_code=400 uri=/trident/v1/node/xxx
But after a while it may take25 seconds:
time="2023-10-11T12:16:33Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=xxx requestID=xxxx requestSource=REST
time="2023-10-11T12:16:33Z" level=debug msg="REST API call complete." duration=25.81268769s method=PUT requestID=xxx requestSource=REST route=AddOrUpdateNode status_code=201 uri=/trident/v1/node/xxxx
Also if I run tridentctl get backends -n trident the command hangs forever. At the same time I can list detail about backend using kubectl get tridentbackend -n trident
I don't see that the node where the controller is running of the controller's pod is having any performance issues. ETCD database is relatively busy (300mb in size) but the control plane nodes have plenty CPU and memory resources.
Does anyone know what could cause such slowness of the controller?
Thanks a lot!