Profiling CUDA kernels in runpod | Runpod | Page 1

novel marlin Apr 3, 2024, 9:33 AM

#

Hi! I'm trying to profile my kernel with nsight-compute and I'm getting error : "==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0."
Which is explained on this page : https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters
and has to be fixed on the host side. Anybody found a workaround for this issue or how to solve it? Thanks!

NVIDIA Developer

scenic fiberBOT Apr 3, 2024, 9:33 AM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

indigo blaze Apr 3, 2024, 9:53 AM

#

you cant do it on RunPod as containers are not provilaged and exposing --cap-add=SYS_ADMIN would cause security risk

novel marlin Apr 3, 2024, 6:31 PM

#

Okay, Thanks for the response. Any idea where can I do this?

indigo blaze Apr 3, 2024, 7:16 PM

#

im not sure what you trying to do

novel marlin Apr 4, 2024, 5:52 AM

#

I'm actually trying to profile few of my custom CUDA kernels running on NVIDIA GPUs

#Profiling CUDA kernels in runpod