Continuous profiling for Dapr sidecar

[viral plinth]

People from Grafana have launched https://grafana.com/oss/phlare/ at the ObservabilityCON 2022. How about enabling pprof on Dapr using annotations in the same way we do for metrics? https://docs.dapr.io/operations/monitoring/metrics/metrics-overview/#configuring-metrics-in-kubernetes

The following annotations can be added:
dapr.io/enable-pprof: "true"
dapr.io/pprof-port: "9090"

Then, we can use the native Go package for profiling: https://pkg.go.dev/net/http/pprof,

Is that useful for someone?

[light pawn]

Doesn't that only work with our Debug containers?

Where do you want to enable it? Where should it be run?

[viral plinth]

We do remove the DWARF debugging information from our non-debugging binary, however this is not a problem if we also generate the dwarf location list on the same compiling process (using -dwarflocationlists) compiler optimizations will continue to be applied and the the dwarf list will help debugging process

Regarding where it should be run: I’m assuming that you are asking about the phlare db right?

If so, the user have to manage it somehow, same way they do for Prometheus, the goal of this feature is to make possible to collect the information from sidecar, users should handle the necessary infrastructure to make it work

[viral plinth]

well, looks like we already have a profiler switch to enable/disable it, and the pprof handlers are already registered as well, https://github.com/dapr/dapr/blob/master/pkg/http/server.go#L162

It should be easier then, it is just a matter of adding the scrape annotations I guess, and include -dwarflocationlists=true in our compilation flags

@golden flicker WDYT?

[viral plinth]

after further investigation I found that the profiling port annotation is missing and defaults to 7777

[light pawn]

Have you read this? https://docs.dapr.io/developing-applications/debugging/debug-k8s/debug-dapr-services/

That enables the profiling port etc. please check whether anything you need is missing there. We don't enable the profiling - and we shouldn't

[viral plinth]

https://docs.dapr.io/developing-applications/debugging/debug-k8s/debug-dapr-services/
Yes, but this is punctually, right? not continuous

We don't enable the profiling - and we shouldn't
Would you mind to elaborate why we shouldn't ?

I think this is the goal of having a continuous profiling database. The idea is to allow users to profile their Dapr sidecars - which currently is possible, but not continuously, thats the case. users can choose to not profile their sidecars, the idea is to provide a way to do this

[light pawn]

We should not enable it in our images by default.. I am not talking about our long haul cluster or similar. There it is fine of course

[viral plinth]

Note that the idea isn't publishing an image without compiler's optimizations or something similar. We will distribute our binary as it is today but compiled with a simple gcflags=-dwarflocationlists=true, the ldflags will be kept as it is currently.

No impact for the binary size: I tested it out with and without dwarflocationlist, the binary size is the same

check it out: https://github.com/mcandeia/dapr/blob/ft/profiling/Makefile#L148

I have tested it out and it worked, if you want to:

1. use the daprd image ghcr.io/mcandeia/daprd:1.11.0-pprof-linux-amd64
2. run with --enable-profiling or with the following annotations:

        dapr.io/enable-profiling: "true"
        dapr.io/sidecar-listen-addresses: "0.0.0.0"

3. use portfowarding 7777:7777 on k8s on run in your local machine go tool pprof http://localhost:7777/debug/pprof/profile\?seconds\=20

[light pawn]

I would suggest a long haul test to compare to be on the safe side 🙂