Discussion Is Dapr mode necessary?

[sleek breach]

Since my early days working on Dapr I've noticed that we differentiate our runtime modes (Kubernetes vs local) for a single purpose (this is an assumption, correct me if I'm wrong before diving into the discussion): having access to the Specs (Configuration, Resiliency, Component and more) and receiving component updates.

When running on standalone mode those specs can be read from disk (using our homegrown disk loader, when running on Kubernetes mode we get those from the Operator APIs.

So, the operator is not only a Kubernetes operator itself, but it also serves as a "remote Manifest Loader".

The process to read the manifests is remarkably like the standalone mode but instead reading those specs from disk, the runtime makes requests to the operator API and retrieves the manifests back (serialized as a []byte)

At the same time, we strive to provide the same set of features for both "modes/platforms" and be prepared to support more platforms in the future.

In addition to that, we have our injector that is responsible for patching the application pod and prepare the environment to get the Dapr and the app running with proper config.

So, my question is: Is there a reason to not consider the Kubernetes mode as a special case of standalone mode where our Operator/Injector handles mounting volumes that contain all the necessary specs? (i.e using ephemeral volumes)

About component updates: We do have feature requests for dynamically loading components for standalone mode

• dynamic loading components

I see a few advantages:

1. do not need to have two different modes (at least from the runtime (Daprd) perspective)

2. do not have to implement with many "platforms" in place, standalone should be the only one

3. do not depend on the operator API (the component update API might be necessary but can be replaced by file system events?)

Also, with the spec resources path we can have a single volume mount holding all specs in there

[woeful vapor]

That is an interesting proposal because it would unblock Dapr in other runtime platforms (Service Fabric?). The code simplification is a good side effect but not the main reason IMO. Glad to hear others on this too.

[stuck vault]

1. How will the Component CRs be dynamically updated in a volume mount without needing a pod restart?

2. How will the volume mounts be populated with the CRs? Please provide a more in depth answer here than "The operator/injector will do it" because K8s doesn't have an API for you to populate (or update) the contents of volumes, ephemeral or other. Only containers that have volumes mounted to them (with proper permissions) can write to the volume, or reference either a secret/ config map, neither applicable in anyway to this use case.

3. For Kubernetes, when fetching component secrets from the k8s secret store, the component files today arrive to the sidecar with the secrets already populated as the operator extracts the secrets and sends them over an encrypted, mTLS enabled connection. If the operator were to somehow (see question 2) populate the files on a volume, you would essentially be keeping secrets as plain text on disk. What's the mitigation for that?

---

This wouldn't help unblock Dapr in other runtimes, as it offers no additional support from what self hosted mode does today. What would help unblock Dapr in other platforms is to implement the operator API for said platforms, which would enable a consistent control plane across them (and be more secure than storing files locally).

[humble yew]

For problem #2, of populating the volumes... Perhaps a "config map projected volume" may work? https://kubernetes.io/docs/concepts/storage/volumes/#configmap In this case the contents of the resources could be passed as strings as values in the configmap. However, that only solves one of the problems highlighted above.

However, I do have questions about the overall premise of the problem. There are differences in Dapr running in self-hosted vs Kubernetes besides how resources like Components and Resiliency are loaded. For example, the injector service plays a crucial role in K8s and zero in self-hosted (and I still think the injector should be merged into the operator, eventually 🙂 )

[sleek breach]

Please, do not take this as a proposal yet, what I’m trying to validate first is the “what” and “why” before diving on “how” it would be implemented, I’m using this forum to avoid discussion about “why/what” in the github proposal. I’m not saying the “how” is not important but having the motivation behind it aligned among stakeholders is way more important IMO, especially when taking into consideration that the discussion about how we would implement can diverge in many ways but the motivation still the same.

Having said that, the questions you did is exactly what I'm expecting when started this discussion, my goal is to bring it up the opportunity to have the Dapr runtime platform-agnostic, and does not means that we don’t need our operator and injector, they still serve as a crucial piece for Dapr on k8s, the same applies to service fabric, there would be a similar service for each platform we support.

[sleek breach]

They would continue to exist, but with a single role of making the Dapr running environment similar to standalone mode. See that the discussion here is nothing more than making our standalone mode more mature as it would be the "only platform" from the Dapr Runtime perspective

[sleek breach]

1. How will the Component CRs be dynamically updated in a volume mount without needing a pod restart?

That's an interesting question actually, because I'm new here I don't know why not applying rolling updates would be an option here.

At least to me it seems to make more sense even to preserve an immutable production environment. In my past microservice days having an immutable environment (changes happen separately and rollout is done progressively - same for rollingback) is an advantage. In addition, some components need a "cleanup" process to prevent them from using resources that are not actually in use, such as database connections, for these reasons it seems to me that rolling update seems to be a great option, I would be happy in reading the proposal and the motivation behind dynamic updates do not need to restart. Which for me is totally ok, and it was a decision already made, my intention here is to learn.

btw: do we "close" connections when replacing components ? https://github.com/dapr/dapr/blob/4c3b1d24753efac15ad196bfe7cd7dae5fc348d1/pkg/runtime/runtime.go#L2218

2. How will the volume mounts be populated with the CRs? Please provide a more in depth answer here than "The operator/injector will do it" because K8s doesn't have an API for you to populate (or update) the contents of volumes, ephemeral or other. Only containers that have volumes mounted to them (with proper permissions) can write to the volume, or reference either a secret/ config map, neither applicable in anyway to this use case.

I do not have the "right" answer yet. But there are few options, like @humble yew points out using configmaps, or even (implement or use the kubernetes native inline) Inline Volumes(https://kubernetes.io/blog/2020/01/21/csi-ephemeral-inline-volumes/),

Another option is having an Init Container that populates the volume reading the specs from namespace

These are non-exhaustive

3. For Kubernetes, when fetching component secrets from the k8s secret store, the component files today arrive to the sidecar with the secrets already populated as the operator extracts the secrets and sends them over an encrypted, mTLS enabled connection. If the operator were to somehow (see question 2) populate the files on a volume, you would essentially be keeping secrets as plain text on disk. What's the mitigation for that?

Do we support referencing secrets on standalone mode? If not, should we support ?

The point is: Or we say that our standalone mode is "less" secure (Arguably not because the environment could be "better controlled") because it requires using plaintext secrets saved on disk or we do allow same set of features for all platforms. A mitigation strategy could be read the secret from an Environment Variable that can be injected using k8s secrets

btw: even secrets has an option to be mounted as data volumes

yet about dynamic component updates: If a component YAML that has a reference to a k8s secret, when the secret value is updated the sidecars receives that updated component ? if not, is this a smell that we should move that logic to the runtime side instead?

[stuck vault]

The how, in this case, is tightly coupled to the what and why, as this change would be purely technical and addresses a differentiation we made due to the way Kubernetes technically operates. If a proposal carries a technical showstopper, it must be addressed from the get go.

[stuck vault]

1. Dapr offers immutable infrastructure in how it operates today, and in addition we've been getting community feedback (hence the issue) to support dynamic configurations which are more native to the K8s experience in how users expect a dataplane to operate when configuration is updated.

2. Config maps are not the right solution for this. As you can see, it's a workaround for the fact K8s does not intend you to mount objects or CRDs (what it calls essential data) on ephemeral mounts (inline mounts won't help). Going against the design beats user expectations, but also carries technical challenges that might be insurmountable for this workaround. Several questions off the top of my head: how does the operator/injector turn CRDs into ConfigMaps? every time during injection? if so, this would carry severe performance penalties and overload the API server. But more concerning, it means the injector/operator need RBAC permissions to access all ConfigMaps and Dapr CRDs in all namespaces 😖 If done through the operator by listening to changes on CRDs on turning them into Config Maps, this means eventual consistency when users roll out their pods which leads to split brain scenarios. All security/perf concerns remain. Doing it through an Init Container is a no-go because it means each and every user pod must be granted access to GET all the components in the namespace, which carries a huge security risk.

3. I advise to read more about Self hosted mode. feel free to ask me any questions any time by the way! Self-Hosted supports reading secrets from any secret store which is not K8s, which is why we have the Dapr mode 🙂 Env Var secret store isn't recommended for production, this won't pass a security audit let alone a maintainer audit 🙂 Mounting secrets as volumes for sidecars is highly discouraged, projects like Istio and Linkerd moved away from that due to security concerns. For Dapr, this would be a privilege escalation regression.

---

To sum up, I don't see this proposal opening up Dapr to other platforms in any way, but instead requiring complex code and workarounds to try and force Kubernetes into something it wasn't made for. My 2 cents!

[sleek breach]

You always bring good points @stuck vault, thanks for the clarification about self hosted mode, I would like just to clarify one thing: The discussion here is not about getting rid of our operator APIs necessarily, but removing the direct dependency from the runtime and provide a single interface to access components/specs/receiving updates regardless of used platform.

For instance, a valid solution would be having the Init container fetching Specs from Operator and writing them on disk, avoiding permissions issues that you mentioned, or even better IMO is passing out as a container environment variable (as base64?) and the init container get the value and write it on disk (a simple shell script can do that) - Again, I'm thinking out loud here.

It would be very nice to not need the operator APIs, especially because it does not provide a good type-safe implementation when fetching CRDs (we return it as a []byte), I see this as a workaround for serving CRDs through APIs that's not the kubernetes APIs itself (occurs to me that we basically act as a kubernetes proxy for those)

clarification for 3: what I meant by saying using an environment variable is basically re-use the Kubernetes feature to inject secrets as env vars and reference them when specifying the component yaml

However, I do not have a good solution to provide dynamic updates rather than be notified from the Operator and write the file on disk (relying on the file system events), I still need to understand the side effects of doing this update "in place/online".

I would feel more comfortable if all our platform dependencies were brought together in a single interface in the code and implemented for each platform, because the way we have today our "mode check" is repeated for each feature we make, and it ends up getting lost in the code (it's not clear which is the way we support that feature in such platform), so it could be a good intermediate solution though

[stuck vault]

"You always bring good points @stuck vault" - not always, but glad that you think so 😄

Using an init container to fetch the the components from the Operator works, but we need to think clearly about the tradeoffs here. We're trading removing several lines of gRPC code from the runtime in favor of adding a completely new binary and executable container to Dapr, with a new entrypoint, build pipeline and versioning. Is it worth it? Leaning towards no at this moment. In addition, an Init Container also means slowing down Dapr boot time considerably, and means apps will take longer before they can actually reach into Dapr - I imagine this is going to have major implications for platforms implementing Dapr sidecars such as Alibaba's FaaS, ACA and others.

[sleek breach]

We're trading removing several lines of gRPC code from the runtime in favor of adding a completely new binary and executable container to Dapr, with a new entrypoint, build pipeline and versioning.

Yup, please don't!

If an init container would be an option, busybox image + simple shellscript would be enough (I think I saw few examples populating nginx config using busybox Init Container)

[muted condor]

Is this thread about the distinction from a code perspective? From an operations perspective it is very different.

Deploying sentry, placement service etc - that all works rather differently in standalone