#Wondering if it's possible ot update Resiliency spec on the fly

1 messages · Page 1 of 1 (latest)

cold osprey
#

Hi,
I've come across Dapr while searching for a ApiGateway-like solution where I can define some resiliency features such as circuit breaker, timeout, retry etc.
We have a system where we define and update timeout durations and retry configs for specific services. Is it possible to retrieve that data somehow and feed Dappr with this? For example, looking at https://docs.dapr.io/operations/resiliency/policies/#timeouts:

spec:
  policies:
    # Timeouts are simple named durations.
    timeouts:
      general: 5s
      important: 60s
      largeResponse: 10s

Is there a way to update largeResponse value to something else on runtime by an API call, or define that as a variable so I can populate that variable time to time?

Thank you,
Gokcen

runic cypress
#

Each sidecar needs to be restarted to pick up changes in the resiliency configuration. This cannot be altered "live" at this time.

cold osprey
#

Thank you @runic cypress .

Is there a way to pass this information (ie. timeout) to the sidecars through an API? Another question, what happens to the ongoing requests during the restart, is there a way to provide blue-green restart/reedploy for sidecars?

runic cypress
#

No there is no API for this - it needs to be configured via the CRD (yaml file) when the Dapr sidecar starts.

So in Kubernetes you can restart things in different ways.. usually Kubernetes will stop routing new requests to a container that it is about to restart. kubectl rollout restart is what I usually use. The sidecar itself is injected when the app deployment is created / started. It uses a Kubernetes primitive feature called a mutating admissions webhook.

Blue / Green deployments are more complicated to configure.. I let @coarse parcel chime in here because he seems to have lots of good production advise. As a maintainer I don't really run Dapr in production.

cold osprey
#

Thank you @runic cypress .

We are considering using http pipeline (ie. middleware) to implement our custom logic: https://docs.dapr.io/operations/components/middleware/

Maybe you or @coarse parcel would have an answer for this:
Can we set the timeout for individiual requests in a custom middleware component? We are thinking of retrieving timeout value from somewhere (like from an API or a state store/DB), then apply the timeout to the request. This brings another question, such as, can we make a separate call during http pipeline execution?

I believe a snippet like this can be used:

            ctx := context.WithTimeout(r.Context(), time.Duration(MY_VALUE * time.Second))
            r = r.WithContext(ctx)
            next.ServeHTTP(w, r)

Looking forward to hear your opinion

coarse parcel
#

Sorry folks, I can't help on this one. Blue/Green deploys is not something we do. I also don't have any hands-on experience with the dapr middleware 😦

runic cypress
#

First of all - if you create a custom middleware component it probably will not be added to Dapr. So you'd have to run your own unsupported Build of Dapr.

In a middleware component you could certainly use any third party API or SDK, but do not try to call into the dapr sidecar from a component itself - that's not the right approach. And middlewares today cannot interact with other dapr components reading or looking up data.

There is a related issue opened recently in dapr/dapr to create a distributed rate limiter middleware

cold osprey
#

I don't know the implications of using unsupported Dapr, can you enlighten me on it?

In theory, is it possible to build Dapr ourselves, by adding custom functionality as in https://github.com/dapr/components-contrib/ middleware components and build Dapr with it & use? Can you direct me to a doc explainig the process?

We are not planning to call Dapr sidecars from there, but, there will be some call to a data store directly (which is actually against Dapr's mission but still).

There is a related issue opened recently in dapr/dapr to create a distributed rate limiter middleware
We won't need rate limiting based on counters but it's about dynamic allowlisting and timeouts based on the source and target of the request. So this wouldn't resolve our issue.