How to set max concurrency per worker for a load balancing endpoint?

[wet elm]

I'm trying to configure the maximum concurrency for each worker on my serverless load balancing endpoint, but I can't seem to find the setting in the new UI.

[lean ocean]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[broken goblet]

its from an env variable right?

what does maximum concurrency do in vllm?

[coarse wagon]

@wet elm , were you initially setting up your endpoint. When creating an endpoint on Serverless, we do the calculation for you. Once then endpoint is setup, you can then edit the endpoint and adjust as needed.

[wet elm]

I mean in the queue based serverless worker, there is a concurrent handler that I can control concurrency for each worker https://docs.runpod.io/serverless/workers/concurrent-handler, I want to know how to do same control for load balancing workers

[broken goblet]

I think it's the request count only

[sterile ledge]

Does this work if we set the concurrency within the fastapi itself, as it supports custom endpoints? Can we process n parallel requests with n concurrency using 1 worker this way?

[broken goblet]

Hmm I'm not sure of that

How is concurrent handlers on load balancing? @coarse wagon is it supported, or what's the logic for balancing to the workers?

[brisk cipher]

I got a question

What happens if worker changes /ping endpoint response code to 204 if the vllm worker is overloaded

Does it make the load balancer not route more requests to it?

[tidal fern]

set requests per worker setting, at the moment its constant, your fastapi should be able to handle that many requests in parallel

[broken goblet]

Oh the scaling config?