Mixed Delay Times | Runpod | Page 1

frosty geyser May 22, 2024, 2:06 PM

#

Hey, what could be the reason for these delay times?

sonic gazelle May 22, 2024, 2:08 PM

#

Could be various different reasons

Not enough workers to handle the number of concurrent requests, so requests sit in the queue
Cold start time (more common)

frosty geyser May 22, 2024, 2:13 PM

#

I'm guessing I can't control the cold start time?

#

I don't think workers are an issue

sonic gazelle May 22, 2024, 2:14 PM

#

You can do things like enabling flash boot, increasing idle timeout, adding active workers etc to improve cold start times.

#

FlashBoot is the only one thats free though.

frosty geyser May 22, 2024, 2:19 PM

#

How does flashboot work?

sonic gazelle May 22, 2024, 3:00 PM

#

frosty geyser How does flashboot work?

"magic"
https://docs.runpod.io/serverless/references/endpoint-configurations#flashboot

Endpoint configurations | RunPod Documentation

Configure your Endpoint settings to optimize performance and cost, including GPU selection, worker count, idle timeout, and advanced options like data centers, network volumes, and scaling strategies.

warm wyvern May 22, 2024, 7:05 PM

#

frosty geyser How does flashboot work?

Basically the tldr though from asking them is it's a caching mechanism, so the more max workers u have, the more requests, the better the cache

#

if u have an active worker, supposedly is even faster, but i dont think is necessary, cause ive heard from people using it in prod that the flashboot is still quite fast normally even with a min worker of 0

#Mixed Delay Times