High delay time | Runpod | Page 1

hasty gorgeBOT Apr 13, 2025, 12:27 AM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

cunning yarrow Apr 13, 2025, 12:41 AM

#

Stuck. Not sure why

grim bronze Apr 13, 2025, 7:07 AM

#

I don't get this, are you trying to download docker image inside a docker(runpod)

#

If so, you shouldn't, because it's not quite supported except cpu pods

#

You should use templates directly on runpod and refer to a registry from runpod, let runpod's system handle the docker image download

cunning yarrow Apr 13, 2025, 9:59 AM

#

grim bronze I don't get this, are you trying to download docker image inside a docker(runpod...

I currently have a vLLM pointing to a model on Hugging Face model, but the requests aren't going through

grim bronze Apr 13, 2025, 10:04 AM

#

I see, Can you check the logs

#

Or via workers tab then check one worker's logs

cunning yarrow Apr 13, 2025, 10:04 AM

#

Okay sure

grim bronze Apr 13, 2025, 10:05 AM

#

cunning yarrow Stuck. Not sure why

But the weird thing is this, it's seems like it's still pulling and its shown on the logs

ember gardenBOT Apr 13, 2025, 10:05 AM

#

cunning yarrow Stuck. Not sure why

@cunning yarrow

Escalated To Zendesk

The thread has been escalated to Zendesk!

grim bronze Apr 13, 2025, 10:05 AM

#

Maybe also create a support ticket for that

ionic coyote Apr 13, 2025, 10:08 AM

#

cunning yarrow I currently have a vLLM pointing to a model on Hugging Face model, but the reque...

what model and the vllm settings

#

please

cunning yarrow Apr 13, 2025, 10:13 AM

#

Model: deepseek-ai/DeepSeek-R1

cunning yarrow Apr 13, 2025, 10:13 AM

#

ionic coyote please

I used the default VLLM settings

#

In the logs:

ionic coyote Apr 13, 2025, 10:19 AM

#

cunning yarrow Model: deepseek-ai/DeepSeek-R1

lol

grim bronze Apr 13, 2025, 10:19 AM

#

cunning yarrow Model: deepseek-ai/DeepSeek-R1

It says out of memory in the lgos, so you definetely need more vram for that configuration

ionic coyote Apr 13, 2025, 10:20 AM

#

you need about 1.5TB of vram

#

you gave it 48gigs

grim bronze Apr 13, 2025, 10:20 AM

#

Yeah alot of people run into this, it can be expensive to run the large typr of that model

cunning yarrow Apr 13, 2025, 10:20 AM

#

I tried to change the model

#

To "mistralai/Mistral-7B-Instruct-v0.1"

#

I got the response, but it took 2 minutes. Will this be the case for every future request?

ionic coyote Apr 13, 2025, 10:22 AM

#

no only if it is a cold start

#

aka if you dont send requests for a while and then send it

#

it will take some time for the worker to start up

grim bronze Apr 13, 2025, 10:23 AM

#

cunning yarrow I tried to change the model

Hmm exit code 1?

cunning yarrow Apr 13, 2025, 10:23 AM

#

Seems to be okay now, it's just the delay times are high

#

Actually, looks good now

grim bronze Apr 13, 2025, 10:24 AM

#

You can use a ns to prevent model re-download for every cold start

cunning yarrow Apr 13, 2025, 10:24 AM

#

What's an "ns"?

grim bronze Apr 13, 2025, 10:24 AM

#

Network storage

cunning yarrow Apr 13, 2025, 10:24 AM

#

Where do I enable that?

grim bronze Apr 13, 2025, 10:24 AM

#

Edit endpoint

cunning yarrow Apr 13, 2025, 10:25 AM

#

grim bronze Apr 13, 2025, 10:28 AM

#

cunning yarrow

There network volume

cunning yarrow Apr 13, 2025, 10:29 AM

#

Got it, thanks. I appreciate your help guys

#

Great support

grim bronze Apr 13, 2025, 10:35 AM

#

Your welcome

lucid zenith Apr 14, 2025, 8:49 PM

#

I'm experiencing the same issue (SDK version 1.7.7)

#

My jobs are just stuck in the queue, so it's not because of my code

#High delay time