Hi! I am new to docker and as well as to serverless. I would like to know what happens during cold start, does my image gets pulled again each time? Or it's just the time required for provising resources and loading models in vram etc?
#What happens during cold start time?
mint ember
cosmic field
Yes, the second one
Image pulling(exc from cold start) in serverless wont be billed, cold start will
mint ember
Awesome. I have another question. I have model weights of 30gbs. Do you recommend adding them in docker image or I should download them using a script to mounted volume?
cosmic field
download them into a mounted volume
mint ember
What's the path of mounted drive?
mint ember
Cool. Thanks for answers
remote hearth
I think flash recommended to add your model a part of docker image whenever possible. It probably better than network volume for performance😀
mint ember
I am super confused now. Lol
cosmic field
What's up
mint ember
How to decide when to add model as part of docker image or download it on mounted drive
cosmic field
I mean if you want you can test them to decide
Which suits better
And time your tests
Lmk if you did that hahah
Your model is quite big right
mint ember
30gbs of disk space
mint ember
Yes, the second one
Image is being pulled on each cold start
cosmic field
No its just the loading of your model, applications
No pulling again
Check the logs
remote hearth
If you think about it, when your Docker image contains everything necessary, the container is ready to go as soon as it starts, with all data stored on the host disk for fast access. In contrast, if you store the model on a network volume, you would need to mount it and connect through the data center’s Ethernet to network storage, which is likely to be slower than accessing the local disk.
mint ember
Check the logs
I checked on my docker hub, I can see how many times my image was being pulled
cosmic field
Okay the logs also say when it's pulled
remote hearth
When deploying your serverless function for the first time requires pulling the image, which can be slow. If you keep sending requests, the container remains active, that avoid cold starts. However, if there’s a pause in requests, Runpod stops the container. When requests resume, a cold start occurs, but it will be faster than the initial image pull.
cosmic field
Yeah cold starts aren't loading image for the first time, cold starts only for when you rarely hit requests to your endpoint
feral gazelle
@cosmic field How long before the endpoint goes “cold”? It doesn't seem to be a constant time, if so do you know what it is?