somber zenith
I have a serverless endpoint which is loading a bunch of loras on top of sdxl, and the time it takes is a lot (more than 500 seconds) on the first load.
This used to work well until I added even more loras, and now it's timing out and "removing container" and restarting it again and again
Any tips to fix this?
mossy salmon
I have a serverless endpoint which is loading a bunch of loras on top of sdxl, a...
any tip's?
kind flower
I have a serverless endpoint which is loading a bunch of loras on top of sdxl, a...
Wow how many lora's do you use? and how do you load it in code?
lou do you have the same problem too?
somber zenith
Wow how many lora's do you use? and how do you load it in code?
I use load_lora_weights
And i load it outside handler
I'm using the sdxl worker serverless template
somber zenith
yes I did
class ModelHandler:
def __init__(self):
self.base = None
self.compel = None
self.load_models()
def load_base(self):
base_pipe = StableDiffusionXLPipeline.from_single_file("./anime/autism.safetensors", add_watermarker=False, use_safetensors=True, torch_dtype=torch.float16, safety_checker=None)
base_pipe = base_pipe.to("cuda", silence_dtype_warnings=True)
base_pipe.enable_xformers_memory_efficient_attention()
compel = Compel(tokenizer=[base_pipe.tokenizer, base_pipe.tokenizer_2] , text_encoder=[base_pipe.text_encoder, base_pipe.text_encoder_2], returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED, requires_pooled=[False, True])
for weight in lora_weights:
print("Loading pose weight:", weight)
base_pipe.load_lora_weights(f"./anime/loras/{weight}.safetensors", adapter_name=weight)
for character in list(character_configs.keys()):
print("Loading character weight:", character)
base_pipe.load_lora_weights(f"./anime/loras/{character}.safetensors", adapter_name=character)
return base_pipe, compel
def load_models(self):
with concurrent.futures.ThreadPoolExecutor() as executor:
future_base = executor.submit(self.load_base)
self.base, self.compel = future_base.result()
MODELS = ModelHandler()
have any idea why this happens? it's a lot of loras I think around 30
it's not a memory issue as far as I know, it's about the first run of the serverless timing out before all the loras load in
somber zenith
what i'm doing is using load_lora_weights outside handler and then using set_adapter() inside the hanlder to activate the lora
if anyone has a clue on how to fix what's happening please let me know
near willow
You can't really fix this, RunPod expects the handler to kick in within a certain time period.
Why do you need to load 30 Lora anyway?
somber zenith
You can't really fix this, RunPod expects the handler to kick in within a certai...
I can't extend this time period?
somber zenith
Why do you need to load 30 Lora anyway?
It's for our image generation service
I dont use all those loras at once, but rather I load them all, then use set_adapter to activate only the ones I require, this way I dont have to load in and load out every lora during every request
kind flower
No i don't really have any idea if this is normal... i never tried loading that much lora's, but i guess its a normal time for loading all of them?
torn sinew
If it's not getting them into memory, that's a problem I'm not sure about. If it's not getting them onto disk from the download before dying, that one I've seen and solved by using a network storage as long as it resumes the download. Then it can work on the downloads until it dies, and pick up where it left of the next time it tries, and eventually finish and not have to download them again afterward.
somber zenith
the files are already on the docker container
@torn sinew
it dies before being able to load everything into ram
torn sinew
Ah. I have no advice on that one unfortunately.
kind flower
I have a serverless endpoint which is loading a bunch of loras on top of sdxl, a...
Maybe it's normal?, did you benchmark somewhere else and it's faster than the performance here
I mean 30 loras is a bunch of loras right, with loading sdxl too
somber zenith
I don't mind it loading for however long it wants but I'd like for it to fully load
What happens is before it loads all 30 loras there's some sort of time out which restarts the worker and it retries loading all of them back in again
And that keeps continuing
Anysy i came up with a different solution to my problem so it's all good now
kind flower
Anysy i came up with a different solution to my problem so it's all good now
Oh how did you solve it?
oh yea i guess its runpod problem, it might timesout because they think it doesn't load
somber zenith
Oh how did you solve it?
I'm loading the lora once at request, and then not unloading it for a new request, It always checks if the lora is already loaded
this way any user who requests a specific lora it only takes extra time once