ComfyUI to API issues

[woven elm]

Hello! My team and I are moving to runpod and are preparing to scale big and I wanted to share what we are doing in hope that someone finds a better way to handle it.

We currently have one universal Dockerfile that prepares the folders, and creates symlinks to models and custom_nodes we have in an S3 volume which the serverless instances have access to.

This S3 volume has all the models and custom_nodes folders, and each GPU on boot installs all dependencies of custom_nodes.

Does all of this make sense or is there a better way to do it?

After spending the entire week building this I saw there is a "ComfyUI-to-API" tool but it prepares a dockerfile that makes each container on each GPU to download everything every time, which doesnt sound very interesting to reduce costs on amount of time spent...

Another big issue:

My workflows tend to make CPU AND GPU at 100%, and im pretty sure CPU being at 100% makes the images be generated with very weird looking artifacts, and the overall composition doesnt make sense, but running the same workflow locally on an even worse GPU works fine...

[woven elm]

Can someone check my conversation with the bot? #1472036951799500923

[woven elm]

Does this make sense?

[woven elm]

Just to be clear…

My issue is the serverless instances are generating bugged images from a certain workflow (images are super blurry), but that same workflow with the same files locally works totally fine

[ember pelican]

This is how serverless will work, from queue load balancer and serverless pod + also usually people use the network storage (similar concept symlink, you get your files in each worker / serverleess pod)

[ember pelican]

did you test by running the docker image locally?

try checking the CUDA version, make sure it matches, any package versions installed are similar

and check if your model isn't corrupted

[woven elm]

How do people load modules from /runpod-volume without creating symlinks in /comfyui subfolders?

[woven elm]

not really, but a similar workflow works with that same image.

[woven elm]

could this be an option though? wouldnt the workflow fail to run if that happened? or would it silently fail and keep "working"?

[ember pelican]

because the loader might think that its valid, keeps loading it, even though some data are missing / invalid

[ember pelican]

oh did you mean load custom nodes in comfyui?, you can symlink it, or another way is using the extra paths.yaml file of comfyui

[woven elm]

would them still be reinstalled again on every cold start? Im trying to plan everything properly because we will scale this to multiple tens of workers and im trying to reduce "dead" time as much as possible

[ember pelican]

yes if its not in the network storage

you should install it in network storage or just in the docker image itself