#SUPIR Upscaling and Restoration

1 messages · Page 1 of 1 (latest)

civic shell
#

Version 2.0.0

NOTE: This needs at least 24GB VRAM for 1x upscale. If you want to upscale more than 1x, you will need more than 24GB of VRAM. 48GB VRAM is recommended.

NOTE: Loading of models on start takes a few minutes, so you can view the log to watch the progress. You will be able to access the port when you see Running on local URL: http://0.0.0.0:3001 in the log.

This template includes:

Get it here.

Note: You can add an environment variable called NO_GPU_OPTIMIZATION if you are using H100 or A100 GPUs with 80GB of VRAM.

solid axle
#

@civic shell You should add note that loading models on boot takes time and it will ui not to work

civic shell
#

Thanks @solid axle , done!

solid axle
#

Cause logs not showing anything and on SPIR logs you might see this

civic shell
#

Yeah don't know why it appears to hang like that for a while

solid axle
#

It's like precompiling models

civic shell
#

Ah ok

solid axle
#

running on H100

civic shell
#

I'm only testing on A6000, A5000 worked fine for small images but not for large images

solid axle
#

Im doing mostly DVD to 4K

#

hmm takes so long to start on H100

civic shell
solid axle
#

suposed to be CUDA 12.2

civic shell
#

This thing actually seems to work fine on A6000

solid axle
#

Going to try with A100

civic shell
#

I scaled up a 1.9MB image x3 on A6000 😁

#

Took 426 seconds though

#

@solid axle did you change the start command to remove all the GPU optimized arguments?

solid axle
#

nope

civic shell
#

You should probably remove them with such a powerful GPU

solid axle
#

where I should look for it?

civic shell
#
fuser -k 3001/tcp
cd /workspace/SUPIR
source /workspace/venv/bin/activate
export HF_HOME="/workspace"
nohup python3 gradio_demo.py \
        --ip 0.0.0.0 \
        --port 3001 \
        --use_image_slider > /workspace/logs/supir.log 2>&1 &
solid axle
#

Prepared lot of images for testing

civic shell
#

I scaled the 1.9MB image up 4x, took like 15 mins but no OOM or anything, and got a 17MB resulting image.

#

So I think A6000 is a pretty good general purpose GPU for my template without modifying the GPU optimizations. I think it will go MUCH quicker on an A100 or H100 though without the GPU optimizations.

#

I can't scale it up x5 though, because then the system/CPU memory is too low and the kernel kills off the gradio app. 62GB RAM and it gets killed off 😆

solid axle
#

A6000 is my to go gpu too most times

#

I would say going over 4 would be usles

#

I'm expert with super resolution

solid axle
#

hmm models load so slow

#

or it might be my internet issues

civic shell
#

No they do load very slowly

solid axle
civic shell
#

Oh yeah thats AMAZING 😍

solid axle
#

I mean it's obscure flv source

vague spoke
solid axle
vague spoke
#

Yes it is sueprb on blurry images. I would like to see more in the style of magnific - where you already have high quality image but need to upscale it and add details

solid axle
#

it is adding details

vague spoke
#

Try it with already nice looking image 🙂

solid axle
#

the model is not made for that. You want use img2img

vague spoke
#

Yes, I guess so. It is just that it was compared to magnific, but it is totally different thing

#

Magnific can't deal with low quality and blurry images well

solid axle
#

I do not seen enywhere where they compare with magnific

#

you could probably add more s-noise

civic shell
vague spoke
#

Reddit too 🙂

solid axle
#

From what I see it's 2 diffrent tools

civic shell
#

Even the title says "SUPIR: New SOTA Open Source Image Upscaler & Enhancer Model Better Than Magnific & Topaz AI Tutorial"

vague spoke
#

People are just desperate for cheaper tool like magnific 🙂

solid axle
#

magnifc is focusing on upscaling + img2img where goes of SUIP is just upscaling

vague spoke
#

Are you sure?

#

Because it could not do it just upscaling. It obviously adds to the image, but just enough and where it needs to

civic shell
#

I don't think its necessarily adding to the image, more like restoring/enhancing it

solid axle
#

yup it's only to go restore existing details from image

civic shell
#

Kinda like CodeFormer does

vague spoke
#

Ok. I too was hoping it worked kind of magnific 🙂 It is still great for upscaling without loosing detail or restoring low quality images

solid axle
#

magnifc is more like auto1111 with img2img

civic shell
#

oops sorry

solid axle
#

do not worry saved image

civic shell
vague spoke
#

How does it "upscale" something that is not there? Just curious 🙂

civic shell
#

I'm busy pushing a new image that uses Juggernaut XL model instead as Furkan said its better..

civic shell
solid axle
#

tried to say it in prompt

civic shell
#

Updated to version 1.1.0

  • Switched out SDXL base model with Juggernaut-XL-v9 SDXL model.
  • Added support for NO_GPU_OPTIMIZATION environment variable when using H100/A100 GPUs with 80GB VRAM.
  • Removed auto-update which breaks things when model is switched out.
solid axle
civic shell
solid axle
vague spoke
#

I was thinking of using some weird workflow with this 🙂

#

Using SD image, then run it through Magnific at x2 (because it costs only 5 credits) and then upscale it big with SUPIR

#

This way would get the added details from magnific and the clear upscale from SUPIR

civic shell
#

Its worth a shot, maybe you can give it a try and let us know how it goes?

vague spoke
#

Trying 🙂

solid axle
#

@civic shell you plan to add that batch support update from SC?

civic shell
#

Hopefully not too many more changes, I need to go to sleep 😆

solid axle
#

in theory I can just git pull

civic shell
#

I am not using his repo 🙂

#

Its using my own fork

solid axle
#

Kill parent with fork

civic shell
#

He disables LLaVA by default, I enable it, I use different LLaVA model etc

civic shell
solid axle
#

i joke

civic shell
#

Yeah forks suck if they are not pushed upstream

#

@solid axle my github repo has been updated if you wanna git pull

solid axle
#

RunPod is breaking

civic shell
#

Dunno why it does that sometimes, its very annoying

solid axle
#

btw is it correct

civic shell
#

Yeah as long as it exists, the value doesn't matter

solid axle
#

hmm intresting I pulled the update but no batch option shows

#

ok now shows

civic shell
#

Updated to version 1.1.1

  • Added batching support.
  • Use step of 0.1 instead of 1 on upscale slider.
solid axle
#

Batch upscaling in progress

solid axle
civic shell
#

Wow crazy 😆

civic shell
civic shell
uneven sparrow
#

I got some errors on a A6000 deployment. 1) python3: can't open file '//gradio_demo.py': [Errno 2] No such file or directory - after the 'native' deploy, guess that's correct, then after finding the right location 2) root@969f1a4c0dd6:/# /SUPIR/gradio_demo.py
bash: /SUPIR/gradio_demo.py: Permission denied - will it copy to my workspace at some point or should just gitclone your repo into my workspace and run it from there? [confused]

civic shell
#

You don't need to start the application manually, the pod starts it for you.

solid axle
#

Btw @civic shell

uneven sparrow
civic shell
civic shell
#

Reddit?

solid axle
#

The app on first run loads multiple models. What if we make model controled by env variable

civic shell
solid axle
#

Guess someone is going to lose income 🫠

#

We would need change ckpt path to load from env

civic shell
solid axle
#

you can set it in CKPT_PTH.py

civic shell
solid axle
#

you can add there to it also works

civic shell
#

Not sure why there are 2 different files for loading models, they should all be in 1 place

solid axle
#

hmm will need to do reaserch on this

civic shell
#

I'll look into it later, busy with other work at the moment

solid axle
#

same

uneven sparrow
# civic shell Did you wait for your pod to complete syncing to workspace?

Hi Ashley, after 30 mins, still nothing in workspace :/ that's the container log: 2024-03-01T10:07:18.808966873Z RunPod Uploader started
2024-03-01T10:07:18.809272594Z Running pre-start script...
2024-03-01T10:07:18.811172902Z Template version: 1.1.1
2024-03-01T10:07:18.821051842Z Existing version is newer than the template version, not syncing!
2024-03-01T10:07:18.821197062Z Updating SUPIR
2024-03-01T10:07:18.823314621Z /pre_start.sh: line 56: cd: /workspace/SUPIR: No such file or directory
2024-03-01T10:07:18.825397470Z fatal: not a git repository (or any of the parent directories): .git
2024-03-01T10:07:18.825603761Z Starting SUPIR
2024-03-01T10:07:18.830640911Z SUPIR started
2024-03-01T10:07:18.830656381Z Log file: /workspace/logs/supir.log
2024-03-01T10:07:18.830809622Z All services have been started
2024-03-01T10:07:18.911569821Z * Restarting OpenBSD Secure Shell server sshd
2024-03-01T10:07:18.946465244Z ...done.
2024-03-01T10:07:18.957892140Z FileZilla SFTP configuration file created at: /workspace/filezilla_sftp_config.xml
2024-03-01T10:07:18.957913660Z Exporting environment variables...
2024-03-01T10:07:18.961883537Z Container is READY!

solid axle
#

You want to see tail command from readme

#

tail -f /workspace/logs/supir.log

uneven sparrow
solid axle
#

is it community or secure cloud?

civic shell
uneven sparrow
#

secure cloud in RO with my network storage attached

civic shell
solid axle
#

ye probaby this is an issue

uneven sparrow
#

ok, got it, a bit inconvenient - any chance to change that?

civic shell
#

should probably stop putting venv in /workspace/venv and move it to the app directory then the storage can be shared but I think its a bit too late for that now 🙈

#

I can start doing that for future template versions

solid axle
#

Probably you would need to be making venv directory named by app

civic shell
#

Official RunPod tamplates also do this though so they should also probably change

solid axle
#

What if we have it like
/workspace/app/venv

civic shell
#

I think just putting it in app folder is probably best though

solid axle
#

or wait better idea
/workspace/venvs/app
/workspace/app/app1

civic shell
#

Oh yeah thats pretty sweet

#

Lots of ways to solve it

#

I just pushed SUPIR template version 1.2.1

#

It removes the auto update via git pull so I can switch the default model back without breaking anything

#

I also noticed Juggernaut screwed up my eyes like the Reddit post mentioned, so I think its better to make base SDXL the default for now until I have time to add the environment variable later

solid axle
#

@civic shell intresting also getting same error with network storage

#

intrestin is that it has //

civic shell
#

Thanks, didnt test it with network storage, will take a look.

solid axle
#

pulling new update will see if it also breaks

civic shell
#

There was a problem with rsync command, I am pushing a fix, so better to wait for the update

#

Will take a while because I have to push like 40GB due to changing model back to SDXL base

solid axle
#

huh I was thinking if we could pack models into docker layers so they can be reused between templates

civic shell
#

They are already in Docker layers but for some reason it is pushing all the models again even though only 1 changed

uneven sparrow
#

Hi guys, sorry for being a pain, but I am again at the 30 minutes mark - this time with SUPIR sync - no network storage this time: 2024-03-01T11:04:20.962474890Z Running pre-start script...
2024-03-01T11:04:20.965771068Z Template version: 1.2.1
2024-03-01T11:04:20.971211959Z Syncing venv to workspace, please wait...
2024-03-01T11:08:56.316211420Z Syncing SUPIR to workspace, please wait...

civic shell
solid axle
#

ashleyk is updating docker image so it might be worth checking

#

I would say in my case syncing to NS was like twice of normal pod

civic shell
#

Still 10GB remaining to push out of 40GB (30GB already pushed).

civic shell
#

I avoid netwotk storage at all costs, even though its useful, I just can't cope with the slow disk speed.

solid axle
#

same

#

I usually store most stuff on my S3 bucket for private models

uneven sparrow
#

Thanks for your great support and work, gentlemen! Really appreciated!

civic shell
#

@uneven sparrow template has been updated to 1.3.1, if you don't mind testing again.

uneven sparrow
#

yessir, on it, running on A6000 with all default, no network storage...

uneven sparrow
#

ok, pod up and running, but: tail -f /workspace/logs/supir.log
python3: can't open file '//gradio_demo.py': [Errno 2] No such file or directory

#

2024-03-01T12:12:50.150042973Z Running pre-start script...
2024-03-01T12:12:50.152047563Z Template version: 1.3.1
2024-03-01T12:12:50.155374950Z Syncing venv to workspace, please wait...
2024-03-01T12:13:01.094569874Z Syncing SUPIR to workspace, please wait...
2024-03-01T12:13:27.045364520Z Syncing models to workspace, please wait...
2024-03-01T12:13:48.262263571Z Fixing venv...
2024-03-01T12:13:48.265274507Z Fixing venv. Old Path: /venv New Path: /workspace/venv
2024-03-01T12:13:48.328065553Z Starting SUPIR
2024-03-01T12:13:48.328744901Z SUPIR started
2024-03-01T12:13:48.328755796Z Log file: /workspace/logs/supir.log
2024-03-01T12:13:48.328793161Z All services have been started
2024-03-01T12:13:48.393349869Z * Restarting OpenBSD Secure Shell server sshd
2024-03-01T12:13:48.422192652Z ...done.
2024-03-01T12:13:48.427568280Z FileZilla SFTP configuration file created at: /workspace/filezilla_sftp_config.xml
2024-03-01T12:13:48.427589512Z Exporting environment variables...
2024-03-01T12:13:48.431467739Z Container is READY!

#

port not ready :/

civic shell
#

Check the log, it might still be loading

#

Mine is still broken on network storage

#

Oh crap I see the problem

#

@uneven sparrow can you do this to start it:

source /workspace/venv/bin/activate
cd /workspace/SUPIR
export HF_HOME="/workspace"
nohup python3 gradio_demo.py \
            --ip 0.0.0.0 \
            --port 3001 \
            --use_image_slider \
            --loading_half_params \
            --use_tile_vae \
            --load_8bit_llava > /workspace/logs/supir.log 2>&1 &
#

Then watch the log:

tail -f /workspace/logs/supir.log
uneven sparrow
#

ok, it's alive - waiting for completion 😉

civic shell
#

I messed up the docker start script, I pushed 1.3.2 image to fix it, but you can use the work-around above to start it for now, sorry about that.

uneven sparrow
#

FileNotFoundError: No such file or directory:
"models/Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors"

civic shell
#

Let me give you the command to fix that

#

Edit /workspace/SUPIR/options/SUPIR_v0.yaml

#

Change SDXL_CKPT line like this:

SDXL_CKPT: models/sd_xl_base_1.0_0.9vae.safetensors
#

then:

fuser -k 3001/tcp
source /workspace/venv/bin/activate
cd /workspace/SUPIR
export HF_HOME="/workspace"
nohup python3 gradio_demo.py \
            --ip 0.0.0.0 \
            --port 3001 \
            --use_image_slider \
            --loading_half_params \
            --use_tile_vae \
            --load_8bit_llava > /workspace/logs/supir.log 2>&1 &
#

I made the template private for now while I fix this, thanks for helping me test it.

uneven sparrow
#

ok, running gradio_demo.py and watching the log...

#

alright, it's up and running! move on to testing. THANK YOU!!!

civic shell
civic shell
#

@uneven sparrow how did it go?

#

I've pushed a new docker image to fix the issues and made the template public again

civic shell
#

Updated to version 1.3.5.

  • Changed SDXL model back to SDXL base model because the Juggernaut model was not good with eyes.
  • Removed password for Jupyter.
  • Added environment variable VENV_PATH to enable setting a custom path for the application venv, with a default of /workspace/venvs/SUPIR.
  • Supports sharing a network volume with other applications.
civic shell
#

Updated to version 1.4.0

  • Bumped to latest version of SUPIR.
uneven sparrow
civic shell
#

Updated to version 1.5.0

  • Bumped to latest version of SUPIR.
  • Bumped runpodctl to version v1.14.2.