▶｜stable-video-diffusion

[fallen plinth]

We got SVD working on imagineapp.co now running on our cluster of 4090s

[silent hinge]

so when can joe average use this video stuffs?

[open heron]

Define joe average

[silent hinge]

me haha

[open heron]

I own a 3080 10gb gpu and it's pretty cool. I think it can run with 8gb tho

[silent hinge]

oh how do you use it? i thought it was like invite only or something

is this just like something to dl on automatics?

[open heron]

Works on comfyui. You just need to download one file and get the workflow

[silent hinge]

interesting. thanks. ill have to look into comfy. quite frankly i thought it was horrible design

first time i tried it and got rid of it

im on a 3080 too so i guess i can run it

[open heron]

It's hard when you need to create a workflow. This one is quite a simple one. You just need to load your picture and hit play

[silent hinge]

interesting

how long can the videos be? 5 second or so?

[open heron]

It's looks like that

[open heron]

A bit less

[silent hinge]

how long until we can prompt a movie haha

5 years? 🤔

[open heron]

A simple short movie? Yes.

[tender nova]

yup, generating the 14frame videos on my 8gb 3070ti in under 2 and a half minutes at 1024x576

[silent hinge]

what format are the files?

[open heron]

New version, look those lights/reflex moving. This thing is really cool

[trim juniper]

looks clean

Anyone getting this silly error "IndexError: list index out of range"

[open heron]

Are you loading the right img2vid model?

[trim juniper]

yes

[glass marten]

SVD + Topaz 3X Slow Motion

[dull thunder]

What you're doing is illegal

[boreal valve]

👀

[polar cobalt]

[boreal valve]

im guessing its because svd is strictly for research and not commercial apps rn?

[dull thunder]

yes

his website is charging for generation using the model

[quick finch]

I'm still having issues with eyes.

[trim juniper]

How could they tell they used the svd

[boreal valve]

read what he typed and also what his bio states lol

[dull thunder]

well first of all he admitted it here in his message, but second, SVD is easily identifiable.

[robust scarab]

what gpu?

how long do these renders take?

[teal harbor]

4090

[teal harbor]

I think like 40-50 seconds, not sure

[robust scarab]

im a bit outta the loop whats the min req on svd?

vram wise

[teal harbor]

I think I've seen people generate stuff with 10 GB VRAM

[robust scarab]

thanks ill look into it

[teal harbor]

#▶｜stable-video-diffusion message

[glass sierra]

I'm going OOM with 16GB 3080 and decoding_t=1, what am I doing wrong?

[open heron]

What workflow are you using?

[dull thunder]

Do not post videos of Hitler on here.

[open heron]

Sorry

[glass sierra]

Just trying to get the default stuff working here:
https://github.com/Stability-AI/generative-models/blob/main/scripts/sampling/simple_video_sample.py

Went OOM on an L40 w/ 48Gb of ram doing the same thing... I'm definitely doing something wrong.

clone, make a venv, pip install, python -m scripts.sampling.simple_video_sample

[open heron]

We are doing mostly this https://comfyanonymous.github.io/ComfyUI_examples/video/ i have a 3080 10gb and works well

[glass sierra]

and download the models, before all that

I never got into comfy, UIs are cancer

I'll look at it though, since it works!

[quick finch]

I'm done for today, it was fun.

[teal harbor]

I was running it basically like this before it was on ComfyUI
https://youtu.be/HMW9hVoQa0M?si=rHYYPW_N_DhPbJ7L

lowvram_mode
https://youtu.be/HMW9hVoQa0M?si=rJ4ZbWkVfjKYEXCx&t=152

[open heron]

It's a really simple UI for this job, maybe it will give hints about what is wrong/different. The original model needs a lot of VRAM

[grim tangle]

couple of low motion bucket grids

and when you go too high for the seed...

[glass sierra]

I actually just got it working on a 48Gb L40, decoding_t=4 seems to take up ~20Gb so I don't get why it's going OOM locally w/ 16Gb... 😦

[teal harbor]

activate the lowvram_mode

[glass sierra]

i'm trying not to use streamlit

[grim tangle]

also fish

[grim tangle]

if you're running the example script, it uses full precision and doesn't offload the model to cpu in between sampling and decoding... it's very badly optimized, comfy's implementation is the only proper one and uses just 8GB for 25 frames

[glass sierra]

ah, I'll just fix it then... I figured it was something like that...
I also figured they'd have tried to make the code runnable on 16Gb in their sample

I guess there's enough momentum they can just release whatever and everyone else will fix it 🙂

[grim tangle]

bit hasty release in that sense sure, but it's just research release

however the implementation in comfyUI is already excellent now

[glass sierra]

cool, I'll look at their code then, thanks!

[icy valley]

just tried it for the first time, really cool!

[sand plinth]

@grim tangle You have already generated many grids, what do you think are the best default parameters for SVD?

[grim tangle]

too early to say, but from samplers euler with karras seemed consistantly good, 12 steps seems nice spot for speed/quality with FreeU_V2

can always increase steps if you get something you like

motion bucket/aug seems to depend on the input/seed, motion amount is consistently controllable with those in most cases

so many things to take into account though, much to learn still!

[icy valley]

fax

[bold wave]

I've been playing with this all day today.

• rife with a 4x multiplier helps, otherwise, it looks pretty jittery
• on a 12 GB card, I can do seemingly as many frames as I want.
• CFG 6.0 seems to yield more 'active' results. I can't help but feel that results vary though haha.
• bucket set to 50 has the best results across the board that I have seen so far
• I moved denoise to .75 instead of 1.0 and it seems to be helping. could be voodoo magic though.

[icy valley]

This is all I seem to be able to do so far

[trim juniper]

will try this now

[icy valley]

I have yet to try img2vid tho

[hazy gorge]

Which graphics card would be better to buy for SDV a gtx 1080ti 11gb or Rtx 3070 8gb?

[bold wave]

here's my setup so far. I have had a harder time managing the SVT_XT for some reason, so I just do SVT. I use facial restore as well, sometimes the facial changes look a little janky, so that normalizes it.

[trim juniper]

I have a workflow Kexo sent

I will add the face restorer to it didnt even think of it

ty

Way better

[harsh storm]

11GB one. More VRAM, more space for AI stuffs. You can also look for some old 2000 cards with 16GB VRAM

[severe moon]

Same input as before but I upped frames to 48 and changed denoise in KSampler to .66 - behold wiggly jello planet

[severe moon]

You want as much VRAM as you can afford to throw at it, it was trained for 40GB server hardware.....

[sterile mesa]

You working with comfy, right?

[bold wave]

I am!

running in cli may or may not have better results though

[sterile mesa]

Cool! Just now seeing your workflow you posted above too

[severe moon]

I got it running in CLI originally but have had best results with ComfyUI so far on the latest portable build. It has a good set of Torch/CUDA built in that you may not already have setup if you run from CLI. So I recommend anyone trying this to start with ComfyUI....

[bold wave]

its certainly easier and more customizable in comfy as well. and - you can throw in interpolation, which looks ten times better.

[severe moon]

yeah it's pretty janky with low fps if you want to render in reasonable time

[sullen harbor]

1000 series nvidia is slow so you should avoid it even if it has more vram

3000 series is going to be much faster

[trim juniper]

so far generations are taking 45 seconds to 1 minute

with 25 frames

[bold wave]

thats amazing. how many samples are you using? mine take a couple minutes at 25 samples

[trim juniper]

samples as in steps?

30

[icy valley]

Pretty insane how easily it can do this with txt2vid tbh

[severe moon]

try throwing some people portraits in it 😉

[icy valley]

Eh I've already seen 25 people doing that

[bold wave]

trying photographs now. cant share the results though haha.

next im going to try scenery shots. should have good results.

[icy valley]

the default values in this workflow made a scenery shot

no idea why this pans right but when I changed the content it starts to zoom in

[trim juniper]

beautiful

[icy valley]

Okay here we go

Starting to get more interesting

It's like it masks and everything for you somehow

[severe moon]

i think it depends on the colors of the input image, stuff with colors that imply depth tend to get more of that layered look and zooming motion from my experiments

[silent hinge]

what are prompts like for this stuffs?

and so you have to use it like image 2 image??

[icy valley]

I'm making these txt2vid

[trim juniper]

in comfy?

[icy valley]

Yup

[quick finch]

yeah, I will try to animate some stuff I created with the Cnet Monster Module to compare with animediff

[icy valley]

I'm not even using control nets or anything

It's able to do this pure txt2img generation with just my LoRA, it's actually pretty sweet

[quick finch]

great, I don't have a Lora for that, so if I need a logo, or Text on an Image I use the monster module https://streamable.com/4tv4sq

[icy valley]

Yeh qrmonster is great, but that's for more placing text on the canvas and then generating over it. With my model, you generate the entire image from scratch without anything else, no control nets, img2img, text from fonts, etc https://civitai.com/models/176555/harrlogos-xl-finally-custom-text-generation-in-sd

[quick finch]

got it, thank you for sharing, I will try tomorrow.

[icy valley]

Awesome! I'd love to see what you create.

OlivioTutorials actually just did a video on it tonight, with a pretty thorough walkthrough for using it

[quick finch]

I just sent u a PM, I just wonder what we can create with these new models with prompt schedulling and other things in the future.

[icy valley]

If you take a peek at the gallery on the bottom of the model page, you will see that people have come up with some insane stuff already

[bold wave]

the best is this one.

[severe moon]

Results may vary

[latent pecan]

oh wow, could script an entire story with this

[icy valley]

LOL this is exactly my experience so far as well

[icy valley]

I'll give it to you, that one is top shelf, but there's so many other good ones

[severe moon]

trying out @icy valley lora

[icy valley]

Rainbow + pixel art =

Hmm I wonder if the graffiti style would have it parallax from the wall 🤔

[icy valley]

Damn this circular motion is dope!!

[latent pecan]

was meant to have the text 404, trying again 🙂

[icy valley]

Here use this https://civitai.com/models/204953/the-404ra-add-on-for-harrlogos

[icy valley]

I dunno what causes that waviness tbh but when I got rid of it, all I get now is zooming straight in and out

[severe moon]

it's just sort of random, the training vids only had a few types of movement so changing the seed for the sampler after the SVD node has the most effect on type of camera movement

[icy valley]

Are you guys doing txt2vid?

[latent pecan]

testing out 404 now

[icy valley]

Although I guess it's the same, you're just making the image to send it in the same workflow

[latent pecan]

well txt to image and image to SV

[severe moon]

sort of....

[icy valley]

that looks awesome 😂

YEP AWESOME

[icy valley]

Welcome to the zoomgang

[severe moon]

the seed before this for the one above was just a straight zoom, same img input....

[icy valley]

although I guess I've defected to GlitchGang

[icy valley]

nailed it

[latent pecan]

you made any loading animation loras?

lora_panrightv2 mixed with your text lora might do it, but probably a bit more technical than that 😄

[icy valley]

Just Harrlogos and the 404ra so far. But Harrlogos is unbelieavably versatile.

Oh are you guys using AD motion loras with it?

[latent pecan]

no, i tried to hook them in but couldnt work out if that works yet, probably doesnt

[icy valley]

Cuz like if you see the first animation I did

the motion is completely different

And ever since adding in my LoRA I havent been able to achieve anything like this again

[latent pecan]

tried to make the text 555

[trim juniper]

how are you adding loras?

[icy valley]

hahahahaha yeah the 404ra is trained to do only that, although a buddy of my made it to 1337 before

[icy valley]

Same as any other Comfy workflow

[noble wolf]

can this run with 8gb?

[icy valley]

Supposedly

[latent pecan]

im running it on 8gb

[icy valley]

then yes

[noble wolf]

Ayoooo lets go

[trim juniper]

so like a normal lora not a motion?

[noble wolf]

I need to learn comfy, yall know a good guide?

[icy valley]

Yes, I'm using my Harrlogos LoRA to do the text

[severe moon]

i have made so many that don't actually say CATGIRLS that I gave up, have a CATRIS instead.... lol

[latent pecan]

maybe do individual letters 😄

[noble wolf]

Is there something like a controlnet for video yet?

[silent hinge]

lol it just came out dude

[sand plinth]

Hmm, SVD works with CGI, not bad...

[icy valley]

You don't have a preview for the img?

[noble wolf]

I'm not expecting there to be anything like contrlnet, I'm just saying it doesn't hurt to check with how fast this shit is going

[icy valley]

I can tell in the first 2 seconds if the spelling missed so I can try again

[severe moon]

oh yeah I just turn off the SVD bit until I get a good image, i'm just not having luck getting the logo part.... :/

[icy valley]

Hm I dunno, rarely takes me more than a couple tries, maybe it's the specific text, or your resolution

[icy valley]

LOL

Science Fiton

My favorite genre

[latent pecan]

yea, futuristic pants you fit on 😄

[icy valley]

Dang bro how you getting this circular motion

Looks awesome

[latent pecan]

wish i had control over it

[icy valley]

Duuuuuude this is CLEAN!!!

[latent pecan]

seed 425324399642679

try that seed out 😄

[icy valley]

Ahah that little ear wiggle

[noble wolf]

yo is there a way to increase font size in comfy ui?

[severe moon]

like, the default size? i jsut zoom in and out, it kinda sucks

[noble wolf]

sadge

I dont wanna zoom in when I'm messing with my prompt, I mean i don't need to but it's just nice to look at

[noble wolf]

Yooooo that's dope

[icy valley]

yooooooooooooooooooooooooooooooooo

[severe moon]

that magic seed man: 425324399642679

[icy valley]

Thats probably the coolest shit I've seen yet

[severe moon]

i like how it just randomly flipped the G for no reason

[icy valley]

That's the best part

[icy valley]

Damn! How did you get the letters to Animate

[severe moon]

it jsut does it

[severe moon]

here's my magic numbers.....

they're really nothing special tho, just seems random how the motion will turn out

[icy valley]

It looks awesome! Thank you for sharing I'm gonna try it when I'm back at my pc

[severe moon]

I think it has more to do with 'chrome' being in the prompt and producing letters with that shiny effect. I bet the training sets had lots of cars and shiny vehicles with reflected motions on them

[icy valley]

That's actually sick because chrome is an activation word for my model I trained to do just that

[noble wolf]

magic man

[severe moon]

just like it always seems to like animating anything cloud or smoke like in an image:

[icy valley]

I bet space stuff will work pretty well

[severe moon]

smokey monster dudes

[icy valley]

That's awesome, what model are you using?

[severe moon]

Midjourney, that's just straight img2vid

[silent hinge]

Is there a way to download stable video diffusion and run it locally?

[robust scarab]

yes that what people are doing here you need to use comfyui though

[robust scarab]

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

[silent hinge]

Thanks. Will it work on my 6gb vram

[severe moon]

you don't need to use comfyui but it's probably the easiest way

[robust scarab]

theres a auto version?

[robust scarab]

not sure but it works on my 8gb card

[silent hinge]

Alr thx

[severe moon]

hmmm. i've seen people getting it to work with 8GB VRAM, no one has mentioned 6.... try it for science and report back to us all!

[silent hinge]

Ok. Btw are there any tutorials for stable video diffusion?

[severe moon]

most of the ones I've seen are jsut for getting it to even run

[silent hinge]

Oooo nice

[silent hinge]

Wdym

[uncut axle]

A classic banger brought back

[severe moon]

like how to install ComfyUI or the streamlit version and set up a workflow that has SVD in it.... there's not a lot about how to get good results yet

[silent hinge]

I got ComfyUI today I just want to be able to get SVD running lol

[severe moon]

there might be some for running a Google Collab version but i haven;t checked any of those out

[latent pecan]

comfyui standalone _=-> comfyui manager -> grab someones image workflow with SV -> install missing custom nodes -> restart comfyui 😄

[noble wolf]

wait where do I download img2vid?

[silent hinge]

Thx.

[robust scarab]

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

theres a tutorial here #▶｜stable-video-diffusion message

[sullen harbor]

For the examples on here you only need comfyui itself, no need for any custom nodes: https://comfyanonymous.github.io/ComfyUI_examples/video/

[severe moon]

yeah it's all built into the latest ComfyUI, thanks @sullen harbor !

[robust scarab]

is it possible to add text conditioning?

[silent hinge]

Can you use custom 1.5 models for stable video diffusion?

[severe moon]

so it's img2vid workflow..... how you get the img is up to you

[sullen harbor]

it's possible to pass these models text conditioning but it might not work very well in practice

[severe moon]

you can easily attach your favorite 1.5 workflow to make the initial image....

[silent hinge]

Alr thx.

[sullen harbor]

you can use the conditioning concat node with "conditioning_from" as the output of a clip text encode connected to the SD2.x CLIP model

[robust scarab]

will try this

[latent pecan]

@next acorn this is their workflow they shared on deforum discord

[robust scarab]

what does augmentation level do?

whats the purpose of the image decoder here if the video generation seemly works fine without it https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/tree/main

[icy valley]

That's cheating!

[river mauve]

SDXL > SVD > TOPAZ

[icy valley]

Damn that quality is wild

[severe moon]

no, using MJ to Photoshop to make green screens to input into txt2vid SVD and then using keying in Premiere to combine layers is cheating (my quick and dirty proof of concept from earlier this afternoon):

[icy valley]

That's goddang double cheating

[trim juniper]

how much upscale are you using

[severe moon]

i made the mistake of fullscreening to see what you were talking about and now i'm going to have nightmares.....

[trim juniper]

lol

[noble wolf]

anyone get this error for img2vid?

[robust scarab]

did you download the svd model?

[river mauve]

1024x576 > 1920x1080

[robust scarab]

can you show a screen shot of your workflow

[icy valley]

Disable FreeU

[tender nova]

So is there a way in comfyui to make a node chain that goes from txt2img straight into SVD img2vid in one generation?

[robust scarab]

#▶｜stable-video-diffusion message

[trim juniper]

Where

Nvm found it ty

Ayway to get the js

[sharp kestrel]

seems like a good time to plug my engulfed by Lora 😛

[noble wolf]

What is freeu?

[severe moon]

most of my shared Peace vids above do that, the vids should have the metadata in them unless Discord stripped em out: #▶｜stable-video-diffusion message

[noble wolf]

and where do i disable it

[devout flame]

hi, regaring svd and full-body videos, i wonder if theres a way to fix the awful faces in them?

is it possible atm?

[sullen harbor]

https://comfyanonymous.github.io/ComfyUI_examples/video/
second example

[trim juniper]

add a face detailer

[devout flame]

does that work in comfyui svds?

[trim juniper]

ive been using it

[devout flame]

oh cool, which one do you use?

[trim juniper]

sent you a screenshot

[icy valley]

I have a simple txt2img workflow if anyone needs

[devout flame]

ok i have those but im getting errors importing them

saying i dont have a module called "segment anything

[icy valley]

Did you disable FreeU

[devout flame]

how do i do that?

[noble wolf]

yes please xD

how do you do that?

I was asking like 5 mins ago

[devout flame]

yeah, couldnt find how to on google either

[noble wolf]

I found like one reddit post but don't think its relevant to this

but maybe idk

[devout flame]

yeah im so confused

[noble wolf]

are you getting the same error I posted earlier?

[devout flame]

no, my error says "module for custom nodes: No module named 'segment_anything'"

[noble wolf]

hmm

[fallen plinth]

we are not live in production for charging using the model yet

[trim juniper]

im getting cursed images when doing t2img to img2vid

[severe moon]

which txt2img models you using?

[devout flame]

can you please elaborate how to disable freeu? nothing is coming up on google or reddit

[trim juniper]

just delete the node

[devout flame]

wdym, i get the error on startup

[trim juniper]

I combined what i had for img2vid with also txt2img

maybe the lora is the thing messing things up

[devout flame]

it happens when i add the custom nodes to the folder, i start it up and it gives me an error saying i dont have segmentanything

[noble wolf]

I just loaded up the default workflow

and its not working

[severe moon]

do the images look high quality before they hit the img2vid portion?

[trim juniper]

not much

might have to add an upscaler

im just using common sense idk what im doing

lol

[severe moon]

sdxl or sd1.5 or.....?

[trim juniper]

1.5

[severe moon]

yeah maybe a double ksampler upscale type workflow to add details

[trim juniper]

let me check

[latent pecan]

this is iskariots workflow , rename this to .json, you may want to change the default resolution to 1024x576 to run on 8gb card

the lora node thing required me to install custom nodes, thou you can disable it, its not needed

put this in your SD models folder

[severe moon]

"goddang 54th dimensional vorpal blades...."

[latent pecan]

damn this didnt animate at all :\

[noble wolf]

master epic E

[arctic reef]

Could you please share your comfyui workflow? These results are fantastic!

[latent pecan]

looks like a great prompt here!

[sharp kestrel]

based on sdxl lora https://civitai.com/models/176367/copperuxl-heavy-metals (i did it)

[sharp kestrel]

can you share topaz's settings?

[devout flame]

ok my next issue, anyone got a workflow json for facedetailer and svd?

[latent pecan]

with comfyui search for faced and install , restart comfy, add face detailer loader and face detailer node just before the final step that puts it all together as a video

if i had a clean workflow id export but i have a whole bunch of purple nodes to ignore for now

[robust scarab]

what does the fps setting do? does it increase the context?

[latent pecan]

installing facedetailer breaks my opencv, i made a custom .bat file for any nodes that break opencv (a lot of nodes seem to! ) .\python_embeded\python -s -m pip uninstall -y opencv-python opencv-contrib-python opencv-python-headless
.\python_embeded\python -s -m pip install opencv-python==4.7.0.72

[trim juniper]

idk txt2img to img2vid not worth that much

[sharp kestrel]

beautiful 🙂
would you upload to civit?

[latent pecan]

you can if you like, i dunno how to 😄

[sharp kestrel]

just like you download a lora, there's a small post button on them... it helps creator stay creative 🙂

the add post button

[latent pecan]

ok done, thou it posted as image not gif

[latent pecan]

(with face restore)

is it worse :D?

face restore probably better for mid shots / sd1.5 models

[latent pecan]

try this, rename to .json and load into comfyui

[noble wolf]

Yooo I finally got it working lets gooo

Now how to download them as mp4...

[latent pecan]

it puts them all by default n comfyui/output

you just change that last node from gif/mp4

[noble wolf]

all I see are those

[autumn spindle]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite

[noble wolf]

Ty brother ❤️

[sterile mesa]

fuck i've been sleeping on XL and comfy. Just installed that shit and I'm jawdropping on the ease of customization

[noble wolf]

yeah it's awesome, especially with comfy, just downloaded it a few hours ago

so poppin

I'm trying to figure out where to place those video format custom nodes xD

[autumn spindle]

Do you have ComfyUI-Manager installed?

[noble wolf]

uh nah?

idk

[autumn spindle]

https://github.com/ltdrdata/ComfyUI-Manager

[noble wolf]

I just installed it and a few other things

[autumn spindle]

It'll help you get the custom nodes

[noble wolf]

ty ty

[autumn spindle]

Especially the Install Missing Custom Nodes command

[sharp kestrel]

and models

[noble wolf]

gawdam it's a holy grail

I'm evolving

[sharp kestrel]

lol

[noble wolf]

and it's so efficient I can do a bunch of shit while generating videos on an 8gb card

lets feckin go

[latent pecan]

how did you get 12s!

[sharp kestrel]

i asked for 100 frames

[latent pecan]

oh 😄

wonder if that works on my 8gb card

[sharp kestrel]

12 sec of pure bad stuff

[latent pecan]

multi verse fairy nitemare 😄

[sharp kestrel]

earlier i was playing with some of the settings

not everything turns out good right away

[noble wolf]

nightmares are my fav yo

deep frying with stable diffusion, truly revolutionary

[sterile mesa]

it is 1am and im just installing nodes and workflows. need to sleep

[noble wolf]

literally same

got my grubby hands on the manager and am downloading everything

Although I wish there was something like tabs you could save as a system instead of workflows, but I guess I'll get used to it

[latent pecan]

i find it hard to get a good naming convention 😄 so i just put every main node in the file description 😄

its pretty awful for qol

[silent hinge]

yo how are u guys generating with this sd video

do u guys have 40 gigs of vram bro

[noble wolf]

comfy ui is super efficient for me at least, using 8gb vram

[silent hinge]

damn so if I have comfy ui I can get svd

[noble wolf]

Yesss

come join us brother

its so ez

https://www.youtube.com/watch?v=AbB33AxrcZo

[silent hinge]

@noble wolf is this the tutorial

nvm it isnt

but is there a tutorial to how to get svd

[noble wolf]

I think someone said it was the unofficial official guide

oh for svd its ez

https://github.com/ltdrdata/ComfyUI-Manager

get this and use the manager to auto install svd on comfyui

once you do that you can grab the workflow to follow off this https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

[grim tangle]

SVD comes built in comfy now, the custom nodes should not be used

[silent hinge]

anyways I only have 6gb of vram but soon im gonna buy new pc

[grim tangle]

comparison grids of fps_id values

[latent pecan]

this is super clean!

[sharp kestrel]

so is anyone keeping track on the seeds and directions or is that a myth?

[arctic reef]

saving raw video generation, and a second one after postprocessing (face swap, upscale)

mostly gets the job done

[latent pecan]

test of 50 frames, tried 100 but OOM

[latent pecan]

was hopium 😄 but yea suspect its the shape of the objects generated *my hypothesis^

[narrow canyon]

Is there a node that will save the generated frames out as individual images?

I just found ComfyUI-VideoHelperSuite, been hunting down that VHS_VideoCombine node I've seen in some workflows

[sterile mesa]

I believe some sort of ffmpeg node would do the trick

[sterile mesa]

Care to share link?

[narrow canyon]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite

Just trying to figure out how to install it

[sterile mesa]

Thanks a ton. I've got a long day of comfy ui setup tmw. Was testing it tonight and very impressed

[dull thunder]

What do you mean, I used your website to generate a video which deducted 20 credits

[robust scarab]

what does the fps setting do if the final frame count is always the set amount?

[clever pendant]

https://comfyanonymous.github.io/ComfyUI_examples/video/
at the bottom is a brief explanation of each

[robust scarab]

does fps increase vram usage?

[grim tangle]

just "Save Image" does that

[narrow canyon]

Awesome, thank you!

It's my first day using Comfy

[grim tangle]

as for installing custom nodes, I recommend the manager: https://github.com/ltdrdata/ComfyUI-Manager

[narrow canyon]

Just found that and got it working! Just figuring out how to apply it to the cog container I'm modifying

[sharp kestrel]

which one of these 2 guys use?

[robust scarab]

model

[grim tangle]

who named those, what

just to see if high values do anything

[narrow canyon]

Is the decoder model needed for the example workflow?

[grim tangle]

no

it's not needed for anything, it's just named awkwardly

[narrow canyon]

Oh good, I can save 9gb then

[grim tangle]

can save even more: https://huggingface.co/becausecurious/stable-video-diffusion-img2vid-fp16/tree/main

[sterile mesa]

Heck yes. Idk how you people keep up with this stuff and find it, but thank you.

[narrow canyon]

Oh blimey, that was quick

[sand plinth]

What achieves the size compression of such models?

[grim tangle]

using half (fp16) precision over full (fp32), for inference it doesn't seem to have any negative effects (so far)

I'm not an expert on that though

[woven fractal]

I'm using comfy with normal stable diffusion

Howdo I get stable video diffusion?

[grim tangle]

update comfy

[woven fractal]

it doen't seem to have any interface like A1111

[woven fractal]

ok

thanks

[grim tangle]

and check the example there: https://comfyanonymous.github.io/ComfyUI_examples/video/

[woven fractal]

So I updated it

and I put th ejosn in

I keep on getting an error

Error occurred when executing SVD_img2vid_Conditioning:

'NoneType' object has no attribute 'encode_image'

that's the error

[grim tangle]

did you change the image?

[sand plinth]

The Lumiere brothers give kudos))

[sharp kestrel]

cfg scaled 2-3 means that you start with 2 and up to 3?

[grim tangle]

It's the video linear node that scales it from 2 to 3 yeah, along the animation, so it's not constant

[sharp kestrel]

so you start at 2 here?

ending in 3 here

?

is 1 to 2 not enough?

[grim tangle]

Depends on the video, sometimes it benefits from more, but it can also burn it

Still testing different values, these aren't meant to be optimal or anything

[icy valley]

they seem to be on a video case basis as well

[grim tangle]

Slow video burns if the end cfg is high while faster can benefit from increased detail against the blurriness that occurs

[wicked wren]

Workflow? 🙏

[grim tangle]

I think this happens if the SVD model isn't correct

[wicked wren]

Awesome, workflow?

[grim tangle]

not too user friendly, just made it for myself to test stuff

[wicked wren]

Cool I appreciate it 🙏

So cfg is something different in video (comfy UI) from cfg when doing stills in auto1111? Correct?

[grim tangle]

not really different, just different scale to use with SVD, I find that it should be between 1-4, usually around 2

[wicked wren]

Alright, thanks!
Can I use any model like with stills or does it have to be video models?

[grim tangle]

only the video model for the video genertion part, for init image it can be anything of course

[arctic reef]

In my previous post, there's json

[zenith depot]

https://cdn.discordapp.com/attachments/1021146407190667264/1177939762666025061/xnviewmp_2023-11-25_12-28-31-2x-RIFE-RIFE4.0-60fps.mp4?ex=657454e4&is=6561dfe4&hm=9c066f581aac3cdb2a3e47adbd107c1e91473e07ead3e03578148d9cf03ef2e9&

[hazy gorge]

So far, which one do you think is better, Stable Video diffusion or Pikalabs?

[sonic stirrup]

is it p[ossible to run svd on an amd gpu?????

[wicked wren]

thanks

[somber oar]

The subtle movement of hair and clothing

[clever pendant]

these are a little rough. fun new toys to play with though

[frigid grail]

Ah, so ComfyUI SVD not working for Max silicon - right? (I get the conv3d is not supported on MPS error)

[arctic reef]

Here's updated workflow, with sound notifications.

[solemn turtle]

Hi, im getting this error while running on the Mac. 'NoneType' object has no attribute 'tokenize'

File "/Users/../ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/../ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/*../ComfyUI/nodes.py", line 55, in encode
tokens = clip.tokenize(text)
^^^^^^^^^^^^^

[granite tartan]

what's the best way to upscale this and improve the quality of the faces?

[haughty garden]

It's a Stable Diffusion problem. The further you get from the camera the more the quality sucks.

[arctic reef]

Take a look at my workflow.

[granite tartan]

thank you 🙏

[flint orchid]

very nice! thanks

[arctic reef]

before and after postprocess (my workflow outputs both versions)

[solemn turtle]

Yo did anyone run stable video with comfy on a mac?

[half hound]

Don't think it's supported yet. Cocktail peanut on twitter mentioned there should be support for it next week.

[solemn turtle]

Thanks

[half hound]

@grim tangle Do you know what nodes you can use to export each frame of the video?

[grim tangle]

"Save Image"

it will save every frame in the batch as .png

[half hound]

thanks!

[silent hinge]

so what do you need for this? comfy and a certain model?

[arctic reef]

more or less

[silent hinge]

how do you prompt? like do you prompt the camera movement?

[arctic reef]

you don't prompt for motion. "motion bucket id" controls amount of motion, but that's it

[silent hinge]

so just prompt like an image?

[arctic reef]

no, you put image and it is a starting point

[silent hinge]

interesting. start image and end image?

[arctic reef]

just the start image (i'm also using face restore image for face correction)

[silent hinge]

looks like algebra

more like confusingai haha

really, that couldnt be any less 'comfy'

[severe moon]

good morning

[frigid grail]

What are you all using for frame interpolation?

[arctic reef]

Not really. It is so flexible, that I can have face reconstructed, upscaled, added interpolation frames and written as 2 separate videos. All with a single click.
This is far from the most simple workflow.

[severe moon]

JJ has a complicated workflow, it can be as simple as this:

[silent hinge]

nice. im dl'd it now so i'll be asking some questions im sure haha

what model do i need?

[severe moon]

there's a SVD and SVDXT model, start with the SVD one: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

[silent hinge]

noice. cheers for the link

[half hound]

Is there a node that can key out green from the image?

[silent hinge]

do i need all the files?

[half hound]

can you layer videos on top of each other in comfyUI?

[severe moon]

If you install ComfyUI then also highly recommend to install comfyui manager, then you can just search for models directly inside comfy, there should be install videos earlier in this chat that cover getting SVD running in comfy....

[flat crystal]

How do you get this layout? This seems amazing

[silent hinge]

any ideas?

was trying to run_nvidia_gpu from comfy to extract

guess not haha

[severe moon]

in theory you could steal that type of workflow from the animtediff crowd using masks and stuff but I haven't seen anyone try that yet. I did try taking my source vids into Photoshop to isolate subjects against greenscreen backdrop and then comp it all in Premier using keying effect. Real quick and dirty proof of concept test I made yesterday:

[severe moon]

it's a .bat file, you just run it by double clicking

[silent hinge]

id id then i got that error i linked

[severe moon]

did you extract the entire .7z file into a new location before trying to run it?

[silent hinge]

maybe i screwed that up

[half hound]

I am looking at ComfyUI Impact Pack and it looks like they have a node for SamDetector which separates parts of the image.

Maybe that can be used

[teal harbor]

hey, this is the workflow

[severe moon]

sounds like you're in the right ballpark

[silent hinge]

oh there it finally worked, i did change the directory first time to windows idk

[half hound]

yeah I'll give it a go

[silent hinge]

ok so now i have comfyui up what is my next step

should i run the update thing first?

[half hound]

is it a fresh install? Than no. Load in the json workflow

[silent hinge]

how do i do that

yes fresh install

[half hound]

https://comfyanonymous.github.io/ComfyUI_examples/video/

[silent hinge]

and img2video model just goes in checkpoints right

[half hound]

download this jsons by right clicking and save link as

then open in comfyui

one of them

you can also take a look at my tut and follow along https://www.youtube.com/watch?v=hoIobzZmNiM

just skip to the end

[silent hinge]

where do i put it when i save it

[half hound]

wherever you want. I put mine in downloads

[silent hinge]

oh ok thanks. do i need all 3 bottom files here: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid/tree/main

and put them all in checkpoint?

[half hound]

just this one svd.safetensors

[silent hinge]

nice

[half hound]

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/tree/main

grab the xt one too

everyone uses that one

[arctic reef]

Thanks!

[silent hinge]

they go in checkpoints like reg models right?

[half hound]

yes

[severe moon]

yeah XT was trained on longer videos, once you get it all running and know your rig can handle SVD you'll probably use SVDXT the most....

[silent hinge]

ive got a 3080 with 12 gig hopefully i can do something and not melt it haha

[severe moon]

that should do fine

[arctic reef]

I'm using 3060 12GB and it is enough.

[teal harbor]

❤️ The face swapping I do with this https://github.com/C0untFloyd/roop-unleashed
It's MUCH faster than doing it with ReActor in ComfyUI
And finally I convert it to gif with this https://www.screentogif.com/
I can share the settings for that also if you're interested

[half hound]

yeah that would be great

[severe moon]

you switching to auto111 to use roop or....?

[silent hinge]

using the example, so you just load an image in and hit queue prompt? nothing more to it, not actual prompting or other guidance?

[half hound]

why do you need to convert to gif? Can't you just use the video combine node and then select gif in comfy

[severe moon]

that's it

[silent hinge]

so roll of the dice it looks good or get something good haha

[teal harbor]

nope I use this https://github.com/C0untFloyd/roop-unleashed
It has a lot of control including face recognition and it's very fast (pretty much realtime for me)

[silent hinge]

and i take it this json i dl'd is optimized no need to fiddle with steps, cfg etc?

[half hound]

well maybe not cfg or steps, but more of the frame data and motion id

[severe moon]

oh I see it's like a standalone thing

[silent hinge]

wheres motion id? and how many fps you using?

[teal harbor]

yes

[severe moon]

nice

[silent hinge]

from image to video already, even if just a few seconds

crazy times haha

[severe moon]

here are my current value settings:

[half hound]

these 2 settings

[silent hinge]

nice. appreciate the example dragon

[half hound]

video_frames max for SDXT is 25

[severe moon]

these models were both trained on 1024x576 pixel videos so you don;t want to stray too far from those dimensions

[half hound]

I got some decent outputs using 1024x1024, but I haven't done that much testing on it and the renders took a lot longer.

[severe moon]

yeah sqaure ouput with sq inpiut seemed to do well. i tried vertical tiktok sized videos and things got weird

when i did sq i did it around 768x768

[half hound]

how did it look at that size?

[severe moon]

you want to multiply the numbers and get close to 1.5M

[silent hinge]

what setting sets the length?

video_frames id guess?

and what is motion bucket for/do?

[teal harbor]

video_frames * fps = length in seconds
motion_bucket_id = amount of movement (higher number means more movement)

[severe moon]

they look fine

[half hound]

man I wish I tested out segment anything more. I am going to have to watch some tutorials on it

is there an SDXL segment anything model out?

[severe moon]

the parameters are covered on the official ComfyUI Video Examples page but here it is again for our discussion:
`Some explanations for the parameters:
video_frames: The number of video frames to generate.

motion_bucket_id: The higher the number the more motion will be in the video.

fps: The higher the fps the less choppy the video will be.

augmentation level: The amount of noise added to the init image, the higher it is the less the video will look like the init image. Increase it for more motion.`

[teal harbor]

I recommend searching this from:kijai on discord, they made a lot of comparisons of different settings so you can clearly see what they do

[severe moon]

in my workflow I output at 8fps from the SVD node and then use interpolation to smooth the framerate back to 24fps:

[bold wave]

I've been doing just 5 fps and adding Rife, is that one better?

[severe moon]

I believe the node is also using RIFE. Before I was making vids and then using Flowframe to do the interpolation but then someone asked how to do it on linux where Flowframes isn't available and it turned out someone had made a ComfyUI node to implement the same tech.

[harsh storm]

The one you used is a different model

[bold wave]

ahhh, that makes sense. Thanks!

[harsh storm]

The node for RIFE is RIFE VFi

[severe moon]

They're your nodes right? Whick ones do you think would work best?

[harsh storm]

Depends on the input. GMFSS for anime, RIFE for both 3D and anime and FILM for large motion (large change between frames)

[bold wave]

OK so - for more realistic shots, upping the resolution to 1024 really helps. however, it gets super janky with artistic images.

also using upscale moved a 24MB file into a 700MB file proooobably not viable lol.

[severe moon]

how much are you upscaling by?

[teal harbor]

are you upscaling it to 16K or what?

[half hound]

good resource if anyone is looking for nodes. https://ltdrdata.github.io/

[bold wave]

lol - I went from 512 and did the esrgan model. in comfy you cant specify - so what does it do, 4x?

[severe moon]

oh i see. i've seen other methods where you take the first ksampler output and then run it through a basic pixel multiplier and controlnet to increase dimensions and then another ksampler with lower denoise value to get more detail. that was for animatediff workflows. i'll probably try that on the SVD workflow today

[bold wave]

oh yeah, ive done that before too, maybe i should give that a shot. this one is where A1111 feels like it has superiority, you just kinda run it.

at that, I could probably apply a little bit of a prompt too.

[severe moon]

yeah

[sharp kestrel]

not sure why, but all my recent ones were super shakey or strange

[sharp kestrel]

using this model

[severe moon]

is this the compressed checkpoint?

[sharp kestrel]

yea

now running one with the non compressed

[severe moon]

are you changing the seed randomly or using same seed for all of them?

[sharp kestrel]

randomly

[severe moon]

some seeds are just shaky

[teal harbor]

those videos are too long, what are your settings?

[severe moon]

what's your motion bucket set at?

[thorny niche]

SVD is nice but soooo slow

[teal harbor]

try normal scheduler

[sharp kestrel]

MJ source?

[teal harbor]

80 frames is way too many, it was trained on 25 max

[sharp kestrel]

i got good 80 frames earlier

will try with normal euler

[teal harbor]

I just noticed, maybe your resolution is too low

[sinful vine]

@sharp kestrel originaly yes but some of that was run through SD afterwards and then into SDV

[sharp kestrel]

it's beautiful. wasnt ment as a jab.

just making sure i'm still on edge 😛

[sinful vine]

No i didnt take it that way 🙂

Thanks

[sinful vine]

ohhh what have i missed in here .... hahah

[sharp kestrel]

anyway, once again plugging my engulfed Lora... it's great for smoke filled scenes...
https://civitai.com/models/196290/engulfedby-xl-your-future-is-shrouded-in-smoke

[teal harbor]

I'm looking forward to the day when we get AI that will make the gears and what not on her head move

[sharp kestrel]

i was trying so hard to get some hears with animated diff last night.

[sharp kestrel]

you get the blinking from SVD?

[sand plinth]

Yeah, sometimes SVD does that

[sharp kestrel]

nice

[sharp kestrel]

did you try the same seed afterwards with different portrait?

[bold wave]

I feel like at 512x512 there's a bit of loss, though. Trying 768 and hoping for a happy medium haha

[sand plinth]

yes, I fixed seed on 425324399642679

[sharp kestrel]

and it works?

[sand plinth]

I hope so 😃

[sharp kestrel]

lol

[silent hinge]

lol now i know why nobody responds to my statements... i was in the openai dalle discord talking about this haha

but damn this is wild, like moving lips, hands, not just pan and scans on some images

[bold wave]

I did my son's school photo and his hair and fingers were moving, it was pretty cool.

[silent hinge]

yeah some of the motions surprise me, i love it. is there a default max length?

i wonder how far away we are from even 1 minute of video

[bold wave]

it was set for 24 frames but I have done 64 just fine.

[silent hinge]

i'll give 64 a shot then

I was trying to generate a video but I got "ComfyUI_windows_portable>pause
Press any key to continue . . ." and when i press a key it stops comfyui what did i do wrong

this happens running both cpu and gpu

[bold wave]

youll have to screenshot your workflow, unfortunately I am not an oracle.

[bold wave]

dang dude. that's amazing. its crazy how a few seconds of video can make it so much more immersing.

[severe moon]

you might have hit your VRAM limit, try lowering some settings maybe

[silent hinge]

like what i'm new to comfyui

[bold wave]

are you trying to just run comfy?

[severe moon]

yeah it's like instant After Effects 2.5 motion

[silent hinge]

yeah dragon that is awesome.

did you use dalle for the image?

[severe moon]

Midjourney

[silent hinge]

noice

what is motion bucket id?

[silent hinge]

Comfy works fine and generates images fine and all that. i have 6gb vram on a 1660 ti. I'll send image of my workflow in a moment

[bold wave]

Mind sharing your workflow? I just want to compare notes. I feel like I am close, but yours feels just awesome.

[silent hinge]

give us the secret sauce dragon haha

[severe moon]

ummm... be good at midjourney... lol

[silent hinge]

im bing/dalle until 6.0 haha

[bold wave]

768 is significantly better, it seems. here's a side by side. only thing is though, lower res seems to move around more.

so maybe 512 with upscale would be the trick.

[silent hinge]

where do you upscale?

[severe moon]

i mean actually though midjourney does all the heavy lifting, I didn't even spell steampunk right:

[silent hinge]

what about your comfy settings though is what i think we meant

[silent hinge]

33 for motion bucket just a random you chose or reccommended?

[severe moon]

for less photorealistic source images lower motion bucket tends to blow up less

[silent hinge]

Here's my workflow

this shiz is fun as hell though

[severe moon]

just having a good source image is half the battle

like i was experimenting with text overlays in photoshop so this without text seems fine

but adding text really changes what happens

[silent hinge]

nice

[bold wave]

I dont see anything that stands out. 14 FPS might be a little high - you can interpolate that. Also, I don't like WEBP at all, if you do video combine, you can save as MP4 or GIF. I do gif for now because it is the best for sharing online.

[silent hinge]

alr thx

[bold wave]

if you can screenshot the error - if you have one - it might help.

[silent hinge]

where do you change the file format?

[silent hinge]

alr

[bold wave]

use that one instead of the webp one

[silent hinge]

i must be blind i dont see it

but im new to comfy

[severe moon]

we're using a different node than from the video examples on comfyui offical page

[bold wave]

oh. do you have a setup for custom nodes?

load mine in... that should ahve the metadata in it, if discord doesnt delete it

[silent hinge]

im using a json i downloaded from somewhere, its been so much going on im not sure haha

[severe moon]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite

[silent hinge]

what do i do with the pic to get it load in?

[bold wave]

save it, then click load, and select my image

if the metadata is there, itll load. if not, i can share the json output instead

[silent hinge]

error

yeah im not sure it changed if you could share the json thatd be tight

im sure im just doing something wrong

when i press a key to continue it just terminates comfyui

and hmm, should i have a VAE or not needed?

[bold wave]

heres where im at right now, though ive just messed with the settings

[sharp kestrel]

no

[silent hinge]

awesome thanks raydestar

[bold wave]

lol the settings just created an abomination so youll have to tweak it - but - it'll get you mostly there.

[silent hinge]

looks like i need some files maybe

When loading the graph, the following node types were not found:
RIFE VFI
FaceRestoreCFWithModel
FaceRestoreModelLoader
VHS_VideoCombine

[severe moon]

do you have Comfy Manager installed?

[silent hinge]

not sure, i just dl'd comfy an hour or so ago so whatever is standard id have i guess

[severe moon]

https://github.com/ltdrdata/ComfyUI-Manager install this first thing it'll make your life 100x easier when working with ComfyUI stuff

[bold wave]

click install missing custom nodes

once you get the manager

[severe moon]

after Manager is installed when you load someone else's workflow it'll help you find all the Missing Custom Nodes and models

[bold wave]

and run updates. lol everything updates all the time, it's a result of being on bleeding edge tech

[silent hinge]

BTW here are my workflow files (they both do the random pause and then when I continue crash. The image I sent is of the txt-img-vid)

[bold wave]

you try updating too. it looks like it's looking for a value that doesnt exist, maybe running updates would fill that in.

[silent hinge]

after updating should i restart comfyui

[severe moon]

when did you first install ComfyUI?

[silent hinge]

yesterday evening

thanks for all this help btw everyone

oh yea thx too

woot and the manager appeared

[bold wave]

it sounds like you're missing some kind of library or something.

[severe moon]

you running the nvidia.bat to start?

[silent hinge]

yes

cpu is unreasonably slow

hold on it might bee working

80 seconds/it lol

[severe moon]

I had problems with my older ComfyUI running the new update after they added SVD so I grabbed a newer build:

[silent hinge]

alr

I'll report back in a few mins when it either finishes or errors out

[barren hound]

Has there been made any locally ran video diffusion yet?

[severe moon]

that one has newer torch/cuda built in which is needed to do run the SVD model

[severe moon]

yeah

[bold wave]

OK this one is using dragons setup. At first I tried at 512x512, and booyyyyy it did not like that. Also, I was using SVD before

[silent hinge]

what model is that instead?

[bold wave]

SVD_XT

[silent hinge]

whats the image decoder version of the model for?

[bold wave]

lol i dont have a freaking clue, i tried running it and it seemed exactly the same.

[half hound]

I think I am close to figuring out being able to create layers in the SVD anyone have any insight in this error? @grim tangle @severe moon

[severe moon]

the tensor's don't match. high level math stuff. something being output is not compatible with where it's going.....

[silent hinge]

btw how can i speed up loading the SVD_img2vid model while keeping it in low vram mode?

[bold wave]

there's that fp16 version, you might try that. I dont know if it offers quicker load speeds, but it could.

once it's cached though, you don't need to wait so long.

[silent hinge]

where does it cache

[bold wave]

lol this turned out just awesome.

[silent hinge]

lol I love it

[bold wave]

lol no idea, honestly, i dont know those kind of tech details. just that once it loads, it doesnt take as long.

[severe moon]

whoa! the chains are moving!

[silent hinge]

Alr. Fingers crossed its not vram

Also were do i download fp16 version

[severe moon]

hmm someone linked to it earlier

[bold wave]

I found this one. As always, use at your own risk, i found the link on reddit. https://huggingface.co/becausecurious/stable-video-diffusion-img2vid-fp16/tree/main

[severe moon]

where did you get compressed checkpoint?

[bold wave]

theyre safetensors though, which is good

[silent hinge]

alr

[wicked wren]

My first ever SVD. The potential is kinda mind boggling

[bold wave]

dang. that one is really good.

[silent hinge]

Oh wow

[wicked wren]

When this thing get easier UIs and better control of motion / expressions etc it will be revolutionary for sure

[silent hinge]

Just wondering, whats the lowest vram someone has been able to run SVD on so far?

[wicked wren]

One thing that would be cool is embedded alpha channels to be able to use layers in a VFX environment

[wicked wren]

Im using 3080 10gb

[surreal haven]

While I was running this piece of code, I encountered an error. Can you tell me what the issue is? How should I go about fixing it?

https://github.com/BalintKomjati/smooth-infinite-zoom#a-colab-notebook-to-generate-infinite-loop-videos-in-minutes

[sharp kestrel]

https://huggingface.co/becausecurious/stable-video-diffusion-img2vid-fp16

[silent hinge]

what does this mean? (You shouldn't move a model when it is dispatched on multiple devices.)

[sharp kestrel]

if you find it for comfy, i'll be happy to know about it 🙂

[silent hinge]

HOLY CRAP IT WORKED ON 6GB VRAM

[severe moon]

awesome thanks

[silent hinge]

I was dumb and didn't set the resolution right so it is cut off but it worked

[surreal haven]

Sure, I'll try to solve this problem on my own. If I find a solution, I'll let you know. Meanwhile, I would greatly appreciate any advice or insights you might have.

[silent hinge]

only took 30 mins lol

[sharp kestrel]

i dont have any, i never used it and i dont use auto1111

[half hound]

I did it boys I figured out how to layer videos in ComfyUI. @severe moon

needs to do some cleanup

but the nodes are working

[silent hinge]

you mean 2 'videos' together? noice

[half hound]

yep

[silent hinge]

sweet

[silent hinge]

you're right its so much faster running it a second time

[sharp kestrel]

what are we seeing here? green screen?

[half hound]

yeah something like that

[surreal haven]

Do you have any suggestions for making an infinite-zoom video?

[barren hound]

Shit, managed to reply to the wrong. Sorry bout that

[half hound]

I am going to try an example with 3 layers and cleanup the nodes then I'll share it

[severe moon]

no worries

[sharp kestrel]

i did it once in MJ outpaint and then used a video editor to stich it up together

[silent hinge]

If SVD can work on 6gb vram, could it theoretically, just very slowly, work on 4gb or even 2gb? I have some cards like that I could try if I wanna suffer lol.

[bold wave]

heck, you can always just run it on CPU. I dont think its a question of if it can, just a question of how long you are willing to wait.

[wicked wren]

Awesome, possible to output a alpha channel you think?

[silent hinge]

My cpu is much slower than my 1660 ti

[barren hound]

So, was this made with comfy? Or some other diffusers?

[half hound]

probably yeah. Use Save Image after vae decode

[wicked wren]

Cool!

[silent hinge]

nice

hae you guys played with samplers or is euler the only one video works on?

[severe moon]

SVD was added to ComfyUI in an update a couple days ago so anyone can run it locally once they have the checkpoints

[severe moon]

they all work

i even tried LCM it looks all blurry most of the time tho

[silent hinge]

which do you prefer dragon?

are you sticking with euler?

[severe moon]

i'll try a different one but i don't think it makes a huge difference so euler seems fine for testing stuff out

[bold wave]

I went through a lot of them after I got it - most of them sucked haha.

[silent hinge]

sticking with euler raydestarr?

[scarlet bay]

https://www.youtube.com/watch?v=NN8jfMZVzZ8
runway x stable diffusion colab

[barren hound]

Can someone link to the custom nodes needed for video diffusion in the pins?

[half hound]

need to update comfyui

[barren hound]

Ah, so i'm simply just outdated :P

[half hound]

yep

[bold wave]

There's a fine line between really bad and really good I hope I didnt just mess up the settings with my next iteration.

[silent hinge]

whoa

Why's it so blurry?

[severe moon]

changed from euler to fancier sampler, definitely some different motion

[silent hinge]

holy shit

i cant believe how good some of the motions are

[arctic reef]

Looks interesting. Could you share the settings?

[teal harbor]

For the Screen2Gif?

[bold wave]

throw in facial restore, it might fix the janky eyes. that looks way good.

[silent hinge]

oh yeah how do i get facial restore again

now that i have the manager

oh i think i see

[bold wave]

I use the model, and the loader.

[silent hinge]

sweet. nice and ez

[severe moon]

euler_a taking a break or somethin......

[silent hinge]

lol

[bold wave]

well. that one sucks

[severe moon]

could just be that seed for euler_a

[silent hinge]

should i insall all facerestore? theres 3 options

[bold wave]

a lot of this is just experimentation and seeing what works best for you. you can always reload my model and then say install missing nodes. you might have to download the face restore model separately.

[silent hinge]

whoa

[severe moon]

yeah euler_a with seed increment added motion

[silent hinge]

those are wild jj

[silent hinge]

so if i installed face restore gfpgan 1.4 once its installed and i refresh im good? nothing to click or check in the ui right

[arctic reef]

I'm digging through my best generations from SD 1.5

[barren hound]

hmm, what can cause the blurryness?

[bold wave]

A lot of things can. CFG, wrong sampler, wrong bucket ID, stuff like that.

[barren hound]

Using the one in the pins for 25 fps

Just set res to 1024, fps to 25, 5 second clip and motion to 255

[sterile mesa]

Awesome!Would love to see the workflow!

[noble wolf]

what do seeds control for img2vid, how it moves?

[silent hinge]

i think the way its moves is kind of random/rng, but that is more i believe motion-bucket-id, sampler, fps, steps

[noble wolf]

motiona-bucket id is how much it moves, fps is just how fast it goes through images, and steps is just how detailed each image is gonna be with newly generated parts no? That's why I was assuming seed had something to do with the "how" of it moving, maybe not idk. It takes too long for me to want to test each individual setting. I need to get more vrammmm

cause this one is only 20 steps, but 100 motion-bucket id, so its moving a bit, but the steps can't keep up to make it look decent, at least I think?

[silent hinge]

oh yeah, how do i change the format for the file again?

[noble wolf]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite/tree/main/video_formats

install that through comfy ui manager

do u have the manager?

https://github.com/ltdrdata/ComfyUI-Manager

and then there should just be a custom node called "video combine vhs"

has lots of formats

[silent hinge]

oh ok i think i see

[arctic reef]

Why face upscale is important (and frame interpolation, too)...

[bold wave]

face upscale? is that different than face restoration?

[noble wolf]

and where do I find both of those, face resto and upscale

[arctic reef]

Face restoration, of course. My mistake.
But upscale is also part of the process - makes detail better and helps face restore find faces (i think).

[silent hinge]

is it this? i installed but still dont see it

[barren hound]

Wait.. I might have read the info wrong. when it states 25 frame gens, that means total frames? Or fps?

[noble wolf]

restart

always restart after new install

"refresh" button doesn't always work

specifically for custom nodes I think? maybe

[silent hinge]

oh ok. does it need a certain workflow json too?

[broken storm]

question here, how do i install SVD with the github repository? i downloaded the file the repository provides, but i dont know what to do after that

[silent hinge]

got it now. thanks

[noble wolf]

you can use the images here for workflows I believe

it has a basic setup for running video gens

or wait maybe not the right link

one sec

[silent hinge]

do you reccomend any other installs for video

[noble wolf]

https://comfyanonymous.github.io/ComfyUI_examples/video/

sorry here

uhm any other installs? hmmm idk sorry honestly I downloaded it last night

[bold wave]

haha just making sure i wasnt missing something.

[noble wolf]

just been messing with it since then xD

[silent hinge]

ahah ok

[noble wolf]

I learned comfy ui last night at 1am 😛

[broken storm]

could anyone answer to this? i do know how to use stable diffusion image gen locally, i dont remember how i did it but i wanna know how to use this locally in my computer but i have no clue

[noble wolf]

its included by default now I believe

https://comfyanonymous.github.io/ComfyUI_examples/video/

click this

drag one of the video onto your comfyui for the workflow

it should be self explanatory, put image in the left, set the settings fps/frames/movement and stuff

[silent hinge]

duskal did you install any face upscalers?

[noble wolf]

Ah no not yet!

[broken storm]

oh

[noble wolf]

I've been messing with forest and shits mostly

not with good results, only face I did was up close

[silent hinge]

not bad!

[noble wolf]

its only not warping the face because I'm super up close

[urban linden]

if anyone is finding a workflow to extend the video, please let me know, I'll light a candle for you or smth.

[broken storm]

so the SVD github download should include it by default

iirc for SD i had to do a lot of setting up for the gui to work

[noble wolf]

extend the vid...

how do you do that I wonder huh