▶｜stable-video-diffusion

[barren hound]

Where's the setting to export as mp4?

[urban linden]

Yeah like this dude did here : https://www.youtube.com/watch?v=Ie2ttY5jELU&lc=UgwofUx3mSiTPXpTnaZ4AaABAg&ab_channel=Purz

But it's just too complex for me...

[broken storm]

yea comfy ui is difficult

[noble wolf]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite/tree/main/video_formats

install that with the comfy ui manager

[broken storm]

all i wanna do is run the SVD github repository locally like i did with SD wich btw works rlly good

[noble wolf]

duckers

you can

just drag and drop the workflow from that link

svd is built in now

[broken storm]

how exactly? i dont use comfyUI

[noble wolf]

Do you have comfy ui up? and running?

[broken storm]

no, i dont even have it downloaded lol wair

[bold wave]

Time to use it - or wait a few days for A1111 to catch up.

[noble wolf]

https://www.youtube.com/watch?v=AbB33AxrcZo

here is a starter guide

its not even an install, just download it and extract the folder

plz use comfy, its so good, more efficient than a1111 imo

[silent hinge]

i still dont see the option to change format 😭

[broken storm]

i do use a111 for SD tho

[silent hinge]

comfys layout is anything but but it seems pretty cool i guess

[noble wolf]

It's only weird at first, but it's quite customizable

so it can be as complex or as simple as you want

you can wait to do video stuff if you want, but if you want sd video you gotta use comfy for now

[sonic stirrup]

HOW DO I SEE THE OPTION TO GENERATE 😭

[noble wolf]

wtf

[sonic stirrup]

HUH

[sonic stirrup]

?

[noble wolf]

this is what mine looks like

[sonic stirrup]

OH UR USING COMFYUI

IM NOT

damnnnn

[noble wolf]

ah yeah my bad what're you using?

[sonic stirrup]

streamlit

[noble wolf]

huh never heard of it my bad

[sonic stirrup]

i watched a yt tut

its ok

[half hound]

just upload an image and the error will go away

but you should get comfyUI easier install and you can do more with it.

https://youtu.be/hoIobzZmNiM

[sonic stirrup]

the error doesnt go

plus im on amd

thyat might be a reason why

[half hound]

ahh yep

uses cuda

[sonic stirrup]

ohh

so it wont work on amd??

[half hound]

not sure

[sonic stirrup]

hm

[half hound]

I see most AI generated content uses cuda

so if you plan on doing a bunch of ai stuff probably should get an nvidea card

[sonic stirrup]

ah ty

[half hound]

but there might be support for amd, I am not sure

[bold wave]

AMD always lags behind. If you wait a few months, there will be support eventually.

[noble wolf]

that's so sad

[bold wave]

When SD first came out, it didnt have AMD support just yet and I went out and bought a card.

But you can go CPU still.

[noble wolf]

cpu is dead slow

[sullen harbor]

my AMD 6800XT worked pretty well on Linux with ComfyUI and SVD

[broken storm]

i mean i dunno

[sullen harbor]

it's slower than it should be because AMD doesn't have flash attention implemented in pytorch

but it works

[bold wave]

it is slow. but at least you can do it. your AMD video card wont do it. so for you, it's the quickest, most optimized solution.

[half hound]

@grim tangle Is there a node that attaches to other nodes that changes outputs? Like If I want the user to select width and height from one node it will change the values of the other nodes?

[silent hinge]

do i need to load a certain json to be able to change format ?

[bold wave]

still trying to save as an MP4?

[silent hinge]

or anything else. i have the video helper suite installed

[half hound]

download VHS extension

o ok

double click type vhs

[noble wolf]

now look up custom node "vhs"

its the videocombine one

[bold wave]

use my solution and go to install missing nodes.

[noble wolf]

can i post nsfw in here I forget

[sullen harbor]

no this discord server is supposed to be 13+ only

[noble wolf]

gotcha

[silent hinge]

can someone screenshot where to change the format then?

[silent hinge]

oooh i see. the ui is so wonky it was way off of my view

[grim tangle]

I do it like this: (the "INT Constant" is the node I set the values, just renamed the titles to Width and Height there)
then you can use the Get nodes anywhere to get those values

[bold wave]

I had to install something to save MP4's. It threw errors until I installed some kind of library.

[silent hinge]

i had to expand to full screen

got it now

any best settings/selection for facerestore?

[half hound]

perfect thanks Kijai!

[bold wave]

I just do base and it works. but if it messes up the face too much, lower the setting down.

[silent hinge]

now though i get this: SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

😭

[half hound]

from the SD jsons?

[silent hinge]

yeah

[half hound]

how are you downloading them?

are you right clicking and save link as?

[silent hinge]

yep then drag in

[half hound]

try clicking load

and then choose from the pop up window

[broken storm]

where does he drag the svd and svd xt checkpoints from? it doesnt show

[half hound]

from downloads folder

download them and then move them over to the checkpoint folder

[broken storm]

the svd install is all of this tho

[half hound]

that's the github repo

download the models

[broken storm]

where could i find the download?

[half hound]

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt

[broken storm]

yea but where do i click for it to download tho

[half hound]

file and versions and pick the svd.safetensors

[bold wave]

click on the download button.

[broken storm]

so i move those two to the checkpoints folder of comfyui yes?

[bold wave]

yes. then hit the refresh button

[grim tangle]

you can ignore the "image_decoder" file (and if you want to do 25 frames instead of 14, get the svd_xt.safetensors from here: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/tree/main

[noble wolf]

we need chest physics

[bold wave]

https://tenor.com/view/bonk-gif-26414884

[noble wolf]

dude wait till we get a control net and can make our own horny bonks

hahaha

remember to include their forehead lmao

[severe moon]

pretty cool how it can just hallucinate details found nowhere in the original image:

[silent hinge]

wow

[severe moon]

wow I turned the denoise value down and sh!t got crazy

[silent hinge]

lol!

[fallen wren]

btw yall, #🐝｜swarm-ui message SVD works in Swarm now :D

[silent hinge]

nice

never used it but might have to check it out

[open heron]

Be afraid

[fair otter]

Has anyone been able to get this to run on silicon macs? If you have would appreciate some help as I can't quite seem to get it.

[spring oak]

A little proof of concept

[severe moon]

Knights of the Sticky Icky

[half hound]

I didn't really think too hard about what all the timings do. Can someone breakdown what all the red highlights mean?

Does the Video Combine frame rate even matter?

does the fps from SVD_img2vid Conditioning overwrite it?

[fallen wren]

frames is total frames in the video, XT uses 25, non-XT uses 14.
fps is framerate the model is going for.
The combine node fps value is how fast it actually renders at. Generally you want both FPS's the same but it can do things if you separate em

[lean gate]

how many videoframes and how many frames per second

[fallen wren]

eg if you do fps=20 in the cond and fps=5 in the combine, you get a slowmo effect

[half hound]

Thanks, and what does the clear cache after n frames and multiplier do in the RIFE node?

[icy valley]

I love your pfp!!

[spring oak]

I created the image by training a LoRa model of a famous TikTok girl and then tried creating a video using that image

[bold wave]

I'm also curious as to what augmentation level does. I only know that if I raise it too high, everything goes incredibly bad.

[icy valley]

Very impressive, I must say.

RART gets his hands on this sht for 2 seconds, and..

[grim tangle]

the fps should be named fps_id, it has nothing to do without output fps, here's one example what it does:

and because things can never be simple, here's one with super high values... I don't claim to understand at all 😛

[half hound]

@grim tangle I am using video combine on 2 sets of videos, now it stitches them together fine with image/gif and image/webm but when I try any other the video formats I get this. Any idea how to fix?

[grim tangle]

how are you combining them exactly?

[half hound]

I can hope into a channel and show you. probably easier to explain it

[noble wolf]

I'm missing some of these, might you guys know where to find them?

[clever pendant]

everything you need to run comfy is contained in the portable folders. just run the update, also in the folder

[fallen plinth]

What is the workflow for this?

[noble wolf]

I ran the update

I can run comfy ui

this is for a custom node

[clever pendant]

do you have comfy manager installed? It can manage all custom node updates for you

[noble wolf]

yes

I installed the custom node through there

that doesn't mean it installs all dependencies and libraries, only some of them

[trim juniper]

sometimes is better to install them manually

[noble wolf]

sure

[clever pendant]

if nodes have dependencies there is likely a readme or an update file in one of the subfolders

[noble wolf]

eh I'll figure it out thanks for the help

what I'd really like to know is if yall know any face restorer custom nodes

tis wonky

[noble wolf]

😮

where'd you get all that maholic

gimme gimme

oh found it yooo

[noble wolf]

That is fucking sick

[grim tangle]

sorry bit too busy for that currently

[bold wave]

Any post-stuff I can do to clean up the image a tiny bit more?

[latent pecan]

prompts :D? fantastic

[narrow canyon]

Is that 25 frames? That feels like longer

[noble wolf]

has anyone tried using this node? I can't seem to get it to work, it says I'm missing dependencies like cuTENSOR, but I tried manually installing those with pip and it's not detecting their path. How do I manually set paths for dependencies on comfy ui?

[latent pecan]

i use this one

no harm installing another interpllator 🙂

[noble wolf]

😄 yeah I'll try that one out

[open heron]

Interpolation

[narrow canyon]

Ah gotcha, it's the lower playback fps that threw me off

[robust scarab]

whats the name of the comfyui node that allows you to mersge multiuple nodes into one node? edit: i found it its called nested node builder

[open heron]

cinematic photo fantasy world female fighter gathering mana to throw a powerful spell, magic, sparkles, movement, action, dynamic angle, lightning earthquake, glowing eyes, energic, vibrating, concentration

[silent hinge]

tried it in Bing image creator Daflon got some cool stuff haha.

guess slightly ot sorry but would make a cool image to use in stable video

[open heron]

Dalle-3 is far better for those action concepts, but then you need to bring to stable diffusion for better quality

[silent hinge]

that one looks pretty solid to me, dalle in gpt results often seem more cgi to me, even though bing is also using dalle its not as restricted

[noble wolf]

You guys know what custom nodes give set / get nodes?

[open heron]

SetNode is the name

[noble wolf]

yeah no

it's called KJ nodes

which adds setnode and other qol nodes

[devout flame]

what is best svd_xt settings for full-body humans for good but balanced motion? its been working well but im struggling to get the right cfg/auglevel/motionbucket. what does everyone use? its either i get completely static images that pan but other times the model goes all over the place blurry af

i just did:
2 cfg linear guidance
130 motion bucket id
0 augmentation level
turned out ok but needed more motion. now doing 135 motion and 1.5 cfg guidance

[frank prairie]

I love how the outside was interpreted as 2D

[silent hinge]

what were your settings on this?

[devout flame]

i also would love to know, posted a question on how to get good motion (but not too much) just above but didnt get any reply 🙃

[noble wolf]

the way it goes is random but adjusting motion_bucket_id changes how much they move

if you want slow and smooth lower the motion_bucket_id

to extend you can try looping by plugging in the final frame and starting the process again

[devout flame]

wb augmentation level?

[open heron]

It still very random. To control motion the better way is...hmmm...fps

[noble wolf]

hmm that I have no idea honestly

if you want smooth like this you can use FILM VFI node

[open heron]

Keep your seed fixed (some movement that you like), control the moviment using fps. high fps = less moviment, low fps = more movement

[noble wolf]

so seed does affect movement! I was right

[devout flame]

what settings do you use mostly?

in terms of svd conditioning node

because that is perfect

[noble wolf]

that's my workflow

if you wanna look it over

[devout flame]

ok thanks!

just trial and error im guessing?

[noble wolf]

for the movement seed yeah thats just trial and error

[devout flame]

thanks, so ill keep trying with those conditioning settings 😛

[noble wolf]

you got this bro!

[icy valley]

thank you for the seed stuff

[bold ocean]

2secs of smooth animation is all i can get, what has most seconds people can get, before animation goes wonky?

[icy valley]

allowed me to make this

[devout flame]

nice!

[noble wolf]

SVD model is trained on 14 frames of motion, svd_xt is trained on 25 frames, anything more than that will get wonky results

try chaining together 14 + 14 + 14 by using the final frame and generating from that

[open heron]

Interpolation and slow motion

and oing pong

*ping pong

[noble wolf]

when I do loop though it gets deep fried... I wonder whats causing that.

multiply the looooop

deep fry her!

[icy valley]

this was my attempt at chaining

[bold wave]

if only the followup text were legible

[grim tangle]

Kinda nature of the thing, quality degrades over time and it stacks up, using same seed and lower cfg can help a bit

[long stream]

just got started on playing around with SVD today

thanks to @astral beacon and @clever pendant for the help 🙂

[noble wolf]

does anyone get a huge lag spike sometimes when moving across comfy ui and loading a gif preview?

and when it lags it moves a few nodes 😦 it's like it's having trouble grabbing data from the workflow or something

[bold wave]

ohh yeah. try doiing vid2vid stuff with 3 controlnets.

[noble wolf]

so thats normal when you have a shit ton of nodes?

just wanted to make sure I didn't download some weird custom nodes from the manager that bugged my comfy ui out

[bold wave]

oh yeah no, since youre running vids etc, its bound to happen.

[clever pendant]

so windy her leg is moving too

[noble wolf]

kk

gal gadot got cramps

[icy valley]

Wonder cramps

[noble wolf]

where do you put the face resto in order of operations?

[noble wolf]

God damn your shit is tight

thanks

[devout flame]

Does higher fps take longer? For some reason my current gen is taking 2 minutes per iteration

On 10 fps

[bold wave]

fps doesnt change how long it takes. frames will.

[open heron]

No. but i belive over 25 makes no difference at all

[noble wolf]

bro daflon this is insane, you upscale your image 3 times ? xD

or am I reading it wrong

[devout flame]

Oof, my gens usually take 10 minutes but now its eta is almost 40

Does comfyui sometimes take slower occasionally? Im on my 100th gen since i opened it at first lmao

Maybe thats why

[noble wolf]

idk, could try restarting ?

[devout flame]

I will after this yeah

[open heron]

This workflow isn't mine, i don't like this way. The first is an upscale, it's 4x, the other is a downscale (usually 0.50) but it will make the final result to big, it's good for mp4 but i like gifs

[noble wolf]

ah fair enough

[open heron]

The very first is a crop. I crop my images to the same size in the beginning

[noble wolf]

fair enough

I like the default rez, and because they warned people not to mess with it

but it seems to do fine /shrug

[open heron]

i try to keep the same. 800x800 / 1024x576 / 576x1024

since most of my image are 1536x1536 i crop them to 800x800 and set the the vid to 800x800

[noble wolf]

is that one of yours?

that's vv nice

[open heron]

Yes. All mine

[noble wolf]

I had to ask

[open heron]

The picture is. I meant i didn't make the workflow, the upscale part

[noble wolf]

Ahh makes sense

my b

[open heron]

That workflow is from someone here, i don't remember. I just did the auto crop/size thing

[noble wolf]

Yall think I should buy a 4060 ti just for stable ?

my 3070 ti is literally faster, just has shitty 8gb vram

[noble wolf]

Holy, water is so good

[clever pendant]

yeah so nice

[noble wolf]

zoooooom

[latent pecan]

wish it didnt melt the character too 😄

[icy valley]

I know I wiffed on the spelling but the effect is pretty neat

[latent pecan]

this is better

[noble wolf]

how do you get a seed out of an image?

[severe moon]

right click, open properties, Details tab, look for metadata. A lot of the posted stuff made in ComfyUI will have metadata under comments, you can copy that text into your code editor of choice and save as a JSON file. This is all if you don't want to try dragging the image into a browser tab with ComfyUI running to see if the nodes all pop up......

[noble wolf]

thanks!

is there a way to make an on/off switch for nodes?

[latent pecan]

30 motion id, instead of 130

[trim juniper]

Trying to go for the "film" look

[severe moon]

nothing simple but you can just select the node(s) and CTRL+B to Bypass them.

Everyone's seen Turkish Star Wars by now, right?

How about Turkish Delight Star Wars?

[hushed coral]

Outstanding

[latent pecan]

60 motion id

[noble wolf]

where does lora go in order?

[icy valley]

After checkpoint

[sharp kestrel]

can you repeat the effect with the same seed and mov id?

[icy valley]

Yesn't.

LOL

So I was having a problem where all I could get it to do was zoom straight in back and forth, and then I saw somebody mention its possibly the seed that controls the motion you get

So when I found a cool effect, on this one

I fixed the seed and have been using the same ever since

But with different content, it does totally different effects

This is with the same seed

And so is this

[sharp kestrel]

yea, make sense that it uses the image to try to underatsnd the direction.
the airplane is very very nice

[latent pecan]

testing an ffmpeg merge, ideally i generate like 100 of these SV in one batch run and it ffmpeg merges them all automatically 😄 thou probably want to cull the blur jank out but maybe not 😄 its all part of the nitemare of the ai

[icy valley]

Thanks!

I do think it's a good idea to stick with a seed that you get cool motion out of

They tend to do interesting motion more consistently across all other content

Just not the same motion apparently

[icy valley]

I tried something kinda similar 😅

[latent pecan]

very melty 🙂

[severe moon]

i see you....

[bold wave]

I'm looking at solutions for the blue jank.

Like. It would be cool to mix in new base images and keep going.

[sharp mesa]

How to create video here?

[severe moon]

Most of us are using ComfyUI on home PC: https://comfyanonymous.github.io/ComfyUI_examples/video/

[latent pecan]

yea it gets very melty :\

[sharp mesa]

Thanks

[latent pecan]

wonder if you can use face detection to auto filter videos out, eg if face detection fails it rejects the video from the workflow

[bold wave]

Several times I was like wow that would be cool if your whole body didn't melt down 🤣

[severe moon]

For the jank have you guys been adjusting the CFG scaling and messing with augmentation_level?

[latent pecan]

ive only tinkered with motion bucket id

30-60 seems ok , the 130 can melt sometimes, others its good for forward/backward motion as it needs more rapid motion

if i just want to jiggle a scene about then less motion isbetter for the current melt wall

[bold wave]

I cranked it up to 800 and it was actually... Fine. It was fine. Not great mind you, but fine.

I tried mixing in normal SDXL - but of course then you go back to SD problems of randomizing clothing.

Too high of CFG and models get fried. So you have to keep it a little low.

[severe moon]

if you don't adjust CFG then it can blur towards the end.... you can offset the frying by lowering CFG withis node and then it will ramp CFG up over the frames

[bold wave]

oh yeah... i think ill try that.

lowering the denoising helps as well, probably for similar reasons.

lol i spotted someone playing with AI in the wild, they were just getting to know AI, they were so excited and all their images were super fried. I was like... that looks nice.

[severe moon]

lowering denoise by a bit seems to help but too much and everything flies apart spectacularly

[keen shuttle]

Where do the videos save in SVD? In Outputs also or somewhere else?

[bold wave]

outputs

if you use VHS at least

[severe moon]

too much denoising:

[bold wave]

lol...

so i think ive gotten a good setup for realistic profiles. but then i move to environment shots and have to start all over. ahh well. i can save multiple setups for things.

[severe moon]

jsut a few notches up from the last one.....

[bold wave]

so what was the denoising on both the animations?

[severe moon]

the VHS Combine node is nice, you can set folders in /outputs or wherever to keep things sorted with the filename_prefix line:

[severe moon]

like .70 and .75

[keen shuttle]

.90 seems good

[severe moon]

yeah it doesn't really seem to change it too much on some images until suddenly it does

[keen shuttle]

So instead of the SaveAnimated Node?

[severe moon]

I was talking about denoising, are you asking about using VHS Combine instead of Save Animated node?

[keen shuttle]

I got it you can double up node outputs , I was talk about denoise level on KSampler

[bold wave]

so the min_cfg and the cfg on the ksampler, those are the same thing? ie you increase the min, and increase the ksampler cfg?

[severe moon]

min_cfg is the starting one and ksampler is end, most workflows will ramp up from low to high CFG, you could also set them the same

[latent pecan]

what do people do for upscaling of SV ? upscale before the final merge?

[bold wave]

upscale is pretty intensive, I would do it before the rife stuff, because the rife will add more frames to upscale.

[hushed coral]

grr how do I add a video like youse

[severe moon]

in the folder select the video and then CTRL+C to copy it, then back here in discord CTRL+V to paste it. or try drag and drop into Discord

[hushed coral]

sorry to sound like a complete idiot, but how do I then post it?

[bold wave]

[severe moon]

when you've pasted it it should show up as an attchment, write your post or just hit enter...

[sharp kestrel]

so who do we talk to to get LCM for SVD?

[long stream]

try h264 format

[hushed coral]

ah

[severe moon]

Not sure what's going on, Discord might not like the format....

[bold wave]

thats why i always save gifs.

[long stream]

h265 too advanced for discord 😄

[hushed coral]

format was the issue thanks folks

[severe moon]

oh wow, SVD can add shadow people

[bold wave]

[long stream]

it's like someone who accidentally walk behind a news reporter

[severe moon]

it must be the image, i tried with a different seed and just got a different shadow person.....

[latent pecan]

ghost in the imageshell 😄

[long stream]

cfg of 4 stopped the faces from blurring

[severe moon]

literally

[latent pecan]

previous ai contaminated that image 😄

[severe moon]

OK, enough of that, that image is obviously cursed....

[long stream]

casper the friendly pillow case

[devout flame]

Whats the maximum length of video can i do without frying it up yall think?

Like, how many stitches?

[latent pecan]

i tried 50 but the person looked like they were very unwell shaking about jittery ;d

[bold wave]

it depends how much movement there is. some images just kind of... breathe... those ones can go on forever.

[latent pecan]

i think the models were very tightly trained

[bold wave]

kind of mad about this one. it would be spectacular if it wasn't so... you know... horrifying. maybe a good candidate to play with img2img stuff with

[devout flame]

Okies, there isnt much movement, but im turning denoising down by .05 every gen to avoid deep frying effect

Trying it anyway

[long stream]

so far it can do panning nicely, but has issues if things rotate

[long stream]

cool lightning actually animated, few images I tried didn't animate the lightning

[hushed coral]

Both disturbing and delicious

[icy valley]

I was really hoping it would, and ecstatic it did

I'm surprised by the amount of things that work when i try them

[severe moon]

most of the time the camera just pans around main subject, moving the background, but not all of the time....

[icy valley]

#▶｜stable-video-diffusion message

like this one too for example

I gave it a jet plane, it animated exhaust behind it and moving clouds

[latent pecan]

fighter jet, clouds in the background , jet exhaust trail as jet flies across

[wicked wren]

Hey all, is it possible to run SVD (comfy) on a silicon Mac yet?

[tiny cradle]

I tried to make it work, got stuck here: https://twitter.com/bryced8/status/1727391254445756450#m

[latent pecan]

nice last of us music

[long stream]

threw a bunch into clipchamp to make this compilation

[solemn turtle]

Yo how much vram do you need train a full stable video diffusion model? 70-80gb Vram?

[sharp kestrel]

is that a borsht monster?

[long stream]

tofu 😄

[sharp kestrel]

my favorite today

[bold ocean]

I tried doing a Highres-fix pass to get a bigger video, but I just got a worse, blurry video, besides Topaz, are there any other good video upscaling and adding detail solutions?

[magic dagger]

How long does it take to get access after signing up to the wait list?

[pine widget]

flying cat

[left geyser]

hi guys, i have a question about stable video diffusion... Im using a infrastructure broker for hardware like replicate.com, im trying to make videos of roughly 5 seconds long from an image... Using replicate.com im just being able to generate 2 seconds with a super low framerate since it is limited to 28 frames total. How are you guys creating those 9-18 seconds long videos with a normal framerate? (24-30fps)
Thank you so much in advance and sorry if i missed the right category to make this question

[silent hinge]

I still cant figure out how to save as a gif. comfy ui is like navigating a maze every use

think i got it, but now I get this: SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

any ideas?

[half hound]

Anyone remember what the max value for motion bucket ID is?

537?

[silent hinge]

thats aweseome funereals

anyone know this error means? SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

or does anyone have a json i can use that has the gif mode? I cant figure out how to set it otherwise

[static tusk]

Perhaps you have a comma after the final part of your prompt?

[sharp kestrel]

if it's for prompt scedule, it means you have a space (or enter) where you shouldnt have

maybe the case with last comma like Leoban said aswell

[harsh storm]

ComfyUI

Its API isn’t straightforward tho. You can wait for diffusers implementation

[unkempt mica]

I just use photopea to convert to GIF

[silent hinge]

I'll have to look into that

what was that other ui that can do video now? comfy i think is just too busy and confusing for a simpleton like me

[unkempt mica]

StableSwarmUI

[silent hinge]

cheers

[unkempt mica]

Has a back end of comfy

[silent hinge]

but an interface for us dummies haha

[unkempt mica]

So all comfy function but nice easy to use UI

[onyx ore]

Try SVD in dreambot

[unkempt mica]

Haha exactly

All you have to do is make sure you have the SVD in the model folder and click the toggle for video. Done. 😂

[silent hinge]

noice

[unkempt mica]

#🐝｜swarm-ui if you need help or links.

[silent hinge]

windows wont dl says it can damage my system 😭

[unkempt mica]

What?

You downloading from GitHub?

[silent hinge]

yeah gave me a warning. yep

https://github.com/Stability-AI/StableSwarmUI link i used

i got it to work though

[unkempt mica]

Microsoft does not want you to be happy. 😂

[forest blade]

Can I use SDV on colab free version ?

[fossil atlas]

how are you all making the output ping-pong?

[lapis mango]

Well, she's sort of alive. Doesn't quite feel like an animator's work though.

Much livelier! Subtle parameter tweaks are the difference between static, fluid movements, and crazy chaos.

[severe moon]

by setting ping-pong to True:

[severe moon]

do you have FreeU node somewehere in your workflow? if so try taking it out

[fossil atlas]

what pack is that node in?

[arctic reef]

Some of my cosplay photos, animated.

[severe moon]

https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite

[silent hinge]

i might, i dont know what that is 😂

I just start to use all this stuffs yesterday so yeah still learning big time

I drag the json someone give yesterday into comfy, then try to do video and get that error so idk

[severe moon]

did you download ComfyUI from scratch yesterday?

[silent hinge]

yes

oh wait, i think i did try it one other time and couldnt figure it out months ago and deleted it

[severe moon]

did you add ComfyUI Manager yet?

[silent hinge]

maybe thats some issue

yes

[severe moon]

you have the portable ComfyUI version?

[silent hinge]

yes

[unkempt mica]

Since Swarm uses comfy you can do all comfy stuff directly in Swarm in the comfy workflow tab. Add manager to the custom nodes folder in the comfy folder and restart.

[severe moon]

what was the exact name of the .7z file if you can still see it? like:

[silent hinge]

new_ComfyUI_windows_portable_nvidia_cu121_or_cpu

[unkempt mica]

No need to have a bunch of separate installs.

[severe moon]

sounds like the latest stable comfy build then

[silent hinge]

do you have a json i could try dragon?

[severe moon]

ok, try using my video I just posted to test the workflow or this image:

[unkempt mica]

The official one is pinned in this channel. But it does not have extra stuff Dragon prolly has.

[silent hinge]

i really am just trying to get it to change to gif

[unkempt mica]

Haha got it. Yeah that's not built into Swarm yet.

[severe moon]

Manager should highlight any missing nodes for you if you load my workflow

there's probably a bunch of extra nodes crap in there from my testing, just ignore it, the main workflow is all smooshed together:

[silent hinge]

so i load that image in manager?

[severe moon]

yeah you can click it here, Open in browser , then in that new tab just drag it onto your ComfyUI window

[silent hinge]

nothing changes when i drag it into comfy

[severe moon]

hmmmm

[silent hinge]

just drag anywhere you mean, or load the image?

[severe moon]

the alien trail dude image?

[silent hinge]

the one you just shared

[severe moon]

that's just a screenshot of my sdesktop

use the alien dude

[silent hinge]

uh what alien dude?

[silent hinge]

oh ok

that worked

[severe moon]

lizard people, alien., i dunno lol

[silent hinge]

got this though: When loading the graph, the following node types were not found:
PrepImageForClipVision
RIFE VFI
FILM VFI
GMFSS Fortuna VFI
Nodes that have failed to load will show as red on the graph.

Close
Queue size: 0⚙️
Queue Prompt
Extra options

[fossil atlas]

does that text concat actually work?

[severe moon]

no Comfy was in here a couple days agao and said " you can use the conditioning concat node with "conditioning_from" as the output of a clip text encode connected to the SD2.x CLIP model"

i was trying to figure it out but i had the dumb

[severe moon]

now use Manager to Install Missing Custom Nodes

[fossil atlas]

i mean, that makes sense, but does it actually put mickey mouse in the video, or just confuse the model completely?

[severe moon]

it errors out IIRC

some bit of the CLIP chain is still missing

[fossil atlas]

hmm... you don't have anything to connect to clip input

clip != clip_vision

[severe moon]

like I said I had the dumb

[fossil atlas]

you could maybe concat two image conditionings into one video... i wonder what that would do

[silent hinge]

wow dragon this is quote a workflow thing you have haha

i think i got it but when i drag an image in it switches it back to my old ui

[fossil atlas]

yeah you have to click "choose image to upload" on the image loader node

dropping images on the browser window always loads the workflow used to make that image

[severe moon]

hmm they probably have embedded json in them, try Opening them instead with the Load Image node, or carefully drag on top of that node

[barren hound]

Hoi, what node do i need to choose output format?

[silent hinge]

ok got that but now i get this: Prompt outputs failed validation
ImageOnlyCheckpointLoader:

• Value not in list: ckpt_name: 'SVD\svd_xt-fp16.safetensors' not in ['svd.safetensors', 'svd_image_decoder.safetensors', 'svd_xt.safetensors', 'svd_xt_image_decoder.safetensors']

[severe moon]

VHS Video Combine

[silent hinge]

oh wait i got it

thanks dragon, i know im not easy to help haha

[severe moon]

yeah you just need to point it where your models are stored

I like storing my main checkpoint in SD15, SDXL and now SVD folders so they're all grouped together

[fossil atlas]

i do that too, but of course my folders are named differently to yours. it would be nice if that could be standardized

[silent hinge]

so you prefer mp4 or gif?

i just wanted to get away from the web format thing

[severe moon]

mp4 will make smaller file sizes

[silent hinge]

nice

[severe moon]

but gif you can jsut upload whereever and it works in msot browsers so....

[silent hinge]

yes I am thinking of sending some silly things to friends so just thinking best universal format they can see

[severe moon]

once you get a cool video you can change the format right away and it will reencode it without doing the whole workflow again, if you change right away

[silent hinge]

nice!

[fossil atlas]

gif is an objectively bad format for video and will produce huge files. the only reason to use it is high playback compatibility.

[severe moon]

and Discord like mp4, very easy to share here

[fossil atlas]

mp4 is good, but make sure to use yuv encoding or discord won't play it inline

apparently the nodes already do that but it isn't the default with ffmpeg as i always forget to my cost

[fossil atlas]

has anyone made a workflow that generates a video and then recurses on the last frame?

[severe moon]

there's some youtube vids up of that sort of thing, it tends to fry everything by the end cause of the CFG artifacting

[silent hinge]

so does loop count to 1 make it, well, loop haha or is 0 do that

i guess i can click a file and find out too

just looking over all these dragon settings

wow dragon that one is incredible

[severe moon]

i think loop count will tell ffmpeg to make it that many loops when it encodes the video, so it will loop 4 times and then stop at the end if you put 4

[severe moon]

just a lucky seed 😄

[silent hinge]

you guys just like the widescreen image ratio or think its more effective?

[fossil atlas]

it might depend on output format. mp4 doesn't have a loop counter but gif does i think

[silent hinge]

ahh

[severe moon]

that's what it was trained on but I've seen plenty of fine results at around 800x800 or flipping the width and height for tiktok vertical vids

[silent hinge]

oh good to know. im using my dalle3 images on it and theyre all square of course so i was just wondering

well i guess not of course, you can do widescreen there

[severe moon]

yeah most of my source images are from Midjourney so I just set my default setting to output 16:9 cause I've been making so much widescreen stuff with SVD

[silent hinge]

I'm waiting for the 6.0 release

[unkempt mica]

Monkey said he recommends 16:9.

[severe moon]

SOON™

[barren hound]

Jeebus, what setting is so damn off which wibble wobbles the whole thing lol. I've set it to use the 25 fps one, but it's so bad lol

[unkempt mica]

O God my EYES

[barren hound]

Please export you guys's workflow so i can see what went wrong with mine lol. Only change from default pinned' one was just resolution and framerate

[fossil atlas]

try 768x768

[severe moon]

try motionbucket around 30 if too wobbly

[barren hound]

Will do. And is there a way yet to describe what image should do in video? Or just random motions for now?

[severe moon]

if you stray too far from the training size you might want to adjust augmentation level as well

[severe moon]

not really, seed change has the most affect on type of motion from my testing

[barren hound]

Speaking of. It mention 25 frame, does it mean max 25 frames? Or 25 fps?

[barren hound]

Ok, so it will read from the image, like ghost on skateboard, it sees device with wheels, that means subject is rolling at speeds, and it proceeds to animate exactly that?

[fossil atlas]

wait, you have that set to 75

[barren hound]

Aye. 3 seconds of 25 fps. that's what i asked, if 25 is max supported total frames, or max framerate video can be at

[severe moon]

video_frames is how many frames total it will produce (determines how long it will be making frames) and fps is.... fps, so how many frames show each second

[fossil atlas]

25 is max for svd_xt

[severe moon]

there's no real max it jsut tends to start degrading after 25

[barren hound]

Ok, so max frames trained for is 25, meaning if i want 4 seconds long and good looking, i need it to be 6 fps?

[fossil atlas]

yes

[severe moon]

basically

[barren hound]

Gotchu. Have my last wibble before the proper wobble

[severe moon]

that's why people are adding interpolation nodes, to produce at lower fps but then smooth the video out back to 24 fps

[fossil atlas]

the fps setting controls the time step, so making it smaller means things move more from one frame to the next... which can produce bad results because it can lose track of what everything is supposed to be

[barren hound]

Aye. I got flowframes, so i can easily interpolate that

Or actually, link me the interpolation node expansion, wanna see how well it can do interpolations vs flowframes :D

[severe moon]

searching for "from: kijai " here you can go back and check their posted video grids comparing all the settings

https://github.com/Fannovel16/ComfyUI-Frame-Interpolation

[barren hound]

One step closer, now there's just a pirhana in there xD

[fossil atlas]

looks like too low framerate

[sharp kestrel]

https://www.instagram.com/p/C0HY5GHR2o-/

gilibenshahar

Choose your fighter!
(Or lawsuit, whichever comes first 🤭)

I used my most favorite Lora from @hellocivitai , it's called Felt Puppet Lora and made by @socalguitarist_ (who also made the model ive used -NightVision XL)
Animations were donne with Stable Video Diffusion - a whole magic by it's own .

Oh, the intro/outro text was created with harr...

Instagram

(might as well try to get some clout while at it 😛

[forest blade]

can anyone help me to use sdv, on either colab or kaggle

[barren hound]

Nope, Too high that i forgot to change

[fossil atlas]

25 is not too high...

i mean it might be too high to produce anything interesting, but the higher you set the frame rate the less motion there should be

[severe moon]

yeah when you adjust fps on the SVD node you need to go through your whole chain and look at all the numbers affecting frame rate

so in my chain it goes from 8fps on SVD, into interpolation node which mutliplies by 3, so final 24 frame_rate on final Combine node

[silent hinge]

wow!

[severe moon]

standard film rate is 24fps

so here's two vids, only changing # of frames, fps and interpolation:

the left one is my usual 8fpsx3>24 workflow

on the right it's 24fps with no interpolation into 24 framerate on the Combine node, that makes it all wobbly

the left one is 24 video_frames and the right one is 72 and they both have ping-pong set so they're both about 3 seconds of produced video before the reversed playback kicks in

the wobbly vid looks worse and takes 3 times longer to produce

[fossil atlas]

because of 72 video_frames. that's the only reason

everything else only affects the duration of the video

[grim tangle]

you misunderstand the fps on the SVD cond node I think

[severe moon]

the only reason? no, that just makes the video short and then hardly anything interesting has time to happen. we avoid wobble by sacrificing time:

[grim tangle]

it does more than just "set the frame rate"

it should be called "fps_id", it's bit confusing the node just says "fps"

[fossil atlas]

no, you just misunderstood what i am saying

video_frames is the cause of wobbling in this example

[grim tangle]

didn't mean you

you are absolutely right about that

[severe moon]

that's exactly what i demonstrated

[grim tangle]

going past 25 is really never a good idea with this

at least by much, a little bit can give like smooth stop effect or something

the fps setting does not affect the number of frames generated in any manner, just what kind of motion you get

[half hound]

@grim tangle is it possible to change node order in ComfyUI?

[grim tangle]

similar to motion bucket id

[grim tangle]

execution order? I haven't found a proper way, why though?

[copper berry]

btw how much time does it takes to gen a video ? (for exemple same size, fps and steps as the one from dragon)
> also with what gpu ?

[severe moon]

exactly, it's video_frames determining how many frames are generated

[half hound]

Trying to load images from a dir in same workflow after it outputs the image seq. Problem is it occurs before the images get generated.

[grim tangle]

why is it a problem exactly?

[half hound]

because it doesn't export to video formats properly in my layering technique, it only exports to images properly. But if I make it an image sequence and load them back in it works fine.

[fossil atlas]

the loopchain node pack can do this... but it is really difficult to understand... i couldn't figure it out, even with the examples

[grim tangle]

you can disable the node and enable after generation I guess

are you using images with alpha or something?

[half hound]

I'll have to take a look

[half hound]

yep

[grim tangle]

you probably shouldn't

[half hound]

probably easier to just open new workflow

[grim tangle]

instead take the alpha to a mask, that's how comfyUI does those things, alpha channel itself is not used beyond loading stuff

[fossil atlas]

just fix the video export node to handle it properly?

[grim tangle]

what would it do with alpha, make it white/black though?

[half hound]

tried that. Video combine doesn't like Alphas in video format. Ok in the image formats though

[fossil atlas]

as a quick fix, just ignore it

[grim tangle]

but why are you saving it in the first place, you need alpha channel in some other software?

[severe moon]

most of my vids are taking 65 seconds on a 4090

[half hound]

to combine videos in comfyui

it works, just too bad it's a bit of a pain to try and make it a video format

but that's ok

[grim tangle]

I'd need to see the workflow tbh

comfyUI just doesn't use alpha channel, you can load mask from it but that's it

all masking, composition etc. is done with grayscale images

the end result you export should never have alpha channel in the first place, I dunno how you get it 😄

[half hound]

I'll share the workflow in a bit. I'll make my video tutorial and then i'll post it.

[fossil atlas]

hmm... SVD conditioning latent output is always an empty latent?

[grim tangle]

yeah you can even use your own empty latents

or any latent, I tried encoding images and got some interesting results

you can even inference without the SVD cond node at all

using just clip text encode, I found some stuff.... but it's whacky 😄

[fossil atlas]

it also seems like leaving the negative unconnected basically has no effect on the result at all

[grim tangle]

I got this out of it (input, unsampled noise, first sampler, 2nd sampler)

[fossil atlas]

or even connecting it to something else entirely

[grim tangle]

I don't know how to explain, so don't ask

just fed unsampled noise noise as latents and text prompt

[left geyser]

thanks for the reply, im a developer looking for an api or sdk to generate the videos, a gui is pretty bad for me, do you have any suggestion? thanks in advance

[fossil atlas]

comfyui is programming, just with nodes instead of text. it's easy to write new nodes or look at what the existing ones are doing if you know python

[bold wave]

ok that one is dope.

[fossil atlas]

there is also a tool that can export a workflow as a python script, although it is a bit janky

[bold wave]

yeah what you want to do is make the workflow, then export it to use as the API. Then, you can alter the settings programatically.

I am using it for a project, the API is actually really simple.

[severe moon]

https://tenor.com/view/spoopy-spooky-halloween-gif-14092772

[copper berry]

ok ok ty 🙂 so at least a 24gb card

[bold wave]

a guy was on here doing 6GB cards. I have a 12 GB card and I do just fine.

[severe moon]

many people have reported success on older cards with as little as 8GB of VRAM, it just takes longer and you don't have as much headroom to do other things while letting ComfyUI render in background

[fossil atlas]

i have 2070S 8GB and video gens take about 2 minutes

but only with the 14 frame model. i can't run the 25 frame one at all

[copper berry]

i think i will do it on a 3080 and some A6500 so should be good

[harsh storm]

Diffusers is the only thing fits your description in this field but it currently doesn’t have SVD. As other ppl said, you can try Comfy’s internal API which uses REST and WebSocket

[bold wave]

comfy is just a UI that runs commands based on what you feed it. If you want to write your own API, with your own switches and toggles, you can do it. I feel like it's not worth the effort, though. using existing framework is going to shave a lot of work off.

[harsh storm]

Diffusers doesn’t have SVD still

[fossil atlas]

there's still a lot of "missing" nodes imo. i wrote a few because they didn't exist...

[unkempt mica]

StableSwarmUI has an API

[harsh storm]

Is it a separated one from Comfy?

[unkempt mica]

It uses comfy as backend so it uses its own API

[harsh storm]

I mean different format or smth

[silent hinge]

Dragon you are getting some great stuffs. Just using your settings you sent me in that json?

[severe moon]

pretty much

[fossil atlas]

what model did you use for clip text encode?

[grim tangle]

https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/open_clip_pytorch_model.bin

[unkempt mica]

I have never used comfy API directly. I just do a websockets request and make a txt2img payload to tell it what perams I want.

[grim tangle]

concating text prompt

[severe moon]

the power of changing seeds

[sharp kestrel]

wait wait wait, this is with svd?

[grim tangle]

yep, same settings, same seed on all

tried 6 seeds now, every time it worked to some extend

with camera movements it seems to add to whatever you get from the image conditioning

dunno if we can replace it with tools available though

[sharp kestrel]

did you use a tool to do it?

[grim tangle]

just comfyUI

[sharp kestrel]

i'm scrolling up to see if you left more clues on how to plug it

[grim tangle]

like here the inital movement is a pan, but it adds the prompts to it

though the already panning doesn't seem to do much

[grim tangle]

the clip used: https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/open_clip_pytorch_model.bin

[tiny cradle]

wow!

[grim tangle]

I suspect we get proper controls for this soon 😄

[sharp kestrel]

probably, but it's fun to play with the toys before

thanks for sharing!

[grim tangle]

prompt weighting doesn't seem to affect it btw

[half hound]

Needs some touch ups but this is with 3 different video layers

[silent hinge]

impressive

who needs green screen haha

[half hound]

the load image input is actually a green sceen lol

[severe moon]

how long does it take with all the layering?

[half hound]

not that long suprisingly

[severe moon]

https://tenor.com/view/ok-all-nice-gif-22927671

[half hound]

can make an infinite amount of layers too

[left geyser]

thakn you guys!

[half hound]

well I am guessing there will probably be a limit based off your VRAM

[left geyser]

appreciate it so much

[bold wave]

Wow this is amazing!

[half hound]

4 mins for 3 layers on RTX 4090

[silent hinge]

4090? nice

until the 50 comes out anyway haha

[fossil atlas]

anyone know where CreateTextMask node expects me to put the fonts?

[open heron]

The girl ate the cup

[fossil atlas]

or what node pack it is from...

[silent hinge]

magic haha

[warm acorn]

what's the best node to interogate an input image (not based on metadata but something like clip interrogate)?

[autumn spindle]

I haven't found any great options, but the WAS suite has Blip Analyze Image https://github.com/WASasquatch/was-node-suite-comfyui

[autumn spindle]

You might be able to find some in here: https://comfyworkflows.com/custom-node-types

[warm acorn]

figured that was still the popular one 🙂

[autumn spindle]

There's also https://github.com/pythongosssss/ComfyUI-WD14-Tagger

[noble wolf]

where does lora go in the order?

nvm figured it out

[sterile mesa]

Where does it go

[noble wolf]

just plug it in right next to the checkpoint

[sterile mesa]

Makes sense thanks

[noble wolf]

gotchu

[sterile mesa]

Was messing with worfows last night on my laptop and realized i need to use a desktop setup. Larger monitor and a mouse goes a long way

Comfy ui is painful on laptop

[noble wolf]

yeah even just a mouse will save your life

being able to move around and zoom in on images with middle mouse without moving shit by accident

[sterile mesa]

Exactly. Haha. Touch screen aint much better since you need to not be in a node to zoom in/out

[autumn spindle]

There are a lot of under-documented tricks, like "hold space to pan" or "alt+click+drag to clone"

[noble wolf]

What they need is a damn undo button, ctrl + z or some shit. I know there's a plugin for that but it breaks shit in the newest update I think

but space bar is actually insane, thank you

so much

also pro tip if anyone ever gets a razer naga pro and the mouse wheel starts jumping, blow in the mouse wheel hole to fix it. Using compressed air for some reason doesn't frickin work

nah but with the new space bar trick + touchscreen, I could see a laptop actually being good for it now

ayo wtf

1.7gb lora!?!

[sterile mesa]

Lol which one? Ive only seen a few that large

Loads of older ones were like 800mb

[noble wolf]

jenna ortega

https://civitai.com/models/116197/jenna-ortega-sdxl

[sterile mesa]

Oh haha i saw that one. Yeah. Ridiculous even for a XL

[noble wolf]

welp it's already downloaded muaha

[sterile mesa]

An embedding might be easier. The 1.5 loras for her are decent and smaller too

[noble wolf]

Does HDD vs SSD affect speed of gens?

and yeah I have all the 1.5 ones for her

[sterile mesa]

Im sure it does

[noble wolf]

wanna try out the xl ones

Yeah I mean I'm assuming it does but if its like 5 seconds I don't really care

bout to plug in this 4gb hard drive I have with my new 4060 ti for 16gb of tasty vram

download all the models before they get taken down~

[robust scarab]

the people here are running on there personal pcs only need to wait if you want to use the online version

[teal harbor]

only first generation, once it's loaded in memory there's no difference

[noble wolf]

epic

Is there a way to quickly snip lines in comfy?

nvm, right click the origin and hit delete all links

[noble wolf]

THATS FUCKING AWESOME

[neat pier]

thanks 🙂

[silent hinge]

damn svd is becoming super good

imagine with topaz

[severe moon]

someone posted some upscaled stuff the other day using Topaz, I'm tempted to get it during their Black Friday sale but it still seems a bit pricey just for upscaling AI videos/images when they'll probably get higher res on their own in short time

[warm acorn]

I think it's pretty safe to say that all BLIP/interrogate tools available now...really have no clue what the input image is...i think it just randomly guesses a set of words from a wordbank, embarrassingly incorrect guesses

[inland swift]

topaz also shows very topaz artifacts. imo it was dated looking over a year ago.

[warm acorn]

couldn't be more wrong

[inland swift]

looks like a sample size of 1 to me

[inland swift]

does my sample size of 1 using the default a1 interrogator mean all of them work 100% of the time?

only part it missed i think is cyclops, but are cyclops not men?

[warm acorn]

dunno, will keep messing with it

[severe moon]

https://tenor.com/view/normal-men-innocent-men-cbbc-hacker-t-dog-gif-3085084613468942384

[silent hinge]

i love the story for normal men and dragon you are the man wow

[robust scarab]

that seems like waifu's tagger pixelart isnt as common in the dataset it was trained on if you try most normal photos or anime images it will be more accurate

[severe moon]

finally some turning gears!

[copper berry]

noice

[merry musk]

topaz $1,000,000 .... remarci foolhardy, 0$

[severe moon]

does remarci have something comparable to Topaz Video Ai?

[merry musk]

foolhardy remarci? its just an upscaler I want making a joke

But really, I can do 99% of what topaz does with inpainting, a lore here and there, about 5 iterations, and then an upscale and it looks flawless compared to Topartifacts

i have topaz, bought the whole bunchg for like 50$ with a student thing they had when i was back at college for a recert.

but rarely use it tbh

[severe moon]

their pay model was a little weird, they have a sub but also a one time 'perpetual' icense but you only get updates for a year....

[merry musk]

yeah they had the license for students, but the updates stopped last year i think.

i hate subscription models, they are garbage,

if i run it on your system with your resources, cool, if i run it on mine, just sell it to me and gtfo

[severe moon]

😉

[copper berry]

ok so as for now AMD GPU don't work with this error

on windows at least

idk on linux

[half hound]

Hey all, @grim tangle @severe moon @silent hinge Here's the multi layer video tut and jsons

https://www.youtube.com/watch?v=ID6IESHjKzE

https://jordain.ca/blog/stable-video-diffusion/

[silent hinge]

awesome! thank for sharing knowledge

[sterile mesa]

Ace. Thanks for sharing.

[prime pewter]

#1072220168534642768 hours flay

[icy valley]

nailed it

[silent hinge]

incredible it can do that 😮

[fossil atlas]

did you see the workflow @grim tangle posted earlier showing how to combine clip text prompt conditioning? i've been playing with it all evening and it seems to help for example in the case where you have a picture where it is raining but the raindrops don't fall down in the svd output. if you add "rain" etc in the text prompt, it seems to help it understand. might help for gears and mechanisms too?

[teal locust]

and you can make some parts of a body moving with the CLIP

[grim tangle]

I have one correction to it btw

I said at one point that prompt weights don't matter, I was absolutely wrong

it just didn't work in that certain seed, went back to the one that clearly worked and tried again, at 1 weight nothing, at 2 it worked, same seed

[fossil atlas]

do you have any thoughts about the three types of conditioning merging? average/combine/concat?

[grim tangle]

yeah tried them all, got nothing useful

tried timestepping too, which was... interesting 😅

[fossil atlas]

average doesnt seem to do anything at all, concat is very weak, combine is too powerful and pushes out the original image too much

[grim tangle]

yeah exactly

can't mess with the conds much, can add to it but changing the existing ones ends up in trash

can mask them though

and area cond works

don't really know the use... but this was two svd conds masked and combiend

mask was a sphere too, not the shape at all.. so it worked

[fossil atlas]

i'm not even sure what average is intended to do, because it seems like it just returns 100% of cond_to and ignores cond_from, based on the results

[grim tangle]

well these conds contain different stuff than we're used to, so those nodes probably just don't work with it

[frank wraith]

Is there some reference/help/tutorial anywhere that explains the parameters and what they do for created vids?

[fossil atlas]

it's just doing very simple array ops so the contents shouldn't matter... and if they're really different shapes then it would just crash

combine is literally just return to + from

[fossil atlas]

there are some reference vids posted in the history of this channel that are helpful

@grim tangle another thing is i managed to remove camera motion with negative prompt... but only for like the first 4 frames, then it started randomly moving around

the "image only" gen was one of those where it just pans across the original like it's a photograph

[grim tangle]

interesting, didn't get far with negatives, need to test more

concating just empty prompt changes the output too btw

[fossil atlas]

i got this result with combine

[severe moon]

Yeah , pretty rad! I’ll have to play with CLIP Later

[fossil atlas]

if you bracket the entire prompt and set weight to like 0.5, combine produces something a lot more reasonable

[golden oyster]

hello which is the best replicate link for 3-4 seconds long stable diffusion videos

[grim tangle]

guess rotation there wasn't feasible anyway

[fossil atlas]

i can see a bit of rotation in the last one

on the tree

[grim tangle]

it's like zoom + pan combined I think

using very low steps to test, could probably make it clearer hmmm

[fossil atlas]

technically what we're calling panning is dollying anyway 🙂

or, actually truck

[grim tangle]

I think they called it panning in the paper

[fossil atlas]

the model probably doesn't know the correct terms anyway if it's trained on random internet stuff rather than by a cinematography expert

personally i just want the camera to be still and have the things in the scene move in natural ways

but i guess that's a lot harder to model

does seem to happen occasionally when you get a lucky seed

[grim tangle]

refering to this, but it's for the loras

also for this to work, aug noise seems to need to be pretty high

that does look like camera panning

[fossil atlas]

for true panning, objects in the background should move faster than objects in the foreground (no parallax effect)

[grim tangle]

hmm yeah

[fossil atlas]

or, well, the same speed really

[grim tangle]

it likes to do those parallax effects often

[fossil atlas]

another thing i noticed is if you put "rotation" in negative prompt, it stops the subject from rotating too

i'm trying "camera rotation" now

[grim tangle]

should have mentioned, I have the other prompts as negative for each

[fossil atlas]

i saw

[grim tangle]

pumped them up to 2 now too

[iron topaz]

how do u ad prompts?

i cant conect the clip text encode

[fossil atlas]

advanced->loaders->load clip, then load this model with it: https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/open_clip_pytorch_model.bin

connect that to clip input on your text encoder, then merge the conditioning using one of the nodes that does that (under conditioning menu)

[iron topaz]

ok thanks i will try it!

[fossil atlas]

there's a workflow posted a few hours ago demonstrating it

but this is the basic idea

[iron topaz]

do u think it would alos be possible to use 3 image frames as imput? or would that requer to mutch work?

[fossil atlas]

no, you could do that and then merge the conditioning. however it probably won't do anything good

if you use concat or average it will just ignore all but the first. and if you use combine the output will be a horrible mess

[iron topaz]

hmm

i thoght that would be cool for longer vids. like you always take the last 3 frames and use that as imput

[fossil atlas]

but who knows. currently the only way to find out is to try it, because this stuff is less than a week old

[iron topaz]

true

i hope that we get controllnets and other stuff and finetuning

[fossil atlas]

putting the last frame back into the input is definitely possible but it is tricky to build a workflow that does it due to how the core of the software works. you could do it with chainloop nodepack, if you can figure out how it works

[iron topaz]

i will look into it. but i think for good results you need more then 1 frame.

so that the motions get good

[fossil atlas]

there's currently no sensible way to extract motions like that, that i know of

so all motions will change every 25 frames or w/e you generate

[iron topaz]

would probably make a funny vid xD

i think its intersting that promting works at all

[fossil atlas]

it barely works

and the effect on the result is rather random unless it's an incredibly simple concept like "rain"

and even then you have to have rain in the input image... prompt can only nudge it in the direction of making the raindrops actually fall like they should, if svd didn't manage to figure that out on its own

[iron topaz]

lol i conected a lora it didt crash bit it loos strage

is this the corect setup?

[fossil atlas]

yes, if you want to ignore the input image completely 🙂

[iron topaz]

and how can i not ignre the imput image xD

[fossil atlas]

use a conditioning merger, like conditioning concat

[iron topaz]

what is the node called

and where can i find it

[fossil atlas]

it's called Conditioning (Concat) in conditioning menu

you can also try combine instead

[iron topaz]

this is a cat ( no imput image)

you can see the cat movement🤔

like this ?

[weak bison]

Hey how do you create the XY plot?

[fossil atlas]

check the prompt conditioning workflow posted earlier. it also does x,y plot with different prompts

[silent hinge]

whats xy plot?

or what does it affects

[fossil atlas]

it's when you join together a bunch of different generations labeled with the parameters into one image or video

here's the link to it: #▶｜stable-video-diffusion message

[tidal pendant]

put a crappy photo of my friend t posing in it

[iron topaz]

when i try to use this it has missing nodes that cant be installed with the manager

[fossil atlas]

probably yeah. it uses a lot of custom nodes

apparently i already had them? i'm not sure what they are though

[iron topaz]

how can i find them?

[fossil atlas]

google search the names and look for github links

[weak bison]

Much appriciated!!

[iron topaz]

it says undefined

[fossil atlas]

in the error popup, it will say the missing names

load it again

[weak bison]

Comfy manager, install it, it will automatically populate them, then you'll have the option to download.

[fossil atlas]

comfy manager doesn't have everything though

it doesn't have my node pack, but people still send me github issues about it so they must be using it for some reason

[weak bison]

I've just installed 3 missing paks, everything works!

[iron topaz]

thats bad

[fossil atlas]

even though it's terrible and unfinished

[iron topaz]

i restarted it and now some nodes show up in the installer

strage

[fossil atlas]

yeah, you have to restart comfy a lot when you install things

it's even worse if you're trying to write a node pack... restart on every code change

[iron topaz]

but id didnt install jet

it showed nothing when i clicked install mising nodes

now it seems to work

i guess restarting solves all proplems xD

[noble wolf]

Aye, yall know if an 8gb card is good enough to train an sdxl lora model?

[fossil atlas]

i don't think it is enough, but this is the wrong channel to ask in...

[noble wolf]

apologies

is this better 🙂

[weak bison]

Sorry to bother, any idea?
Ran that custom workflow of your's -

[fossil atlas]

looks like a possible node name conflict?

svd nodes are in core now

[iron topaz]

it could work with nvidia system memory fallback but it will be slow

[noble wolf]

I got that error when I was using this node

It went away when I started using this one

[fossil atlas]

no idea.. things get messy when you have loads and loads of custom nodes

[weak bison]

Yeah, thank you.

[noble wolf]

lmao

[iron topaz]

have you selected a imput image

[weak bison]

Yeah, it's there, I've loaded it.

[iron topaz]

and teh video model ( it audo selects a random model)?

[noble wolf]

wait are you using an svd model?

[weak bison]

Damn, missed that thing, thought it was for image gen..
Fixed, you're my hero.

[noble wolf]

can you even use other models with svd?

[fossil atlas]

no

[iron topaz]

no but it auto selects other models and that maes errors

[fossil atlas]

yeah, that's the problem for sure

well spotted

[noble wolf]

ahh yeah I was looking at him using an sdxl model and I was so confused

[iron topaz]

i had teh same error xD

[weak bison]

Btw, this whole workflow it's quiet optimised runs around 1.4s/it
Thank you for sharing.

[iron topaz]

its not my workflow

[noble wolf]

what about it's optimized? does that custom node freeu_v2 do something?