🐝｜swarm-ui

[dim sonnet]

i've never seen it fail for any reason other than "the backend doesn't have the model file" so this'll be something new

[neon atlas]

thanks

[neon atlas]

fresh installed working all at default still have my old swormui

[neon atlas]

only fresh install works for me thanks

[balmy orchid]

Sorry I wasn't clear, I shouldn't have mentioned grids. I didn't mean using the arrow keys to move the scroll bars in the grid view window, that works fine.

When we click on the image main view window view, and that image goes fullscreen and greys out the rest of the interface, could we use the arrow keys to advance us to the next image?

[dim sonnet]

oo

[static fulcrum]

somenodes on stableswarm break compared to just using straight comfy

[worldly oxide]

I normally come to Discord to complain about a bug or an error, but today I just want to say thanks to the team, StableswarmUI has given my ageing 980ti 6GB card a new lease of life!

[regal imp]

Thanks, I was missing the CNTL+F5 bit. Are there SWARM docs somewhere?

[static fulcrum]

@dim sonnet https://github.com/ltdrdata/ComfyUI-Impact-Pack/issues/114

is it because stableswarm cant take in some of the facedetection /seg models

?

[umbral cypress]

A beautiful woman is looking lonely at the sunset outside the window in a dim room

[dim sonnet]

work-in-progress but yes https://github.com/Stability-AI/StableSwarmUI/tree/master/docs

[dim sonnet]

that is extremely unlikely to relate to swarm at all, i think you just selected the wrong thing when trying it in swarm vs comfy

[static fulcrum]

but i didnt change anything techincally..

[regal imp]

When loading the control-lora-recolor-basic_example.json workflow, the ColorCorrect node is missing and the ComfyUI Manager cannot find it. (the other three example workflows are fine 🙂 Fix anyone?

[neon atlas]

To see the GUI go to: http://0.0.0.0:8188
Error: OpenAI API key is invalid OpenAI features wont work for you
QualityOfLifeSuit_Omar92::NSP ready
got prompt
Failed to validate prompt for output 9:

• CheckpointLoaderSimple 4:
  • Value not in list: ckpt_name: 'Stable-diffusion\sdvn7Realartxl_beta2.safetensors' not in ['sdXL_v10RefinerVAEFix.safetensors', 'sd_xl_base_1.0.safetensors']
    Output will be ignored
    invalid prompt: {'type': 'prompt_outputs_failed_validation', 'message': 'Prompt outputs failed validation', 'details': '', 'extra_info': {}}

ruining confyui on 2 computers but getting this error

[dim sonnet]

looks like you don't have your models on the 2nd computer?

[neon atlas]

same setup for both all assets

[dim sonnet]

That is not correct. One of your comfy instances has the following models: ['sdXL_v10RefinerVAEFix.safetensors', 'sd_xl_base_1.0.safetensors']
Which doesn't include the one you're using, 'Stable-diffusion\sdvn7Realartxl_beta2.safetensors'
if the files are definitely there, you might need to restart the comfy instance, or adjust its settings.
A comfy self-start on the local machine will automatically point into swarm's model dir, but a remote via URL you have to manually align the model dir

[neon atlas]

thank you for the info

[static fulcrum]

im wondering if its just that swarm doesnt provide the directory option for some of the sam models, and thats why its failing?

[dim sonnet]

what

[static fulcrum]

im assuming the reason why my comfy workflow didnt work on swarm is because it requires sam models for face detection and segmentation, which have their own model folders to put in comfy...but with swarmui i cant point to those directories so it fails when it gets to the node

thats my assumptions

the current workflow im using in comfyUI which works.. but when i test it on swarmai it generates an image.. but fails at the face segmentation part

[dim sonnet]

no, swarm just doesn't care and will pass through the comfy value exactly how comfy expects it.

for unrecognized inputs it will literally just take raw data from comfy and pass it back how comfy had it

[static fulcrum]

but i dont get it though... if its literally running comfy exact values.. why does it break on swarm but not comfy... didnt change anything for it to break

[pliant crag]

Just used "Stable Diffusion Swarm UI App Icon as an Prompt" and created a MacOS app icon 😄

Another Version, with bees, though like the other one more

[neon atlas]

on second computer with amd rx 6800 xt 16 gb card using this args python main.py --directml --listen --dont-upcast-attention its ok without the Face Detailer

[static fulcrum]

thats kinda of the same issue i was getting also its something to do with the models inputs being wrong in swarm

[ruby glacier]

Hi Gui s I think that StableSwarm is a shit , it doesnt work for me , I am a Macintosh user since year 1998. My advice is to use Googles Tensorflow to unify the process ...... Some people dont have RTX so the future of Stable Diffusion is to make it more universal......not only for Windows users....... No hate at all, but make it more universal! So we can all take a profit of it ...... I am really tired to find a workaround for every little thing....

[ruby glacier]

TensorFlow and PyTorch are two of the most popular deep learning frameworks, and they both have their own strengths and weaknesses.

TensorFlow is generally considered to be better for production deployments, as it has a more mature ecosystem of tools and libraries. It is also more optimized for certain hardware platforms, such as GPUs and TPUs. However, TensorFlow can be more complex to learn and use, especially for beginners.

PyTorch is generally considered to be better for research, as it is more flexible and allows for more experimentation. It is also easier to learn and use, especially for beginners. However, PyTorch is not as well-optimized for production deployments, and it does not have as mature an ecosystem of tools and libraries as TensorFlow.

I like Python for his simplicity... but with Torch 2.0 we have some optimizations...... I dont understand but I want it to work on AMD Radeon,M3 , etc Everywere like Adobe Firefly...... You have to rewrite everything from scratch, to make it for everyone , I dont know how , but I dont like to doesnt work

[dim sonnet]

is this some weird meme? Wat?

StableSwarmUI has swappable backends, it is not inherently dependent on python/pytorch/etc. at all, it's up to the backend to choose things like that

if somebody made a TF backend swarm would easily and happily be able to link to it

the default comfy backend does work on AMD graphics cards and on Macs btw

[hearty notch]

what I am messing up here? if I want to change the location to my models and Checkpoints?

cuz I am getting this error too @dim sonnet

Hi btw, it's been a while I am away, hope that I am not missing out on too much fun

[hearty notch]

no hate? but you start the conversation with using the word 'shit'

[dim sonnet]

er, have you restarted since changing that?

[hearty notch]

yeah

[dim sonnet]

currently doesn't propagate updates fully til a restart

and the error: FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\SD_Checkpoints\\Stable-Diffusion'

doesn't match the screenshot

[hearty notch]

ah ok

like hard reset, i thought only CTRL+F5

[turbid veldt]

can stable swarm be used to break a single generation job up across multiple GPUs? My problem with SDXL is that it's quite slow (due to the high resolution 1024 x 1024). It usually takes 30 seconds - 60 seconds to generate 4 images (with refiner). Would it be possible to cut this time in half using two GPUs?

[ruby glacier]

The backend works, but SS don't generate images, there are bugs on Mac OS , but nevermind.... tnx for the answer 🙂

[dim sonnet]

the time to generate 4 images can be cut in half with two GPUs, yes

the time to generate one image cannot be

but as long as you want more than 1 image, swarm can do it faster

[ruby glacier]

I'm using Automatic1111's webui on my M2 MBP and it's working great but I wondered if there was any way to use the CoreML models that Apple released (e.g. https://huggingface.co/apple/coreml-stable-diffusion-v1-4/tree/main/original/compiled)? They're not in .ckpt format so I assume I can't just drop them in the /models folder.

[ruby glacier]

because I am experiencing bugs .... for example, the UI doesn't save the Server setting after a restart , very annoying issue , and lot more ..... why it is not tested on different OS.....?

[hearty notch]

Yeah, but you can not just say it’s shit! This is not polite to the effort from the developer

And honestly, this is one and only developer who is listening to the community feedback

So just be nice

That’s all

If you pay money for that UI and you didn’t get what you want, I agree with you, you can call it out

But getting something like this for free and then attacking the developer, it’s not nice .. at all

Everyone here reported issues, it was sorted out so quick, and again I am just reminding you that you are getting this for free!

Are we done here????

[ruby glacier]

I sorry about that, I know it is not easy to synchronize it on every browser and OS, but it not professional and waste of time , it is a alfa version , but it is the official SD ui ....

[hearty notch]

Are you paying any money to the developer!?

If the answer is no! Stop trolling and leave!

[dim sonnet]

it doesn't save what exactly?

[hearty notch]

I think this person is just trolling it’s very obvious

I am just reading what they are saying and it’s very incoherent

[ruby glacier]

No comment ! , Much respect for the developer, I don't think he is the only one, perhaps it is team .... but Why I have to pay money..., there are UIs like A1111 or Vlad made professionally and everything works fine.... I like the idea of SS to use different backends and multiple GPUs , but there is a lot to refine....

[hearty notch]

Then go use that UI!!! Why you are arguing!!!!??? If you have something positive to say! Say it! If not! Just f@&$ off!!

[dim sonnet]

anyone in here not following the Feature Announcement Thread really should go follow it -- important announce there:

css/format rework is in-progress, on latest commit things will look quite a bit differenty

[worldly oxide]

How could I add multiple LoRAs to that interface setup?

[dim sonnet]

click the loras tab, click all the loras you want

loras tab is at the bottom tabs, same as before, hasn't moved

[barren moat]

I'm stuck at 512x512 and can only change the aspect ratio. How do I change the image size?

[dim sonnet]

either (A) select a model that has the size you're wanting (ie select an SDXL model if you're wanting 1024) or (B) set the Aspect Ratio to Custom at which point you can see width/height freely wherever you want

[barren moat]

I figured it out. I had to choose “use comfy workflow” and then I was able to switch the image size. After that I clicked the “disable comfy workflow” to go back and the image size stayed at 1024x1024.

The adjustment sliders are missing from the new UI to adjust the image size.

[dim sonnet]

... What.

you just... set Aspect Ratio to custom

you don't have to do the rest of that

or select an SDXL model so it uses 1024

if that's not happening you have some form of bug

[barren moat]

My mistake, I'm an idiot lol I didn't know you have to set it to custom to use 1024x1024

[dim sonnet]

you don't, that's just how you customize it

it will use 1024 if you have an SDXL model selected

[barren moat]

as soon as I changed it to 1:1 it switched back to 512x512

[dim sonnet]

do you have a model that isn't SDXL selected

[barren moat]

no sir, I only have sdxl models loaded

[dim sonnet]

can you screenshot the Models menu showing what model you have selected

[dim sonnet]

click Models

[barren moat]

they're all sdxl models

[dim sonnet]

that button

Ah

metadata isn't loading?

[barren moat]

no, it was before but now it's not since the update

Maybe it's not loading because I'm using symlinks???

[dim sonnet]

uhh, maybe? That's an interesting possibility - symlinks should work

[dim sonnet]

are there any errors in the server console?

oh and to be clear: metadata is how it knows what model type it is, so that's the root cause of the resolution issue

[dim sonnet]

wait, what, .symlink files? tha'ts new

never seen windows render symlinks that way

[dim sonnet]

0.0

uh

are

are you okay

[barren moat]

I'm using an app called Symlink Creator

[dim sonnet]

is your computer okay

is that screenshot as cursed looking to you as it is to me

[barren moat]

yeah it's fine as far as I can tell

[dim sonnet]

l-

look at

the

the screenshot??

what happened??

[dim sonnet]

am i high or is that screenshot haunted

[barren moat]

there is errors for the meta data

[dim sonnet]

Oh. Huh.

Uh.

That's new.

[barren moat]

I can dump the symlinks and just change it in the server settings

[dim sonnet]

Can you

• stop the server
• delete model_metadata.ldb from the models folder
• restart

[barren moat]

ok

they showed up! but there are still some errors in the console

[dim sonnet]

:D

what errors

[barren moat]

there's more but it's pretty much the same for each model

maybe from the symlinks??

[dim sonnet]

oh it's still that

uh

you should probably either (A) just point the folder in settings or (B) make a symlink properly

I think the unusual symlink format you're using might not be handled properly when it's symlinking the database file

[barren moat]

sounds good I'll just point it to the correct location. I'm new with symlinks

Awesome thanks for the help!!

[dim sonnet]

for future reference if you need symlinks, the recommended/standard way is to open a command line and use mklink /J LinkFile TargetFile (and fill in the two filepaths)

specifically with folders, not individual model files

[barren moat]

ahhhh ok got it. I'll try that

Thank you

[barren moat]

I'm just getting errors now for the controlnet models. I have them loaded directly in swarmUI directory under:
D:\StableSwarmUI\Models\controlnet but still get the errors. Any suggestions?

[dim sonnet]

For whatever reason, the database engine (LiteDB) is having trouble reading the metadata storage files (model_metadata.ldb) from your model folders. I'm not quite sure why? But, uh, make sure there's nothing weird (like odd symlinks) and when in doubt just delete the file and restart

I should probably set it up to automatically nuke the files tbh

[barren moat]

Ok thanks, I’ll try that. Yeah I probably messed up something trying to connect everything with the smylinks app.

@dim sonnet Well hot damn! That did the trick!! Thank you, again!

[balmy orchid]

omg I had not opened the Grid Viewer Page link this entire time - It has all the features I need, it's all good. Is that in the tutorial readme somewhere? I was naively expecting it to be in the main UI somewhere, and overlooked it for weeks.

[ruby glacier]

... perhaps you are too stupid to understand what Constructive Critic is !!!!!!!!
How to say something positive, it does not work on Mac OS!!! Do you understand!?

you are a poor funny shit! * hahahaa

[dim sonnet]

can you not

[hearty notch]

Meh, it’s ok, just ignore it 🤭 it’s ok

Btw, @dim sonnet I was thinking if it maybe a good option to have some presets for the comfyUI, where people can just load the workflow directly from the generate page

Can adapt some basic workflow for different things

Like image to image or upscale or so

I am not so sure how that can be possible from a coding point of view tho

[dim sonnet]

there's a save/load workflows menu, and it wouldn't be difficult to add premade ones to it

I intend at some point to make it so whatever settings you have on the Generate page, you can click a button and be sent to the Workflow tab and see a workflow that does exactly the same thing

so eg getting an img2img workflow would be as easy as enabling an init image on the generate page, then click view workflow and boom there you go

[hearty notch]

Yeah, that will be great, and yet I am still keeping my eye peeled over the weight changing options using shift and arrow keys, I can see that active in comfyUI, and it will be a great addition tbh to see that in the generate tab

[barren moat]

Swarm needs an icon!

made with... swarm!

[barren moat]

Good morning!

Is there a way to have the UI load per-defined settings? For example, I always use a certain model, and 2 certain loras with a set amount of steps. Is there a json/config file I can edit to always load those setting by default?

[dim sonnet]

technically it has one, it's just bad

that was generated like... sdxl beta? as just "software interface icon" or something lol

[barren moat]

it's better than nothing

[dim sonnet]

i'm indecisive about what the replacement should be, leaning towards something involving bees or a beehive for consistent branding

maybe a beehive but purple (ie stability's brand color)?

[barren moat]

nice

[dim sonnet]

Yes: under the Presets tab, make one, and name it Default

anything configured there will be considered your user defaults

not sure if that applies Loras properly, loras are a bit weird. If it doesn't lemme know, can fix if so

[barren moat]

ok great! I'll try it now

[barren moat]

I can't get it to add in the loras

[dim sonnet]

oh woops i know why

also new favicon replacement is a purple robot bee

still weird but a lot better than the old placeholder

[dim sonnet]

The oddities there went deeper than I expected (due to lora being internally a multi-select) but it should work now!

Lora and lora weights are under a new part of the preset menu:

alongside the main selected Model and and some other oddities

oop, weights doesn't work

[dim sonnet]

fixed, works now

[barren moat]

Awesome! Thank you!

[barren moat]

Is anyone else having issues with image previews on the side bar and in the image generation screen?

T

hmmmm maybe it has something to do with the model??? It's only happening when I use a certin model with a crazy long name.

I changed the name of the model to something not as long and that fixed the problem

maybe it was because of the special characters that were throwing it off. The old name was:

0.75(3n1_SBC) + 0.25(juggernautXL_version2).safetensors

[dim sonnet]

ooo that's possible yeah, special characters do weird things sometimes

might need some escaping

[umbral siren]

Oh, can I not post? edit: I can

Hi everyone 🙂

I started Dockerizing Stable Swarm UI and believe I've hit my (very low) limit of knowledge of C#.

Dockerizing was straightforward so it's not like I've contributed a whole lot, but I'd like to learn! Here is my repo, prepping hopefully for a PR (and feel free to contribute):
https://github.com/surfingdoggo/StableSwarmUI

I guess posting code blocks is prohibited

Here's the branch comparison, probably the next best way to do this:
https://github.com/Stability-AI/StableSwarmUI/compare/master...surfingdoggo:StableSwarmUI:master

I'm going to start looking into troubleshooting C#, but if there's anything obvious to anyone, I'd appreciate it! I'm a DevOps/MLOps/Cloud/Python guy primarily. Open to any feedback as well, I'd love to get a working PR in on this

[dim sonnet]

That launch-docker.sh uses some... weird publish+copy hack, it should just do what launch-linux does instead

it also, uh, doesn't appear to ever even actually launch for that matter

the "running with docker" in the readme, i'm not sure why there's... so many different commands?

[umbral siren]

I believe I've hit my (very low) limit of knowledge of C#

[dim sonnet]

if the publish hack is to generate a container, the rest of the container stuff is redundant i believe

per https://devblogs.microsoft.com/dotnet/announcing-builtin-container-support-for-the-dotnet-sdk/ you can pretty much just publish and run in theory

[umbral siren]

Thanks! I'll give it a look and see what I can come up with

[umbral siren]

I cleaned up the Dockerfile following that guide and even ran it natively on the fresh cloud instance I'm working on. I'm also running the container on a local WSL vm. It builds and runs in all cases, but the browser doesn't do anything.

[dim sonnet]

it runs but you can't connect? or the browser self launch doesn't work (if the latter: yeah docker's gonna very much break that)

[umbral siren]

I can run other things in WSL locally, let me verify on this cloud VM

but, okay I kinda figured the self launch wouldn't work. I'm assuming that's intended behavior of the C# app?

[dim sonnet]

self launch is intended behavior of the normal app.
It not working is expected of docker because that's the type of thing docker was designed to prevent happening

[umbral siren]

You're talking about the browser tab opening automatically? I'd expect that to not work on a remote VM running Docker. I'm talking about anything working at all if I visit http://<ip addy>:7801/Install or on localhost.

What I was getting at earlier was where to begin troubleshooting. Setting log levels to debug isn't generating anything, either.

The docker container works, and I'm happy to open a PR or make any changes, but I'd love to see the app running inside of it with full funcionality, assuming that means digging into the C# code though.

[dim sonnet]

if
(A) the container is launching
(B) the SwarmUI instance is running and launches properly, and
(C) you are not able to connect to it
Then:
you have a docker issue, not a C# issue

but the question is, is it actually launching and running properly inside

if it is, you should see

15:44:58.969 [Init] Starting webserver on http://*:7801
15:44:58.996 [Init] Program is running.

in logs

[dim sonnet]

docker issue, or networking issue*

pay attention to network addresses

eg if you have it loading onto localhost instead of * or 0.0.0.0 that may be an issue

[umbral siren]

I was wondering about that, thank you a ton for confirming.

If you're around, it's now connecting and showing a blank page:
http://150.136.123.37:7801/Install
but I'll be shutting down this instance here in a bit

docker run --rm -it --gpus=all -p 7801:7801 stableswarmui StableSwarmUI.dll $@ --launch_mode none --host "*"

[wispy marsh]

Okay @dim sonnet I know that I'm always doing hacky stuff and should be ashamed of myself, but I am wondering. I only use A1111 for a couple of extensions, and SadTalker is the main one. I realized that it was running under the "bloat" of A1111, and didn't have to, so I used their conveniently included webui.bat shortcut to install a local version of it. Now I can start it locally and it does its thing. But because I am an inquisitive and adventurous young woman, I wondered what would happen if I used it as a backend in Stable Swarm. So I linked the folder to the webui.bat in the server section, and yep it started. . .well sort of. . .the log says: Self-start AutoWebUi on port 7822 is loading. . . and the interface has that ever popular pulsing green bar at the top and says some backends are ready but some are still loading. . . But here is the thing, If I look on http://127.0.0.1:7860/ where Gradio likes to start SadTalker, it is alive and well. . .well I'm assuming that it is well. . .I haven't yet generated anything with it. But my question for you oh guru of the swarmey stuff, is: By running it this way as a backend, am I gaining the multi-gpu benefit? And is there a way to make the pulsating message bar stop pulsating and messing me? I know that you have nothing better to do in the middle of you r programming than to answer my silly questions, but for some reason you do. . .Thanks for that.

[grand sleet]

how exactly do you use this?

[balmy orchid]

hmm -so you'd like it to split the sadtalker frame sequence across multiple gpu's? interesting - sounds technically feasible, but I'll let Alex weigh in.

[dim sonnet]

if you set it to run multiple copies of the backend (one per GPU) then yes it works as multi-GPU (when you interact through swarm, not the 7860 address). If it's forever "loading" that means either (A) something is failing when launching (--api has been known to be unreliable at times). or (B) you have multiple backends and one loaded but others are stuck.
idk what SadTalker is but if it's not self-activated for all gens, or integrated to swarm, it doesn't use swarm's splitting

[wispy marsh]

Thanks, I think I understand, surprised that you don't know what SadTalker is, I guess that you are too busy with your swarmy stuff to animate avatars. That causes me another question, so with Comfyui, are you saying that it only leverages the gulti GPU if I use it on the tab of the "Image Generation" GUI but if I use it at local: 7821, it isn't using multi GPUs? That seems strange to me as they both show up in the same command window. . . If that is the case, I will change the way I work because I often opt for the original since I don't really like you buttons and stuff getting in my way. . .

[dim sonnet]

correct, only the generate tab uses multiple gpus

[wispy marsh]

So are you saying that if I am using the Stable Swarm GUI but I'm in the Comfyui edit workflow tab, that if I hit "Queue Prompt" that it will not leverage the multi-GPU support? If that is the case, then I have a HUGE problem. Almost all of the work I do involves animation, which requires running batches and in most case starting from a node in the workflow. The generated tab doesn't offer these resources. Please clarify!!!

[balmy orchid]

Sadtalker was never meant to run in a "self-activated" way, it really wasn't written like that, it's basically just a cmdline script with a gradio interface. Sounds like you are already using Comfyui for animation (love to see that workflow)? What you'd want is someone to script up some Sadtalker Comfyui node(s), then you would get the StableSwarm multigpu features for free.

[wispy marsh]

Thanks for your thoughts. I think you and I have a different memory of the chronology of SadTalker. Since the first alpha commit of "opentalker,"(SadTalkers predecessor) in 2022, it was and remains a stand alone application. Vinthony, the head dev. has been super helpful to me throughout the many commits. It only recently was integrated into A1111. Many if not most users just it alone with the Gradio GUI. I runs just fine and dandy that way, even without the help of xformers, which it doesn't seen to be able to leverage in A1111, anyway. It is even now available for use on discord, for the more casual user. Unfortunately for the sake of this discussion, it wouldn't be helpful for me to get someone to "script up" some nodes, as they would require for Stable Swarm to run in batch mode. Which it doesn't. I would be tantamount to running any of my other custom workflow, that I "scripted up." I could generate lots and lots of the same frame over and over again, or I could generate one image, and then go back to the backend and advance all the animation nodes and the resave the workflow and generate another frame. With many of my animations being in the range 2000-6000 frames this would be rather time consuming. Batch generation just hasn't been a priority for @dim sonnet to this point. It is a shame, because leveraging my three GPUs on this machine and the two other on my api networked box would really be awesome in these huge renders.

[dim sonnet]

correct, using the comfy workflow tab does NOT split multiple GPUs. If you use "send to generate tab" and then generate there it does use multiple GPUs and otherwise executes exactly the same thing.

[dim sonnet]

how do you use ReVision? Just drag images into the prompt box and it applies ReVision. The Model parameter is only needed if you have an alternative ClipVision model, which, uh, i don't think anybody does

[dim sonnet]

anything in comfy can be ran with multiple GPUs, so if you get comfy nodes for it, that does work. The difference is which interface you use to start the generation (the Comfy node editor vs the Swarm generation manager)

[wispy marsh]

Really? MTB's Animation Builder and Fannolove's Loop chain both need to be started from within the workflow, with each you set the numbers of loops and frames and press queue. If you start the generation from the generation manager you will generate one loop and then stop. For those of us that do animation these are staples. So since you seem convinced that I'm dumb, please explain to me, like I'm a five year old, how one starts an 3000 frame animation from the generate tab. I have, all the nodes that I need to do what I need to do, they just can't be started from the generation tab. Incidentally, you can't start them in the node editor window via the queue Prompt Button. As a workaround, I've tried to replace these advanced nodes with primitives that advance the frame with each iteration, but these only work with a batch, not a series of individually queued generations. If this isn't clear to you, I can send you a capture video of what I'm talking about, or should I take your short, matter of fact answers as your telling my that you would rather I go away? You and your team have unlocked the frustration that so many of us have had in allowing image generation across multiple GPU's. No one doubts that you are a world class programmer, but there are many of us that actually use these applications, that might know a little about a little. I love that Stable swarm will generate ONE image much faster that I can generate it in stand alone compyui, and magnitudes faster than in A1111. But that ain't my gig, dude! I make video animations. If you can show me how to do that from the generate tab, then I will shut up.

[dim sonnet]

oh, if they're very specifically custom modifications to the Comfy UI, that can't be used, they'd have to make a Swarm version.

that's re you can't start them in the node editor window via the queue Prompt Button indicating that it's not actually just a comfy node, but rather a custom interface hijacking into the comfy interface

Anything that is a comfy node can be used via swarm, but things that are modifications to the comfy interface not necessarily

Animation software would likely in general need to be ported to swarm to work properly

Stable swarm will generate ONE image much faster that I can generate it in stand alone compyui that, uh, isn't the case? 0.o

Swarm can't generate one image faster - it can generate many images faster

the core concept is simple: if you have 2 GPUs, and generate 2 images, each GPU will generate one image. Or if you have 3 GPUs and generate 10 images, each GPU will generate 3 and 1 will take the last.

It can't split a singular image across several GPUs, as SD is a linear process

A properly integrated animation tool in swarm, when you have multiple GPUs, would be way faster than a single GPU, as the purpose there is massive batches, which is exactly what swarm does best

(if any relevant developers are interested in building swarm integration for that, i'd be happy to help them do it)

[dim sonnet]

It's very specifically the + symbol, because MS webtech magically double-parses that symbol in particular. For some reason. It's fixed now.

[dim sonnet]

Sorry for the delay - this is a feature now! In the main generate tab, both in normal view and fullscreen preview, you can use the arrow keys to move between different images in the current batch

[grand sleet]

drag?? thanks alot 👍

[wispy marsh]

Thanks for your reply, and also on the github issue. I am glad that you are seeing the difficulty that I am having, and it is clear that you can see the potential if we could generate these animations within the generate tab. Ha Ha, I don't know what qualifies someone as a "relevant developer," but I am on a team that specifically nodes for animation, similar to Deforum, for ComfyUi. Perhaps you would be willing to consult with us on how to make these changes.

[balmy orchid]

By "self-activated" I was using Alex's terminology. I meant "meant to be run as a node". Both the Gradio and the A1111 versions act as standalone operations, as you noted. I also need a multi-gpu capable sadtalker for an upcoming project of mine. I was planning on doing this the hard way, by creating my own app until I heard of StableSwarmUI, which got me into playing with ComfyUI. A1111 days feel numbered to me. I've not used Sadtalker for as long as you, but I have tried out several very similar open-source projects that do the same thing (get 3MM poses from frame sequences and apply FOMM and or a Audio2Lip solution to each frame and then concatenate back to video). I didn't mean to suggest "scripting up" a comfy implementation as if its an easy thing to do - I was actually hoping someone else would think it was simple so I don't end up having to do it myself! This is well within my skillset, but I wanted to avoid having to take the time to figure out the Comfyui API and reverse-engineer Sadtalker, as it is really just a step in my end goal (as I imagine it is for you). But I will if nobody else does it!

[wispy marsh]

This is "near and dear to my heart." I basically stopped using A1111, because of its limitations and discovered comfyui. I mainly migrated in an effort to develop workflows and custom nodes to implement deforum like animations within comfyui. I have been fortunate to be able to work with some great DEV's to accomplish this, and although in infancy, these nodes are rapidly coming to fruition. (see the recent animation offerings from CompyRoll.) Likewise the potential for SadTalker and/or Wav2Lip like nodes is HUGE! but will take time to develop. But if, and when, these nodes become available, and the workflows are devised, leveraging Stable Swarm would change the entire landscape. Since I generate many many SadTalker videos for my YouTube channel, and for outside work, this is a priority for me. My hope is that soon we will be able to work with @dim sonnet to animation controlers working thrugh the Swarm front end. I am talking with others about how to move forward, but since these types of nodes use javascript, there are programing challenges. Watch this space. . .

[dim sonnet]

couple big updates today: Parameter config tab https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-6905973

and grid gen searching/tab-complete https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-6907828

[barren moat]

Awesome!!

[sharp ginkgo]

Is there some way I can save my images in comfy such that Swarm can open them with the grid page? Or a documented format I could attempt to replicate with a node?

[dim sonnet]

huh? i don't understand what you're asking

[sharp ginkgo]

Currently I have to generate images in StableSwarmUI to be able to use the grid page. I’m hoping to keep using the existing comfy, with a custom save node such that I can open the folder in swarmUI to make the grids

[dim sonnet]

"the grid page" as in the Grid Generator tool? or the Generate tab batch view? Or Image History?

[sharp ginkgo]

Pretty sure the grid generator tool

[dim sonnet]

either way you can load custom comfy workflows to the main tab by just clicking the big purple button at the top left of the workflow editor tab

if you want a grid of different workflows, you can save them (with the save button at top left) and then use the Comfy Workflow grid axis

these buttons

[sharp ginkgo]

Hmm, I’ll give it a try later.
One more question…
I have a node that generates conditionings with built-in functions for modifying the embeddings(“A scale(cat|1.2)” - generates a cond where cat is scaled by x1.2) is there someway I can register “scale-cat” as a grid parameter, and then return the value from my node(or set some metadata) for swarm to associate it with the outputted image? Again, for building xy grids

[dim sonnet]

If you use "Use This Workflow in Generate Tab" the grid axis list will automatically update to list all inputs to all your nodes and let you grid any of them

also for prompt related things you can use the Prompt Replace grid axis to replace specific parts freely

[sharp ginkgo]

I’ll do some testing in a bit, but I’m not sure thats exactly what I’m looking for. Another example function I have is setDim(cat|180,0.065) that sets the value of dimension 180 in the embedding for “cat” to 0.065, and ideally I’d like to be able to build a grid where I can see all the gens where I modified dimension 180. For example, x axis would be seed, and y axis would be dimension 180s value to try and visualize the impact of that dimension at certain ranges. So in theory.. the node would need a way to set some metadata saying “cat-dim-180” is “0.065” or other arbitrary key-value pairs

[dim sonnet]

if you have a node with inputs, you can grid that input, it just works

[dim sonnet]

i published a convenient preset file of the official SDXL presets, in swarm-ready format with icons and all pre-included https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-6920544

[nimble zephyr]

This is so cute well done and thank you!

[dim sonnet]

i wrote release notes for the 0.5.4 release https://github.com/Stability-AI/StableSwarmUI/releases/tag/0.5.4-Alpha ... it's a lot, lol

[nimble zephyr]

Hi everyone! New on here. I am strugling to get my to access swarm on my LAN. Bassically I want to do is sit on my lap top in front of the tv and be able to access my desktop with swarm.
With Automatic and Comfi all I neede to do was add the --listen arg to my .bat files.
Ive tried adding it to the Swarm bat file.
I've tried adding "-- listen" as an extra argument on the "Backends tab too.
That is the main this I am trying to do right now.

I would also love to set up the multiple gpu's usage in my house but I am not sure if I need to install swarm on all the other computers in my house too ?
Although th git doc is well written, all the network jargon goes over my head. I think I need an idiots guide Not sure if step by step tutorials exists anywhere yet ? Thank you !!! https://github.com/Stability-AI/StableSwarmUI/blob/master/docs/Basic Usage.md

[dim sonnet]

to make Swarm accessible across LAN, you can add --host 0.0.0.0, that's the equivalent of what other UIs (inaccurately) call --listen

To use more backends across LAN, you need backends on your LAN - the standard/recommended backend is ComfyUI, so you need ComfyUI running on those other machines

you can then add them as backends via the Server->Backends tab, just click to add "ComfyUI API By URL" and add the LAN URL for those other instances

yarr there's a doc about it https://github.com/Stability-AI/StableSwarmUI/blob/master/docs/Advanced Usage.md

[nimble zephyr]

Cool thanks @dim sonnet ! Ill give it a try again in the AM thank you!

[dim sonnet]

nother feature announcement: https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-6924557

ps bonus fun fact, i built that tokenization code a while ago, and when i compared it to openclip's python code it executed orders of magnitude faster than openclip did, with otherwise equivalent results. Which is neat. (it's fast either way tho, not hard to tokenize)

[grand sleet]

ive encountered some issues, is there a simple way to fix this?

[dim sonnet]

oh, oops, my bad, i didn't commit the tokenset file properly - fixed now

[rapid ruin]

i just downlaoded stableswarm UI. Did I do something wrong? or is the tokenset file not fixed yet? (i'm new to this... sorry if it's a silly question)

[barren moat]

Seems to be working over here. Run the update.bat file and that should fix it

[rapid ruin]

Thanks a lot! fixed now

[dim sonnet]

more feature announcements because i can't sleep so im working til i collapse https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-6932449

(i'm at ~40 hours of baddecisionmaking now)

[wispy marsh]

I'm no sure but I think this ia an answer to prayer!

[barren moat]

That's a great feature!

[dim sonnet]

lmao.
So, first of all, refer to this user who actually looked at it rather than spreading fearmongering:

the watermark used in the upstream repos is literally just a yes/no type of thing that says "yes this image came from stable diffusion", as a way of slightly more reliably identifying than vs. the AI detection thingies out there that try to guess

definitely no IP or anything else lol

and, uh, whether it's present or not in Swarm entirely depends on what backend you use, swarm doesn't bother with any of that. I don't think comfy uses that at all

[barren moat]

Oh crap! No, I definitely do not want to start spreading fear mongering. Maybe I should just delete the question? I was just curious.

[dim sonnet]

yeah no worries on you, this is the danger of fearmongering, it works by taking advantage of perfectly reasonable people. Those posts look genuinely scary if you don't know better

[gray geyser]

what api do you guys reccomend

for img2img

inpainting

but the painting happens on my own site

[nova sparrow]

colab is working?

[dim sonnet]

should be

[nova sparrow]

i have errors like this

and module torch not found

[dim sonnet]

those aren't real errors, that's just cloudflare being dumb

[nova sparrow]

oh

okey

[dim sonnet]

idk why cloudflare struggles so bad to establish a connection while it's launching

[dim sonnet]

oop the torch not found part was a real bug though, fixed now

[nova sparrow]

a so that's why it was working, I thought it was an unimportant error haha, thanks for the attention.

oh

again

[dim sonnet]

make sure you fully disconnect and close any runtime and load it up fresh and execute the commands in order (minus the first one)

the fix was specifically an extra line in the code that inits the drive folder option

(more specifically it deletes data that colab breaks and stores as broken in your drive for some reason)

[nova sparrow]

okey

i gonna try again

nop

when i installing, that error happens

[nova sparrow]

it finally worked, thank you.

[nimble zephyr]

Hi. @dim sonnet or if anyone knows. I have managed to get swarm working on my network thanks to @dim sonnet Now I have a new chalenge my for some reason my Swarm doesnt seem to have an interface on my server (main) PC .
Its working well on my laptop (host ? ) Macbook but it seems that I can only acess the COMFY interface.
The GENERATE , COMFY WORKFLOW EDITOR , USER, SERVER tabs are missing as is the yellow WORKFLOW , SAVE,LOAD tab.
Ive zoomed in and out and set the interface to 100%. I can however see the comfy manager and use it.
Anyone know how to access the swarm tabs ,when this happens ?

[dim sonnet]

that sounds like you opened the wrong port?

default swarm port is 7801

default backend port is 7821

[nimble zephyr]

Ahh lol so easy!! thank you

[sterile gazelle]

I keep getting this error while trying to make a tutorial video on stableswarm, how to fix this? "Some backends have errored on the server. Check the server logs for details" (I tried it on the colab, comfyui doesnt load on it and generally cannot generate with loaded model)

[dim sonnet]

as it says in the message, check the server log

[sterile gazelle]

I did.

and here it is

its in the colab version

ComfyUI Self-Starting

[sterile gazelle]

whats next?

[dim sonnet]

i hate colab so much

[dim sonnet]

I pushed a fix - close the colab and reopen, should work now

(google colab broke more things again, not even swarm's fault)

[sterile gazelle]

what was the work around? I guess thats not unusual, but it is a good trial at least to get familiar.

Thanks

[dim sonnet]

disabled venv, as colab just... breaks venvs now i guess

[sterile gazelle]

thanks

also quick question, does swarmui consider text:1.5 formated weight assignments on the prompt section?

[dim sonnet]

Depends on backend you're using, but both the default comfy and the optional auto backend if you for some reason want that, support (text:1.5) format

idk if stabilityapi does offhand, i don't think so

[sterile gazelle]

yeah I use everything default from the colab because I make a video demo as an intro guide, so I had to be accurate, if it does then we are good.

[sterile gazelle]

by the way how to get preprocessors installed? (and where to place them)

[dim sonnet]

https://github.com/Fannovel16/comfyui_controlnet_aux are the main ones

[sterile gazelle]

ah yeah from the official comfyui colab, how do we install them inside the swarm setup though? will installing them inside comfyui's backend folder be enough?

[sterile gazelle]

also where can we do inpainting/outpainting? as part of the tutorial will include that as well

[lean coyote]

this run on AMD?

[dim sonnet]

Interface for that is not yet built

I'm wanting to do it right not just slap together a bare minimum like auto's img2img tab, so it'll take time

[dim sonnet]

Modern cards: yeah should work fine
Older AMD cards: yes but you might need to do some manual pip install stuff to make it work

[dim sonnet]

rip this question/entire topic -- google colab banned any/all usage of SD today :(

[sterile gazelle]

I just noticed, that is disappointing since I just composed a 1 hour video two times which is now partially rendered obsolete, it makes sense due to the ammount of people delving into it and overusing its resources but then again no idea how to now use it in an alternative way, their paid plans seem inferior to others and I do not prefer it as a paid option for now.
on that note yes I noticed it crashing while the instance is active.

[dim sonnet]

no they banned all usage of SD on colab - the paid plans are included in the ban

options now are (A) run on your own PC or (B) use a rented cloud server (runpod/vastai/etc)

[sterile gazelle]

well too bad I got the last batch of back in the day for a premium gtx980 so thats a nono unless I wait a couple of years to render lol

[sterile gazelle]

there is also kaggle o_O

[sterile gazelle]

comfyui by default seems to not crash it for now at least.

[wheat sonnet]

Are you sure, I've been running Diffusers scripts on the free tier today, yesterday and the day before.

[dim sonnet]

they might've missed Diffusers? (Considering it's different from other SD)? But people have been saying all the different SD options are breaking, and even some other AI tools like RVC -- https://www.reddit.com/r/StableDiffusion/comments/16eyr8x/google_seems_to_be_intentionally_restricting/ https://www.reddit.com/r/StableDiffusion/comments/16f6g5i/alternatives_to_google_colab_for_running/

that or reddit's wrong lol idk?

i just tried the swarm colab and it's working, at least for now

[lean coyote]

any idea how to run this or automatic111 web.ui on cpu and not gpu?

[dim sonnet]

comfyui can run on cpu (and swarm can use that as a backend) but it will be very very slow

in swarm just go to Server tab and edit the comfyui self-start backend, add to the extraArgs slot: --cpu

[lean coyote]

comfy ui doesnt support windows till later stil year right? im using web.ui with automatic1111 and didnt know if there was a control option to use cpu rather, i tried one of the cpu settings, but it would only give me 1-2 actual pictures then rest of frames were static black and white dots

@dim sonnet really appreciate your help

but it was super quick

[dim sonnet]

huh? comfyui works great on windows

[lean coyote]

it can make videos too?

[lean coyote]

.

[lean coyote]

is there anyway to partially use gpu with cpu like split load?

[sterile gazelle]

seems like google stopped the colab support for guis mostly

[wheat sonnet]

I kinda understand, it not really what colabs are for, especially the free tier
Hopefully the GUI's were not doing anything really stupid like having a process continuely running to keep the instance attached.

[dim sonnet]

Feature update that i'm very happy about: lora/controlnet list now clearly separates models that are compatible or not with your current model
https://github.com/Stability-AI/StableSwarmUI/discussions/11#discussioncomment-7022809

[nimble zephyr]

Hi Guys and girls . I am getting the following error in my launch 12:52:16.645 [Error] [WebAPI] Error handling API request '/API/ListBackends': Request input has unknown session id. I also get this message "Some backends are ready, but others are still loading..." on my swarm UI.
I have refrenced Automatic 11111 by loading backend: (1): Auto111 SD-WebUI Self-Starting
C:\stable-diffusion-webui\webui.bat
My args are :--api --reinstall-xformers --medvram-sdxl --theme dark --disable-nan-check
I can connect to comfy and Swarm UI on remote macbook via my LAN but can not connect to Automatic on my server using URL on remote lap top.
I saw @dim sonnet note abour New backend type: Swarm-API-Backend. where do I put that ?

thank you!

[dim sonnet]

• unknown session id just means you restarted the server without reloading the webpage, it's fine
• the backends still loading message ... literally explains itself very clearly?
• if something is actually broken please specify precisely what you're trying to do, what's broken about it, and the full logs
• the new swarm api backend is ... a backend type option, under the backends list?

[nimble zephyr]

Ok fair enough. So I want to be able to access automatic on my remote laptop. I can access swarm and comfy via the URL but am not able to figure out how to get to see my automatic UI on my remote computet. I think have used the correct URL. So the reason I posted about the "backends still loading message" was that I though it might be ralted as it never goes away. Shall I post the full log?

Should I redact my IP ?

[dim sonnet]

if you're trying to direct access the ui itself for auto through a remote connection you'll need --listen

[nimble zephyr]

Ok Ill add that to the arg. Thank you

It worked thank you Alex.

[barren moat]

Hey Alex with that new update it's tell me the Stability Offset Lora isn't compatible with quite a few of my SDXL models?? Is that possibly a bug or am I screwing something up which is probably the case. 😂

[gray geyser]

I came here to ask about an issue I was having then I see that same quest from@barren moat

err **question.

[dim sonnet]

oh, I forgot to account for, uh - are you sure that's an SDXL 1.0 model?

SDXL 0.9 models are considered different and I forgot to account for that

(the interface will happily let you ignore the incompat and use it anyway if you want btw)

but yeah i need to set it up to recognize arch intercompatibility

[dim sonnet]

Done, there's now an intercompat listing that will let it understand that an SDXL 0.9 model can use SDXL 1.0 loras

[gray geyser]

@dim sonnet thanks. yea it was ones listed as 0.9 vs 1.0 models. that I was having issues with.. ( funny thing is that all the 0.9 models I have was listed on civitai as 1.0 models not 0.9. )

[dim sonnet]

any models that are in SDXL architecture format, but spec-incompliant, will be assumed to be SDXL 0.9. Being spec-incompliant might mean they trained the model on outdated software, or that they went out of their way to strip core metadata.

You can edit the model's metadata within swarm if you want to fix stuff like that

just click the lil

set architecture to stable-diffusion-xl-v1-base and set all the other values however you like

[gray geyser]

nice, thanks for the info on that.

[barren moat]

It’s listed as a 1.0 model on CivitAI, is there anyway to tell for sure?

Never mind, I see the info above. Thanks for all your work Alex

[hearty notch]

btw, I love the new option to load CUI workflow directly form the gen tab, however, it's always giving an error msg

[dim sonnet]

what error?

[hearty notch]

hi @dim sonnet

Invalid data: Unknown param type request 'comfyrawworkflowinputmodelcheckpointloadersimplenodeckptnamec'

this is what I am getting when I activate any workflow,

[dim sonnet]

ooh, i see why, fixed

[hearty notch]

Btw, controlnet is dope!!!

However, I am thinking on the possibility to have the same preview we are getting on the classic webui controlnet as it’s helpful to understand what the model is annotating

Not sure if this is something that can be done

And I will always bug you over and over about the tokens weight control by using shift and arrows on the keyboard😝😝

[boreal lintel]

hi @dim sonnet, can we use ControlNet with the StabilityAPI? doesn't seem to work for me,

[dim sonnet]

no, stabilityapi doesn't support controlnet

[dim sonnet]

definitely possible ye. I'd probably want to put the preview somewhere separate rather than dumping into the output tho. Will need to think about how to organize that best

(if you want to see raw outputs in the meantime and you're comfortable (heh) with comfyui, you can do a custom workflow and just link a preview node up to the preproc)

[hearty notch]

Yes I totally agree there is no need to have it with the output, could be (and that just my rough idea) to have it on a separate tab, like the one you added recently to for the token count, or even in tools next to batch and grid

As this is an option that can be used to have a starting point only

[dim sonnet]

that suggestion made me realize i could just

yeet a lil button

not the prettiest thing, but, uhm, fixing design aesthetic is nitrosocke's job not mine >.> <.<

[hearty notch]

You guys are awesome, and really the UI is blooming so beautifully slowly

Btw, are we going to see some API soon where we can connect it to softwares like revit, or sketchup or any other software are that more on the architecture design side of the world?

[dim sonnet]

Swarm has a pretty clean API already, just haven't fully documented it or pushed for its general usage

i've used it in a personal project already: https://github.com/mcmonkey4eva/SimpleDiscordAIBot

(Discord bot that's mainly a Language model runner, but it can connect to swarm to have the LLM help generate images)

it's pretty cool

connection to other specific software tools is gonna depend on, er, well, devs that use those tools building integrations there

[dim sonnet]

the sins i must commit to account for different ways people might select text...

but, anyway, added as Control+Up/Down arrow

[hearty notch]

Well, I have always many ideas to share, and really been enjoying my time as a user for swarmui, can’t think of it as a beta user but it’s more of a sharing experience and pushing it 😝

[dim sonnet]

what would i do without github copilot running while i'm trying to write docs

(i need to find a way to auto-disable copilot in .md files before it gives me a stroke lol)

oo there's a setting for it

[hearty notch]

I guess soon we will be able to see some friendly bots around the house nagging us for doing chores

I will have a look at this for sure, but the way I see sdxl is working it’s more about communicating simple words of what you want or don’t want to see

While others are using these long paragraphs of prompts thinking it will tweak the results, I find it personally more about keeping it simple,

[dim sonnet]

er, shouldn't post nsfw images here

[hearty notch]

Esh! Totally slipped my mind, let me censor it

[dim sonnet]

https://github.com/Stability-AI/StableSwarmUI/discussions/11 posted a feature announcement about prompt syntax support

[hearty notch]

Was always wondering if there is a way to randomize presets on the main prompt without the need to click on them from the main UI, is there any way around it?

[dim sonnet]

oo

[hearty notch]

is this normal?

and where these previews are saved? incase I wanna delete them to avoid cluttering

[barren moat]

I’ve been waiting for this! This is fantastic!!

[dim sonnet]

not saved at all, in-memory

and ... how did you get three previews 0.o

[dim sonnet]

a cat <preset:<random:sdxlofficial/photographic, sdxlofficial/fantasy art, sdxlofficial/anime, sdxlofficial/neonpunk>>

slightly hacky but you can use prompts to muck with presets, meaning you can apply random to it

[hearty notch]

looks like i hit preview three times,

and the ui registers every click

[dim sonnet]

oh you just clicked very quickly?

well

uhm

don't? :)

.>

[hearty notch]

a cat <preset:<random:sdxlofficial/photographic, sdxlofficial/fantasy art, sdxlofficial/anime, sdxlofficial/neonpunk>> <random:sdxlofficial/photographic, sdxlofficial/fantasy art, sdxlofficial/anime, sdxlofficial/neonpunk>> so I guess using this prompt will randomzie 2 presets for every gen?

[dim sonnet]

• okay it's actually fixed to not do that now

[dim sonnet]

you'd have to do <preset:..> <preset:...> to use two

[hearty notch]

ah yeah this is what I did

portrait of a man, shaved head, black eyes, wearing leather shirt <preset:<random:Fantasy Illustration, Nightmare Phonography, CyberNoir, Anthopsycho, Drippy Dark, Coupage, Surrypaint, Pholiograph, Mechparth, Basicg, OldHunter, RubberCup>> <preset:<random:Fantasy Illustration, Nightmare Phonography, CyberNoir, Anthopsycho, Drippy Dark, Coupage, Surrypaint, Pholiograph, Mechparth, Basicg, OldHunter, RubberCup>>

these are the names of my presets

but it seems that it's not randomizing

I am just getting the sanme over and over

oh wait you just updated the UI? let me try to update and try again

[dim sonnet]

yes that was added when i sent the screenshot of it

[hearty notch]

now this is dope it's more room to play with, really cant wait to see what other controlnet will be released soon

[dim sonnet]

(also, protip: because presets can modify any param, this can be used to randomize any param, as long as you're willing to spend a minute building presets of params you want to play with)

[hearty notch]

oh yeah this is what I usually do, but I keep some of them clean as I am using with controlnet heavily

this is awesome for real

[neon atlas]

how can i fix this pls

[neon atlas]

some screenshots

[grand sleet]

bro tryin to scam us 💀

[dim sonnet]

I'm... not sure how you managed to get a null parameter value? 0.o but i pushed an update that will give a warning rather than crash when that happens

[neon atlas]

thank you for your help

[dim sonnet]

DynamicThresholding is now supported natively out-of-the-box in swarm :D

[hearty notch]

just incase

[dim sonnet]

?

[hearty notch]

it took me few minuts to find it haha

[dim sonnet]

oh

yeah i forgot to mention where it was lol sorry

[hearty notch]

btw, which sampler you recommend using with SwarmUI, and are we going to see a dedicated upscaling tab? the refiner upscale is doing a good-ish job, this is not a rant, but I understand that there is more power to pull out of SDXL when used for upscalling normal images

[dim sonnet]

Sampler: it defaults to Euler+Normal and that's generally a very safe bet with SDXL. For SDv1 I prefer DPM++ 2M Karras but it's not a huge difference.
Upscaling tab: probably something like that as part of the image editing update whenever the time comes to push that

upscaling using Stable Diffusion is a bit of an intense task btw. VRAM & Time grow exponentially as image size grows

[hearty notch]

i have 24 4090, I think I will be safe playing around with it

[rare crag]

technically they only grow quadratically 🤓

and tiling upscale can be done in constant VRAM, though that would really need tiling controlnet which I don't think SDXL has yet

[stable vine]

Is there any doc about using multiple GPUs? Is it possible to utilise multiple GPUs for rendering Deforum animations via stableswarm-ui?

[hearty notch]

mulit-gpu is rendering 1 img per backend, I do not think there is a way to do what you are asking for

[hearty notch]

yep, exactly

[hearty notch]

crossing fingers since I saw the raw power of ControlNet LoRa from stability

[dim sonnet]

not quite correct - there's multiple sources of exponentiation that compile together, the raw pixel-size growth is just the first factor

attention layers have their own exponential growths, and that's before talking about hardware limits

(also obviously in non-precise-math-speak exponential is a broad term that includes quadratic and anything like it)

[hearty notch]

damn I am feeling dezzy reading this haha

[dim sonnet]

Technically something like that* is possible, though there isn't a dedicated tool for it and it would really need one

* Deforum animations in the raw basic form are explicitly sequential, each frame is derived from the last, which is very difficult to split between GPUs due to being sequential.
Other variants of animating that way aren't necessarily sequential and are easier to split, eg:

• Video2Video, where you have an init video and each frame is separately img2img'd. That can be done in swarm via the Image Edit Batcher tool.
• Separated-scene video generation in parallel: you could eg generate key frames 5 seconds apart for a longer video, and then generate the frames in between the 5 second points - the keyframes have to run sequentially, but then the frames in-between exist in multiple discrete 'scenes' where each scene can run in parallel (this needs a tool for it built in swarm to function)

• Or you can use multi-GPU techniques for a quality boost even with sequential methods, eg have 4 GPUs each generate a different seeded frame and have an aesthetic scorer pick the best frame of the 4 to keep and discard the rest (this would also need a tool built to work)

[hearty notch]

so are there any method to pool the VRAM GPU?

that will be a good way moving forward with upscale maybe?

[dim sonnet]

only ones with horrific performance penalties

for upscaling, tiling approaches can split GPUs in theory

though as genrtose mentioned that'd work a lot better with a controlnet for it

[hearty notch]

what kind of penalties we are talking about?

[hearty notch]

the controlnet tile was a decent one, looking forward to see it with SDXL

[silver osprey]

any idea how to get this to do batch size?

I got my AIT workflow ported, but batch size isn't working

like, the option for batch size is gone

[dim sonnet]

generally Images count is preferred over batch size, but, batch size was added a week or two ago, just update and it should work

i've never tested ... opera? is that? , some of that ui looks a lil off

especially the prompt boxes 0.o

will have to add opera to testing i guess

[silver osprey]

I just updated it, no option for batch size

[dim sonnet]

disable & reimport the workflow

the default batch size opt is under Advanced

[silver osprey]

plus, even if there was such option, will live preview show the batch like A1111 used to do? or will it be like live preview on ComfyUI

[dim sonnet]

it just displays whatever previews comfy outputs

[silver osprey]

Comfy always shows 1 out of the 4, so I guess that answers my question

[silver osprey]

still not there =\

[dim sonnet]

does it appear without a workflow?

[silver osprey]

@dim sonnet am I missing something here?

[dim sonnet]

oo

oo i see why

old code to remove batch size was still present

[silver osprey]

I did manage to make a workaround for this by using a node I made for batch size 1 to x, but again, live preview is only for the top left image

is that a limitation caused by ComfyUI?

[dim sonnet]

Prooobably? can't right now but i'll do some testing later today and see if that's something that can be bypassed

[silver osprey]

also without this UI, the live preview from ComfyUI ALWAYS shows only the top left image of the batch

but when it outputs it shows all 4, so that's fine.. but it would be cool to have the live preview show the entire preview of the batch

if this would have batch size and show the entire batch's live preview, I'll use this all the damn time, but well- it doesn't =[

[dim sonnet]

batch size itself should work if you re-update

previews, well, we'll see?

it can do multiple live previews if you use Images rather than Batch (and have enough backends to run em simultaneously)

worst case scenario can do a custom node that emits more previews i think?

[silver osprey]

I did, that didn't help

with the workaround batch size does actually work, but it shows only the top left image out of the batch again; just like normal ComfyUI

[dim sonnet]

0.o?

[silver osprey]

@dim sonnet maybe we could reverse engineer the way live previews in A1111 work? Because that does show the entire batch on the live preview

[dim sonnet]

i slapped together a workflow with two EmptyLatentImage and

[silver osprey]

Only issue with that is, well, ComfyUI is just more efficient due to things like AIT

[dim sonnet]

main one became Batch Size, second one is within the node

oo, i need get AIT running so i can add native compat

[silver osprey]

Yeah, that's really similar to what I did, but does that show the entire batch preview?

[silver osprey]

I can help with that actually, if you would make AIT a default thing you would also need to either ship with the UI precompiled modules, or make standalone scripts that do this if it detects compatible GPUs

AIT's capabilities are 64-2048px, 1-4 batch

You might theoretically be able to push the batch size to 8, but you would need to compile it on a GPU that can actually go above batch 4 with SDXL

[dim sonnet]

Please one issue at a time

that is about the Batch Size setting

preview support is a separate issue rooted in how comfy works that likely will require modifications to comfy

yeah comfy expressly only transmits the first image of the batch

a custom sampler node could easily transmit the rest tho

[silver osprey]

is there a way StableSwarm could bypass that to show the entire preview of the batch? or will modifications to ComfyUI itself be required

this is what I'm currently working with, but again; live preview only shows the top left image

[silver osprey]

yeah, this works now. but live preview still shows the top left image

[dim sonnet]

i'm going to eat you

[silver osprey]

tf?

[dim sonnet]

^

so anyway @silver osprey good news: I can in fact preview a batch ... bad news: by modifying the comfy sampler code, which won't be compatible with your custom AIT workflow

[silver osprey]

I can probably adapt, if the preview method is the same as the normal one it shouldn't be a problem. AIT doesn't have a different node for sampling

can we test that theory?

[dim sonnet]

needa clean up the code before i push, then ye

[silver osprey]

alright, tag me here after you do so

[dim sonnet]

pull latest commit, and swap your KSampler for SwarmKSampler and try it

idk what'll happen if you run it directly on the comfy workflow view tho lol since it's outputting data that ui doesn't know how to process

[silver osprey]

like I mentioned, I'll adapt the workflow to support this if needed

[silver osprey]

uh, where is that node? I pulled the commit but the ComfyUI tab doesn't have that node

[dim sonnet]

are you using Comfy Self Start?

if so it should just be available

[silver osprey]

no..?

[dim sonnet]

if you're using comfy-api-by-url it's a lil awkward and you have to port it

[silver osprey]

how do I do that

[dim sonnet]

i'd recommend self-start

Server -> Backends

[silver osprey]

can I set that to my local ComfyUI install? AIT requires an older Commit of ComfyUI

[dim sonnet]

oo

uh, so, yes, but make sure to disable AutoUpdate

and set StartScript to your local path

[silver osprey]

can I set that to the local path of my bat start script? I built torch and xformers from source in a VENV

[dim sonnet]

(if you want to use api-by-url, you'll have to add an extra-model-paths.yaml file and add custom_nodes: (yourpath)\StableSwarmUI\src\BuiltinExtensions\ComfyUIBackend\ExtraNodes)

[dim sonnet]

if you have a venv, and not python_embeded, it will use the venv automatically

but if your folder is this hacked maybe you should just use api-by-url lol

[silver osprey]

guess I'll do that then

@dim sonnet so this?

[dim sonnet]

that's a valid api-by-url backend yes

and is i assume what you already had

[dim sonnet]

^ need to do this part

[silver osprey]

where do I write that?

[dim sonnet]

inside comfy's folder

there's a extra_model_paths.yaml.example

remove the .example and edit it

alternately you can have swarm launch a self-start comfy once at any time and it'll automatically generate a valid model paths yaml in the swarm Data folder, and you can copy that over

[silver osprey]

my brAIN HURTS

hold up

can't I just add that K_sampler as a custom node?

@dim sonnet won't it be possible to make the SwarmKsampler a custom node? this solves compatibility issues like this

because I can't do the "self start" thing, and api-by-url doesn't allow it to use the SwarmKsampler thingy

[dim sonnet]

it is a custom node

the yaml file is just to link it properly

[silver osprey]

why can't the api-by-url allow it to use that node? that would make things way easier

[dim sonnet]

wh?

it can, you just have to set up the yaml in comfy

[silver osprey]

yeah, but that causes it to spit errors I can't fit into discord..

the yaml solution isn't optimal

[dim sonnet]

what errors?

the yaml solution is optimal if done right

that's the main reason for the self-start being recommended, self-start automatically configures the yaml

if you provide your remote comfy instance with a valid yaml it'll more or less work out the same

[silver osprey]

I don't get it

this is probably just me being dumb

I give up, man

maybe it's not that important to have the live preview show the entire batch

[silver osprey]

@dim sonnet can you possibly port the swarm Ksampler into a custom node that you just got clone into comfyui/custom_nodes and than use in ComfyUI and the ComfyUI tab in the UI

[dim sonnet]

that error is unrelated to the yaml, that's you using comfy self-start

[dim sonnet]

for the third time it literally is that

[silver osprey]

It won't import.. I wouldn't have asked if it was that simple

[dim sonnet]

look in StableSwarmUI/Data - because you tried a self-start, it should have a file named comfy-auto-model.yaml

rename that file to extra_model_paths.yaml and put it inside the ComfyUI folder

then launch comfy as normal

it should just work, and launch with the swarm nodes enabled

[silver osprey]

yeah..

gotta love it when the error can't fit in a normal discord message

I give up man, I'm probably too dumb for this anyways haha

ohh

I had to run it once from Comfy itself

now batch preview works

but it's not in a grid form for some reason

alright

it works, it was just a pain to set up

@dim sonnet any idea why 1 out of 4 previews has higher res than the other 3 and why aren't they all showing up where the final image with the grid would usually update?

honestly, I'd argue this section is useless and if where the image in the middle is placed there would be a grid with the entire batch (like outputting batch in normal ComfyUI) that updates after it's finished

and possibly before it switches with a grid like that have the live batch preview in the middle there, just like A1111 used to do @dim sonnet

[dim sonnet]

pushed a change to catch errors like that and at least display more clearly what happened

[dim sonnet]

that section is pretty handy when you're using swarm normally and not trying to monobatch your hacked comfy

a larger grid in the center might be nice for when you're doing large parallel gens yeah

that's been on the todolist for a while, and you're now the second person to ask for it lol

[silver osprey]

Yeah, you know what I mean? Like A1111 used to have

Like if batch size is enabled have it show the live batch in a grid form, then after it outputs replace the preview grid with an output grid where you would select an image to view (just like A1111's preview used to function)

[dim sonnet]

• Swarm has FreeU support now i guess lol

[hearty notch]

what is that?

btw, side note i did notice that there is a 30% increase in speed when using a haigher batch size

gen time drop from 10 seconds to 7 seconds when using batch size of 3

[dim sonnet]

whatever this paper's on about https://arxiv.org/abs/2309.11497

claims to improve quality with some multipliers

results are, uh, variable

but it's an option in swarm under Advanced for anyone that wants to try it

[dim sonnet]

Depends on more variables than that - prompt length, image size, your GPU, model architecture

[hearty notch]

i guess i will, let me pull the update

[dim sonnet]

(prompts that exceed 75 tokens essentially have extra batch size pre-used from the prompt itself)

(and SDXL and SDv1 respond very different to scaling)

[hearty notch]

oh i now rememeber, when I try to update comfy backend, it shows that there is a new pytorch i think 2.2?

but the update is ending with some errors

let me share it,

i am using the bat file for comfyui and python dependeencies when I try it

` note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for multidict
Failed to build aiohttp multidict
ERROR: Could not build wheels for aiohttp, multidict, which is required to install pyproject.toml-based projects

D:\StableSwarmUI\dlbackend\comfy\update>pause
Press any key to continue . . .`

this will be the end of the update and never more forward after that

[dim sonnet]

0.o?

i haven't used the updater in... ever. I've never used the updater.

Lemme try it lol

well! Good(?) news - i get the same

[hearty notch]

hahah which means?

shall we take it to the ComfyUI people?

[dim sonnet]

cc @cunning holly auto-updater when it pulls reqs seems to be pulling a version of aiohttp that wants to compile itself but can't? aiohttp/_websocket.c(4): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory same for multidict

oo /Fobuild\temp.win-amd64-cpython-311\Release\multidict/_multidict.obj -O2 there's python 3.11 refs in here

same for aiohttp

[cunning holly]

yeah that's why updating with dependencies isn't really recommended

pip packages break way too often

[dim sonnet]

so recommended update route is just git pull and no more touchy?

[hearty notch]

and is there any use of updating to pytorch 2.2?

[cunning holly]

yes, just git pull

[hearty notch]

ok let me try

[cunning holly]

you can update to pytorch2.2 if you want

it probably won't change much

[hearty notch]

is there any guide on that??

[hearty notch]

freeU and Dynamic turned on with the img to the left, the results are ... well need to test it more

but yea! huge variables

[dim sonnet]

psst, you didn't hear this from me but

<.<

.>

we clear? okay

for sdxl base

1/1/0.2/1

or 1/1/1/0.2

both do cool things

[dim sonnet]

and yeah so per ^ swarm's builtin autoupdate for self-start comfy backends is already doing all you need to update properly and you just don't need to touch updating it at all :D

[hearty notch]

I will try it for sure

[dim sonnet]

went looking for AITemplate on ComfyUI and who do i see

is FizzleDorf's repo the latest?

and that's why it requires older comfy, cause it's not maintained?

huh, i can use aitemplate for the unet but not the vae

Seems like just adding Load AITemplate in the middle and leaving the rest the same works really well?

it gens faster and otherwise works the same

8.6 it/s vs 5.6 it/s, nice

:( when not executed directly from workflow view, ait does a full reload, even though it doesn't need to

still noticeably faster tho

[dim sonnet]

oo, vram gets tight

[tepid garden]

105458332365504512

[dim sonnet]

those sure are some numbers you typed there

oh not just any numbers that's my discord user id

[hearty notch]

https://tenor.com/view/hackerman-mr-gif-12747764

[silver osprey]

Oh, the workflow I made is designed specifically for this. If you want, I can give you the version of it I'm planning on releasing

It's even faster than the featured version, 11s per image

[dim sonnet]

sure?

i'm not using any proper workflow about it at all atm

literally just shove aitemplateloader between model and ksampler and runs twice as fast

didn't even use the aitemplate vae because that brokey

if what you got is faster and not brokey that'd be awesome

[silver osprey]

Is you have torch nightly and Xformers nightly VAE is already instant

This is why I haven't provided precompile modules for vae, it's already instant- no need for AIT there

[silver osprey]

this also has a styles feature I added

[dim sonnet]

hwha

[silver osprey]

images get too big for discord, you can bypass this if you want

@dim sonnet also, did you do the batch grid preview thing yet?

[dim sonnet]

so what's the trick here

that makes it faster

this looks like... just using aitemplate, and a really really long nodepath to a disabled refiner lol

[silver osprey]

look at the results =]

[dim sonnet]

67 steps 0.o

well i can't actually run it because you have a bunch of nodes i don't have

and either way results don't answer the question

[silver osprey]

what's the question?

[dim sonnet]

^

what's done here that makes it any faster than just shove aitemplateloader in the middle

[silver osprey]

more efficient settings, and it's a more refined workflow

[dim sonnet]

wh

that's not an answer either

"the workflow is more efficient because it's more efficient"

???

[silver osprey]

the sampler settings, the upscale stuff, all that

[dim sonnet]

... oh

i think

the answer is

it's not more efficient lol

it's just you previously posted a not-very-efficient workflow, and i never looked at that, and you fixed that

for context my comparison point here is

i did that

and it went 2x faster

[silver osprey]

listen, I wouldn't have shared my workflow if it wasn't requested. I am the one that provided the modules for windows

[dim sonnet]

i was confused, i thought you were saying you had a workflow that found a way to be more efficient than what i was already doing

but you were in fact doing a separate-but-related thing

[silver osprey]

also set that to disable, it will break if you switch models with that setting enabled

[dim sonnet]

oh yeah i already hit an error from that before and swapped to disable in swarm's code

which is sad tho :(

on a 4090 it runs much faster with enable

[silver osprey]

that's not how it works

wait, 4090? you shoud be getting like 14it/s with that

[dim sonnet]

it is how that works lol. it's so fast that the load time with it disabled adds a noticeable entire extra second

[silver osprey]

oh yeah, you also need AITemplate empty latent image

[dim sonnet]

2s vs 3s is a big (relative) difference

[dim sonnet]

that didn't make a difference when i tested earlier

[silver osprey]

it won't make a difference, it just makes sure you won't pic incompatible resolutions

[dim sonnet]

incompatible resolutions didn't break anything when i tested earlier lol

or at least silly random resolutions didn't

[silver osprey]

for instance, if you do with AIT 1025^2, you can notice that half of the image won't get diffused correctly

[dim sonnet]

you can't do 1025

[silver osprey]

with the normal empty latent you can

[dim sonnet]

nope

[silver osprey]

then do 1032^2, that would also not get diffused correctly

[dim sonnet]

oo when you do batches with silly res it goes weird, but as long as batch_size=1 it doesn't matter

i always do batchsize=1

[silver osprey]

@dim sonnet , this is what the empty latent node is for, it won't let you do that res- if you use that, it won't diffuse normally

[dim sonnet]

kinda irrelevant in my case either way

[silver osprey]

you would notice that the top left quadrent of the image is latent-looking

[dim sonnet]

cause my resolutions are chosen by

swarm figuring it out for me

[silver osprey]

ah, that solves it then

[dim sonnet]

or the manual sliders which are compatible reses (x64) anyway

[silver osprey]

but must be 32px increments

[dim sonnet]

have you go out of your way to force to incompatible res there

which, well, if someone does that, they're making an active choice to get silly results

[silver osprey]

you could switch it to 32

[dim sonnet]

nnnno

no

the resolution sliders in swarm are designed to make it very easy to land on proper resolution values

rather than just being raw finegrained messes

they snap to reasonable values and are pot-scaled

[silver osprey]

yeah, I saw, handy

[dim sonnet]

such that there's an even space between 256, 512, 1024, 2048

since nobody in their right mind ever wants 2048 but wait go 32 pixels back

and if you're in your wrong mind and want it anyway you can hold Shift to bypass or just type into the number box

[silver osprey]

anyways, this is what I'm working with here.. @dim sonnet any plans to have the live previews be in a grid form in the middle?

[dim sonnet]

^

that's not "just done the minute you ask" type of request cause that's complicated webcode muckin'-with

[silver osprey]

I see. anyways, if you plan on making AIT a standalone thing in Swarm; you will either need to make precompiled modules or even have it execute a compile script when it detect a compatible GPU

[dim sonnet]

i mean, for right now, just, use the AIT ext for comfy is close enough

and hook it up to this nice lil checkbox

[silver osprey]

also, I made an image blending and zero shot subject workflows that work on AIT I plan on implementing into swarm

[dim sonnet]

oooo

[silver osprey]

I'm already saying, I might not be providing modules for AIT anymore, so we might have to resort to making standalone compile scripts

[hearty notch]

do we need to enable it from settings?

I can not see it on my UI

[dim sonnet]

it will only appear if you install https://github.com/FizzleDorf/AIT to your comfyui/custom_nodes folder

[hearty notch]

let me give it a try

am i missing something else to do? the gen time is increased

after turning it off it will error out and stop generating

18:39:56.269 [Warning] ComfyUI stdout: return torch.nn.functional.linear(input, self.weight, self.bias) 18:39:56.269 [Warning] ComfyUI stdout: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 18:39:56.270 [Warning] ComfyUI stdout: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_CUDA_addmm) 18:39:56.333 [Info] No images were generated (all refused, or failed).

[dim sonnet]

uh

well

something brokey

what GPU do you have?

if it's not a modern nv card AIT might just not work right on it

[silver osprey]

seems more like an issue with their ComfyUI and AIT commit

if the GPU wasn't supported it will say something along the lines of "!! no modules found for SM[your SM] XL 1024 4!!"

@dim sonnet any idea why does the UI show only 3 out of the 4 batch size?

the live preview did show the entire 4, but then the 4th disappears and one of the images from the batch stays in live preview format while it also shows the output of it above it

@dim sonnet this is how it behaves, I think this would be perfect if where the output images show up the live previews will show up in grid form, then update to another grid where you can toggle 1 out of the batch and it will show you that one, just like in A1111

[hearty notch]

4090 with latest drivers

when I generate after lunching with AIT disabled I am getting 10 seconds, per img, and after enabling it I am getting 12 seconds,

now, if I disable it, it will error out, and I need to either keep it on, or kill the server and restart again

hence why I am asking if I am missing anything, like do I need to install anything else besides the costum nod?

[dim sonnet]

(that's a question i don't have a good answer to, Tdg8 might)

[hearty notch]

@silver osprey well, let's wait and see if he will share more insight

[silver osprey]

this is a conflict with your ComfyUI commit and the AIT commit

[hearty notch]

and what you adivse?

[silver osprey]

you need a specific commit of both for it to work properly

[hearty notch]

can you share more info on how to get it done? like a link or a guide?

[silver osprey]

just merge this with your ComfyUI install after you back it up incase anything goes wrong

https://github.com/comfyanonymous/ComfyUI/tree/bc76b3829f5fbba7c5a439c7833d313a3ca87398

it's from over a month ago tho, so idk if this is an optimal fix

for me, the other compatible versions of ComfyUI are unacceptably slow, so this is what I use

[hearty notch]

I think a good option is to install it separatly and create a new backend to swarm with this one

[silver osprey]

yeah, probably a good idea

[hearty notch]

I will give it a go now

[silver osprey]

wait, there is also a commit of the AIT node you might need to switch to

this is what I use for the node

https://github.com/FizzleDorf/AIT/tree/ca8f0627992649eb2211bbdc2f269484b054e62b

[hearty notch]

humor my lack of tech knowledge, and if you have the time to guide, how can I install a spesifc commit?

git clone https://github.com/comfyanonymous/ComfyUI/tree/bc76b3829f5fbba7c5a439c7833d313a3ca87398

like so?

[silver osprey]

you just download the zip and replace all conflicts

[silver osprey]

this won't work likely

just download the commit as a zip, then replace all conflicts

[dim sonnet]

i have latest of both and it's fine 0.o

[silver osprey]

yeah, but 8it/s on a 4090? no. you should be getting much more with AIT

[hearty notch]

so I will install the latest one, as a fresh install, then replace with the old commit and overwrite the files

[silver osprey]

that should do it

[hearty notch]

awesome

[hearty notch]

and for this will be git clone https://github.com/FizzleDorf/AIT.git in the custom nods? right?

[silver osprey]

no, you extract the zip in the custom nodes

[hearty notch]

like so?

[silver osprey]

yep

[hearty notch]

ok let me fire it up and see

ummm, gen time is going up, compared to what I have before

also, @dim sonnet when I created a new backend with the old commit the FreeU is gone from the gen tab

[dim sonnet]

yeah freeu is brand new

so running old comfy = can't have the new thing

[hearty notch]

oh I see,

[silver osprey]

freeU seems to make results slightly worse tbh, idk why would it be relevant for SDXL

it might be relevant for 1.5, but idk about SDXL

[silver osprey]

I think also latest ComfyUI+latest AIT could work, but the latest ComfyUI updates are broken on the 4000 series except for the 4090

This is due to it attempting to use more VRAM, which when the 4000 series gets over 90% full, the speed turns into a small fraction of what it used to be

Again, Nvidia is the one to blame about this, not ComfyUI

Nvidia always finds new ways to screw with us

[vale turret]

why are my checkpoints shwoing up but not my Loras

[hearty notch]

hi @dim sonnet i did try AIT with many scenarios now, seems to work fine when using basic settings only with a high batch number, there is a very good time reduction there

however, once you add a lora things will go sideways so quick

also

it will error out if you disable the AIT option and you can not gen anything unless you fully close the server

and start from scratch

this is what I observe after adding a lora to the gen process

I am not sure if this is normal, you can see the last one when adding a lora

[dim sonnet]

nvidia was trying to do a good thing to be fair - they made it so you could go past VRAM limits without crashing, which is great for a lot of cases

it's just bad for redlining AI software lol

it's probably possible to correct for at the software layer (ie add an artificial vram cap in software a lil below the hardware vram size)

also driver dependent, my poor rtx 4080 server-laptop just yeeted a cuda oom when i pushed a gen in ait too hard on it :(

[dim sonnet]

check Server -> Server Configuration, your Lora path might be wrong?

depending on where you've been putting your loras and models

[dim sonnet]

outside of AIT-specific issues, the time to gen with a lora the first time might be slow because it has to load the lora file, after the first it should be cached and be quick again

re AIT-specific, ima test and see if i get anything weird

aitemplate seems happy to use loras for me

[silver osprey]

that sounds like an issue with your commits actually. I tried updating everything yesterday and my 4070ti is also doing that, so I reverted to commits from a month ago

again, no denying Nvidia are assholes for using tech from 2012 in the 4000 series's VRAM

they cucked the entire 4000 series except for the 4090 and 4080ti

this is why the rest of the 4000 series can't go above 90% VRAM usage

so if you have a 4000 series with 12gb VRAM, it's more like 10 because you are unable to use 2 out of the 12

[silver osprey]

also, I didn't test FreeU yet, does FreeU change anything about the architecture?

if it doesn't- it should be compatible with AIT right off the bat

[dim sonnet]

what?

I... pushed gen settings too high and ram out of VRAM on a machine running linux with probably slightly old nv drivers for linux, and it OOM'd

[hearty notch]

Btw, what is happening with tensorrt, I did notice that there is one from stability on hugging face

[dim sonnet]

has nothing to do with my code

[hearty notch]

I did try tensorrt before with sd1.5 and that was a real game changer for nvidia

[dim sonnet]

trt is really capable and really fast but it is hell to get working properly

[hearty notch]

What is oom? Can you elaborate?

[dim sonnet]

the trt file on HF is so anyone who wants to go through the joys of fighting with trt to make it work, has at least some of the work done in advance

[silver osprey]

TRT isn't as fast as AIT, but it allows for OneFlow- which is faster than AIT

[dim sonnet]

OOM = Out of Memory

[silver osprey]

currently OneFlow is fastest, than AIT, than TRT

[dim sonnet]

iiii've seen graphs that say different

[silver osprey]

?

AIT is right behind OneFlow

[eternal walrus]

1

[hearty notch]

With 1.5 I home baked a model with my old 3060 nvidia and the results are wild, hope to see someone do something for that sdxl

[dim sonnet]

is that... an old graph, pushed by oneflow or ait? for SDv1 that's just lying about it?

cause 16.7 it/s in pytorch for ... any version of SD on an a100 is very wrong

[silver osprey]

it's a graph on OneFlow's website

[dim sonnet]

yeah

they're just lying about numbers to sell a product lol

[hearty notch]

Yeah this is so true

[silver osprey]

hold up, I'll look for a graph outside of OneFlow's website

[dim sonnet]

to my understanding of trt (i haven't dived in much so this is secondhand) "homebaking" the model is basically obligatory, as every card has to compile it slightly differently

[dim sonnet]

eg a 3060 and a 3070 or whatever require two different trt compilations

and the files aren't cute lil 5 mib chunks like ait has, the trt files are full fat models

[dim sonnet]

yeah that looks closer (albeit outdated)

[silver osprey]

I think OneFlow is also architecture specific instead of being checkpoint specific like TRT is

[dim sonnet]

oneflow is hyper specific

[silver osprey]

oh, really?

[dim sonnet]

it's like

you need one full fat megamodel file for 1024x1024

and a different one for the same model but 16:9

[silver osprey]

well, at least we got AIT and it's at the very least close to stuff like that

[hearty notch]

I recall it was limited to my 12gb vram, and based on the res you bake it, you are stuck with that, which means no upscale and you are always and you must be 512x512

[hearty notch]

Wait! Wai! I am a 4090 user! These numbers are bull stinky shite

[silver osprey]

I also heard from FizzleDorf they managed to get TRT and AIT working together and the speeds where over 200it/s

this is what FizzleDorf somehow did

idk how the fuck, but it's possible

[hearty notch]

Wait hold up! You can run on 26it/ s with 4090??

[silver osprey]

those are benchmarks on 512^2 on SD1.5

[hearty notch]

What they are rendering? 256x256?

[silver osprey]

not SDXL

[hearty notch]

Well this is why these numbers are in my personal opinion make no sense

Controlled numbers are like Alex was saying to sell something

But reality is what matters

[silver osprey]

try it then, they seem to do- I know people that have 4090 and that's similar to what they get with those settings

[dim sonnet]

that particular graph is somewhat decent, independent testing, it's just outdated

[hearty notch]

Can you share a guide? I am usually just using out of the box settings

As I put my faith in the ui developer

[silver osprey]

so far this is the fastest inference though. idk how the fuck FizzleDorf pulled that off, but yeah- that's the fastest speeds that are currently possible (probably)

[dim sonnet]

since you're using Swarm with comfy backend you're fine as far as anything that chart can achieve

that chart comes from Auto WebUI era where you have to manually tweak every internal setting to make it work. Comfy mostly does it for you other than advanced things like ait

[hearty notch]

This is the same game with games benchmark, running the test on a controlled environment with liquid nitrogen coolers! Lol

[dim sonnet]

barring... a lot more to that, probably what happened is they accidentally broke the model to not run properly and measured the inference speed of generating garbage lol

[hearty notch]

But in reality no one will do that

[dim sonnet]

(and shared it for the sake of the meme as opposed to "this is the results we get now")

[silver osprey]

I know FizzleDorf, they managed to get AIT working with TRT on 1.5 and that was the speed

[dim sonnet]

there is however a lot of room of optimization hacking - look no further than exllamav2 to see what happens when a dedicated optimizer focuses in on it

[hearty notch]

Btw, going back at our conversation about the Lora @dim sonnet I know that changing any setting in swarm usually will slow down the first gen like you shared before

But what I am starting to notice that different Lora will also change the speed of the render

[silver osprey]

I heard about that- doesn't that do degradation though?

[hearty notch]

Not counting the first gen

[dim sonnet]

exllamav1 had somewhat degraded quality, exllamav2 has a new technique to actually improve quality over other quantized executors

[hearty notch]

I am not saying the opposite, I know for sure they did it, but what I am interested in is the result of that run with 200/it

[silver osprey]

that result is identical to the result without optimizations on the same 1.5 model

that's just 1.5 being 1.5

[silver osprey]

peculiar. does exLLaMa use some kind of engine for speeding stuff up? or is there something different happening there?

[dim sonnet]

the main thing is they just wrote their own cuda kernel for processing things

there have been multiple different projects in succession that have done that for llms

eg gptq-for-llama had a custom cuda kernel variant that blew any/all competition out of the water including, er, whatever the llm equivalent of ait from meta was called i forget

triton, that'n

[silver osprey]

it was from microsoft and it's called DeepSpeed I think

[dim sonnet]

deepspeed's differenty

also unbelievably cursed

[silver osprey]

probably true

anyways, I'm thinking if stuff like exLLaMa don't use engines; we might be able to implement AIT on top of that

[dim sonnet]

it doesn't use an engine because the engines would be slower

it's a layer deeper than "use an engine"

it's "we just wrote our own hardware calls"

[silver osprey]

really? so it's just the kernal doing the speed up?

maybe we could get exDiffusion eventually haha

it would be nice to have the same boost exLLaMa gives on diffusion models, I tested exLLaMa2 just a moment ago; autoGPTQ was 8T/s, exLLaMa2 was 62T/s, insane.

almost squaring the speed there

[hearty notch]

btw, can someone recommend some guide on dynamic thresholding? and how to properly use it? or this is just an/ off switch?

[dim sonnet]

https://github.com/mcmonkeyprojects/sd-dynamic-thresholding/wiki

[hearty notch]

oh that was your baby! oh damn,

ok time to read

[hearty notch]

do you have any recommendation on the settings? or you advice on experimenting ?

[silver osprey]

when using the latest ComfyUI commit I get 4s/it with batch 4 with AIT, and on old commits I get 3.2it/s with batch 4 also with AIT. what happened Comfy?

I was about to test FreeU, but the ComfyUI commit supporting FreeU is a fraction of the speed of older commits

[hearty notch]

which gpu you are using?

i tired the commit you shared with me before for both comfy and the ait nod

but there was no speed improvement

[silver osprey]

4070ti. I think the newer commits just try to use all the VRAM, while gpus like 4070ti fuck up when VRAM is over 90%

[hearty notch]

i am on 4090, which ideally should perform even better, but I really can not see it

[silver osprey]

@cunning holly is this true?

[hearty notch]

first snap is with AIT off, second is On, and that's my VRAM load with AIT on, this is with all up to date ComfyUI and AIT nod @silver osprey thought you may find this interesting

4 img batch of 1024x with 20 steps

[silver osprey]

if you're using the AIT integration built into swarm, idk. but in vanilla ComfyUI it's about double the speed

not on latest commits though, idk what Comfy did there that messed up AIT so badly.

[hearty notch]

yeah I am using the one from Swarm

[hearty notch]

I did try the one you shared with me,

it was almost the same

I am just sharing what I did try

[silver osprey]

again, you used the swarm integration, it doesn't seem to work the same way as normal ComfyUI

[hearty notch]

ah ok

I will try it then only with comfy