🆕｜sd3

[hallow lion]

Comfyui nodes and workflow will arrive 5.2 seconds after the weights drop.

[sterile pendant]

I don't know about how long kohya will take to get it up and running with the UI, but I'm sure there will be some update to the accelerate and other python libraries that will support it and when that happens, you'll be able to do it manually with a CLI script (all koyha's interface really does is gives you a convenient UI to make the command, then runs it in the command prompt)

[hallow talon]

Unfortunately I have no idea what that means. I don’t know much about Python itself.

[neon wagon]

SD3 loras = 2 week after the weights
SD3 finetunes = 1 Month after weights

but good finetunes merges after weight need 3 month minimum.

[dry wave]

kohya sd-scripts is a cli script.

you don't need updates to accelerate (you don't even need accelerate if you use single gpu). You need updated on the training script

[rain current]

Where can I see the results of 2B?

[bitter hearth]

"the villain, a villainous villain making villain stuff, in a dark alleyway"

very villain

[storm saffron]

2 weeks? 3 months? Why so long? Fine tuners will already have datasets ready to go.

[sterile pendant]

I know it is, I already said it's just a UI to make things easier to build the command. There's even a button to print the command in the command window so that you can use it without the interface if you desire. I said accelerate because that's literally what gets called when you run kohya, it sorts the rest out. But yeah, the other libraries, like I mentioned, will need to be updated before anything can propagate downstream from it.

[sterile pendant]

Should only take a few days to weeks before you can train though.

[dry wave]

yes, and that's wrong

kohya_ss is the python script that does the training. It's not an ui

there are several UIs on top of kohya_ss, that is what you are referring to

accelerate had nothing to do with that. It's just a wrapper script that simplifies things like multi gpu training

[sterile pendant]

See what I said about other python libraries...

And accelerate can then call those other libraries. But yeah, I should have worded things a little clearer. Accelerate is like saying diffusers in the sense of it simplifies all the platform/hardware stuff and simplifies the calling of other libraries. Can you manually call those the hard way, sure. The training scripts will be updated pretty quickly, but won't work their ways into UIs until things like accelerate are updated, since you'll have people trying to train on a variety of PC environments

[dry wave]

yes and that is still wrong lol

you don't have to update accelerate

accelerate is really just a "simple" wrapper that allows to switch between different precision types and gpu setups.

[sterile pendant]

and you're 100% postive that it will be able to handle switching to the requirements that the DiT architecture has?

[dry wave]

doing that manually is not more effort or complicated than using accelerate. The idea of accelerate is that you can easily turn off these things without changing the code

yes

DiT is just a transformer architecture running on pytorch

and we had this discussion already but: DiT is using transformers, sdxl is also using transformers. You call the same pytorch kernels under the hood

[sterile pendant]

i suppose we'll see, just figured there'd potentially be compatibility issues with SD3 due to all the components that will have to be shuffled around like if you're trying to train sd3+encoders+t5 at the same time. i imagine they will add in some kind of control over where to keep the various models, like having the T5 stay in ram instead of vram and so on. but yeah, at the core, they are all going to just call pytorch at the end of the day

[dry wave]

yes, but accelerate is not doing these kind of things

[sterile pendant]

https://huggingface.co/docs/accelerate/en/package_reference/accelerator

i'll read through a bit more, but it definitely looks like it handles where shit sits

from a wrapper point of view

But anyways, again, other libraries are going to have to be updated, along with training scripts, etc etc, before you'll see UIs implment them. Your average user isn't going to want to go into command prompts with a scalpel to manually train. 99% of people are going to rely on things like koyha and onetrainer

So it will likely take days to weeks for all the stars to align to be able to just do it through a UI

[storm saffron]

You can freeze the embeddings for the text encoders and the image latents before training to save vram. All the stuff for training DiTs is in pytorch already otherwise pixart and Hunyuan couldn't train.

[sterile pendant]

that's true, so there's already a leg up there

[dry wave]

stuff like cpu offloading, pre-conputing and so on have to be done by the library itself (diffusers or kohya_ss)

but you are right for sure that it will take some weeks until everything runs smoothly for the end-user

[storm saffron]

Yeah not everyone but the enthusiasts and early adopters will want to fiddle with cli scripts.

[sterile pendant]

right, again, the main point is that people want to be able to do this in UIs like koyha and onetrainer, so all the various libraries used in those are going to need to be updated. there's a reason why linux only has a 2.3%(counting steamdecks) usage rate for home PCs, people hate CLI bullshit(even though modern versions don't make you do cli stuff really anymore, people still have the bad taste in their mouthes from when they tried linux ages ago lol)

[storm saffron]

We're just seeing new people starting to train pixart now OneTrainer added support.

[sterile pendant]

i saw that they finally added support for it. personally, i like onetrainer more than koyha

[storm saffron]

WSL does kinda help with that though if you truly wanted to train a lora the day SD3 comes out (assuming they release the LoRAs code same day)

[sterile pendant]

even if they don't release it day one, i'm sure people would try to frankenstein something up anyways

[dry wave]

if you have fine-tuning code going to lora is trivial

[storm saffron]

I'm sure. The code will be there in diffusers, just gotta glue it together.

[sterile pendant]

it's not some new architecture or anything, llms have been using it for many years now, so a lot of the functions are already there. obviously, you just have to get the steps in the right order for the layout of how SD3s specific flavor works though

[storm saffron]

Wonder what happens if you give T5 and clip completely different prompts entirely that contradict each other when inferencing. 🤔

[wide pagoda]

Similar things have ready been done with SDXL
Anyway I assume clip would dominate

[silver sluice]

So after following the conversation, from my understanding, even tho people already have early access to the weights, none of those people have knowledge or ability to integrate or update the UIs

[cobalt moon]

comfyanonymous :

[silver sluice]

Seems like another missed opportunity by SAI, not giving early access to devs who integrate stuff and fine tuners, seems like a strong case for that VIP room

[cobalt moon]

Bro literally SAI staff

[silver sluice]

So we’ll have to rely on CLI scripts to generate anything after release? I realize someone said to expect workflow for comfy quickly after release but it seems prudent to just have given access to devs who would be making this work so it can be refined and ready for launch

[cobalt moon]

Lykon who also got access on SD3 Medium said he will train new Dreamshaper based of that model

[silver sluice]

I’m a big fan of dreamshaper, is lykon the guy who made it?

[cobalt moon]

Yes, he just said that in Civitai Discord server

[storm saffron]

I believe comfy will be updated same day as launch to support SD3 (it's literally what they are using to test and generate the images you see on X and discord)

[cobalt moon]

SAI literally use Comfy to generate SD3 images yeah

[sterile pendant]

Maybe one in a hundred thousand to million users train stuff. It's really not as high of a priority as you'd think. We're just in an echo chamber of enthusiasts. Plus, I'm sure SAI just can't wait to see all the degenerate stuff people will immediately train; that further gives AI a bad rap and further pressures governments to want to crack down on generative AI even harder...

[cerulean sphinx]

Gradio is such easy to use that updating the UI to train SD3 should be quite quick once you've got a CLI script to train it.

[silver sluice]

So comfy supports SD3 internally and they’re just going to release the weights along the software update I’m gusssing?

[silver sluice]

I’m not concerned or interested with training stuff just wanna do basic level stuff like use it with comfy lol but I do see your point

[cobalt moon]

Yes

[cerulean sphinx]

But personally I think using config files are easier in the long time than dealing with any UI. The only benefit the UI serves really is reasonable presets for training.

[cobalt moon]

Well I dont think it is mattered that much for high quality finetune creators

[cobalt moon]

This is what you get without regulations

Everyone ( at least here ) hate regulation, but they must be there, for legal purpose

[silver sluice]

So you prefer editing a file and running a command over using a UI?

[cobalt moon]

You do not wanted people to easily generate porns with your face on and share it online

[fiery wharf]

funny that they use that excuse to regulate ai models but ai military models used to bomb other countries are not regulated

[cobalt moon]

AI Military Model? Care to share one of them please

[sterile pendant]

Yep, and so long as people are consentlessly making vulgar loras of people, along with other endless amounts of illegal content, governments will feel more and more pressure to regulate and enact laws

[cobalt moon]

If you cannot show it then it is just a scaremongering

[fiery wharf]

https://interactive.aviationtoday.com/avionicsmagazine/december-2019-january-2020/artificial-intelligence-efforts-for-military-drones/

[sterile pendant]

if you live on land within the planet earth, you are subject to the rules of who ever owns that land. they make the rules, end of story. military offense/defense is critical in keeping a nation going. making waifus and training ||drippy butthole || loras are not.

(i was legit horrified to learn that it wasn't just a meme, that these were actual loras people trained...)

[fiery wharf]

oh yea bombing people with ai drones is better than some nude b

[sterile pendant]

look, i get it, you have a toy and you don't want people telling you how you can and can't play with it. but you're comparing two completely different things

[fiery wharf]

not really we are comparing goverment regulation of ai and why they arent touching ai military models

reading comprehension 101

[sterile pendant]

oh i get what you're trying to say, but what i'm saying is that you're too immature and self-centered to realize that they aren't even remotely in the same league

this isn't checkers, it's chess

national safety is exponentially more imporant to a country than you playing with ai waifus

[fiery wharf]

government wants to regulate ai but doesnt touch ai military models,there i kept it clean and simple for you seems like english is not your forte

if they truly cared about security they would regulate both

[sterile pendant]

it's actually my third language, yes. the governments want AI for the military for national security. it's an arms race... protecting your country from other countries that want to harm your country. AI will make a larger difference in physical and virtual warfare, than going from the sword to a drone strike.

[sterile pendant]

no, they will regulate the one to prevent internal stability issues that would arise from people making fake content that could cause economic/political issues within a country. like deepfaking a politician doing or saying something

[storm saffron]

I mean that's already been possible for years and years with Photoshop premiere, etc.

[sterile pendant]

not easily

[desert garnet]

these reads like one of those evangelical clowns in america telling me why killin ppl is better and safer than sex

[sterile pendant]

and not doable by 99% of people that can type simple words out

[sterile pendant]

i'm an atheist, but reality is reality. if you want to keep a country running, you need to stay a step ahead of your enemies and rivals

[storm saffron]

No, but they don't really matter in the big scheme of things, instability comes from large actors who have enough money to be consistent

[tiny otter]

@storm saffron i need to create images where should i go

[storm saffron]

I dunno.

[desert garnet]

yes the ai drone that killed the humanitarian workers from belgium in gaza helped to keep my country safe

[sterile pendant]

do you know how many civilian casualties there have been, historically, in wars throughout the past century?

it absolutely sucks and ideally should be zero

but modern warfare has cut it down astronomically

[tiny otter]

Anyone know ??

[desert garnet]

thats true,to increase that number we should train more models to automatically target ppl based on an algorithm

[sterile pendant]

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8581199/

i can dig up even more credible stats like this

[desert garnet]

perfect argument for ai robots in wars

[sterile pendant]

which is why im all for it in militaries. let the bots duke it out with the other bots and leave the people out of it. it would just be the modern version of siege warfare, minus wasting human lives

[desert garnet]

yea if your family dies because of a misplaced airstrike, its just collateral damage bro,man up

[sterile pendant]

the point is, that number will be far lower vs historical statistics

[cobalt moon]

And to target you and your family?

Yeah, sarcasm 100%

[sterile pendant]

so yes, governments are going to continue to strengthen AI usage in their militaries and yes, they are also going to crack down on what citizens are allowed to do with AI usage. you can cry into the wind about it, but it is how it is

[cobalt moon]

We went from SD3 to whole AI military regulation and usage

[desert garnet]

well there was someone cryin over waifus so it always goes down that way

[sterile pendant]

yeah, it was a dumb false equivalency fallacy thing that someone brought up, like usual

[silver sluice]

can you do the same image but monochrome fuzz pets and instead of kids replace it with men in black suits? lol i dont know if you're taking requests or not but that one came out raelly nicely looks great

[patent acorn]

i would better see the mice facing to the side instead

[cedar gale]

You can mount your hamster however you like.

[patent acorn]

no way

[urban arch]

Ribbed, for HER pleasure.

[raven fern]

awww that's cute :3

[frail shoal]

i'm curious, is the SD3 prompt comprehension better than pixart sigma that has a 20GB T5 model ? Has anyone compared them both ? I'm currently using sigma with sdxl refiner and it can understand things like concepts, emotions and feelings and represent them visually like a real artist. Curious if SD3 text comprehension can do that

[sterile pendant]

The issue isn't so much the size of the t5, it's how well it maps to the image generation model. Think of t5 like chatgpt and how well it can handle and understand conversational input. Since it can better understand semantics like a red ball on a green cube on a blue table, it can then do a better job(in theory) of pairing those colors to those shapes when you go to generate it. That 20gb is it in fp32, there are also fp/bfp16 versions of it that are around 10gb. Pixart sigma is awesome, but is pretty small and not all that trained yet(in comparison to say sdxl finetune level image quality). SD3's t5 will likely be on par with pixart's, but the image creation half will blow it out of the water since it has a far larger dataset and more training to map the concepts

[frail shoal]

base image quality in sigma is shit, that's why i use sdxl refiner

[sterile pendant]

Also keep in mind that SD3 has three input encoders: the same clips from sdxl and t5. I'd imagine you can run it with any combination of the three, maybe even pure T5, but I can't remember if they've verified that or not

yeah sigma is meant to be more of a research paper kind of model, they don't have the resources to fully "train" train it all the way

It costs a lot to make quality base models

but sigma is great for hashing a scene out to then be resampled with sdxl. it has kinda been the poorman's sd3 while we wait for sd3 to come out

[silver sluice]

how many gigabytes do you guys expect SD3 2B model to be? looking for a ballpark estimate

[trail frost]

Is there any open source image generation models other sdxl 1.0 and pixart

[gusty trail]

hunyuan-dit, lumina-t2i, Kandinsky 3.0

[low stone]

Hunyuan has been great. Has different composition than pixart so its adds variety to use both

[trail frost]

Thank you so much

[frail shoal]

exactly, however i find that with the poorman's sd3, i prefer the images this workflow does than midjourney. Midjourney does not do what you want most of the time

[frail shoal]

also i'm using 20 gb model since i can use it in cpu. The half precision one only works on gpu with comfyui

and i need gpu room for sdxl refiner

i only have 6gb vram

[lucid swift]

living avian theropods

[cerulean sphinx]

Yes, that makes it easier to test out different parameters when training.

[sterile pendant]

Was kinda just hoping it would spit out a chicken since they are living descendants of them lol

(well all birds are living descendants of therapods)

[lucid swift]

yes xD

[lucid swift]

another result xD

[lucid swift]

what the hell

[cedar gale]

Lol thats a dualtops.

[cedar gale]

😄

11 / 10.

[lucid swift]

😎

[cedar gale]

FBI catfish?

[lucid swift]

xD yes

the thinnest smartphone that exists android

the thinnest smartphone that exists android 1999

[cunning lintel]

New china model (Lumina-Next-SFT) https://github.com/Alpha-VLLM/Lumina-T2X
If these devs keep this progress up this lumina thing might start be be actually good

[lucid swift]

looks impressive

sd3

[fallen swallow]

WTF ... this is awesome!

[silver sluice]

i could see that, where you need to batch things so you quickly script a thing that'll batch a few different scenarios to examine the results. sounds about right?

[trail frost]

Anyone here using Hunyuan dit if so can you please share some examples

[sterile pendant]

That's the new finetuned version, right?

I need to reinstall it to mess around with it again

[lucid swift]

cool

[frail shoal]

can you use it in comfyui ?

[dreamy sundial]

this starts to look good

[storm saffron]

If nobody has answered... 4Gb

[silver sluice]

thanks i've been super curious about that, so the 2 billion parameter model will be around 4 gigabytes and the fine tuned models are expected to be 6-7 kind of like SDXL right?

[storm saffron]

Nope, 4. SDXL was around 6 due to also including CLIP and VAE in the safetensors file, if they also include the VAE and CLIP in it then it'll end up around 5.5Gb (ish)

SDXL was 2.6B and ended up at 5Gb just for the UNET

Without VAE and CLIP

[lucid swift]

Lumina-Next-SFT promt: This image is a captivating digital artwork that portrays a surreal scene set in a misty, swamp-like environment. The dominant color palette is a deep, neon purple, which bathes the entire scene in an otherworldly glow. The atmosphere is thick with fog, obscuring the details of the trees and vegetation that surround the scene. The water in the foreground reflects the eerie light, creating a mirror-like effect that adds to the sense of depth and mystery.

In the center of the image, there is a figure seated on a motorcycle, poised as if ready to depart. The rider is clad in a dark, form-fitting outfit that merges with the shadows, and their helmet has a reflective visor that mirrors the intense purple hue of the surroundings. The motorcycle itself is sleek and modern, with a design that suggests speed and agility. The rider's posture exudes a sense of anticipation, as if they are waiting for the perfect moment to make their move.

Lightning bolts pierce through the dense fog, creating a stark contrast with the otherwise monochromatic scene. These bolts of electricity add a dynamic element to the image, suggesting a storm or a supernatural event. The lightning's jagged lines and the way they illuminate the mist create a dramatic and intense atmosphere, enhancing the overall sense of suspense and wonder.

The image is a blend of natural and surreal elements, creating a dreamlike quality that invites viewers to immerse themselves in its mysterious world. The neon purple color scheme and the ethereal lighting contribute to a feeling of otherworldliness, making the scene both captivating and unsettling." image

[storm saffron]

at least 8 GPUs are required for full fine-tuning of the Lumina-T2X 5B

Ouch.

[lucid swift]

i think ther is a 2b model?

at least it says to b in the web gui

[storm saffron]

There's a bunch of models and it's not that clear how to actually use any!

[lucid swift]

yes they shuld have sepereated models into diffrent githubs

but i guesse more stars or someting

[storm saffron]

no requirements.txt either for pip

[lucid swift]

💀

Lumina-Next-SFT

[storm saffron]

What's SFT?

[lucid swift]

i think that means its finetuned to look better and its not a base model

[storm saffron]

Not just the base model safetensor's version?

[lucid swift]

"Lumina-Next-SFT is a 2B Next-DiT model with Gemma-2B serving as the text encoder, enhanced through high-quality supervised fine-tuning (SFT)."

[storm saffron]

Ahhhh ok

It doesn't look that fine tuned from the pic above. 😄

[lucid swift]

xD

[storm saffron]

What was your prompt?

[lucid swift]

i think it uderstands natural laguage better so my promt was not optimal "photograph of ghost special force agent, adorned in all-black human anthropomorphic furrsona fish in fursuiter at a con highly detailed, the interplanetary from "2001 a space odyssey"

[storm saffron]

I'm gonna compare it with my PixArt finetune.

Have you tried their 2K model?

[lucid swift]

no

[storm saffron]

There, my pixart FT

[lucid swift]

this one is more like a fish

"photograph of ghost special force agent, furrsona detailed, the interplanetary"

[storm saffron]

Anthropomorphic, ✅

[turbid grotto]

I have seen some disagreement about sd3 75 tokens due to clip or 512 due to T5 a while ago, is this still concern or it is not a problem?

[lucid swift]

its still a porblem

[storm saffron]

OK now it's gone more furry since I changed furrsona to fursona

[lucid swift]

we are not sure how it will behave

[lucid swift]

these models dont like mispellings

[storm saffron]

Comfy needs a spellchecker node

[turbid grotto]

someone noted "clip stacking" or somthing like that, could it be a solution?

[storm saffron]

I guess it's going for Ghost Recon?

Considering it's a 600M param model that I badly finetuned, it's doing alright. 😄

[lucid swift]

i am not sure.

yes thats smaller then 1.5

[lucid swift]

it might allow 512 tokes

[storm saffron]

Well DiTs behave more like models 1.5 - 2x the size of their UNet counterparts of the same parameter count.

[lucid swift]

what do you mean

[storm saffron]

I mean they behave more like models bigger than the equivalent UNET model.

[lucid swift]

oh

[storm saffron]

Hard to explain. A 2B parameter DiT is more like a 3-4B parameter UNET in capability

[lucid swift]

vs sd3 but sd3 was only able to fit a small part of the promt

[lucid swift]

daim

i think one problem of the lumina model is that they use sdxl vae

[storm saffron]

That could be an issue, especially with small faces and text.

Same problem with PixArt

IDK if I can be bothered to get Lumina working with SD3 just a few days away

[lucid swift]

xD if i would be home i would install it xD but you can also just use the web demo

lumina vs sd3 (but sd3 is like 8b and lumina is like 2b ) The image depicts an alien-like creature with a large, elongated head and dark, almond-shaped eyes. The skin appears textured and rough, reminiscent of reptilian or amphibian skin. The creature is sitting in a body of water, partially submerged, with its legs and lower torso hidden below the surface. The background is foggy, adding a sense of mystery, and features tall, thin reeds and barren trees, creating a marsh-like or swamp environment. The overall atmosphere is eerie and otherworldly, with muted colors and low light enhancing the creature's unsettling appearance.

[silver sluice]

thanks for the technical answer i appreciate the insight

[storm saffron]

I like the composition and light of the 1st one.

[lucid swift]

yes

but the second one looks more like a photo

but i never said photo in the prompt

[storm saffron]

PixArt. (600M param)

[lucid swift]

this looks also interesting

[storm saffron]

I'll do a widescreen version, which I think would enhance it a bit

[lucid swift]

"The image shows a cozy, eclectic room with a vibrant, colorful ambiance. The ceiling is draped with multiple tapestries featuring intricate designs, including mandala patterns and depictions of plants and celestial motifs. The lighting is soft and atmospheric, with various sources contributing to the overall mood: Ceiling Lighting: There are red, pink, and purple lights that illuminate the tapestries, highlighting their patterns and adding a warm glow. String Lights: Multi-colored string lights are draped around the room, adding to the festive and relaxed atmosphere. Television: A flat-screen TV on the wall displays a scene from the animated show "The Simpsons"

[lucid swift]

cool

[storm saffron]

Lumina seems pretty undertrained or not at all finetuned.

[lucid swift]

yes this look under trained

[lucid swift]

impressive

[storm saffron]

PixArt constantly impresses me for its size

[lucid swift]

did stabillety improve fursuits?! "The image shows a person dressed in a colorful fursuit against a plain pink background. The fursuit features a large, stylized head with prominent, pointed ears that are white on the inside and purple on the outside. The head is covered in bright red fur with a white stripe across the face and purple fur around the muzzle. The eyes are large, black, and oval-shaped. The person is wearing a black shirt with long sleeves and a purple skirt. The sleeves have blue and purple striped arm warmers extending to their paws, which are covered in purple fur. The lower part of the fursuit includes purple fur-covered legs and feet. The overall style is vibrant, playful, and highly stylized, typical of fursuits often seen in the furry community."

maby the lumina model wants diffrent promting or something

[storm saffron]

I guess they would have trained it on fursuits to please a certain demographic?

[storm saffron]

It's definitely NOT a fursuit. It's a weird combo between the two. Hmmm... Try "Mascot Costume"

[lucid swift]

idk but stabilety models are normally very shit at fursuits. i know that xD

[storm saffron]

Mascot costume is not as bad, but sill not good. 😄

notes If I did a retrain on my finetune of pixart add more furries.

[lucid swift]

the only one good at furrys is open ai here my tests xD

[storm saffron]

My dreamMODE model for SDXL is pretty good at furries.

[lucid swift]

also fursuits?

[storm saffron]

I'll try your prompt in it

[lucid swift]

the promot is just made with chat gpt so it might not be a good one

[storm saffron]

It'll probably miss out half of it due to being SDXL

[lucid swift]

xD

[lucid swift]

classic clip

its better then bais. but it does not look like a real fursuit xD

[storm saffron]

It's close, hold on.

Better?

A person in a grey wolf fursuit at a comicon

[lucid swift]

yes it looks better

[storm saffron]

that's dreamMODE CosXL on civit if you want it. 😄

[lucid swift]

i just asked chat gpt to generate the image with the promt and it made this

[storm saffron]

Why so much fire?

[cedar gale]

I have no idea I tested your prompt.

[storm saffron]

That seems like a lot of fire for it not being in the prompt. 😄

[lucid swift]

https://tenor.com/view/shannon-sharpe-shay-nope-nah-nuhuh-gif-14856215

[cedar gale]

Ok ok I admit, I might have spiced it up a little.

[storm saffron]

did you just add "on fire"

[lucid swift]

✅

[storm saffron]

😛

felt puppet nightmare, night, fog

[cedar gale]

Noice.

Pikachu edition.

[lucid swift]

lumina cant do text

[lucid swift]

Lumina, idogram, sd3 "The image is a top-down view of a photograph placed on a wooden surface. The photograph appears to be in black and white and has a vintage quality. The edges are clean and straight. The subject of the photograph is a person standing and wearing a long dress with a high collar and buttons down the front, with their hands clasped together in front. The person is wearing a mask or headpiece resembling a goat's skull with large, curved horns. The background behind the subject is dark, suggesting the use of a flash which highlights the subject and makes them stand out against the darkness. The photograph itself is not dirty or damaged. Below the photograph, handwritten text reads: "Fear is weakness.""

[storm saffron]

Hands are no better on Lumina either

PixArt (once I added polaroid)

[storm saffron]

I mean there are some sub-sets of furry culture to also try prompts for, but that might get removed. 😄

[cedar gale]

Tell me more.

[sterile pendant]

at the moment? no, not that I'm aware of.

[storm saffron]

I'd rather not. 😄 Have a bat minion instead

[cedar gale]

The information will be used very wise. 😄

As u know me. 😉

[storm saffron]

Research is left to the reader. 😉

[cedar gale]

I do thank you kindly for the nice bat minion tho. 😄

[bitter hearth]

gimme prompt for these 2

lmao the explosion became a christmas tree

[silver sluice]

can the SD3 API generate stuff that's like licensed/copyrighted? Like if you ask for it for super mario riding a skateboard on a sidewalk will it do that or will that throw an error?

[silver sluice]

oh wow so like fully unlocked lol cool thanks for the share

[vapid radish]

The Monster was:
In a foggy mountain pass, a repulsive creature emerges, but its form is indistinct and partially obscured by the dense mist. The winding road disappears into the thick fog, flanked by towering pines that create a tunnel-like effect. The fog muffles sound, creating an eerie silence and a sense of mystery and solitude. The creature's grotesque appearance is only vaguely visible through the haze, blending seamlessly with the eerie atmosphere of the journey through the unknown

The Cat:
cinematic photo film still of a little cat, (close shot), snow treading, rain dripping, fog filling, .shallow depth of field, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy, cinematic photorealistic, 8k uhd natural lighting, raw, rich, intricate details, key visual, atmospheric lighting, 35mm photograph, film, bokeh, professional, 4k, highly detailed

[low stone]

So they just released a new version of Lumina-Next-SFT which is kind of impressive and more prompt following. I prodded city96 to see if we can get comfy support for it.

[bitter hearth]

@vapid radish am I doing it right

[vapid radish]

Would you rather fight the monster-sized cat or the cat-sized monster?

[bitter hearth]

[lucid swift]

bs has nothing to do with furrys and it also exists in every other group of people

[storm saffron]

I'm confused. Anyway let's move on.

[dull star]

Yes I love that stuff like pop culture characters and etc weren't omitted, and gore/blood neither!
And yet bunch of artists were opted out and it doesn't change the quality of the model that much

And fine-tuned models are yet to come

[patent magnet]

Side by side comparison: left is SD 3 Ultra raw, right is the Upscaled + Eye-Corrected version

Side by side comparison: left is SD 3 Ultra raw, right is the Upscaled + Eye-Corrected version

Side by side comparison: left is SD 3 Ultra raw, right is the Upscaled + Eye-Corrected version

Prompt: anime art, 1girl, kemomimi, harpy, curvy, [white|black] hair, looking at viewer, [yellow|aqua] eyes, hood, fur coat, underbust, corset, night, moonlight, castle, sitting, squatting, Large_wings, hands between legs, from below, rooftop, wide_shot

[silver sluice]

I prefer the left on all occasions as far as general image composition but the right has better eyes, if i could take the eyes from the right and the composition from the left that'd be the winner, feels like upscaled is making the image overall worse

[patent magnet]

The upscale toned down the colors a bit, and some parts of the lighting made it worse, I agree. But the final quality is still superior to the original.

Yes, I could do that. Correct the eyes without the upscale. Very easy to do. 👍

[silver sluice]

so you have access to the sd3 weights and you're doing this in comfy or something?

[patent magnet]

I'm using the official Ultra API colab for the SD3 generation. And the post in Fooocus

[torpid forge]

I'm just making little goats

and clowns

[severe phoenix]

soo cool, pls whats the prompt for these

[brisk bay]

a fat man 40 years drinking beer funny picture

[fading fiber]

nice

[fleet meteor]

Amazing gens

[sick cedar]

can anyone tell me what version of the T5 model SD3 uses?

[patent acorn]

https://tenor.com/view/rick-and-morty-rick-laughing-terrifying-morty-gif-10529203898491388485

yo sd3 make me an image of maniac laughing rick from rick and morty

[low stone]

Llama3 prompt (it's so creative) and SD3. June 12th is gonna be great: A grotesque Trump mask made from human skulls and twisted metal hangs upside down from a rusty chain suspended above a trash-strewn alleyway illuminated by flickering fluorescent lights casting eerie shadows on crumbling brick walls.

[hallow lion]

2 more days

😄

now it's not 2 weeks!

just 2 more days

[desert garnet]

after that we only need to wait 2 weeks for 8b 👍

[bitter hearth]

I miss the good old days

[compact forge]

whats 8b?

[desert garnet]

8B is a unit in NieR:Automata, and one of the targets in the "YoRHa Betrayers" quest. 🤖

[compact forge]

great, cant wait

[bitter hearth]

@desert garnet since when did you become the sd3 ceo

[desert garnet]

2 weeks ago

[fallen swallow]

^^

[severe phoenix]

dude pls whats prompt for this

are thee maade witth ideogram?? 😲

[noble coyote]

Yes 🙂

[muted dove]

These are with SDXL, but I used national geographic documentary style shot of a living avian theropod

[hallow lion]

No it's real photo.

[remote holly]

Have developers of tools like a1111, fooocus or comfyui received the sd3 models in advance or will we have to wait a bit for integration?

Or can we use the model directly?

[jolly abyss]

I am pretty sure comfy has them and is ready for the "launch". I doubt that A1111 and Foooocus will support SD3 from the start.

Oh and swarm obviously too.

[remote holly]

ah too bad, in any case I can't wait to see the workflows that it will be possible to do in comfyui

[mild bramble]

Request: Can anyone try to push sd3 to its maximum capacity for realism in ai images

[lucid swift]

living avian theropods

[low stone]

You can use artisan in the other channel for that

[turbid grotto]

made a mistake in word "truck"

lykon is teasing with further trained 8b

[past flame]

Ah yes

The Hort

[patent acorn]

thats SO good

could have some simpsons comic parodies in sd3

[low stone]

news reporter with a microphone standing in front of a chaotic scene where a hideous gigantic monster is destroying buildings and stores

[lucid swift]

what is your promt?

[noble coyote]

"Sonic the Hedgehog does Sears Catalog?!?!?" 😄

SDXL, not SD3

[lucid swift]

i dont think thats the prompt

[patent acorn]

who pinged?

[sage burrow]

#artisan-3 message

[lucid swift]

nice! thank you

[steep widget]

New system available ♥ - https://www.youtube.com/watch?v=-mTT49TDxIE

[noble coyote]

SD3@ClipDrop

[rain current]

[noble coyote]

Too Daze alreddie

[pseudo stone]

12th june in 2 days tho

[raven fern]

just like Naruto says in the dub: "Believe it!"

[sterile pendant]

What if it was a dyslexic mixup in d/m/y vs m/d/y and they actually meant December 6th?

[dull star]

so close 🙏

its so good that we have a date

and 8B will come later and will be released as well

[mortal mesa]

is it known that its comfyui ready?

[mild bramble]

what fr?

[mild bramble]

it'll be like sdxl I guess so yea

[raven fern]

as far as i understood, it will be day 1 ready on comfyui

or rather, available day 1

[mortal mesa]

ya just lots of things have been said, wondering if reconfirmed

[raven fern]

maybe ask comfy directly :3

[ancient cape]

interally they are using comfy. so yes, its ready day 1

same for stableswarm, if you wanna generate grids

[dull star]

exacly, comfyui and stableswarm are day 1 things

anything else is probably gonna take a few days if not weeks, don't know

we should also know that controlnets are not going to be day 1 things, but we might expect 1.5 quality if not better due to the MM part of MMDiT

it's just that it may take more time for people to do research about it first

as long as regional prompting is day 1 and they figure out a pos embed fix so that we can get proper highresfix, I'm perfectly satisfied for the time being

[raven fern]

i dont care about controlnets right away, il be happy enough with t2i and i2i for now 🙂

[dull star]

oh btw they might've already started training 8B

[raven fern]

😮

[dull star]

Lykon's been posting pics

[turbid grotto]

I dream about memes finetune

[dull star]

then again, these must've used Ultra's workflow with like highresfix and everything

[dull star]

if finetuning is made easy, that's LITERALLY the first thing I'm doing

I can't wait

and since its 2B, we can actually train it

[turbid grotto]

hiresfix can be done only with sdxl rn?

[dull star]

no, with any Unet model to my knowledge

but with DiT models such as pixart and now SD3, highresfix is broken

[turbid grotto]

I will try too!

[dull star]

for Pixart, the entire image is noisy and distroted, and with SD3, the area outside the base resolution part is a blurry mess

[turbid grotto]

tiled upscale breaks bokeh and blur for me

[dull star]

if you go out of resolution range without fixing the positional embedding handling or using tiling, it does this (clear image in center, distortion on the outer edges)

from Alex (mcmonkey)

[dull star]

this is exactly why I am worried about them saying "just use tiled upscal"

then agian, lykon's image must've been with upscaling, and they look wonderful

maybe low denoising is okay? 🤷‍♂️

I really hope its just a case of fixing a part of the code and not a limitation with DiT

[turbid grotto]

maybe sd3 just smarter)

also tiled controlnet might be more effective

[dull star]

no its definitely upscaling

but the VAE is superior though

[tropic plume]

guys

[dull star]

this an image withotu highresfix and the woman's face doesn't look like a mess

the eyes look perfect

[tropic plume]

do i create images here or can i invite bot to my dms

[dull star]

I actually don't know, but you can use #1237459938901491852 channels of course
check out #artisan-faq

[tropic plume]

thankyouuu

[turbid grotto]

will it be slower?

[dull star]

no idea actually

[turbid grotto]

hope not 4 times slower

[dull star]

I hope its mostly vram difference and it can be solved with a tiled VAE

but I hope its not 4 times slower

[tropic plume]

it wont let me type anything in that channel

[teal fossil]

Is he using the 2B we'll get or a version of the 8B?

[dull star]

these are 8B pictures

[teal fossil]

Meh. That tease right before the 2B release is in bad taste.

[dull star]

honestly doesn't look any better than 2B right now

[hallow talon]

Has there been any update on what time on Wednesday we should expect the model to be released?

[dull star]

don't recall

just simply expect the worst and you won't be disappointed

I would wager midday to night in the US

I'd be surprised if they released it around greenwich midday time

[hallow talon]

as long as it comes out at some point on wednesday as promised I'm happy lol.

thanks 🙂

[dull star]

it cannot possibly be delayed, the model is trained well now

[hallow talon]

oh yeah, wasn't expecting it to be! Might've worded that wrong. I was just curious what time to expect it 🙂

[dull star]

https://github.com/comfyanonymous/ComfyUI/commit/8c4a9befa7261b6fc78407ace90a57d21bfe631e

GUYS

YESSSS

YESS THANK YOU COMFY

excellent!!!!!

we can leave prompts emtpy to try it all out

[bitter hearth]

Comfy

(the UI)

[hallow talon]

hopefully someone will make a workflow cause I can use comfy but I don't know exactly how to connect everything together properly (especially now that it got more complicated with SD3)

[remote holly]

How many gb takes sd3 to memory including all components ?

[gusty trail]

cliploader cliptextencoder not a very good name for t5

[dull star]

yeah idk why its still cliptextencodesd3

when there's a T5 as well

but I suppose its for familiarity

[dull star]

https://comfyanonymous.github.io/ComfyUI_examples/ not now, but later it will probably be here

[vapid radish]

Do we know how big the SD3 model file is going to be yet? I did just order another 4TB SSD just in case 😀

[dull star]

2B isn't going to be big, especially since its going to be at bf16 or fp16

it's T5 that's gonna be massive of course

which is 10 GB if you haven't installed it

[turbid grotto]

why T5 is so huge 💀

[dull star]

its a large LLM model (large large language model kek)

even if we are only using the encoder part of it

[teal fossil]

Don't jinx it now. 😉

[dull star]

I mean if they train it more, I don't mind

okay maybe I do, cause I kinda want to try the model offline now 😔

[teal fossil]

Do you know what Shift refers to?

[dull star]

no idea, but cascade also needed a shift node

yayy its alex!!!!1!

[teal fossil]

Pretty neat, but like early SDXL prompting where we experimented with G or L only or custom prompts for both... it'll probably turn out not that useful.

[viral plaza]

lolyeah had that conversation earlier

[dull star]

I think T5 will make a difference

[viral plaza]

swarm too

[teal fossil]

The noodles don't scare me no more. I have become one with the noodle.

[dull star]

oh absolutely 🔥

[viral plaza]

probably on launch day yeah

[viral plaza]

if you use Swarm it autogenerates workflows for you

great for getting set up and still lets you take the workflow itself and muck with it after at will

[teal fossil]

Alex linked a T5XXL version that is only the Encoder. It's smaller and apparently loads faster.

[viral plaza]

SD3-Medium-fp16 is a 4GiB file, it relies on a separate 1.3 GiB CLIP-G, 0.24 GiB CLIP-L, and optionally a 9.5GiB T5-XXL.
When downloading finetunes expect to usually only download variants of the 4GiB file, not the textencs

[teal fossil]

Hi Alex. 🙂

[viral plaza]

Also it's possible to store the model in FP8, making it only a 2GiB file when you do that

[dull star]

yeah its the encoder part only at like fp16 so its around 10GB

[turbid grotto]

what about q4?

[dull star]

it loads WAAAY faster than fp32

[viral plaza]

I don't think we have the software tooling ready for launch day for T5-4bit

hypothetically if we did it should work fine and would be a ~2.5GiB file

[dull star]

I wonder if a ggml implementation of encoder models such as T5 would help 🤔 (I'm a broken record)

[viral plaza]

however you can just not use T5 at all as an option, imo probably the best launch-day option

yeah t5 works well in ggml 4bit, just need a convenient way to shove that into comfy

[bitter hearth]

[dull star]

there's llama-cpp-python pip package or whatever its called

[teal fossil]

That's interesting... for SDXL (and v-1.5) training the TENC alongside the Unet makes all the difference. You think that's unnecessary now?

[turbid grotto]

thank you for the info ♥️

[teal fossil]

Basically the difference between mere Dreambooth and Finetunes / multi-concept-LoRA's

[viral plaza]

Sigma Shift, if you're familiar with sigmas in SD it literally just offsets those values, see here https://github.com/comfyanonymous/ComfyUI/commit/8c4a9befa7261b6fc78407ace90a57d21bfe631e#diff-6c3064a93127b01542c5772a797c9d356b876fc9940ec14951f95ff8ea270656R172-R204
or for a simplified version of the same code:

[teal fossil]

Taggui can apparently run Salesforce/blip2-flan-t5-xxl in 4bit. Maybe that helps?

[dull star]

well there's bnb4bit

we can load it in 4-bit, but storing is a different question

[bitter hearth]

no clue what im reading

[viral plaza]

it's quite possible that training the textstream will replace the power training the tenc.

probably still training the tenc is a powerful tool, but it's less valuable with the streams, and also harder with the 3 tenc setup dealio, so the tradeoff made the most sense to just not include tencs

[teal fossil]

I've been hearing about Sigmas more and more for a few weeks, but apart from "something something detail" I don't really get them. So changing the Shift will have a direct impact on the output and should be experimented with?

[viral plaza]

and for reference when I say this, I'm basically personally the reason SDXL has tencs in the model lol, others wanted the tenc separated for XL but i fought to include it because training it is so worthwhile and the tencs are only 1 out of the 7 gigs of space the model takes anyway

balance is different for SD3 so I can't really argue it on this one

[viral plaza]

bnb is so weird and jank with all the limitations like this. ggml and exllama just do things so much better

[teal fossil]

So of course they would have to be loaded alongside SD3 2B to be trained with it (excluding T5 since that would need more than consumer hardware)?

Textstream == the new Ddit architecture?

[dull star]

I saw in the terminal that apparently it's an immovable brick due to bitsandbytes, so could it be that you cannot offload it to RAM after you have used it?

[teal fossil]

It's ~1.5GB for L & G now. Are they exactly the same Clip's SDXL used or custom ones?

[viral plaza]

Uh so the short of it is:

• in early diffusion models, we had timesteps 1-1000 exactly
• in the modern era of diffusion, we now have dynamic steps (eg 20 step or 50 steps or etc) and that worked by converting to approx timesteps (so eg for 50 steps, you multiply your step by 50 to get a timestamp value) and adding the sigma value to represent the timesteps in a way the model can now actively process
• the shift effectively curves the timestep space, so it can spend more time in the early (structural) steps or more time in the later (detail) steps

If you've ever used my Dynamic Thresholding toolkit in auto/comfy/swarm, the CFG Scheduler feature is very similar to sigma shift (albeit of course using the CFG rather than sigmas to push this preference schedule)

in the case of SD3 I think sigmas are basically just linear by default until you apply the shift (vs other models had more of an algorithm to em)

(the sigma goes through an embedder to turn into latent magic inside of the model)

[teal fossil]

So Sigma Shift 0.5 -> more time for structure Vs Shift 2 -> more time for detail?

[viral plaza]

TextEncs don't have to be loaded, you can precalculate embeddings separately. Better for VRAM that way. Kohya and other trainers suppor this out of the box iirc

the textstream is part of the model, it can't be separated. Presumably trainers will let you select which parts of the SD3 model you want to freeze or unfreeze

[teal fossil]

Like we could (later down the road once it's fixed) go 0.5 Shift for the initial image and Shift 2 for the HiresFix / Ultimate Upscale / whatever?

[viral plaza]

the "mm-" in "mm-dit" is "multimodal", referring to the multiple streams - in SD3 base there's text & image streams, but hypothetically if you go to add eg a controlnet, you'd just add a controlnet stream

[viral plaza]

SD3 is, for all official purposes, CLIP-G + CLIP-L + T5-XXL. While I think in practice everyone's gonna just yeet t5 out a window and focus clips-only, the official reference publication stuff is all including T5

[rugged nova]

https://huggingface.co/Alpha-VLLM/Lumina-Next-T2I Uses RoPE scaling to generate at higher resolutions. Any idea if SD3 will also be capable of this?

[dull star]

#🆕｜sd3 message

this was 5 days ago, idk if something changed

but we can do tiled upscaling for the time being

[viral plaza]

this graph shows blue = no shift
red = shift 3
blue = shift 0.5

red as you can see runs through timesteps much faster early on in structural phase, then spends more time slowly on the later detail steps

SD3's default reference recommended shift is 3

it makes sense to play in the range of 1.5 to 3

you probably wouldn't ever do below 1

[dull star]

nice, this is a good demonstration

[viral plaza]

not at launch, and probably won't be the exact same code (I think RoPE doesn't apply here iirc, the posembeds are weirder) but most likely someone will find a way

("RoPE is "rotary positional embeds" and "RoPE Scaling" is a technique to scale RoPE, SD3 has its own pos embedding logic)

[teal fossil]

I still have to wrap my head around the fact that very early day embeddings are back on the menu... xD

[viral plaza]

TI embeddings have been awesome the whole time

they are incredible on XL

i wish people would use them more / write good tools for them

it's theoretically possible to train a small TI in a few seconds (some people even published code that does this before, then deleted it off github argh)

LoRAs are more flexible obviously but for single-concept training TIs are hard to beat for speed/quality/usability

[dull star]

also aren't they going to work across all 3 (or 4) model sizes?

since they use the same clip?

[dull star]

the only time I remember using TIs for fun were in SD2 days

idk what happened, but stylistic TIs were really good in quality

we had like midjourney TIs and Greg Rutkowski TIs

they were nice and super small in filesize thanks to TI's nature

would making a TI for SD3 have nearly the same procedure as making a TI for SDXL?

[fleet meteor]

we´ll see in 2 days 😁

[dull star]

eh but I'm inexperienced in training 😔

I'll have to wait for random rentry blogs lmao

all I did is use google colab and then later some super braindead easy GUI to train Loras for SDXL

[fleet meteor]

I think I was decent in training untill I tried training a lora / checkpoint with more than 30 images

My model broke

[dull star]

damn

https://github.com/Nerogar/OneTrainer okay this has a super easy installation method and it has an SDXL Embedding preset

[turbid grotto]

It is super easy to use

and convenient

[raven fern]

oh shit.. waddup 😮

[dull star]

ah, CEST has 1 day left

[dry wave]

I found TIs amazing in SD 2.1 but in SDXL they didn't worked well for me

also I had the feeling that TIs overfit much worse than unet training

[viral plaza]

Yes. For that matter an SDXL TI trained today will work on SD3 when it's out

[viral plaza]

that's not how timezones work but lol

[dull star]

excellent, thanks

[raven fern]

damn son.. give me :3

[wild remnant]

@raven fern lol

[coral sable]

hype is unreal, last day (or 2) of waiting

[raven fern]

[wild remnant]

@raven fern

[raven fern]

❤️

[coral sable]

release will be as planed on schedule?

[raven fern]

all according to keikaku

[agile hornet]

When it drops on huggingface I should be able to just grab the model and toss it in A1111 right?

[low stone]

Nope

There'll be comfy support at launch. Not a111

[agile hornet]

noooooo

[low stone]

Comfy is written by stability ai. The other guys are their own separate devs so they'll have to work on integrating afterwards.

[fleet meteor]

Aaa I can´t wait to try it

[raven fern]

132k lines changed 😮

[molten valley]

it will be on huggingface right ?

[dull star]

very epic new nodes

[low stone]

Ooooh very exciting. Didn't realize they pulled it into main branch already

[viral plaza]

you can use it in Swarm as well on day 1, which has a friendly interface like auto1111 does

[viral plaza]

ye

[viral plaza]

most of that is just the tokens file for t5 lol

[viral plaza]

yep lol everything goes straight to master for comfy

[low stone]

Sshhh don't say that, it's all code that you guys toiled away on. Just go with it. 🙂

[viral plaza]

8B started training months ago but yes it's regained focus for the training team now that 2B is about ready

[dull star]

excellent!

[viral plaza]

(scrolling through backlog sorry if this already got an answer) T5-XXL, encoder only, as found eg here https://huggingface.co/mcmonkey/google_t5-v1_1-xxl_encoderonly/tree/main

[coral sable]

updated and rdy ^^

[viral plaza]

if you update swarm you can get a valid SD3 workflow out of it too lol

[low stone]

Is there a new ksampler as well or still using the same one?

[dull star]

nope

all of the nodes I've shown you are the all of the nodes I've shown you

🔥

[low stone]

Ah

[agile hornet]

Not sure if I can run Swarm, I got a 2080TI with 8GB of ram and all the other ones work well but I heard swarm uses more recources to run is that true?

[viral plaza]

same ksamplers as normal

[viral plaza]

Swarm uses less resources than auto does lol

it's way more efficient

[agile hornet]

nice

im gonna set it up today then

[dull star]

what about schedulers?

can we use all samplers and schedulers?

how does the flow matching influence it

[viral plaza]

You generally want Euler+Normal, but any Sampler that isn't Ancestral or Stochastic (ie SDE) should work

ancestral/stochastic are incompatible with flow

[low stone]

That's a good question. Is there any benefit for the perturbed and align your steps nodes with sd3?

[raven fern]

go with the flow 🙂

[dull star]

we'll see, about that..........

[viral plaza]

AYS won't work out of the box, but can be retrained for SD3

[raven fern]

so nvidia has to release something new for sd3?

[dull star]

it's gonna be cpu by default?

[viral plaza]

alternate guidance options like SAG and PAG will probably just flat not work as-is and need new research to make things like em

[raven fern]

yea some new research

[dull star]

and freeu neither, cause its specifically for unets

[viral plaza]

for AYS they published a unique list of sigma values to use per model arch, so they'd need to rerun it and get a list specific to SD3

[raven fern]

yea basically anything unet related wont work

[viral plaza]

oh yeah freeu is entirely out

as is cacheysamply :(

my precious baby, killed to death twice in a row

[raven fern]

right

i personally like the Inspire pack ksampler for ays, cause it's already built in, dont have to use a lot of nodes for AYS

[dull star]

at least we can experiment with stuff on day one, like messing around with text encoders, textual inversion embeddings, tiled upscaling

what about regional sampling?

[coral sable]

ok I'm convinced to try SwarmUI, isn't it on Pinokio, cant find

[low stone]

I should try my stone man shooting money from his hands regional prompt on sd3. Seeing as it understand placement keywords, it can probably just do it natively

[raven fern]

wait, i just noticed comfy did a small fix to cosxl edit models, does this mean they fixed the artifacts? im gonna check 😮

[dull star]

oh shit I just realised

top-bottom memes will work perfectly due to regional sampling

[carmine blaze]

is API call the only way to test SD3 right now?

[dull star]

currently, yes

but just wait till 12th and you can use it offline

[carmine blaze]

sweet thx

[dull star]

you're welcome

[raven fern]

didnt fix the artifacts, welp

actually wait, il try with some other pics

[dull star]

hmm I wonder when we'll get inpaint models

ngl it would be perfect for SD3

[raven fern]

man greatness awaits 🙂

[viral plaza]

SwarmKSampler (built in with swarm) has it too

[raven fern]

nice

[viral plaza]

swarm is super easy to install natively, you don't really need an install manager like some other uis need https://github.com/Stability-AI/StableSwarmUI?tab=readme-ov-file#installing-on-windows

[low stone]

I think I was able to achieve just directly what I did with regional promoting with this one. One of the things that's so impressive about sd3 so far is the variation available between seeds. It's not just the same image from a slightly different view each time. Every one is very different which is great.

[raven fern]

@viral plaza do you know when we will get an edit model for sd3, like cosxl edit?

[viral plaza]

was just a bugfix

[viral plaza]

don't need an inpaint model really

[viral plaza]

here see for example i did an inpaint with sd3 medium release candidate directly

my lil terminator kitty looks awesome

i set the creativity a bit too high so there's some discoloration on the edges but not too bad unless you're looking really really close

this isn't some SD3 magic btw this is just Swarm's inpaint code working well lol

can do the same with XL

[viral plaza]

iiiddunno. Hopefully somebody finds a better way to make one than just ip2p. If not... well somebody's gonna make an ip2p on sd3 at some point

[raven fern]

yea

also, look at Alex flexing with the release candidate version 🙂

[dull star]

wow that works well for a base model

like yeah its 0.9 strength, but still

[viral plaza]

can of course also just do the mask better to prevent the discoloration

[dull star]

kek

[raven fern]

nice kitty

[viral plaza]

oh actually turn off Mask Shrink Grow and this looks really good

[dull star]

how come a1111 users didn't switch to stableswarm yet, it even has these convenient features similar to it

[viral plaza]

... anyway tldr point is, if you use a good UI like Swarm and fiddle settings a bit, you don't really need an inpaint model

[viral plaza]

main reason for most people aren't using Swarm is just cause they don't know about swarm or don't really comprehend what swarm is

most people once they try swarm never go back to anything else

[dull star]

its not 1:1 to a1111, so therefore it must be bad

[viral plaza]

comfy users it's a 100% no-brainer to use swarm, auto webui there's differences to learn but ... like, most of the differences are improvements so lol

the one pain point for auto users is if you have old auto extensions you really like - you can usually find a comfy equivalent, but if you don't like the noodles, it's awkward

[coral sable]

waiting installation to finish :c

[viral plaza]

swarm is generally less reliant on extensions though, a lot of stuff is built in

you only need extensions when you're getting really really crazy

[coral sable]

meanwhile octopussy v2 (SD3)

[raven fern]

stable swarm seems very well built, il try it one day

[coral sable]

not sure if she be eating herself or that's how she is

when lambo sd3🤣

[ocean lance]

would this work too? i tried with pixart and the output was identical https://huggingface.co/city96/t5-v1_1-xxl-encoder-bf16

just saw yours is roughly the same size

[viral plaza]

that looks like it's the same thing just bf16 instead of fp16, so yeah

[ocean lance]

do you know why the team chose xxl over xl? xl seems to be more popular just from searching for fine tunes of xxl

[viral plaza]

they literally just grabbed the biggest one they could find im pretty sure lol

[ocean lance]

bigger is better 😄

[hallow lion]

Prepare the consistency nodes for the arrival of SD3.

the only thing left to conquer - consistency.

we have it in lighting btw

😄

we just need shape/texture consistency now

1 day left

dont predict the arrival of the mesiah

well in this case we can

[faint breach]

can sd3 make it 2 more days from now though? can it time shift?

[frozen lynx]

guys we gotta think of something to make time go faster so it's 2 days from now

[viral plaza]

no but effectively-time-travel has been proposed before

("anticausal" = model that's trained to solve in the reverse of the direction of time)

[faint breach]

using improbility fields? thonk

[viral plaza]

i was thinking MM-DeLorean

[faint breach]

this is heavy

[frozen lynx]

like .. if u move fast enough u time travel to the future right? we all just gotta get on a spaceship that can travel at near light speed

[faint breach]

naw. that'll just age us infinitely before we get back to the same moment

[faint breach]

if i eat that will it be 2 days from now?

[ocean lance]

maybe from a food induced coma

[compact forge]

whos ready for unexpected delay on release day like last time 😆

[sterile pendant]

And then all the selfish people that will complain about their free toy being late

[compact forge]

obviously if you delay something right as people expect it to come out

[frozen lynx]

I think I figured it out. so like time is a measurement of light moving, so if we can move at light speed, it'd be like pressing pause on a video game but then we could just skip to any point in time instantly so to get to SD3 launch all we gotta do is go fast like sonic

[desert garnet]

damn selfish ppl who pay for api access

[sterile pendant]

That already have access to it and whose credits are still going to be in their accounts...

[compact forge]

yeah man fucking leeches

i hate people so much

[ocean lance]

is this the first model that hasn't leaked?

[desert garnet]

we truly live in a society

[sterile pendant]

But for real, people will complain about anything. Even when it does come out on time, people will complain that they can't run it on their 2gb vram gpu and 8gb ram laptop from 10 years ago

Or that it "suck"

Because no bobs n vegine

[compact forge]

thats valid tho

[desert garnet]

too many bobs and vegana,ban model pls

[sterile pendant]

To who? 90% of the complainers are just people from countries where prawn is banned, like India and China.

They just happen to come from the two largest countries on the planet

[compact forge]

lol you think they gen images cause they have no access to porn? 😆

[sterile pendant]

It's illegal in both those countries.

[desert garnet]

didnt u know? ppl in china and india have an iron mask on their head with tiny wires coming out of it

[compact forge]

so im asking if they are using ai as a workaround?

[viral plaza]

we still got 2 days for somebody to be stupid about it

like the person that saw "stable audio open" uploaded to HF and stole it and reuploaded a "leak" with the word "open" removed

(i really wonder what the goal was. Was it, like, clout chasing? or... why tho?)

[compact forge]

do they even have a set of closed beta testers like sdxl?

[viral plaza]

nope, only a few business-tier partners this time

vs XL for example was sent out to anyone with a .edu address

[ocean lance]

so why not anyone with .com address? ;p

any sensible training parameters to recommend for training a lora?

[viral plaza]

that'd be a @lavish osprey question

[bitter hearth]

Is there a free way to use sd3?

[ocean lance]

when the weights are release you can run it on your own computer for free

[bitter hearth]

Thanks

[noble coyote]

Fabian@GLIF

[junior dune]

What’s the estimated vram requirement for sd3?

12gb and 32gb ram?

[junior dune]

Cause auto1111 lets me remotely share and access the website on any network

[ocean lance]

my guesstimate is an 8gb gpu and 32gb of ram should be okay

since T5 can be loaded in CPU ram

pixart is a 0.6B model, requires 1.5 GB of VRAM

so roughly 5gb for the mmdit

loading clip g and clip l uses 1.8gb of vram

[viral plaza]

you can just not use t5

at which point the reqs are <= sdxl's reqs

[viral plaza]

swarm does too!

here's the docs about it https://github.com/Stability-AI/StableSwarmUI/blob/master/docs/Advanced Usage.md#accessing-stableswarmui-from-other-devices

[ocean lance]

comfy only uses 3.6gb to run sdxl

does that make the 2B model excluding the clip models less than 1.8gb?

[junior dune]

Awesome! Thank you

Does it generate a link you can access on mobile on another network like A111 does?

[viral plaza]

SDXL minus CLIP is ~5.5GiB, SD3-Medium (2B) minus textenc is ~4GiB

[viral plaza]

mostly yep, check the docs i linked, use cloudflared option probably

mobile support is a liiil wonky atm

it works but not a great experience tbh

[ocean lance]

is comfy offloading the clip to cpu maybe?

[junior dune]

Sick

[viral plaza]

if you're using only 3.6GiB of VRAM usage then yes comfy is offloading things for you

[ocean lance]

so might be able to run on my 6gb laptop card then :>

[viral plaza]

yeah probably

[ocean lance]

[ocean lance]

do you know if it's any quicker for our friends with apple computers than sdxl?

[viral plaza]

https://x.com/argmaxinc/status/1790785157840125957

argmax (@argmaxinc) on X

On-device Stable Diffusion 3
We are thrilled to partner with @StabilityAI for on-device inference of their latest flagship model!

We are building DiffusionKit, our multi-platform on-device inference framework for diffusion models. Given Argmax's roots in Apple, our first step

Twitter · May 15, 2024, 4:43 PM

[visual hamlet]

哈哈哈

[teal fossil]

@viral plaza Does Swarm's Inpaint put the whole image through the Vae (degrading the image over time) or is it using a stitch-approach to mitigate that?

[viral plaza]

there's an Advanced param under Init Image that controls that, defaults to enabled (ie stitches the image back to prevent VAE damage)

[teal fossil]

Perfect.

I think I'll try to get used to Swarm more with SD3. (as I can still use my overconvoluted crazy Noodle workflows in that tab)

[ocean lance]

does swarm have tiled upscale?

[teal fossil]

Kinda like KoboldCpp + SillyTavern.

[remote holly]

It will be possible to use sd3 online tomorrow ?

Online and free

[haughty jasper]

Where can I use SD3?

[cobalt moon]

tomorrow. OK?

ComfyUI just update its compatibility to SD3

the Midnight is coming

uhhhh... actually are they going to just do the license announcement along with open weight

[viral plaza]

yknow not built in currently in a convenient way but i oughtta add that before sd3 launch eh

(can do in comfy tab of course)

[haughty jasper]

Is SD3 available for download now? I thought I could experience it on Discord.

[cobalt moon]

nah the Discord is one is kinda like Midjourney, which need a subscription

[haughty jasper]

But I couldn't find the SD3 bot in Discord.

[cobalt moon]

Artisan. I suggest you to wait til tomorrow when the announcement is here

[haughty jasper]

okk

[remote holly]

Sd3 on hugging face ?

[viral plaza]

added support for https://github.com/BlenderNeko/ComfyUI_TiledKSampler - if you install that, under Refiner you'll get Refiner Do Tiling checkbox. It's not perfect but if your RefinerControlPercentage isn't too high it does its job well

[cobalt moon]

yeah

[viral plaza]

#🗣｜artisan-support-feedback

[remote holly]

Ha cool , i hope it will run correctly because stable cascade have lot issues with diffusers lib

[ocean lance]

Should've told you I made a tiled sampler 😛 It adds the tiles together using a feathered mask technique which is pretty seamless. https://github.com/bash-j/mikey_nodes/blob/main/mikey_nodes.py#L3015

[viral plaza]

ooo
any chance you'd be willing to PR the bare minimum version of it into swarm? (I'm intentionally trying to keep swarms dependencies at bare minimum, no large packs except when absolutely necessary)

[ocean lance]

I can try, where would it go? I have a function that splits the image into tiles, then a function that stitches the tiles back together

[viral plaza]

as a parameter to, or variant of, SwarmKSampler: https://github.com/Stability-AI/StableSwarmUI/blob/master/src/BuiltinExtensions/ComfyUIBackend/ExtraNodes/SwarmComfyCommon/SwarmKSampler.py#L115

[storm saffron]

@viral plaza do you know if SD3 will work with tensorRT (not sure anyone would have tried it yet)?

[viral plaza]

it will yes, nvidia will publish stuff for it

[coral sable]

bro Swarm has everything, you were right. I'm rdy for SD3

[bitter hearth]

has cat

[coral sable]

lots of cat

[lavish osprey]

params for lora training? It would greatly depend on the type of lora, the dataset size, the goal, etc

at the moment we didn't test any style or character lora (well, we briefly did but then we had to move out to other stuff).
We did lots of aesthetic or "fix" low rank training

[prisma zenith]

So I read thru the history to try to understand. But I still have a few questions. Woudl appreciate if someone could fill me in 😄

SAI is releasing SD3 Medium, but there could possibly be a small and a big? Also what is the impact of the 2B vs the 800M vs the large? Is the impact more on the model's ability to translate text into images and understand what it's trying to create? Or will it have an impact on image quality too? (my understanding is that it's a bit of both because it will have less latent's to draw from right?)

Finally what is SD3 Ultra? is it a comfy UI workflow when using the SD3 Api? Or is it the large model?

Thanks to anyone who takes the time to answer!

[lavish osprey]

It's style aligned well enough to allow most types of lora training imo

[lavish osprey]

param numbers impacts all type of things, from general knowledge to inference capabilities (ie the ability to create new concepts from the ones you know)

[lavish osprey]

that being said, 2B mmdit is likely all a human can ever need.

[bitter hearth]

I also did that prompt and got a similar image lmao

[coral sable]

SD3, same prompt(lots of cat)

[prisma zenith]

thank you for clarifying!

What about what SD3 Ultra is? Is it a workflow? or the larger model?

[lavish osprey]

it's only "Ultra" in theory, not "SD3 Ultra"

[prisma zenith]

but what is it? a model? or a workflow to make the most of the output from the model currently on the API

[lavish osprey]

what's in Ultra is a trade secret, you can think about Core and Ultra in terms of msg, which is salt on crack, while base models are just salt.

[prisma zenith]

I see, so Ultra is basically the model that is available via API is that it?

or is that the one available via stable assistant?

[lavish osprey]

ultra is not a model

there is no model named ultra

[prisma zenith]

so what is ultra then? a workflow?

[lavish osprey]

ultra refers to an api endpoint, what's behind that is not for me to say

[prisma zenith]

or was is this a meme/joke that i'm not getting xD

[prisma zenith]

I see! i get it now

thanks!

[lavish osprey]

https://tenor.com/view/uncle-roger-king-of-flavour-gif-18146501

enjoy some ai generated pizza

[bitter hearth]

[prisma zenith]

this is the one on fireworks right?

[coral sable]

that's evil, I'm hungry

[lavish osprey]

no, Ultra is provided directly by us, not Fireworks

[lavish osprey]

then enjoy a dog on a ball on a surfing board

[lavish osprey]

@viral plaza give me the yellow name so I can start banning people who put pinepple on cheese and tomato sauce

[prisma zenith]

got it! i'm looking at the api now. This one has Sketch & Canny as controlnets right?

[lavish osprey]

elaborate on "this one"

[prisma zenith]

https://platform.stability.ai/docs/api-reference#tag/Control/paths/~1v2beta~1stable-image~1control~1sketch/post

[coral sable]

tested some MJ prompts with SD3

[lavish osprey]

I don't remember if Structure has canny to be honest

[bitter hearth]

runs away

[viral plaza]

For legal reasons I would like to clarify that Stability AI does not compare its services to crack, Lykon's choice of phrasing is entirely his own

[viral plaza]

for not-pissing-off-reddit reasons I'd like to clarify this is also Lykon's own phrasing gasjlasg

[lavish osprey]

let's just piss off reddit

[lavish osprey]

https://tenor.com/view/uncle-roger-msg-cocaine-of-cooking-gif-14767225509662422148

[teal fossil]

I so hope that this is true.

I can't wait to try Lora & FineTuning...

Can't wait how different the Textstream (& Imagestream?) will train compared to the XL Unet+Tenc.

And I wonder how long it'll be until Kohya & OneTrainer add manual Clip-G/L training for concepts.

And dedicated simple Embedding training.

[lavish osprey]

you don't really need to finetune TEs for concepts

historically NAI used SD1.4 TE without any change

and Pony finetuning TEs kind of destroyed them, to the point it lacks basic knowledge like "a bank"

the first 5-6 months of SDXL lora finetunes didn't touch the text encoders and they mostly work very well

even with made up words as activation

(as a matter of fact, I'd suggest to keep the tes untouched and just use them for preprocessing)

[ocean lance]

say I have a 50 photos of my cat and I want to train a LoRA, where should I start? Would it be any different to training a LoRA on 100 paintings by Monet?

[storm saffron]

Is that because the text encoders will just output tokens for whatever you put in there anyway? Even if it's completely nonsensical?

[teal fossil]

And it has problem with colors and color bleed.

I have high hopes that SD3 got rid of most color bleed.

[teal fossil]

I'll definitely try that.

For SDXL at least training the Unet only was never enough for new concepts (at least during my tests) - adding the Tenc always got it into the right direction.

[dry wave]

each token gets an embedding, yes

that's how the original dreambooth paper worked: they used a "non-sensical" input token and trained on that

instead of using one token you can also just use a bunch of tokens (like your name)

as more rare (and non-sense) the token is as harder it is for the model to learn from it but there is also much less overfitting and damaging then

[dry wave]

it worked also for sdxl, but training the unet takes much more time than training the te

[lavish osprey]

tokens are the input, not the output, but your intuition is correct.

[lavish osprey]

^ this is mostly correct. The model will produce an embedding of your prompt depending on the tokens. So there sre still going to be vectors that the Unet/DiT can "catch" to understand new concepts

with SD3 you also have more stuff you should check when training, like "did I ruin text understanding" or "did I create conflicts among the various text encoders"

[amber nexus]

If the SD3 2B is released tomorrow, will I be able to use the inpainting or upscaling features directly in ComfyUI?

[outer cloak]

A quick dumb question is there a fixed time for the release like 12PM PDT or something?

[teal fossil]

Another thing - I started pruning my LoRA's per Weights - for some concepts that was highly effective and countered the detrimental effects of my LoRA's on the base Models.

Will that be possible / necessary with the new architecture, or is it structured differently?

[dry wave]

what do you mean with "per weights"?

but loras work technically the same everywhere. They are not specific for diffusion methods

[teal fossil]

You can load LoRA weights individually (can't remember what the Comfy Node was called exactly).

[teal fossil]

But aren't they like a "UNET jank" for XL?

[dry wave]

each matrix has it's own pair of lora matrices. You can remove them individually, yes

[dry wave]

I don't know what you mean with unet jank.
But they are not limited to unet anyways. You can have text encoder loras, too

[bitter hearth]

[vapid radish]

It's funny that if you mess up the prompt with SD3 it really tries to follow it, with hilarious results.
"vampire with fangs, wearing a black cape and black with red lining inside shoes"

[bitter hearth]

https://tenor.com/view/unico-uni-mario-galaxy-goomba-unicouniuni3-fight-punch-cat-gif-835764120138785088

its a goomba

[desert garnet]

its wednesday 12th on new zealand already,where SD3 ?

https://tenor.com/view/discord-who-asked-me-looking-for-who-asked-travolta-meme-travolta-confused-gif-27133622

[gusty gale]

two weeks =]

[desert garnet]

[teal fossil]

True. But it was just established we should try not training the Tenc first.

We'll see I guess.

Someone else wrote "jank" as in LoRA's interact with Unet's in a strange way.

[dry wave]

I don't know what you mean 😅

if you train the TE then you should train it first

or you train it together with unet

but training unet first and then te sounds wrong

regarding lora and unet: I have no clue what this "strange" interaction should be

in the end a Lora is doing nothing strange, it's just a weight update

[dull star]

Generic phone selfies look so good YESSSSSS, I hope CCTV and other styles work as well as 8B, this is good news

[bitter hearth]

you can just take a selfie yourself

why gen them

[sick cedar]

Yeah. Thanks for this m8!
I was making sure i could run it, because i was having troubles running Pixart with the same encoder.
I turned out in the end that i was just doing something wrong with the workflow, because it works great off of thwe CPU now.
Thx again for the reply. 🤗

[tropic aspen]

So, just want to mention we're around 1 day away, and we still don't know how the licensing for SD3 works despite being told it would be explained before SD3 launches

[storm saffron]

Generic selfies aren't exactly difficult.

[dull star]

I don't like how this one looks

why did the AI dude use portrait mode 😔

[dull star]

this one looks 300x more convincing to me

generic ceiling light, no depth of field, just a dude smiling

[storm saffron]

There, badly lit, mildly out of focus, terrible selfie

[dull star]

this one's better than the last

but I still prefer SD3's selfies

[storm saffron]

I'd prefer to see examples of things that can't be done already in SDXL.

[bitter hearth]

[bitter hearth]

Tell me one thing SDXL can't do

[storm saffron]

Bow and arrows.

SDXL

[bitter hearth]

Kek

Weapons are annoying indeed

Don't think this basic version available can do any better

[storm saffron]

It can do it, sometimes, a bit better

[desert garnet]

6 dogs standing next to each other,each one is of a different breed and has a different fur color

[storm saffron]

Or a woman talking to another woman in the street, one has blonde hair and is wearing a red dress, the other is wearing a purple dress and a baseball cap

I mean it was fairly ambiguous.. 😄

more specific (for SD3 purposes)
a woman talking to another woman in the street, one has blonde hair and is wearing a red dress and a fedora, the other has brown hair is wearing a purple dress and a baseball cap

SDXL has mixed hair colours now, and only baseball caps

[lavish osprey]

but this doesn't look anywhere as realistic as the one I posted.

[lavish osprey]

this is likely the most realistic one you posted but still looks like "good cgi" to me and not a real photo