Implementing Deepfloyd Support

[proven trench]

Heh, I figured I should open a thread on this, though it's likely been discussed elsewhere already.

Is there any knowledge of how difficult it would be to implement Deepfloyd support into InvokeAI, either currently or with the upcoming Nodes setup? I'm not really in the loop with how different it is from a typical Standard Diffusion setup, but I have seen the early results from it and it's definitely a remarkable advancement.

Be curious what the prospects are for having it be supported in Invoke are. 🙂

Here's a good video covering just was DF has to offer!

https://youtu.be/4Zkipll5Rjc

[tardy parcel]

https://tenor.com/view/whisper-surprised-shocked-gif-11804296

nodes

[wide parrot]

Just a note: if you look at the specs it needs 16GB VRAM MINIMUM (24 for full function) which means most people can't run it. Also a slap in the face yet again to our being forced to use Nvidia (which has bad Vram/cost) and not AMD.

[ancient pawn]

On top of that Stability isn’t letting others fully use it yet - licensing is for research only

Bit disappointing

[shut moat]

I’m guessing the answer is “let’s finish the core stuff first” but is there anything specific planned around nodes extensibility in the direction being able to like clone “my-node” repo into a folder and the node just shows up?

(related to this topic since deepfloyd is probably the best example of something y’all wouldn’t want to merge in at this point for the licensing reason)

[tardy parcel]

That’s how it works. Add a node (python code) and it shows up in the node editor. Having this show up in the classic ui is a wip, it’s a bit trickier to engineer that.

[solid valve]

It would be cool if you could create your own node workflow, then make it show up in the tabs along the left and look like the regular text2img/img2img interfaces...

[shut moat]

Yeah I got the basics, I hadn’t actually looked at the loading logic - how does it know where nodes are? (Don’t really need an answer here, I can just look myself later)

[tardy parcel]

At the moment, if they are in the invocations folder (in the repo) they will be picked up. We’ll make that more approachable for end users

[shut moat]

Awesome

[proven trench]

Well, it's got an CC - BY NC license on it, so you can use it, just not for commercial yet

To be fair, I heard a rumor that somehow part of DF will eventually replace OpenClip in the Stable Diffusion models? I have no clue if that's true, but I heard that DF and SD may not remain separate eventually

[ancient pawn]

Right. And the license when downloading it emphasizes research

But, fair, could just end up being used to research spaghetti text.

[shut moat]

So I poked around at this since I’ve wanted to hack more on invoke and play with Deepfloyd. I’ve got it working with basically a node per stage of the pipeline which is basically t2l -> l2l -> l2i.

The 3rd stage is just using an existing stable diffusion upscaler, so I probably can make that work without a new node.

For the other two, I’m happy with having them be their own nodes (especially since they probably aren’t something to have in the main codebase?), but I’m curious what the “right” nodes thing would be. I started thinking about trying to genericize latent.py, ModelManager, and probably a handful of other places which seemed like the right direction?

[ancient pawn]

I think you did the right approach

I.e., creating dupes of t2l or l2l with deepfloyd. At least as I understand it, we’d want to treat different pipelines as different nodes, not try to merge it all into one massive node

As far as the SD upscaler, I believe that hasn’t been incorporated into Invoke, so adding as it’s own node would also probably be the thing to do

[wide parrot]

PromptEmbeds has "latent1" and "latent2" ouputs. I assume based on where they go those are pos and neg but need to be labeled as such. Also latent outputs going into embed inputs with the same color (these could be correct, but if so they need to share labels or it looks like a mistype).

[shut moat]

PromptEmbeds has "latent1" and "latent2" ouputs.
Yeah I was playing around with something generic there and didn't fix label

[tardy parcel]

my god it’s beautiful

[wide parrot]

Also for much later feature enhancement: multi-model sequential stage latent node.

[sweet silo]

For me looks like embeds it's conditioning, but not sure for real what resends
And first prompt embeds looks like compel node

Also to test it for me I need "Enable sequential offload" flag unfortunately 😄

[shut moat]

I haven't looked at compel much, conceptually I think they're the same thing, but the compel node looks tightly coupled to CLIP and SD pipelines, where Deepfloyd uses T5 encoder and has some slightly different pipeline methods.

[sweet silo]

Just to be clear - there is no pipelines from diffusers in nodes, except for generation nodes(l2l, t2l, ...) and even here it's custom pipeline and it can be disappear

[ancient pawn]

I think his point is that compel may be more of an SD implementation vs being workable with Deepfloyd. @scarlet skiff have you thought or been asked about deepfloyd + compel yet?

[scarlet skiff]

i haven’t, but it deliberately wrote compel to be text encoder agnostic so @shut moat you ought to be able to give it the t5 text encoder and it’ll Just Work

i mean, assuming that deepfloyd also uses the TE’s last hidden layer, which it may not do

[shut moat]

That’s awesome, I’ll try it out tonight.

[shut moat]

Looks like there’s a few things to peel through to get it to work. The compel node uses model manager which expects SD. Bypassing that and I hit a couple other things. I’ll get a better look when I have more time.

[tardy parcel]

Model manager is currently being rewritten to support individual models - May be easier once that is released

[shut moat]

Heh. Looks like it will, but not quite yet:

[shut moat]

I have to RMA my graphics card, so my progress on anything for this will have to pause 😭

Checked in code here - https://github.com/aluhrs13/iai_node_deepfloyd - should work for folks if you just drop it in to your invocations folder and follow the steps to agree to the DeepFloyd terms and stuff from their blog post.

[ancient pawn]

Good luck on the GPU - 🪦

[shut moat]

Also I looked through the model manager change in more detail. Probably won’t even need new nodes for DeepFloyd, might just need tweaks to model manager, t2l, l2i, and upscaling.

[ancient pawn]

@sweet silo could probably opine more on that, but i do think as much as possible we want to keep things from getting bloated with all the conditional logic. That's just my "design/architecture" hat being worn

[shut moat]

Bumping this again - @sweet silo do you have a feeling if DeepFloyd will fairly easily drop in to the new model manager?

[sweet silo]

my opinion about deepfloyd itself is - something that hard to run 😄

with such high requirements

but now it possible at least