Inconsistent Lora Effects

[visual monolith]

I am running v2.3.4 and enjoying testing out lora support. I am finding that many of them from civitai have a very mild effct or no effect at all. Some of them do a little bit more when used with a specific trigger word, but in many cases, the quality of the output is not great. I find that using the deliberate v2 model, most of the time the results are either the same, or better without the lora. There are a few exceptions.

This was surprising to me because the existing supported textual inversion embeddings seemed to all have an impact without using special prompt words (other than the embedding trigger itself). E.g. <style-glass>, <style-hamunaptra>, <style-empire>, <style-pnmagic> all apply a universally noticeable change to almost any prompt. I have only found a few lora that behave in this way.

I am using the new lora drop down list to insert the lora trigger in the positive prompt.

Are some of the lora just not compatible with the invoke ai implementation? Is the issue the deliberate v2 model I am using them with? Am I doing something else wrong?

[obsidian holly]

It's possible some loras might not be supported but we've tested most of the common ones and haven't come across any that weren't working. Could you link us to some that do not work and maybe we can take a look and see why they aren't and maybe dish out a fix.

[visual monolith]

I am off to bed, i will come up with a few specific examples and post back tomorrow. I do want to add that in the console, it always indicates the safetensors file is being loaded before generating the image, then it says unloaded after it is complete, so there is no obvious failure, the issue is just that it doesn't seem like many of them have any effect. Anyways, again, ill post tomorrow with specifics

[obsidian holly]

I noticed that some of them required a higher strength. It could be that or probably not. But I'll give the examples you provide a try later and see if theres any reproducible issue that needs to be fixed. Thanks.

[visual monolith]

here is an example:

https://civitai.com/models/8031/rfktrs-darkdreammini-v10

i used their prompt as a starting p oint for the most part:

Prompt:

award winning portrait photo of a female rogue assassin, wearing biomechanical techwear armor made of human bones, backlighting+, digital painting, concept art, smooth, sharp focus, rule of thirds, dark fantasy, intricate details, medium shot, (shallow depth of field)+, by sandra chevrier, art by RFKTR_rootrex withLora(rfktrsDarkdreamminiV1_v10,0.75) [bad art, jpeg artifacts]
Seed: 3744525392
Sampler: k_dpmpp_2
Steps: 30
CFG scale: 9.5
Width: 768
Height: 960

there are tiny subtle changes, but nothing much difference than turning on "variations"

This is with the Deliberate v2 model, but the lack of impact is similar for vanilla sd 1.5

i tried bumping up the lora strength as high as 2.0 without any difference

here is a text file with console output for this generation showing the lora was loaded, then unloaded without showing errors

i also tried using k_euler_a sampler with similar results

[charred halo]

Thanks for the report. I’ll investigate. Possibly some Lora’s are not loading properly.

[charred halo]

I have played with the darkDreaming LoRA a bit and agree that it doesn't have the expected effect on the image. It's not doing nothing, though, because if I turn the weight way up it does change the image. Just not in the way I expected. Do we know that this LoRA works with AUTO1111?

@jaunty python I wonder if you have any insight into this?

[icy anchor]

Same issue, some loras dont seem to affect anything at all

for example https://civitai.com/models/18177/gigachad-diffusionlora

[visual monolith]

Not sure if this matters, but I noticed that with the textual inversions that I have such great luck with, putting them at the beginning of the prompt has much more impact than at the end. With loras, it seems to be a special keyword that doesnt matter where it is in the prompt. Is it possible there is an ordering in the underlying tech that the UI is unfortunately hiding control of? Is it possible that in the backend of InvokeUI, the loras are simply being added to the end of the pipeline and they could be added earlier and thus have more of an impact similar to how embeddings have more impact in the beginning?

[jaunty python]

@visual monolith just tried with a1111 to generate image with this parameters - same output as in InvokeAI, so in this case I don't think that there any problem, also your lora has trigger word - rfktrstyle but you not using it

this lora tries to do some style... but it's quite hard to notice

for example with lora:

and without lora:

generation settings for image with lora:

without lora - just remove rfktrstyle, and withLora(rfktrsDarkdreamminiV1_v10,0.75)
also I disable here xformers to test in more similar conditions with a1111
//nsfw added only as there some eyes behind me.... to be sure that all will be fine...)))

[jaunty python]

it's works too
with lora:

without lora:

@visual monolith one more thing - if lora looks work only a bit, you can just increase lora's weight above 1))
(no lora, and lora with weight=4 and rfktrstyle++ token)

[visual monolith]

I tried their trigger word and also their sample prompt with the gallery image. It didn't make any difference.

[visual monolith]

thats def showing something, but its still not even close to the sample image they showed with that prompt. Maybe its just a poor lora, but the issue is I have downloaded about 20 loras and most of them have mild to no effect, nothing like the effectiveness of textual inversions which seem to consistently have impact

i do wnat to give an example of one that "just works", https://civitai.com/models/22437/abstract-disco-diffusion-look - this one works like good textual inversion, you just add it to any prompt, and it has an effect, you dont have to craft a prompt that somehow brings out its purpose or heavily boost trigger words

[jaunty python]

hm... ¯_(ツ)_/¯
i'm tested this on more then 10 loras (top ~12 on civitai + several random)
most works fine
worstest results got from locon version of:
https://civitai.com/models/11701
//I'l debug one more time -__-

[charred halo]

In the case of a textual inversion, the exact trigger term is always inserted when the TI is used in the prompt. When you’re using a LoRA you need to include both the withLora() and whatever prompt terms the Lora was trained on. Some of them seem to want very specific (combinations) of prompt terms, while others are more forgiving. I would say that suspect Loras should be tested with auto1111 to distinguish between a Lora problem and an InvokeAI problem.

I’m not saying this isn’t an InvokeAI issue, but the fact that one Lora loads and works fine and another in the same format loads but has no effect makes me suspicious.

[jaunty python]

@charred halo I not sure, but I think I found reason - compel uses original model text_encoder, but only before generation it's patched by lora

compel must use text_encoder patched by loras
otherwise we drop all patches to this

@muted flame

[charred halo]

Great detective work. This sounds like a problem because it changes the order of events considerably.

[muted flame]

yes.

i concur

[charred halo]

On the “it’s a problem part?”

[muted flame]

oh with all of the above

[charred halo]

I was afraid of that.

[muted flame]

@jaunty python 's right that the text encoder needs to be patched before compel generates embeddings

[charred halo]

So two passes over the prompt.

[jaunty python]

this lead us again to - "lora not part of prompts"

[muted flame]

it's not a big deal, it just means that the functionality that patches the model needs to be activated around the call to compel.build_conditioning_tensor_for_prompt_object

the prompt parsing is already split into, (1) parse the prompt to a tree-like structure followed by, (2) use the tree like structure to generate embeds

we have all required lora info after (1) so can just wrap (2) in a contextmanager that activates the loras

[jaunty python]

lora applied deep inside generation code as I know

now it's just - generate image, and inside it's load loras

[muted flame]

no, only one pass needed

[charred halo]

Oh good. I feared we were going to go back to the earlier design of pulling the withLora() out of the prompt before passing to compel.

[muted flame]

nope!

should be enough if InvokeAIDiffuserComponent.custom_attention_context was made stateless (static)

or was split into stateless (static) and non-stateless components

[charred halo]

Makes sense. How much state does it carry around now?

[muted flame]

then just put these lines of get_uc_and_c_and_ec()

    uc, _ = compel.build_conditioning_tensor_for_prompt_object(negative_prompt)

inside a with InvokeAIDiffuserComponent.custom_attention_context():

[muted flame]

not much, i think it's just a convenient way of storing/accessing the CrossAttentionControlContext

no reason why that can't be an arg and/or a return value

actually this should be an easy fix, i can take at a look at it today

been meaning to tidy that code up for a while

[charred halo]

That would be great!

[muted flame]

🙇

thanks for debugging @jaunty python , for some reason i just assumed loras only touch the unet

[charred halo]

This is good timing because I was about to port the 2.3 LoRA code to main.

I’ll wait until this is resolved.

[jaunty python]

so, it's looks that you should do like:

with custom_attention_context(InvokeAIDiffuserComponent.ExtraConditioningInfo(tokens_count_including_eos_bos=0, cross_attention_control_args=None, lora_conditions=lora_conditions)):

or create extra condition with zeros and Nones at beginning,
after (1) set lora_conditions
use it
and after (2) set other fields

[muted flame]

@charred halo remind me if 2.3 supports ckpts or if it's all diffusers

because if it's only diffusers then there's some stuf fhere to clean up

[jaunty python]

it's converts all to diffusers now

[muted flame]

2.3 does or main does?

[jaunty python]

2.3
nothing heard about this at main

it's simply force --ckpt_convert to True now

[muted flame]

ahh nice thanks

[muted flame]

gee i really do not like the amount of side effects of the LoraCondition's __call__ method

it leads to a situation where we've got a method that takes only a unet as an argument that may, depending on which loras have been requested, futz with the text_encoder too

[jaunty python]

@muted flame is this fine for tests?

[muted flame]

idk what you mean by tests but i'm already working on a branch that changes stuff deeper inside the custom_attention_context

it's cleaner that that

ready in a few minutes hopefully

[jaunty python]

hope that there some mistakes in my quick patch as I still can't get to work this locon no mater how apply it - our forward code, injected forward as in lycoris or patched weight as in a1111

[muted flame]

https://github.com/invoke-ai/InvokeAI/pull/3214

[jaunty python]

crap... updater can't update to branches in forks))

[charred halo]

We only support diffusers models. When user tries to load a ckpt it is converted to diffusers on the fly.

[charred halo]

Nice! I’m on my way to airport and will test against the problematic Lora’s as soon as I can.

[muted flame]

i noticed that the arcaneStyleLora_offset i was trying to test before now actually does something! which it didn't before. so i think it's working

oh heh the lora weight isn't clamped. that's nice. negative weights work too

[jaunty python]

@muted flame I compared c/uc with a1111
seems similar

output also close

if i do model weights patch instead of modifying result in hooks it's get even closer(and other differences can be result of small difference in uc/c)
but it's a question - where more calculation errors)

[jaunty python]

is there some kind of... optimize/cache algo?
i'm tried to generate on clean install in wsl and first image a bit different from generated images after(i disabled xformers, so no random in images)

first gen:

after this even if i restart invokeai I get a bit different:

@visual monolith I retested your lora, with this fix it's a lot easier to say that lora works:

[visual monolith]

thank you for all the effort, looks better for sure, do you know when the next official release will have this? (no rush), just curious

[jaunty python]

Ask @charred halo )
But I think it's better to release fast, so users will use lora normally)