Weird LoRA behavior (really three issues but they seem highly related)

[topaz gate]

This is a fresh-ish install of swarmui, it did not show this issue before previous to Swarm Version: 0.9.8.0. Three weird behaviors ive observed is that loras all seem to have this format in the Loras tab of a prompt: lora:weight(-1). From what I could find that was "related to text encoding" except it was not, because when it manually changed the lora weight text encoding it had an entire category for it. Not only that but the prompt I got on the same seed ended up being completely different with no changes except the lack or the presense of that -1. It seems completely unrelated to text encoding but somehow deeply deeply affects the prompt and I can't control when it shows up and why. I use ui Loras, so theyre not inserted in the prompt, but trying to add the (-1) in both places seperately and together didnt get much of a yield except when it did and it finally got me a copy of the prompt I was trying to reuse for testing. SO this is somehow affecting my prompt but theres absolutely no documentation iI find after thoroughly searching.

Another issue that seems strongly related to this is that sometimes, when this -1 is in the prompt, it will completely change the actual weight of the lora. It will make a lora 1 when the weight I used was 0.3 for example. I can't control the outputs of what I'm doing and its quite frustrating. At last, the weirdest issue by a little really is the fact that it will duplicate loras despite them only being added once, grey them out and treat the 0.3 and 0.3(-1) versions of the lora to be completely different. I can promise you I did not insert the same twice. I can't test things like this

It started happening in this version, seemingly reusing parameters does nothing and deleting the lora then reading it "fixes" it somewhat. Not really. It still adds it to random Loras for no reason. I can't tell if I'm reporting a bug or a question on this honestly given the complete lack of documention of this being a normal thing.

these are all examples of the weird behavior im talking about

the last two show what i mean where it completely edits the weight of the lora which i didnt put in

i cant quite test prompts and find something that works when seemingly everything is really fighting me here. i was hoping it was text encoding but even manually adding the -1 not only did it make the gen significatly worse compared to the one i was reusing the exact same parameters for, but it didnt even work because it didnt show it in the lora field meaning whatever this is is a feature that either hasnt been documented or it hasnt been documented good enough for google to scrape anything on it i could use. text encoding really was the closest thing i could find

[topaz gate]

after troubleshooting this myself and with a friend for three days straight the issue seems to be is that swarm will randomly hate you for not setting text encoding weights yourself, and turn them to -1, not returning the exact weight you used for the lora but -1 specifically, which wouldnt be much of a problem to have to set manually but then that also fries the image too. so setting it to 1.0 fucks with the style of the image, and setting it to -1 also fucks with the style of the image. so you end up having to set it to 1.0 and -1 simultaneously by inserting the lora twice with different tenc weights. thats the only way ive been able to fix this issue and make it work consistently again. what's weird is that this issue is not easily replicable, for some of my friends it happened, for others it did not. for some it happened quite a few updates ago, for me its happening NOW. so this is seemingly random behavior that completely fucks up your prompts.

this is noticeably decreasing the quality of my gens and requires me to go through some insane roundabout way to fix it. i understand people are likely busy or might not understand what bug is happening here, but i'm not the only person being affected by this and this is an actual issue seemingly.

it's causing issues with the image not following the lora at all, or even doing the opposite of the lora as if i set the lora weight to -1, kind of defeating the entire purpose of a lora.

[topaz gate]

i assumed it looking better mightve been weight stacking so i also tested that out, to be clear. completely different image again even with the same parameters. so it's not weight stacking because of stacking multiple loras, it has to do with the tenc weight regardless.

[dense gust]

that's a weird issue for a niche thing (setting lora text enc weights separately, tbh I thought it was for older models with CLIP TE only)

idk where you saw that 1(-1) syntax but it doesn't exist, the in prompt syntax is:
<lora:lora_name:model_strength:text_enc_strength>

your second issue when putting X(-Y) in UI lora is that Swarm expect a number here, when provided with anything else it assumes 1 (there's a warning log about this)

and just in case, -1 doesn't mean ignore or default or something, it applies the negative of what it supposed to do.

last tip: when troubleshooting advanced Generate tab features, if you know Comfy a bit you can go to Comfy tab and "Import From Generate" to see the generated workflow and how it changes when you change parameters

[topaz gate]

see i'm not putting anything

that is SWARM doing it

automatically

randomly

completely fucking up the prompt

forcing me to fix it through changing it manually in the add prompt thing

but when i do that, it breaks anyway in some other weird way

unless i force it to word by manually setting it to -1, then setting to i believe 0 when you dont set any weight?

my friends have had this issue too. we have no idea why it happens

i came here because i found no documention of it doing that

i had to troubleshoot the issue on it own and i still dont know why it happens

1(-1) is NOT me it is swarm, i dont use text enc it is forcing me to

the only solution ive found to this is setting the text encoding twice

[dense gust]

oh, got it

yeah so extra weird, never had Swarm do that for me

[topaz gate]

ive had friends who have had this issue with swarm

i genuinely thought it was a me issue somehow

but one of my friends mentioned that yeah, it broke for her back in the anima 2 preview days

it eventually fixed itself, on its own seemingly, but it was fine for me for a few months

and then randomly started happening

i have not messed with any settings, this is a fresh install, i have used anima models before and this didnt happen. we use the same model and it didnt happen to her again, she couldnt replicate it

i also couldnt replicate it in some of my testing

[dense gust]

can you share any generated image so I can get the params see if I can try to recreate it on my end?

[topaz gate]

sure

this was a test image i did with a prompt i use for testing new models/loras. it has the issue for me on the parameters

you can see it is rather random which loras it picks for that -1 behavior

i have little hope of you being able to replicate the behavior though, it seems so random when it chooses to break.

to tell you everything ive done to mess with this; i tried deleting the loras from the gui, it seems to at times remove the -1 once you refresh them by placing them one again, in some cases it does not. i have tried reusing parameters, it does not include the -1 once you do that, it will still randomly pick one of the loras to attach to. i changed the name of two of my loras, but the issue was happening previous to that and to loras which i didnt change the name of, so it is not it somehow decoupling or something because of the changed name. i reset the server multiple times, i updated to the most recent version again, i tried a different model and loras which seemed to briefly fix it but if it was happening to someone with different models and loras anyway then that's not a related problem.

[dense gust]

I just grabbed 5 random loras from civitai cause I don't use Anima much, and so far so good, no random negatives

[topaz gate]

i changed the vae from automatic to qwen, i turned on the qwen model in the advanced model addons tab and tried changing the text encoder to the recommended one by anima, i went back to the default settings of automatic vae and text encoder. issue was happening regardless. it is something with swarm specifically.

[topaz gate]

yes it is stupidly difficult to replicate because it seems to be random behavior, at least behavior i have found no pattern with

which is a problem, because then i can't really figure out how to fix it. i have to do the roundabout way of setting the weight multiple times and thats just really fucking annoying to do unfortunately, specially given theres no way of doing it in the gui

[dense gust]

digging further: even when setting the text enc strength explicitly, it is ignored as Anima loras only use "Load LoRA" instead of Model and CLIP

[dense gust]

can you go to Comfy Workflow tab and "Import from generate" then share the generated workflow?

[topaz gate]

is there a way to do that with previous images? because its using the most recent one

[dense gust]

yeah, have your previous image opened, click "reuse parameters"

[topaz gate]

that is the workflow of the sent image

issue is, if text encoding is really not affecting anything, that doesnt explain why the image comes out completely different with and without it. when the bug happens and when it doesnt.

[dense gust]

correct and that's why it's extra weird

anything interesting in Server > Logs > View: Debug ?

[topaz gate]

visual example, if needed. same prompt, same parameters, just one has text enc and the other one does not. neither has the bug.

both do not look very polished, specially the second. which i found weirdly happens when the bug doesnt.

[topaz gate]

i will check hold awn

nothing important seemingly

i did, to be clear, recently reset the server because i was starting to get annoyed with the situation and being unable to find a solution

had i remembered the logs i wouldve sent something earlier

the only error of note i remember seeing is something about compatibility with one of my loras, and it could explain the issue if other people i knew werent also using it and not having the same problem

this is just a stumping situation

[dense gust]

yeah lora compatibility detection is not 100% perfect as each trainer has its own naming scheme, one of those I grabbed is not detected correctly but it works fine anyway, just UI things

[topaz gate]

yeah i remember that being something someone reported to here. i checked pretty much this entire channel before submitting anything

i wanted to make sure i wasnt missing some super instructive information that could solve my problem or some documentation i didnt see in the past week ive been dealing with this issue.

see now this is even stranger

how do i have 10 tenc weights but only four loras

i think something is severely wrong with the way swarm is handling tenc weights and that might be the core of the problem here. an issue nobody would notice because nobody ever uses tenc weights these days.

and neither do i, but its setting it by itself to my stuff, so i kinda have to to solve the issue.

it really said fuck these random guys in specific huh

[dense gust]

hmm seems fine for Chroma at least

[topaz gate]

yeah i WILL say both of us who have experienced it experienced it with anima models, but i would not be surprised if we werent the only ones. my friend didnt even mention it was something she saw until i did because she gave up trying to fix it on her own. i don't doubt more people looked up a solution, found nothing and kinda just gave up afterwards.

i honestly wouldve had i not gotten so annoyed i couldnt fix it myself.

[dense gust]

yeah this is beyond me, you gonna need the big boss @merry shell to look at it

[topaz gate]

🫡

i appreacite you for trying to help, some of the info you tried to extract hopefully helped to narrow down the issue]

[merry shell]

I don't track the root issue here but what I've read of this thread is barking down weird rabbit holes

the (-1) was a minor UI misrendering and can be ignored, that just meant you had prompted and UI selected loras at the same time. the (...) slot in the UI is section confinement, and default for unspecified is -1 meaning global-but-go-first

textencoder shouldn't be relevant

i think somewhere down this thread you had a side issue of inputting weird values experimentally and then getting them locked into strange places, to which the primary solution is, uh, just don't do that lol

[topaz gate]

i mean yeah duh of course, but such ui thing wasn't documented anywhere i could find, so i couldn't find what i was supposed to do in that situation, hence messing around. the -1 did appear in cases where the lora was only ui selected and not added to prompt. in that case, does it show up when you also use keywords? because if not, then it definitely wasnt happening in what sounds like the intended case.

text encoder was very much weirdly relevant, hence my further confusion on what was going on, specially when it apparently isnt even a field youre supposed to be able to use with anima loras?

it's barking down weird rabbit holes because weird behavior was happening, and unless i'm misunderstanding, not just in the cases you've said its intended to happen. and given zero documention exists of it anywhere (that i could find, i'm open to someone on reddit somewhere having asked and me just not having used the right keywords to find it), you kind of have to shoot in the dark to figure out what it is. most shots probably wont hit

also, for the can be ignored part, i did want to! i couldn't ignore it because i could see it was actively messing with my images whenever it did or didn't show up. trust me, i did want to ignore it when it was happening because i couldn't find what it was related to at all.

[merry shell]

You misunderstand my point.

The (-1) showing up can be ignored, it is not related to any issues you see, it is not related to text encoder stuff. It just means you have UI and Prompt loras at the same time.

If issues only appear when the (-1) is there, then the issue is potentially related to the usage of UI & Prompt loras at the same time, and is not related to the text encoder nor to the display of a (-1) in the metadata.

I'm trying to give you mechanical information about how SwarmUI works so you can better understand what you're seeing and better narrow down what's actually happening for you

[topaz gate]

i see! i was saying it did show up when i didnt have ui and prompt loras at the same time, hence the confusion about what was going on and why.

because if it's meant to show up only then, then it isnt, and that might be an unrelated bug in that case.

it does help me to fuck around with it some more though to know theyre very likely unrelated problems.