LoRA Easy Training Scripts Redux

[trail tree]

I'll stick to 4 or so

time to do xl lora of my shitty doodles

[trail tree]

whats allat

[magic peak]

random shit that you don't have to worry about

more or less just torch yelling at itself

[trail tree]

cant do training now

my pc crashes like 10 minutes in

weirdo

[magic peak]

oof

that sucks

I wonder why?

[trail tree]

maybe cuz I set power limit to 75%

in addition to the undervolt I already had

or it was random

either way Icould finish the last one I did

peaked at 90c for memory

[trail tree]

ok got 110 images for mutsuki dress

[trail tree]

Btw seems like tag and toml doesn't save

Eh, 107 images with 2 repeat and 4 bat j gives like 70 epochs for 2k steps

I'll just save every 200

[trail tree]

Hm, I set it to save only every 200 steps but it shows 11 epoch saves

Oh I forgot to change sample to per step

I'm only at 600 steps now but the epochs around 200 already seemed fine

[magic peak]

Oh for dev? Yeah I know, I need to fix it

When I was working on the complete rewrite, I just flat out neglected to recreate those systems lel

[trail tree]

[trail tree]

[magic peak]

Oh no

[trail tree]

didnt go that well this time, even the last epoch doesnt get it right every time

oh well I'll redo it later

[magic peak]

Oof

[trail tree]

interesting though

batch 4 repeat 2 2222 step run

batch 2 repeat 2 run so like 1122 steps

the 4 batch one took longer to do something and still isnt great at last epoch

wtf

I ran the trainer while sd was launched with an xl mode lloaded

💀

150 mb away from oom

[magic peak]

150 isn't oom though lel

[trail tree]

sadge

[trail tree]

maybe I left in some shit pics

meh

dumbshits tagged different type of dresses for her

with the same teags

dumb af

ok should be fine

but had to delete a bunch

oinly left with 101 images now

[foggy comet]

kek this got me

Derrian sees Tony heading his way and just panicks

[magic peak]

I mean, it doesn't matter how much I try to help him

like, I've tried way more then the average person would even attempt to

I don't know what to tell him

[trail tree]

mf literally made a personal branch for llil bro

💀

my latest favorite is this

I won't go back 😠

oh no how will we ever survive

[magic peak]

oh god...

you know what? I would love for him to never come back

note to self, NEVER MAKE WHAT HE WANTS POSSIBLE

[zealous glacier]

https://github.com/kohya-ss/sd-scripts/pull/1151

from the furry server

[magic peak]

I use 0.41.1 though

[trail tree]

https://fxtwitter.com/bdsqlsz/status/1766670707835257035

[magic peak]

Good, finally can update it

[trail tree]

bruh

why does it want to download cu118 torch

well I'm not using xformers anyways

[proven kernel]

can someone help me to use this correctly?

[magic peak]

I have it set to 118 because there was some compatibility issue, but it's probably resolved at this point, so I'll change it to 121 and see

[proven kernel]

sup

short question

how many steps should u have with 500imgs?

8k or 200

i got 2 configs and im a bit confused on the high difference

[trail tree]

style?

maybe like 3k

[trail tree]

I didnt know how to do torch 2.0 with 121 so I just downloaded the latest stable

no xformers is the downside

[magic peak]

doesn't 2.2.0 have support for 121 though?

[trail tree]

dunno, I just didn't know how to download it

[magic peak]

oh

alright

[trail tree]

but I didnt see any performance being worse with sdpa

so was fine for me

btw what would you normally use deepspeed for

[magic peak]

isn't deepspeed for mutli gpu?

[trail tree]

seems so

[magic peak]

yeah, so not really useful for the average person

[trail tree]

gonna get 3 moer 3090s soon

[timid yoke]

when using lora trainer, and being over the limit, what happens to the additional tokens? The settings have a max token length of 225, but my file has in total 400tokens ?

[magic peak]

It truncates to 225 tokens

[timid yoke]

oh thank you, do you also know by chance what may be causing this blur? I have it in almost all epochs

[trail tree]

#ai-art message

[foggy comet]

ya looking at dev branch commits it should be working as of at least 2 days ago in za UI

[trail tree]

[magic peak]

Kek

I've been testing Dora since it was fixed lel

Honestly, it seems worth the hype

[trail tree]

What setting do you use

[magic peak]

Still figuring that out, they are slightly different then normal lora, so I've been messing around with testing them. I have a very prototype toml I can send you once I get home from work

[foggy comet]

HandL HandR

[magic peak]

Don't worry, I know you've been wanting it too lel

[foggy comet]

RinMlem

[magic peak]

Wasn't that before the fix?

[trail tree]

maybe

no clue

[magic peak]

Yeah, it was

The fix was 4 days ago

The "way smaller lr" was 100% related to a bug in the code

Considering I still nan'd even with lr 1e-6 lel

[trail tree]

bruh

what would be good to try for finetuning?

like not singular topics/concepts but a bunch

[foggy comet]

First time using it through the backend feature and it works flawlessly, sasuga Derrian

ikekneel

I think printing the local port that uvicorn binds to could be a good idea though unless it confuses the newbs

and maybe a way to change the port through like an ENV var

[magic peak]

Np, lots of work went into making it easy to install and use lel

[foggy comet]

AYAYAY

[magic peak]

Oh yeah, that was actually something I was thinking about

I have a config.json, so I can just add that as a config option

[foggy comet]

[magic peak]

I also need to look into adding more tunnel options too, right now it's ngrok, and cloud flared, but I also want to add zrok, and local tunnel

[foggy comet]

[magic peak]

Zrok might be annoying though lel

[magic peak]

Btw, for most use cases, the user is either using it locally, and likely doesn't even realize there is a backend running as a separate process (in which case they use 127.0.0.1:8000 (pre populated), or they are using a tunnel, which doesn't have a port to enter in

So printing the port is generally not very useful

[foggy comet]

yee I just forgor uvi port off the top of my head

[magic peak]

Fair

[foggy comet]

ah zrok

I tried zrok

the install process was fairly annoying

[magic peak]

It seems less limited then ngrok, but it also seems like a pain to install

[foggy comet]

yeah it was a pain, I think I had to re-do it at one point

[magic peak]

Either way, glad the backend is working fine for you

[foggy comet]

I don't even know if it's working now on my vps

[magic peak]

Oof

[foggy comet]

imma be honest I don't even know why I did it

coz I have nginx and shit

do I have dementia

I might have dementia

[magic peak]

That bad

[foggy comet]

yeah I absolutely cant tell why would I ever need it

AH OKAY I GOT IT

nvm

I was seeing if I can tunnel something to the world using zrok like that, from my local pc

[magic peak]

Ah I see

[trail tree]

would you filter out multi char pics for char loras? I have like one with 500 images but half of the pics have like 2-3 other chars on it. saw u did that duo blue archive green hair train chars shit so maybe u have some insight?

[magic peak]

I usually try not to include art which has multiple characters, unless it explicitly has both of the characters in it, and I explicitly tag both

when I run grabber, I usually make sure to have 1girl, and neg the other characters I am training, so that the images I do get in that search result are explicitly only the expected character

[trail tree]

https://cdn.discordapp.com/attachments/1179903144659005521/1228401722322321418/291f93cfde854d50f4a21096bd96b99d.gif?ex=662be941&is=66197441&hm=814a7756aaa78074c4eb729951ebbb85facb8cdaccdca505d2acfb1d06587a9a&

trying lion and rex

forgot to update

whatever only doing same run as last time

:\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\lib\site-packages\diffusers\models\attention_processor.py:1259: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)

steps: 1%|▍ | 24/2596 [01:50<3:17:10, 4.60s/it, avr_loss=0.0734]

if i disable gradiant checkpointing I'll oom

even tho it only uses 12gb with it on

[magic peak]

Huh

[magic peak]

Are you training these 3?

Because if so... I have a version of them trained for ponyxl...

Or are you only training ein?

[foggy comet]

what I do is I have a folder with those

and basically overprune the tags quite a lot

just so the chars are reimu, cirno, pose, location and outfit

and I set lower repeats on those

[magic peak]

Interestingly, I do the opposite, I usually make sure all of their tags are there

And it usually works well enough if you have those images in their own folder with less repeats

[foggy comet]

hmm, I can see that working as well, yeh

I could maybe run some comparisons at some point

[trail tree]

just 1

[magic peak]

Humu, yeah I would only include images of that one character

Any image that has more then 1 of them is an immediate toss

[trail tree]

fuck

I forgot to change the sample prompt

[magic peak]

But I'm the madman that would train all 3 into one lora lel

So

[trail tree]

left mutsuki dress prompt

[magic peak]

Actually, I'm the madman that trained all 3 into one lora in 1.x lel

[trail tree]

I'll keep the datasets for future finetune maybe

[magic peak]

Fair enough

I kept all of my datasets with the expectation that I would be using them again eventually

[trail tree]

fuck I also left the wrong model

-1 hour lulw

[magic peak]

Oof

[magic peak]

I've only trained on pony, so I only have one model to worry about

[trail tree]

don't use pony a lot so I train for all the other xl models instead I'd gen a blank char in pony in the setting I want and then inpaint over it in other models instead

at least for the face

hmm

what card do u train on again

[magic peak]

A6000

[magic peak]

I don't gen really in general, outside of lora tests, so I just train on what most people are using

[trail tree]

dayum

u got 48gb vram?

[magic peak]

Ye

A solid batch 2 with dora

[trail tree]

how much was it

[magic peak]

I'm not kidding non checkpointing is actually almost 48gb of vram

For batch 2

[magic peak]

4k

[magic peak]

With checkpointing though... it's 11gb

[trail tree]

same

11gb with oft unet only

[magic peak]

You use oft?

[trail tree]

yea

[magic peak]

I didn't bother with it because it didn't actually look like an improvement

To be fair, dora is the first alternative algo I have used

[trail tree]

i'll try it later as well

[magic peak]

All of my lora I've baked have been dora

Well, most

[trail tree]

I only tried normal lora locon lycoris and oft

even the old ones?

[magic peak]

Locon, and dora when it came out

[magic peak]

Not the 1.x ones

Those were either lora or locon

[trail tree]

I havent updated yet

[magic peak]

It's only on dev, and it's not a selectable algo

[trail tree]

y

[magic peak]

It's a modifier on lycoris locon, loha, and lokr

[magic peak]

Because kohaku

[trail tree]

[trail tree]

[trail tree]

hmm

epoch 1 vs 2

why buba

wasnt sure what to adjust for lion and rex scheduler

so I just ran same shit as before

2e-5 unet and 1e-6 minimum

[fluid dune]

Do it

[trail tree]

dont have 4k now

[trail tree]

limbs where

[magic peak]

lion huh?

[trail tree]

ye

wait

I confused it

with the other one u told me about

FUCK

I didnt see the fucking scroll bar

meh

last epoch sample

[trail tree]

and I did bare lora

what afucked up day

eh, eyes are fudge

gonna redo

[magic peak]

Ein has really odd eyes, so I would not expect it to learn perfecfly

[trail tree]

could be nice if the artstyle was cleaner

or adetailer

need to alternate outfit to be able to send full pic

[trail tree]

beta1, beta2, beta3 = group['betas']
ValueError: not enough values to unpack (expected 3, got 2)

3rd one is 0.999?

Consider setting beta3​ between 0.9995 and 0.99995 if beta1​,beta2​ are set to 0.9,0.999.

Beta3 (often denoted as (\beta_3)) is specific to the CAME optimizer.
It determines the confidence level for the adaptive memory update.

I'll just do 0.9995

similar memory use

11,800mb

[magic peak]

I just use the default betas for came

[magic peak]

I had a feeling you would say that

[trail tree]

it didnt change for me

[magic peak]

I mean, I just deleted the arg from the optimizer args

[trail tree]

ah

[magic peak]

and use whatever is packed into came

[trail tree]

hm 1e-5 might be too low lr

[magic peak]

I use 1e-4 lr

I don't train te usually

[trail tree]

same now, at least for diag oft

cuz it doesnt work

[magic peak]

oh I don't use diag oft, so I can't give you any params

[trail tree]

I just use whatever

I meant if u train te for diag oft it wont work in webui

so gotta disable it anyways

[trail tree]

oh yea its not even close

came diag oft epoch 1

1girl, ein \(blue archive\), blue archive, by quasarcake, [by ssambatea], solo, black background, simple background, best quality, masterpiece, absurdres <lora:ein3-000001:1>

bruh this artist tag forces muscle bodies

eh, so close on the eyes

[trail tree]

gonna check tmrw which one gets the eye right the most

[magic peak]

using artist tags lel

yeah, nah, it's definitely me that's the weird one for absolutely refusing to use artist tags

[trail tree]

maybe, i dont realy give a shit personally, I'll do almost anything to get good looking images if I really cared about artist tags I wouldn't even be using ai in the first place since the entire thing is a mix of artist tags

sec

[magic peak]

well, I should say, I don't use artist tags when testing lora

[trail tree]

oh yeah

[magic peak]

but then, I don't gen outside of testing lora, so I never use artist tags

[trail tree]

[magic peak]

I just enjoy training

[trail tree]

its fun, I prefer making imgs tho

maybe if I had an a6000 I'd do more

[magic peak]

heh

it is nice being able to do batch 2 with a dora without checkpointing

a solid 41gb of vram when not training te

[trail tree]

dunno dont even play games much so its not like the resources are needed elsewhere

but I couldnt gen while training or do llm

[magic peak]

when i'm not training, i'm working on the UI, and when i'm not working on the UI, then i'm playing games with friends

[trail tree]

how is it that much

[magic peak]

I dunno, dora just uses a ton of vram, but somehow compresses down to 11gb with checkpointing

I don't make the rules

[trail tree]

oh well, fairly happy with this

[trail tree]

oh u added a toggle button for dora?

[trail tree]

read that on reddit ubt not seeing it

oh I used update instead of pull

[trail tree]

wait does higher batch size mean slower train

[magic peak]

It can if you have to use a bunch of grad acc and checkpointing, but usually it's faster

[magic peak]

On dev, yeah

[trail tree]

didn't see it so probably didn't update properly

[magic peak]

It's just a checkbox on the network args I believe

[trail tree]

[magic peak]

You aren't on dev lel

Either that or you aren't on the newest dev

[trail tree]

`45bf41ff988e99304cbe716ad1642571288712e4 branch 'dev' of https://github.com/derrian-distro/LoRA_Easy_Training_Scripts

f595d1e5430e1037688cf1d09efa3c0733840c1f not-for-merge branch 'main' of https://github.com/derrian-distro/LoRA_Easy_Training_Scripts

9855d2ee1a2ea9ed82df0b35cb17a6f723ff5365 not-for-merge branch 'old-scripts' of https://github.com/derrian-distro/LoRA_Easy_Training_Scripts`

I used the update bat

i'll just redo it

[magic peak]

If you are swapping from main branch you have to do a complete reinstall

[trail tree]

I'm on dev

[magic peak]

But did you just git checkout to dev?

Or did you install new in dev

[trail tree]

D:\LoRA_Easy_Training_Scripts  dev  git status
On branch dev
Your branch is up to date with 'origin/dev'.

no

I just cloned it at that point

and thne used update bat

in backend and in the base folder

[magic peak]

You only need to touch the base folder, and the update folder expects you to have already run the install.bat

(In the root folder, it handles both)

[trail tree]

nice

[magic peak]

Yeah, you shouldn't ever need to go into the backend

[trail tree]

I only did to use a different torch version and just saw there was also an update bat there so I ran it

[magic peak]

It installs torch 2.2.1 I think, which is only one version behind

[trail tree]

yeah this was back before 2.2.0

[magic peak]

And I think I swapped it to using cu121 at some point too

[trail tree]

pog

anyways I'll just reinstall

[magic peak]

I think I might change the update files to just nuke the venvs and reinstall from scratch everytime

Easier then having people ask questions

[trail tree]

does it come up often

[magic peak]

Enough times I have people who don't actually run the installer before running the updater

Or they installed everything then realized they weren't on dev

[trail tree]

I mean it kinda makes sense that u only run the installer when you're first setting up so from that angle I get it

[magic peak]

I still have a ton of work to do on the UI before I want to move it to main

But time makes it so my work is somewhat sporadic

[trail tree]

wtf

u did that much in like 2 months

[magic peak]

Yes lel

[foggy comet]

Derrian is a machine

he writes code how I write shitposts:

famst

[magic peak]

kek

[magic peak]

FAMST

[trail tree]

welp that didn't work

[magic peak]

What exactly were you trying to do?

[trail tree]

wanted to try if boft would work since it seemed to use the same parameters as diag oft but I blew it up

[trail tree]

can I just type the names of these for lycoris preset in the preset box?

https://github.com/KohakuBlueleaf/LyCORIS/blob/main/lycoris/config.py

tried but didnt do much oh well done for now, was worth a try either way

[magic peak]

oh yeah, it will be fine with those

[trail tree]

#1198937785961291808 message

[trail tree]

ok it works after turning off rescale

[magic peak]

I don't have access to that server

[trail tree]

ah

its just the lycoris server

https://discord.gg/QxvWG3Rg ✅

but its not relevant anymore, since it works

seems like boft is more prone to pick up style with same settings compared to oft

but tbh i'M not sure what settings are good for it, 0 info pretty much anywhere

thats the downside of brand new or not much used stuff

[trail tree]

ok epoch 1 of th eboft is really fucking good

lmao char is just defaulting to nude with empty prompt, based

[foggy comet]

boft deez nut

[magic peak]

Ah I see, yeah I'm not in that server because I don't actually use lycoris stuff often, at least not until dora came out

[trail tree]

bofa?

what is bofa

https://www.youtube.com/watch?v=KZfq9amWaJg&t=30s

[trail tree]

uhh

I think I know why boft took so long

just 10gb worth of checkpoints

and I only saved every 2 epochs

oft with same settings was 92mb

the fuck

4x size

[trail tree]

https://github.com/facebookresearch/schedule_free

[magic peak]

that's not getting added until it's in kohya

i'm not touching that with a ten foot pole

(i've already tried to implement it)

[trail tree]

I just saw it in another server

[magic peak]

fair

[trail tree]

what's factor?

like for lokr

[magic peak]

Factor?

[trail tree]

One of the options just dunno which

[magic peak]

Oh, don't know about it then

I don't train lokr, they aren't worth it

[trail tree]

It's worth it compared to full finetune

But I don't think it's a setting for just lokr

They're just calling it a different name than what's in the UI orobablxy

[magic peak]

I mean, you do you, but I definitely can't agree

lokr isn't a finetune replacement

[trail tree]

It's what was used for the kohaku models so that's enough for a result for me to try

[magic peak]

hmm, I guess we look for quality in different ways, because I don't really consider those models quality

[trail tree]

Yeah I don't consider pony quality either yet here we are

[magic peak]

neither me, but at least it's trainable

big reason why I don't use animagine is that it's annoying to train

[zealous glacier]

What's wrong with animagine training

[magic peak]

it's just really annoying to train on

it's too "fiddily" I guess

not very easy to get good results from it

[trail tree]

I wonder if kohaku fixed oft and boft training time to try

[trail tree]

wtf didnt realize there was an option for textual inversion

[trail tree]

I can almost hear it

📉

trying hubris loss

[magic peak]

Yeah, some random furry pr'd it a while ago now to add support for it

I just decided to clean it up a bit and add it in

[magic peak]

I'm only on 3.0.0dev4 so it's not the most up to date version

I need to get around to updating everything

[trail tree]

I updated to latest and it worked fine

Didn't get to test though, ran a setup I know worked, undertrained though

[trail tree]

D:\LoRA_Easy fork\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\lib\site-packages\torch\utils\checkpoint.py:90: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
steps: 0%| | 5/3456 [01:30<17:15:28, 18.00s/it, avr_loss=0.0604]

is that warning okay

[trail tree]

wanted to try different setup

batch size 4 accumulation step 2 128 epochs is not the way to go for boft

[trail tree]

actually not sure, maybe more effect than last time but also changed other settings

https://cdn.discordapp.com/attachments/1218640456636698736/1264777416413282345/image.png?ex=66a11582&is=669fc402&hm=0b90f88ec12760fc62f3a66cd950f62b31eec1ca6ff09499f63b64fce498f6d6&

dataset has some really pretty pictures tho

[trail tree]

can u fit in the option for boft sometime? I think it just needs an entry in the algo list and ui and that's about it

[magic peak]

Probably can, yeah. I also have another arg I need to add at the request of somebody else anyways. I might actually add another drop down to allow adding of random args people might want to use but doesn't make sense anywhere in the UI

[magic peak]

BOFT is added to the algo list

[trail tree]

cool

[hardy sluice]

Today I learned REX uses d=0.9 by default and not 0.5 as noted in the paper

Why was 0.9 chosen as the default rather than 0.5? Was it personal preference

[hardy sluice]

I also noticed that d isn't a parameter that you can adjust in scheduler args

perhaps i'll do a pull request for it to be allowed

[trail tree]

flux

what d?

[hardy sluice]

d is a scheduler parameter for the custom scheduler REX

it affects how aggressive the curve fall-off is (or not aggressive?)

[trail tree]

do u have a link? I couldnt find it

[hardy sluice]

https://arxiv.org/abs/2107.04197

[hardy sluice]

https://github.com/derrian-distro/LoRA_Easy_Training_scripts_Backend/blob/92e696b0bd47bab5742dd695f5848da7a5dee8c2/custom_scheduler/LoraEasyCustomOptimizer/RexAnnealingWarmRestarts.py#L24

https://www.desmos.com/calculator/qzh3xa0rqu

The effects of d on the curve of the scheduler. The paper sets d=0.5

I'm also just speculating, but I think the REX scheduler works because the overall learning rate is higher, and that at later steps, it's at such a low learning rate that it literally does nothing to the model.

So perhaps derrian choose d=0.9 to aggressively cut off the useless and low learning rate that isn't used, and that d=0.5 isn't enough. I personally chose d=0.7. I might pull request for my changes to allow users to set d values as scheduler args.

[magic peak]

It was the default in the implementation that kohya didn't accept the pr for, I didn't read the paper, and it worked fine, so I never bothered to change it

I actually didn't do any number crunching or theorycrafting for this one. I was asked to implement X version, so I just did lel

I actually couldn't quite figure out what d was doing from the code alone, so I didn't touch it

[magic peak]

I already have plans on updating the UI, I'll probably do that this weekend

[hardy sluice]

d is how bias the scheduler is towards enforcing higher LR towards the end/middle of learning.

first image is when d=0
second image is when d=0.5
third image is when d=0.9

implemented default is 0.9, paper default is 0.5. I think something around 0.7 (fourth image) is maybe a bit better, but i idk

when I get the chance I'll make the changes and make a PR. it just exposes d to the init function of the scheduler.

[magic peak]

No need, I'll be working on the UI today, so I can add it as an arg

But I've had no issues with 0.9

It works very well

[hardy sluice]

cool beans, thank you

0.9 is probably still a good value at medium-low learning rates

[magic peak]

Well, it also works nicely for longer training runs

[hardy sluice]

though if you use a rather high lr, it never decreases enough to learn and finalize the bake

[hardy sluice]

i think 0.9 has it's uses, yes

[magic peak]

I train at 7e-5 normally

[hardy sluice]

i did a few trainings at 1e-3

[magic peak]

I would never think of using that high of an lr even when I wasn't using rex lel

[hardy sluice]

i was trying out a very specific technique of copier loras

essentially, you overfit a lora on exactly one image, then merge it, and train the a new lora on the difference from the merge

for style traininng... it's a mixed bag, but i've seen it work for consistent image filters (e.g. greyscale)

[magic peak]

Yeah, I know of them. I thought the lr for them was smaller

[hardy sluice]

they can be, i think it's just a longer train to overfit. i'm sure that high of an LR has implications on the final lora even if it's just to lock image generations

i'm still trying to wrap my head around it... namely when training the second LoRA; if it should have exactly the same params as the initial train or train as if it's a normal LoRA

or if, again, that high of an LR on the initial copier lora has implications on the final result

i'll have to formally run a trial to test this

[trail tree]

nice, it's weekend from tomorrow

[magic peak]

It'll be the weekend starting in an hour or two for me

So I'll be working on it soon

[trail tree]

[fluid dune]

Libadwaita update?

[magic peak]

what the fuck is that

[fluid dune]

Just Gnome’s GTK 4 layer

[magic peak]

ah

can you tell I don't use linux?

[fluid dune]

No?

And I have done the Libadwaita joke to a few Windows users

[magic peak]

oh, alright then

[trail tree]

libtard waiter

[fluid dune]

Yes I forgot you were delayed in French sorry

[magic peak]

https://github.com/derrian-distro/LoRA_Easy_Training_Scripts/tree/flux

[magic peak]

ok, fixed the github submodule issues

now it should work correctly

[hardy sluice]

[hardy sluice]

Was debias estimation implemented yet?

[trail tree]

[magic peak]

Extra args

Basically, any arg I didn't know where to place, or didn't feel needed to be a default value got shoved to the extra args, basically just put the arg and the variable in and it will load it into the args

[hardy sluice]

👍

[trail tree]

does cformers work with 2.4.0 on windows

c section

[magic peak]

Xformers doesn't, no

You have to use sdpa

[trail tree]

I'll just use that then

[hardy sluice]

I don't think I have too much of a diff between xformers and sdpa in terms of performance, idk

[magic peak]

there isn't

[trail tree]

I wonder how much juice it uses

my memory junction is 76 just from caching lull

[trail tree]

enable fp8 training.
Traceback (most recent call last): File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\flux_train_network.py", line 408, in <module> trainer.train(args) File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\train_network.py", line 594, in train t_enc.text_model.embeddings.requires_grad_(True) File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\venv\lib\site-packages\torch\nn\modules\module.py", line 1729, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'T5EncoderModel' object has no attribute 'text_model'

the watface

guess I'll try a different one

[magic peak]

yeah, you need the T5xxl_fp16 from SD3 medium

[trail tree]

oof 9gb

https://github.com/kohya-ss/sd-scripts/pull/1452

nvm

justr gonna disable it, would be a tight fit anyways

maybe my config from before is overwriting something

[magic peak]

yeah i've got no clue tbh

I didn't encounter this bug at all

[trail tree]

gonna go through a bunch of settings and barebones

ok I forgot to change main args

was on boft

do you need to load t5xxl even when you disable te training

[trail tree]

ok yeah thats not it

happens even when I use thhe sd3 one

but I checked and you're already up to date with that pr I linked so I don't get it

apparently a workaround is cache te outputs but then I get another error

[magic peak]

yeah not sure

[trail tree]

alright got a working setup I think

https://github.com/bmaltais/kohya_ss/issues/2701 was reading this

[magic peak]

oof

[trail tree]

this time it will work

ok it went back to optimizer got empty param lllist

after I removed relative_step=False scale_parameter=False warmup_init=False

so it's probabllymissing a setting

or I need to try the other optimizers

[magic peak]

are you using came? came doesn't have those args

[trail tree]

yeah

[magic peak]

I was using came for my tests, so I know it works

[trail tree]

I don't get why it says that then

[magic peak]

no clue

[trail tree]

do you have any of your test configs?

ok I got it I tihnk

got further

Traceback (most recent call last):
File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\flux_train_network.py", line 408, in <module>
trainer.train(args)
File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\train_network.py", line 1099, in train
noise_pred, target, timesteps, huber_c, weighting = self.get_noise_pred_and_target(
File "D:\LoRA_Easy_Training_Scripts\backend\sd_scripts\flux_train_network.py", line 323, in get_noise_pred_and_target
assert network.train_blocks == "single", "train_blocks must be single for split mode"
AssertionError: train_blocks must be single for split mode

prob most of the problems were related to trying to use lycoris modes

[magic peak]

That should be handled by my UI

[magic peak]

Also yes, lycoris is completely not supported

[trail tree]

yeah that was on lora

disabled split mode and now I'm stuck at 24gb

[magic peak]

Yeah, that's related to the split_mode arg

There was a bug related to it like 4 hours ago

But I fixed that quickly

[trail tree]

yeah saw

updated

[magic peak]

Well, to be fair, it's experimental for the biggest reason of it can break at any moment and I've no clue why

[trail tree]

just gotta keep spamming

until a config works

[magic peak]

I'll give you a config that worked for me when I get home and finish eating

Keep in mind you'll have to tweak it

Because it's built for 48gb of vram

[trail tree]

pokemon

[magic peak]

What model are you trying to load, btw

Fp8 dev?

[trail tree]

yeah

[magic peak]

No good

[trail tree]

uh

[magic peak]

Load the bf16 version directly from the flux repo

[trail tree]

alrighty

oh right there is an fp8 mode anyways

[trail tree]

I get to this point with both fp8 and bf16 ver

and both give this error

I'll change huber to something else

alright it's running

changed loss to l2

[magic peak]

oh yeah, l2 loss only

[trail tree]

changed to batch size 2 since didn't make much of a difference being on 1

shouldve put saving per steps instead of epoch with 2k images but whateva

[trail tree]

this makes me want to get a 48gb card even more

[trail tree]

[magic peak]

oh yeah, this happened to me

the defaults are really bad lel

best to setup the flux args to be something close to this

[trail tree]

huge

thanks

[magic peak]

np

[trail tree]

was running this lul

wonder if I can speed it up than what I did llast time

was like 30 minutes for 100 steps

[trail tree]

huh, my files and sample images aren't saving to the designated folder/wandb and I don't see any warnings or error messages

[magic peak]

Odd, things save to the expected folders for me

[trail tree]

yeah just randomly happened

hopefully not on 2nd epoch

welp

ggs

dunno how thhat happened

-1 hour I guess

[trail tree]

huh

it didnt save again

am I retarded

[magic peak]

!configs?

[trail tree]

yeahh

those are saving properlly

just theh safetensor and sample images arent

[trail tree]

yeah it just wont save

weirdest shit i've ever seen

guess I'll reinstallll tmrw

[magic peak]

so long as kohaku hasn't updated lycoris for kohya, there is nothing for me to add

[fluid dune]

Make easy training a multi backend project

[magic peak]

I would love to see how you would do that

[fluid dune]

you turn it into swarm

[magic peak]

i'm not learning C#

[fluid dune]

[magic peak]

too java like

[trail tree]

is this factor thing a setting somewhere?

fuq it düd

ok yeahh it doesnt matter what I set it to

as long as factor is -1

[hardy sluice]

that's a big dim 👀

[magic peak]

That is new, I'm pretty sure, and considering I don't use lokr at all, I didn't notice the error

[trail tree]

not just lokr, I see it in others too

[trail tree]

here

[magic peak]

So long as it isn't locon or lora, I haven't used it at all to know something was wrong lel

(And clearly nobody else has either, considering my absolute lack of any sort of bug report)

[trail tree]

I did ask once before

[magic peak]

Gonna be honest, I tend to forget if it's not made an issue on the github

[trail tree]

I didn't know it was an issue

I never tried lokr either, it only came up in boft warnings before

[trail tree]

I still don't know if it's an actual customizable option, but from reading the lycoris server it seems to come up fairly often when talking about settings for networks

and it doesn't make sense that lokr is locked at -1

it gives a 10mb file even if I set dim to 10000

[trail tree]

ok I thought there were only like 30 new teri pics that I got from boorus

but on cn forums there's like 3 times that in just a month

[magic peak]

I have no clue who that is or where they come from

[trail tree]

it's teriteri just sleepy

now my total imaages is 36+42

not bad

[fluid dune]

issue or not issue
On most git host making an issue helps track suggestions, bugs and other stuff

[magic peak]

I think i've seen some art of her. I definitely dont know her though

[trail tree]

shes cute

wtf is this

zorro lookin ahh

https://tenor.com/view/zorro-sword-slash-gif-14863752

[trail tree]

lost mask only to become zorro again

[trail tree]

turned up lr and dim a bit but maybe it's too much, quality is ass now

[trail tree]

overtrainging now, making the plushies without even prompting

[trail tree]

overbaked sadge

[trail tree]

https://github.com/KohakuBlueleaf/LyCORIS/blob/8ad8000efb79e2b879054da8c9356e6143591bad/docs/API.md?plain=1#L56

[trail tree]

fuck me for updating I guess

now none of the commits work, even the one I was on

💀

[trail tree]

@magic peak

[magic peak]

Already fixed

I was asleep

[trail tree]

[trail tree]

Is it hard to set up remote use?

[magic peak]

no, just install the backend on the remote server + a tunnel if you need one (it has 2 to choose from, but you can just use your own if you prefer) and then copy paste the url to the frontend

(which is installed on your computer)

[trail tree]

Pog

[pulsar dome]

what gpu do u use for training flux?

[trail tree]

3090 in my pcc

[pulsar dome]

how many it/s do u get on that?

[trail tree]

depends on the setup

I think 3 hours for 200 steps

withh 23gb

vram use

again depends on setup

[pulsar dome]

oh well i was looking for some other settings to speed it up on a 3060 12gb but idk if 10secods per it is fastest u can get on kohya using adafactor/constant

[trail tree]

not really a good metric to use for comparisons unless you compare against someone running the same exact settings

what's your vram usage?

[pulsar dome]

8.9 peak

[trail tree]

try telling ai your exact settings with that amount of vram and ask what to increase to get more speed out of it

[pulsar dome]

do u think increasing resolution of 512 vs 1024 or batch size would increase speed? or maybe switching to lion?

[trail tree]

worth a try

all I know is that increasing from 512 x768 to 1024

made mine use 10gb more vram

for some reason

[pulsar dome]

but when u trained loras for sdxl/pony do u notice that training speed on flux is 2x slower or 3x times slower?

[trail tree]

well the models are bigger

its not much slower for me, just uses more vram. I was used to 3 hours~ for sdxl

[pulsar dome]

well ill just wait maybe they will optimize scripts later

[trail tree]

https://github.com/woct0rdho/triton-windows

[magic peak]

I had triton for it for a while, never actually worked, so I won't be adding this

I mean it's just a package in the end, so people can install it themselves, if it ever gets onto pypi then I'll think about it

I don't want to manage auto downloading the correct wheel

Especially because the updates will continue to change the names

[trail tree]

just linked cuz saw it not that you have to add it

[trail tree]

how do the samples work if you train lora on vpred model?

[magic peak]

no clue, that's definitely a kohya question