Bunch of copy (differential) LoRA | 東方Project AI | Page 1

daring geyser May 31, 2023, 1:59 PM

#

https://huggingface.co/Aotsuyu/LoRA/tree/main/diffLora
These are trained in an unique way. I first generated the Reimu image, trained a LoRA on it to the point it's overfit and only outputs the image, regardless of prompt (32/16 alpha/dim, adamw8bit 5e-4, 3 epochs, 300 steps, 1 batch size).
I then edit the image with a variety of effects I could think of, let's take the second one as the example, the wide one.
I train a similar lora on that image, and then I merge the two LoRA at -1 and 1 strength, while resizing to 8dim.
Doing this pretty much removes the Reimu from this merged lora, so you're left with a LoRA that applies the effect.
The effect isn't something amazing that you can't achieve without the LoRA usually, yes, but I've been having fun mixing these and hope to see more usage out of this method.
A more detailed comparison can be seen in the schizo grids directory, so I just post a relatively small grid here.
Previews generated with Aurora.
91 lora at the time of writing, some are admittedly pretty garbage (as in, they dont do anything that interesting), since I'm just throwing shit at the wall.

Aotsuyu/LoRA at main

#

A more involved example, this is how I edited the reimu doing style transfer on Van Gogh's Starry Night

#

Ending up with a LoRA that does this

#

Interesting thing is that these also have sometimes great effects at strengths that we don't usually use, that being anywhere from -2 to 2 strength, sometimes even bigger values.

#

#

chalkandcharcoal effect

#

If nothing else, these can be used for meme LoRA (not quite sure why did the example turn out so lewd but it does showcase the effect)

#

Some more loose examples

#

#

#

This one has additional lora but it's something I liked, the invert lora made her a glowie

#

wheat fossil May 31, 2023, 2:15 PM

#

https://cdn.discordapp.com/emojis/770360988985589791.gif

daring geyser May 31, 2023, 2:19 PM

#

https://rentry.co/kopiki_lora Is where I got this method from, thank you @atomic goblet for posting it -w-

コピー機LoRA学習法

※cweb-pixが死んで画像がリンク切れしてたのでmajinaiに上げて修正しました
はじめに
コピー機LoRA学習法とは、過学習させたLoRAを基に別のLoRAを作成する手法。アウトラインの調整などの細かい変化をさせるLoRAを作成できる。
LoRAの「モデルと教師画像の差分を学習する」という仕組みを利用したもの。
このメモでは作成手順の紹介と作成の記録をする。
手順および作成記録
1.元となるモデルで画像を出力する
なんでもいいから適当に出力する。出力した画像は「元画像」とする。
2.元モデルで元画像を学習させる
過学習で完璧なコピー機LoRAにする。ここで作成したLoRAは「コピ...

void spire May 31, 2023, 2:39 PM

#

So you basically only use 2 images for each lora?

#

Or rather 1 for each and then merge them?

daring geyser May 31, 2023, 2:48 PM

#

yep

#

so for these I used 1 image for the base lora and 1 each for the others so.. 92 images for 91 lora

#

my settings are probably not perfect either, it might be beneficial to actually train at some giga dim like 256

#

to make the overfitness overfit even more

#

because the more overfit it will be the less overfit the result will be... if that makes sense

#

basically you want to approach 0 loss with training

#

the values I posted should be good for a test run though, since it should take like 15 minutes to make the resulting lora

wheat fossil May 31, 2023, 2:54 PM

#

daring geyser May 31, 2023, 2:54 PM

#

oh and this is the master of these https://huggingface.co/2vXpSwA7/iroiro-lora/tree/main/release

#

do try out some of these, they are great

void spire May 31, 2023, 3:12 PM

#

Cool stuff, gotta try making some KoishiSip

daring geyser May 31, 2023, 3:13 PM

#

I theorize it could also be used for stuff like accurately generating tattoos for a character

#

or different pupil shapes, with pretty much no style impact

#

maybe even stuff like genning a fucked up hand with the 'v' gesture

#

fixing that

#

and doing the diff

#

suffice to say, there's potential

#

so far my attempts at tatto have failed tho

void spire May 31, 2023, 4:01 PM

#

Could be potential for inpainting lora, the ones I've made are basically just overtrained

daring geyser May 31, 2023, 4:04 PM

#

that was what I was thinking as well, since making an inpainting lora is usually a pain

daring geyser May 31, 2023, 5:31 PM

#

ah I forgot to mention

#

in my repo the

#

shit named with just numbers

#

are random gens I used

#

5,6,7 can be interesting

#

and the psycho ones are just different color vomit patterns

vital solar Jun 3, 2023, 4:59 PM

#

Popcorn

daring geyser Jun 3, 2023, 4:59 PM

#

mygod

robust grove Jun 3, 2023, 7:07 PM

#

mygod

noble dirge Jun 3, 2023, 7:08 PM

#

mygod

wanton pecan Jun 3, 2023, 7:23 PM

#

mygod

daring geyser Jun 3, 2023, 8:41 PM

#

so I did some tests on spiral eyes, and I regret training at 512, I think that was the issue

#

I can try doing it on 876 tonight, I think I can do that on batch size 1

#

#

#

my initial edit kinda sucked as well I guess

#

lemme try and g enerate a cowboy shot xy and then I'll upscale one of these with adding @_@ tag

#

also I cba to try these out with inpainting right now

#

#

also I tried different settings for this one, ones I've seen iroiro use

#

which is 5k repeats, 1 epoch, 128 dim 64 alpha

#

1e-4

#

cosine

#

took a fair bit longer to make than my previous settings

wanton pecan Jun 3, 2023, 8:56 PM

#

chenThink

daring geyser Jun 3, 2023, 9:02 PM

#

wdym 5 files

#

lemme redo

#

first try, lora 1.5 strength, cowboy shot, @@ in prompt, adetailer off
second, same settings, adetailer on, denoising 0.45, `@@, pink eyes in adetailer prompt
third, portrait, adetailer off
4th, portrait, adetailer on

#

then

#

5th, og generated without lora or adetailer (still had @_@ in prompt to not fuck the seed up),
6th, 5th but manually selected and inpainted (512x512, pink eyes, @@ in prompt, inpaint only masked, 0.8 denoising, lora at 1.8 strength)
7th, 5th but 1024x1536, 1girl, portrait, pink eyes, @@`, inpaint whole, 0.5 denoise, 1.5 lora strength

#

so it aint really amazing for inpaint

#

thinkeroo

#

the portrait one generated as-is could be good at proper strength maybe

#

@void spire you may be interested in results

void spire Jun 3, 2023, 9:06 PM

#

Did you try inpainting?

daring geyser Jun 3, 2023, 9:07 PM

#

yesh, the second set is that

void spire Jun 3, 2023, 9:07 PM

#

Curious to see how consistent it is

daring geyser Jun 3, 2023, 9:07 PM

#

it seems to generate some bullshit with inpaint masked

void spire Jun 3, 2023, 9:07 PM

#

daring geyser yesh, the second set is that

Nice

daring geyser Jun 3, 2023, 9:07 PM

#

this was trained on a portrait though

#

lemme find the set

#

#

as you can see the edit is kinda meh

#

it could be better at #1 trained at higher res

#

and #2 different crop of the eye

#

I did up the current ver to da repo

#

but it's not really in a usable state

void spire Jun 3, 2023, 10:11 PM

#

Hmm hmm interesting

woven oak Jun 4, 2023, 11:35 AM

#

this is interesting

vital solar Jun 4, 2023, 11:38 AM

#

void spire Hmm hmm interesting

new method for your sharingan lora I assume

cold hemlock Jun 9, 2023, 12:42 AM

#

@daring geyser Not sure if you are still looking at this thread - but I'm quite intrigued. I wanted to test this myself but I haven't had much luck, maybe you can point me in the right direction.

The rentry you linked seemed to need more steps and follow a more annoying process - so I followed you formula instead. I'll repeat it here and you can maybe let me know where I went wrong.

I generate an image using a model (ie. dreamshaper)
I train a lora on that image (dreamshaper as a base) to the point where it becomes overfit and only produces that image. This becomes loraA
I modify the original image and train a second lora on the modified image (drameshaper as a base) to the point where it, again, becomes overfit and only produces that image. Exactly the same settings as the previous model. This becomes loraB
I merge both loras (I'm using kohya_ss GUI) together without a base model, just the two loras. loraB with a weight of 1 and loraA with a weight of -1. This becomes loraC.

Is this process right? I ended up with a loraC that is also overfit and generates a more mangled version of the original image.

daring geyser Jun 9, 2023, 12:44 AM

#

cold hemlock <@443060245804089374> Not sure if you are still looking at this thread - but I'm...

This process seem right indeed, do both of the loras consistently produce the same image and the settings are identical?

#

down to the lora seed

cold hemlock Jun 9, 2023, 12:45 AM

#

yes, the lora seed was 123 for both. The images they produce don't look identical to each other (except for the mods) but they are very close. I'm not sure how busy you are but if you have a couple of minutes I can dm you and share some of what I'm seeing so that I don't spam this thread 🙂

daring geyser Jun 9, 2023, 12:45 AM

#

also it could be the merge + resize that's not working well, I use python .\networks\svd_merge_lora.py --save_precision "fp16" --precision "float" --save_to "new8dimLoRA.safetensors" --ratios -1 1 --device "cuda" --new_rank 8 --models .\LORAofBASEIMAGE.safetensors .\LORAofEDITEDIMAGE.safetensors from command line

#

I mean it's no spam, really, since some people might learn from it down the road

#

but my DMs are open if you prefer that

cold hemlock Jun 9, 2023, 12:48 AM

#

maybe I should give that a shot first. I'm using bmaltais' gui for kohya's scripts so I'm not the most familiar with the CLI interface for kohya's scripts, but I can give it a quick shot.

#

I had to edit his GUI so that it'd allow me to go to -1 on the merge weight and maybe that gets clamped elsewhere 🤔

daring geyser Jun 9, 2023, 12:49 AM

#

well this one is also doing a resize to 8dim so that affects stuff but since bmaltais is just a wrapper for the scripts

#

there shoul be a 'sd-scripts' folder in there

#

if you go in there with a terminal and do .\venv\Scripts\activate this will activate the virtual environment that should just allow you to run the command I posted previously

#

with values adjusted to your lora names

#

okay I took a glance at the bmaltais repo and there's no sd-scripts folder, but you can do it from the 'base' folder

#

as in just .\venv\Scripts\activate and then the lora merge command

cold hemlock Jun 9, 2023, 12:52 AM

#

tyty, merging it now 🙂

#

ok that works perfectly now @daring geyser -- I'll dig into what the difference is with what I was doing with bmaltais' wrapper and post it here in case it's helpful to others.
It might be the resize, but I don't know why that'd make a diff 🤔
EDIT: Not resize - most likely svd merge like mentioned below

void spire Jun 9, 2023, 12:57 AM

#

just gonna chime in here, I tried doing with with #1073837013758902355 and that didn't work either.

daring geyser Jun 9, 2023, 12:57 AM

#

bmaltais is probably using the normal merge_lora script and this one uses svd_merge_lora, what's the difference? no clue

#

SVD is basically a matrix factorization technique, which decomposes any matrix into 3 generic and familiar matrices.

#

wanton pecan Jun 10, 2023, 5:35 PM

#

when the

vital solar Jun 12, 2023, 11:44 AM

#

https://media.discordapp.net/attachments/1019446913268973689/1117477272153231431/1099736523622064138.gif

daring geyser Jun 12, 2023, 9:10 PM

#

I just realized I have probably made a big blunder by training the tenc even though there were no tags

#

manually setting text encoder weight to 0 yields usually better results with most

#

now I wonder if that can yeet the tenc somehow

daring geyser Jun 12, 2023, 11:14 PM

#

toolkit can do it

vital solar Jun 13, 2023, 9:38 AM

#

Yep yep

daring geyser Jun 13, 2023, 2:14 PM

#

I ripped out the tenc so it's only natural that I post a grid

vital solar Jun 13, 2023, 2:15 PM

#

315mb ReisenBlank

daring geyser Jun 13, 2023, 2:21 PM

#

ehehe

vital solar Jun 13, 2023, 2:21 PM

#

daring geyser Jun 13, 2023, 2:21 PM

#

https://a.yuri.fun/bchzl.png I like this one

surreal stump Jun 13, 2023, 9:44 PM

#

Looking at these has made me realize I'm not creative, I just want to copy other people's artstyles

daring geyser Jun 14, 2023, 1:22 AM

#

surreal stump Looking at these has made me realize I'm not creative, I just want to copy other...

I mean these are just random filters kinda thing

surreal stump Jun 14, 2023, 1:23 AM

#

Yeah but I'm too brainlet to put them together to make somethign unique

daring geyser Jun 14, 2023, 1:32 AM

#

at least some of them have some usage in horror

sullen rock Jun 20, 2023, 10:28 AM

#

Who decides what is creative?

#

Arent you making something new out of something old?

mossy vector Jun 28, 2023, 12:00 AM

#

daring geyser Ending up with a LoRA that does this

this is mint

daring geyser Jun 28, 2023, 12:00 AM

#

mossy vector this is mint

🏸 I don't have a mint emote

mossy vector Jun 28, 2023, 12:01 AM

#

fuknmint

woven oak Jul 15, 2023, 2:43 AM

#

Smokeleaf

void spire Jul 18, 2023, 11:32 AM

#

@slender briar Is it possible to do this with your trainer? I tried using your old sd scripts but couldn't get it working, maybe I'm missing something.

daring geyser Jul 18, 2023, 11:36 AM

#

void spire <@1083542322396008553> Is it possible to do this with your trainer? I tried usin...

just overtraining the lora is doable for sure since that's what I've been using Thooking