2.3.1

[jagged oasis]

Please use this post to discuss experiences and issues with version 2.3.1, which is now at release candidate stage: https://github.com/invoke-ai/InvokeAI/releases/tag/v2.3.1%2Brc1

We need testing of installation and operation on three platforms:

1. Windows
2. Linux
3. Macintosh OS

NOTE that the Windows install has several issues that I'm working on. I will post RC2 later this evening.

[civic coral]

Trying to update a manual install, I'm getting ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python >=3.10; 0.1.1 Requires-Python >=3.10; 0.1.2 Requires-Python >=3.10; 0.1.4 Requires-Python >=3.10; 0.1.5 Requires-Python >=3.10; 0.1.6 Requires-Python >=3.10 ERROR: Could not find a version that satisfies the requirement compel (from versions: none) ERROR: No matching distribution found for compel and Traceback (most recent call last): File "/home/who/InvokeAI/.venv/bin/invokeai", line 5, in <module> from ldm.invoke.CLI import main File "/home/who/InvokeAI/ldm/invoke/CLI.py", line 12, in <module> from compel import PromptParser ModuleNotFoundError: No module named 'compel'

tagging @fallow osprey

Python 3.9.2```

So, I guess we only support 3.10 now?

[civic coral]

Also, doing a new manual install, I see that the page now mentions MPS only for M1/M2 macs, but MPS mode apparently also works on at least some intel macs with AMD gpus.

[jagged oasis]

<@&1020322889704149052> v2.3.1-rc2 is now available for testing. A bunch of windows install bugs have been quashed. New issue is that the recently-added compel library requires Python > 3.10. https://github.com/invoke-ai/InvokeAI/files/10810719/InvokeAI-installer-v2.3.1%2Brc2.zip

@civic coral Confirmed! Neither manual nor automatic installs work with Python 3.9 systems. @fallow osprey , are we compelled to use compel ?

[fluid hearth]

nodes will require 3.10 anyway. Is there a reason we want/need to stay on 3.9?

[jagged oasis]

Just to avoid an upgrade for a lot of users who are still on 3.9. Also a bit of documentation needs to change.

[fluid hearth]

(I feel like the 3.10 requirement is for type hints for nodes, but there may be some generics-related stuff it requires as well)

[civic coral]

It's sounding more and more like invoke should explicitly install python for itself, rather than counting on the system to already have a compatible version installed.

[jagged oasis]

Ubuntu 20.04 LTS, which I still have on my laptop, only has Python 3.9. I need to upgrade the OS in order to get Python 3.10, or else compile from source.

(or add a custom ppa, not too bad)

[civic coral]

I installed the latest debian a few weeks ago, and it still comes with 3.9. I had to install 3.10.9 from source.

[jagged oasis]

I suppose 3.11 stll isn't working with torch? It makes me a bit sad to have to tell people that we only support one particular version of the interpreter.

The docker container build is failing as well, due to the python 3.10 requirement in compel @fathom comet @fallow osprey See https://github.com/invoke-ai/InvokeAI/actions/runs/4249924106/jobs/7390529945

I'm knocking off for the night. I hope there won't be too many unhappy users tomorrow. @tidal elbow @river shuttle

[fallow osprey]

ah. i don't actually think i need 3.10 for compel

@jagged oasis if you bump the compel version to 0.1.7 in pyproject.toml python 3.9 works

[jagged oasis]

Excellent. Sorry for being grumpy. It was an unexpected hitch.

[jagged oasis]

I did not even realize that compel was your own package.

An a slight tangent, do you know why we are asking for numpy < 1.24? It's a slight annoyance to watch torch install 1.24, and then Invoke uninstalls it and installs 1.23. Since numpy is big, you have to wait while it installs, uninstalls, and then reinstalls.

[frozen pollen]

If we want to work with Invoke (generate images) during the migration to nodes, which branch should we pull from?

v2.3.1?

[tidal elbow]

I think its probably best for the team to play with both - but yeah "full feature functionality" would probably be 2.3.1

[jagged oasis]

<@&1020322889704149052> Hey everyone, 2.3.1-RC3 is out. You can use either Python 3.9 or 3.10 with it, unlike RC1 and RC2 which required 3.10. Zip file here: https://github.com/invoke-ai/InvokeAI/releases/download/v2.3.1%2Brc3/InvokeAI-installer-v2.3.1+rc3.zip, and release notes here: https://github.com/invoke-ai/InvokeAI/releases/tag/v2.3.1%2Brc3

@fallow osprey I've got compel pinned at 0.1.7 in pyproject.toml, so you can release future versions without worrying about breaking invoke python 3.9 users. We'll make python 3.10 the minimum requirement for main when nodes is merged in.

I think people are more accepting of large requirements changes when a major release comes out, in contrast to a bugfix point release.

[fallow osprey]

i hope you mean 0.1.7

you do, all good.

[jagged oasis]

Yes!

[frozen pollen]

Did you check this in yet?

[jagged oasis]

It's still in PR state awaiting review. However the zip file I posted has the fix.

[fathom comet]

can look into that

[fathom comet]

But if we require Python 3.10 now, we should also update the pyproject.toml and set requires-python = "==3.10"

Ahhh - sorry, did not read the whole thread - suppose the 3.10 problem already got resolved 😅

[jagged oasis]

Problem was fixed, thankfully. I was not enthusiastic about asking users to install a new python in order to accommodate a bugfix point release.

[cunning valley]

I didn't realize Debian stable doesn't have 3.10 packaged yet.

Looks like we'll see a new Debian stable, uh, sometime this year, probably.

[frozen pollen]

Not holding my breath!

[wary solar]

error while testing the auto-update after updating to RC3

[wary solar]

well, It's not working anymore, after crashing due to vRam issue, and restarting didn't solved. tried with the option "clear ram after every use" and still no effect.

[jagged oasis]

@wary solar Thank you so much for finding and reporting this. There is a Linux/Mac dependency in this bit of code. @scarlet oak - this is the code you wrote at my request and which I approved!

@scarlet oak I've reverted for now, just in order to move the release along. The commit was 89239d1c54b67dda883773fc65ba08615db8aa27

@wary solar I'm sorry your system is broken, but it should be easy to fix.

The error message suggests that you have a broken model file. I can't tell from the message which model it is, but if it is a diffusers model, you can find it in INVOKEROOT/models/diffusers. Delete the entire directory. When you relaunch invokeai and try to load the model, it will re-download it from the internet.

If it is a checkpoint file, then choose (5) from the menu and re-download it.

Problem is trinart-2.0 comes in both checkpoint and diffusers forms.

If this is happening with all models, then let me know.

[wary solar]

discovered, the issue was caused by the Accurate option on

I changed it from fast, now, after changing back it's working fine

Accurate is always broken, never worked, but in 2.3.1 it didn't even start rendering

[verbal pine]

I'll give this a test later on this evening and let you know how RC2 installs 🙂

[jagged oasis]

@keen basin Just became aware of this bug report about the Accurate rendering of intermediate images not working. I had also encountered the same issue in the CLI's handling of intermediate images, and the solution was:

@@ -1262,10 +1263,13 @@ def make_step_callback(gen, opt, prefix):
     os.makedirs(destination, exist_ok=True)
     print(f">> Intermediate images will be written into {destination}")
 
-    def callback(img, step):
+    def callback(state: PipelineIntermediateState):
+        latents = state.latents
+        step = state.step
         if step % opt.save_intermediates == 0 or step == opt.steps - 1:
             filename = os.path.join(destination, f"{step:04}.png")
-            image = gen.sample_to_image(img)
+            image = gen.sample_to_lowres_estimated_image(latents)
+            image = image.resize((image.size[0]*8,image.size[1]*8))
             image.save(filename, "PNG")

This is not exactly the solution you're looking for since I also changed from "accurate" to "fast" in this fix, but the essential problem is that the callback is now being handed a PipelineIntermediateState object, rather than the latent image and the step.

[keen basin]

Thanks for the Intel. Could you please make a gh issue for me so it doesn’t get lost?

[jagged oasis]

Will do.

[wary solar]

I noticed, after doing 10 images for the context, running out of memmory, even asking in the option to free the memory each time after a image is generated, the only way to recover from it is loading another model and returning. It's a bug?

it's increasing until running out

[cunning valley]

yeah, "Current VRAM" is expected to generally end up at the same place every time

[wary solar]

after some images I need to restart

[frozen pollen]

How much VRAM do you have?

[hot idol]

I'm not in #dev-chat but gogurt's problem and KinGabe's problem sound like what I was having in #1078096357916684338 except I don't have xformers

[wary solar]

6GB

[wary solar]

it's 512x512 ddim

for the challenge

[civic coral]

Ok, so I've confirmed this VRAM build up issue. As a workaround until the devs fix it, you can reset the buildup by switching models.

[wary solar]

I'm just updating it inside the model manager

works faster

[civic coral]

Ah, that causes a reload of the model, essentially switching to the same model that's already active, which has the same effect.

[fallow osprey]

looks like https://github.com/invoke-ai/InvokeAI/pull/2780 didn't make it into the 2.3.1 official release?

the python3.9 fixes for my 3.10 type hinting 🙊

[frozen pollen]

Merged 18 hours ago

BUT into main.

So that PR needed to be merged into v2.3 instead.

@jagged oasis @river shuttle

I'm sure there's some git way to fix this.

[tidal elbow]

think we can cherry pick the commit into the 2.3 branch or something

[frozen pollen]

And then do a quick re-release of 2.3.1...

[jagged oasis]

I missed that. I'll pick into 2.3.1 and release a 2.3.1.post1

[tidal elbow]

I looked into it at #archived-💿installer-live-chat and it seems like this was merged in

as as I can tell, looking at the commit

[jagged oasis]

It is merged in as far as I can tell.

I have to do a patch release anyway, because model merging is broken. As soon as I'm done with meetings, I'll start up my laptop, which is running 3.9, and see what the issue is.

[tidal elbow]

It is possible there isnt an issue, fwiw.

[jagged oasis]

As I recall (now), the PR was against main and I merged without realizing it at first. Then I saw the error and did a cherry pick. However, I never tested against 3.9 because @fathom comet 's docker build uses 3.9 and started working again.

I will confirm, just in case.

[frozen pollen]

It got merged back in, I believe.

[jagged oasis]

This was a couple days ago.

[frozen pollen]

requires-python = ">=3.9, <3.11"
  "dnspython==2.2.1",
  "opencv-python",```

  "compel==0.1.7",```

[jagged oasis]

In any case, merging was broken due to a mistake on my part and I've fixed it in 2.3.1.post1 Looks like I caught the problem before too many users complained - at least I haven't seen any reports yet.

[jagged oasis]

Just went through a full install on a system with Python 3.9.5 and all went smoothly, so I think the user must have been on an RC.

[wary solar]

Wow! I am absolutely blown away by the amazing work you all have done! Thank you so much for your incredible passion and competence in solving the update bug, accurate preview, and memory issues in record time! Your dedication to excellence is truly awe-inspiring, and I can't express how much I value it. Your hard work and dedication have not gone unnoticed, and I can confidently say that you have set the bar incredibly high. Once again, from the bottom of my heart, thank you all so much!

[keen basin]

thanks and thanks for helping with our portuguese translation 🙂

[verbal pine]

Hey, sorry it took me a few days, but the install went fine for 2.3.1! 🙂

One thing I notice, after having it download any extra models, is I see three 2.1 models: 2.1, 2.1-768, and 2.1-base. The 2.1 model also seems to have the wrong description with it. It says that it's trained on 768px images, which isn't correct unless that's supposed to replace the 768 model. Otherwise, might wanna fix the description for it

Uhh, offhand based on what I see that might actually be a 768 model. the results at 512x512 look horrible...

Ok, yes I can confirm the 2.1 is actually the 768 model. Here are the results I got with testing.

Prompt: A painting by Thomas Kinkade
Seed: 12345

2.1 at 512x512

2.1 at 768x768

2.1-768 at 512x512

2.1-768 at 768x768

2.1-base at 512x512

2.1-base at 768x768

So yeah, something got messed up with the 2.1 model and it's actually the 2.1-768 model, like the description says. This definitely needs to be addressed to avoid confusion. Or perhaps it's a mistake given there is already the 2.1-base model?

[verbal pine]

Gotta be honest too, I don't seem to be having as much luck with the inpainting-2.0 model either. Is it supposed to be a 512 or 768 model?

[cunning valley]

I think https://huggingface.co/stabilityai/stable-diffusion-2-inpainting is probably 512px

oh, did you add 2.3.1 recommended models to your existing earlier set of models?

it does sound like you have some duplicates in there under different names.

looks like the current https://github.com/invoke-ai/InvokeAI/blob/main/invokeai/configs/INITIAL_MODELS.yaml doesn't have 2.1-base in it at all

[wary solar]

Setting 1000 steps break the program, and changing it doesn't make it recover =\

it's a bug or I messed it up? Before setting steps to 1.500 was possible

[cunning valley]

you got me there, I never tried anything >= 1,000 steps when working on 2.3.

Hell, I never tried anything over 100 steps. And at CFG 50??? Why?

[wary solar]

testing random stuffs

sometimes bigger steps avoid the artifacts in the image above, and stick to the prompt too

[fallow osprey]

at CFG 50 though you're basically throwing a rubber band yoyo into space and waiting until it slingshots back from jupiter

the "sometimes" indicates whether your yoyo comes back or flies off to Sirius

[frozen pollen]

I can confirm that 1000 steps at any CFG crashes.

[cunning valley]

I would expect 1,000 to work, but maybe there is an offset somewhere in there that means we only get 999 in practice.

I'm not surprised that things over 1,000 don't work because I think there are schedulers that assume timesteps can be whole numbers (and the training range for Stable Diffusion was 1,000, so going over that puts you in a place where the distance between steps is less than 1.0).

[frozen pollen]

It should absolutely not crash, however.

It requires a restart of Invoke to fix.

[jagged oasis]

I just did 2000 steps at CFG 200 and it didn't crash. The image was pretty horrid, though.

[cunning valley]

it may vary with scheduler

[jagged oasis]

When curating the starter models just before the 2.3.1 release I decided to offer the 768 pixel model as plain-old stable-diffusion-2.1 and not to include the 512-pixel base model. If you previously had -768 and -base installed, this will get slightly confusing. Is there a good reason to include 2.1-base as well? I don't find it sufficiently better than 1.5 to justify including it.

Probably best to cap steps and CFG at reasonable max values.

[jagged oasis]

It's easy enough to fix this.

[cunning valley]

I think the argument for 2.1-base is something like:

• You want something that prompts similarly to 2.1 (with LAION's Open CLIP), with the performance characteristics that SD 2.x has, but at the lower resolution because you don't have the resources to run 768px all the time.

(I make no claim that's of broad enough interest that it deserves a spot in the default model list. That's another question. Just offering considerations other than "better/worse than 1.5.")

[tidal elbow]

Think it's valid to offer any/all SD foundational models as defaults, even if they're not necessarily recommended

[keen basin]

This issue with models is that the are duplicated. The new model installer looks for a the same models eg sd2.1 768 but with new names in models.yaml.

[jagged oasis]

Not a problem. I'll get another patch release ready.

I'll go back to the original naming scheme and offer both -base and -768.

[keen basin]

So if you had sd2.1 768 installed previously, and it’s entry in the yaml has “768” in the name, the model installer doesn’t see it as already installed

[keen basin]

Sounds good and a simple fix

[jagged oasis]

Yes, I get that.

[cunning valley]

or it could refrain from adding a "new" model if there was already one with a matching source.

[jagged oasis]

https://github.com/invoke-ai/InvokeAI/pull/2820

[jagged oasis]

This would work. For now, we'll just go with restoring the previous names. I guess what threw me is that the repo_id for the 768 model is stabilityai/stable-diffusion-2-1, and that the base model is stabilityai/stable-diffusion-2-1-base

I didn't have the repo ID for the 2.0 inpainting model right, either, so that's fixed here too.

[keen basin]

Thanks!

[jagged oasis]

This will require a review from @scarlet oak and @river shuttle as it changes an install file and bumps up the version number. If there's a delay in approving, I can force it.

[frozen pollen]

So why can't I generate @ 1000 steps with ddim? Is it a GPU memory issue since I'm on a lowly 12GB?

Hm. Seems to be generating now for some reason. We'll see if it goes all the way.

[cunning valley]

memory usage is pretty independent of step count.

it's because num_train_timesteps https://huggingface.co/stabilityai/stable-diffusion-2-1/blob/main/scheduler/scheduler_config.json is 1000 and so if you try to go past that you're asking for fractional timesteps, and some schedulers may not know how to cope with that

[verbal pine]

Yes I did! But that said, the 2.1 is still definitely a 768 model. The description is actually correct, just not the name

[verbal pine]

I tend to default to 512, unless I'm going for a larger image. I'd still have it be an option for people. That said, 1.5 likely is better, but I still think choice is good for that. Or at least explain to people about that, namely cause otherwise they won't realize that's a 768 model when the canvas defaults to 512 at startup

[verbal pine]

You can probably drop the -base part. Just have it be 2.1 and 2.1-768px

[civic coral]

I'm seeing a lot of reports of things breaking because users end up with file paths in invokeai.init that include spaces. In one case, they even said that quoting the path didn't work. If someone doesn't have anything to work on because of the code freeze, fixing the string handling logic for those might be low-hanging fruit.

[jagged oasis]

Can you confirm if quoting the path fixes the problem? It ought to, and it is an easy fix.

[jagged oasis]

Here is a PR that puts quotes around the file paths in invokeai.init: https://github.com/invoke-ai/InvokeAI/pull/2827

@scarlet oak or @fathom comet could one of you give this a quick review? It's only three lines of code.

[jagged oasis]

Thanks! 2.3.1.post2 is now released. It corrects the spaces-in-filenames issue. Funny how I'm always bitten by that bug and never learn.

[cunning valley]

err, I guess the correctness of that depends on what invokeai.init is. Is that any kind of standardized file format or is it more of an ad-hoc thing?

[jagged anchor]

DIDN'T fix --output dir crashing when starting up

[civic coral]

Can you post the contents of your invokeai.init file?

[jagged anchor]

fixed it

i had yo remove the --output from the invoke.init file reinstall the post2 it, then add the --output file

then run 7

is there a way to get a history in Iterm2 ?

[wary solar]

Hello!

I'm having an issue with SAEs, it's giving this erro: vae-ft-mse-840000-ema-pruned.vae.pt' is not a valid JSON file

it's happening for any VAE I have, even .bin and .pt and .safetensors

[tidal elbow]

You're using VAEs for ckpt/safetensor files, but are likely trying to use that for a diffusers model, aye?

put this in the VAE repo ID for that model stabilityai/sd-vae-ft-mse - that should be the diffusers version of the VAE

[wary solar]

it's already there.

I followed a tutorial from the help files

I'm using VAE in all my models and diffusers. Automatic1111 readed alright, but InvokeAI always give me this error

If I remove the offline location, it complains about not being able to reach the site

and this is a diffuser of my face, show more messages

[tidal elbow]

You probably don't want to have both the "VAE Location" & the repo id both - its intended to be one or the other.

[wary solar]

will try without, but the diffuser doesn't have both and don't load too

give that error and don't load the diffuser, only without VAE

it will load

[keen basin]

the VAE is a .ckpt file, not a .pt file

[wary solar]

I downloaded a safetensor, and didn't worked

[keen basin]

PT files are used for training embeddings - is this a training embedding or an autoencoder?

[wary solar]

reading the help files, a phrase was telling to change it to vae.pt

[keen basin]

what is the filesize of vae-ft-mse-840000-ema-pruned.vae.pt

[wary solar]

inside the file is also calling it a .pt file

319MB

[keen basin]

hmm

that's 330 KB, not MB

that's not a VAE

and its in your embeddings folder. i don

~~this is the VAE: https://huggingface.co/stabilityai/sd-vae-ft-mse-original/blob/main/vae-ft-mse-840000-ema-pruned.ckpt ~~

err, sorry THIS one: https://huggingface.co/stabilityai/sd-vae-ft-ema-original/tree/main

but yeah, where did you get that embedding and why is it named like a VAE? something's not quite right

[wary solar]

it's this file, and was on the diffuser folder, but didn't worked and I followed a tutorial on the help files saying to change it to vae.pt, but the place for pt files is in another folder, so I changed it

what a mess, lol

will download again to show the error

[keen basin]

to use it as an embedding, rename it something else. if its an embedding trained on your face, just rename it to like 'telles0808face.pt' and leave it in embeddings

then delete the paths to that file in the model manager. it shouldn't be in the VAE slot

[keen basin]

oh wait LOL is 326.799 actually 326,799?

sorry, number formats are different!

so you can't just rename the file i don't think. its a different format.

[wary solar]

still giving error

[keen basin]

im sorry i need to go to work, but what works best for me is to let the invoke script, when u choose the install models mode, do all the downloading and stuff for you

@tidal elbow can you pls help get the models paths sorted?

i think we need to clear out the exist VAE entries via model manager, run the install models script and then update the vae locationss in the model manager, but maybe there is any easy way

[tidal elbow]

yep!

[wary solar]

will try installing via prompt

[tidal elbow]

I would do this

I would delete the model from the model manager (removes the entry, does not delete the file)

re-add it, and (assuming it is a diffusers model) only input the VAE repo ID

[wary solar]

both, models and diffusers are giving the same error

for this VAE with 560000

some diffusers using custom VAEs are okay

[wary solar]

anyway, thank you very much for taking time to look into 🙂

[jagged oasis]

What tutorial? Hopefully not one of ours.

[jagged oasis]

From the thread I can't follow what the sequence of events was that led to the VAE being placed in the embeddings directory. What is the current issue with the models file entries? Under 2.3.1 newly-created models.yaml no longer have any top-level VAE entries, and the installer quietly installs the legacy ckpt VAE in the right location for use by legacy ckpt files that need it.

[tidal elbow]

I think this was a lot of manual manipulation into broken state

[jagged oasis]

Can you post an excerpt of your configs/models.yaml file covering the model that is giving you the issues?

[river shuttle]

ill take a look soon

[wary solar]

Right now it's working, after deleting all the models and diffusions and installing again using the prompt. I used the webui to install some diffusers before and the problem started.

I watched a tutorial from Automatic1111 and thought it would work similarly, but it didn't. Everything started with a parsing problem first and an offline issue, even though I wasn't offline.