ComfyUI execution error: mat1 and mat2 shapes cannot be multiplied (1x2304 and 2816x1280)

[steel storm]

I am using the segmentation feature, this is my prompt:

orc warrior on a horse, close up
<segment:human face,0.5,0.4> detailed orc face
This works correctly if I do not change the "Segment Model" setting. But if I select a different model in the "Segment Model" setting, I always get this error:
ComfyUI execution error: mat1 and mat2 shapes cannot be multiplied (1x2304 and 2816x1280)
my regular model is Flux Dev, and in the "Segment Model" I selected some SDXL checkpoint.
It seems it's not supported to use a different model architecture for the segment than for the regular generation? Is that correct? Because if the base model and segment model are both the same SDXL model, it works correctly. Only if I use Flux Dev as base and SDXL as segment model, it does not work.

[frail knoll]

Hey, it looks like you need help with SwarmUI. If so, here's what to do:

First, hit CTRL+F and search to see if anybody else has already asked the same thing, and check if it's in the Troubleshooting Doc.

If not, open a post in #1255969955808022679.

When making your post, go to your Swarm interface, click Server, then Logs, then the Pastebin button, then Submit. It will generate a link - copy that link, and include it in your #1255969955808022679 post.

Make sure to also describe the problem in your post, and include any relevant information, such as screenshots of the interface or your gen parameters. .

[pliant pivot]

oh no i see the issue

segment logic wasn't properly built for arch swaps

this oughtta cover it better

updated the internal code to understand arch swaps properly, and added a section for param overrides (since eg Flux wants CFG=1 but SDXL wants CFG=7ish)

[steel storm]

nice, thank you! another thing, I really wish the segment prompt would be its own prompt box on the left in the segment settings. because adjusting the <segment> part of the prompt in the main prompt makes SwarmUI think that the whole prompt changed and makes it re-run the whole generation (so also the base generation), and not only the segment part of the generation. while if I change a parameter in the segment settings on the left, only the segment part of the generation is re-run which is much faster.

[pulsar spade]

Have you tried to put nothing between the last character of your prompt and the "<"?

[steel storm]

I like to put a line break or two line breaks between the end of the regular prompt and the <segment> part, otherwise it's very hard to read

[pulsar spade]

Yeah but I think that's what triggers the new gen

[steel storm]

why should that make any difference in if it re-runs the whole generation or not when changing the segment part?

[pulsar spade]

the text enc will never see the <segment> part because that's only for Swarm itself

unllike white space

Oh sorry

I misread what you wrote.

[steel storm]

what I'm doing is like changing

orc warrior on a horse, close up
<segment:human face,0.5,0.4> detailed orc face

to

orc warrior on a horse, close up
<segment:human face,0.5,0.4> very detailed orc face

and that re-runs the whole generation then, also the base flux generation

I assume there is some code somewhere that checks if the input to the prompt box changed, and if yes, re-runs everything since the prompt box affects everything

[pulsar spade]

Yeah I missed the part where you said that you only changed the segment part.
Thought you added the segmentation after your initial gen and then added white space to the prompt.

Contrary to what I said, this doesn't even happen to me if I add a segment or white space after the initial generation.
Testing right now with a different arch for the segment, maybe it's specific to that.

Hmm, seems to work for me. Did you make sure to fix your seed?

[pliant pivot]

yeah the caching is based on nodes in the comfy backend not the swarm parameter contents