Z-Image-turbo: black output after few generations

[rough monolith]

I have a weir bug using Z-image, and only Z-image. I do generations around 1024x1024 res, they come out fine, then suddenly everything breaks and all I get is black output. Like fully black image, from the first step. Restarting the back-end helps.

SwarmUI v0.9.7.2
Windows 11 24H2
RTX4090, 7950X3D, 64GB DDR5

[jolly viper]

(https://discord.com/channels/1243166023859961988/1443782840033738772)

[rough monolith]

This doesn't seem to be related? I tried the proposed solution anyway and issue still happens.

[jolly viper]

I'm sorry. I saw this in a post and thought it might help you. 😯

[rough monolith]

I appreciate the sentiment 🙂

[scarlet python]

I'm having the same issue on MacOS. Trying to figure it out.

[jolly viper]

I generated about fifty images with z_image_turbo_bf16, without any problems (with Swarm 0.9.7.3 , RTX4060Ti 16GB VRAM, 64GB system RAM). 🤨

[rough monolith]

"It works for me" is not helpful. We know most people don't have such an issue. The same way I don't see errors that other people report here - it doesn't mean I should just dismiss them and comment "well I don't have this problem".

[rough monolith]

I found some suggestions on-line that adding "--force-upcast-attention" as ExtraArg in backend could help. Testing now.

[jolly viper]

I just wanted to let you know my test results. 😀 . btw, have you tried with the fp8 version?. Or try server->backends->edit comfy backend->ExtraArgs-> add --force-fp32->save (mcmonkey)

[scarlet python]

Yes, forcing fp32 worked for me, hopefully it works for you!

[rough monolith]

Can't say if it works well as it's trippling my generation time. It's much faster for me to restart the back-end every 10 gens if it breaks. But thanks for confirmation this is where the issue lies.

[tardy prairie]

fp32 shouldn't be needed for you on a 4090

you have bf16 and fp8 support on a 4090, both of which should dodge the fp16 range error that a mac may hit

[rough monolith]

I assumed so, but alas. Both bf16 and fp8 result in the same issue. I guess I'll redownload models to see if that helps.

[tardy prairie]

can you replicate a gen that goes black, and then

[mighty kindle]

Hey, it looks like you need help with SwarmUI. If so, here's what to do:

First, hit CTRL+F and search to see if anybody else has already asked the same thing, and check if it's in the Troubleshooting Doc.

If not, open a post in #1255969955808022679.

When making your post, go to your Swarm interface, click Server, then Logs, then the Pastebin button, then Submit. It will generate a link - copy that link, and include it in your #1255969955808022679 post.

Make sure to also describe the problem in your post, and include any relevant information, such as screenshots of the interface or your gen parameters. .

[rough monolith]

As if to mock me, it ran 30 gens back to back with no errors, --fast as the arg (my usual setup). I will update this thread if it happens again.

[tardy prairie]

https://github.com/comfyanonymous/ComfyUI/commit/daaceac769a1355ab975758ede064317ea7514b4

this commit should make fp16 work potentially

so, worth a fresh try @scarlet python and possibly others