Corruption or Glitches | Invoke | Page 1

coarse lion Oct 30, 2022, 5:23 AM

#

Not sure where it's going wrong, but I need to restart the Python script at times to fix what I can only describe as a glitch or corruption somewhere. When running, I'll get great results, up until it starts spitting out weird images (see attached).

The GPU is a 3070 which doesn't have any cooling issues, runs games without issues and I've even run memory tests on it without any issues. Latest drivers are installed (Windows 11) and zero errors in the event logs.

All of the generation works as expected, ie there's no errors shown in the console... just that the result isn't right. If I stop and start the invoke script, it's immediately happy again for what could be 50 images or could be 500. Any tips on what to check next?

native gull Oct 30, 2022, 5:28 AM

#

coreco's AI Dreamer cluster has seen this come up on multiple RTX 30xx cards.

#

oh you're in that thread 😅

#

I guess next I'd want to check if it's sufficient to unload and reload the model, or it's only a whole new processes that fixes it.

viscid spear Oct 30, 2022, 5:45 AM

#

I'm suspicious. This apparently started happening recently. Could this be related to model caching or the new VAE? Those are two major features with deep hooks... That said, that really looks like a video card issue.

native gull Oct 30, 2022, 6:15 AM

#

its appearance varies, but sometimes it does retain a fair bit of the color and composition of the expected result. Or it obeys properties like "seamless."

coarse lion Oct 30, 2022, 6:29 AM

#

Yep it's happened with AI Dreamer and pure Invoke, I can't make it trigger when I want it to either!

#

Certainly can't rule out bad hardware as a cause.

cyan cradle Oct 30, 2022, 4:38 PM

#

rats I can’t find the other thread… IIUC, CUDA cores are a physically separate entity from e.g, shader cores and texture units, so games and 3D benchmarks will not accurately uncover CUDA hardware issues. I’m pretty sure that there are CUDA benchmarks out there (other than SD 😆), but I don’t know of any specifics

cyan cradle Oct 30, 2022, 6:53 PM

#

I can't vouch for this one way or the other, but it looks interesting https://sourceforge.net/projects/cudagpumemtest/

SourceForge

CUDA GPU memtest

Download CUDA GPU memtest for free. A GPU memory test utility for NVIDIA and AMD GPUs using well established patterns from memtest86/memtest86+ as well as additional stress tests. The tests are designed to find hardware and soft errors.

native gull Oct 30, 2022, 7:37 PM

#

ah, projects still hosted on sourceforge in 2022

cyan cradle Oct 30, 2022, 7:45 PM

#

Yeah, there is that...

native gull Oct 30, 2022, 7:49 PM

#

maybe https://openbenchmarking.org/suite/pts/nvidia-gpu-compute ?

#

it's not memtest (which would be nice) but there's non-gaming benchmarks

cyan cradle Oct 30, 2022, 7:51 PM

#

I tried looking at that, but it wanted to install PHP (🥲 😅 😲 👿 I mean seriously)

native gull Oct 30, 2022, 7:56 PM

#

oh here's a memtest: https://github.com/GpuZelenograd/memtest_vulkan

#

oh, yeah, the phronix benchmark stuff is php. which is, um, a thing. but you don't have to expose it to a web server, iirc.

#

and here's a currently maintained fork of that sourceforge cudagpumemtest: https://github.com/ComputationalRadiationPhysics/cuda_memtest

cyan cradle Oct 30, 2022, 8:03 PM

#

Which happens to have the coolest GH org name in history 😄

coarse lion Oct 31, 2022, 1:29 AM

#

the memtest_vulkan was finally able to show an error after a. few thousand iterations

#

Can therefore confirm this is a hardware issue and that'd be the tool I recommend for testing

native gull Nov 28, 2022, 10:24 PM

#

I now have a new-in-box RTX 3060 here!

#

time to find that test suite and run it before I get too attached.

#Corruption or Glitches