#Backends unavailable between generations today

1 messages · Page 1 of 1 (latest)

earnest stone
#

Not sure what happened. This is what I'm seeing:
20:27:02.507 [Warning] [BackendHandler] backend #0 failed to load model model/train3/model/Train3.safetensors
20:27:02.508 [Error] Self-Start ComfyUI-0 on port 7827 failed. Restarting per configuration AutoRestart=true...

glass brambleBOT
#

Hey, it looks like you need help with SwarmUI. If so, here's what to do:

First, hit CTRL+F and search to see if anybody else has already asked the same thing, and check if it's in the Troubleshooting Doc.

If not, open a post in #1255969955808022679.

When making your post, go to your Swarm interface, click Server, then Logs, then the Pastebin button, then Submit. It will generate a link - copy that link, and include it in your #1255969955808022679 post.

Make sure to also describe the problem in your post, and include any relevant information, such as screenshots of the interface or your gen parameters. .

earnest stone
lucid rock
earnest stone
#

ok thanks i'll try that

earnest stone
#

got it installed but same error I think:

21:23:05.925 [Error] [BackendHandler] backend #0 failed to load model with error: System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory`1 destination)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
#

at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken) at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) --- End of stack trace from previous location --- at SwarmUI.Utils.Utilities.ReceiveData(WebSocket socket, Int32 maxBytes, CancellationToken limit) in D:\StableDiffusion\SwarmUI\src\Utils\Utilities.cs:line 290 at SwarmUI.Builtin_ComfyUIBackend.ComfyUIAPIAbstractBackend.AwaitJobLive(String workflow, String batchId, Action`1 takeOutput, T2IParamInput user_input, CancellationToken interrupt) in D:\StableDiffusion\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUIAPIAbstractBackend.cs:line 315 at SwarmUI.Builtin_ComfyUIBackend.ComfyUIAPIAbstractBackend.LoadModel(T2IModel model, T2IParamInput upstreamInput) in D:\StableDiffusion\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ComfyUIAPIAbstractBackend.cs:line 890 21:23:05.932 [Warning] [BackendHandler] backend #0 failed to load model model/train3/model/Train3.safetensors 21:23:06.516 [Warning] [BackendHandler] No backends are available! Cannot generate anything. 21:23:06.518 [Error] [BackendHandler] Backend request #1 failed: No backends available! 21:23:06.519 [Warning] [BackendHandler] No backends are available! Cannot generate anything. 21:23:06.520 [Error] [BackendHandler] Backend request #2 failed: No backends available! 21:23:07.926 [Init] Self-Start ComfyUI-0 on port 7822 is loading... 21:23:12.512 [Init] Self-Start ComfyUI-0 on port 7822 started.

lucid rock
#

so, what i can tell you is:

  • that's a "sudden hard crash" (ie the backend is crashing with no error message, the log you have is just swarm getting confused that the backend disappeared), usually caused by out of memory errors, but not always. It looks like you have a lot of memory so I wouldn't think it'd be that, but maybe the model file is superhuge?
  • rtx 50xx is not very stable so there's risks there
  • you're trying to do some wonked out gen, and getting like 10 "lora not found" errors. Maybe try simplfying your param inputs and stripping out the nonsense?
  • The "train3" name implies its your own custom trained model file? if so, maybe there's something wrong with the model file, and normal models work?
#

I don't have a solid definite answer as to what went wrong, but that's the range of options I can think of

earnest stone
#

thanks. It is my own model. I tried a previous checkpoint of it but also encountered the backend error. This happened after several succesful generations today, so it's strange. The generations were using those same lora callouts that were being ignored from a prompt I just copied and pasted that I saw on civitai.
But I'll try what you've suggested.

lucid rock
#

... actually, other theory re the memory bit

#

CPU Cores: 32 | RAM: 61.61 GiB total, 26.45 GiB available,

#

you have over 30 gigs of ram in use while swarm is loading (ie before swarm has loaded models or anything, ie swarm isn't what's using all that)

#

did you leave the training software up or something by accident maybe?

earnest stone
#

no it was trained long ago. Guess I should have attempted a restart before looking for help though. I will soon.