H100 SXM5x dumped in the middle of an experiment

[quick radish]

Is there some way to know why a pod suddenly discos and terminates? I donʻt see anything in the log. Just lost an experimental run due to a sudden termination and Iʻd like to avoid doing that again

[brazen arch]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[grand coyote]

Disconnects and terminates?, do you perhaps used a spot pod type?

or shared your account as team, or shared your API key

or maybe your account ran out of balance

[quick radish]

Is on demand the same as spot? Can being within an hour of running out of balance trigger a termination? None of the other things

[magic crest]

it is not. If you select SPOT pricing then you are agreeing to your pod being terminated at any time - when there is a demand for your chosen GPU and none are available

was this your case @quick radish ?

[grand coyote]

Running out of balance can stop pods actually, not specified within an hour or any duration after it.

But yeah spot pods can be terminated (interruptible anytime )