🔒 SLURM, Joblib Parallel, Lustre and BrokenPipeError [Errno 108]

[icy stratus]

@radiant stag

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[radiant stag]

Hello everyone,

I recently encountered random errors while running a SLURM-orchestrated distributed computing task using Python with Joblib's Parallel module. I’m trying to determine whether this issue is entirely due to the system configuration or if there could be other contributing factors.

Specifically, I’d like to understand: 1) Under what conditions this error might occur, and 2) whether it could result from something on my end, such as a misconfiguration in my script or resource requests, or if it’s likely due to the system setup (e.g., node communication issues, resource allocation, or SLURM configuration).

As a client in an HPC system, I’d like to provide constructive feedback to the system administrators. Any guidance or troubleshooting tips would be greatly appreciated!

Thank you!

https://pastebin.com/z4AhC9dK https://pastebin.com/rHX98DBx

[icy stratus]

@radiant stag

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.