Disk reading unacceptably and mind boggingly slow

[fierce fox]

I thought I had figured out where to put my data:
pay for extra disk space in /
move the data from /workspace (which is a network drive) to / and from there it can be read fast enough.

But today, I tried the same thing in this pod:

RunPod Pytorch 2.1
ID: yuicz8xifys47n
1 x A100 SXM 80GB
16 vCPU 250 GB RAM
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04

and the reading speed from / was as horrible as reading from /workspace.
the pod is down now cause obviously i'm not keen to pay for useless machinery

So what are we supposed to do I don't understand how anyone can do ML like this.

[hallow glacier]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[opal gust]

Network drive is garbage, I agree. Someone else had this issue, and I think the way they solved it is they use network drive as a storage area, and then copy out into the container disk area, and they found substantially sped up their workflow

[fierce fox]

Hi Justin, I'm pretty sure I am that someone since I remember chatting with you!
And the problem is that yesterday even though I copied my data to the container disk area ( at the root of the filesystem if I'm not very mistaken: "/" ), the reading was just as slow as from the network disk.!!

Same problem today

RunPod Pytorch 2.1
ID: 5vzts37ixeqaz7
1 x A100 80GB
8 vCPU 188 GB RAM
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
On-Demand - Secure Cloud

My data is in /scratch

and the script that just reads records from the database one by one works at 2 iterations/second.
On a computer where the .db is read from an SSD this runs at 600 iterations/second.

So i'm pretty sure there's pods where even / is on the network ?