#Extremely slow upload to HuggingFace

7 messages · Page 1 of 1 (latest)

marble night
#

For the past 10 hours or so I've had issues uploading to HuggingFace on my pod (jhw1d9hmjb8d3v).

speedtest-cli shows acceptable speeds, but specifically uploading to HuggingFace goes below 1MB/s often.

sterile agateBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

faint gyro
#

try enabling hf transfer, then rerunning it

#

Use hf_transfer: this is a Rust-based library meant to speed up uploads on machines with very high bandwidth. To use it, you must install it (pip install hf_transfer) and enable it by setting HF_HUB_ENABLE_HF_TRANSFER=1 as an environment variable.
You can then use huggingface_hub normally.

Disclaimer: this is a power user tool. It is tested and production-ready but lacks user-friendly features like progress bars or advanced error handling. For more details, please refer to this https://huggingface.co/docs/huggingface_hub/v0.27.0.rc1/package_reference/environment_variables#hfhubenablehftransfer

Environment variables
marble night
#

hf_transfer was installed and enabled. I uploaded everything to a S3 bucket and uploaded to HF from another pod. It's quite annoying though, because this can become very costly very quickly.

faint gyro
#

I see yeah, are you uploading from a network storage?

terse larkBOT