Is there an option for more disk space? One of my tasks will need to download two very large files at times, easily surpassing 10GB.
#10GB Disk Space - Option for more?
wintry trench
burnt saffron
Would be nice also for me. 10gb is close to what i need, but having just a little bit more would be very convenient (possibly 20gb)
silk hemlock
What machines are you guys running on? https://trigger.dev/docs/machines
The easiest change (that we could do fairly easily), would be to have the smaller machines have less disk space and scale it with the machine size.
How much disk space do you need @wintry trench?
We have a plan to use distributed disks so there are no disk space limits. There would be some paths you can write to that are actually backed by something like JuiceFS. That would also allow you to write files that are shared between your runs. But this is at least a few months away from happening…
silk hemlock
We've had a few other people ask for this and we might change the disk space on the machines soon. Let me know what you need and what machines you're using right now, and we'll see if we can make it work.
burnt saffron
@silk hemlock share files across runs 😮 amazing! The fact that you are even just considering this -- is just great. Can't wait for that change 😄
@silk hemlock that means, no more S3 storage for storing output, I guess.
silk hemlock
We'll continue to support large payloads/outputs invisibly using cloud storage like we do now. But they're limited to only 10MB anyway…
The other thing we're thinking of doing is that you can take any of these files and generate a presigned URL from them with a simple function call.
So you can basically write files to "disk" normally, then at the end of the function if you want to make that file available elsewhere you can return the presignedUrl for it.
In the shorter term, we will change the disk space so the smaller machines have a bit less disk space and the larger machines have more.
burnt saffron
@silk hemlock Currently I have small-1x which uses 10gb, reducing it would be quite a breaking change for my runs 😦
Given that my run is responsible is downloading files from a source into another as a proxy. Increasing machine performance just for storage is also quite expensive.
However JuiceFS solution that you mentioned that would allow us to share storage across runs (hopefully with high speed), it super good news .