#Help with zip-rs write optimization

19 messages · Page 1 of 1 (latest)

dusk mango
#

Hi there! So I was reconstructing a zip file (well, a jar) with zip-rs after working with the original zip's data. It seems that the write speed is very slow, even compared to the other operations i did on the data which were much heavier but still finished faster, its taking around 4-5 seconds to write a total of 114mb of data, this shouldnt be a storage speed bottleneck as ive gotten 2gb/s on the same drive before. I've attached my code, i think ive done something wrong to cause such a slowdown

unknown.png
#

forgot to add the tags, woops, added

#

There are 28837 files that are written

#

actually, no i made a mistake, in this code i have ignored a few files (1159)

#

so 27678 files are written in around 4-5 seconds, is there a way to optimize that?

tepid bear
#

and this is with --release?

dusk mango
#

yea

dusk mango
ionic path
#

I wonder if it would benefit from a BufWriter between the ZipWriter and File

dusk mango
ionic path
#

it would be like ZipWriter::new(BufWriter::new(File::create(...).unwrap()))

dusk mango
#

ah okay, thanks ill try it out

dusk mango
dusk mango
#

still no idea why it takes longer to write than to actually encrypt, maybe its just my hashmap iteration way of doing it thats extremely inefficient?

ionic path
#

ZipWriter defaults to performing compression, but compressing encrypted data is usually futile (no significant savings), so you might want to set the FileOptions to disable compression

#

also, you can "pipeline" the work by, instead of collecting all of the data in a HashMap and then writing it, putting each file processed by fire_threads in a channel, which your for loop can read from to write the zip

#

that way you can start writing as soon as one file is ready

dusk mango