Incrementally build and write to disk a tf.data.Dataset

[tawdry hull]

1. My goal is to save to disk a tf.data.Dataset without ever having to load all the data into memory

2 & 3. I've tried the following:

for ...
    data = generatemydata(...)
    ds = tf.data.Dataset.from_tensor_slices(data)
    ds.save(path)

This doesn't work because each .save() saves over the previous dataset in the loop. At the end if I try to load this, only the last dataset survived

[hasty sail]

I used the from_generator function to generate my dataset without OOM'ing my system memory BUT I dont know if it will work with your save call.

https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator

[tawdry hull]

Ok, that's an annoying interface that you have to pass an generator...

.. but even so. Suppose I do that, create the tf.data.Dataset, and save to disk. Then later on I want to add more data to the same Dataset. Then this approach wouldn't work... That's what I meant by incrementally.

[echo hornet]

@tawdry hull has leveled up! (0 ➜ 1)

[hasty sail]

Then I'm not sure. How much time would it cost you to reprocess the entire dataset (both seen and new data)? If it's not too much, then I'd just do that.