#🔒 Asynchronous writing to a large NumPy array

25 messages · Page 1 of 1 (latest)

warm raptor
#

Hi everyone, I have a large number of NumPy arrays (say, N) of the same size (say, M) and dtype. I want to combine them into a single large NumPy array (N x M) and save it in a file. How can I do that efficiently without assigning the values one row at a time? Can this be done asynchronously? Also, each input NumPy array is an attribute in a dataclass object. Any help or suggestion is much appreciated. Thank you!

spring zephyrBOT
#

@warm raptor

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
boreal pilot
#

I encountered this quirk when working on an oscilloscope processor and the output stream got glitchy because apparently my line drawer was faster than i could output

warm raptor
#

Thanks! I was also just looking at zarr. It supports async read/write. Could that be a better alternative than doing it only in numpy?

limpid bone
#

What if you allocated the single big array, and then did a direct memory copy of the underlying array data of the individual arrays into the underlying array of the single big numpy array. You'd need to get the layout right, but then you could just blit the data.

#

Not sure why doing this asynchronously (if you mean Pythons async stuff) would help.

warm raptor
#

Thanks! I mean, I'd like to parallelise it somehow.

limpid bone
#

I'm just not sure parallel access would be faster. But I guess multithreading the blits might be a speedup, if the blit releases the GIL. (And I'm not sure how I'd do the blit - probably trying big_underlying_array[memory_indices] = individual_underlying_array[all_of_it].

warm raptor
#

I'm also worried that there's not enough RAM. Another requirement is that I'd like to do random access on the N x M array when reading later on.

limpid bone
#

If you're genuinely short on RAM you could put it all in a file an mmap it? Another thing I've never tried with numpy. But I have mmaped a file as an array.

#

(Keeping in mind in our heads the distinction between a numpy array and the underlying array.array it's wrapping to store the values.)

warm raptor
#

I want to write the combined array to disk really. Maybe memmap is the way to go?

limpid bone
#

So possibly there's a

  • make the big file for the big array (just seek to the offset and write one byte, on UNIX anyway)
  • mmap it
  • make a numpy object supplying the mmap as the basis for the internal array
limpid bone
warm raptor
#

Ah, great. I'll give that a try then.

limpid bone
#

SO you make the file, mmap it, write data into the mmap (a buffer, looks like RAM). All done.

warm raptor
#

Read should be easy if I want to access specific rows?

#

Ah, I can just load it as if it were a normal numpy array.

limpid bone
#

Bearing in mind that whle it looks like memory, only what will fit will be in actual memory - things get paged in/out as you access this. If you've got lots of RAM you're fine, but after that there's the usual costs of using virtual memory.

#

I'd hope that sequential access would be pretty performant.

#

Need to go AFK, sorry.

warm raptor
#

Thanks a lot both!

spring zephyrBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.