#🔒 NaN from non-NaN values when slicing a float16 NDArray

22 messages · Page 1 of 1 (latest)

eager turret
#

I'm having a problem where, when slicing a float16 NDarray loaded from a HDF5 file with indices provided for the slice, all the numbers in that array when read are NaN.

def read_dataset(group, dataset):
    d_m = group["%s" % (dataset)].attrs['d_m']
    size = group["%s" % (dataset)].shape
    read_start = [abs(d) for d in d_m]
    read_end = [s-abs(d) for d, s in zip(d_m, size)]
    if len(read_end) == 2:
        data = group[dataset][...]
        read_data = data[read_start[0]:read_end[0], read_start[1]:read_end[1]]
    elif len(read_end) == 3:
        data = group[dataset][...]
        read_data = data[read_start[0]:read_end[0], read_start[1]:read_end[1], read_start[2]:read_end[2]]
    else:
        raise NotImplementedError("")
    print("Dataset", dataset, "has dtype", read_data.dtype)
    print("Data:")
    print(read_data) # gives NaNs
    print("Original data:")
    print(data) # no problem
    return read_data

Group is a HDF5 Group object, and both data and read_data are NDarrays. The purpose of the function is to strip out halo points from the results of a CFD simulation-in this specific case, read_start = [5, 5, 5], read_end = [134, 134, 134], and the shape is (139, 139, 139), and it does so without issue for float32 and float64 datasets. The console output for any float16 dataset looks something like the attached images.

image.png image.png
rustic jasperBOT
#

@eager turret

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
tall crescent
#

[134, 1334, 134] ?

#

Is that a typo in discord, or also in the code

#

@eager turret

eager turret
#

that's a typo, my 3 key is failing on my keyboard

#

sorry about that

tall crescent
#

Does it work if you do data = group[dataset][...].astype(np.float32)?

eager turret
#

no

tall crescent
#

Can you print data.dtype, and data.shape?

#

and type(data)

eager turret
#

yep that's float16 and (139, 139, 139)

tall crescent
#

Could you try data = group[dataset][...].copy() ?

eager turret
#

Still giving NaN unfortunately

tall crescent
#

I'm not very familiair with HDF5

eager turret
#

yeah I confirmed, HDF5 has its own Dataset object that when sliced with [...] returns an NDarray of the data contained by it

tall crescent
#

I'm not sure then. A NDArray should not give nans when slicing, it would just give a view. If I were you I would use the debugger to make sure all dtypes and values are as expected, and the data contains no NaNs.

eager turret
#

That was a good idea, I checked the indices [0, 0, 0] and [5, 5, 5] and the latter is getting turned into a NaN even if I index the Dataset directly, so I think the problem must be with H5py instead of Numpy. The problem also persists in other scripts like this example code

import h5py

f = h5py.File("opensbli_output_250000.h5")
block0 = f["opensbliblock00"]
rhoE_B0 = block0["rhoE_B0"]
print("rhoE_B0[0, 0, 0]", rhoE_B0[0, 0, 0])
print("rhoE_B0[5, 5, 5]", rhoE_B0[5, 5, 5])```

And I know for sure there's no NaNs in the dataset because I can preview it in a HDF5 viewer + the application that produced the dataset has a NaN check before writing data
#

Not sure if this might end up being a bug in H5py

rustic jasperBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.