🔒 Potential Bug with the polars library ? Unexpected Behaviour

[unique trench]

I was working on fixing a bug in my data pipeline and i noticed i was getting unexpected values for a column,
I have abstracted the operation into a simple dataframe with 4 rows for the sake of replicating the bug.

Basically the condtion for the last row is not working as expected,
Comparing a NaN value should yeild False for the first 2 condtions, yet I still end up with the first condition passing through.

Code -

import numpy as np
import polars as pl
data = {
'x' : [1,2,3,4],
'y' : [4.33,-4,54.7,np.NaN]
}

df = pl.DataFrame(data)

df2 = df.with_columns(
pl.when(pl.col('y') > 0.02).then(pl.lit('P'))
.when(pl.col('y') < -0.02).then(pl.lit('Q'))
.otherwise(None).alias('z')
)
df2

[wheat aspen]

@unique trench

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[ocean geyser]

That does seem like unexpected behavior. There is a discord server for Polars you may want to ask there? As a work around, you should be able to fill_nan(None) first?

[unique trench]

yep , fill_nan seems to be the go to approach

[stuck mauve]

the polars devs replied in the polars server, but quoting here for context

looks correct, see https://docs.pola.rs/user-guide/concepts/data-types/overview/#floating-point

Any NaN compares equal to any other NaN, and greater than any non-NaN value.
it is a weird feature, but a feature nonetheless

feel free to !close here

[frail shale]

It's both expected and unexpected

Because some of the docs show using np.nan, which will give False on both np.nan > 0.02 and np.nan < 0.02

I was only able to replicate the behavior by using ham = pl.lit('NaN', dtype=p.Float64) and comparing with ham

It's bizarre

[wheat aspen]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.