🔒 When I export the pandas dataframe as CSV and read it back, the working system of the code changes

[zinc falcon]

When pandas reads the dataframe from csv, the table gets corrupted.

data = {
    'time': ['08:20', '08:21', '8:22'],
    'meters': [[[824, 827]], [[825, 826]], []]
}

df = pd.DataFrame(data)

for index, row in df.iterrows():
    for pair in row['meters']:
        print(pair)

output:
[824, 827]
[825, 826]

when I export this dataframe as csv and read it back

df.to_csv('test.csv', index=False)

df = pd.read_csv('test.csv')

for index, row in df.iterrows():
    for pair in row['meters']:
        print(pair)

output:
[
[
8
2
4
,
...

[vapid vapor]

@zinc falcon

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[lofty warren]

fascinating

the data is the same, the index is the same, the dtypes too

[coarse wigeon]

this was asked in another channel and the issue is having Python lists in the cells of a frame

it's not a natural element to store in a frame

it's not automagically read back as lists from the CSV, but as strings

a remedy is .apply(ast.literal_eval) on the column of interest

[lofty warren]

ohhh

[coarse wigeon]

while also questioning the choice of storing lists in a dataframe

[lofty warren]

so dtype O is used for both strings and lists?

[coarse wigeon]

indeed

[lofty warren]

ahhh

[coarse wigeon]

stands for "object"

anything non-numeric and non-datetime essentially

[lofty warren]

Out[29]: '[[824, 827]]'

In [30]: df["meters"][0]
Out[30]: [[824, 827]]```

it really is list vs str

TIL, thanks @coarse wigeon

[ripe jacinth]

afaik the usual way of storing multidimensional data in a dataframe would be via multiindex rather than lists

[vapid vapor]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.