#🔒 Trouble with Pandas Dataframe and nested Lisits/Dictionaries

12 messages · Page 1 of 1 (latest)

weary nova
#

Hello everyone, thank you for taking time to read my post.

I am trying to create a Pandas dataframe and am having trouble with the determining the best approach. I have nested data as shown in the picture. The list "CleanProductCollection" is a list with 12 dictionaries. Each of the 12 dictionaries has two keys, one of which is a list called 'data'. These lists contain 50 dictionaries each.

It annoyingly nests further than that; however, I can work with the data in the dictionaries from the list 'data'. I am struggling to understand how to best extract all of the 50 dictionaries data from each of the lists. If I do the following -
df=pd.DataFrame(CleanProductCollection[0]['data'])

And then df.to_csv, it gives me a csv in an acceptable format; many of the cells have a lot of data, due to further nesting. How can I perform this for all dictionaries in all nested 'data' lists?

image.png
inner tideBOT
#

@weary nova

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
blazing urchin
#

uhh, sorry, json_normalize

#

!docs pandas.json_normalize

inner tideBOT
#
pandas.json\_normalize 

pandas.json_normalize(data, record_path=None, meta=None, meta_prefix=None, record_prefix=None, errors='raise', sep='.', max_level=None)```
Normalize semi-structured JSON data into a flat table.
weary nova
#

I will read this over again. I had looked at it, but it didn't seem to do anything different. As in, the csv looked the same. But I may have not used it correctly.

blazing urchin
#

You could also use something iterative like, say,

lst = []
for el in CleanProductCollection:
    lst.extend(el["data"])
df = pd.DataFrame.from_records(lst)
weary nova
#

I tried json_normalize and it did give different results, I just did not notice.

df=pd.DataFrame(CleanProductCollection[0]['data']) - 50 rows x 76 columns
json_normalize - 50 rows x 137 columns

I am going to try your suggestion as well.

#

when I used the iterative approach, it gives 343 rows x 76 columns. I will need to review it and play with this.

inner tideBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.