🔒 Dataframes being weird and annoying

[copper cairn]

I'm working on a web scraper. I have a dataframe df, it has its titles ready and I want to create the rows for it, I have a for loop which adds the rows into the df. But for some reason it will only add the rows until the 36th, when I try to add any more rows it gives an indexing error. The list col_data is 86 values long.

col_data = col_data[1:]
for i in range(0,30):
row = col_data[i]
row_data = row.find_all('td')
row_data1 = [data.text.strip() for data in row_data]
df.loc[i] = row_data1
df

additionally I can print out all the rows and they all come out just fine, it only gets messed up when I try to add all of them in.

[honest canyon]

@copper cairn

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[icy ingot]

for i in range(0,30): perhaps I'm misunderstanding but isn't your for loop only 30 numbers long?

or is that just the code for the example?

[copper cairn]

right now this works, here's what happens if I give it a bigger number

it works for numbers smaller than 37 for some reason

[icy ingot]

so I assume col_data[36] is the culprit, what is in that data point?

[copper cairn]

this is what happens when I print out all the rows

[copper cairn]

it looks the same as the others

[icy ingot]

mb, seems like 37 is the culprit afterall, there are a few N/A values. When you do [data.text.strip() for data in row_data] on that line what do you get?

[copper cairn]

I made it go until 37, the last value is 37

[icy ingot]

what if you copy the text of the Ice Spirit line and try to assign it to the dataframe? like

df.loc[37] = eval(<that line as a string>)

[copper cairn]

so like imma try adding it to the df, after I get the first 36 values in?

[icy ingot]

I wonder if pandas automatically turns N/A into None and then throws the error? but that would be an odd behaviour

[icy ingot]

or just put it in as the first value, make the df and then try to put it in

[copper cairn]

alr

yo, so sorry my pc just doed

died

IM on my laptop now

36 seems to work just fine

[icy ingot]

try with the Ice Spirit input

wherever that is

[copper cairn]

thats the next one i think

I think I discovered something weird

aha!

[icy ingot]

heureka!

[copper cairn]

it adds more than one when I change it by one

[icy ingot]

one thing I was thinking about is that you're using .loc[] with indices, but there is .iloc[] for indices... maybe that might help? not sure

[copper cairn]

so sorry to keep you waiting, Im gonna get some good pictures and come back

ok so my aha moment was a fluke caused by my df not getting reset

sorry Im back I got my pc running again

iloc doesnt work at all

[icy ingot]

do you think you could send the code here so I could run it and see what can be done?

[copper cairn]

sure I can send u the full thing

[icy ingot]

the whole thing including the requests to the website

[copper cairn]

want me to copy paste it or send you the notebook file?

.ipynb

[icy ingot]

copy paste

but please use the appropriate formatting

!code

[honest canyon]

Formatting code on Discord

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

For long code samples, you can use our pastebin.

Hey @copper cairn!

It looks like you're trying to paste code into this channel.

Discord has support for Markdown, which allows you to post code with full syntax highlighting. Please use these whenever you paste code, as this helps improve the legibility and makes it easier for us to help you.

To do this, use the following method:
```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
You can **edit your original message** to correct your code block.

[copper cairn]

from bs4 import BeautifulSoup
import requests

url = 'https://clashroyale.fandom.com/wiki/Cards'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html')

table = soup.find_all('table')[0]
table

world_titles = table.find_all('th')

world_table_titles = [title.text.strip() for title in world_titles]

import pandas as pd

df = pd.DataFrame(columns = world_table_titles)

col_data = table.find_all('tr')

df = pd.DataFrame(columns = world_table_titles)
col_data = table.find_all('tr')

col_data = col_data[1:]

#wont work past 37
for i in range(0,37):
  row = col_data[i]
  row_data = row.find_all('td')
  row_data1 = [data.text.strip() for data in row_data]
  df.loc[i] = row_data1
df

aha finally got it to send]

[icy ingot]

aaah I see the issue, notice the row with Guards, the last value is missing

[copper cairn]

lemme check

oh yeah it is, let me try a quick fix

actually it isnt missing, it has an extra one

after count

weird...

[icy ingot]

yeah just noticed it's extra

[copper cairn]

bruh this is like an error in the actual spreadsheet

theres a blank space here

[icy ingot]

for i, row in enumerate(col_data):
  row_data = row.find_all('td')
  row_data1 = [data.text.strip() for data in row_data]
  try:
    df.loc[i] = row_data1
  except ValueError:
    row_data1 = [data for data in row_data1 if data]
    df.loc[i] = row_data1
df
``` just a simple, rough fix that should work for now

[copper cairn]

I was thinking about cutting off that last value

for i in range(0,38):

row = col_data[i]
row_data = row.find_all('td')
row_data1 = [data.text.strip() for data in row_data]
if i == 38:
row_data1 = row_data1[0:9]

[honest canyon]

Hey @copper cairn!

It looks like you're trying to paste code into this channel.

Discord has support for Markdown, which allows you to post code with full syntax highlighting. Please use these whenever you paste code, as this helps improve the legibility and makes it easier for us to help you.

To do this, use the following method:
```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
You can **edit your original message** to correct your code block.

[icy ingot]

0:10

[copper cairn]

still there

[icy ingot]

oh did I mess up? was it 9 columns in df?

[copper cairn]

ye 9 columns

[icy ingot]

sorry you were right

[copper cairn]

AHA

fixed

for i in range(0,38):
  
  row = col_data[i]
  row_data = row.find_all('td')
  row_data1 = [data.text.strip() for data in row_data]
  if i == 37: 
    row_data1 = row_data1[0:9]

  print(row_data1)
  # df.loc[i] = row_data1

no way it actually works now

THANK YOU SO MUCH

BUT ALSO IM SO PISSED, ALL OF THIS BECAUSE THE WEBSITES SPREADSHEET SUCKS

[icy ingot]

this is a completely normal occurrence when working with real-life data. Treat this as a new skill/experience rather than annoyance

[copper cairn]

understood!

guess thats smth u gotta look out for when web scraping

[honest canyon]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.