#🔒 BeautifulSoup Help

8 messages · Page 1 of 1 (latest)

wet ledge
#

I'm trying to use BeautifulSoup to collect all the sentences in a website as well as the section they fall under (sections fall under "<strong>" tags). However, I have to account for the fact that sentences can appear in a variety of elements: <p>, <table>, <blockquote>, even inside of a <div>...

I have to account for nested tags too, so as to make sure that I don't collect a sentence more than once. I have tried using soup.find_all(text=True) to bypass that, but that doesn't seem to necessarily go through the website in the order I want (top to bottom). Any advice?

sterile baneBOT
#

@wet ledge

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
wet ledge
#

For example, I have done:

        for section in all_sections:
               if section in elem.get_text():
                    curr_section = section
        if curr_section != None:
                dictionary[curr_section] += elem.get_text().replace(curr_section, "")

The issue with this one is that it does not go through the website from top to bottom. It seems to first go through all the <p> tags, and then at the very end it goes to a <table> tag (which is closer to the top of the page).

#

I'm also not aware of a library for webscraping that better deals with this issue.

steel egret
# 
from bs4 import BeautifulSoup as bs

example = """
<html>
<ul>
   <li>Lorem ipsum dolor sit amet, consectetuer adipiscing elit.</li>
   <li>Aliquam tincidunt mauris eu risus.</li>
   <table>
   <tr>
   <td>banana</td>
   </tr>
   </table>
   <li>Vestibulum auctor <b>hi!</b>dapibus neque.</li>
</ul>
</html>
"""


soup = bs(example, features="lxml")
for child in soup.recursiveChildGenerator():
    if isinstance(child, str):
        tag = child.find_parent().name
        if child := child.strip():
            print(tag, repr(child))

# output:
# li 'Lorem ipsum dolor sit amet, consectetuer 
# adipiscing elit.'
# li 'Aliquam tincidunt mauris eu risus.'      
# td 'banana'
# li 'Vestibulum auctor'
# b 'hi!'
# li 'dapibus neque.'```
wet ledge
sterile baneBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.