Pupeteer unable to find element (dev tools show the element)

[waxen lily]

Hello everyone,

I'm currently working on a web scraping tool that extracts data from the following web page: https://www.handelsregister.de/rp_web/documents-dk.xhtml.
However, I'm running into some unexpected issues and I'm not sure how to debug them.
Here's a brief overview of the issue:

• I'm using Puppeteer to scrape data from the web page.
• The tool is able to navigate to the correct page and locate the element that I want to click.
• However, when I try to extract the element using document.querySelectorAll(), I'm not getting the expected results.
• Specifically, I'm using the following code to extract the first toggle span:

const firstToggleSpan = await page.evaluate(() => {
  const toggleSpans = document.querySelectorAll(
    "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-s",
  );
  return toggleSpans[0]; // prints `{}` to   the server logs
});

• However, firstToggleSpan is an empty object,
• whereas running the same code in the Puppeteer DevTools console, it returns the element that I want to click.

How can I debug pupeteer's page selectors and why doesn't it behave as the querySelector in the dev tools?

[waxen lily]

for whatever reason the handler functions didn't work.

However, page.evaluate() did the trick in the end.

  const firstToggleSpan = await page.evaluate(() => {
    const toggleSpans = document.querySelectorAll<HTMLSpanElement>(
      "span.ui-tree-toggler.ui-icon.ui-icon-triangle-1-e",
    );

    if (!toggleSpans.length) {
      throw new Error(
        "Could not locate the toggle for `Documents on register number`",
      );
    }
    // toggle
    toggleSpans[0]?.click();
  });