Adding to an array without overwriting current value | JavaScript Mastery - Programming & Coding Community | Page 1

Hello,

I'm trying to work with some scraped data and can't figure out how to accomplish my goal. I've done some research on my own, but maybe i'm not approaching
this correctly. Essentially the site im scraping from only displays 30 articles at a time so I want to scrape > map to array > check number of articles > if less than 100 > load more >scrape > repeat

"dependencies": {
"playwright": "^1.39.0"
},

// Function to retrieve articles
const getArticles = async () => {
return await page.$$eval('.athing', articles => {
return articles.map(article => {
const titleElement = article.querySelector('.titleline');
const timeElement = article.nextElementSibling.querySelector('.age');
return {
title: titleElement.innerText.trim(),
time: new Date(timeElement.title).getTime()
};
});
});
};

let articles = [];
let previousLength = 0;

while (articles.length < 100) {
// Get current articles
articles = await getArticles();

console.log(`Loaded ${articles.length} articles`);

// Check to see if the target number of articles has been reached
if (articles.length < 100) {
  const moreButton = await page.$('a.morelink');
  if (moreButton) {
    await moreButton.click();
    await page.waitForTimeout(3000); // Wait for new articles to load
  } else {
    break; // No more "more" button, exit loop
  }

  // Confirm length increased
  if (articles.length == previousLength) {
    console.log('No new articles loaded. Exiting.');
    break;
  }
  previousLength = articles.length;
}

}

// Slice the first 100 articles

articles = articles.slice(0, 100);

I fell like maybe i'm overcomplicating this as I've been staring at the issue too long. Feel free to withhold solution in leu of pointing me in the wright direction. I'm not looking for someone to fix my issue, but rather some guidance

#Adding to an array without overwriting current value