Puppeteer and Proxy Question | Trigger.dev | Page 1

junior barn Dec 16, 2024, 7:49 PM

#

Hi all! I understand that in order to do web scraping, I need to use a proxy. However, I'm not sure if what I intend to do is actually web scraping, and I'm looking for advice... (sorry if this is dumb and the answer is "yes, this is obviously web scraping").

I'm hoping to export data about my Garmin activities from connect.garmin.com. On each activity page, they have an option in a menu to "Export to TCX/GPX". I'm hoping to use Puppeteer or Playwright to login, navigate to the most recent activity, and click that menu option. Is this web scraping? Or just using automation?

Thanks!

polar falcon Dec 16, 2024, 8:06 PM

#

I would personally say that it is scraping

#

Furthermore, since Garmin has an API that they offer for the Garmin Connect product, they may not take too kindly to said scraping

junior barn Dec 16, 2024, 8:11 PM

#

Sounds good! I'll set up a proxy

gritty depot Dec 16, 2024, 8:11 PM

#

Yeah this is scraping because you don’t own the Garmin site, you don’t have permission and they might block the IP address.

But I do sympathise that you just want to access your data. We will have dedicated IPs at some point and will probably have a solution which doesn’t require using a 3rd party

polar falcon Dec 16, 2024, 8:12 PM

#

Have you tried applying for API access first? They say it's for business use, but sometimes you can get approved for those with a well-worded application.

That would be the path of least resistance probably

junior barn Dec 16, 2024, 8:14 PM

#

Unfortunately the api is not available for personal use - I did try applying but was rejected

#Puppeteer and Proxy Question