Just tried out a t3.small instance without much luck (running out of memory). tried a r3.large wich looks better but seems to be week on cpu. any hints?
#which ec2 instance type is best suited for crawling?
marble charm
coral boltBOT
@marble charm just advanced to level 4! Thanks for your contributions! 🎉
cedar token
Depends a lot if you are using HTTP based or browser based. HTTP 1 GB + 1/4 CPU is ok, for browser at least 4x as much
main dagger
Depends a lot if you are using HTTP based or browser based. HTTP 1 GB + 1/4 CPU ...
Just curious to know if someone wants to scrape 8-10k pages per day then will 4GB RAM VPS be sufficient? 8-10k pages over period of 24hrs not in a single go
cedar token
It should be for sure. But as I said, optimizing your scraper first can give you much more
main dagger
It should be for sure. But as I said, optimizing your scraper first can give you...
Any rough estimate like how much ram would need if I want to scrape 20k pages per day? Each request will spin up new browser instance and close it. 20k requests per day.