lucid kernel
Hey, we currently scrape research content (eg PDFs, web articles, videos and podcasts) to our database from various free research providers. We take the source url, the title, a description, the authors and company.
We use a combination of methods including Octoparse but they are all quite unrealiable and time consuming to setup for new sources. I'm wondering if anyone could advise me whether it's feasible to use Firecrawl instead and whether that would be more robust? It would be amazing if someone has experience with both Octoparse and Firecrawl - or just similar projects.
We are an extremely lean startup - myself and a co-founder. So just knowing if this is worth investing our time in changing processes would be extremely helpful!
Many thanks!
uneven thorn