Change Tracking

[lime rivet]

As I understand it, the current change tracking scrapes the url content and then reports whether its changed or not. Is there a way to check for a change or when the site was last updated without fully scraping the url content again?

[somber iris]

I'm curious why you're trying to avoid doing a full scrape. Are you concerned about the credit cost, or something else?

Realistically, the only way to know if something has changed is to fully scrape it (which is cheap), and then compare it against a previous scrape.

[lime rivet]

i'm mainly concerned about speed / latency in my application. firecrawl is consistenly taking 10-20 seconds per url. it'd be ideal if i could get it down to a few seconds by maybe storing the data after scraping it and then just having some tool that can quickly check whether i need to rescrape or just return what's in the database