Creating WARC files with crawlee | Apify & Crawlee | Page 1

tepid echo Feb 3, 2026, 9:59 PM

#

I just read this amazing blog post about creating WARC files with crawlee: https://crawlee.dev/python/docs/guides/creating-web-archive.

I was considering using the wayback proxy to constitute my archive collection.

Still I am wondering, if I need to set up some datacenter proxy to bypass some rate limit, is it possible to use it along with the wayback setup?

Like, can I use the wayback proxy for archiving on top of other proxies? Does crawlee allow such set up?

Thanks 😘

abstract gorgeBOT Feb 3, 2026, 9:59 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

finite forge Feb 4, 2026, 10:35 AM

#

From our internal support:


It should be possible to setup fixed proxy for the recording server. I think the server looks at some environment variables (HTTP_PROXY, HTTPS_PROXY  ???) . Then it will use the same proxy for all the requests.I am not sure if it is easily possible to do dynamic per requests proxies. This would have to be experimented with and I do not know it without try it. Probably not possible out of the box and you would need to write some subclass of the default recording server to handle dynamic proxies.```

#Creating WARC files with crawlee