change storage dir programaticly | Apify & Crawlee | Page 1

grand talon Jun 10, 2024, 12:29 PM

#

Hi All,

I want to use crawlee in kubernetes. I want my jobs to be resumable if a pod gets evicted so I have set up a PV for storage. This however poses an issue if I have multiple pods running at once. To solve this I want to change the storage dir programaticly when instanciating crawlee. I know I can do this through ENV vars however id prefer a more programatic solution if possible?

I have looked at the constructors for the (Playwright) Crawler and the Configuration class but I dont seem to be able to set it.

Thanks!

lime nymph Jun 11, 2024, 7:48 AM

#

Hello,

There is https://crawlee.dev/api/core/interface/ConfigurationOptions, you probably have to pass a new StorageClient https://crawlee.dev/api/core/interface/StorageClient > https://crawlee.dev/api/memory-storage/interface/MemoryStorageOptions

StorageClient | API | Crawlee

Represents a storage capable of working with datasets, KV stores and request queues.

MemoryStorageOptions | API | Crawlee

dense valley Jun 11, 2024, 7:55 AM

#

You can define storageClientOptions in the Configuration constructor, where you can use the MemoryStorageOptions, see the links above. Then, you can pass the Configuration object to the crawler constructor.

#change storage dir programaticly