How to use polling in a horizontally scaled application.

[devout lynx]

Hi, I use my Nestjs backend to receive leads via webhook or to collect them via polling (if the provider does not offer the possibility to send us the leads).

I currently use Cron from @nestjs/schedule for polling. If I now scale my Nestjs application horizontally, e.g. via docker scale or kubernetes, the job would be executed several times over the number of instances.

I have read that with Redis you can use Redlock to lock the job for all but one instance so that it is not executed multiple times.

[slate solar]

Multiple options:

1. Use env variables set in only one replica that would enable the schedule there and disable it otherwise

2. use scheduled (delayed) jobs with redis and Bull

3. implement distributed locking (as you suggested)

4. use an external scheduler application that runs as a single instance and periodically pings a service endpoint of the scaled app which triggers the pull

5. same as 4, but use a scheduler as as service (most cloud providers offer this)