Best practices for syncing data from one 3rd party service to another using http requests.

[stark plaza]

Hello,
I'm trying to sync data from one service to another. I pull down the data since the last time I ran the sync code. Then loop through and send each item to another service. There is no bulk upload so I have to make the requests one by one.

Currently I'm looping through and sending off the request then saving locally when the request is successful.

I'm thinking of changing it to storing the data locally first then saving it to the other service and marking it as synced. That way I can always re-try if there is an error.

However, I'm wondering what are some good practices while making http requests in a loop?
When one request fails, do I keep going?
Do I fail silently and let the user know what ones didn't sync.
Do I implement auto retries.
Do i try and run the whole process in a job or make each sync it's own job ?
Do I run it as a background process with sleeps to not overload the other service throttles.

I'm using Laravel Saloon for my http request manager

Thanks.

[fervent frost]

Hello @stark plaza

I haven't used Laravel Saloon yet but I might give you some advice I think?

• I proposed that you might want to use beginTran so if you encountered a failure you can rollback the saved data
• For the fail parts it depends, if you wanted you can tell the user. If the data is sensitive you might want to reconsider it. If you wanted to check for the failed ones you can simply use Logging
• You can auto retries but with set maximum attempt
• Again for the jobs it really depends, if the data is massive I would use queue and job to sync the data, for the follow up I usually do it by batches