Webhook ETL question

[lyric hazel]

Hi all! we are trying to replicate the Mollie payments, including status updates to a BigQuery environment for reporting purposes. However the webhookurl is already configured for production purposes. We tried using the payments REST endpoint but it has no modified_at timestamp so it now gives us the full reload of the table including all history, which is of course very inefficient.

Question: We are now thinking to fetch GET Payments filtered on created_at timestamp (e.g. 2 weeks to make sure statusses are mainly final) and then the refunds and the chargebacks seperately. Are we then missing another status update that needs to be included?

Any other tips on how we could set this up most efficiently? Thanks!

[distant field]

hi hello!
Just to understand: are you also in control of the application that currently processes the webhooks? If so, I'd suggest duplicating the webhook yourself. You could have a look athttps://github.com/fjbender/mollie-webhook-multiplexer
In case you cannot apply the above, periodically polling the different transaction endpoints (Payments, Refunds, Chargebacks, Captures) is something we see happening often. If you do that once a day, your BigQuery environment isn't that much behind

[lyric hazel]

Thank you! We are not in control of the application indeed. We simply replicate the data for reporting purposes. Alright, thanks we will follow up on this approach. However its a bit of a pity in this case we need 4 endpoints (whereas the refunds and chargebacks are also in the payments endpoint) - so in case there would be a modified timestamp in the payments API that would save us time