What Services Should I use? | Cloudflare Developers | Page 1

sterile pollen Nov 3, 2023, 5:45 PM

#

Hello everyone. I'm looking to build an application that can achieve the following:

Receive event data from Shopify's web pixel API (https://shopify.dev/docs/api/web-pixels-api)
Send the event data to storage (ideally to BigQuery but open to something else)

I'm unsure what all the tools I would need for this.

Should I use a worker to create an endpoint for ingesting the events?

Then connect the worker to BigQuery? BigQuery only allows 1500 "load jobs" per day so I couldn't insert events 1 by 1. How could I batch and load say every minute?

Or should the worker be bound to D1? R2? (not sure what the major differences are there).

Thank you to anyone with thoughts and ideas 🙂

daring magnet Nov 3, 2023, 5:51 PM

#

D1 is sqlite based database, R2 is S3 supported storage (basically for your files). So if you wanna run SQL commands on your data you better go with D1. Or you can use Workers-KV (key value store) if you need to access them fast. Question is how much data you're gonna push to your desired infrastructure?

sterile pollen Nov 3, 2023, 5:52 PM

#

It's going to be a lot. Think all the events that happen on an ecommerce store. Page View, view item, add to cart, purchase, etc.

daring magnet Nov 3, 2023, 5:54 PM

#

Then you need to use another cloud database because cloudflare has some limitations on D1 (https://developers.cloudflare.com/d1/platform/limits/) And since it's still in beta...

sterile pollen Nov 3, 2023, 5:56 PM

#

Right, so I was thinking BigQuery. Any thoughts on that? And batching for inserts every minute?

daring magnet Nov 3, 2023, 5:58 PM

#

I've never used or dig into it sorry. But you can check aws db products. also I didn't put them side by side.

sterile pollen Nov 3, 2023, 5:59 PM

#

Thank you so much for your inputs MK! Really appreciate you jumping in.

Wondering if anyone else has some thoughts to?

#What Services Should I use?