#nickdnk-metadata

1 messages · Page 1 of 1 (latest)

vapid vortex
#

Hi there 👋 no, metadata is only available as a field on objects. That being said, if you can find a way to use that field to help with your logging/flows, then you can do that.

charred birch
#

Alright

#

I might be able to d othat

#

I was just wondering if there was like a general "request custom field" I could use instead

#

But i'll just add to the objects

vapid vortex
#

Gotcha, yeah there is nothing like that currently, but it's an interesting idea.

charred birch
#

It could help with tracing your own requests to the logs at least.Can you perhaps tell me how far apart req_B4yxmVEzJMV3Zi and req_Qif5lX8XYIt1Oa is more accurately than seconds, if you have that data?

#

like if you have higher resolution

#

I'm expecting you don't

#

just checking

vapid vortex
#

Most of our tooling doesn't go that granular, but taking a closer look they seem to be about 3 ms apart, with req_B4yxmVEzJMV3Zi being listed first.

charred birch
#

Alright thanks

#

That's awfully close

#

As you may have guessed I'm having some issues with webhooks processing twice

#

very rarely

vapid vortex
#

Aahh gotcha

charred birch
#

It's inserting the event into into a unique column in mysql, then grabs the row with a lock on it before proceeding, so processing it twice should be impossible, but somehow it still happens

#

back to the drawing board I guess

vapid vortex
#

Hm, yeah I'd assume that lock would help prevent this. Are you relying on the data in the webhook, or are you just using them as a trigger and retrieving the relevant object from Stripe during processing?

charred birch
#

So the flow is:

webhook takes the webhook data and sends it to SQS (AWS worker queue), which does not guarantee FIFO or only-once processing. Multiple workers could get the same job. So to prevent that if only-once is required:

Insert the event row with a unique constraint on the Strip Event ID. This can only succeed if it's not already there. MySQL should guarantee this or the world would implode, so I'm pretty sure that works. There's also a column called "success" which has either a 1 or a 0. This starts as 0 when inserted. We ignore duplicate errors on the insert.

Then we proceed to SELECT success FROM webhooks WHERE ID = ? FOR UPDATE and if we get that lock (which is exclusive), we continue processing the webhook. We don't release the lock (commit/rollback transacton) until the webhook is completed. Before releasing, we set the success column to 1 OR we delete the row entirely if there was an error. This should result in:

Job comes in twice:

  1. Inserts, success

  2. Inserts, duplicate (ignores error)

  3. Grabs lock, processes webhook

  4. Waits for lock...

  5. Processes webhook, updates success to 1, commits and releases lock

  6. Gets lock, sees that success is now 1, returns and discards the job

So somehow we still end up with the job running twice within 3ms. Am I overlooking something obvious here?

#

I tested this, obviously and it does wait and get success 1 then discards, when I run it locally

#

And to answer your question: It depends. Sometimes we fetch from the API, sometimes we use objects directly. It depends on the case.

#

I just realised that "waits for lock" could actually also be "row not found" if the first jobs fails and deletes the row, however that does not explain this 3ms race condition, and in this particular case the first job did succeed anyway.

vapid vortex
#

Apologies, it's been a while since I've thought through DB lock flows like this, but the process you're describing sounds like it would be effective. The only thing that comes to mind is if that lock is behaving in an optimistic fashion, but if it's exclusive then shoots a hole in that theory.