#nickdnk-pm
jovial herald
Actually I think I see it
But feel free to confirm
I assume req_kYhIgsCRhcz5Up is it
Huh this is weird.
Looks like we temporarily consumed the paymentmethod after it was cloned
Looking more
robust raft
req_kYhIgsCRhcz5Up
Yes
And yes it's very unexpected. I code pretty defensively and have not had this happen before, ever, and we re-use payment intents all the time for this case
jovial herald
Ah okay actually the issue here is that the paymentmethod was never attached to the customer being passed in the request (cus_LDwSwXD2hZNmil)
Wait, no, that's not right...
There already was a successful clone with that paymentmethod and customer: req_NcLxzqmSY0sydD
And that is why it was consumed. Because it wasn't attached to the customer first.
So yeah, you'll want to take a look at why the paymentmethod was not attached to that customer @robust raft, or why the clone request got duplicated if you aren't attaching the paymentmethod to a customer first (though I would expect you are since you passed the customer ID in the clone request).
jovial herald
It did work
You successfully cloned, and then you attempted to clone again.
Which errored
robust raft
Can you send me those request IDs?
robust raft
Okay
jovial herald
That is the successful clone
robust raft
so that succeede
jovial herald
If you inspect cus_LDwSwXD2hZNmil, you will see the paymentmethod was never attached to that customer
robust raft
So yeah, you'll want to take a look at why the paymentmethod was not attached to...
So what you mean is that I should find out why I attempted to clone and attach twice
as that's the only thing that fits
Since we're doing that in one request
or
jesus this is so hard to debug because it's half connect half platform
jovial herald
You mostly just want to look at your platform here
Are you expecting the paymentmethod to be attached to the customer?
But also yeah, you will want to find out what triggered a duplicate request.
Looks like you have had this error happening sporadically for a while now (you can see it if you filter your platform logs for outgoing Connect requests and status: failed and endpoint: /v1/payment_methods
robust raft
When I receive a payment method and a "yes, save the card" boolean I fetch all current payment methods and do nothing if it's the same card (if they re-enter their card again I don't need to do any connect-replication). If it's not the same card I just attach it to their customer immediately. Maybe if this request gets sent or processed twice for the same payment method object we'd end here
Yeah I see, it was just never reported to our error logging before
req_nwFcbpOQRi0qwO and req_53srizsBLZtX4U have the same payment method ID
So it seems that's probably what's happening
but it's weird because that should just land them in my "this is already your saved payment method" logic I just mentioned
jovial herald
Yep that customer did already have that card attached. pm_1KXUyLL7ilRdQXxEx5jGRhXF has the same fingerprint
jovial herald
but it's weird because that should just land them in my "this is already your sa...
Yep that's probably where you want to debug
robust raft
use Stripe\Card;
use Stripe\PaymentMethod;
$newCard = $paymentMethod->card;
/** @var PaymentMethod[] $paymentMethods */
/** @noinspection PhpUnhandledExceptionInspection */
$paymentMethods = PaymentMethod::all(
['customer' => $user->getStripeCustomerID(), 'type' => Card::OBJECT_NAME]
)->data;
foreach ($paymentMethods as $existingPaymentMethod) {
/** @var Card $existingCard */
$existingCard = $existingPaymentMethod->card;
if ($existingCard->fingerprint === $newCard->fingerprint
&& $existingCard->exp_year === $newCard->exp_year
&& $existingCard->exp_month === $newCard->exp_month) {
// Card already exists as payment method. No update needed; return existing payment method.
return $existingPaymentMethod;
}
}
Anything jumping out to you there?
With two failing requests being 7 seconds apart it seems unlikely that it's a race condition
jovial herald
Oh you are looking at expiry too
I don't think fingerprint cares about expiration
Just card number
robust raft
Exactly, so I include it
regardless, same payment method object ID would be identical
In this case expiration is irrelevant; all parameters of the card would be the same
jovial herald
No I'm saying that they won't have same expiration but will have same fingerprint?
No they aren't
Let me double check but pretty sure expiration is different
robust raft
Why wouldn't they have the same expiration?
For the ame paymen method object ID?
How?
robust raft
I'm sorry but how would the client send me the same payment method ID referencing a different card?
jovial herald
The client never sends you a payment method ID. They enter their card details, we create a payment method ID.
robust raft
And they do that once
jovial herald
Okay let me walk it through
robust raft
We're working with one payment method ID
jovial herald
Your customer enters their card which ends with 4242 and expiry 12/23 this creates pm_123 and has fingerprint: abc
Then they come back and enter their card again later. This card ends with 4242 (same card number) and expiry 4/25. This creates pm_567 and had fingerprint: abc
Because the fingerprint is the same but the expiration is different, this breaks your loop
robust raft
Yea, and I'd want to replace that card
exactly because it's not the same
But that's not what's happening here
jovial herald
Okay I see. I thought you were updating based on the above, not passing over.
Okay so then yeah nothing jumps out about that code
robust raft
No I return from that loop if it's the same payment method you already have
robust raft
Because in that case we have no reaason to update it
No problem you couldn't know
So yeah, somehow I get past that loop and send the "please connect this payment method ID to the customer" twice
Logically I just can't see how that would happen, especially not with requests being like 7 seconds apart
jovial herald
I'm seeing the clone requests ~30 seconds apart
robust raft
Exactly that just makes it worse
Because the check should filter it out
as it's already attached
is it perhaps because I filter by type 'card' ?
Like some cards have a different type and get excluded?
jovial herald
Wait. Why didn't the card get attached in this case?
jovial herald
is it perhaps because I filter by type 'card' ?
No don't think so
robust raft
$paymentMethods = PaymentMethod::all(
['customer' => $user->getStripeCustomerID(), 'type' => Card::OBJECT_NAME]
)->data;
Hm
robust raft
Wait. Why didn't the card get attached in this case?
Why it didn't or why it shouldn't? It shouldn't because it already was
And if it already was, that loop should find it?
robust raft
uhm
req_nwFcbpOQRi0qwO and req_53srizsBLZtX4U
same payment method ID
no way that can be different expiration
jovial herald
Right right, but that card wasn't already attached to the customer. Ignore the duplicate request for a second.
robust raft
Alright
jovial herald
Isn't the flow that you should attach the card to the customer if it is not identical?
Then clone it?
robust raft
Yes
(just confirmed in the code so we're on the same page)
I update platform customer if card !== same expiration and same fingerprint
then clone
if it's the same payment method (same fingerprint and expiration) I just fetch the current payment method from customer and clone that
jovial herald
Okay this is interesting...
The expiration was not actually different (I was wrong before)
BUT
The customer had TWO previous paymentmethods identical to that one and both were attached....
As in... one was attached first
Then it was removed and the other was attached
Then for this third one it was not attached (but still cloned)
robust raft
Still cus_LDwSwXD2hZNmil ?
jovial herald
Yes
robust raft
But that sounds right doesnt it?
Attach because different, then clone
repeat (maybe switching between cards)
Receive identical card and only clone
jovial herald
- pm_1KXYhJL7ilRdQXxEthiLpn5s is the paymentmethod from the error (not attached but cloned)
- pm_1KXUZpL7ilRdQXxEfxZFX0Cn was the initial default on the customer (attached on customer creation)
- pm_1KXUyLL7ilRdQXxEx5jGRhXF was attached and replaced pm_1KXUZpL7ilRdQXxEfxZFX0Cn
robust raft
and are pm_1KXUZpL7ilRdQXxEfxZFX0Cn and pm_1KXYhJL7ilRdQXxEthiLpn5s identical?
jovial herald
Yes all three are identical
robust raft
okay
jovial herald
all have 5/23 expiration
robust raft
Then I still don't know what the problem is
because that sounds like what I want
except for the dupe
no wait pm_1KXUyLL7ilRdQXxEx5jGRhXF was attached and replaced pm_1KXUZpL7ilRdQXxEfxZFX0Cn shouldn't happen
if it's identical
jovial herald
Right
That's why I was saying interesting.... as it doesn't follow your expected behavior
robust raft
It's so difficult to debug when the dash says stuff like A card payment method ending in 1122 was detached from customer cus_LDwSwXD2hZNmil
instead of including the object ID in the row
A card payment method (ID HERE) ending in 1122 was detached from customer cus_LDwSwXD2hZNmil
would help
robust raft
and yes I know
cards are not the same tough for alle these requests
created card is 05/23 oeWELOVOujTCnzL0
replaced by 09/25 PoUA9vT7bL2sEv7m
jovial herald
OH there was a card inbetween
robust raft
Back to square one 😄
jovial herald
lol 😦
robust raft
I mean this all looks right to me
jovial herald
Yeah everything seems expected except for the duplicate clone request
robust raft
although I'm unsure how they ended up with "detached"
jovial herald
What is the code to trigger the clone?
jovial herald
although I'm unsure how they ended up with "detached"
Not sure what you mean by this?
jovial herald
When?
robust raft
cus_LDwSwXD2hZNmil no card?
robust raft
soryr I was on the event log
yea that all looks correct
$payment_method_connect = PaymentMethod::create(
[
'customer' => $user->getStripeCustomerID(),
'payment_method' => $paymentMethodId
],
[
'stripe_account' => $reservation->getStripeAccountId()
]
);
so $paymentMethodId is either the saved payment method (if they send us the same card) or the new one we just attached
jovial herald
What is triggering that though? I think that is what you need to investigate. I think the frontend action must come into play here with the root cause
jovial herald
Sure, you can only clone server-side. But what hits that endpoint?
jovial herald
What client action is causing all of this? The "save card" button on your UI?
robust raft
the purchase button
you can save while paying
and that's locked so you can't click twice
and we use the stripe SDK for idempotence
30 second timeout
on client
and if they refresh it would generate a new payment method ID from stripe.js
same if they click again after a failed attempt or whatever
And if it's a race condition it has to be so fast that the loop won't catch it
robust raft
Which logs exactly?
jovial herald
You have a duplicate confirm happening for the payment
Take a look at your platform logs for the time of the error
~11:25pm yesterday
robust raft
req_36IMCLU9h1iqFY ?
This is the user clicking "pay" again and again as far as I can see
until they get rate-limited
unless you're looking at something else
jovial herald
Yeah that one
robust raft
That's expected I'd say
jovial herald
So they hit pay again... but you are saying this wouldn't re-trigger the server-side logic to clone again?
robust raft
Only if they use a different card
jovial herald
The requests are pretty far apart so that may just be a red-herring
robust raft
Oh clone, yes, always clone
sorry my brain keeps thinking update == clone
we always clone the payment method
and if they retry, they do it with a new payment method
jovial herald
So that would cause the duplicate clone request then, no?
robust raft
as you can see in req_9E4QTw9gMkHo53 and req_OffDSaJR4Ks18v
no cause it's a new payment method
stripe.js generates a new one for each attempt
look that those two
same payment intent
confirmed twice
two different payment methods
each cloned once
so the card might be the asme
but it's a new pm object
same for req_JjDgV4KHufDhb5 and req_A7wIs5FfM5aJUd
this is expected
jovial herald
Ah okay yes different paymentmethods.
robust raft
happens all the time and way more frequently than that error we're chasing
we sell nightclub tickets to teenagers
and apparently they're all broke
so they try a lot of different cards
jovial herald
Gonna take a look at the last time this error happened
Feb 15
Ugh we need timestamp filters in DB logs
robust raft
Yea and I imagine the tools you have at your disposal are better than mine
so you can imagine how hard it is for me to find 😄
I still don't see how this can happen. If I attach the pm and immediately clone it, then this should only go wrong if my server somehow receives the same pm object twice
I mean I could just override the idempotency key for the attach and clone requests with the payment method ID hashed in there
instead of a random uuid
for alle the requests in the flow
hmm... maybe dangerous
Then I'd have to handle the case and not do anything on my end if idempotency was used but somehow still respond with the order
Can probably work that out though if that's the case
jovial herald
So... the duplicate from Feb 15 also happened almost exactly 30 seconds apart...
Wondering if that 30 second timeout you were talking about has something to do with things here
robust raft
A clue
It might. But how. I mean if it times out and the user clicks it again, we're back to a new payment method
but the frequency of the error does indicate that something like networking could be the culprit
jovial herald
Yeah networking seems like it could be it... the other times this error has occurred (when it happens more than once in a day) are all within 1 minute
robust raft
but all 30 seconds apart also?
Our origin timeout on cloudfront is 30 seconds
I should probably up that
I'm gonna bump that to 60
jovial herald
No not all 30 seconds apart
robust raft
all above 30?
jovial herald
Yeah
robust raft
I don't know, I still don't understand how this means we end here
Our backend code does not stop running, the client just gets a timeout
but it should complete
and if they click the button again, they generate a new payment method and start over
jovial herald
Yeah I don't know. Something is causing your backend code to get hit twice with the same objects
robust raft
(and we have mysql locking in place to ensure they can't purchase twice)
it might even be infrastructure
like a load balancer resending the request to a different application server
I have increased the origin timeout from 30 to 60 seconds, maybe that helps
And I'll have someone check the frontend code to make sure it's impossible to submit the same payment method object
jovial herald
Seems like a good plan
robust raft
It's a pretty rare error (considering the number of payments) so I can afford debugging it like that
jovial herald
It also has no real impact... right?
robust raft
yeah they just get an error and try again
jovial herald
Or so it seems
robust raft
I would assume
And those on really poor connections are probably aware
like if you're in the subway trying to buy a ticket
you expect it to maybe go wrong
and you'd retry
jovial herald
Yeah. Client might honestly see absolutely nothing related
Alright I'm outta here.
👋
robust raft
Thanks for your help