#ihobbes
vivid condor
Can you describe the behavior in more detail? Do you have a screenrecording/screenshots/logs/etc?
frail orchid
Hello! Here's what chrome dev console is spitting back when we try to run a transaction.
vivid condor
How are you integrating with us?
Payment Element, etc?
What function call causes that error to happen?
frail orchid
I'm not sure of the function call we are trying to narrow it down now. The call hangs whether we are trying to process a payment without creating a customer and saving a card and also when we try creating a customer and saving a card--neither is successfull and nothing registers on Stripe's side.
We are using StripeJS
vivid condor
Recommend adding debug statements to narrow down the exact function call this is occurring at
Also recommend spinning up a test site you can share with us so we can reproduce the issue
frail orchid
This is an embedded web app on a kiosk that accepts payments, so I don't believe we can spin up an instance for review. Only try to diagnose at the endpoint.
vivid condor
Hm ok
That makes it hard for us to reproduce and help
Recommend adding those console logs to get further in debugging
frail orchid
Working with our dev now to see about doing that. From your end, were any updates pushed yesterday that could be causing this issue? Again, the app has worked flawlessly for years and this is the first issue we've had out of the integration.
vivid condor
We constantly make changes, but nothing is jumping out at me that would cause this
Would need a little more info from your dev
Want to know exactly which line it's hanging on and if there are any other warnings/error messages anywhere
frail orchid
Thank you--working on this now
radiant harnessBOT
frail orchid
Hi Rubeus..any new info from your side on this? We are still tracking down error on this side. As I explained before, our web app hasn't changed in years and just stopped interfacing with Stripe yesterday evening.
uncut moat
No, I'm just getting caught up. Does your app do anything out of the ordinary with iframes or anything like that?
frail orchid
I do not believe StripeJS is loaded into an iframe but it has been years since I looked at it. Am I able to share our Stripe implementation code with you?
uncut moat
But you don't know what part of this code is throwing the error?
frail orchid
We are trying to track it down now. It's very old code and we are having trouble spinning back up to figure out what's going on since it's been working properly for 5 years or so.
uncut moat
Are you loading Stripe.js from js.stripe.com?
uncut moat
Hm. Next step is to figure out which specific part of your code triggers the error.
frail orchid
We are trying to add debug lines to the purchase function now.
uncut moat
Any luck?
frail orchid
you can ask if anything has changed with generating tokens. Looking at Stripe, I most recently see errors for /v1/tokens:
parameter_invalid_empty - referrer
You passed an empty string for 'referrer'. We assume empty values are an attempt to unset a parameter; however 'referrer' cannot be unset. You should remove 'referrer' from your request or supply a non-empty value.
5:02
Causes a 400 - api version 2015-01-26
uncut moat
Can you give me an example request ID? Here's how you can find a request ID: https://support.stripe.com/questions/finding-the-id-for-an-api-request
frail orchid
Users with a saved card are able to purchase fine. Users trying to use or save a new card cannot and stripe hangs. Sure one sec.
The code is waiting for that call to return a token, but it never returns so it just hangs until it times out.
I'm not seeing any of the failed calls in the Stripe dashboard logs.
uncut moat
I'm not sure if this issue is related to the error about the frame mentioned earlier. Sounds like it could be two separate things, but will know more once I see an example request.
You passed an empty string for 'referrer'.
Where are you seeing this?
frail orchid
IN the stripe logs. Here's an ID:
req_OVITVK8SYYUu0S
could be unrelated
THose errors began popping up the aafternoon of the 27th--hadn't seen one before then.
uncut moat
Looking, hang on...
Is that when your issue started?
Or did you see the problem pop up before or after that?
The exact time and date (UTC) the issue started would be helpful.
frail orchid
We aren't sure. We only start getting busy on Friday's so it's possible the issue started a few days ago and we are just now catching it. Our first known issue is yesterdya evening.
uncut moat
What does "yesterday evening" equate to in UTC? I don't know where you are or when evening happens for you.
uncut moat
Thanks. I'm investigating further. In the meantime, I am seeing a lot of successful Token creation requests, like this one: https://dashboard.stripe.com/logs/req_JpWwKVZGLc4foU
Are those payments going through just fine?
radiant harnessBOT
frail orchid
The only payments going through right now are for users who already had a card saved on their account. If they try to enter a new card to pay Stripe response hangs.
uncut moat
So these successful Token creation requests (of which there are many) are still resulting in failed payments?
frail orchid
That token request you linked to looks like the last one that went through before 3:30 EST that day on the 27th when we got the first referrer error 400
uncut moat
What about this one? https://dashboard.stripe.com/logs/req_6ymjWZqHzqvbhG
That's more recent.
And comes after the latest batch of errors.
Looks like that one resulted in a successful payment, right?
Ah, okay, so I think it's two different issues.
The /v1/token request issues are one problem, which seems intermittent. The other problem, which is the client-side error you're seeing, seems to prevent /v1/token requests from happening in the first place.
So the /v1/token error issue is actually a known problem we already released a fix for, so that shouldn't happen again. That brings us back to the client-side error which seems to be preventing /v1/token requests in the first place.
Which we need more information about, like which specific part of your code is throwing that error?
frail orchid
Understood. Trying to add more debug code now to see if we can get an error back from the Stripe call.
So because our code hasn't changed, does that mean there's something that's changed with stripejs in the last 2 days which changed and is causing this issue? We'll keep looking
uncut moat
We modify Stripe.js all the time, often several times per day. Those changes shouldn't cause an issue like this though, unless your integration changed or is doing something quite abnormal.
frail orchid
That's the strange part. Like I mentioned, the codebase for this kiosk app has been dormant and working well for 5 years or so.
uncut moat
We think we may have found something on our end. How long will you be around?
frail orchid
Until resolved.
Lost a big chunk of revenue today unfortunately and weekend is busiest time so I'll be around as long as it takes!
twilit garden
I'll keep you posted as I have updates for you
frail orchid
feral wolf
@frail orchid any chance you can give a bit more details about your set up? Like you mentioned a "kiosk" what is the set up like OS/browser/concept, etc.
frail orchid
Sure. It's an Electron web app running Chromium on Linux.
We are using StripeJS for payments.
feral wolf
Can you give the exact versions?
frail orchid
Sure thing--hold tight. It's a pretty old app. We have a newer app as well that uses a much newer Stripe Elements implementation but have not been able to upgrade some older locations yet and theya re still using StripeJS.
I show two Stripe API versions in our dashboard and they both get similar numbers of calls though I'm not sure why all calls aren't using the default.
feral wolf
I don't want API versions sorry
I want the operating system and browser exact versions
feral wolf
it's a system/compat issue between our JS code and you runtime environents
frail orchid
And yes I was trying to get everything to you here. Been a long day! One moment.
feral wolf
no worries
frail orchid
Looks like Chromium 83
Example system:
Linux 5.10.17-v7+ #1421 SMP Thu May 27 13:59:01 BST 2021 armv7l GNU/Linux
feral wolf
thanks
frail orchid
If it matters, Electron version is 9.1.1
feral wolf
thanks, this is not a common environment to load Stripe.js in. We're still investigating
frail orchid
Understood--thank you very much!
frail orchid
We cannot find anything in the documentation about setting the value of "referrer" when passing the transaction info to Stripe. This new 400 error in the last two days could be a red herring, but it's suspicious that we've never seen that error before in our log history and started getting many of them in the last two days. The hang is definitely happening on createToken either way.
feral wolf
you don't have to set anything, it's supposed to be all automatic, we just had a bug for some reason with your set up
frail orchid
Ah!
feral wolf
right now we haven't been able to reproduce in any way but we're rolling back a change to fix it while we figure out the real root cause
frail orchid
Understood. Will stand by for testing. Thanks for all your help! We haven't had to hop in the Discord before but this is incredibly useful. Otherwise we'd be dead in the water over here.
feral wolf
Glad we could help 🙂
uncut moat
@frail orchid 👋 Hello again! Long shot, but do you know if anything in your app or your Electron config might be modifying window.origin, especially in iframes?
frail orchid
I'll ask our dev. Hold on!
We have a chat support page that loads an iframe, and some transition bumper pages that load iframes--but the buy time page where we load Stripe.js is not one of them. Still poking around to confirm we didn't miss anything.
uncut moat
So Stripe.js adds iframes to the page. Your JavaScript shouldn't be able to modify the content of those frames, but because this is an Electron app the app itself might be changing things in those frames.
From what we can tell the root cause of this is that deep inside Stripe.js we're checking window.origin inside one of our own iframes and not getting back what we expect, so we're wondering if Electron itself might be modifying it inside our frames somehow.
frail orchid
Dev says Setting an http origin/referrer in electron isn't possible as far as he is aware. I'll shoot him back what you said.
uncut moat
Yeah, like I said, a long shot, but figured it couldn't hurt to ask.
@frail orchid Okay, we just deployed a change that may fix this. Can you see if there's any change on your end?
uncut moat
You may need to reload/restart the app depending on how caching works inside Electron, so you pick up the new version of Stripe.js.
If you're still seeing an issue knowing if the underlying error in the console is the same or different would be helpful.
frail orchid
Rebooting equipment and testing now
Problem persists.
Dev looking now.
Debug info to follow. The above error may mean nothing.
New error in console though:
feral wolf
thanks we're looking into it
Do you or your developer know of anyeasy to reproduce/test your code outside of your kiosk?
frail orchid
It doesn't look like the hang failure results in a referrer failure in the Stripe log everytime though. There's not a referrer failure since 6:27 EST but my onsite people have tested transactions recently that hang.
uncut moat
Can you have someone type window.origin in the console there, hit enter, then let us know what the value is?
We determined the referrer failure is mostly unrelated and not directly associated with these errors.
frail orchid
window.origin returns 'undefined'
uncut moat
What about document.location.href?
frail orchid
Testing outside of the kiosk hardware has proven difficult since the electron app is an old version, getting all the dependencies that it uses and work on that specific Linux operating system breaks lots of things if we try to run the app in another environment.
feral wolf
gotcha
frail orchid
Checking on document.location.href
file:///opt/kiosk-user-app/app/index.html#/buytime
uncut moat
Okay, one more thing that would help us out. In the dev tools can you you change from top to the __privateStripeController... frame, then check window.origin and document.location.href again?
frail orchid
On it
Only seeing __privateStripeMetricsController
On page load. Should we try to process card payment?
uncut moat
Yeah, sorry, I should have specified that you need to do this on the page where the Card Element is loaded.
frail orchid
That's the only thing showing on the page with the stripe cc entry field.
For metrics controller window.origin is undefined and document.location.href =
uncut moat
That helps, thank you!
frail orchid
Yes
uncut moat
Okay, yep, that's the issue. We're working on a fix now.
frail orchid
uncut moat
If possible can you try window.location.href in there too?
frail orchid
Sure one moment.
uncut moat
We think that one should come out to the js.stripe.com URL.
uncut moat
Perfect, thank you!
frail orchid
Standing by if you need anything else. We owe you all big time.
uncut moat
I think that's the last thing we needed for now.
Okay, we're roughly just shy of two hours until the fix goes out and is live. We have high confidence this fix will do the trick, but you can never tell until you actually see it working. 🙂
frail orchid
Oh that is amazing. I'll set an alarm and test once it goes live.
uncut moat
I'll flag if it goes out sooner than expected.
I'm going to step away for a little while, but I'll return later as we approach the expected deployment time.
frail orchid
Sounds great! We manage eSports and Social Gaming facilities for Service Members around the country so they have something fun to do when they aren't training and this would have kept us out of commission over the weekend. They'll certainly appreciate not having to sit in the barracks! Is there a survey or something I can fill out for you and the team there or an email address I can send something?
uncut moat
This thread will remain open in the meantime.
We log these threads, and I can surface whatever feedback you have, so just saying whatever you want here works. 🙂
frail orchid
Awesome. See you closer to go-live!
uncut moat
@frail orchid Okay, deploy is ready, can you try again?
frail orchid
Hello! Yes. Trying now. Will take a few minutes to reboot.
Seems to be working!
Trying a few more tests.
frail orchid
Indeed it does appear to be working!
We'll run it through it's paces in the morning with users but in my small testing transactions are working just fine again.
uncut moat
That's great news! Sorry for the trouble!
frail orchid
I'm just so thankful you all were able to diagnose the issue AND patch it in just a handful of hours. I honestly can't believe it. We've been a Stripe customer for 12 years and this is the first time we've needed to use technical support, and you and your team have provided top notch service. You could have easily written off our issue as a fringe use case, but you took ownership of it instead and solved our problem on the spot. Thank you very much! Please forward this feedback along!
I'll reach out in the morning once users start filing in if we see anything else strange in the Stripe logs but I think we are good to go.
uncut moat
Will do, thank you for your patience and helping us figure out the core issue!