justinadkins | Stripe Developers | Page 1

onyx pondBOT Sep 28, 2022, 6:18 PM

#

eternal python Sep 28, 2022, 6:19 PM

#

Are these generating logs in your Stripe Dashboard? Where are the 500s coming from (e.g. Stripe or your internal server)?

plucky pulsar Sep 28, 2022, 6:19 PM

#

We're seeing it on our network monitoring dashboards

#

DataDog / Embrace etc

eternal python Sep 28, 2022, 6:20 PM

#

Okay. Are they generating log entries here though? https://dashboard.stripe.com/logs

Stripe Login | Sign in to the Stripe Dashboard

Sign in to the Stripe Dashboard to manage business payments and operations in your account. Manage payments and refunds, respond to disputes and more.

plucky pulsar Sep 28, 2022, 6:22 PM

#

No, but does that dashboard show BBPOS / reader endpoints? (e.g.: firmware update checks etc)

#

That log list looks like it's mostly geared towards payment / charges etc

#

So no, they don't appear there @eternal python

eternal python Sep 28, 2022, 6:24 PM

#

Okay, just checking to see if this was an internal happening (e.g. 500s from the Stripe side). How many readers are failing to connect?

plucky pulsar Sep 28, 2022, 6:26 PM

#

Not a lot because we're slowly rolling out to M2 devices (this appears to only happen for M2 readers) -- past 3 hours we've seen 25 HTTP 500 errors

#

That said we have ~100 rolled out and concern is if they retry connection / have an unexpected disconnection customer would be stuck

eternal python Sep 28, 2022, 6:31 PM

#

Do you have the serial number of one of the affected devices so I can try and dig a bit more? What version are they on?

plucky pulsar Sep 28, 2022, 6:32 PM

#

Couple serials:
133C106006
142A100159
147A101531

#

Will double check SDK version

#

2.12

dense crypt Sep 28, 2022, 6:34 PM

#

From what we are seeing, it is affecting all M2 readers that are trying to connect in our fleet

eternal python Sep 28, 2022, 6:42 PM

#

Digging more now and will circle back shortly

#

Okay, it looks like this is on our end. Your account admin should have received an email, but here are the details:

From 16:31 - 17:30 UTC on September 28th, we observed increased failure rates for firmware updates on Terminal devices, which in turn would prevent usage of those devices. We have made a configuration change on our end to allow devices to connect and collect payments, and skip firmware updates. We will re-enable firmware updates as soon as possible.

plucky pulsar Sep 28, 2022, 6:52 PM

#

Could we get confirmation for when that configuration change is rolled out?

#

(Our account admin is also out of office on October 11th)

eternal python Sep 28, 2022, 6:55 PM

#

Yup! Happy to update you here, though I don't yet have an ETA on resolution.

@somber holly is taking over, but they'll be monitoring internal comms to make sure you're notified when the resolution is rolled out.

plucky pulsar Sep 28, 2022, 6:55 PM

#

Okay, thank you @eternal python

violet warren Sep 28, 2022, 7:04 PM

#

Shouldn't https://status.stripe.com/ reflect this issue?

Stripe system status

Check Stripe system status and see recent API status updates.

plucky pulsar Sep 28, 2022, 7:17 PM

#

Regarding the account admin email, we checked our account and never got that email

#

Not sure if it meant our Stripe account admin or an email address we've provided, but if the latter then didn't receive it

dense crypt Sep 28, 2022, 7:19 PM

#

We haven't seen a 500 from api.emms.bbpos.com/tms2/sdk-service/terminal-version-getter in the past 40 minutes but are still receiving 500 from api.emms.bbpos.com/tms2/sdk-service/login

#

Some M2 Serials that are experiencing this:
STRM26139021898
STRM26138002894
STRM26138002896

somber holly Sep 28, 2022, 7:26 PM

#

Checking with the team on this

plucky pulsar Sep 28, 2022, 7:28 PM

#

Oh I gave the wrong serials, hah my bad, yeah those are the correct reader serials

somber holly Sep 28, 2022, 7:50 PM

#

So yeah this is part of the same incident

#

Efforts to resolve it are still ongoing

#

Hello, it looks like we just made a change that should help mitigate this. If you try again with your devices are you still seeing this issue?

plucky pulsar Sep 28, 2022, 8:10 PM

#

It does look like we haven't seen the issue arising now for about 20 minutes according to our logs

dense crypt Sep 28, 2022, 8:11 PM

#

We are seeing devices recover, any insight on the change? Was this a data problem?

#

It seemed to affect only a percentage of our readers

upper fog Sep 28, 2022, 8:14 PM

#

Not a data problem from your side. The issue was related to firmware updates on these terminals, we temporarily disabled firmware updates here, are working to resolve the underlying issue, and will reenable updates once we have addressed it

#

Great to hear your devices are recovering!

dense crypt Sep 28, 2022, 8:18 PM

#

So if an M2 reader did not need a firmware update, would it have been unaffected?

upper fog Sep 28, 2022, 9:00 PM

#

@dense crypt Hey huge apologies for the delay in reply, I missed your message. As far as I can tell this issue largely affected devices that needed an update but could affect transactions on other devices.

#

For the three serials IDs you provided, they all look to be issues from the firmware update issue

dense crypt Sep 28, 2022, 9:04 PM

#

The devices you were noticing having issues during a transaction, would they disconnect? I think the majority of ours were the firmware update issue but just collecting information now in case we see different issues and to document in our post mortem

#justinadkins