#crackingtreys
acoustic shell
Do you have a Dashboard URL or a Webhook Endpoint ID I can look at to see the errors in your logs?
ancient ledge
This is the endpoint we are using https://api.dev.swiftraize.com/v1/webhooks-handler/wh_2XgBzTWY5Zeb2uqVWiCesFquqCs
If you see here, some succeed and some fail and we get the Failed to connect to remote host. If I retry sometimes it works and sometimes it doesn't
acoustic shell
Ah, so that's definitely on your server's side. My guess is that your webhook handler was updated and isn't ingesting those webhooks or sending back an HTTP 200. Are you able to do a dump of your network logs to troubleshoot further?
Unfortunately we're a bit blind when it comes to this, since the server configuration is the problem
ancient ledge
Atcually, it does not reach the load balancer that sits in front of our backend. We checked the logs for load balancer. For some webhooks, Stripe can reach and for other it does not.
The error from above, "Failed to connect to remote host" indicate that it could not resolve the host at all
We check DNS Nameserver
acoustic shell
If you click resend does it still not connect?
ancient ledge
as well and it seems everything propogated
ancient ledge
If you click resend does it still not connect?
sometimes it works and sometimes it does not
finally we did load testing on the webhook endpoint and it was able to handle high throughput
acoustic shell
Can you copy/paste that Event ID here so I have an example of one that failed? The ID that looks like evt_abc123
ancient ledge
There you go
evt_3O8TtECfaW1k9MRD1VLgCTq4
It's not a universal error, our load balancer is not receiving any calls at all
acoustic shell
Hmmm, I'm not sure. Let me grab someone with more network experience and see if they can weigh in
acoustic shell
As far as we can tell, there's nothing we can find that indicates this is on our end. It has to be on your server's side. Did you do a network log dump of all requests in the time range of these failures? If you're doing load balancing you might have to do multiple different dumps from different servers.
green robinBOT
ancient ledge
The problem here is that for those failed webhook it does not even reach the load balancer
to even get the logs
This an example of logs from load balacner
https 2023-11-03T21:23:03.598298Z app/DevApiSwift/5cb37e0378c07ea2 54.187.205.235:20405 10.0.6.155:8080 0.000 0.036 0.000 202 202 1463 92 "POST https://api.dev.swiftraize.com:443/v1/webhooks-handler/wh_2XgBzTWY5Zeb2uqVWiCesFquqCs HTTP/1.1" "Stripe/1.0 (+https://stripe.com/docs/webhooks)" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:ca-central-1:772918164852:targetgroup/ECSCre-appli-QR4NN0BZYISO/79b2e87c4426a421 "Root=1-654564b7-5dc5ef3b2ce7bb87596e44ae" "api.dev.swiftraize.com" "arn:aws:acm:ca-central-1:772918164852:certificate/2d10e303-fc8e-4eff-a301-8d2995deee93" 0 2023-11-03T21:23:03.501000Z "forward" "-" "-" "10.0.6.155:8080" "202" "-" "-"
swift kayak
HI 👋
Stepping in as @acoustic shell needs to go
ancient ledge
Hi!
swift kayak
Looking at the event you shared, it looks like your host becomes unavailable sometimes
These events appear to be making it to the Stripe edge network. Apparently this is often the result of a DNS issue mapping to the URL you provided.
ancient ledge
so the assumption here is that DNS NS has issues
swift kayak
That is a frequest culprit in these situations
ancient ledge
which it may explains that that it resolves on your stripe randomly and not
is it possible
to "refresh"
swift kayak
These aren't servers we own or control
ancient ledge
Hmm. Ok in your experience, they would probably have some TTL
swift kayak
You update the DNS records for your servers and these propagate across the internet
Yes in my experience there can be a TTL of 1 minute to multiple hours