#Urgent: Two production services down due to database connection issue

106 messages · Page 1 of 1 (latest)

night dust
#

@woven imp@terse creek could I get some urgent assistance

woven impBOT
#

Project ID: N/A

night dust
#

My project service is unable to connect to SurrealDB, it appears to be a wss (websockets secure) issue with railway

#

Another person in SurrealDB discord server is having this issue as well. I have made no changes to my configuration and my database is down for two different services

shut lanternBOT
#

New reply sent from Help Station thread:

This issue occurs also in Surrealist. When using WSS , the connection breaks but HTTPS works fine. This was not previously the case, both worked correctly before.

You're seeing this because this thread has been automatically linked to the Help Station thread.

past olive
night dust
#

https works fine

#

This was previously functional with no issues. Another user is experiencing the same issue with railway and running on a different database version

past olive
#

Are you available to check if it's working in a few minutes ?

night dust
#

Yes

past olive
#

ok

#

try now

night dust
#

Working! @past olive

#

What was the issue

past olive
#

I will have to troubleshoot it. We are moving everyone over to a new edge network and I just put your application back on the legacy network.

night dust
#

I still have one service down, oddly

#

(same project)

past olive
#

what's the issue exactly ?

night dust
#

Exact same connection error, I'll give it a minute to see if it might be some sort of pre-existing connection issue. I'll trigger a re-deploy for the project now

past olive
#

I've just changed DNS settings so existing connections will continue to talk to the previous network

#

we have migrated 400k applications to our new network and this is the first issue I've seen :/

night dust
#

Websockets can be a bitch

#

I suspect many other SurrealDB users are experiencing the same issue

past olive
#

I will try to repro right now

night dust
#

Okay, so my other service is online now after a redeploy

night dust
past olive
#

the migration process has been ongoing for the past 2 weeks.

#

slowly moving apps over

past olive
night dust
#

Do you know roughly when our project might've been migrated?

#

Just would like to know how much downtime there might've been.

it must've been a few days since I started having this issue on one service before the other one started showing issues (probably the existing connection)

past olive
#

I tried checking but I don't have enough data to know for sure.

#

I might be able to check 1 more place

night dust
past olive
#

the IPs resolve to different regional load balancers

#

so geodns is picking the closest LB

night dust
#

ok thanks just checking

past olive
#

I can confirm the date

#

just will take time as I grep the log file

night dust
#

If you'd like to reproduce my setup,

I was running

surrealdb/surrealdb:v1.4.2 as the docker image

/surreal start --auth --allow-guests --allow-funcs --deny-funcs "http" --allow-scripting --deny-net --no-banner as the custom start command

Region: US West (Oregon)

Runtime: Legacy but also tried V2 with same issue

#

The other person having the issue was on 1.5.2

To reiterate, this issue only occurred with wss connection within Surrealist GUI

Not with https

image.png
past olive
#

ok, your app was migrated excatly at 2024-08-26T02:32:51Z

#

utc

#

so about 48 hours

night dust
#

Yeah that sounds about right

past olive
#

close to 24 hours

#

I know websockets work

#

I'm sus about wss:// requiring TLS but our proxy doesn't do TLS to your app.

#

https:// worked though so maybe not that

past olive
night dust
#

@past olive are you saying data isn't encrypted during transit?

past olive
#

it is encrypted all the way to your app, just not using HTTPS.

internet <-- HTTPS --> Edge Network < -- WireGuard encrypted tunnel -- > Your App

#

did you configure tls certs on your app ?

night dust
#

hmm you might be right,

surreal start has the following option

HTTP server:
      --web-crt <WEB_CRT>
          Path to the certificate file for encrypted client connections
          
          [env: SURREAL_WEB_CRT=]

      --web-key <WEB_KEY>
          Path to the private key file for encrypted client connections
          
          [env: SURREAL_WEB_KEY=]
past olive
#

if so the edge network could be talking over HTTPS (and also wireguard)

#

most people don't have TLS terminating at their application

#

most is read as, I've never seen it

night dust
#

Well, I think TLS is enabled for https most definitely, but what about Websockets? How does websockets interact with proxies?

image.png
past olive
#

TLS would work for ws the same since that happens before the application (websocket/http) right after tcp

night dust
#

Anyhow, thank you for resolving the issue as fast as you could. I would investigate this as I imagine other users besides the one I spoke with is having this issue

past olive
#

yup, i'm working to reproduce right now

#

sorry you had this issue.

night dust
#

No problem - these things happen, fortunately I'm not running a critical service

#

When you are able, I would love an update on what occurred, whether you were able to reproduce/fix the issue with the new network

past olive
night dust
#

You can probably use the web GUI if its easier

#

otherwise downloading the application

past olive
#

oh I see. the service is just the DB and the screenshot is a frontend to it

night dust
#

That's right yeah

#

Let me know if you have any questions regarding connection setup, or whether you need me to test with my project once a fix is deployed

#

I used Root authentication method via the credentials generated by railway, any namespace and database should work for the connection

The URL should just be the host, and wss for the protocol

shut lanternBOT
#

New reply sent from Help Station thread:

hello, i have the same issue in my project can you please help me

You're seeing this because this thread has been automatically linked to the Help Station thread.

shut lanternBOT
#

New reply sent from Help Station thread:

@bachargit, what is your domain and I can take a look.

You're seeing this because this thread has been automatically linked to the Help Station thread.

shut lanternBOT
surreal mauve
#

Hello.

I cannot connect to my deployed surrealdb instance on railway: wss://surrealdb-production-c358.up.railway.app/rpc, even though it was working a couple of days ago.

Here is my projectID: 320c6ccd-0556-4d5f-9f87-05d51b1043ed

past olive
#

do you still have the issue if you use https:// protocol in the url instead of wss:// ?

#

I am aware of this surrealdb issue and wss://. Working to reproduce and fix.

surreal mauve
#

i see no issues with the https:// protocol, only the wss://

feral trail
#

Hello,
I've been experiencing the same issue. Any way to resolve this?
My ProjectID: 09be9421-29f2-4e16-b051-751c7fb2b16a

past olive
#

Make surrealdb client use https:// protocol instead of wss://. The connection will upgrade to websocket anyways.

feral trail
#

how was this resolved for Albert and bachargit? is there something else I can try?

past olive
#

The issue is with the edge network being used by the application. We are moving on from our old proxy but I can put your app on it while we try to resolve this issue for surrealdb

feral trail
past olive
#

wss:// should work for you in a few seconds.

feral trail
#

yeah it works, thank you for the help

surreal mauve
#

Hey @past olive, can you do that for my app as well?

I started needing the wss:// protocol now

surreal mauve
subtle dove
#

please do not ping team members, we are currently on a day off

subtle dove
subtle dove
#

Hey @night dust @surreal mauve @feral trail - We have moved all domains over to the new edge proxy, including the domains attached to the surrealdb databases.

night dust
#

awesome @subtle dove thanks for the update

subtle dove
#

sorry i should have been more clear, we have not patched away whatever surrealdb is doing incorrectly with websocket connections

night dust
#

It appears to be working fine for me @subtle dove ?

subtle dove
#

well you are on the v2 proxy, so thats awsome, are you using wss or https?

night dust
#

actually it may be https but I am not too sure

#

I guess I'll find out when I migrate my new update using wss