Private networking - webhook error | Railway | Page 1

maiden beacon Jul 3, 2023, 4:42 PM

#

I'm working on a new callback pattern for realtime GraphQL subscriptions that utilizes a webhook like pattern. I want to use the internal networking in railway to secure that callback url, but I'm getting an error:

request to http://{private}.railway.internal:6607/callback/71d75f513901349ee7dfe50054409d4b019887cdd9c6f79e41b8ee752d293ae7 failed, reason: getaddrinfo ENOTFOUND hosted-router.railway.internal

I'm currently listing my public url as: http://{private}.railway.internal:${env.PORT:-4000}/callback

Is there anything jumping out why this wouldn't be working? I already have everything working successfully when I use the public domains avaialble.

quiet spadeBOT Jul 3, 2023, 4:42 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

final vergeBOT Jul 3, 2023, 4:42 PM

#

Project ID: 38b0d466-9168-4f9f-bf7f-c09650f565e8

maiden beacon Jul 3, 2023, 4:42 PM

#

38b0d466-9168-4f9f-bf7f-c09650f565e8

lament fog Jul 3, 2023, 4:50 PM

#

the private networking interface takes about 100ms to become bound, and another 2-3 seconds for private dns resolution to come online, any internal network requets during this time will very likely fail

maiden beacon Jul 3, 2023, 4:57 PM

#

Does it mean we need ~3s on pod startup or for each of the network calls?

lament fog Jul 3, 2023, 4:58 PM

#

if they are internal network calls, yes, a 3 second grace period is needed, or auto retries is an option too

#

railway will be working to shorten or eliminate the need for this pause

maiden beacon Jul 3, 2023, 6:07 PM

#

Okay I've tried adding some stuff here but I'm still getting errors.

I added some code to wait for the service to be ready:

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://{private}.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    }
  }, 1000);
});

I hardcoded the port that my other service is in and I'm seeing this error in my other service (project id: 19b30993-43c6-4719-b45a-81d55968ec20):

{"cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"hosted-router.railway.internal"}}
```

#

I figured I should get any positive http response (even 404 is fine) and then it's good to continue my request chain

lament fog Jul 3, 2023, 6:08 PM

#

so uh whats with {private}

#

is that just you hiding information?

maiden beacon Jul 3, 2023, 6:08 PM

#

I guess?

#

I mean yes, is it hiding anythhing really?

lament fog Jul 3, 2023, 6:09 PM

#

wdym you guess? lol
is that what your code actually has written in it?

maiden beacon Jul 3, 2023, 6:09 PM

#

I have this http://hosted-router.railway.internal:7718

lament fog Jul 3, 2023, 6:10 PM

#

okay so what was the point of {private}

maiden beacon Jul 3, 2023, 6:10 PM

#

So I didn't put hosted-router in discord, but I'll just change that later

lament fog Jul 3, 2023, 6:11 PM

#

are you using nixpacks

maiden beacon Jul 3, 2023, 6:12 PM

#

Currently only the defaults when I deploy, I see Nixpacks in my settings

#

actually the service that I'm exposing the internal domain on is docker

#

the service it's being reached from is using Nixpacks

lament fog Jul 3, 2023, 6:13 PM

#

everything is docker

#

but the image is built with nixpacks

#

do you have internal networking enabled

maiden beacon Jul 3, 2023, 6:14 PM

#

Screenshot_2023-07-03_at_11.13.51_AM.png

#

that right?

lament fog Jul 3, 2023, 6:16 PM

#

i assume you are just seeing that json error log indefinably?

maiden beacon Jul 3, 2023, 6:16 PM

#

yup, still going in the logs

lament fog Jul 3, 2023, 6:16 PM

#

and that code is from a nodejs server

maiden beacon Jul 3, 2023, 6:16 PM

#

yup

lament fog Jul 3, 2023, 6:17 PM

#

well im stumped

maiden beacon Jul 3, 2023, 6:17 PM

#

This stuff is hard

lament fog Jul 3, 2023, 6:17 PM

#

its always just worked for me lol

maiden beacon Jul 3, 2023, 6:17 PM

#

I know that feeling so well 🤣

lament fog Jul 3, 2023, 6:19 PM

#

wanna try awaiting a setTimeout so that it acts as a blocking pause before any network requests are done?

maiden beacon Jul 3, 2023, 6:19 PM

#

yeah like 3s?

lament fog Jul 3, 2023, 6:19 PM

#

yes, max time ive seen is 3s

#

average time is 1.5s but 3s is a good safe time

maiden beacon Jul 3, 2023, 6:22 PM

#

no luck

Screenshot_2023-07-03_at_11.21.57_AM.png

lament fog Jul 3, 2023, 6:23 PM

#

i dont see a pause in the timestamps?

#

also, node 20??

maiden beacon Jul 3, 2023, 6:24 PM

#

lament fog also, node 20??

🤷‍♂️ I just say above 18

  "engines": {
    "node": ">=18.0"
  },

maiden beacon Jul 3, 2023, 6:25 PM

#

lament fog i dont see a pause in the timestamps?

Sorry I misunderstood and put the sleep before the timestamps, not in them. I'll change that

lament fog Jul 3, 2023, 6:25 PM

#

where did you put the pause? sleep 3 && npm run start

maiden beacon Jul 3, 2023, 6:26 PM

#

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
await sleep(3000);

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://hosted-router.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    } finally {
      await sleep(3000);
    }
  }, 1000);
});

👆 that's just in my code, it's a web server that is already started up before the request issue

lament fog Jul 3, 2023, 6:27 PM

#

and same issue?

#

as a last ditch attempt, add ENABLE_ALPINE_PRIVATE_NETWORKING set to true in your service variables

maiden beacon Jul 3, 2023, 6:34 PM

#

Okay there was some success in that, I get a response error:

{
    "data": null,
    "errors": [
        {
            "message": "HTTP fetch failed from 'playback': HTTP fetch failed from 'playback': 503: Service Unavailable",
            "path": [],
            "extensions": {
                "code": "SUBREQUEST_HTTP_ERROR",
                "service": "playback",
                "reason": "HTTP fetch failed from 'playback': 503: Service Unavailable"
            }
        }
    ]
}

#

This is now something else I need to look into that might not be railway, since this is the Apollo Router receiving a response from a downstream service and that is the actual gql response coming back

lament fog Jul 3, 2023, 6:35 PM

#

one more thing

#

in the hosted-router.railway.internal service variables do you have a fixed PORT set?

maiden beacon Jul 3, 2023, 6:36 PM

#

I do not, I just have hardcoded in the subgraph I'm testing that port hardcoded

#

everything else listens on the $PORT environment variable

#

(which I don't set)

#

(but I might need to)

lament fog Jul 3, 2023, 6:36 PM

#

since railway assigns it randomly, that means it makes it hard to do an internal request to a service on a random PORT

#

so yeah might want to set that to a fixed value in the service variables

maiden beacon Jul 3, 2023, 6:37 PM

#

ya good call, I will do that for sure

#

I at least have something more to investigate and I'll report back my findings end of day. I'm hoping to trim that code down into a small block I can reference and ideally delete in the future when y'all get to that point ❤️

#Private networking - webhook error