#Nginx Reverse Proxy with Private Networking

207 messages · Page 1 of 1 (latest)

cold cape
#

I am trying to setup an nginx server to reverse proxy two other services backend & frontend, and expose it publicly via this nginx. I set port of both backend & frontend at 80, and both are not public, but I want to merge them such that / serves the frontend service and /api/ serves the backend service. But when I setup the nginx conf so and used the private networking address, it crashes and doesnt work. Please help

steep tulipBOT
#

Project ID: 9960c53b-53ce-401e-8708-8404250e7ad9

winged stagBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

cold cape
#

9960c53b-53ce-401e-8708-8404250e7ad9

#

    listen 80;
    server_name localhost;

    server_tokens off;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;

    location / {
        proxy_pass http://zivah.railway.internal;
    }
    
    location /api {
        proxy_pass http://zivani.railway.internal;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;

        proxy_connect_timeout       70s;
        proxy_send_timeout          86400;
        proxy_read_timeout          86400;
        send_timeout                86400;
    }

}

My nginx config

light inlet
#

try adding resolver fd12::10

#

in the server block

#

that's the dns resolver the containers use to resolve internal addresses, it seems nginx needs to be explicitly told that

#

untested though

cold cape
light inlet
#

also, it's preferable if you used a non privileged port like 8080 instead of 80

cold cape
light inlet
#

resolver doesn't accept ipv6 addresses?

cold cape
light inlet
#

maybe this is the syntax you need to use? I'm not too familiar with nginx myself
resolver [fd12::10]

cold cape
#

I am using nginx alpine byw

light inlet
#

shouldn't be a problem, I use alpine or distroless in all my apps

cold cape
#

went back to the same old error now

light inlet
cold cape
#

yes

light inlet
#

interesting, well I'll do some messing around and get back to you!

cold cape
#

let me know if you need anymore details,

light inlet
#

will do

cold cape
#

I tried with Caddy to see if it has to do with nginx, but same result

#

but caddy instead throws a 502 directly and doesnt crash

light inlet
#

just let me refine the config a bit and ill send it over soon

cold cape
light inlet
#

frontend: https://proxy-production-9e87.up.railway.app/
backend: https://proxy-production-9e87.up.railway.app/api/
nginx.conf:

worker_processes 5;

worker_rlimit_nofile 8192;

events {
    worker_connections 4096;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';
    access_log /dev/stdout;
    error_log /dev/stdout;
    sendfile on;
    keepalive_timeout 65;
    tcp_nopush on;
    server_names_hash_bucket_size 128;
    server_tokens off;

    resolver [fd12::10] valid=10s;

    server {
        listen 3000;
        listen [::]:3000;
        server_name localhost;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;

        location / {
            set $fontend frontend.railway.internal;
            proxy_pass http://$fontend:3000;
        }

        location /api {
            return 302 $http_x_forwarded_proto://$host/api/;
        }

        location ^~ /api/ {
            set $backend backend.railway.internal;
            proxy_pass http://$backend:3000;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;

            proxy_connect_timeout       70s;
            proxy_send_timeout          86400;
            proxy_read_timeout          86400;
            send_timeout                86400;

            rewrite /api(.*) $1 break;
        }
    }
}
#

i haven't tested this super extensively, but it does work

cold cape
#

let me quickly plguin and check for mycase

light inlet
#

i think i might make a template out of this, but use caddy instead, that nginx config is kinda bulky for what it does

#

since ive seen a good few request from people who have seprate frontend and backend services but want to serve them both from the same domain

cold cape
#

I guess a little problem?

light inlet
#

show me your dockerfile?

cold cape
cold cape
light inlet
#

even though you have an nginx proxy, would you be interested in the caddy version of this?

cold cape
light inlet
# cold cape

this is my dockerfile

FROM nginx:1.24.0-alpine-slim

COPY nginx.conf /etc/nginx/nginx.conf
cold cape
light inlet
#

yeah i agree, much more info about nginx than caddy, your choice makes complete sense

cold cape
light inlet
#

ive just used that dockerfile for anything nginx related ive ever done

cold cape
#

hmm pretty clean

#

no error now

#

but

#

same for frontend too

light inlet
cold cape
light inlet
#

yeah is that okay? that has nothing to do with the proxy

#

also the config i gave you sets nginx to listen on port 3000

cold cape
#

ahh, wait, it should be 80 / 443 right?

light inlet
#

no

#

please keep it 3000

cold cape
cold cape
light inlet
#

no

#

just set PORT = 3000 in the service variables

cold cape
#

yea gotcha

#

now we have an nginx page and error

#

so someting to do with my config

light inlet
#

your backend is returning 503 though

#

you cant expect your proxy to work, if your backend alone isnt working

cold cape
#

it works with its own public domain

#

oh you mean the error

light inlet
#

you cant expect your proxy to work, if your backend alone isnt working

cold cape
#

ok i need to do the migration and check then

#

but it should show this page instead right?

#

or does it only take 200?

light inlet
#

no, nginx will not proxy a 503 through by default, same with caddys proxy

cold cape
#

hmm got it, let me migrate and and check

light inlet
#

proxy_intercept_errors off;

cold cape
light inlet
#

i think

cold cape
light inlet
#

show me the error log for that request please

cold cape
light inlet
#

makes sense

#

wild guess, is this a spring backend

cold cape
#

nope, django

#

WSGI

light inlet
#

way off

cold cape
cold cape
# cold cape

this is the command that runs backend (zivani)

light inlet
#

-b [::]:$PORT

cold cape
cold cape
light inlet
#

yeah you need to bind to all interfaces, not just all ipv4 interfaces, since internal networking is ipv6 only

cold cape
#

giving it a shot here

light inlet
#

you really should be setting that start command in a railway.json file

cold cape
#

it still went out to build the docker image 🤦‍♂️ , i was thinking to save time

light inlet
#

uh yeah I've talked with railway about that, every little thing you do rebuilds from scratch

cold cape
cold cape
light inlet
#

gotcha

cold cape
cold cape
#

building it all the time, just costs railway more $$$

cold cape
#

We actually build and keep our images in ECR, then EKS picks it up from there, I can actually use those images

light inlet
#

I assume you have your github repo linked to the service, you'd have to unlink the repo and then you'd see the button to add the image, but I don't think it would be applicable for you, it only supports public images and the docker and github image repositories

cold cape
#

ah they need to support private images,

light inlet
#

I'm sure they will, it is still beta after all

cold cape
#

GHCR might support private, need to try and see

light inlet
#

yeah but there's no way to give railway credentials to pull a private image

#

okay so what's the status with gunicorn

cold cape
cold cape
#

it just keeps loading now

#

-> gunicorn --timeout 30 --max-requests 1000 --max-requests-jitter 50 --workers 5 -b [::]:80 --log-level=error framework.wsgi is what I gave in

light inlet
cold cape
#

hmm

cold cape
#

ah 🤦‍♂️

#

let me set it to 8000 everywhere now

light inlet
#

every service gets a PORT = 8000 variable set, and every service that you can configure to listen on $PORT do so

cold cape
light inlet
#

slow your horses, one thing at a time

#

backend, now that it's listening on port 8000, show me a deploy logs screenshot please

cold cape
#

2023/07/25 18:54:13 [error] 30#30: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.2, server: localhost, request: "GET /api/healthz/ HTTP/1.1", upstream: "http://[fd12:4f6a:612e::79:437e:9411]:80/healthz/", host: "arena-nginx-production.up.railway.app"

#

:8000 went missing

#

ok my bad, the dockerfile wasnt pushed since I made it to 8000 from 80

#

2023/07/25 18:56:59 [error] 31#31: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://[fd12:4f6a:612e::ac:6c18:14fb]:3000/favicon.ico", host: "arena-nginx-production.up.railway.app", referrer: "https://arena-nginx-production.up.railway.app/api/healthz/";

#

ah well it goes to 3000

#

did I mess upsomething again?

light inlet
#

I know why

light inlet
cold cape
#
worker_processes 5;

worker_rlimit_nofile 8192;

events {
    worker_connections 4096;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';
    access_log /dev/stdout;
    error_log /dev/stdout;
    sendfile on;
    keepalive_timeout 65;
    tcp_nopush on;
    server_names_hash_bucket_size 128;
    server_tokens off;

    resolver [fd12::10] valid=10s;
    proxy_intercept_errors off;

    server {
        listen 3000;
        listen [::]:3000;
        server_name localhost;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;

        location / {
            set $fontend zivah.railway.internal:3000;
            proxy_pass http://$fontend;
        }

        location /api {
            return 302 $http_x_forwarded_proto://$host/api/;
        }

        location ^~ /api/ {
            set $backend zivani.railway.internal:8000;
            proxy_pass http://$backend;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;

            proxy_connect_timeout       70s;
            proxy_send_timeout          86400;
            proxy_read_timeout          86400;
            send_timeout                86400;

            rewrite /api(.*) $1 break;
        }
    }
}
light inlet
#
worker_processes 5;

worker_rlimit_nofile 8192;

events {
    worker_connections 4096;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';
    access_log /dev/stdout;
    error_log /dev/stdout;
    sendfile on;
    keepalive_timeout 65;
    tcp_nopush on;
    server_names_hash_bucket_size 128;
    server_tokens off;

    resolver [fd12::10] valid=10s;
    proxy_intercept_errors off;

    server {
        listen 3000;
        listen [::]:3000;
        server_name localhost;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;

        location / {
            set $fontend zivah.railway.internal:3000;
            proxy_pass http://$fontend;
        }

        location /api {
            set $backend zivani.railway.internal:8000;
            proxy_pass http://$backend;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;

            proxy_connect_timeout       70s;
            proxy_send_timeout          86400;
            proxy_read_timeout          86400;
            send_timeout                86400;
        }
    }
}
cold cape
#

ah nice, lets see now

#

!!!

light inlet
#

frontend time

cold cape
#

Youre awesome!

#

2023/07/25 19:04:15 [error] 30#30: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.2, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://[fd12:4f6a:612e::ac:6c18:14fb]:3000/favicon.ico", host: "arena-nginx-production.up.railway.app", referrer: "https://arena-nginx-production.up.railway.app/api/healthz/";

light inlet
#

well first

cold cape
#

2023/07/25 19:05:23 [error] 30#30: *32 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://[fd12:4f6a:612e::ac:6c18:14fb]:3000/favicon.ico", host: "arena-nginx-production.up.railway.app", referrer: "https://arena-nginx-production.up.railway.app/initialize";

light inlet
#

what framework is that frontend

cold cape
#

NextJS

light inlet
#

what's your start command

cold cape
#

let me get you the command that runs it, a sec

#

do we need that [::] ipv6 thing?

#

something like this?

light inlet
#

next start -H :: -p $PORT

cold cape
#

awesome, let me try that now

light inlet
#

make sure you have a service variable PORT = 3000

cold cape
#

byw, this should be fine right? (like if I want to give a default)

light inlet
#

nope

#

delete line 33, 34

cold cape
#

how do I set a default?

cold cape
light inlet
#

and what's your new start command

cold cape
#

being deployed

light inlet
#

okay just checking, since you like to change things from what I say

cold cape
#

service port is already set since a while

light inlet
#

I'm happy I could help

cold cape
#

You did an amazing job at helping me

light inlet
#

thank you

cold cape
#

railways team themselves didnt bother much

#

Now, I can start migrating several of our core services into railway,

light inlet
#

I mean they're right, it is better suited for discord

#

that's your aws costs?

cold cape
#

Railway should probably get you something

light inlet
#

bro they won't even give me a sticker

#

I've been asking for stickers for so long

cold cape
light inlet
#

I'm just a community member

cold cape
cold cape
light inlet
#

I'm too silly for them

cold cape
#

ahh, dont tell so. Where are you from byw? what do you do for a living?

light inlet
#

maple syrup land, and I help people in the railway server for a living

cold cape
#

I suggest you make a few templates and contribute them to railway, like the nginx and caddy ones

light inlet
#

haha made you google

#

I have made a few templates, and I'll be adding the caddy template in the future

cold cape
light inlet
#

I currently have 2$

cold cape
#

hmm...

light inlet
#

I don't know how it works, but oh well

light inlet
#

pretty damn clean with caddy

{
    admin off
    persist_config off
    auto_https off
    log { format console }
}

:{$PORT} {
    reverse_proxy frontend.railway.internal:3000

    handle_path /api/* {
        reverse_proxy backend.railway.internal:3000
    }
}
light inlet
hollow epoch
#

Sorry to necro this old thread btw