Garbled response / Gzip issue? | Railway | Page 1

loud fern Sep 11, 2024, 3:56 PM

#

I've been having an issue I can't get to the bottom of, so finally asking here.

Context:

I have one Railway project, with several services: one frontend (NextJS, App Router, mainly SSR), one backend service (Django), a database (postgres)
Frontend tries to connect to the backend via private URL, and falls back to public URL if the initial connection fails (like when in client side, or during build)
Frontend calls various APIs, as you'd expect, APIs return JSON
There is no auth between services
The backend has django.middleware.gzip.GZipMiddleware and corsheaders.middleware.CorsMiddleware
The frontend has Cloudflare DNS, which I use to manage the domain
I'm using the new proxy

Issue:

I made a change to remove pagination in one of my APIs, and removed the associated pagination logic on the frontend
After deploy, I get the attached error from my frontend during build

Things I've tried:

Calling this API on the cmd line/postman works fine, and I get Content-Encoding = gzip in the response headers
Disabling GZipMiddleware in Django, get the same result
Mocking the response - returning hardcoded JSON - get same issue
When I make the hardcoded JSON smaller, it often works - this makes me think it's a Gzipping issue (which triggers when the response is >200 bytes)
Trying the legacy proxy - same result
Setting Accept-Encoding = gzip, deflate, br and Content-Type=application/json on the request in Next
Adding DEFAULT_RENDERER_CLASSES = rest_framework.renderers.JSONRenderer in Django
Praying

c18400df-5dee-4fe7-bc56-a98db8d475bd

📎 output

scenic tulipBOT Sep 11, 2024, 3:56 PM

#

Project ID: c18400df-5dee-4fe7-bc56-a98db8d475bd

loud fern Sep 11, 2024, 3:57 PM

#

Additional:

Another API is called during build that returns an extremely small response (JSON with ~5 entries) and this works fine.

loud pasture Sep 11, 2024, 4:10 PM

#

do you have an endpoint i could curl to reproduce this issue?

loud fern Sep 11, 2024, 4:10 PM

#

curl -s -X GET 'https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/' | jq

loud pasture Sep 11, 2024, 4:11 PM

#

is that supported to work?

loud fern Sep 11, 2024, 4:11 PM

#

ha - yes - what are you seeing?

loud pasture Sep 11, 2024, 4:12 PM

#

json

loud fern Sep 11, 2024, 4:12 PM

#

yes

loud pasture Sep 11, 2024, 4:12 PM

#

can i have an endpoint that reproduces the issue, im not sure what i should be doing with an enpoint that works fine

loud fern Sep 11, 2024, 4:13 PM

#

yes... I mean that's the issue

The endpoint works fine, every time, when called in curl/postman.

But in build it gives the output I shared in the first post. This is what I can't understand.

loud pasture Sep 11, 2024, 4:14 PM

#

during build of your FE?

loud fern Sep 11, 2024, 4:14 PM

#

yep

#

are you able to see my build logs?

loud pasture Sep 11, 2024, 4:14 PM

#

how is this url being called?

loud fern Sep 11, 2024, 4:14 PM

#

one sec, getting the code

#

📎 callstack

#

forgive the terrible loggin

loud pasture Sep 11, 2024, 4:19 PM

#

did you say you have disabled GZIP on the BE and still got the same garbage response?

loud fern Sep 11, 2024, 4:19 PM

#

yep - I thought the logs look like printed gzip

#

so I removed GZipMiddleware, same issue

loud pasture Sep 11, 2024, 4:20 PM

#

is GZIP currently disabled (removed)?

loud fern Sep 11, 2024, 4:20 PM

#

no i renabled but can disable it again

loud pasture Sep 11, 2024, 4:20 PM

#

lets leave it for now

#

just trying to parse through these logs

loud fern Sep 11, 2024, 4:24 PM

#

good luck

#

LMK if I can help

loud pasture Sep 11, 2024, 4:27 PM

#

i dont see content-encoding = gzip anywhere in these FE build logs, does response.headers.entries() strip that out?

loud fern Sep 11, 2024, 4:28 PM

#

take a look at the backend logs - I'm logging the headers on inbound requests:

loud pasture Sep 11, 2024, 4:29 PM

#

right but content-encoding = gzip should be returned as a response header

loud fern Sep 11, 2024, 4:30 PM

#

yeah agree it's not there

#

but why?

loud pasture Sep 11, 2024, 4:32 PM

#

curl -I "https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 11 Sep 2024 16:31:23 GMT
Referrer-Policy: same-origin
Server: railway-edge
Vary: Accept-Encoding, origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Railway-Request-Id: nmUVBEMHRwKJVioSmvMM1Q_2020806880

misbehaving middleware?

but you said it was the same with the middleware removed?

loud pasture Sep 11, 2024, 4:48 PM

#

either way, can you disable / remove the gzip middleware for now?

loud fern Sep 11, 2024, 4:51 PM

#

dude sorry, back at keys

#

disabling now

loud pasture Sep 11, 2024, 4:52 PM

#

for context, curl was showing that it got the gzip response header -

C:\Users\Brody>curl --location -I "https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/" --header "Accept-Encoding: gzip, deflate, br" --header "Content-Type: application/json"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Encoding: gzip
Content-Length: 826
Content-Type: application/json
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 11 Sep 2024 16:40:05 GMT
Referrer-Policy: same-origin
Server: railway-edge
Vary: Accept-Encoding, origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Railway-Request-Id: 1bO2oXJ6Sn-rCTujPm4rCw_882434190

loud fern Sep 11, 2024, 4:52 PM

#

this is what led me to think it might be something to do with the proxy

#

communicating with the end point directly works

loud pasture Sep 11, 2024, 4:53 PM

#

the proxy doesnt touch any of these headers afaik

loud fern Sep 11, 2024, 4:54 PM

#

https://help.railway.app/questions/new-proxy-stripping-accept-encoding-head-56fb16a0

Railway Help Station

The new proxy is stripping the Accept-Encoding header

On the legacy proxy, requests correctly returned the gzipped response when browsers send the Accept-Encoding: gzip, deflate, br header. However, this does not work on the new proxy. It appears that the Accept-Encoding header is stripped before reaching my server.

#

aware this is now solved, but wondering if related?

loud pasture Sep 11, 2024, 4:54 PM

#

loud pasture the proxy doesnt touch any of these headers afaik

well besides the headers it does touch, you know what i mean lol

loud fern Sep 11, 2024, 4:54 PM

#

btw, I noticed that when I deploy the backend, I have to commit/push to force a rebuild on my frontend, rather than just trigging a deploy from last commit

loud pasture Sep 11, 2024, 4:54 PM

#

loud fern https://help.railway.app/questions/new-proxy-stripping-accept-encoding-head-56fb...

Hello all, we have pushed a change to our proxy, we no longer strip out that header!

loud pasture Sep 11, 2024, 4:55 PM

#

loud fern btw, I noticed that when I deploy the backend, I have to commit/push to force a ...

what happens when you do deploy from last commit?

loud fern Sep 11, 2024, 4:55 PM

#

loud pasture > Hello all, we have pushed a change to our proxy, we no longer strip out that h...

I know, but just wondering if could be a related thing?

loud fern Sep 11, 2024, 4:55 PM

#

loud pasture what happens when you do deploy from last commit?

I'll show you

loud pasture Sep 11, 2024, 4:55 PM

#

loud fern I know, but just wondering if could be a related thing?

at this time, i dont think so

loud fern Sep 11, 2024, 4:55 PM

#

backend is deployed

#

frontend deploying now with deploy from last commit

loud pasture Sep 11, 2024, 4:57 PM

#

i can confirm with postman the BE response is no longer gzipped, no encoding header and resonse body is larger

loud fern Sep 11, 2024, 4:58 PM

#

agreed, see the same

loud pasture Sep 11, 2024, 4:58 PM

#

interesting

loud fern Sep 11, 2024, 4:58 PM

#

ok - frontend built correctly with trigger last commit

#

but same issue

loud pasture Sep 11, 2024, 4:59 PM

#

lets use the private domain then

loud fern Sep 11, 2024, 4:59 PM

#

in build?

loud pasture Sep 11, 2024, 4:59 PM

#

yeah

loud fern Sep 11, 2024, 4:59 PM

#

oooh

#

have you just opted me in

#

fine if you have

loud pasture Sep 11, 2024, 5:00 PM

#

yeah, just trying to move fast here, i will ask next time

loud fern Sep 11, 2024, 5:00 PM

#

no don't worry, just wonderd

#

didn't remember doing it and I've been battling this for 1.5 days... starting to forget what life was like before this bug

#

I think it's worked

#

either way - being able to use private urls in build is a win

#

shall I renable gzip on the BE?

loud pasture Sep 11, 2024, 5:06 PM

#

oh, it worked...

#

i was kinda hoping it wouldnt

loud fern Sep 11, 2024, 5:06 PM

#

yeah, as is we haven't learnt anything

loud pasture Sep 11, 2024, 5:06 PM

#

exactly

loud fern Sep 11, 2024, 5:06 PM

#

just that the new builder works for some reason

#

enabling gzip

loud pasture Sep 11, 2024, 5:07 PM

#

hold on

#

not yet please

loud fern Sep 11, 2024, 5:07 PM

#

oops

#

aborted the build

loud pasture Sep 11, 2024, 5:07 PM

#

oh wait, this is a production site, can you afford to keep this offline as we debug?

loud fern Sep 11, 2024, 5:07 PM

#

yes

#

all my user will be ok with it

loud pasture Sep 11, 2024, 5:08 PM

#

are you sure? we could spin up a dev env in railway

loud fern Sep 11, 2024, 5:08 PM

#

yes don't worry

loud pasture Sep 11, 2024, 5:09 PM

#

alright, sounds good, but yeah lets keep gzip off for now, it eliminates variables

loud fern Sep 11, 2024, 5:10 PM

#

anything I can do to help debug?

#

I would like to remove the hard-coded response, and replace with DB query

#

ok for me to try?

#

I'll leave gzip off

loud pasture Sep 11, 2024, 5:11 PM

#

yeah thats fine

loud fern Sep 11, 2024, 5:17 PM

#

that's deployed

#

redeploying FE

loud pasture Sep 11, 2024, 5:19 PM

#

well my fridge doesn't cool anymore, I gotta go AFK while I transfer everything to another fridge outside

loud fern Sep 11, 2024, 5:21 PM

#

good luck

#

I'm going to try renabling gzip while you're fridging

#

FYI database connection worked

loud pasture Sep 11, 2024, 5:21 PM

#

they didn't before?

loud fern Sep 11, 2024, 5:22 PM

#

ah, it was just one of many things I've been trying to debug the issue - hardcoding the json response in my backend

#

was looking for special characters/fixing the size of the repsonse, etc

#

gzip works

loud pasture Sep 11, 2024, 5:28 PM

#

okay I have some more things I'd like to try when I'm back

loud fern Sep 11, 2024, 6:19 PM

#

where are we on the fridge outage, Brody? P1? Users affected? Is there an incident channel we should join?

loud pasture Sep 11, 2024, 6:24 PM

#

there's an incident? I thought we left off in a working state with the new builder?

loud fern Sep 11, 2024, 6:25 PM

#

was a joke about your fridge, ignore

#

anyway, new builder working - I'm back to coding, thanks for your help

loud pasture Sep 11, 2024, 6:26 PM

#

oh I thought you were serious 😅

loud fern Sep 11, 2024, 6:26 PM

#

rarely

loud pasture Sep 11, 2024, 6:26 PM

#

I thought you were like, screw your fridge, help me with this gzip stuff

loud fern Sep 11, 2024, 6:27 PM

#

haha

#

would never priotise work above food

loud pasture Sep 11, 2024, 6:28 PM

#

I'll be back at the computer as soon as I finish eating and moving the food to the other fridge

#Garbled response / Gzip issue?