Internal Server Error (500) on Production Standalone Mode in Alpine Image - Next.js 13.4.12, 13.4.19 | Next.js | Page 1

terse sorrel Aug 27, 2023, 12:30 AM

#

Hello! I'm currently having issues when trying to deploy a NextJS app using standalone mode in a Docker image. I've spent all day trying to figure out the issue and apply 'fixes' that people have found online, but none of them seem to work for me.

On Next.js 13.4.12, the whole website returns the Internal Server Error 500 error, with ECONNREFUSED 0.0.0.0:36993 (or a similar port) errors in the logs.

On Next.js 13.4.19 (latest), the website loads however just the image endpoint receives the error instead.

Looking online, this seems to have been a common issue introduced in 13.4.13 and fixed in 13.4.19, but I'm thinking this may be a different issue?

Will post additional logs and info in the thread.

final thicketBOT Aug 27, 2023, 12:30 AM

#

Post created!

🔎 This post has been indexed in our web forum and will be seen by search engines so other users can find it outside Discord

🕵️ Your user profile is private by default and won't be visible to users outside Discord, if you want to be visible in the web forum you can add the "Public Profile" role in id:customize

✅ You can mark a message as the answer for your post with Right click -> Apps -> Mark Solution
(if you don't see the option, try refreshing Discord with Ctrl + R)

terse sorrel Aug 27, 2023, 12:31 AM

#

Dockerfile:

FROM node:18-alpine

ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY ./public /app/public
COPY --chown=nextjs:nodejs ./.next/standalone /app
COPY --chown=nextjs:nodejs ./.next/static /app/.next/static

USER nextjs

EXPOSE 3000

ENV PORT 3000
ENV HOSTNAME "0.0.0.0"


WORKDIR /app
CMD ["node", "server.js"]

(Have tried adding
ENV NODE_TLS_REJECT_UNAUTHORIZED 0
and ENV HOSTNAME "127.0.0.1" but did not work)

#

Logs on 13.4.12

(Summary for web forum index)

- info Loaded env from /app/.env
Listening on port 3000 url: http://0.0.0.0:3000
- info Loaded env from /app/.env
Error: socket hang up
    at connResetException (node:internal/errors:720:14)
    at Socket.socketOnEnd (node:_http_client:525:23)
    at Socket.emit (node:events:526:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ECONNRESET'
}
...
Error: connect ECONNREFUSED 0.0.0.0:36993
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '0.0.0.0',
  port: 36993
}

📎 message.txt

#

Logs on 13.4.19

(Summary for web forum indexing)

website-v6-dev.1.9adq5q5216lm@Panavision    | - ready started server on 127.0.0.1:3000, url: http://127.0.0.1:3000
...
ready started server on [::]:3000, url: http://localhost:3000
website-v6-dev.1.c0xg42joylic@Panavision    | - info Loaded env from /app/.env
website-v6-dev.1.c0xg42joylic@Panavision    | - info Loaded env from /app/.env
website-v6-dev.1.c0xg42joylic@Panavision    | TypeError: fetch failed
website-v6-dev.1.c0xg42joylic@Panavision    |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
website-v6-dev.1.c0xg42joylic@Panavision    |     at async invokeRequest (/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
website-v6-dev.1.c0xg42joylic@Panavision    |     at async invokeRender (/app/node_modules/next/dist/server/lib/router-server.js:254:29)
website-v6-dev.1.c0xg42joylic@Panavision    |     at async handleRequest (/app/node_modules/next/dist/server/lib/router-server.js:447:24)
website-v6-dev.1.c0xg42joylic@Panavision    |     at async requestHandler (/app/node_modules/next/dist/server/lib/router-server.js:464:13)
website-v6-dev.1.c0xg42joylic@Panavision    |     at async Server.<anonymous> (/app/node_modules/next/dist/server/lib/start-server.js:117:13) {
website-v6-dev.1.c0xg42joylic@Panavision    |   cause: Error: read ECONNRESET
website-v6-dev.1.c0xg42joylic@Panavision    |       at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
website-v6-dev.1.c0xg42joylic@Panavision    |     errno: -104,
website-v6-dev.1.c0xg42joylic@Panavision    |     code: 'ECONNRESET',
website-v6-dev.1.c0xg42joylic@Panavision    |     syscall: 'read'
website-v6-dev.1.c0xg42joylic@Panavision    |   }
website-v6-dev.1.c0xg42joylic@Panavision    | }

📎 message.txt

#

If I ssh into the container and use netstat, there doesn't seem to be anything on the port that's having the issue on 13.4.12. On 13.4.19, there's a next-render-work process on the port

#

Running the standalone server locally on my Windows development machine works fine - the issue seems to just be on the Docker image (tried both node:16-alpine and node:18-alpine). I have a GitLab CI/CD pipeline that successfully installs deps, lints and builds.

Sharp is also installed in the package.json. I'm using Prisma linked to a read-only MySQL database and a couple external apis. Also using google fonts using the new next/font/google system.

Thanks in advance!

#

Internal Server Error (500) on Production Standalone Mode - Next.js 13.4.12, 13.4.19

#

Internal Server Error (500) on Production Standalone Mode in Alpine Image - Next.js 13.4.12, 13.4.19

hearty crater Aug 27, 2023, 1:53 AM

#

Have you tried a different node version? My alpine docker image is set to 18.17.1 and it's working

#

There's been a whole slew of issues with standalone builds and things that affected docker deployments but it all seems to be resolved as of 13.4.19 although I had to make a bunch of changes and ensure I'm passing in ENV HOSTNAME 0.0.0.0 (AWS ECS deployments set this to the internal AWS host/address if you don't specifically set it yourself and it needs to be 0.0.0.0)

#

Where are you deploying it?

terse sorrel Aug 27, 2023, 1:56 AM

#

It’s deployed on a Docker Swarm that goes through a nginx reverse proxy on another swarm service. The env var should already be set in the dockerfile.

#

Will try updating the node version tomorrow

hearty crater Aug 27, 2023, 1:58 AM

#

Yea just ensure the swarm isn't overriding it by default. I assume it wouldn't but worth checking

#

It's really been a nightmare over the past two weeks with all the bugs that appeared related to standalone

terse sorrel Aug 27, 2023, 10:37 AM

#

hearty crater Have you tried a different node version? My alpine docker image is set to 18.17....

It worked for the images on the first page then stopped working again with the same error

#

On alpine 18.7.1 with next 13.4.9, have confirmed the hostname is 0.0.0.0 in the env

terse sorrel Aug 27, 2023, 2:08 PM

#

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeRequest (/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
    at async invokeRender (/app/node_modules/next/dist/server/lib/router-server.js:254:29)
    at async requestHandler (/app/node_modules/next/dist/server/lib/router-server.js:475:24)
    at async Server.<anonymous> (/app/node_modules/next/dist/server/lib/start-server.js:117:13) {
  cause: Error: connect ECONNREFUSED 0.0.0.0:41955
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '0.0.0.0',
    port: 41955
  }
}

For some reason it seems to be trying to fetch a port that doesn't have anything listening on it

#

The line on router-server.js seems to think there is a worker running on that port, which there isn't.

#

So it appears that when the container boots then the port is correct, but there is this error.

Then after a bit the next render worker program stops and starts on a new port, but the new port is not updated in the router worker so it then shows the other error

terse sorrel Aug 27, 2023, 3:09 PM

#

Reproduced on a full node 18.17.1 docker image, same issue. Also tried directly installing sharp within the docker image.

If I take the .next directory and manually run it on my windows machine it works fine

terse sorrel Aug 28, 2023, 11:27 AM

#

So done a bit more testing, it seems if I run the Docker image on the debian linux server in the swarm (through nginx) or individually (directly) it doesn't work, but if I run the exact same image on my windows machine with wsl2 it's fine

#

Running it on my ubuntu linux server is also fine. Have also tried running the container in privileged more and it doesn't make any difference. The ubuntu and debian docker inspect responses are pretty much identical. Only diff apart from the OS is my debian install is on docker 24.0.4 and ubuntu is on 24.0.5

terse sorrel Aug 28, 2023, 12:34 PM

#

Hello - managed to resolve this, but just an update if anyone else has this issue/is interested:

The issue was (I believe) with the server having an older CPU - some sort of Intel Xeon 3GHz 4 core one, not sure on the model. I managed to fix this by pinning the service on Docker Swarm to a different host.

I think it was completely unrelated to all the other issues introduced in Next 13.4.13.

In hindsight, I remember having a similar problem with the next build command in the GitLab CI/CD runner - I was receiving a random Illegal Instruction Core Dumped fault on this node, which I accidentally discovered the fix when I restarted the runner and it restarted on a different host. But as I couldn't see any errors or logs anywhere and as that was done a while ago, I'd completely forgotten about it.

Thanks for anyone who helped!

#Internal Server Error (500) on Production Standalone Mode in Alpine Image - Next.js 13.4.12, 13.4.19