Unable to run in AWS ECS/fargate - MODULE_NOT_FOUND

[hazy lodge]

Hi! As said in here https://github.com/backstage/backstage/issues/10093#issuecomment-1530769283 Facing similar issue while running backstage in AWS ECS/fargate

5/2/2023, 9:50:53 AM    node:internal/modules/cjs/loader:936    backstage
5/2/2023, 9:50:53 AM    throw err;    backstage
5/2/2023, 9:50:53 AM    ^    backstage
5/2/2023, 9:50:53 AM    Error: Cannot find module '/app/packages/backend'    backstage
5/2/2023, 9:50:53 AM    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)    backstage
5/2/2023, 9:50:53 AM    at Function.Module._load (node:internal/modules/cjs/loader:778:27)    backstage
5/2/2023, 9:50:53 AM    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)    backstage
5/2/2023, 9:50:53 AM    at node:internal/main/run_main_module:17:47 {    backstage
5/2/2023, 9:50:53 AM    code: 'MODULE_NOT_FOUND',    backstage
5/2/2023, 9:50:53 AM    requireStack: []    backstage
5/2/2023, 9:50:53 AM    }

the image runs perfectly fine in local, same image from registry, but unfortunately it fails in fargate and I am unable to understand why
image is created with DOCKER_BUILDKIT=1 as requested due to the --cache command of the packages/backend/Dockerfile
tried with backstage 1.13.0/1.13.2/1.14.0-next.0 all having same behavior.

Unable to run in AWS ECS/fargate - MODULE_NOT_FOUND

[fiery ginkgo]

Hi, @hazy lodge, we run Backstage the same way in Fargate and don't have any issues. We've deployed 1.13.0 and are currently running 1.14.0-next.0.

I'll and see if I can think of anything you could look at. The one thing that does seem odd is that you have multiple versions of the same packages. That's generally a bad thing so I might clean that up to start

[hazy lodge]

Hi @fiery ginkgo thanks for the response. Which multiple versions do you refer to?

That is for the cli 0.22.6 version

[fiery ginkgo]

@olive orchidstage/core-components, @olive orchidstage/integration and @olive orchidstage/plugin-catalog-react

[hazy lodge]

Oh I see, let me check why that is there, thanks

[fiery ginkgo]

I don't think that's the issue but it's just the only thing I can think of right now. Could you post your Dockerfile? Maybe that might have some clues? Feel free to obfuscate anything sensitive

[hazy lodge]

I am using same image provided by backstage https://github.com/backstage/backstage/blob/master/packages/backend/Dockerfile

yarn install --frozen-lockfile
yarn tsc
yarn build:backend --config ../../app-config.yaml

information listed here https://backstage.io/docs/deployment/docker

BTW: it is ../../app-config.yaml and not app-config.yaml found in the documentation as you run the command build:backend from root and there is where the app-config.yaml is. But anyway that s not the issue.

@fiery ginkgo just cleaned the dependencies and left the right ones, no duplications and still same error.

Tested in EKS as well same images same error 😢

within the image:

-rw-r--r--   1 node node    1459 May  2 03:14 app-config.production.yaml
-rw-r--r--   1 node node    3864 May  2 03:14 app-config.yaml
drwxr-xr-x 612 node node   20480 May  2 03:16 node_modules
-rw-r--r--   1 node node    1464 Jan  1  1970 package.json
drwxr-xr-x   1 node node    4096 Jan  1  1970 packages
-rw-r--r--   1 node node 1018280 Jan  1  1970 yarn.lock

node@b99d23b8c960:/app$ ls -la packages/backend/
total 24
drwxr-xr-x 1 node node 4096 May  2 03:16 .
drwxr-xr-x 1 node node 4096 Jan  1  1970 ..
-rw-r--r-- 1 node node 2176 Jan  1  1970 README.md
drwxr-xr-x 2 node node 4096 May  2 03:16 dist
-rw-r--r-- 1 node node 2056 Jan  1  1970 package.json

[hazy lodge]

@fiery ginkgo do you have an example of a Dockerfile I can try?

[hidden nebula]

Hi @hazy lodge , You could use yarn why to check for any duplicate dependencies. Might be helpful.

[hazy lodge]

thanks @hidden nebula I did fix that after @fiery ginkgo 's comment now there are not duplicates.

[fiery ginkgo]

Sorry, been busy today. We use the same Dockerfile from the multistage docs, basically just below what you are using in the docs

The build command we use is: yarn --cwd packages/backend build from the root. We don't pass it the --config at all. But it may be picking that up by convention like in the build process itself. Not super sure on that

[fiery ginkgo]

This bugs me a lot. Like it doesn't make sense. The error you are hitting feels like a compiled code type of issue so it should be everywhere you run the container. The only new thing that came to mind was permissions

[hazy lodge]

It doesn't make sense at all. I have been sure to provide right permissions doing chmod +x passing --chown=node:node --chmod=0755 everything you can imagine.

I have been trying everything

and getting [FATAL tini (7)] exec /app/entrypoint.sh failed: Permission denied or /bin/sh: 0: cannot open /app/entrypoint.sh: Permission denied even extracting the node call to a entrypoint file

just to verify

now what I did was to keep the file as it is, but in the entrypoint have a simple tiny server to listen at 7007, (no backstage) to see if it was related to backstage packages.

got the same but now with the server I put in place

this is the most annoying problem I have found in months, also because testing is so tedious until you can verify the run.

in local everything works fine, just it fails in ECS and EKS