✅ - The PR previews stopped working overnight

[frank silo]

One day one PR was compiling and two days after (25th August) it stopped working along with the other previews. No configuration changes. The main branch (no preview) compiles and deploys perfectly.

The only error came from the backend log several times:

2023-08-25T09:39:50.524Z [WARNING]: - Initializing your environment: dev
2023-08-25T09:39:51.313Z [WARNING]: - Building resource api/proppexV1
2023-08-25T09:39:51.354Z [WARNING]: - Building resource auth/proppexAuth
2023-08-25T09:39:51.527Z [WARNING]: - Building resource auth/userPoolGroups
2023-08-25T09:39:51.732Z [WARNING]: - Building resource storage/proppexPublicBucket
2023-08-25T09:39:51.765Z [WARNING]: ✔ Initialized provider successfully.
2023-08-25T09:40:13.991Z [WARNING]: ✖ There was an error initializing your environment.
2023-08-25T09:40:13.995Z [INFO]: 🛑 Executing custom build scripts is not yet implemented
2023-08-25T09:40:13.995Z [INFO]: Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

Attached a picture showing that the same commit first worked and then didn't.

[jovial mist]

@frank silo is this still not working?

[topaz cypress]

@bleak terrace

[frank silo]

I had to change the build script in the Amplify console to make it work somehow... but I have my own amplify.yml in the root of the project. Wich I guess it's being ignored

So, I made it work, but I don't know how exactly

@jovial mist

[frank silo]

Update: I tried to deploy to another branch and it failed with the same error

it is a go-live deployment and Amplify let me down

😞

[frank silo]

BTW, I tried again today. The session id was 5868dca4-46ea-42b3-a946-43fddb1d76d1

[frank silo]

Hello, sorry for being pushy...
This is the GitHub ticket that I opened about this issue:
https://github.com/aws-amplify/amplify-hosting/issues/3670

@bleak terrace maybe you could shed some light here?

[bleak terrace]

Hi @frank silo 👋 , I am investigating into this and will keep you posted

[frank silo]

Thank you!

[bleak terrace]

Just to confirm, does the issue occur in the following workflow?

1. Amplify app with multiple branches and backend environments:

• main
• dev
• stage

2. Enable PR previews against main branch:

• Create new backend environment or select an existing environment to use

3. Add amplify.yml to the root of the repository:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

4. When a new PR is created against main , backend build fails with "🛑 Executing custom build scripts is not yet implemented".

[bleak terrace]

A couple of follow questions:

• Have you enabled full stack CI/CD deployment feature on the main branch: https://docs.aws.amazon.com/amplify/latest/userguide/amplify-config-autogeneration.html ?
• Are you updating the backend via amplify push or via CI/CD deployments? If via amplify push what CLI version are you using locally?

[pliant rover]

When I run into this error message, it's always in amplify CONSOLE CI/CD. It happens with amplify cli 12.2.3 or latest.

Running an amplify push locally with THE SAME COMMIT will SUCCEED with amplify cli 12.2.3.

I run into this error message in console when I add a new function as a 'nodejs/npm' function with a 'build' entry in the new function's build.json.
This USED TO work. I have several functions that work with this configuration, in the same backend.

I also run into this error message in console when I add a new function as "nodejs/custom build script," which puts the build script in the new function's amplify.state.

The next permutation of things that should work that I'm going to try is the typescript examples in amplify documentation.
https://docs.amplify.aws/cli/function/build-options/

You can see my application d5i79jkckfr0q for lots of examples of this error breaking backend builds.

[bleak terrace]

Thanks for sharing that information @pliant rover , we are investigating into this further.

[pliant rover]

New test case -- Just ran 'amplify add function' and did a git commit. No custom build steps at all. Just whatever 'amplify add function' added to my project.

Ran git push, and console backend build failed with 'Executing custom build scripts is not yet implemented' again.

[hallow forge]

@pliant rover could provide us the function/<function-name>/ amplify.state?
Additionally, is the function connected to a layer?

[pliant rover]

Three different failures

{
"pluginId": "amplify-nodejs-function-runtime-provider",
"functionRuntime": "nodejs",
"useLegacyBuild": true,
"defaultEditorFile": "src/index.js",
"scripts": {
"build": "npm install --no-bin-links --production"
}

{
"pluginId": "amplify-nodejs-function-runtime-provider",
"functionRuntime": "nodejs",
"useLegacyBuild": true,
"defaultEditorFile": "src/index.js",
"scripts": {
"build": "echo 'function build a' && rm -rf ./src && mkdir -p ./src && echo 'function build b' && mkdir -p ./ts/src/copypasta/ && echo 'function build c' && cp -r ../copypasta/opt/* ./ts/src/copypasta/ && echo 'funciton build d' && npm install && echo 'function bulid e' && tsc && echo 'function build f' && cp -r ./ts/src/copypasta/* ./src/copypasta && echo 'function build g' && cp ./ts/package.json ./src && sed -i 's/tximportexport/jximportexport/' ./src/package.json && echo 'function build h' && cd ./src && npm install && echo 'function build i' "
}

{
"pluginId": "amplify-nodejs-function-runtime-provider",
"functionRuntime": "nodejs",
"useLegacyBuild": true,
"defaultEditorFile": "src/index.js",
"scripts": {
"build": "npm install --no-bin-links --production"
}
}

[pliant rover]

Not connected to a layer

[pliant rover]

A lot of my custom build steps are about replacing functionality that could otherwise be done with a lambdalayer because using layers breaks something in amplify...I'm trying to remember what... amplify api mock?

[bleak terrace]

Can you share some insights on the following:

• Are you using a monorepo setup?
• Do you have a buildSpec file (amplify.yml) at the app root level in the repository? If yes, can you share that?

[pliant rover]

Yep, using a monorepo. This is the amplify.yml that is relevant for the backend builds that are failing.

version: 1
applications:

• backend:
  phases:
  preBuild:
  commands:
  - echo "THIS IS THE GIT VERSION OF amplify.yml"
  - npm install -g typescript
  - yum install jq -y
  - which jq
  - pwd
  - ls
  - cat ../backend/amplify/team-provider-info.json
  - jq {"${USER_BRANCH}"} ../backend/amplify/team-provider-info.json
  - jq {"${USER_BRANCH}"} ../backend/amplify/team-provider-info.json > temp.json
  - cat temp.json
  - cat temp.json > amplify/team-provider-info.json
  - cat amplify/team-provider-info.json
  - echo "env"
  - env
  build:
  commands:
  - '# Execute Amplify CLI with the helper script'
  - amplifyPush --simple
  frontend:
  phases:
  preBuild:
  commands:
  - npm ci
  build:
  commands:
  - npm run build
  artifacts:
  baseDirectory: build
  files:
  - '/*'
  cache:
  paths:
  - node_modules//*
  appRoot: backend

There are some prebuld steps to fix an issue I've been having with initializing new backends. Removes all the non-relevant team-provider-info.json entries. I set up new backends a lot. Like four times related to this error.

The echo at the top is to make sure the repo amplify.yml is being run.
If you have a monorepo, it looks for amplify.yml at /<this-app-dir>. IIRC I couldn't find that detail in the docs. Suggest you add that.

The build fails at the 'backend' stage, so I doubt it's failing at frontend.phases.build.commands "npm run build"

[bleak terrace]

Thanks for sharing this detailed information @pliant rover . We are continuing to actively investigate into this and will keep you posted with any updates.

[ivory fossil]

I am having a simlar issue with just the standard ci/cd flow. I have an env for my main branch, and it is not getting triggered at all when something gets merged into main.

[ivory fossil]

Getting this as the reply from aws

{
    "message" : "Error: Invalid GitHub Webhook request"
}

It looks like Amplify sends this error on merged PRs, but it works if I just push up a new commit to main

@bleak terrace ^

I can send you the payloads sent by github directly, but they look very similar, and are structurally the same.

[bleak terrace]

@pliant rover Can you remove the node live package update from your app d5i79jkckfr0q and then redeploy?

The build container by default supports Node 16.

[pliant rover]

remove the node live package update

How would I do that?

[bleak terrace]

@frank silo Can you add and set the amplify-cli live package update to point to latest for your app dg7jtgfxh9v63? Please redeploy your app then

[pliant rover]

Should I remove the node version from build settings?

[bleak terrace]

How would I do that?

https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#configuring-live-updates

[pliant rover]

OK removing node from build settings

[frank silo]

Sorry, I just saw this. It's a little late for my TZ

[frank silo]

Omw

btw, I'm not using a monorepo. I have the Amplify backend in a repo and the frontend in a different one

[frank silo]

Isn't latest the default config?

[bleak terrace]

The build container by default uses Amplify CLI v10.5.2. Thus, you need to add the live package override and set it to latest

[frank silo]

oh...

is that a magic version? is it documented somewhere?

btw, the build worked!!! 🎉

[bleak terrace]

Unfortunately, this information is undocumented and we are working on reflecting it in the console.

Glad to hear that your build succeeded 😃

[frank silo]

I'm asking my devs to push something to the PRs to see if the previews work again too

lookin good

hard to track, easy to fix...

but why the main branch deployed correctly?

[pliant rover]

I have a few examples of this failure and I'm rebuilding them all...

• Just amplify add function and git push -- seems to be working now 👍
• Putting my custom build steps in amplify.state (aka declare custom build script) -- is now breaking for some other unrelated reason (policy document too long) -- so probably 👍
• Legacy style: putting my custom build steps in /../function-name/ts/src/package.json -- custom build script seems to run and fail because it's janky -- so probably 👍

will try to actually get these to succeed...

[bleak terrace]

Thanks @pliant rover !

[pliant rover]

I'll hopefully have those backends working tomorrow. Any idea what the root cause is? Is there like a dependency that changed but we can still build with the node version that happens to be on that build image right now?

[pliant rover]

I'm still having a weird issue with the 'declare custon build script' approach. With an amplify.state that looks like this

{
"pluginId": "amplify-nodejs-function-runtime-provider",
"functionRuntime": "nodejs",
"useLegacyBuild": true,
"defaultEditorFile": "src/index.js",
"scripts": {
"build": "echo 'function build a'"
}
}

I get this error:

2023-09-06T23:10:21.599Z [INFO]: 🛑 Received error [Error: Command failed with ENOENT: echo 'function build a'
spawn echo ENOENT] running command [echo 'function build a']

I don't know if this is new because I've never used the amplify.state.scripts feature before I started getting the "Executing custom build scripts is not yet implemented" error.

Should I be able to run the same shell commands (like echo) in amplify.state.scripts that I've historically been able to run in package.json.scripts? Should I put my build script into a file that I reference from amplify.state.scripts ?

In any case, I'm no longer blocked because my old workflow with package.json.scripts works again. Thanks!

[hallow forge]

hey @pliant rover , we were able to reproduce the issue. Executing custom build scripts is not yet implemented error occur when creating a function using Amplify CLI versions greater than 12.2.0 and building the functions in Amplify CLI version less than 12.2.0 in this case the CICD backend build.
To mitigate the issue,

1. Ensure that the build settings -> Live package updates are currently set to Amplify CLI version is greater than 12.2.0
2. If the Live package updates has a Node.js version overridden to a specific version, remove the override. We have identified a issue where the backend build is being tied to an older version of the CLI.
   could you try just adding npm install in the scripts build and rerun the build

[pliant rover]

Cool.

could you try just adding npm install in the scripts build and rerun the build

In amplify.state? I'll get to that today...

[hallow forge]

yes, to the amplify.state

[pliant rover]

I put 'npm install && echo 'function build a' into amplify.state.scripts.build

2023-09-07T15:34:34.349Z [WARNING]: ✖ There was an error initializing your environment.
2023-09-07T15:34:34.349Z [INFO]: 🛑 Received error [Error: Command failed with ENOENT: npm install && echo 'function build a'
spawn npm ENOENT] running command [npm install && echo 'function build a']
2023-09-07T15:34:34.349Z [INFO]: Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

oh maybe i'm missing a package.json in the directory

[hallow forge]

hey, can you remove the echo. It appears the echo may not work here. I ran into this as well. in my state file just the following worked and the builds was succesfull. have to dive into why this occurs.
for example

{
  "pluginId": "amplify-nodejs-function-runtime-provider",
  "functionRuntime": "nodejs",
  "useLegacyBuild": true,
  "defaultEditorFile": "src/index.js",
  "scripts": {
    "build": "pnpm install"
  }
}

[pliant rover]

2023-09-07T16:26:15.144Z [WARNING]: ✖ There was an error initializing your environment.
2023-09-07T16:26:15.145Z [INFO]: 🛑 Received error [Error: Command failed with ENOENT: npm install
spawn npm ENOENT] running command [npm install]
2023-09-07T16:26:15.145Z [INFO]: Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

[hallow forge]

interesting, is it ok i take a look at the function folder over a private DM?

[pliant rover]

sure

[bleak terrace]

Hey folks, since this thread has had a lot of comments I am going to briefly summarize our findings in this comment.

Pain Point

Amplify backend builds for standard and preview branches were failing with "Executing custom build scripts is not yet implemented".

Root Cause

The custom build feature was introduced by amplify cli v12.2.0:

• https://docs.amplify.aws/cli/function/build-options/
• https://github.com/aws-amplify/amplify-cli/releases/tag/v12.2.0

This particular feature does not seem to be backward compatible with older amplify cli versions.

The build container in an Amplify app uses amplify cli v10.5.2 and Node v16 by default. This can be overridden using live package updates feature by setting amplify-cli package to latest.

We identified this behavior as a bug where if you set the amplify-cli package version as latest the build environment continues to use v10.5.2 and this breaks the backend builds as we are not patching the amplify-cli version correctly.

Workaround
There are a couple of workarounds that you can leverage to unblock your builds:

Option 1

• Remove the node live package update
• Set the amplify-cli to latest

Option 2

• Remove the amplify-cli live package update
• Add the following as the first two commands in the preBuild section of the build spec:

rm -rf ~/.amplify
npm i -g @aws-amplify/cli@latest

[pliant rover]

Also, it looks like amplify.state's script.build field seems to always execute as a single command with arguments, not as a shell script.
So, if I have a build script "echo 'beginning some stuff' && npm whatever" it'll complain that echo can't find '&&'

[bleak terrace]

@pliant rover Do you mind creating a new post/thread describing the issue that you have been running into with the amplify.state build script field?

Since this seems like a different issue than the original one (backend build failing due to mismatch in amplify-cli version), I'd recommend tracking this in a separate thread.

cc @hallow forge

[pliant rover]

yeah it's a different issue, just wanted to make sure you caught it.

[hallow forge]

hey @pliant rover , I filed an issue about this behaviour here: https://github.com/aws-amplify/amplify-cli/issues/13216
the issue occurs if we add a command say echo(for example) after the build command which seems to be incorrectly passing && to the command.
but adding the echo command before the npm install builds as expected
for example echo 'hello world' && npm install
observed this on a local push as well

[pliant rover]

Oh you're right -- I forgot, the issue I ran into with putting echo first was that it couldn't find 'echo'

[bleak terrace]

Please refer to this GitHub issue to track further updates regarding the amplify-cli version mismatch in the build container: https://github.com/aws-amplify/amplify-hosting/issues/3670

[rotund ermine]

✅ - The PR previews stopped working overnight