#Deployment from local (mac) works but deployment from console fails

33 messages · Page 1 of 1 (latest)

delicate jasper
#

Hey,

So I've been building an app using sandbox and everything was working perfectly, so I decided to tear it down and deploy through the proper pipeline in the console.

But it failed...

I have a lambda function that is deployed with a bundled package (onnxruntime-node) and a lambda layer containing a small ml model. When I deploy it successfully from local, it has a package size of 75MB. But when deployed through the Amplify console, it errors complaining that the unzipped size is over the maximum of 250MB.

I've narrowed it down to the packaged onnx model specifically (when I remove it, the console deployment succeeds).

So... What gives? The only thing I can think of is that esbuild is packaging it differently on Amazon Linux compared to macOS... But I can't think of why...

Anyone come across this before?

hasty bear
#

https://github.com/aws-amplify/amplify-backend/issues/2539
did notice you did open a github issue as well. added a comment to the issue.
I would suggest forcing a docker build in your bundling option, this would help confirm if this is indeed installing some platform specific add-ons and keep consistency between the local and cicd build

GitHub

Environment information System: OS: macOS 15.1.1 CPU: (11) arm64 Apple M3 Pro Memory: 159.00 MB / 18.00 GB Shell: /bin/zsh Binaries: Node: 22.13.0 - /usr/local/bin/node Yarn: undefined - undefined ...

delicate jasper
#

I'll test out docker tomorrow morning to verify

#

I might also try a custom build in case the bindings are the problem

hasty bear
#

Just a initial thought from the doc did notice the linux x64 does support both CPU and CUDA while MacOS arm64 onyl supports CPU. the default image on the console uses amazon Linux which could be installing both
https://www.npmjs.com/package/onnxruntime-node
additionally did notice this information

To use CUDA EP, you need to install the CUDA EP binaries. By default, the CUDA EP binaries are installed automatically when you install the package.

delicate jasper
#

That could be the culprit, thanks, I'll take a look in the morning

delicate jasper
#

onnxruntime-node's module size when installed locally and when installed through the Amplify console are both 180MB

#

This is whether or not I enable the install flag that skips CUDA installation

#

Which is still weird... The function without the onnx module is 1.5MB, the onnx module is 180MB, so it should still be under 250MB even if it doesn't compress the package at all

delicate jasper
#

Can't even uninstall docker

delicate jasper
#

Same error of the lambda being too big

hasty bear
#

hey, with a local docker build i was able to reproduce this behavior. Amplify depends on CDK to bundle the resources and runs cdk deploy under the hood.
https://github.com/aws-amplify/amplify-backend/blob/main/packages/backend-deployer/src/cdk_deployer.ts#L35
with the following


const myLambda = new NodejsFunction(stk, "MyLambda", {
  runtime: Runtime.NODEJS_20_X,
  entry: handlerPath,
  architecture: Architecture.X86_64,
  environment: {
    ENV: "dev",
  },
  bundling: {
    nodeModules: ["onnxruntime-node"],
    forceDockerBundling: true,
    format: OutputFormat.ESM,
  },

  memorySize: 3000,
  timeout: Duration.seconds(900),
});

it does appear the package includes additional addOns as part of installation.
While the CUDA maybe the reason the for the size increase there may be other addons that the package installs as part of its installation. but do note the 250 MB limit also includes the Lambda layers being attached to the function.
to test this i removed the node module which deployed the function with unpackaged size of 1.7 MB
additionally, switching to architecture type to ARM_64 deploy successfully with docker indicating the there are addOn's being installed as part of x64
You could use a custom container image which does provide a quota upto 10 GB and customize the package installations.
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs.BundlingOptions.html

for additional, information you could also try reaching out on the CDK repository on the Lambda function constructs
https://github.com/aws/aws-cdk

GitHub

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code - aws/aws-cdk

GitHub

Home to all tools related to Amplify's code-first DX (Gen 2) for building fullstack apps on AWS - aws-amplify/amplify-backend

delicate jasper
hasty bear
#

Hey are you referring to the build image on Amplify hosting? Currently only supports images with x86-64 architecture.
The results provided are from local testing to emulate the bundling on different architectures and check the package size
To reduce the size of the packaging with onnxruntime-node the following are couple of options that may help with your use case

  1. As provided using a custom docker image with the function should provide you a higher limit of 10 GB
    https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.DockerImageFunction.html
  2. instead of the nodeJs function construct use the Lambda construct with custom commands in bundling options. I was able to deploy with following
new Function(stk, "say-hello", {
  handler: "index.handler",
  runtime: Runtime.NODEJS_22_X,
  timeout: Duration.seconds(20),
  architecture: Architecture.X86_64,
  code: Code.fromAsset(handlerPath, {
    bundling: {
      image: DockerImage.fromRegistry("dummy"),
      local: {
        tryBundle(outputDir: string) {
          // Create a package.json in the output directory
          execSync(`cp ${path.join(handlerPath, "package.json")} ${outputDir}`);

          // Install dependencies in the output directory
          execSync(
            `cd ${outputDir} && npm install onnxruntime-node --onnxruntime-node-install-cuda=skip`
          );

          // Copy the handler code
          execSync(`cp -r ${handlerPath}/* ${outputDir}`);
          return true;
        },
      },
    },
  }),
});

reference: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3_assets.AssetOptions.html
3. Use custom pipeline such as codebuild with Arm64 image for cdk to bundle with docker and deploy the backend
https://github.com/aws/aws-codebuild-docker-images

GitHub

Official AWS CodeBuild repository for managed Docker images http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html - aws/aws-codebuild-docker-images

delicate jasper
#

@hasty bear Thanks for the thorough investigation! I'll see about testing out some of these options tomorrow

#

Still haven't been able to find any reference to how big these extra architecture add-ons are though, but that's an issue with onnxruntime-node I guess

hasty bear
#

while i havent had a chance to look at the addOns with the option 2 did notice these without the CUDA

delicate jasper
#

Jeez this package is bloated

delicate jasper
delicate jasper
#

Specifically, it can't find my handler

#

It says it's looking for the handler at /var/runtime/index.mjs

hasty bear
#

Hey, as this a Lambda function with custom bundling. You would need to define the tsconfig, run the tsc command in the bundling. by default Lambda expects a index.mjs file as the handler file, you would need create a index.ts .

new Function(stk, "say-hello", {
  handler: "index.handler",
  runtime: Runtime.NODEJS_22_X,
  timeout: Duration.seconds(20),
  architecture: Architecture.X86_64,
  code: Code.fromAsset(handlerPath, {
    bundling: {
      image: DockerImage.fromRegistry("dummy"),
      local: {
        tryBundle(outputDir: string) {
          // Create a package.json in the output directory
          execSync(
            `cp ${path.join(handlerPath, "package.json")} ${outputDir}`,
            { stdio: "inherit" }
          );

          // Install dependencies including TypeScript
          execSync(
            `cd ${outputDir} && npm install onnxruntime-node typescript @types/node`,
            { stdio: "inherit" }
          );

          // Copy the handler code and tsconfig
          execSync(`cp -r ${handlerPath}/* ${outputDir}`, { stdio: "inherit" });

          // Create a minimal tsconfig.json if it doesn't exist
          const tsConfig = {
            compilerOptions: {
              target: "es2022",
              module: "node16",
              moduleResolution: "node16",
              esModuleInterop: true,
              strict: true,
              outDir: "dist",
            },
            include: ["*.ts"],
            exclude: ["node_modules"],
          };

          fs.writeFileSync(
            path.join(outputDir, "tsconfig.json"),
            JSON.stringify(tsConfig, null, 2)
          );

          // Transpile TypeScript
          execSync(`cd ${outputDir} && npx tsc`, { stdio: "inherit" });

          // Move compiled files to root and cleanup
          execSync(`cd ${outputDir} && mv dist/* . && rm -rf dist`, {
            stdio: "inherit",
          });

          return true;
        },
      },
    },
  }),
});

While i am defining all in the bundling, you can create a folder with the required files on lambda function then define the compilation steps in the bundling
https://github.com/aws-samples/cdk-lambda-bundling/blob/main/lib/cdk-bundling-lambda-stack.ts
https://docs.aws.amazon.com/lambda/latest/dg/lambda-cdk-tutorial.html

wanted to quickly understand your use case. is there a requirement to use onnxruntime-node in your function?

GitHub

Contribute to aws-samples/cdk-lambda-bundling development by creating an account on GitHub.

delicate jasper
#

And when I re-deploy it with the compile command, it still can't find the generated index file

#

I think I'm gonna come back to this fresh on Monday

delicate jasper
delicate jasper
hasty bear
#

hey are you seeing an errors? how is it currently setup now?