how to add a prefix filter on an s3 trigger lambda | AWS Amplify | Page 1

mortal mountain Jun 28, 2023, 11:18 PM

#

i created added storage to my amplify app and chose the option that generates a lambda trigger.
it's all working fine, the lambda is called when i add a file to the s3 bucket.

but i was hoping to set a filter on the trigger so my lambda only gets called when files appear in the "/inbox" folder, for example.

do i have to manually edit the amplify/backend/storage/myname/build/cloudformation-template.json?

is there a way to do it in the console and pull it down?

it seems there needs to be a Filter.S3Key.Rules { Name: "prefix", Value: "inbox/" } somewhere in there...

mortal mountain Jun 28, 2023, 11:45 PM

#

i was able to add this rule in the console, but it seems wrong to not have amplify handle this?

twin marsh Jun 29, 2023, 3:15 PM

#

@mortal mountain if you added it via the "aws console" and not within "amplify studio" then you've made changes outside your project which could potentially be clobbered by a future push and/or won't apply to the bucket when you deploy into a new environment. You can use a tool like "Former2.com" (or dig into the AWS docs) to see the CloudFormation changes that were applied to define the filter. It can also generate CDK snippets for you.

#

Next, you inject the config you need as CDK code by adding an amplify/backend/storage/<resource>/override.js file as described here: https://docs.amplify.aws/cli/storage/override/#customize-amplify-generated-s3-resources

mortal mountain Jun 29, 2023, 3:18 PM

#

the issue is that the storage folder created by amplify cli does not have a cloudformation-template for me to edit.

twin marsh Jun 29, 2023, 3:20 PM

#

Right. I had the same issue for a rest-api. That override.js file should do the trick.

mortal mountain Jun 29, 2023, 3:20 PM

#

is there seriously no way within the amplify tooling to do something as simple as setting a prefix filter?

#

i appreciate the pointers, and will for sure do it... i'm just constantly amazed how amplify does things in a certain way, and everything else is left up to these manual methods... it's seems like it could do more.

twin marsh Jun 29, 2023, 3:24 PM

#

Yeah. It was stupid easy to make the trigger, and if that's all you wanted it would have saved you a ton of time. But if you want a trigger + filter, you have to learn all the details of how to set that up, but then ALSO learn how to cram it into Amplify. It ends up being more work than if you just used CDK or SAM or similar. Enjoy the journey I guess. It's a grind.

mortal mountain Jun 29, 2023, 3:28 PM

#

yeah... i came from SAM to amplify and have been generally disappointed... they make the super-vanilla-demo-app that no one will ever actually use, SUPER EASY to deploy and build and demo... and everything else you want to do makes you wish you didn't adopt amplify.

#

do you recommend this former chrome extension? sounds like its necessary for s3 config

twin marsh Jun 29, 2023, 3:33 PM

#

"recommend" is a strong word. I sometimes use it so I can export a resource, make a change in the console, export the updated resource and see what changed. That can be a shortcut vs. wading through all the documentation without knowing exactly what you're looking for. I'm not sure i'd leave it installed after you're done using it.

#

I wish AWS has "export to cloudformation" buttons in the console. Former2 is the next best thing to that.

mortal mountain Jun 29, 2023, 3:35 PM

#

got it. now i just need to find a credential that will work...

twin marsh Jun 29, 2023, 3:38 PM

#

If you're using SSO (recommended) I think you'll make a 'Permission Set' called Former2 (or whatever you like) and then associate it with a user. Otherwise, make a new 'former2' api-only user.

mortal mountain Jun 29, 2023, 3:38 PM

#

i sorted it...

#

ok, there's the yaml

#

you said i can get it to generate that override cdk code?

#

i don't get it... the code generated doesn't look like an override?

#

i guess i don't know what to do with the code generated by this tool... how can i use the cf yaml to make amplify know about my filter?

mortal mountain Jun 29, 2023, 3:59 PM

#

wow... and what's the story with the generated function being in javascript and you have to do all sorts of manual steps to convert to typescript, but this amplify override storage command generates typescript... in a completely different way that the recommended lambda way?

twin marsh Jun 29, 2023, 4:05 PM

#

So, I just updated my config and did a "scan". Then found "S3" and selected my bucket. I click "Add selected" button. Next you click the "Generate" button. And then you can pick the "output" you want from the side-bar (e.g. CDK (V2)).

#

The catch is, that's the code to generate the resource, but Amplify already does that. So you really just want to look to see what resource and/or which attributes of that resource you need to modify to apply the filter. Then craft CDK code to just do that bit.

#

For example, here's my override.js that modifys my rest-api:

Object.defineProperty(exports, "__esModule", { value: true });
exports.override = void 0;
function override(resources) {
    resources.restApi.description = "PixelsToPostcards 3rd Party API";
    resources.restApi.apiKeySourceType = "HEADER";
    resources.restApi.body.securityDefinitions = Object.assign(Object.assign({}, resources.restApi.body.securityDefinitions), { "require-api-key-in-header": {
            "type": "apiKey",
            "name": "x-api-key",
            "in": "header",
            "x-amazon-apigateway-api-key-source": "HEADER",
        } });
    const custSecurityRules = [{ "require-api-key-in-header": [] }];
    Object.values(resources.restApi.body.paths).forEach((path) => {
        path.options.security = custSecurityRules;
        path['x-amazon-apigateway-any-method'].security = custSecurityRules;
    });
}
exports.override = override;

mortal mountain Jun 29, 2023, 4:22 PM

#

you just dropped that in by hand? no amplify override?

twin marsh Jun 29, 2023, 4:23 PM

#

That is my override.js file.

mortal mountain Jun 29, 2023, 4:23 PM

#

understood... i'm just grappling with the typescript setup that amplify override storage just dumped into my project...

twin marsh Jun 29, 2023, 4:24 PM

#

So I think you want to add overrice.js to /amplify/backend/storage/<yourbucket/override.js

#

And then you want to add a filter... So something like

mortal mountain Jun 29, 2023, 4:24 PM

#

i don't think they've really thought through how typescript works in all of this...

twin marsh Jun 29, 2023, 4:25 PM

#

`resources.bucketName.NotificationConfiguration.LambdaConfigurations.Filter = "????"

#

For this snippet, I'd stay away from TS.

#

console.log resources.yourbucket to see the code executes and you are looking at your resource. Then drill down until you can set the Filter.

#

Otherers may have a better way. I just baby-stepped it. Got it working and ran away. Maybe once it's working, update this issue for the next person: https://github.com/aws-amplify/amplify-cli/issues/2268

GitHub

Allow storage trigger filters · Issue #2268 · aws-amplify/amplify-c...

Is your feature request related to a problem? Please describe. Storage Triggers currently invoke the Lambda function on any ObjectCreated or ObjectRemoved event. This means that the function will b...

mortal mountain Jun 29, 2023, 4:30 PM

#

having that issue sit open for four years feels pretty typical of this package... :/

#

maybe i can ask about this on office hours today...

twin marsh Jun 29, 2023, 4:33 PM

#

In 30 minutes they have "office hours" when you can talk to the team. However, they'll probably just point you to the 'override.js'. From there, it's "just cdk".

mortal mountain Jun 29, 2023, 4:33 PM

#

an "override" seems like an odd approach vs just configuring the resources correctly in the first place.

#

"here's our super helpful cli tool that made a bunch of random resource configuration choices for you that you don't want, and our solution is... we install a poorly configured typescript config into your project and tell you to undo what our handy tool did with an override!"

twin marsh Jun 29, 2023, 4:41 PM

#

import { AmplifyS3ResourceTemplate } from '@aws-amplify/cli-extensibility-helper';

export function override(resources: AmplifyS3ResourceTemplate) {
  resources.s3Bucket.notificationConfiguration.lambdaConfigurations.filter = "????"
}

#

Getting closer. Jus have some squiggles under lambdaConfigurations, not sure why.

twin marsh Jun 29, 2023, 5:34 PM

#

import { AmplifyProjectInfo, AmplifyS3ResourceTemplate } from '@aws-amplify/cli-extensibility-helper'

export function override(resources: AmplifyS3ResourceTemplate, amplifyProjectInfo: AmplifyProjectInfo) {
  console.log('\n\n\n')
  console.log('\n\n\n')
  console.log('amplifyProjectInfo:')
  console.log(JSON.stringify(amplifyProjectInfo, null, 2))
  resources.s3Bucket.notificationConfiguration = {
    lambdaConfigurations: [
      {
        event: 's3:ObjectCreated:*',
        function: "arn:aws:lambda:us-east-1:123456789012:function:my-function",
        filter: {
          s3Key: {
            rules: [
              {
                name: 'prefix',
                value: 'your-prefix'
              },
              {
                name: 'suffix',
                value: '.txt'
              }
            ]
          }
        }
      }
    ]
  }
  console.log('\n\n\n')
  console.log('\n\n\n')
}

#

Then amplify build to test and amplify push to see if it updates the resource properly. Not sure how to get the function arn as a property in the override.ts file.

#how to add a prefix filter on an s3 trigger lambda