vast sapphire
I'm experiencing a similar issue to the one mentioned here:
https://discord.com/channels/705853757799399426/1169393091275399238
My stack is currently stuck in UPDATE_ROLLBACK_FAILED, and continuing the rollback doesn't resolve it. It seems to be a problem with the ConnectionStack, but I'm not sure how to resolve it and I'm wondering if there's a better solution than blowing away my entire stack and having to rebuild all that data.
limpid kraken
Hey @vast sapphire
Not sure how familiar you are with CloudFormation, so excuse me if coming across the wrong way.
A recoverable situation is one where drift was introduced. Drift is just a change was made to the underlying physical resource outside of CloudFormation. The path forward is to change the physical resource back to what CloudFormation expects it to be.
Documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-update-rollback-failed
Rarely, the situation is irrecoverable - but it has happened to me before. In those cases, we change the deletion policy, skip the resources, and re-create the stack with an import OR we prepare an adjacent stack. Neither is particularly pleasant, especially if dealing with a production system.
vast sapphire
Hmmm looks like it's saying a few of the app sync resolvers for specific relationships aren't found. There's only three of them so I should be able to just manually create those right? I'm just not sure how to do that. Is there an AWS CLI command I can use to copy the config of one of the others and then tweak it to make the missing ones?
Or worst case, I can probably create manual backups of my tables, then do amplify api rebuild then restore the tables, no?
limpid kraken
aws appsync help will list out relevant commands - sounds like you may want aws appsync create-resolver help
One thing I’d he curious of is how exactly resolvers got deleted. I’d check audit logs in CloudWatch if you have them
But yeah worst case you could do a DB dumb and restore, but that’s the nuclear option. Many things can go wrong in those types of operations, so to the degree you can, I’d try to fix the api layer connections
vast sapphire
It's looking like I'm gonna have to blow it away and start over. I've got the DB backups going. How can I protect the s3 buckets and Cognito users?
vast sapphire
Hmmm I'm making some progress, but I'm currently stuck on the API stack. It says one of the types (ChatMessage) is failing because of this error:
Output 'transformerrootstackChatChatClientOwnerDataResolverFnChatClientOwnerDataResolverFnAppSyncFunctionBE26EA1AFunctionId' not found in stack 'arn:aws:cloudformation:us-east-2:148371460060:stack/amplify-documentdashboard-dev-135217-apidocumentdashboard-UB2QVD6SPI69-Chat-HLCZAOSCQ0T7/4b1e2dd0-8352-11ee-a415-0a9b7927b467
I fixed a few other issues, but this one has a lot going on and I can't make heads or tails of it
midnight geyser
Same error here!
vast sapphire
More progress. Turns out, the cloudformation template I was uploading to try and get it to update the stack was referencing a specific "S3DeploymentRootKey" that was outdated. Its parts of the s3 key used to find the templates to build for all the nested stacks. Each attempt makes a new one of these, but once I update it to the most recent one, I'm getting new errors
vast sapphire
for posterity, some of the ones its using "statically" are located at s3://amplify-your-stuff-here-deployment/amplify-cfn-templates. Some of those templates are relying on parameters provided by the top level template. Those parameters are telling it where the s3 objects are for the templates of all the other resources. That pattern looks something like this: s3://amplify-your-stuff-here-deployment/amplify-appsync-files/fccfe9887a36432985248e6616b1c62e94a77c5e/. These also depend on parameters that are set by the other templates. But that fccfe9887a36432985248e6616b1c62e94a77c5e was being set in the top level template.
vast sapphire
small update: It's not using the updated version of the templates I uploaded to s3://amplify-your-stuff-here-deployment/amplify-cfn-templates, but I got it to a point that I think I can work with
vast sapphire
Looks like I may have made some progress. I ran into a snag with lambda layers where I just ended up deleting the python one and tweaking the permissions and dependencies for the node one just so it would trigger a rebuild of those. I also had to delete the deployment state json from the bucket and force push my changes from my local machine (with the "allow destructive schema changes" flag). This post was very handy https://stackoverflow.com/questions/58168051/aws-amplify-resource-is-not-in-the-state-stackupdatecomplete
Still not out of the woods yet though
midnight geyser
Do you know why the deployment state file does not exist in the bucket? it is my case, I read about this solution but this file does not exist in my deployment bucket