Template execution stuck

[trail smelt]

Guys I'm having an issue with my production Backstage, and only there, the QA and DEV are fine. And this is on ARO ( Azure RedHat Openshift)

I can create with templates but then after a while it gets stuck in the creation of another.
It doesn't even start the debug one. None of the 5 templates works after this happens

I give you some data

Request URL: https://mydomain/api/scaffolder/v2/tasks/16b66ac7-7b0f-4525-901f-510836f83269/eventstream
Status: 504 Gateway Time-out
Content-Type: text/html
Remote Address: 10.xx.xx.xx:443

Backstage v1.45.2

I'll be glad to know any ideas or comments, thanks

[shell fjord]

Hi maybe you can find more content in this issue here https://github.com/backstage/backstage/issues/27217
But issue has also happened to me but updating @backstage/plugin-scaffolder-backend to "^2.1.0", at leas i can continue to run other templates and the template that get stack after some time it will be canceled

a temporary fix would be to manually restart the pod on prod

this is very hard if i can say imposible to replicate locally but based on my experience with this error i suggest to check if you are using multiple version of @backstage/plugin-scaffolder-backend and make sure to be the latest one compatible with the backstage version you have

[trail smelt]

@backstage/plugin-scaffolder-backend 3.0.1, 1.33.0

I just realized I have 2 versions, maybe that's it. The weird thing is why QA and DEV environments this issue never show up ( they are the same build as PRD)

[shell fjord]

i know i know trust me its crazy but when updating the @backstage/plugin-scaffolder-backend to the latest version this error still ocures but this time has a better handeling because you can still you can run other templates and the template trigered that got stack automatically get canceled in the db

as i said for the moment the only easy fix to let prod template work again is to restart the pod manually and then take care of updating to the latest version of @backstage/plugin-scaffolder-backend

in your case make sure to have only @backstage/plugin-scaffolder-backend 3.0.1,

Also one last thing to avoid this errors with package version i would suggest to dedicate sometime to configure your backstage instace to use Backstage Yarn plugin to avoid such situations in the future for all other backstage packages not only the scafolder
https://backstage.io/docs/getting-started/keeping-backstage-updated/#managing-package-versions-with-the-backstage-yarn-plugin