404 being returned from Astra web console after upgrading ACC from 22.08 to 22.11

[ocean sail]

Hello,

We are currently running an ACC PoC on premises using an OKD/OpenShift 4.10 cluster. We wanted to test the upgrade process, so we started by installing ACC 22.08, and then upgraded to ACC 22.11 afterwards. When we first installed, we set telemetry to false, but the installer still deployed the telemetry pod. The telemetry pod went into a crash loop, but Astra was working correctly. Our assumption was that the telemetry pod was in a crash loop because our firewall was blocking the pod from communicating to anything outside our network. When we upgraded ACC the telemetry pod was deployed again, but some new additional related pods were deployed, and also went into a crash loop (metrics-facade, public-metrics, telemetry-service). Post upgrade when we hit the ACC route we receive a '404 page not found'. The route we are using looks correct, normally we receive a different message if the route is pointing at an unavailable pod. There is some speculation that this is occurring because the telemetry service is not healthy. We are using a temporary license for testing purposes. I am looking for feedback on how to resolve the issue.

[dense swan]

Hey clutch, @tiny mantle may be your best bet here, but he is in the UK and it's a little late there. Hang tight and we'll get you an assist.

[ocean sail]

Hey @dense swan, that's not a problem, it's late here now also. I am mountain time. I don't mind if the response isn't immediate. Thanks.

[tiny mantle]

Hey, sorry for the delayed response, busy start to the week. Do you have any logs or information from the failing containers?

When you say you set telemetry to false, where was this done? In the deployment customisation?

We don't require internet access, it just limits some functionality, it shouldn't stop the pods from starting so will be interested in any reasons for the failure. https://docs.netapp.com/us-en/astra-control-center/get-started/requirements.html#access-to-the-internet

[tiny mantle]

Thanks I have them, I've removed them for you now I have them.

[tiny mantle]

It looks like the upgrade hasn’t completed correctly, that’s likely why the 404. The failure with the telemetry service looks to have carried over into the upgrade. I need to dig a bit more in to that, I’ll speak to a few people later today when the US folks come online

Are you working with anyone from NetApp on the PoC?

[ocean sail]

Hey Jason, this was actually the same conclusion we came to after reviewing the logs. We have an account with NetApp, but it was recommended I try reaching out here by my team because we have had some good experiences in the past using the discord server.

[tiny mantle]

That's fine thanks, I just wanted to make sure I wasn't skipping anyone that was initially involved. I have raised this with engineering as we have a similar report from another customer as well. I will keep you updated.

[ocean sail]

Glad to hear it's not just us. Thanks for removing the attachment as well.