Multi Cluster - Single PVC Support (RWX)

[noble mirage]

Hi,

I stumbled across the following article to support the provisioning of a single RWX volume, that is accessed by 2 different cluster instances (both of them running the trident operator): https://netapp.io/2022/03/31/one-pvc-and-two-kubernetes-clusters-with-netapp-astra-trident/. Everything worked out without any flaws as defined in the article.

Unfortunately I am not sure on how to proceed in a use case, where the "master / manager / owner" cluster is not reachable and I want to perform operations (such as changing volume size, creating snapshots, ...) in the left over cluster, that has imported the volume with the --no-manage attribute.

• Do I need to re-import the volume without the flag?
• Do I need to delete the volume upfront for this (leading to a downtime)?
• In addition how do I recover from the scenario, when the master cluster comes back online and both are trying to "manage" the volume?

I know this is a quite complicated topic, but I just want to assure that these scenarios are handled correctly.

Thanks and BR,

Philipp

[pearl storm]

Hi Philipp - I wrote that article, and you are correct that these are complicated scenarios. If you want to manage the PVC, you would need to disconnect it and re-import with without nomanage. This will cause a problem with your master cluster though, and you would need to re-import it back into that cluster if you want to manage it from that one down the road. You can only manage the volume from one cluster at a time.

However you want to tweak the scenarios, you should definitely test it out in a lab setting and make sure that you understand the steps and what the result will be. That is what I did for the specific scenarios in this article, so any change to the process should involve testing for that change.

[noble mirage]

Hi Michael,

thanks for your input. To sum it up for others and ask some side questions:

• Do I need to re-import the volume without the flag?
  Yes
• Do I need to delete the PVC upfront for this (leading to a downtime)?
  I understood that we need to disconnect the volume, which means to shut down the pod consuming the PVCs. Do I also need to delete the PVC and PV?
• In addition how do I recover from the scenario, when the master cluster comes back online and both are trying to "manage" the volume?
  What happens when the old "master" comes back online and both are trying to manage the volume?
  From what I understood the old master will not find it's volume (as it was renamed by the new cluster). Will it go into an error state or provision a new volume (as it is now missing). The necessity for re-importing the volume in un-managed state on the old "master" cluster is clear to me (as we now have a new "master"). But this needs to be done once the cluster is back online and already doing it's thing :).

We are anyhow planning to test these scenarios, I just would like to understand what will happen under the hood here.

Thanks and BR,

Philipp

[pearl storm]

Hey Philipp - there would be no need to delete the actual data volume on the NetApp storage system, but you likely need to remove the PV/PVC from being visible on the K8s cluster if you want to import it again without nomanage.

You are right that the old master then won't find this PVC when it comes back online, and you will need to import on the old master (with or without nomanage depending on which K8s cluster you want to end up owning the management of the PVC).

These are my educated assumptions at how this will work, but again I have not tested these specific scenarios and you will want to verify by testing with non-production datasets.