Trident 23.01 / OpenShift 4.11 - ontap-san multipathd not running

[fast bone]

After upgrading to Trident 23.01 from 22.01 we are unable to mount iscsi volumes.

The events for the PV shows that multipathd is not running.
{"@timestamp":"2023-03-14T19:30:39Z","level":"error","message":"GRPC error: rpc error: code = Internal desc = failed to stage volume: multipathd is not running","requestID":"a49fca6c-b2bc-4a24-8fa1-9abfff9477d0","requestSource":"CSI"}

We did not see any errors in the driver-registrar container, but do see the following in the trident-main container of the csi pods:

{"@timestamp":"2023-03-13T19:59:58Z","level":"error","message":"Failed to add portal  to self-healing map; err: portal value cannot be empty","requestID":"2cb64538-54e7-4d91-88a1-606db18cd5df","requestSource":"Internal"}```

We are unsure why it is seeing the portal list as empty now and is seeing multipathd as not running. (Is there anything specific that we need to do for OpenShift 4.11?)

[amber flower]

Hi,
did you modify the multipathing configuration according to ;
https://docs.netapp.com/us-en/trident/trident-get-started/requirements.html#critical-information-about-astra-trident-23-01

[fast bone]

Hi, I will give that a try.

Since we're using RHCOS it appears they do not include that (multipath.conf) by default. I'll create a machineconfig for it and get that rolled out to see if it works.

[fast bone]

Initial tests look good. I was able to mount the iscsi pvc as expected.

For those having a similar question for OCP, I loosely followed this kb https://access.redhat.com/solutions/5607891

kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: worker-iscsi-configuration
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,ZGVmYXVsdHMgewoJcG9sbGluZ19pbnRlcnZhbCAJMTAKCXBhdGhfc2VsZWN0b3IJCSJyb3VuZC1yb2JpbiAwIgoJcGF0aF9ncm91cGluZ19wb2xpY3kJbXVsdGlidXMKCXVpZF9hdHRyaWJ1dGUJCUlEX1NFUklBTAoJcHJpbwkJCWFsdWEKCXBhdGhfY2hlY2tlcgkJcmVhZHNlY3RvcjAKCXJyX21pbl9pbwkJMTAwCgltYXhfZmRzCQkJODE5MgoJcnJfd2VpZ2h0CQlwcmlvcml0aWVzCglmYWlsYmFjawkJaW1tZWRpYXRlCglub19wYXRoX3JldHJ5CQlmYWlsCgl1c2VyX2ZyaWVuZGx5X25hbWVzCXllcwoJZmluZF9tdWx0aXBhdGhzCQlubwp9Cg==
        filesystem: root
        mode: 420
        path: /etc/multipath.conf
    systemd:
      units:
      - name: iscsid.service
        enabled: true
      - name: multipathd.service
        enabled: true