Problems with FAS3240-Power on I/O expansion module is degraded: Sensor detecting voltage anomalies

[hasty geyser]

Hello everyone,

I am experiencing issues with a FAS3240 controller

Every now and then, we lose high availability and have to activate it manually. One of the logs that I'm not sure what it refers to is:

Power on I/O expansion module is degraded: Sensor detecting voltage anomalies.

I would appreciate it if you could help me with your experience.

info

netif.linkDown

Sun Mar 9 02:13:42 UYT

Ethernet c0a: Link down, check cable.

info

netif.linkDown

Sun Mar 9 02:13:42 UYT

Ethernet c0b: Link down, check cable.

info

ctrl.rdma.heartBeat

Sun Mar 9 02:13:42 UYT

High-availability interconnect status: Missed heartbeat to 192.xxx.x.xxx

info

ctrl.rdma.heartBeat

Sun Mar 9 02:13:42 UYT

High-availability interconnect status: Missed heartbeat to 192.xxx.x.xx

notice

cf.fsm.takeoverByPartnerDisabled

Sun Mar 9 02:13:42 UYT

Failover monitor: takeover of FAS3240-3A by FAS3240-3B disabled (interconnect error).

notice

cf.hwassist.takeoverTrapRecv

Sun Mar 9 02:13:44 UYT

hw_assist: Received takeover hw_assist alert from partner(FAS3240-3B), system_down because l2_watchdog_reset.

info

cf.fsm.stateTransit

Sun Mar 9 02:13:45 UYT

Failover monitor: UP --> TAKEOVER

notice

cf.fm.takeoverStarted

Sun Mar 9 02:13:45 UYT

Failover monitor: takeover started

notice

scsitarget.vtic.down

Sun Mar 9 02:13:47 UYT

The VTIC is down.

info

coredump.host.spare.none

Sun Mar 9 02:13:47 UYT

No sparecore disk was found for host 1.

info

raid.vol.replay.nvram

Sun Mar 9 02:13:47 UYT

Performing raid replay on volume(s)

notice

raid.replay.partner.nvram

Sun Mar 9 02:13:47 UYT

Replaying partner NVRAM.

info

raid.cksum.replay.summary

Sun Mar 9 02:13:47 UYT

Replayed 0 checksum blocks.

info

raid.stripe.replay.summary

Sun Mar 9 02:13:47 UYT

Replayed 0 stripes.

[cinder helm]

You’re running this past design life and it’s not out of the question that the IOXM is beginning to fail. This system is now at least 12 years old.

[tiny parcel]

Did you check c0a-c0b ports back of the controllers? seems that FAS3240-3A is having interconnect problems

[fickle stratus]

the interconnect is probably down because the other node panicked or otherwise failed, if only the vtic was down it wouldn't have to do a raid replay on the partner volumes. It's a full-blown takeover and without the (console) logs of the other node it's hard to tell what's wrong

[gray kraken]

Pretty sure it looks more like this... with a controller and the IOXM module installed, so two chassis for a HA... and it looke like the IOXM has some problems... coincidentally this is the only controller where I have ever replaced a faulty chassis... but in this case the system must have been running for about 10 years now, so cannot blame it for having issues I guess? One could always try to reseat the modules etc. 🙂