ENABLE_SOROBAN_DIAGNOSTIC_EVENTS parameter on validators

[heady bone]

My colleague stumbled on a controversial information when looking at ENABLE_SOROBAN_DIAGNOSTIC_EVENTS param here:
https://github.com/stellar/stellar-core/blob/40a02cf03d603b013a7dc5c956af8c1cba4f5f96/src/main/Config.h#L566

... The default value is false, and this should not be enabled on validators.
Does it mean that we can enable this options on our validator nodes, but it will result in higher resources utilization? Or we shouldn't enable it under any circumstances because our nodes will fall out of consensus?

We heavily rely on diagnostic events internally.

[heady bone]

We are getting error on our validator nodes

2024-02-20T15:56:44.886 GA23K [default FATAL] Got an exception: ENABLE_SOROBAN_DIAGNOSTIC_EVENTS is set, NODE_IS_VALIDATOR is set, and RUN_STANDALONE is not set
2024-02-20T15:56:44.886 GA23K [default FATAL] Please report this bug along with this log file if this was not expected

Are ENABLE_SOROBAN_DIAGNOSTIC_EVENTS and NODE_IS_VALIDATOR mutually exclusive?

[tidal gorge]

yes

diagnostic events run in a secondary budget that is not the one used for fees (i.e. turning diagnostic events on is an indication that you want to let ledgers close slower than usual) and that's prohibited on a validator, only allowed on watchers

to avoid validators falling behind