AIQUM performance counter collection fails (9.14) | NetApp | Page 1

cyan frigate May 25, 2024, 12:00 PM

#

Hi All,

we're trying to build a lab with ONTAP 9.14.1p4 and AIQUM 9.14 (vSims & vApp).
Discovery works fine, health data is found, performance data is missing.

2024-05-25 08:55:38,369 INFO [JmsSession [ID:5ccbf921-38ee-43b2-a0b4-8d46af8f86df:1:1] delivery dispatcher] c.n.a.d.a.DataCollectorConnection (DataCollectorConnection.java:172) - clusterUuid=0b3f2b34-18e5-11ef-ab73-005056990101, action=COUNTERS_REQUEST_STATUS_FAILURE, payloadDir=null, success=FAILURE
2024-05-25 08:55:38,369 INFO [JmsSession [ID:5ccbf921-38ee-43b2-a0b4-8d46af8f86df:1:1] delivery dispatcher] c.n.a.d.a.DataCollectorConnection (DataCollectorConnection.java:147) - DataCollectorConnection:signal() clusterUuid = 0b3f2b34-18e5-11ef-ab73-005056990101, type = COUNTERS_REQUEST_STATUS_FAILURE
2024-05-25 08:55:38,369 INFO [JmsSession [ID:5ccbf921-38ee-43b2-a0b4-8d46af8f86df:1:1] delivery dispatcher] c.n.a.d.a.DataCollectorConnection (DataCollectorConnection.java:152) - key=0b3f2b34-18e5-11ef-ab73-005056990101?type=COUNTERS_REQUEST_STATUS_FAILURE
2024-05-25 08:55:38,369 INFO [async-perf-pool-1] c.n.a.d.a.DataCollectorConnection (DataCollectorConnection.java:133) - DataCollectorConnection: wait() 0b3f2b34-18e5-11ef-ab73-005056990101, removing message from messageMap
2024-05-25 08:55:38,370 INFO [async-perf-pool-1] c.n.a.d.a.DataCollectorTrigger (DataCollectorTrigger.java:300) - Successfully ask for counters from cluster: 0b3f2b34-18e5-11ef-ab73-005056990101

I was already going through the knowledge base, tried all sorts of workarounds (e.g. installing AIQUM on our "DC" (Windows) - that's where the log is taken from). No good. No performance data showing up.

Any ideas?

robust basin May 25, 2024, 5:35 PM

#

That looks like an issue with the cloud agent connection, but I haven't seen that issue myself in my own lab. Looks new.

does it work if you locate server.properties on the um server and change the cloudagent option to false? You'll need to restart the services too. I don't exactly recall where that file is in windows, so you'd need to search for it.

if you snapshot the um vm before making that change, or are fine with removing and readding the clusters in your lab (this will remove all historical data so it's not recommended for normal troubleshooting) you can create a case and we'll did further into the issue and possibly create a bug as well. Ping @urban cloud or I with the case number on Monday/Tuesday if you do create one.

cyan frigate May 27, 2024, 9:06 AM

#

I disabled "Active IQ Portal Events" in AIQUM.
That's the only thing I saw, that could use the "Cloud Agent Connection"...

We're completely self-sufficient here, it's just a local lab, there should not be any cloud connectivity. The cluster is "right next" to AIQUM (same subnet). Still performance data collection fails.

#

All certificates are also up-to-date...

#

Re-Started the services, too

robust basin May 27, 2024, 6:39 PM

#

with aiqum 9.14 and an ontap version able to do the agent connection (anything after 9.9, but um just does this with 9.14 i believe).

this ' cloud agent connection' is not in the gui, and is not related to the aiqum portal events. you'll only be able to stop using it by disabling it in that file i mentioned, since it does this on the backend.

it is functionally similar to the defunct CI data collector that use cloud agent connections too, if you used CI a couple of years ago. calling it a cloud agent is a leftover from that time period - sorry for the confusion. there is no actual cloud connectivity in the connection between um and the cluster.

you can check the status of this connection on the cluster side by going into advanced mode and running the commands:

::> cluster agent connection show
::> cluster agent connection show -instance

C:\Program Files\NetApp\essentials\conf would be the default location for server.properties in windows.

cyan frigate May 28, 2024, 8:26 AM

#

I'll double-check later. I'm supposed to get a fresh lab instance today again (workshop finished successfully yesterday).

Our current suspicion is ONTAPI vs RESTful API being switched off by default in a fresh 9.14 install.
I'll try "system services web ontapi modify -suspended false" to check on that first...

If that doesn't work, I'll try your suggestion.
(AIQUM is originally installed as a VM "next to" the DC I'm mainly using as my access point in the lab. Meaning access to the AIQUM innards is tricky, since I can only SSH into it (no access to the VMware console). I'll have to do a parallel Windows install and only then I'll have access to configuration & log files...)

cyan frigate May 28, 2024, 9:37 AM

#

--> Results (preliminary)

Against all info in e.g. https://kb.netapp.com/on-prem/ontap/DM/REST-API/REST_API_KBs/FAQs_on_ZAPI_to_ONTAP_REST_API_transformation_for_CPC_(Customer_Product_Communiques)_notification

ONTAPI is not switched off by default in our fresh 9.14.1 (upgraded to 9.14.1P4) Sim Install:

`cluster1::system services web ontapi> show
ONTAPI Suspended

false
cluster1::system services web ontapi*> mod -suspended true

Error: command failed: ONTAPI cannot be manually suspended. Use the "vserver
services web" CLI command to disable the ONTAPI service.

cluster1::vserver services web*> show -name ontapi
Vserver Type Service Name Description Enabled

NAS_svm data ontapi Remote Administrative API true
Support
SAN_svm data ontapi Remote Administrative API true
Support
cluster1 admin ontapi Remote Administrative API true
Support
3 entries were displayed.`

#

--> Results (Cluster Agent, ONTAP side)

`cluster1::*> cluster agent conn show -instance

                                          Name: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3
                                          UUID: 2d55fd05-1c07-11ef-99af-005056990101
                               Destination URL: amqpwss://aiqum.demo.netapp.com:56443
                                         State: connected
                                     Use Proxy: false
                             Subscribe Address: ontap.agent.manager
                               Publish Address: ontap.agent.cluster
                         Auth Certificate UUID: f6d49e90-19c3-11ef-8bcf-005056990101
                         Auth Certificate Name: -
                                  Auth CSR URL: -
                                 Manager Token: -
                                      Username: clus-agent-WZac
                                          Role: readonly
                        IPspace for Connection: Default
                               Last Error Code: 138281000
                            Last Error Message: AMQP transport failed for connection "UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3". Reason: Error during websocket handshake: conn fail: 61
                               Last Error Time: 5/28/2024 08:44:03
                          Validate Certificate: true
                  Initial Reconnect Delay Msec: 10
             Reconnect Backoff Multiplier Msec: 2
         Maximum Delay between Reconnects Msec: 300000
  Time Connection Must Remain Established Secs: 5
                   Accept Self-signed SSL Cert: false

...`

I'll switch off certificate checking and check the results...

cyan frigate May 28, 2024, 2:52 PM

#

--> Results (Cluster Agent)

Removed Cluster from AIQUM, Cluster Agent connection was gone, re-added the cluster to AIQUM, switched certificate checking off:

´cluster1::cluster agent*> conn mod -validate-certificate false -ssl-allow-self-signed true -name *
1 entry was modified.´

Now there's no more errors in the cluster agent connection... OK, but

#

`cluster1::cluster agent*> con show -instance

                                          Name: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3
                                          UUID: 0c4e1076-1cd9-11ef-bae5-005056990101
                               Destination URL: amqpwss://aiqum.demo.netapp.com:56443
                                         State: connected
                                     Use Proxy: false
                             Subscribe Address: ontap.agent.manager
                               Publish Address: ontap.agent.cluster
                         Auth Certificate UUID: 0ac9bc75-1cd9-11ef-bae5-005056990101
                         Auth Certificate Name: -
                                  Auth CSR URL: -
                                 Manager Token: -
                                      Username: clus-agent-YmYl
                                          Role: readonly
                        IPspace for Connection: Default
                               Last Error Code: -
                            Last Error Message: -
                               Last Error Time: -
                          Validate Certificate: false
                  Initial Reconnect Delay Msec: 10

...
Accept Self-signed SSL Cert: true
...
Total Messages Received from Manager: 509
Total Manifest Messages Received from Manager: 1
Total baseline.request Messages Received from Manager: 20
Total counters.request Messages Received from Manager: 488
Total Connection Status Messages Received from Manager: 0
Total Connection Modify Messages Received from Manager: 0
... Total Messages Sent to Manager: 510
Total manifest.request Messages Sent to Manager: 1
Total Baseline Messages Sent to Manager: 20
Total Counters Messages Sent to Manager: 0
Total Connection Status Messages Sent to Manager: 1
`

#

looks like performance counters are still not being sent.
I'll create some traffic and report further...

cyan frigate May 28, 2024, 5:42 PM

#

--> Result: AIQUM seems to send ONTAPI (!) requests for counters. ONTAP just does not send them, but gives an unspecified error ("Error Message count"), but without a clear message ("Last Error Message"). AIQUM seems to be OK (it's asking), but some ONTAP setting is missing?? (Counter Schedule State: inactive??) - but baseline is inactive also and is being sent...

#

`
cluster1::cluster agent connection*> show -instance

                                          Name: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3
                                          UUID: 0c4e1076-1cd9-11ef-bae5-005056990101
                               Destination URL: amqpwss://aiqum.demo.netapp.com:56443
                                         State: connected

...
Username: clus-agent-YmYl
Role: readonly
IPspace for Connection: Default
Last Error Code: -
Last Error Message: -
Last Error Time: -
Validate Certificate: false
Accept Self-signed SSL Cert: true
Node Owning the Connection: cluster1-01
Total Messages Received from Manager: 817
Total Manifest Messages Received from Manager: 1
Total baseline.request Messages Received from Manager: 32
Total counters.request Messages Received from Manager: 784
Total Connection Status Messages Received from Manager: 0
Total Connection Modify Messages Received from Manager: 0
Total Restapi Messages Received from Manager: 0
Total agent.status.request Messages Received from Manager: 0
Total Subscription Create Messages Received from Manager: 0
Total Subscription Delete Messages Received from Manager: 0
Total Unknown Messages Received from Manager: 0
`

#

` Total Messages Sent to Manager: 818
Total manifest.request Messages Sent to Manager: 1
Total Baseline Messages Sent to Manager: 32
Total Counters Messages Sent to Manager: 0
Total Connection Status Messages Sent to Manager: 1
Total Restapi Status Messages with Success HTTP Responses Sent to Manager: 0

          Total Error Messages Sent to Manager: 783

Total Errors Encountered Attempting to Send/receive: 0
Total ONTAP PubSub Topic Messages Sent to Manager: 0
Total agent.status Messages Sent to Manager: 0
Client Application: Unified Manager
App URL: -
Last Connection Established Time: 5/28/2024 10:00:01
Baseline Schedule State: inactive
Counter Schedule State: inactive
Override Scheduled Collection of Baseline Data: false
Override Scheduled Collection of Counter Data: false
Keep the Agent Alive: -
`

robust basin May 28, 2024, 8:06 PM

#

so this is where we get into the weeds. we in support don't yet have a whole lot of information available on this connection type to do a lot of troubleshooting, other than 'turn it off'. To dig much further, i'd need a case I could send to engineering.
i do not recall how long the baseline took. I'd expect a schedule to be there, but i assume the baseline has to complete first. i also am not sure how UM's implementation of this connection type differs from CI's yet. it's too new.

i'd expect these counters requests to be REST based for the cloud agent, since that was the original intent of the connection type. but if it's not working, UM may be falling back to the default ontapi method to collect the ccma files, and certainly will be using ontapi for collecting the ccma files for performance if we turn it off.

you could try checking the cluster logs for these keywords and see if they shed any light on the issue.
EMS messages :
• mhost.ca.connect.failure
• mhost.ca.connect.cert.error
• mhost.ca.connect.success
• mhost.ca.connect.delete
• mhost.ca.msg.delivery.fail
• mhost.ca.rehost.success
• mhost.ca.node.offline

MGWD messages:
• amqp_mgmt_agent
• cloud_agent_rdb_callbacks
• cloud_agent_scheduler
• dc_client_ui
• dc_instant_baseline
• dc_manifest

in unified manager, you could check the au.log (in C:\ProgramData\NetApp\OnCommandAppData\au\log) and the ocumserver.log (in C:\ProgramData\NetApp\OnCommandAppData\ocum\log), but i don't have any keywords to check there other than the cluster name or the ip you added it with.

#

and just as an fyi, i don't believe completely disabling ontapi is viable yet if you need to use System Manager. this is in 9.15RC1:


Warning: The service "ontapi" supplies features required by: "sysmgr". Disabling "ontapi" will disable all dependent services.
Do you want to continue? {y|n}: y

dawn-cl::*> vserver services web show -name ontapi
Vserver        Type     Service Name     Description                   Enabled
-------------- -------- ---------------- ----------------------------- -------
dawn-cl       admin    ontapi           Remote Administrative API     false
                                         Support
dawn-dr       data     ontapi           Remote Administrative API     true
                                         Support
dawn-nas      data     ontapi           Remote Administrative API     true
                                         Support
dawn-san      data     ontapi           Remote Administrative API     true
                                         Support
4 entries were displayed.

dawn-cl::*> vserver services web show -name sysmgr
Vserver        Type     Service Name     Description                   Enabled
-------------- -------- ---------------- ----------------------------- -------
dawn-cl       admin    sysmgr           OnCommand System Manager      false

dawn-cl::*> version
NetApp Release 9.15.1RC1: Tue May 14 10:56:50 UTC 2024```

I don't think we'll be 100% rest based until 9.18.

cyan frigate May 29, 2024, 11:58 AM

#

robust basin and just as an fyi, i don't believe completely disabling ontapi is viable yet if...

I guess the timeline shifted a bit...
Don't think anybody will mind.

But that explains, why it's not switched off by default (yet), like the Knowledge Base article stated...

cyan frigate May 29, 2024, 12:03 PM

#

robust basin so this is where we get into the weeds. we in support don't yet have a whole lot...

Last EMS message:

mhost.ca.connect.failure: Cluster agent connection of the client: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3 is not healthy. Attempting to reconnect. Error: AMQP transport failed for connection "UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3". Reason: Error during websocket handshake: DNS NXDOMAIN.

Sequence number
6081

Description
This message occurs when the cluster agent connection is not in a connected state. Reconnect attempt to establish a connection will be initiated automatically.

Event
mhost.ca.connect.failure: Cluster agent connection of the client: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3 is not healthy. Attempting to reconnect. Error: AMQP transport failed for connection "UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3". Reason: Error during websocket handshake: DNS NXDOMAIN.

Action
Verify that the connection path to the destination is in healthy state and network security polices are configured as needed to permit connectivity. If this behavior persists, contact NetApp technical support.

-->After switching off certificate checks & allowing for self-signed certs these messages stopped occurring...

#

mgwd.log:

0000000a.0003ac36 000d30a1 Wed May 29 2024 08:34:49 +00:00 [kern_mgwd:info:3727] 0x840c5e500: 0: ERR: amqp_mgmt_agent: sendError:src/tables/amqp_mgmt_agent.cc:700 sendError on subject 'counters.request' for 'UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3': entry doesn't exist 0000000a.0003ac37 000d30a1 Wed May 29 2024 08:34:49 +00:00 [kern_mgwd:info:3727] 0x840c5e500: 0: ERR: mgmt_agent_collections_api: handle:src/tables/mgmt_agent_collections_api.cc:810 Failed to collect and send counters for specified datastore "opm" for cloud agent instance "UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3". Reason: entry doesn't exist 0000000a.0003ad50 000d35f3 Wed May 29 2024 08:37:05 +00:00 [kern_mgwd:info:3727] 0x840c5e500: 0: WARNING: cloud_agent_utils: collection_file_cleanup_rotation:src/tables/cloud_agent_utils.cc:148 Failed to cleanup collection history for client: UnifiedManager_1f8a4437-d706-4821-a68c-a2d6ec3112e3. Client either does not exist or is not a REST api service client
That's all the relevant entries from mgwd.log (occurring over & over with different time stamps...)

drowsy bough May 29, 2024, 12:24 PM

#

DNS NXDOMAIN
That's your problem, right there. DNS resolution is not working correctly

cyan frigate May 29, 2024, 12:29 PM

#

The environment is really simple: fresh 9.14.1 Sim install, updated to 9.14.1P4. Fresh 9.14 AIQUM OVA install. Some load generator on the connected virtualized DC. Everything in it's own 192.168.0.0/24 subnet.

No "case" necessary to replicate, I guess...

#

Double-checked, if Performance archiving is active:
`cluster1::statistics archive config*> show

           Is the Performance Archive Enabled?: true
    Base Path of Performance Archive Datastore: /mroot/etc/log/stats/ccma
       Days to Retain Performance Archive Data: 28

Maximum Percentage of Root Volume Used for Performance Archive Data: 5%
Minutes Between Performance Archive Data File Rotations: 2
Minutes Between Perfstat Collections: 0
Date Perfstat Collection Will Be Turned off: -
Minutes Between OPM Data File Rotations: 5
Minutes Between ASUP Data File Rotations: 30
Days to Retain ASUP Archive Data: 28
Hours to Retain Agent Archive Data: 12
Software Client of Installed Image: rtpbuild:R9.14.1PxN_240418_0624
Enabled Performance Preset Names: asup-hourly, asup_cfg,
copy_offload, css, default,
mcc_performance, nas, nvmf,
opm, ps, san, statit,
svm_migrate, sysstat_all,
sysstat_mp,
waffinity_stats,
wafl_bufstats, wafl_cpm,
wafl_spinhi, wafl_susp`

And checked in the cluster1-01/etc/log/stats/ccma/crs directory: loads of files present in 2 minutes interval timestamps, just as expected...

robust basin May 29, 2024, 7:54 PM

#

cyan frigate **mgwd.log:** `0000000a.0003ac36 000d30a1 Wed May 29 2024 08:34:49 +00:00 [kern...

these 'does not exist' mgwd messages look like what i get when i try to run commands on a connection that has been removed. is it still listed with 'cluster agent connection show'? looks like um might have tried deleting it and wasn't able to cleanup everything.

the ccma files path um is looking for is /etc/log/stats/ccma/kernel/opm/

for the DNS issue, you could also try to add a host entry on the cluster to resolve um's dns name.
::> vserver services name-service dns hosts
create delete modify show

what I found for the DNS error is the cluster cannot resolve UM's name/fqdn in the amqpwss URL used for the agent connection. UM provides this name to the cluster rather than the cluster looking it up. in this case, it seems like it was checking the certificate as well as DNS since it stopped after disabling the certificate checks

cyan frigate May 30, 2024, 2:18 PM

#

`Index of cluster1-01/etc/log/stats/ccma/kernel/opm/

Name Last Modified Size
Parent Directory/ Thu May 30 10:16:12 America/New_York 2024 -

000941_000300_1717077630104_0263281.ccma.gz Thu May 30 10:05:30 America/New_York 2024 16337

000941_000300_1717077630104_0263281.ccma.md5 Thu May 30 10:05:30 America/New_York 2024 158

000941_000300_1717077630104_0263281.ccma.meta Thu May 30 10:05:30 America/New_York 2024 338437

000942_000300_1717077930104_0263281.ccma.gz Thu May 30 10:10:30 America/New_York 2024 16296

000942_000300_1717077930104_0263281.ccma.md5 Thu May 30 10:10:30 America/New_York 2024 158

000942_000300_1717077930104_0263281.ccma.meta Thu May 30 10:10:30 America/New_York 2024 338437

000943_000300_1717078230104_0263281.ccma.gz Thu May 30 10:15:30 America/New_York 2024 16771

000943_000300_1717078230104_0263281.ccma.md5 Thu May 30 10:15:30 America/New_York 2024 158

000943_000300_1717078230104_0263281.ccma.meta Thu May 30 10:15:30 America/New_York 2024 338437

000944_000000_0000000000000_0000000.ccma.gz Thu May 30 10:15:29 America/New_York 2024 0

000944_000000_0000000000000_0000000.ccma.meta Thu May 30 10:15:29 America/New_York 2024 332029

`

#

Cluster Agent connection still active, Counter requests still sent, but not answered (error messages instead.

#

dns

#

`cluster1::vserver services name-service dns*> show
Name
Vserver Domains Servers

NAS_svm demo.netapp.com 192.168.0.253
cluster1 demo.netapp.com 192.168.0.253
2 entries were displayed.

cluster1::vserver services name-service dns*> ping aiqum -node local
aiqum is alive

`

#

Name resolution works...

#

==> Big hint from colleague caring for the production network:

The production AIQUM that was updated from 9.13 to 9.14 (that gave performance data for our production systems) is still giving performance data.

The fresh 9.14 AIQUM install that he tested in production does not give performance data, just like in the lab.

The Lab colleagues want to test installing a fresh 9.13 in the lab, then updating it to 9.14...

robust basin May 30, 2024, 5:40 PM

#

new connections (adding a cluster to um 9.14) will attempt to utilize cloud agent first.
existing connections using password or mtls will not be changed to cloud agent connections when um is upgraded.

i assume your prod has been existence for quite a while and was using password or mtls auth before to 9.14, and that is why it is still working. a new cluster being added to your prod could run into these same issues.

Are you seeing broker failures in au.log?
https://kb.netapp.com/data-mgmt/AIQUM/AIQUM-Issues/CAIQUM-5673

NetApp Knowledge Base

CAIQUM-5673: Cluster acquisition in AIQUM 9.14 RC1 is failing due t...

cyan frigate May 31, 2024, 9:17 AM

#

robust basin new connections (adding a cluster to um 9.14) will attempt to utilize cloud agen...

No Errors or Warnings, just INFOs, connectivity seems fine. It successfully asks for counters, it just does not get any back

2024-05-31 01:05:51,697 INFO - clusterUuid=0b3f2b34-18e5-11ef-ab73-005056990101, action=COUNTERS_REQUEST_STATUS_FAILURE, payloadDir=null, success=FAILURE 2024-05-31 01:05:51,697 INFO - DataCollectorConnection:signal() clusterUuid = 0b3f2b34-18e5-11ef-ab73-005056990101, type = COUNTERS_REQUEST_STATUS_FAILURE 2024-05-31 01:05:51,697 INFO - key=0b3f2b34-18e5-11ef-ab73-005056990101?type=COUNTERS_REQUEST_STATUS_FAILURE 2024-05-31 01:05:51,697 INFO [async-perf-pool-5] c.n.a.d.a.DataCollectorConnection (DataCollectorConnection.java:133) - DataCollectorConnection: wait() 0b3f2b34-18e5-11ef-ab73-005056990101, removing message from messageMap 2024-05-31 01:05:51,698 INFO [async-perf-pool-5] c.n.a.d.a.DataCollectorTrigger (DataCollectorTrigger.java:300) - Successfully ask for counters from cluster: 0b3f2b34-18e5-11ef-ab73-005056990101 2024-05-31 01:05:51,707 INFO - --------- Received Message --------- 2024-05-31 01:05:51,708 INFO - JMSMessageID: ID:c266b430-ce93-4ca2-953b-7c94a2ef44aa:7:1:1-7626 2024-05-31 01:05:51,708 INFO - JMSTimestamp: 1717142751697 2024-05-31 01:05:51,708 INFO - JMSCorrelationID: null 2024-05-31 01:05:51,708 INFO - JMSReplyTo: null 2024-05-31 01:05:51,708 INFO - JMSDestination: um.mega.manager 2024-05-31 01:05:51,708 INFO - JMSDeliveryMode: 1 2024-05-31 01:05:51,708 INFO - JMSRedelivered: false 2024-05-31 01:05:51,708 INFO - JMSType: counters.request 2024-05-31 01:05:51,708 INFO - JMSExpiration: 0 2024-05-31 01:05:51,708 INFO - JMSPriority: 4 2024-05-31 01:05:51,708 INFO - JMSCorrelationIDAsBytes: null

cyan frigate May 31, 2024, 9:22 AM

#

drowsy bough > DNS NXDOMAIN That's your problem, right there. DNS resolution is not working c...

As mentioned, PINGing "AIQUM" works fine. (name resolution via DNS server, no local host entry)

After certificate check was softened, THIS error disappeared. Seems to have been trouble with e.g. the self-signed cert...

cyan frigate May 31, 2024, 11:42 AM

#

robust basin That looks like an issue with the cloud agent connection, but I haven't seen tha...

Finally switched off Cloud Agent in server.properties:

Re-started the services, Still no performance data.
No error messages either.
The Cloud Agent connection was predictably deleted upon user request

What's worse: Health data collection seems to have stopped, too!

We're going the "install 9.13 and upgrade" route, and check how that's working...

cyan frigate Jun 3, 2024, 1:03 PM

#

It seems to be an ONTAP 9.14 problem...
Our production colleague tested with AIQUM 9.13 and 9.14:
Our ONTAP **9.12.1 **systems delivered performance data to both.
Our ONTAP **9.14.1 **systems delivered performance data to none...

robust basin Jun 3, 2024, 11:27 PM

#

support is looking at issues with adding 9.14 and 9.15 to UM 9.14
there are some known issues with mtls that we're presenting to engineering, but we have no data on the cloud connection issues.
i can tell you how to disable mtls and the cloud connection, but making them work without disabling is hit or miss unfortunately. disabling is the only sure fire method to get polling working, which of course fixes nothing long term.

cyan frigate Jun 4, 2024, 12:49 PM

#

Feedback from our Lab guy:

ONTAP 9.14.1 RC1 works, ONTAP 9.14.1GA does not

#

Please forward to the relevant ONTAP guys, it seems they broke the performance collection between RC1 and GA and obviously even P4 does not fix it...

cyan frigate Jun 5, 2024, 4:41 PM

#

@stark lily is this thread something for you?

robust basin Jun 5, 2024, 9:06 PM

#

I work with Hasnain and these issues with adding clusters are among the things we've been discussing internally. I don't think he's on discord often.

#AIQUM performance counter collection fails (9.14)

`cluster1::system services web ontapi> show ONTAPI Suspended

`cluster1::system services web ontapi> show
ONTAPI Suspended