Volume User (Or IP, or both) IO Metrics in Harvest?

[bronze sail]

Example query from REST API:
https://cluster/api/storage/volumes/Volume-UUID/top-metrics/users?top_metric=iops.read&return_records=true&return_timeout=15

Example Response:
{
  "_links": {
    "next": {
      "href": "/api/resourcelink"
    },
    "self": {
      "href": "/api/resourcelink"
    }
  },
  "incomplete_response_reason": {
    "code": "111411207",
    "message": "Partial data has been returned for this metric report. Reason: The activity tracking report for this volume is not available because the system is busy collecting tracking data."
  },
  "notice": {
    "code": "111411207",
    "message": "No read/write traffic on volume."
  },
  "num_records": 1,
  "records": [
    {
      "iops": {
        "error": {
          "lower_bound": 10,
          "upper_bound": 11
        },
        "read": 10,
        "write": 5
      },
      "svm": {
        "name": "vserver_3",
        "uuid": "42ee3002-67dd-11ea-8508-005056a7b8ac"
      },
      "throughput": {
        "error": {
          "lower_bound": 2,
          "upper_bound": 3
        },
        "read": 2,
        "write": 1
      },
      "user_id": 1876,
      "user_name": "Ryan",
      "volume": {
        "name": "vol_9",
        "uuid": "c05eb66a-685f-11ea-8508-005056a7b8ac"
      }
    }
  ]
}

Cant find nothing in Prometheus.

[orchid acorn]

@bronze sail Harvest doesn't collect top-metrics. What do you mean that there is nothing in prometheus?

[bronze sail]

i meant that what you said 😄
We have Customers that would like to see how many IOPs users are generating in our CIFS Env. and i havent found nothing in the metric browser on our Grafana.

[orchid acorn]

We have two open stories for similar requests: https://github.com/NetApp/harvest/issues/2591 https://github.com/NetApp/harvest/issues/2118

Could you +1 these stories if they are relevant to your use case?

[bronze sail]

I have upvoted.
@orchid acorn How realistic is it that i will get those Dashboards in one of future releases?

My customer is asking me regularly if its possible 😄

[orchid acorn]

@bronze sail we have discussed this requirement in detail recently, and here is the summary:

Let's take the top client as an example, which can be collected via the endpoints /storage/volumes/*/top-metrics for volume level or /svm/svms/{svm.uuid}/top-metrics at the SVM level. At the volume level, it will be an expensive call for ONTAP, especially when using the wildcard *. For SVMs, we will also need to make multiple calls to ONTAP to collect these stats. Also, each metric, such as iops.read, iops.write, throughput.read, and throughput.write, must be queried separately, which would quadruple the number of API calls.

Even if we make these calls, given the nature of the unique time series, this may generate for Prometheus, it is not good for Prometheus cardinality (https://last9.io/blog/how-to-manage-high-cardinality-metrics-in-prometheus/). Every time we may get different top clients, resulting in a lot of time series generation in Prometheus.

One of the possible solution for this is to not publish these metrics to Prometheus but to expose them via the Harvest CLI. We can call SVM endpoints via the CLI and display results via stdout only. But that is not something we are planning to implement currently.

[bronze sail]

Thank you for the update 🙂

[orchid acorn]

@bronze sail Top-metrics feature is available in Harvest 24.11 has added Top Clients and latest nightly has added Top Files collection. More information here https://github.com/NetApp/harvest/discussions/3130

[gloomy lintel]

resurrecting an old thread...

I ran across the blank panels for Top Files & Top Clients in the Volume dashboard in grafana, followed to https://github.com/NetApp/harvest/discussions/3130, and saw that we currently need to do some nabox customization to enable this. Given that it mentions a link to enabling it in the harvest UI, is there any intent to enable this from within the NABox UI? It seems like a "feature" rather than a customization, at least from my viewpoint.

[warm harness]

@atomic willow

[atomic willow]

Well, It seems I should prioritize this !

"Given that it mentions a link to enabling it in the harvest UI" I'm not sure I understand ?

[warm harness]

I believe kwinkelmanbw is saying that the Harvest documentation describes how to enable this off-by-default-feature here https://github.com/NetApp/harvest/discussions/3130 including steps for NAbox4. They would like NABox4 to surface the enable/disable part in the UI and not require them to edit files. Please correct me if that explanation is incorrect @gloomy lintel

[gloomy lintel]

that is correct @warm harness