#NAbox Alert : CapacityLow [/data] has less than 15% available.

51 messages · Page 1 of 1 (latest)

soft frost
#

https://nabox.org/faq/#delete-data
Managing Metrics --> Delete data

We've got Cohesity snapshot-integrated backups currently enabled which generates a large number of clone volumes that prepends all volume names with cohesity_*

Is there a way to query delete all those old clone volume metrics?

Delete metrics for volume myvolume

curl -k -s
'https://nabox_ip/vm/api/v1/admin/tsdb/delete_series?match[]={volume="myvolume"}'

Labels
alertname = CapacityLow
alertgroup = NAbox Rules
device = /dev/sdb
fstype = ext4
instance = node-exporter:9100
job = node-exporter
mountpoint = /data
severity = critical
Annotations
description = [/data] has less than 15% available.
summary = Mountpoint [/data] low capacity

surreal fog
torpid wharf
surreal fog
#

So maybe that’s a NAbox problem. Was it added recently ?

torpid wharf
soft frost
soft frost
surreal fog
#

Yes you should be able to use volume=~”cohesity_.*”

#

But I’m confused, did you see the link I shared or did you check the query you mentioned earlier ?

#

Ok I was confused. Chris harvest should probably ignore cohesity clones right ?

soft frost
# surreal fog Yes you should be able to use volume=~”cohesity_.*”

$ curl -k -s 'https://x.x.x.x/vm/api/v1/admin/tsdb/delete_series?match[]=\{volume=~"cohesity_.*"\}'
cannot delete time series: the number of matching timeseries exceeds 1000000; either narrow down the search or increase -search.max* command-line flag values (the most likely limit is -search.maxUniqueTimeseries); see https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#resource-usage-limits

VictoriaMetrics: Single-node version

Documentation for VictoriaMetrics, VictoriaLogs, Operator, Managed VictoriaMetrics and vmanomaly

surreal fog
#

Try narrowing it down by putting something after _. Not sure what the naming is

soft frost
#

seems to be fluctuating - will give it some time to settle down...

#

@torpid wharf - what syntax do you suggest we add to exclude all cohesity_* volume clones?

torpid wharf
#

I'm not sure what all of your names look like, but from the info you have provided something like the attached screenshot.
That assumes that all the volumes start with cohesity. If they don't, you can adjust to something like cohesity_. If that doesn't work for you, can you provide more example volume names of what you want to exclude?

surreal fog
soft frost
soft frost
torpid wharf
#

That should work, but depends on all the ways they are named. You can check your regex and example names here https://regex101.com/r/3tZkOu/1. The lines highlighted in blue and shown in the Match Information panel are the lines that are matched. The other aren't. If you want those other lines matched too, share some examples that aren't working and we can help you with the regex.

soft frost
#

Found the config file in /data/packages/harvest/conf/zapi/cdot/9.8.0/volume.yaml

torpid wharf
#

to make these changes in nabox4, try this instructions https://github.com/NetApp/harvest/discussions/3904 click on the nabox4 twisty. In step 3, you do not need to run the sed command. Instead edit the copied template and add the volume cohesity line to the exclude_regex plugin like in your screenshot above

#

are you using Zapi or Rest?

soft frost
#

Added - volume cohesity_.+

torpid wharf
#

since you are using nabox, you need to be mindful of how you edit the template files, otherwise when you upgrade your changes will be overwritten. The link I shared above describes how to make changes to Rest templates that aren't touched by an upgrade. If you're using Zapi templates, there is a slightly different way that I can share.

For both Rest and Zapi, you want both the config and performance collector to ignore these volumes. Since you edited the conf/zapi template, you should see them ignored in the volume_labels metric, but you will still collect their performance metrics since the conf/zapiperf template was not also updated.

surreal fog
#

I'm adding I'm adding it to nabox (- volume cohesity_.+)

torpid wharf
#

@surreal fog if you are considering adding it to nabox out-of-the-box, doesn't that imply that cohesity belongs in the list of excludes that Harvest already ships with? If so, we can add it there

surreal fog
#

I need to double check that indeed. I'm not sure how it works now, it's possible at some point I removed the nabox implementation in favor of letting harvest apply defaults

torpid wharf
#

we discussed that, but I haven't checked what your overrides are

surreal fog
#

Ok I checked, NAbox is still doing it :

admin@localhost /etc/nabox/harvest/active/zapi $ cat custom.yaml
objects:
    CIFSSession: cifs_session.yaml
    Volume: exclude_transient_volumes.yaml
admin@localhost /etc/nabox/harvest/active/zapi $ cat cdot/9.8.0/exclude_transient_volumes.yaml
plugins:
  LabelAgent:
    # metric label zapi_value rest_value `default_value`
    exclude_regex:
      - volume `.+_CVclone`
      - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
      - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
      - volume `sdw_cl_.+`
      - volume `MDV_CRS_.+`
      - volume `MDV_aud_.+`
#

now, how does that conflicts/complete Harvest default list ?

torpid wharf
#

given how nabox sets up confpath, the out-of-the-box templates will be loaded first. You can confirm by restarting havrest and then searching the logs for best-fit You will see where the template is being loaded from. e.g.

msg="best-fit template" Poller=umeng-aff300-01-02 collector=Rest:Volume path=/harvest/conf/rest/9.14.0/volume.yaml

vs
msg="best-fit template" Poller=umeng-aff300-05-06 collector=Zapi:Volume path=/harvest-conf/active/zapi/cdot/9.8.0/exclude_transient_volumes.yaml

soft frost
#

Still trying to delete almost 2,000 old cohesity flexclone volume entries but wildcards aren't working - unless I have a syntax error?

curl -k -s 'https://<nabox_ip>/vm/api/v1/admin/tsdb/delete_series?match[]=\{volume="cohesity*"\}'
What's the best way to do it?

surreal fog
#

Are you getting an error ?

#

Did you try cohesity_.+ like mentioned earlier ?

soft frost
#

@surreal fog - no not receiving an error - it doesn't delete all the stale volume clone names
Yes, added the following line: - volume cohesity__.+

  # Ignore transient volumes, e.g. SnapProtect, SnapManager, SnapCenter, CommVault, Clone, and Metadata volumes
  exclude_regex:
    - volume `.+_CVclone`
    - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
    - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
    - volume `sdw_cl_.+`
    - volume `MDV_CRS_.+`
    - volume `MDV_aud_.+`
    - volume `cohesity__.+`
soft frost
#

@surreal fog , @torpid wharf - ran a large number of curl commands to delete the old Cohesity clones but still showing /data around 88% used.
What's the best way to both list and delete all the old cohesity_* clone volumes?

surreal fog
torpid wharf
#

you probably want to use a regex to find these cohesity clones - in which case, you need to use =~ like below. Before doing a delete, first do a select to make sure that you have the correct selector:

curl --insecure 'https://10.193.49.126/vm/prometheus/api/v1/series?match[]=volume_labels\{volume=~".*234"\}'

soft frost
#

@torpid wharf - ty. Yes, that came back with the listing of all the cohesity clones
However ran the following and still showing /data at 88% used...

curl --insecure 'https://<redacted>/vm/prometheus/api/v1/series?match[]=volume_labels{volume=~"cohesity."}'
curl --insecure 'https://<redacted>/vm/api/v1/admin/tsdb/delete_series?match[]=volume_labels{volume=~"cohesity."}'
curl --insecure 'https://<redacted>/vm/internal/force_merge'

torpid wharf
#

try this:

curl -v http://<vmselect>:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series -d 'match[]=volume_labels{volume=~"cohesity."}'

soft frost
#

failed to connect - our network team has locked down things
Port 8481 for Victoria Metrics?

surreal fog
#

With NAbox, replace the base endpoint to https://nabox_ip/vm/ (no port)
So that would be curl -v https://nabox_ip/vm/delete/0/prometheus/api/v1/admin/tsdb/delete_series -d 'match[]=volume_labels{volume=~"cohesity."}'

That's if you have guest access enabled, otherwise you'd have to create a token for VM and send it as Bearer token in headers

soft frost
#

Already have Victoria Metrics guest access enabled - still erroring out:

$ curl -v https://1.2.3.4/delete/0/prometheus/api/v1/admin/tsdb/delete_series -d 'match[]=volume_labels{volume=~"cohesity."}'

  • Trying 1.2.3.4:443...
  • Connected to 1.2.3.4 (1.2.3.4) port 443
  • ALPN: curl offers h2,http/1.1
  • (304) (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/cert.pem
  • CApath: none
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (IN), TLS handshake, Unknown (8):
  • (304) (IN), TLS handshake, Certificate (11):
  • (304) (IN), TLS handshake, CERT verify (15):
  • (304) (IN), TLS handshake, Finished (20):
  • (304) (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
  • ALPN: server accepted h2
  • Server certificate:
  • subject: C=US; ST=New York; L=New York; O=Xylem Inc; CN=example.com
  • start date: Jun 24 15:41:48 2025 GMT
  • expire date: Jul 19 15:40:48 2026 GMT
  • subjectAltName does not match ipv4 address 1.2.3.4
  • SSL: no alternative certificate subject name matches target ipv4 address '1.2.3.4'
  • Closing connection
    curl: (60) SSL: no alternative certificate subject name matches target ipv4 address '1.2.3.4'
    More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

surreal fog
#

Yes when you don’t have a valid certificate setup you need to use curl -k

surreal spoke
#

Hi all — I'm a colleague picking this back up. Found the root cause.

The delete kept failing (exceeds 1000000) because there are >10M historical cohesity_* series (~2yr of daily FlexClone churn) — even per-last-digit batches were all >1M. With /data at 96%, force_merge felt risky, so we grew the /data VMDK 210→527 GB (per the FAQ) for headroom.

But the real issue is we're still collecting the clones. On NAbox 4.1.0 our pollers collect Volume via REST, yet /usr/share/nabox/harvest/exclude-transient-volumes/ only ships zapi/ + zapiperf/ — no rest/restperf. So the exclude never applies to REST collectors (and the list has no cohesity_ pattern). Confirmed still collecting: count by(volume) = 13 for volume_labels{volume=~"cohesity_.+"} and 14 for volume_read_data{...} (would be 0 if excluded).

We can't self-patch: /usr/share/nabox is read-only, and files in /etc/nabox/harvest/active/ get wiped by the startup regeneration.

best-fit (Volume = REST, not ZAPI):

 Rest:Volume      /harvest/conf/rest/9.14.0/volume.yaml
 RestPerf:Volume  /harvest/conf/restperf/9.12.0/volume.yaml

package ships only zapi/zapiperf (no rest/restperf):

 exclude-transient-volumes/zapi/{cdot/9.8.0,7mode/8.6.0}/...  + custom.yaml
 exclude-transient-volumes/zapiperf/{cdot/9.8.0,7mode/8.2.5}/... + custom.yaml

Asks:

Add rest/+restperf/ to exclude-transient-volumes (mirroring zapi/zapiperf) + a cohesity_.+ pattern?
Or is there a supported /etc/nabox/harvest/user/ layout for REST excludes that survives regeneration/upgrades? The README there only links the generic Harvest docs.

Thanks!

torpid wharf
surreal spoke
#

Thanks @torpid wharf — that explains it. We're on NAbox 4.1.0, which bundles Harvest 25.02.0-1, so we are a bit outdated from the v26.05.0 default exclude. What's the recommended way to get Harvest 26.05.0 (or newer) on NAbox? A NAbox release that bundles it, or a supported in-place Harvest update?

torpid wharf
#

either will work, but I would proably upgrade nabox since there have been a number of improvements since then