Prometheus Consuming Huge Space / chunks | NetApp | Page 1

remote knoll Nov 3, 2023, 4:20 PM

#

We are monitoring 38 clusters, mainly 2 nodes but have several 6 node clusters.

I have the Prometheus retention set to 180 days but I'm constantly running out of space. We are now at 600GB for the data drive and at 93% as I just increased it from 580GB.

I'm looking at the /data/prometheus/data directory consuming 538GB and running a du - h /data/prometheus/data I see there are roughly 30+ 01HXXXXX directories. Shouldn't prometheus be compressing the data out? WAL is only 3.8GB currently.

Just to give an example:
36G /data/prometheus/data/01HE6B4XWAWWEA41FHM1V4YF45/chunks
38G /data/prometheus/data/01HE6B4XWAWWEA41FHM1V4YF45

Any suggestions?

tropic dew Nov 3, 2023, 7:44 PM

#

That is actually really good IMO.

#

If you think a standard perf archive for support purposes is about 100 MB compressed, times 68+nodes...it sounds about right.

#

PAs have a lot more data overall...

fierce blaze Nov 3, 2023, 10:04 PM

#

You might want to disable detailed metrics collection, there will be a checkbox for this in 3.4

remote knoll Nov 3, 2023, 10:18 PM

#

@fierce blaze How would I go about doing that? Also, we don't really need to monitor quotas and possibly even qtrees, is there a simple way to prevent polling those and delete the metrics it's captured?

fierce blaze Nov 4, 2023, 10:46 AM

#

I would recommend you update to 3.4b, unfortunately it's not available at the moment

#

back up in a few days probably

quiet dragon Nov 6, 2023, 11:05 AM

#

@remote knoll Can you send the logs to ng-harvest-files@netapp.com. We can identify and disable templates that are doing extensive data collection and are not relevant to your use case. For more information, please refer to the documentation at https://netapp.github.io/harvest/23.08/help/log-collection/#nabox.

remote knoll Nov 8, 2023, 1:53 PM

#

@quiet dragon I've emailed logs to the address you provided.

quiet dragon Nov 8, 2023, 2:03 PM

#

Thanks Received.

smoky tusk Nov 8, 2023, 2:32 PM

#

remote knoll <@323825973718155265> How would I go about doing that? Also, we don't really ne...

@fierce blaze have you documented how to disable a handful of the default templates?

fierce blaze Nov 8, 2023, 2:49 PM

#

It’s generated with every start unfortunately. You can add things but not remove from defaults

smoky tusk Nov 8, 2023, 2:51 PM

#

would a temporary workaround be to make the changes in /opt/packages/harvest2/conf those will persist across restarts until you upgrade Harvest?

remote knoll Nov 8, 2023, 2:52 PM

#

I'm having an issue with Prometheus replaying WAL, times out then restarts again. I have space and increased the RAM to 64GB.

quiet dragon Nov 8, 2023, 3:07 PM

#

@remote knoll Based on the logs, there are 2 pollers where Qtree and CIFSSession has high object count. We can disable the Zapi Qtree and ZapiPerf Qtree templates, as well as the Zapi CIFSSession template, if you are not using them.

remote knoll Nov 8, 2023, 3:10 PM

#

We are not so I would imagine that would help with performance improvements

#

And those clusters are actually the same. The 01 was renamed to the -GEN. In NABOX Admin I updated the cluster name but kept the 01 address so we wouldn't lose historical data but I think we ended up losing it anyway

#

@quiet dragon Will you be sending an email on disabling those?

quiet dragon Nov 8, 2023, 3:16 PM

#

Do you mean steps how to disable them in NABox?

remote knoll Nov 8, 2023, 3:17 PM

#

Yes

quiet dragon Nov 8, 2023, 3:19 PM

#

Sure I'll share in a moment.

remote knoll Nov 8, 2023, 3:20 PM

#

Thank you

quiet dragon Nov 8, 2023, 4:00 PM

#

@remote knoll Here are the steps https://github.com/NetApp/harvest/discussions/2470

remote knoll Nov 8, 2023, 4:01 PM

#

Thank you kindly

#

I have implemented the changes. Is there a way to delete the stats for these to free up space?

smoky tusk Nov 8, 2023, 4:15 PM

#

https://nabox.org/faq/#managing-metrics has instructions on deleting metrics and old data

remote knoll Nov 8, 2023, 4:17 PM

#

I have seen that. Do I need to do it for all qtrees/CIFS sessions or is there a wildcard I could use?

smoky tusk Nov 8, 2023, 4:22 PM

#

the underline part is a normal prometheus query so I would recommend refining that query in Prometheus query editor to make sure you have the labels you want and then use the delete_series API

There are some good pointers here too about using promtool or tsdb-status to find the largest series https://tanmay-bhat.github.io/posts/how-to-drop-and-delete-metrics-in-prometheus/

remote knoll Nov 8, 2023, 4:24 PM

#

@smoky tusk Thank you very much! I was just looking at updating that metric and looking for the correct syntax so your response was very helpful.

#

The issue I have at the moment however is that Prometheus will not come up, it's keeps replaying the WAL so I'm unable to get the correct query.

smoky tusk Nov 8, 2023, 4:37 PM

#

It may take awhile to start - another person was facing a similar long time to start in this thread (not sure you saw it) #1156924701621366794 message

remote knoll Nov 8, 2023, 4:39 PM

#

It'll run then eventually show :Error: Error fetching Runtime Information: Bad Gateway. If I refresh the Prometheus web interface it'll restart the Replay

#

Looks like they experienced the same I am. I can wait and see if it finishes but if not I'll do what was recommended and delete the WAL files.

#Prometheus Consuming Huge Space / chunks