Data retention on harvest | NetApp | Page 1

sinful socket Jan 29, 2025, 1:08 AM

#

I have setup harvest and its been working well, but when I am trying to see the historical data, afteer 28 days it goes blank.

I have checked the prometheus config and I have it set for 1y, or may be I am not chelking in the right place.

I edited /etc/systemd/system/prometheus.service

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus
--config.file /etc/prometheus/prometheus.yml
--storage.tsdb.path /var/lib/prometheus/
--storage.tsdb.retention.time=2y
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries

is there a different place I need to check ?

also If I do the curl query (i made the retention to 2y and restarted prometheus) I still see 1y

curl -s http://localhost:9090/api/v1/status/runtimeinfo
{"status":"success","data":{"startTime":"2025-01-29T00:46:14.341118477Z","CWD":"/","reloadConfigSuccess":true,"lastConfigTime":"2025-01-29T00:46:18Z","corruptionCount":0,"goroutineCount":71,"GOMAXPROCS":48,"GOMEMLIMIT":9223372036854775807,"GOGC":"75","GODEBUG":"","storageRetention":"1y"}}

acoustic stirrup Jan 29, 2025, 6:38 AM

#

@sinful socket
After editing the service file, make sure to reload the systemd daemon to apply the changes.

sudo systemctl daemon-reload
sudo systemctl restart prometheus

sinful socket Feb 3, 2025, 6:02 PM

#

i did, i even rebooted the host.!

wind pagoda Feb 3, 2025, 6:04 PM

#

what does sudo systemctl status prometheus show?

sinful socket Feb 4, 2025, 6:14 PM

#

thats says 1 year

[root@harvstsvc0001 ~]# sudo systemctl status prometheus -l
‚óè prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2025-01-28 16:46:14 PST; 6 days ago
Main PID: 30859 (prometheus)
CGroup: /system.slice/prometheus.service
‚îî‚îÄ30859 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --storage.tsdb.retention.time=1y --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

wind pagoda Feb 4, 2025, 7:17 PM

#

thanks. the prometheus.service shows storage.tsdb.retention.time=1y, what would you like it to be? You should be able to change that value by editing the appropriate argument in /etc/systemd/system/prometheus.service

Also looks like you have vendor preset: disabled which means when the machine reboots this service will NOT be restarted automatically. If you want Prometheus to restart after the machine reboots run systemctl enable prometheus

sinful socket Feb 4, 2025, 7:34 PM

#

oh thats a miss on my end, I want it to start back.! and made the necessary changes

I have it changed to 2y and I can see the service now shows 2 years, but when I see grafana host, I dont see any metrics after 30 days.!

#

[root@harvstsvc0001 /]# sudo systemctl status prometheus -l
‚óè prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2025-02-04 10:15:54 PST; 1h 19min ago
Main PID: 4899 (prometheus)
CGroup: /system.slice/prometheus.service
‚îî‚îÄ4899 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --storage.tsdb.retention.time=2y --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

#

Screenshot_2025-02-04_at_11.36.28_AM.png

wind pagoda Feb 4, 2025, 7:56 PM

#

Has Harvest been running longer than 30 days? In other words, Prometheus will only contain metrics for the pollers that it has been scrapping

#

so if a poller has only been running for one week and Prometheus was scraping that poller - you won't see more than one week's worth of metrics for that poller

#

Harvest does not back-fill histocial metrics

sinful socket Feb 4, 2025, 9:25 PM

#

yes harvest and al lthe pollers are running for more than 30 days, the only time the harvest goes down is for the patching . Pollers are un edited.!

wind pagoda Feb 5, 2025, 12:36 PM

#

Let's skip Grafana for the moment and go straight to Prometheus. If you search for something like node_labels then click the Graph button and expand the time range do you see data? For example, on the system below there is data going back to March

sinful socket Feb 5, 2025, 6:18 PM

#

Screenshot_2025-02-05_at_10.18.35_AM.png

#

doesnt look like its retaining 1 year

#

Screenshot_2025-02-05_at_10.21.29_AM.png

wind pagoda Feb 6, 2025, 12:30 PM

#

Do you recall what Prometheus's retention was before you changed it? It should have been 15d since that's the default. https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

When you changed the retention from 15d to 1 year, from that point forward, Prometheus will start retaining more data. Prior to your change, Prometheus only has that last 15 days of data saved

Storage | Prometheus

An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

sinful socket Feb 7, 2025, 5:27 PM

#

I actually don’t remember what it was before, so now that the Prometheus reflects higher retention, we are good right?

Let me soak it for few weeks and see how it goes

#

Thank you for the help

wind pagoda Feb 7, 2025, 5:37 PM

#

yes, you should be good. Let us know in a few weeks if you don't see data

grim scarab Jul 2, 2025, 9:48 PM

#

@wind pagoda Hello all, I know this is old but why start a new thread, right. So, with only the basic install of NAbox and Harvest we can still set the retention period for data collectors for up to 2 years? By editing the /etc/systemd/system/prometheus.service? Should we not need a seperate Prometheus server installed and Harvest pointing to that or it's not needed?

acoustic stirrup Jul 3, 2025, 6:11 AM

#

NABox 4 comes with a 2-year default retention period. For more information, visit NABox FAQ.

If you are using NABox 3, you can change its Prometheus retention as well. @past lodge can help with the steps.

grim scarab Jul 3, 2025, 4:47 PM

#

@acoustic stirrup I didn't see a default retention or even the line entry in the env.custom file so I added the line. So that means from today's date it will start retaining 2 years worth of metrics? GF_SERVER_DOMAIN=10.x.x.x
GF_AUTH_LDAP_ENABLED=true
ALERTMANAGER_LINK_URL=
GF_SMTP_ENABLED=false
GF_SMTP_HOST=
GF_SMTP_USER=
GF_SMTP_FROM_ADDRESS=
retentionPeriod=2y

wind pagoda Jul 3, 2025, 5:49 PM

#

@grim scarab in nabox4, the two years retention is set in /etc/nabox/.env and /etc/nabox/.env.custom is used to override those values. If I check https://nabox-ip/vm/flags I see -retentionPeriod="2y" even though I do not have a .env.custom If you back out your change and restart, you should see the same

grim scarab Jul 3, 2025, 5:51 PM

#

@wind pagoda Oops then let me remove it from the /etc/nabox/.env.custom. Because I see it in the directory you mentioned already. Thanks!

#

@wind pagoda This link is a little confusing because it doesn't mention the file you stated, it only says to edit the .env.custom file - https://nabox.org/faq/#change-default-retention

wind pagoda Jul 3, 2025, 5:57 PM

#

I think you tagged the wrong Chris. @past lodge can update the documentation to be clearer

#Data retention on harvest