Nabox 95% full, How do I purge old data? | NetApp | Page 1

brave remnant Mar 29, 2023, 8:14 AM

#

As per above, NAbox is full and not reporting. How do i remove old data

brave remnant Mar 29, 2023, 8:37 AM

#

I have followed the guide on the website but getting a 404 page not found from putty

#

frigid ravine Mar 29, 2023, 11:01 AM

#

@brave remnant You would need to start prometheus with --web.enable-admin-api to be able to use delete_series endpoint. More information here https://sbcode.net/prometheus/delete-timeseries/#:~:text=There may be a time,earlier%2C then you are able.

brave remnant Mar 29, 2023, 12:34 PM

#

I used done this as per the FAQ on the nabox website, but still 404

#

frigid ravine Mar 29, 2023, 3:29 PM

#

@brave remnant I have tried enabling the --web.enable-admin-api in nabox and it works fine. DO add your time range in below query.

curl -k --location -g --request POST 'http://localhost/prometheus/api/v1/admin/tsdb/delete_series?match[]={job="harvest2"}'

Did you restart prometheus after changing prometheus configuration? You can check if this setting is applied like below.

brave remnant Mar 29, 2023, 3:40 PM

#

Could you post that command in so I can just ctrl c it please 🙂

frigid ravine Mar 29, 2023, 3:42 PM

#

docker exec -it prometheus sh
ps -ef | grep prometheus

brave remnant Mar 29, 2023, 3:44 PM

#

keen crescent Mar 29, 2023, 3:45 PM

#

that probably means at mistake was made in changing the prom file and it is in a restart loop. Double check the edit

frigid ravine Mar 29, 2023, 3:46 PM

#

try running docker logs prometheus and see if any errors

brave remnant Mar 29, 2023, 3:47 PM

#

keen crescent Mar 29, 2023, 3:48 PM

#

ah not able to restart because of no free space issue. hmmm will need to cleanup something else so Prom can restart so you can delete some data

brave remnant Mar 29, 2023, 3:48 PM

#

Is the harvest install file stored anywhere? Could I remove them?

keen crescent Mar 29, 2023, 3:49 PM

#

yeah let us check and see what large file(s) we can remove

#

@storm merlin any pointers on what @brave remnant can delete to try to recover. I'm not seeing much fat to get rid of. Maybe /data/graphite if you aren't using Harvest 1.X but may be small if you aren.t @brave remnant can you increase the underlying disk?

brave remnant Mar 29, 2023, 3:57 PM

#

Can do but we will get charged

keen crescent Mar 29, 2023, 4:00 PM

#

i generally reach for ncdu when I want to find the top things to delete, but you probably will not be able to install it even though it's small. You could try by sshing in and running apk add ncdu then ncdu /

#

var/log/docker should be safe to remove

#

/var/cache/apk/* is safe

brave remnant Mar 29, 2023, 4:03 PM

#

I cant get out the network to install stuff 😦

storm merlin Mar 29, 2023, 4:06 PM

#

I'm guessing growing vmdk is not an option ?

#

df -h says what ?

keen crescent Mar 29, 2023, 4:08 PM

#

@brave remnant sent you a DM

storm merlin Mar 29, 2023, 4:08 PM

#

I usually go with

cd /data
du -hs *
# cd into directories causing issues
du -hs *
# etc...

keen crescent Mar 29, 2023, 4:09 PM

#

yep ncdu does all that for you

storm merlin Mar 29, 2023, 4:09 PM

#

yes much easier with ncdu but if not possible to install...

keen crescent Mar 29, 2023, 4:10 PM

#

it's a 67.8 KB file we can send to Watson

storm merlin Mar 29, 2023, 4:10 PM

#

good point, let me get that for you

keen crescent Mar 29, 2023, 4:10 PM

#

that's the DM I mentioned above

storm merlin Mar 29, 2023, 4:11 PM

#

cool

#

if /data is real full that might be an issue because I don't see what we can delete in prometheus data without breaking it

#

might be worth checking what size is /data/graphite/

#

or maybe tune2fs and remove the 5% reserve it might have

keen crescent Mar 29, 2023, 4:12 PM

#

I sent you a question in teams about python versions

#

yes our nabox system does not have much fat

storm merlin Mar 29, 2023, 4:14 PM

#

Doesn't need fat really, if something goes wrong it's usually in the logs, that you can clean anyways

#

well, /data is another story

brave remnant Mar 29, 2023, 4:15 PM

#

Yeah its the data part thats rammed

keen crescent Mar 29, 2023, 4:16 PM

#

yep i see that's a different mount so cleaning up anything on / does not help

storm merlin Mar 29, 2023, 4:16 PM

#

anything you can delete like /data/graphite ?

#

another way which is some work is copy over the prometheus data and run it so you can clean it u, then copy it bac

#

or mount a NFS share, copy over, and do the same

#

I'd like to see du -hs /data/* anyways

brave remnant Mar 29, 2023, 4:19 PM

#

storm merlin Mar 29, 2023, 4:19 PM

#

me might be able to remove the partition reserve

#

tune2fs -l /dev/vg_data/lv_data|grep Rese

keen crescent Mar 29, 2023, 4:21 PM

#

and as mentioned earlier, if you are not using Harvest 1.X you can rm /data/graphite - that might be enough to allow Prom to restart

#

i would try that first

storm merlin Mar 29, 2023, 4:21 PM

#

exactly, 173M might be enough to fix it

brave remnant Mar 29, 2023, 4:21 PM

#

Harvest 2 Im using so I could give that a go

storm merlin Mar 29, 2023, 4:21 PM

#

make sure you stop harvest pollers before

#

Now if the goes it to clean up all prometheus data you might as well delete everything in /data/prometheus

#

removing graphite makes only sense if you want to selectivvely delete prom data like indicated in https://nabox.org/faq/#delete-data

brave remnant Mar 29, 2023, 4:26 PM

#

Pollers are not even loading, the web gui for the systems is just blank

storm merlin Mar 29, 2023, 4:26 PM

#

all right

#

dc stop nabox-harvest2; rm -rf /data/graphite/*; dc start prometheus

brave remnant Mar 29, 2023, 4:31 PM

#

Same error, not enough space removed

storm merlin Mar 29, 2023, 4:31 PM

#

dang it

#

tune2fs -l /dev/vg_data/lv_data|grep Rese

brave remnant Mar 29, 2023, 4:32 PM

#

storm merlin Mar 29, 2023, 4:33 PM

#

tune2fs -m 1 /dev/vg_data/lv_data then start again

brave remnant Mar 29, 2023, 4:37 PM

#

thats done the trick!

storm merlin Mar 29, 2023, 4:37 PM

#

Quick !!!

brave remnant Mar 29, 2023, 4:42 PM

#

storm merlin Mar 29, 2023, 4:47 PM

#

Don't forget curl -k -L -X POST http://localhost/prometheus/api/v1/admin/tsdb/clean_tombstones

#

And tune2fs -m 5 /dev/vg_data/lv_data

#

FYI I fixed the blank page when graphite/prom/grafana aren't responding

brave remnant Mar 29, 2023, 5:54 PM

#

Thank you 81GB back 🙂 until the let us buy more GB of storage Would it be an easy job to use NFSV4 as the area?

storm merlin Mar 29, 2023, 9:34 PM

#

https://nabox.org/faq/#reset-password

#Nabox 95% full, How do I purge old data?