supple pelican
Hey all, I'm trying to graph out SMB usage over a very large period of time. I get this error indicating I need to increase the max value, but I didn't want this to be a global change (worried about performance impacts overall) I was just hoping to be able to run this manually, but my knowledge of VictoriaMetrics is fairly limited. Is there an easy command I can run to expand that limit as a one-off process in /vm/vmui or something?
I'm just now noticing that I seem to be pretty lmited on a couple of the SMB graphs (I assume that these are massive datasets due to the number of SMB connections) but I'm pretty underutilized on the NABox appliance, would increasing this max value affect anything other than the cpu/memory utilization of the NABox appliance?
dull kayak
hi @supple pelican my read of the docs on https://docs.victoriametrics.com/#resource-usage-limits imply that you can safely increase search.maxUniqueTimeseries .
If you increase it too much, you may hit out of memory errors. I found more details here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6930
That issue also implies that if you give Nabox more memory, this value will automatically increase too
supple pelican
Thanks Chris, I might try that since I've already got 12g RAM, 6 cpu and it's barely using what's there. I would have thought it would have increased to utilize what's there already...
It looks like the "connection count" graph is already running into the memory issue at 7 days, but when I increase to 30, it claims to be hitting the max limit. However it doesn't actually seem to be utilizing that RAM (since the value doesn't increase)...
Ok, I'll look into seeing what the increase will do.
Do you know where the commands for increasing these configs are off the top of your head? If not, I can dig more into the documentation to see if I can get those specifics
dull kayak
I think you're using Nabox, right? If so, @frank coral can chime in where they are configured there
supple pelican
I think you're using Nabox, right? If so, <@323825973718155265> can chime in whe...
Thank you! Yes, Nabox. Just upgraded to 4.0.9
frank coral
That’s a Grafana error or a VM error ?
dull kayak
Grafana showing a warning/error from VM
frank coral
Ok makes sense.
supple pelican
it looks like a VM error passed through the Grafana interface.
frank coral
You can pass custom arguments to VM from within /etc/nabox/compose.yaml but not sure it’s reasonable that you’re hitting that limit
supple pelican
It certainly seems like I'd be utilizing more memory on the system before running into memory errors... I can walk the timing up, increasing by a day, and eventually it just gives the 'out of memory' error, but it never uses all the memory up until that point, so I'm not sure if it's just checking to make sure the memory is there as a safety blanket or something... 🤣 but I don't seem to be resource constrained
dull kayak
that issue I shared, made it sound like VM sniffs the amount of memory and sets that limit at start-up. Another option would be to check the VM logs and see what value it is currently using and increase that value without adding more memory
you should be able to skip Grafana altogether and go straight to VM via https://$vm-ip/vm/vmui/ and then use the cifs_session_connection_count query.
Ryan, I agree with your suspicion that this template is causing a large number of time-series due to the fact that it includes labels for client_ip and connection_id. That means if you have lots of clients collecting, you will create a time-series for each of them. That's one of the reasons we disable this template by default
supple pelican
you should be able to skip Grafana altogether and go straight to VM via https://...
It's definitely not a Grafana issue, I need to look into increasing the search.maxUniqueTimeseries parameter. @frank coral is there a way to do this within the Nabox implementation of VM?
frank coral
Yes you can add that flag to compose.yaml like I suggested earlier. There is a section for Victoria metrics with already some command line arguments used at run time, you can add your own.
dull kayak
/etc/nabox/compose.yaml
supple pelican
ah, sorry, I got confused with the 'not sure it's reasonable you're hitting that limit' statement, so I thought that wasn't the right place.
Is there any way to see what the existing number is?
or is it the 2084775 from the error message?
Please let me know if this seems like it's the right format for passing this command in:
platform: linux/amd64
image: victoria-metrics
command:
- -search.maxUniqueTimeSeries=2500000 <==added this line
- -envflag.enable
- -vmalert.proxyURL=http://vmalert:8880/
- -http.pathPrefix=/vm
- -promscrape.config=/config/prometheus.yml
container_name: victoria-metrics
hostname: victoria-metrics
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "5"```then I assume a dc restart havrest
(is the havrest a typo that just survived forever, or an inside joke about harvest?! 🤣 it messes with my brain every time I have to type it)
supple pelican
Sorry gents, I think this is a @frank coral question, I just don't want to mess it up.
supple pelican
“- -arg”
Updated! I wasn't sure if the - - was something like commenting it out and those weren't active or anything. Does that look right now?
supple pelican
@frank coral, I updated the compose.yaml with the line below, and did a dc restart havrest but when I run the query, the error is still giving the 2084775 number as the limit...
platform: linux/amd64
image: victoria-metrics
command:
- -search.maxUniqueTimeseries=2500000
- -envflag.enable
- -vmalert.proxyURL=http://vmalert:8880/
- -http.pathPrefix=/vm
- -promscrape.config=/config/prometheus.yml
I even tried updating the compose.yaml.local file with a similar (but slightly different) number and resetting again, no change. Let me know if there's another location I should be adjusting this value (or if I should be looking at another value...)
frank coral
dc restart victoria-metrics
Watch for the indent too. Make sure it’s aligned with envflag
supple pelican
dc restart victoria-metrics
Screenshot to show the indenting, i think it's correct.. still getting the old number on the error message, even after the restart of victoria-metrics
supple pelican
Should I look at increasing the memory.allowedPercent counter too?
How can I check the current values of these VM values?
supple pelican
Ok, I got it to load the settings from the compose.yaml file, but it required a full reboot. dc restart didn't do it. I needed to reboot the whole system (I used the GUI)
I'll play with these numbers, but it would be nice to be able to see what the current victoriametrics settings are for them...
dull kayak
you should be able to see some of the values in the log files according to that issue I pasted earlier. docker logs victoria-metrics
supple pelican
I'll go look, Thanks @dull kayak
frank coral
All right I'm back from business travel, I can spend some time on this
worldly hull
Hi There, just to share the setting i have for victoria metrics - "--search.maxQueryLen=99513250"
- "--search.maxUniqueTimeseries=4900000"
- "--search.maxPointsPerTimeseries=190000000"
- "--search.maxSamplesPerQuery=300000000"
- "--search.maxSeries=30000000"
- "--search.maxQueryDuration=300s"
- "--search.logSlowQueryDuration=60s"
- "--maxInsertRequestSize=67108864"
i'n not in nabox, it is a VM with 64g of RAM, when it is really loaded i can go to 6O% memory utilization, but we need to take into account the numbers of clusters pooled, i have about 100. In Any case i used "canary testing" 🙂