Prometheus Remote Write configuration | NetApp | Page 1

carmine zinc Jan 22, 2024, 9:09 AM

#

Are there any templates for Prometheus Remote Write for ONTAP data and is the Remote Write configuration preserved when updating NAbox?

wintry carbon Jan 22, 2024, 10:36 AM

#

Remote write configuration wouldn’t be preserved across updates. I could probably implement an option to send metrics to a remote endpoint. In that case you’d be running only Harvest in NAbox and that might not be worth it ?

carmine zinc Jan 22, 2024, 10:47 AM

#

We wish to send a subset of data in Prometheus to another Prometheus environment and I'm told this can be done with Remote Write. Does that make sense?

carmine zinc Feb 11, 2025, 11:40 AM

#

@wintry carbon Any update here? I'm looking at this again and want to send Victoria metrics data to a Prometheus instance.

wintry carbon Feb 11, 2025, 11:42 AM

#

Sorry I missed your message. Yes, that's what remote writes are for. I need to check how to implement that in NAbox so it's not reverted on upgrades though

carmine zinc Feb 11, 2025, 11:43 AM

#

No worries, it happens. But do you know how to add it non-persistent in NAbox v4?

#

We managed to do it in NAbox v3 but not with the flatcar thing I'm lost

wintry carbon Feb 11, 2025, 11:55 AM

#

tricky one actually. VM decouples things a lot more than prometheus, meaning that they have more specialized components than just a single software.
Remote writes are handled by vmagent, not victoria-metrics main software. vmagent is shipped with NAbox, as it is used to anipulate metrics coming from remote NAbox 3 instances during migration, but that's it, harvest itself is collected by victoria-metrics, not vmagent.
The solution, I believe, would be to collect metrics with the agent instead of vm and that's not a trivial change. I'm not 100%, let me do some more research

carmine zinc Feb 11, 2025, 12:49 PM

#

Interesting, thank you for the explanation.
Would it be a better idea to do the other way around, to let the remote environment fetch data from NAbox?
A pain point in that case would be that we need to limit the data they have access to.

broken solar Feb 11, 2025, 3:09 PM

#

I do remote write from prometheus to VM, and only send a subset of data. Works great. Feel free to ping me

#

Not with NAbox though, so that probably complicates things

carmine zinc Feb 11, 2025, 3:26 PM

#

I want data to flow from VM (in NAbox) to Prometheus and the NAbox part is probably complicating things, yeah 🙂
I think @wintry carbon will have to act as mediator here as I am not familiar with the components involved.

broken solar Feb 11, 2025, 4:01 PM

#

For me, it's just a few lines in the prometheus conf file, and the target (VM in my case) doesn't even need to know about it. VM just accepts it by default

#

If I had to guess, you'd have to manually update your nabox VM config file to point it to your remote write target and do the relabeling for only sending a subset. Upgrade/revert would probably involve manually adding those config items back to the file, but I'm not sure how NAbox handles the VM config files for uograde/revert 🤔

#

Oh I just saw Yann's message about VM itself not being a sender of remote write. Ok maybe don't listen to anything I have to say. Sorry 😟

carmine zinc Feb 12, 2025, 8:13 AM

#

What you are saying was true for NAbox v3, in v4 it works a bit differently.

carmine zinc Feb 19, 2025, 11:44 AM

#

@wintry carbon Sorry to ping you but have you taken any time to investigate if it would be possible to implement Remote Writes?

wintry carbon Feb 19, 2025, 1:30 PM

#

Sorry I haven't honestly ! Can't make any promise for now, but I'll open an issue for it

carmine zinc Mar 5, 2025, 1:20 PM

#

Is that an issue on some github or internally?
Was wondering if I could follow it.

wintry carbon Mar 6, 2025, 9:38 AM

#

Internal I'm afraid

carmine zinc Jun 9, 2025, 2:53 PM

#

wintry carbon Internal I'm afraid

Any news here or is the issue dropped?

wintry carbon Jun 10, 2025, 11:22 AM

#

Not dropped, there is an issue opened for it but not started either !

wise trout Aug 7, 2025, 8:40 AM

#

Hello Team, i find myself in the same situation: meaning i would need to remote-write metrics data from the NAbox to another server for metrics collection and looking for best practices & options, any suggestions or feedback is appreciated! cheers //KG

young walrus Aug 7, 2025, 9:42 AM

#

vmagent is a good option, as discussed above, for reading metrics from the NABox VictoriaMetrics instance and writing them to another server.

wise trout Aug 7, 2025, 11:10 AM

#

thank you for the quick response Rahul, might be a stupid question: but would i have to re-lable the metrics using this way? my intention would be to copy the needed dash-boards from the internal grafana on NAbox to our other grafana instance, so i can avoid having to set them up manually

young walrus Aug 7, 2025, 11:23 AM

#

There is no need to relabel. Metrics will be copied as-is to the new VictoriaMetrics using vmagent. Your new Grafana instance will simply point to the new VictoriaMetrics server.

wise trout Aug 15, 2025, 2:04 PM

#

Hello guys, I’m still having some issues here, (and im also new to this, so please bear with me) anyway im trying to run a second vmagent container that scrapes the same metrics as the internal vmagent on the NAbox and remote-writes them to our centralized metrics server.

What I’ve tried:

Separate YAMLs (vmagent.yml for scrape, remote_write.yml for remote_write) → error: flag provided but not defined: -remoteWrite.config

Combined YAML (scrape + remote_write) → error: at least one -remoteWrite.url command-line flag must be set

Using CLI flags directly (-remoteWrite.url=…, -remoteWrite.header=…) → errors: flag provided but not defined

Environment: NAbox v4.0.10, Grafana 11.5.2, VictoriaMetrics 2.24.0, vmagent v1.112.0

Goal: Have a second vmagent push the same metrics to a central server.

Any recommended approach or documentation that would put me on the correct path of solving this? Do we need a newer vmagent/NAbox, or is there a supported method for this setup? cheers, Karl

remote cobalt Aug 15, 2025, 2:10 PM

#

@wintry carbon

wintry carbon Aug 15, 2025, 2:13 PM

#

Allow me some time to review the whole thing 🙂

wise trout Aug 15, 2025, 2:15 PM

#

cool 🙂 tnx

wise trout Aug 21, 2025, 11:22 AM

#

Hello again guys,

I’m considering a possible workaround for my use case. The idea is to set up a new server running a more recent version of vmagent (one that supports the necessary remote_write flags, including custom headers). This vmagent would then scrape metrics via the NAbox proxy endpoints and forward them to our centralized metrics server.

Do you see any issues with this type of setup, or would you consider it a recommended approach? Any feedback is much appreciated.

Cheers, Karl

carmine zinc Aug 21, 2025, 1:52 PM

#

wise trout Hello again guys, I’m considering a possible workaround for my use case. The id...

If you get this working I am very much interested in a guide for how to do it, I don't know much about setting up that kind of thing so I'm glad NAbox exists.

remote cobalt Aug 21, 2025, 6:11 PM

#

@wise trout I've not tried it, but that sounds reasonable. Another option, edit the /etc/nabox/compose.yaml file on nabox and add another - -remoteWrite.url=http://victoria-metrics:8428/vm/api/v1/write to your destination. The caveat with this is when you update nabox it will overwrite your edits

wise trout Aug 22, 2025, 12:02 PM

#

thank you Chris, yes i did try to edit the compose.yaml but it seems vmagent would not launch with the needed remotewrite flags, it just kept crashing. anyway i believe a separate server setup will be easier to handle for future upgrades, as you said any changes will be overwritten on the NAbox itself

remote cobalt Aug 22, 2025, 12:04 PM

#

At any time in the future, if you end up wanting help editing the compose file, let us know. That should work

wise trout Aug 22, 2025, 12:06 PM

#

ok tnx, appreciate it, ill let you know how everything turns out

wise trout Aug 29, 2025, 9:39 AM

#

Hey, I’m still working on forwarding NAbox metrics to a central Metrics-collector. I now have a second RHEL server running with vmagent, but I’m having trouble reaching the Harvest proxy endpoints on the NAbox server.

If I hit https://<nabox IP>/havrest/proxy/<ports defined in harvest.yml>/metrics with curl, I always get redirected to /login.

Basic auth (-u user:pass) doesn’t work (still redirected).

Direct http://localhost:<port>/metrics on the NAbox host is refused (not bound).

Is there a way to authenticate a non-browser client (like vmagent) directly against the Harvest proxy endpoints, or should I use another approach?

cheers!

slow crescent Aug 29, 2025, 10:10 AM

#

@wintry carbon

wintry carbon Sep 2, 2025, 9:54 AM

#

Let me take a look, if you can, upgrade to 4.0.11 which brings API token management and make those easier

wise trout Sep 2, 2025, 1:33 PM

#

Hey Yann, ok i will have a look at updating the NAboxes

wise trout Sep 10, 2025, 6:28 AM

#

Hey Guys, updated NAbox, the API token is not accepted by harvest proxy endpoints, seems it still expects a web session cookie, not a bearer token or basic auth.

remote cobalt Sep 15, 2025, 11:39 AM

#

@wintry carbon

wintry carbon Sep 16, 2025, 7:06 AM

#

Checking

#

Seems to be working here.

❯ curl -v -k -I -H "Authorization: Bearer 3558f12e920f41e49d3121a41343af06" https://nabox4.home.lab/havrest/proxy/12990/metrics
* Host nabox4.home.lab:443 was resolved.
* IPv6: (none)
* IPv4: 10.1.1.23
*   Trying 10.1.1.23:443...
* Connected to nabox4.home.lab (10.1.1.23) port 443
* ALPN: curl offers h2,http/1.1
[...]
* [HTTP/2] [1] OPENED stream for https://nabox4.home.lab/havrest/proxy/12990/metrics
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nabox4.home.lab]
* [HTTP/2] [1] [:path: /havrest/proxy/12990/metrics]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [authorization: Bearer 3558f12e920f41e49d3121a41343af06]
> HEAD /havrest/proxy/12990/metrics HTTP/2
> Host: nabox4.home.lab
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer 3558f12e920f41e49d3121a41343af06
>
* Request completely sent off
< HTTP/2 200
HTTP/2 200
< date: Tue, 16 Sep 2025 07:18:56 GMT
date: Tue, 16 Sep 2025 07:18:56 GMT
< strict-transport-security: max-age=315360000; includeSubDomains; preload
strict-transport-security: max-age=315360000; includeSubDomains; preload
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: DENY
x-frame-options: DENY
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
<

wise trout Sep 16, 2025, 8:22 PM

#

hum ok, ill try again tomorrow to see whats going on for me. cheers

wise trout Sep 18, 2025, 7:55 AM

#

[<hidden> ~]$ curl -v -k -H 'Authorization: Bearer c4a98beb82db4f608029ace4673a50cf' <hidden>/havrest/proxy/12001/metrics

Trying <hidden>:443...
Connected to <hidden> (<hidden>) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/pki/tls/certs/ca-bundle.crt
TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS header, Certificate Status (22):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS header, Finished (20):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.2 (OUT), TLS header, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS header, Unknown (23):
TLSv1.3 (OUT), TLS handshake, Finished (20):

#

SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
ALPN, server accepted to use h2
Server certificate:
subject: C=US; ST=NY; L=New York; O=NetApp; OU=NAbox; CN=nabox
start date: Apr 30 08:56:27 2025 GMT
expire date: May 1 08:56:27 2026 GMT
issuer: C=US; ST=NY; L=New York; O=NetApp; OU=NAbox; CN=nabox
SSL certificate verify result: self-signed certificate (18), continuing anyway.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.2 (OUT), TLS header, Unknown (23):
TLSv1.2 (OUT), TLS header, Unknown (23):
TLSv1.2 (OUT), TLS header, Unknown (23):
Using Stream ID: 1 (easy handle 0x55c123a48fd0)
TLSv1.2 (OUT), TLS header, Unknown (23):

GET /havrest/proxy/12001/metrics HTTP/2
Host: <hidden>
user-agent: curl/7.76.1
accept: /
authorization: Bearer c4a98beb82db4f608029ace4673a50cf

TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.2 (IN), TLS header, Unknown (23):
Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
TLSv1.2 (OUT), TLS header, Unknown (23):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.2 (IN), TLS header, Unknown (23):
TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 302
< date: Thu, 18 Sep 2025 07:42:54 GMT
< location: https://<hidden>/login?redirect=/havrest/proxy/12001/metrics
< strict-transport-security: max-age=315360000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-xss-protection: 1; mode=block
< content-length: 0
<
Connection #0 to host <hidden> left intact

#

still getting redirected to login page, tried using an incorrect token and it gives the same response

#

and i did a full reboot of the Nabox server after the in-place upgrade to the latest version

#

any suggestions on how to continue the troubleshooting?

wise trout Sep 22, 2025, 2:17 PM

#

ping: any suggestions here? is there any settings in the proxy that i need to do to allow token authentication?

young walrus Sep 23, 2025, 7:37 AM

#

@wintry carbon

wintry carbon Sep 23, 2025, 7:40 AM

#

Thanks Rahul

Karl, can you check your DMs you should have a link I sent last week to book some time in my calendar

wise trout Sep 23, 2025, 1:35 PM

#

cool will check

#

booked now, sorry i did not notice my DM:s before.

versed hull Oct 30, 2025, 2:08 PM

#

Hello @Karl did you solve your issue with 2 remote write config ? I'm actually testing it to keep my nabox vm things, and push it to an other grafana.

carmine zinc Apr 14, 2026, 2:18 PM

#

@wintry carbon Hey! Is the Remote write from NAbox perhaps more approachable now that Harvest supports VictoriaMetrics in push mode?
Not sure I understand if it is related but thought I'd ask.

remote cobalt Apr 14, 2026, 2:42 PM

#

hi @carmine zinc these aren't related, but you might be able to use Harvest's VM push mode to replace your need for remote_write

carmine zinc Apr 14, 2026, 3:02 PM

#

Ooh ok, are there any documentation on how to enable this and make it persistent in NAbox?

#Prometheus Remote Write configuration