Prometheus remote_write question

[tawny grove]

Hi, we’re doing a proof of concept for using Harvest with test systems and visualizing and analyzing and archiving metrics from our test systems.
I posed this question on the CNCF Slack channel for #prometheus, but I figured I should ask here too:

I am new to Prometheus, but I have a use case question, and I am not able to find any info about if my idea is on the right track or even possible.

I have a Prometheus instance that scrapes data and stores it locally. It is collecting metrics for a system under test. Then after some time, a different process will make a decision (i.e. "did the test pass?"), and based on that yes/no decision, I want to take the data that was collected/stored locally and send that off to a more permanent Prometheus instance. I only want to send the data if the decision is a yes. I'm not sure if "remote_write" is the answer. Is it possible to store the data locally for some time, and then restart the instance with a remote_write config option? Will it send the already-collected local data? Or does it only send the real-time scraped data?

Are there any other options for transferring data from one Prometheus instance to another? I know this is barbaric, but perhaps some sort of file copy from the docker volume “harvest_prometheus_data” ?

[tawny grove]

It seems that remote_write only sends the real-time metrics and not the historical TSDB blocks on disk. The replies on the Prometheus slack channel seem to indicate that the more “Prometheus” way to do what I’m looking for is to collect everything and then decide later what to keep. Like one idea is to use the delete API to remove labels that are not wanted.

[cursive shell]

It's also possible to backup/restore an instance of Prometheus as outlined https://github.com/NetApp/harvest/blob/main/docs/MigratePrometheusDocker.md But that's not quite what you're asking. You're asking can you backup instance1, instance2, etc. and restore all of them on instance99 That I have not tried.

Might be worth checking out VictoriaMetrics too. It may better support your usecase