Containerized harvest upgrade

[crude karma]

Hello, I was following this procedure to upgrade harvest https://netapp.github.io/harvest/23.05/install/containers/#upgrade-harvest

Since I would like to mantain historical data (source version was 22.05.0-1) I followed that doc:https://github.com/NetApp/harvest/blob/main/docs/MigratePrometheusDocker.md

I have created harvest_prometheus_data, copied old data, but when I start everything it seems prometheus is using volume harvest-230324-nightly_linux_amd64_prometheus_data.

How can I link the volume harvest_prometheus_data to prometheus container?

Thanks a lot

[gusty nacelle]

hi @crude karma can you paste your prom-stack.yml file?

[crude karma]

Hi @gusty nacelle , the prom-stack.yml had some customization so it's copied from the old version

volumes:
    prometheus_data: {}
    grafana_data: {}
    harvest: {}

networks:
    frontend:
    backend:

services:
    prometheus:
        container_name: prometheus
        image: prom/prometheus:v2.33.1
        volumes:
            - ./docker/prometheus/:/etc/prometheus/
            - prometheus_data:/prometheus
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--storage.tsdb.retention.time=90d'
            - '--web.console.libraries=/usr/share/prometheus/console_libraries'
            - '--web.console.templates=/usr/share/prometheus/consoles'
            # - '--web.enable-admin-api' # Enable to delete time series data from Prometheus see https://www.robustperception.io/deleting-time-series-from-prometheus
        ports:
            - 9090:9090
        networks:
            - backend
        restart: unless-stopped

    grafana:
        container_name: grafana
        image: grafana/grafana:8.3.4
        depends_on:
            - prometheus
        ports:
            - 3000:3000
        volumes:
            - grafana_data:/var/lib/grafana
            - ./grafana:/etc/grafana/provisioning # import Harvest dashboards
            - /etc/ssl/stardadaeu:/etc/ssl/stardadaeu:rw
        environment:
[...]
        networks:
            - backend
            - frontend
        restart: unless-stopped
        labels:
            kompose.service.type: nodeport```

[gusty nacelle]

that's probably the issue - prom-stack.yml included changes for this prometheus volume change. Try changing your local copy to match this https://github.com/NetApp/harvest/blob/5824aaadacf52d2fda7b6dbd2612852588cfa77b/prom-stack.tmpl#L4

[crude karma]

@gusty nacelle I tried the change but unfortunately I have still that links 😦

harvest-22050-1_linux_amd64_grafana_data             0         3.177MB
harvest-22050-1_linux_amd64_prometheus_data          0         59.8GB
root_prometheus_data                                 0         0B
root_grafana_data                                    0         0B
harvest-230324-nightly_linux_amd64_prometheus_data   1         72.62MB
harvest-22050-1_linux_amd64_harvest                  0         0B
harvest_prometheus_data                              0         59.8GB
harvest-230324-nightly_linux_amd64_grafana_data      1         3.934MB
harvest_grafana_data                                 0         0B```

[gusty nacelle]

after editing your prom-stack.yml did you stop the previously running Prom container?
docker stop (docker ps -fname=prometheus -q)

[crude karma]

yes

docker inspect give me the following mounts

       "Mounts": [
            {
                "Type": "volume",
                "Name": "harvest-230324-nightly_linux_amd64_prometheus_data",
                "Source": "/var/lib/docker/volumes/harvest-230324-nightly_linux_amd64_prometheus_data/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/opt/harvest-23.03.24-nightly_linux_amd64/docker/prometheus",
                "Destination": "/etc/prometheus",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

[gusty nacelle]

docker inspect prometheus "Mounts": [ { "Type": "bind", "Source": "/Users/cgrindst/opt/harvest/docker/prometheus", "Destination": "/etc/prometheus", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "volume", "Name": "harvest_prometheus_data", "Source": "/var/lib/docker/volumes/harvest_prometheus_data/_data", "Destination": "/prometheus", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" } ],

these are the only differences between out of box prom-stack.yml and yours. And you changed lines 4-6 to match the left side. Hmmmm

your docker inspect does not match your prom-stack.yml though which is why I thought maybe it was not stopped/rm-ed from earlier per this step. https://github.com/NetApp/harvest/blob/main/docs/MigratePrometheusDocker.md#stop-prometheus-container When I say it does not match, I would expect your mounts.name to be harvest_prometheus_data Maybe you stopped the Prometheus container but did not remove it?

[crude karma]

I'm pretty sure it was removed, maybe I can rebuild it

[gusty nacelle]

yeah, trying to think of other reasons the inspect doesn't match the compose yml

[gusty nacelle]

We've not been able to recreate locally. When you get a chance, can you try again? Something like this. If you can capture the output and paste here that would be helpful with troubleshooting
`docker stop (docker ps -fname=prometheus -q) && docker rm (docker ps -a -fname=prometheus -q)
docker ps -a
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
docker ps -a

if you have jq installed

docker inspect prometheus | jq '.[].Mounts'

otherwise

docker inspect prometheus`

[crude karma]

Hi Chris, I think we fixed! Maybe the issue was I started prometheus the first time with the wrong prom-stack.yml, so your first suggestion was right 😉

I'll send you the output via pm if you would like to review