#NABox Migration, Unable to find image 'victoriametrics/

1 messages · Page 1 of 1 (latest)

hoary ridge
#

While migrating from NABox 3.5.4 to 4.0.7, unable to migrate historical telemetry

This is an internet disconnected site with ~870GB of historical telemetry to move to the new instance.

Automated Migration successfully adds ‘pollers’, but fails the Prometheus snapshot.

/usr/local/nabox/docker-compose # migrate
Restarting Prometheus with admin API enabled... OK
Waiting for Prometheus.................... Failed
Taking snapshot...time=2024-12-19T00:28:44.667Z level=ERROR source=migrator.go:59 msg="Failed to take Prometheus snapshot" error="Failed to take prometheus snapshot: %!w(<nil>)"

The Manual migration has a successful snapshot but fails with the following errors

/usr/local/nabox/docker-compose # docker run --rm -v /data/prometheus:/prometheus -it --network docker-compose_default victoriame
trics/vmctl:heads-master-0-g93a29fce4e prometheus --prom-snapshot=$SNAPSHOT_PATH --vm-addr=https://$NABOX_IP/vm/ --vm-insecure-sk
ip-verify -s
Unable to find image 'victoriametrics/vmctl:heads-master-0-g93a29fce4e' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host.
See 'docker run --help'.
/usr/local/nabox/docker-compose #

Deviations from documented process include the following

  •      Add snapshot path as per what is returned

  •      Add target IP to URL in ‘docker run’ string (instead of variable)

  •      Add ‘dash’ to container name (ie, victoria-metrics/vmctl)

  •      Append ‘heads-master-0-g93a29fce4e’ with container ID (this is might be nonsensical, we don’t know a lot about containers)

Tests performed

  •      Curl to victoria-metrics from origin is successful. (ie. curl –k https://$NABOX_IP/vm/)

  •      Curl to https://registry-1.docker.io/v2/, fails (disconnected site)
compact fossil
#

@analog stump

analog stump
#

Please re apply NAbox 3 update, the image is missing it happens when upgrades partially fails.

hoary ridge
#

Thankyou, the migration has now started, may have to leave it over the weekend.

hoary ridge
#

The migration completed successfully, however Grafana is no longer functional on the 4.0.7 side with 'Bad Gateway' error in the browser when attempting to reach the Dashboards. Attempted reinstall of 4.0.7 with no effect. Grafana version in 'About', also not reporting a value (ie. '?').

analog stump
#

what does dc logs -f grafana say ?

hoary ridge
#

Some extracts from the grafan logs

#

[36mgrafana | [0mlogger=plugin.store t=2024-12-23T08:10:26.406093186Z level=info msg="Plugins loaded" count=54 duration=35.707894ms
[36mgrafana | [0mError: ✗ invalid email address for SMTP from_address config
[36mgrafana | [0mGrafana server is running with elevated privileges. This is not recommended
[36mgrafana | [0mlogger=settings t=2024-12-23T08:11:26.743916209Z level=info msg="Starting Grafana" version=11.2.0 commit=2a88694fd3ced0335bf3726cc5d0adc2d1858855 branch=v11.2.x compiled=2024-12-23T08:11:26Z

[36mgrafana | [0mlogger=plugin.store t=2024-12-23T08:11:26.813222484Z level=info msg="Loading plugins..."
[36mgrafana | [0mlogger=plugins.registration t=2024-12-23T08:11:26.847596209Z level=error msg="Could not register plugin" pluginId=xychart error="plugin xychart is already registered"
[36mgrafana | [0mlogger=plugins.initialization t=2024-12-23T08:11:26.847628556Z level=error msg="Could not initialize plugin" pluginId=xychart error="plugin xychart is already registered"
[36mgrafana | [0mlogger=local.finder t=2024-12-23T08:11:26.847694934Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled

analog stump
#

mmm, the plugin thing doesn't seem right

hoary ridge
#

Thankyou. I have uploaded a chunk of the log file. The full (80MB) file requires additional process endorsement. Hopefully this extract helps.

analog stump
#

Looks like there is an invalide email address in SMTP configuration, can you fix that ? What does it look like in NAbox email settings ?

hoary ridge
#

Oh Ok, so would this cause the ‘Bad Gateway’ error?

analog stump
#

It's not clear in the log if Grafana actually crashes, so I'm assuming a "error" level log is terminating Grafana