I have one node that can't get up to the newest version from the edge repo.
i've tried the kickstart script multiple times but still not there.
#ubuntu 20.04 kickstart installs old version of agent.
light edge
mossy orbit
Can you share what version number you are getting on the affected system?
light edge
yes
mossy orbit
Also, just to run through the usual checklist:
- Is the affected system behind a proxy?
- Is the affected system geolocated differently from any you see working correctly?
- Are there any other differences between the affected system and any working systems in terms of network setup?
light edge
no proxy
all from same datacenter
10% identical servers
so, @mossy orbit in the UI it reports as v1.36.0-211-gc805a9afa
but there are no pkg installed when looking in dpkg
Sorting... Done
Full Text Search... Done
netdata/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (metapackage)
netdata-apache2/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (apache integration)
netdata-core/focal 1.19.0-3ubuntu1 amd64
real-time performance monitoring (core)
netdata-plugins-bash/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (bash plugins)
netdata-plugins-nodejs/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (nodejs plugins)
netdata-plugins-python/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (python plugins)
netdata-web/focal 1.19.0-3ubuntu1 all
real-time performance monitoring (web)
Hmm, that looks like you don’t even have the Netdata repos at all there.
light edge
i don't have the apt key either
mossy orbit
Can you share the contents of /etc/netdata/.install-type from the affected system?
light edge
that folder is not created
:S
# ps aux |grep netdata
netdata 412899 2.1 0.0 318276 103908 ? SNsl 13:26 8:26 /opt/netdata/bin/srv/netdata -P /run/netdata/netdata.pid -D
netdata 412918 0.0 0.0 44684 1472 ? SNl 13:26 0:00 /opt/netdata/bin/srv/netdata --special-spawn-server
netdata 413141 0.0 0.0 44880 28896 ? SNl 13:26 0:16 /usr/bin/python3 /opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin 1
netdata 413145 0.1 0.0 770732 46568 ? SNl 13:26 0:40 /opt/netdata/usr/libexec/netdata/plugins.d/go.d.plugin 1
root 413154 1.2 0.0 57308 5240 ? SNl 13:26 4:55 /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin 1
root 413187 0.0 0.0 182000 14352 ? SNl 13:26 0:16 /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1
netdata 531393 0.1 0.0 1908 1424 ? SN 19:26 0:02 bash /opt/netdata/usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
root 543164 0.0 0.0 9168 660 pts/0 S+ 19:59 0:00 grep --color=auto netdata
let me try remove that folder
and run the kicstart
mossy orbit
OK, looks like it’s falling back to a static install for some reason.
light edge
so
mossy orbit
Next thing I was going to ask was to see the output from running the kickstart script.
light edge
i just removed /opt/netdata
and running the curl script
look at this.
then i hit y.
mossy orbit
Odd, that looks like installing the keys is not working correctly...
mossy orbit
Yeah, it’s falling back to a static build, which puts everything under /opt/netdata by default.
light edge
yup
mossy orbit
Can you try removing Netdata again, downloading the kickstart script, and then running it like:
sh -x kickstart.sh --native-only
The --native-only switch will force it to only attempt to use native packages, while the sh -x will turn on execution tracing (which should hopefully tell us what’s going on here).
mossy orbit
OK, I think I know what’s going on here.
Can you double check and see if the APT configuration on the affected system includes an APT source entry using a URL starting with https://packagecloud.io/netdata/netdata?
light edge
sources.list.d/netdata-edge.list:deb https://packagecloud.io/netdata/netdata-edge/ubuntu/ focal main
sources.list.d/netdata-edge.list:deb https://packagecloud.io/netdata/netdata-repoconfig/ubuntu/ focal main```i've added that manually
copied from another server that works
mossy orbit
Can you try removing that and rerunning the kickstart script?
mossy orbit
APT doesn’t do automatic key management like DNF or Zypper do, so adding a source means the key needs to be added at the same time, otherwise you get repository metadata update failures.
However, prior to installing the repository configuration, the kickstart script explicitly runs a metadata update because APT does not do this automatically when you tell it to install a package (and not doing so causes dependency resolution to fail if metadata has never been updated)., hence the failure we were seeing above.
Yep, looks like it worked that time.
Can you double check that Netdata is actually installed through APT?
light edge
yup
ii netdata 1.36.0-211-nightly amd64 real-time charts for system monitoring
ii netdata-repo-edge 1-2 all Configuration for the official Netdata Edge package repository.```seem good now
so what now? since i ran it with native-only
mossy orbit
1.36.0-211 is indeed the latest nightly.
Though I just now noticed that that’s also what you mentioned above.
light edge
well that was what the cloud reported
mossy orbit
Either way, re-running the original kickstart command should claim that node properly (and check for updates).
mossy orbit
Our nightly versioning gets a bit confusing at times though because of it not tracking patch releases due to how we actually handle patch releases.
For future reference, you can check what the latest nightly is by looking at the version listed in https://github.com/netdata/netdata/blob/master/packaging/version (that gets updated on every release and nightly build), though only the part before any -nightly is relevant (some install methods have a proper -nightly at the end, others end up with a -gxxxxxxxx indicating the commit they were built from).