Harvest installation has been a bit more of a challenge than I had anticipated or hoped. At his point I think I have Grafana and Prometheus working, but I can't, for the life of me, figure out why the i can't get the endpoint discovery listener up without manually starting it with 'harvest admin start'
#Admin listening on port 8887 doesn't start with Harvest
main wagon
zenith lagoon
hi @main wagon sorry to hear you're having setup issues. Can you say how you installed Harvest: Docker, RPM, Deb, Native, nabox? Or share the doc link that you're using to set things up if you have that handy
main wagon
well, since the rpm for prometheus on RH required too much wierd voodoo with systemd, I just did a manual install in /opt/prometheus and hacked the startup scripts... seems to work... graphana was an rpm and harvest was also rpm
using your docs except where the examples fail... so i had to did a bit for some of the prometheus parts
so, the official harvest docs
at this point grafana and prometheus aren't issues if i can't get the harvest autodiscovery listener to start
i guess i could enter targets manually
zenith lagoon
thanks, that context helps. We can get the admin node working for you. Not sure if you saw this, but it has some good trouble shooting steps for Prometheus HTTP service discovery https://netapp.github.io/harvest/nightly/help/troubleshooting/#prometheus-http-service-discovery-doesnt-work
main wagon
the collectors work
just not the autodiscovery listener
yamllint is fine, doctors sees no problems
i only get a listener if i manually start it with 'harvest admin start' ...
zenith lagoon
And I'm guessing that you followed these instructions https://netapp.github.io/harvest/nightly/prometheus-exporter/#prometheus-http-service-discovery
Back to your problem, you said "i can't get the endpoint discovery listener up without manually starting it with 'harvest admin start"
You need to run bin/harvest admin start to start the admin node. Once it is started, the pollers that start/stop will communicate with the admin node to update the list of running Harvest pollers.
Maybe your question is how do you start the admin node automatically instead of manually?
main wagon
yeah, i guess I assumed that when the configuration includes sufficient info for the "admin node" to start, it should do so...
i don't remember seeing anything that autodetection needed an extra manual start
zenith lagoon
gotcha - yes, nothing will start that process for you. Typically folks running native or package installs use systemd to start that process on machine restart. It's process managment, which is why I asked how you installed, since docker, packages, and native all have slightly different ways
main wagon
ok, so it needs to be an extra systemd script
could just be an option to the main script START_ADMIN_NODE=yes or the like, but I'm not the architect
zenith lagoon
which main script do you mean?
zenith lagoon
https://github.com/NetApp/harvest/blob/main/service/harvest.service
does systemd support environment variable conditionals? Not everyone wants to use admin node, some prefer controlling the static targets manually or by generating the list. I'm open to adding START_ADMIN_NODE=yes but I don't think systemd supports conditional logic. It could be implemented with another level of indirection by having the harvest.service exec a shell script that decides what to do based on envvars
If you start the admin node manually, is everything working for you?
main wagon
i cut-n-pasted the service script from github and it's up
i'm not a daily l33n0x admin anymore and sort have always hated sysV start scripts... i'd have to look to see what sort of conditional variables are necessary...
or harvest itself would just have to know if the listener is defined in the main harvest.yml file and start the admin if it defined... (ip/host:port)
now i just need to beat prometheus into listing to ipv4 ...
zenith lagoon
glad to hear everything working now 🎊 we looked at doing your 2nd suggestion but there's a race condition with doing it that way since each poller is independent and does not communicate with each other. Each poller reads the same harvest.yml but no communication between them
main wagon
yeah, i've specified 127.0.0.1:9090 in the config file but it is still only listening on :::9090
it's a linuxism to promote tcp6, but i hate policy... i'd rather just have tools
zenith lagoon
127.0.0.1:9090 should do the trick - check the prom log files and confirm that prom is using the config file you're changing and/or not overriding with a CLI option
main wagon
ok... the linuxism is that netstat displays things inconsisstently...
if only :::9090 is displayed, it's implicitly ipv4 and ipv6 ... yet you'll see explicit listings as well... 0.0.0.0:22 and :::22
this will all be containerized in the end and run on k8s, i just wanted to get something working
the docs could perhaps be more specific about what Harvest is... it's a set of pollers with an optional autodiscover daemon and some example dashboards... which is a load of great work, mind you, but the user needs to specifically install prometheus and grafana
i found some of the examples more than confusing, but i'm more a learner through osmosis... one good consistent example configuration is usually enough for me to get it
zenith lagoon
appreciate the feedback, we'll work on making that clearer. Docs are hard, we have a mix of all four types and depending on the person, their experience, etc they want different things https://documentation.divio.com/
main wagon
yeah, it's very persepective and experience related... we learn from what we already know
main wagon
i think i borked the dashboard import somehow... I keep getting an error: Templating: Failed to upgrade legacy queries
zenith lagoon
which version of Grafana did you pick?
zenith lagoon
Grafana 10.2.3 changed something about the case of datasources (we discovered this last week, still researching), I know several customers are using 10.2.2 without issue. Can you use 10.2.2 instead?
main wagon
I guess I could try, but it'll be a bit later... have to run out for a while...
main wagon
yeah, that didn't help...
I can do individual queries to the prometheus datasource in grafana, but all of the dashboards result in the same error
now when importing dashboards, one has the option "datasource" and the ambiguity here is that is this the "type" of datasource, i.e. Prometheus, or the "name" of the datasource? Me, being me, trying to think ahead a bit, chose to name the datasource "Harvest" and i see in the imported panels that the UUID of the query datasource is Prometheus by default, which I assume in this case is both the name and type of datasource...
nocturne trellis
@main wagon Do you mean import didnt work for 10.2.2 as well? Can you try updating datasource name in grafana to Prometheus. It may be prometheus (small p) in 10.2.3.
main wagon
again, it's more about what "datasource" means during the 'harvest grafana import' and it means the name of the datasource and not the type... if you call the datasource Prometheus and its type is Prometheus then you don't have issue... but if you call a Prometheus type of datasource something else, like "Harvest", then you have to use the --datasource Harvest to get the dashboards imported correctly
nocturne trellis
Yes it is name of datasource
You can pass name of datasource during import command if it is different than Prometheus
main wagon
yes... it's more about what "datasource" means... type or name... and it's name
i'm just trying to point out the ambiguity in the docs... the --datasource tag refers to the name used when setting up the datasource in Grafana, and not the type of datasource used.
when one calls them both the same thing, then the problem doesn't arise, but it's still there
nocturne trellis
Understood we'll fix our docs. So are you all set with harvest setup?
main wagon
yeah, things are running at the moment as expected
going to move onto a containerized version with docker first, then to k8s
nocturne trellis
Great. Can you point me to the docs where this confusion is there?
main wagon
i guess it would be here: https://netapp.github.io/harvest/23.11/dashboards/
nocturne trellis
Got it thanks!
main wagon
in the "By default..." paragraph
it's simply to point out that the datasource type in all of the examples is Prometheus, but when one sets up a "Datasource" in Grafana, one can call it "My_favorite_datasource" which uses Prometheus, and when importing, one would then need to use the --datasource My_favorite_datasource switch/option
nocturne trellis
How about below
By default, the dashboards are connected to a datasource named `Prometheus`, which is of the Prometheus type, defined in Grafana. If you have a Prometheus datasource with a different name, you can specify this name using the --datasource flag during import/export.
i might write it as "... which is also of the Prometheus type of datasources, defined in Grafana, but it can have any name regardless of the type. If you called your Prometheus type datasource something else, you need to specify this name ... "
nocturne trellis
okay thanks. We'll update this for our next release.