Hi All, I get a "Welcome to Grafana" message when I login into the Grafana host with the browser, although I would have liked to see anonymous access or atleast a screen asking for credentials. I know this is easy to resolve with "disable_login_form" and "anonymous access", but I am struggling to find these options with no ability to login into the GUI. I can't find the option to change this on the CLI of my AWS EC2 harvest cloud instance.
#"Welcome to Grafana" message when I login into the Grafana
halcyon hare
worn tartan
hi @halcyon hare can you say more about how you installed Harvest and Grafana?
halcyon hare
Hi Chris, it was done by creating an EC2 instance using this template : https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/monitoring-harvest-grafana.html
It has been working well for several months, and both our dev and prod instances turned to @Welcome to Grafana@ message losing our ability to supply credentials or do an anonymous login
worn tartan
thanks. That's odd, I don't think that template has changed and the ansible workflow that template uses has not changed either. Are you seeing this when you visit http://EC2_instance_IP:3000 or something else?
halcyon hare
This is what I see Chris. I havent done any upgarde ( or chnaged anything for that matter )
worn tartan
from the CLI can you check disk space df -h and sudo docker ps -a, then sudo docker logs grafana
halcyon hare
Hi Chris, the docker instances and capacity is all good - but I have got in what should be a good clue from the docker grafana logs
"failed to lock and execute cleanup of old login attempts" logger=cleanup error="database or disk is full"
and "failed to lock and execute cleanup of expired auth token" logger=auth error="database or disk is full"
The reason for this might be that they capacity had become full in the last few weeks and the disk had been expanded..
worn tartan
that looks promising. You could try restarting Grafana and check to see if the "database or disk is full" msg happens after restart
halcyon hare
The message does not happen after the restart, but I continue to get the same login screen
worn tartan
any other interesting log messages after restarting Grafana? How did you restart via docker or Ansible? This is the Ansible script that was used to set it up https://github.com/NetApp/harvest_install/blob/main/manage_harvest.yml
halcyon hare
I just did a docker restart, but I could try an ansible/harvest restart
yes - the ansible restart did the trick..many thanks Chris 😉
halcyon hare
Hi @worn tartan , I have another host where I tried the same things as on the host we resolved last week ( see discussion above ) but to no avail. I have restarted the grafana docker and ran the ansible script "manage_harvest"
The logs such as below appear, and I have disabled "COOKIE_SAMESITE" but that hasn't worked either
lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=x.x.x.x time_ms=1 size=29 referer=
Do you have another suggestion on what could be tried or any other logs that could be looked at ?
worn tartan
hi @halcyon hare the host is showing the same problem as last week? which is when you go to http://EC2_instance_IP:3000 you are presented with the splash screen but no progress?
halcyon hare
yes thats right Chris - no option to eneter credentials and nowhere to go
worn tartan
well phooey 🙂
can you paste docker ps -a and email this file docker logs grafana &> grafana.txt to ng-harvest-files@netapp.com
which ansible task did you run?
received
halcyon hare
I have sent the logs Chris. I started the ansible script that refrashes the docker instances and the prometheus container. This had resolved our problem last week.
worn tartan
understood, what I meant was which of the ansible tags in that script did you run? for example: ansible-playbook manage_harvest.yml --tags setup
can you also try
ansible-playbook manage_harvest.yml --tags api
halcyon hare
Sorry for the incomplete reply Chris - yes that is what I run - tags "api"
worn tartan
looks like this host ran out of disk space too. I'm guessing you increased, like the host from last week? I agree nothing jumps out in your log files. If I logout of my Grafana instance and then goto http://10.193.49.74:3000/login I see a similar request as you
t=2024-01-29T11:14:27-0500 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=10.249.134.57 time_ms=0 size=29 referer=
Grafana stores some data locally in the browser's local storage. Let's rule out a problem with cached local state by opening a private browser window and trying from there
halcyon hare
yes thats the thing I have been trying through the day Chris - starting Incongnito private windows, new browsers and asking my colleagues ( who have never logged in before ) to access this - nothing has worked
worn tartan
👍 that rules out local browser state
halcyon hare
okie that might mean the status of "302" is acceptable. I have been looking at this error code with respect to grafana which suggested disabling cookies in grafana.ini - but I dont think that is relevant/applicable here
worn tartan
the 302 is expected because it is trying to redirect you to /login. What happens if you go to http://$ip:3000/login directly? Another idea, what does curl show you with curl -v http://$ip:3000
I see what I expect which is a 302 with Location: /login
halcyon hare
Hi @worn tartan - so the interesting bit is this - If I set disable_login_form to FALSE ( which is not what I actually want ) - I get the login screen as expected, I can login with my credentials and get into grafana and all working OK. No problems.
But I actually want ANONYMOUS access and have run this for a few months so far - so I TRUE the disable_login_form again wanting to just get my grafana dashboard without having to log in, and that is where it just "welcomes" me and lets me do nothing
So in conclusion I think it is just the ANONYMOUS login aspect that is broken
worn tartan
halcyon hare
thats right Chris. Many ways to do this I think. The first time I set this up was to login into the GUI with default credentials - and edit the anonymous access section as explained above.
Today - with no ability to access the GUI - I changed the grafana.ini file from the CLI and modified "disable_login_form"
worn tartan
can your email your grafana.ini and I'll try locally with the same?
received thanks
have you tried uncommenting this and changing to true?
worn tartan
when I make these edits to my local instance and restart, I'm able to hit the page without logging in
halcyon hare
Many thanks Chris - yes it works now 🙂 ...quite strange that it got changed to non-anonymous login by itself