Invalid character error when adding storagegrid to harvest | NetApp | Page 1

simple adder Nov 7, 2023, 3:29 PM

#

I tried adding a poller for storagegrid, but i get an invalid charachter after restarting the service.
Configuration:

    datacenter: "Location Here"
    addr: 10.x.x.x
    username: "user-name"
    password: "password"
    use_insecure_tls: true
    exporters:
      - prometheus1    
    collectors:
      - StorageGrid

Error:

{"level":"warn","Poller":"gridname01","error":"invalid character 'I' looking for beginning of value => failed to unmarshal storage grid err","collector":"StorageGrid","object":"Prometheus","caller":"poller/poller.go:665","time":"2023-11-07T15:25:29+01:00","message":"init collector-object"}

Did i mis a step when configuring the poller?
The poller i configured for ontap works without any issues.

undone obsidian Nov 7, 2023, 5:02 PM

#

hi @simple adder what version of Harvest are you using? And thanks for the log snippet. Looks like SG is returning something unexpected. If I know the version I can check which API is returning the unexpected json

simple adder Nov 7, 2023, 5:10 PM

#

harvest version 23.08.0-1 (commit 874a1f87) (build date 2023-08-21T08:35:30-0400) linux/amd64
checking GitHub for latest... you have the latest ✓

undone obsidian Nov 7, 2023, 5:15 PM

#

I suspect the response from the endpoint we're hitting is not returning JSON and that's why the error is invalid character. In terms of configuring - did you follow the prepare steps here? https://netapp.github.io/harvest/23.08/prepare-storagegrid-clusters/

#

Let's try something outside of Harvest and a request that does not require auth. Can you run this on a system with a recent(ish) version of curl? Replace ip with yours
curl --insecure https://10.193.174.194/api/versions

simple adder Nov 7, 2023, 5:35 PM

#

Seems like the randomly generated password i was using was the issue %#8mdrUcPANDY#ei
If i had to guess its probably the %. Its working now with a new password.

undone obsidian Nov 7, 2023, 5:37 PM

#

i see you quoted that password in your yaml snippet above, which means it should work fine from a Harvest POV. We can troubleshoot that more if you want by trying the same password via curl or maybe you want to change the password and move on?

simple adder Nov 7, 2023, 5:48 PM

#

Thats what i expected when i put the password in quotation marks. Do you have a url on hand that requires auth? Im also fine with just keeping the password changed.

undone obsidian Nov 7, 2023, 5:52 PM

#

Here's one. Notice user and pass are defined in two places. The auth header and the payload
curl --insecure --request POST --url https://10.193.174.194/api/v3/authorize --header 'authorization: Basic user:pass' --header 'content-type: application/json' --data '{"username": "user","password": "pass"}'

simple adder Nov 7, 2023, 6:03 PM

#

Both passwords work when using curl

undone obsidian Nov 7, 2023, 6:11 PM

#

thanks for the confirmation. there were no other relevant warns/errors earlier in the log file before the one you pasted at the top?

Given that curl works, sounds like there's a Harvest bug. With the changed password are you seeing metrics now? If you're OK with the current workaround, I'll open an issue on GitHub to track (unless you want to open it), and we'll get this fixed

simple adder Nov 8, 2023, 8:28 AM

#

There were no other errors in the log file. The only thing that was logged before and after is the poller starting, and stopping because no collectors initialized.

Metric collection is working now, i see data in the grafana dashboards

You can open a github issue to fix it, the workaround works fine for me

undone obsidian Jan 10, 2024, 4:27 PM

#

Hi @simple adder wanted to share that we figured out the issue here. It was a bug in Harvest - we failed to include the Content-Type: application/json header during StorageGRID auth and that caused StorageGRID to treat the payload as a form, but only when the password included a % ¯_(ツ)_/¯ Either way, easy fix

#Invalid character error when adding storagegrid to harvest