luns in qtrees | NetApp | Page 1

inner gulch Nov 16, 2023, 8:57 PM

#

We have luns in qtrees, to aid in some accounting. It looks like all of harvest isn't really expecting that.
Looking at lun.yaml, i see it's expecting the lun as the 4th element. Is there a way to make it use the last element, rather than a numbered one? Or is there a better way to go at it?

near mist Nov 17, 2023, 7:34 AM

#

@inner gulch lun.yaml would be handling the cases where luns were created within qtree. If it's within qtree then qtree info will be available, else the volume info available where it's resides. attaching one screenshot of one of the local cluster, where only few luns were belongs to qtree not all.

Could you share some more detail regarding the lun as 4th element and/or 1st element which you would be expecting ?

inner gulch Nov 17, 2023, 1:43 PM

#

near mist <@1105851379244867616> lun.yaml would be handling the cases where luns were crea...

in lun.yaml, it says:
# path is something like "/vol/vol_georg_fcp401/lun401"
# we only want lun name, which is 4th element
split:
- path / ,,,lun

and the LUN page is using the qtree name as the lun name, even though we have a multiple luns in the same qtree, so the paths would look like:

/vol/vol_georg_fcp401/lun401/lun-1
/vol/vol_georg_fcp401/lun401/lun-2
/vol/vol_georg_fcp401/lun401/lun-3

but we just see them as

svmname/vol_georg_fcp401/lun401

spiral swallow Nov 17, 2023, 4:51 PM

#

hi @inner gulch we may need to create a plugin to handle this case, in the meantime, if you amenable, you could try this. Comment out or remove the split: and add this instead then bounce the poller.
split_regex:
- path ^\/\w+\/\w+\/((?:\w|-)+)$ lun
- path ^\/\w+\/\w+\/\w+\/((?:\w|-)+)$ lun

inner gulch Nov 17, 2023, 7:05 PM

#

It doesn't like the format:
WRN labelagent/parse_rules.go:251 > (split_regex) rule has invalid format [path ^/\w+/\w+/((?:\w|-)+)$ lun]
WRN labelagent/parse_rules.go:251 > (split_regex) rule has invalid format [path ^/\w+/\w+/\w+/((?:\w|-)+)$ lun]

spiral swallow Nov 17, 2023, 7:20 PM

#

it worked locally so probably a copy/paste error introduced by Discord. What version of Harvest are you running?

inner gulch Nov 17, 2023, 7:22 PM

#

harvest-23.11.0-1_linux_amd64 in containers

#

hrm. quotes need to be on the string ?

#

or backticks?

spiral swallow Nov 17, 2023, 7:24 PM

#

cool, can you screenshot your snippet - the screenshot I pasted above works. Yes the backticks are required

inner gulch Nov 17, 2023, 7:27 PM

#

i think mine matches now. now for the waiting game to see if it shows up.

inner gulch Nov 17, 2023, 7:44 PM

#

Looks like i'm getting no luns at all now. Is there an easy way to directly query prometheus?

spiral swallow Nov 17, 2023, 7:45 PM

#

yes, see if you can hit port 9090 in a browser. Alternatively you can curl the poller directly too

#

if you want to curl the poller directly curl -s localhost:$port/metrics | grep lun replace $port with your poller's port

#

docker ps -a should show your the poller's port if you exposed it

inner gulch Nov 17, 2023, 7:50 PM

#

yeah, no sign of the new named luns.

spiral swallow Nov 17, 2023, 7:54 PM

#

Did you end up curling or using Prometheus? Any errors in the Harvest logs? I haven't found a local cluster with luns in qtrees to test with so I created a unit test with your example "/vol/vol_georg_fcp401/lun401/lun-1" and the split_regex correctly creates the label lun=lun-1

inner gulch Nov 17, 2023, 7:55 PM

#

using prometheus. i just flipped to using 'lun_size' as a query, and the luns are there, but missing a lun field.

spiral swallow Nov 17, 2023, 7:57 PM

#

I assume you have a mix of luns? some in qtrees and some not? I did try the template changes I gave you on a cluster with normal luns and they are still collected fine

inner gulch Nov 17, 2023, 7:57 PM

#

on this lab host, it's just a small set of luns in qtrees.

spiral swallow Nov 17, 2023, 7:57 PM

#

ah ok

inner gulch Nov 17, 2023, 7:58 PM

#

and i haven't figured out where to find the harvest logs

spiral swallow Nov 17, 2023, 7:58 PM

#

and they have (well had) this shape? /vol/vol_georg_fcp401/lun401/lun-1
docker logs poller-name if you're using containers

inner gulch Nov 17, 2023, 8:00 PM

#

there's in a format like:

/vol/foo_luns1/bar-lab/host1-1
/vol/foo_luns1/bar-lab/host1-2
/vol/foo_luns2/bar-lab/host2-1
/vol/foo_luns1/bar-lab/host3-1

ok, yeah, i did figure out those logs. Nothing exciting is coming out of there now that i have the backticks in place

spiral swallow Nov 17, 2023, 8:01 PM

#

a small victory since no luns either 🙂

#

oh i see the problem - you know the joke about regexes? The one I gave you does not match /vol/foo_luns1/bar-lab/host1-1 because of the hyphen in the 3rd element. Let me fix that and give you a new one if you want to try

inner gulch Nov 17, 2023, 8:06 PM

#

fun fact - the regexes in old harvest also don't expect spaces in qtree names.

spiral swallow Nov 17, 2023, 8:06 PM

#

- path ^\/(?:\w|-)+\/(?:\w|-)+\/((?:\w|-)+)$ lun
- path ^\/(?:\w|-)+\/(?:\w|-)+\/(?:\w|-)+\/((?:\w|-)+)$ lun

#

we can totally make spaces work

#

For those that have not seen it before...

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

#

does the new regex work for your luns?

inner gulch Nov 17, 2023, 8:24 PM

#

it appears to. i need to apply it to zapiperf as well

spiral swallow Nov 17, 2023, 8:25 PM

#

nice!

inner gulch Nov 17, 2023, 8:32 PM

#

wait a sec. the lun name is the entire path, not just the lun name.

#

so i've screwed up zapiperf, even though i copied the lines directly from one to the other

#

missed the second pattern

spiral swallow Nov 17, 2023, 8:35 PM

#

looks like zapiperf is slightly different since it extracts volume also

inner gulch Nov 17, 2023, 8:39 PM

#

testing with

split_regex:
  - path `^\/((?:\w|-)+)\/(?:\w|-)+\/((?:\w|-)+)$` volume,lun
  - path `^\/((?:\w|-)+)\/(?:\w|-)+\/(?:\w|-)+\/((?:\w|-)+)$` volume,lun

spiral swallow Nov 17, 2023, 8:41 PM

#

since you are pulling out two vars (volume, lun) I believe you need to capture groups. Let me try on my local cluster.

This should also work for your zapiperf lun.yaml - use split, as long as all luns are in qtrees
lun / ,,,volume,lun

inner gulch Nov 17, 2023, 8:42 PM

#

wouldn't it be / ,,volume,,lun ?

spiral swallow Nov 17, 2023, 8:43 PM

#

yes you're right since it is /vol/vol-name/qtree-name/lun-name

inner gulch Nov 17, 2023, 8:43 PM

#

and, yes, there's a capture group in there for the volume.

#

but i think i did something wrong

#

it's a nice capture group for "vol".

#

but
- path ``^\/(?:\w|-)+\/((?:\w|-)+)\/((?:\w|-)+)$`` volume,lun
- path ``^\/(?:\w|-)+\/((?:\w|-)+)\/(?:\w|-)+\/((?:\w|-)+)$`` volume,lun

is not doing what i expected

#

(ignoring the backtick situation)

spiral swallow Nov 17, 2023, 8:53 PM

#

yeah im trying that locally with my non-qtree enhanced luns

inner gulch Nov 17, 2023, 8:53 PM

#

lun

#

- lun needs to be at the front

spiral swallow Nov 17, 2023, 8:56 PM

#

hmmm if so that looks like a bug. capture group1 should be the volume match and group2 should be the lun match

inner gulch Nov 17, 2023, 8:56 PM

#

#

because i coped from zapi to zapiperf, i had path instead of lun

spiral swallow Nov 17, 2023, 8:56 PM

#

oh oh right, i see what you mean. Not the capture groups but the input label

inner gulch Nov 17, 2023, 8:57 PM

#

so now the dashboards are looking better. Qtrees are dropped, which shouldn't matter with our scheme

spiral swallow Nov 17, 2023, 8:58 PM

#

excellent!

inner gulch Nov 17, 2023, 8:59 PM

#

thanks for the help!

spiral swallow Nov 17, 2023, 9:00 PM

#

you bet. Have a good weekend!

#

And if you want to look at the space problem you mentioned earlier we can do that next week

spiral swallow Nov 21, 2023, 5:11 PM

#

I created a pull request for this bug @ben we'll get it reviewed and checked in https://github.com/NetApp/harvest/pull/2502

#luns in qtrees