Snowball Word Stemming in Gleam

[hushed glen]

https://hexdocs.pm/snowball_stemmer/index.html

After being nerd sniped into doing this by Louis last week, I implemented the Porter2/Snowball English word stemming algorithm in Gleam! It works on all ~40k test words from the Snowball Project and is ready to use should you need word stemming for text search, etc.

This is my first time publishing a Gleam package and my second proper project with Gleam, so I'd greatly appreciate any feedback, especially but not limited to, performance. My stemmer is decidedly slower than Louis' porter_stemmer even though I've gone through the low-hanging fruit of using as much pattern-matching and splitter as possible instead of pop_grapheme, etc. Not entirely sure what can be done from here

Thanks in advance!
-# Happy word stemming! (lol)

I suppose the initial hope was for this to provide better word stemming for the packages site, but I'm not sure if the perf drop is worth the nicer search. (It's not ridiculously slow or anything, it can stem all ~40k test words in under 0.2 seconds on my laptop)

[uncut comet]

That's plenty fast enough I think

[lime robin]

If yoshie says it, it must be true

[sage spade]

tell that to @sweet nacelle

[uncut comet]

Literally making it faster right now

[sweet nacelle]

Oh my good I look amazing

[tame coral]

Humility is for losers

[sage spade]

why did you get the most goblin photo of me

ffs 💀

[odd geyser]

ooh you're using splitter to find patterns but not actually split on them

We could have a function for that, avoiding the work of actually splitting the strings

[hushed glen]

yeah! I got the idea from what you said in #general the other day

[odd geyser]

Could expand all the bool.guards to case expressions to make it bit faster

[hushed glen]

that would be great! would also make the pattern matching easier to understand

[hushed glen]

because of function call overhead?

[odd geyser]

aye

Are you also using it to check if it starts or ends with patterns?

Could you open an issue for checking if a pattern is in a string please

[hushed glen]

yeah, because the prefix/suffix being "" indicates starting/ending with the pattern. it works, but feels a bit janky because when you have things like #(_, "", "") and #("", _, ""), it's not immediately obvious what it's actually matching for

[hushed glen]

will do! i did think of gleam_regexp to do that, but then I'd have to compile more regular expressions, so it would be good if splitter had that functionality itself

[odd geyser]

For starts-with pattern matching will be faster

Splitter won't ever have regular expressions as it is fast, and regular expressions are slow on Erlang

[hushed glen]

oh, I thought splitter used regexp under the hood, don't know where I got that from 😅

[hushed glen]

in some cases I was just being lazy and didn't want to write out 20 alternatives to the match (b, c, d, ...) but I can go back and change those

[lime robin]

It uses Regexp on the JavaScript target, but not Erlang

[hushed glen]

ah, that must have been it

[hushed glen]

just did! once you've decided on the exact API you want for this functionality, I wouldn't mind trying my hand at a PR for it, if that's alright with you

[odd geyser]

You can match just the vowels and then the catch all is the consonants

[hushed glen]

oh. right, yeah, that's smart

[hushed glen]

hey @odd geyser, I was wondering if I could try making the packages site work with snowball_stemmer and maybe make a PR?

[odd geyser]

Sounds good! Interesting to see how it compares