I'm going to be keeping updates here, about the LLM memorize π project. Tomorrow, or the soonest I can I will make a summary post on what's happening, the current results, future plans, etc.
The Repo: https://github.com/SrGonao/memorizer
#GπT
fierce jungle
odd pecan
@fierce jungle curious what happened with this, any success?
This reminds me of this paper:
https://openreview.net/forum?id=qYb0CANLGC
fierce jungle
Oh I started doing other stuff, but just got it to learn 100k digitis
odd pecan
Oh I started doing other stuff, but just got it to learn 100k digitis
that is actually pretty remarkable. This updates me in favor of transformers being able to get really good at arithmetic
How much did you explore changing the tokenizer??
fierce jungle
Oh I switch to mamba actually. My tokenizer was just the numbers and "."
odd pecan
oh ok that makes more sense
fierce jungle
that is actually pretty remarkable. This updates me in favor of transformers bei...
I dont think this shows that at all. It was literally just memorizing a sequence
I think i got to 25 or 50k with transformers
smoky trellis
I would be impressed if its actually able to discover an algo for generating pi rather than just memorizing it...
fierce jungle
it's not, it fails at the first digit that is not being shown
smoky trellis
that seems unlikely tbh