normal flame
https://arxiv.org/pdf/2305.07759.pdf
In the current environment May 2023 is a long time ago 😂 but I've been pondering this paper for a while. There are some (I think) unsupported claims, but also some really fascinating insights into what transformers are learning and why, along with an interesting dataset. Anyone who read this still thinking about it? I'd be interested in everybody's take.
normal flame
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
tough pebble
@normal flame Thanks for positing a really interesting paper. I'm still working my way through it, but I have one worry: "... a synthetic dataset of short stories ..., generated by GPT-3.5 and GPT-4". I fear that we are risking learning GPT-speak rather then "English as she is spoke" https://www.fishpond.co.nz/Books/English-as-She-Spoke-Fonseca-Jose-De-Pedro-Carolino/9781617204531
normal flame
Oh yeah, there are a series of issues with the dataset, premise, and conclusions. I think the main takeaways for me were the concept for the task itself (e.g. short stories in very basic English, where completion is the goal), and the analysis of what transformers are learning across different model sizes.