tulip grail
https://arxiv.org/abs/2501.00663
This seems like a big step at first glance!
"We observe that our Titan architecture
outperforms all modern recurrent models as well as their hybrid variants (combining with sliding-window attention) across
a comprehensive set of benchmark"
gulp!
nocturne relic
Sounds like this could be big. Seems to perform better with larger models. But we've also seen some earlier architectures (like samba) that didn't really scale that well in the end. Wonder how this performs on truly large models.
nocturne relic
Who thinks this is part of the magic of gemini's insane Context length?
real bison
Who thinks this is part of the magic of gemini's insane Context length?
Nearly impossible that it is.
tulip grail
Who thinks this is part of the magic of gemini's insane Context length?
Yeah the old record of 2M tokens mentioned in the paper is the gemini family. This is definitely a new approach.
nocturne relic
Yeah the old record of 2M tokens mentioned in the paper is the gemini family. Th...
Kind of wild that they'd publish this research if it is potentially so consequential for performance
tulip grail
Yeah that's been bugging me too. But then again they also released the transformer paper. But there you could argue they weren't aware of the importance which shouldn't apply now.
tough radish
Google isn't a single coherent organization. Teams can operate with various degrees of autonomy, so if there wasn't an order from above stating that specific types of research cannot be published, the research teams want to publish.
reef egret