So you end up with a very complex nonlinear layer function that is easily optimized because the inputs are effectively constrained by Pre-LayerNorm
#d_ff/d_model + swiglu tests
rose vapor
soft bobcat
my primary takeaway is that the best nonlinearities seem to have significant dependencies on the surrounding hparams, such as LR and problem type
my second takeaway is that there is some way to make sin work, which I tried but always failed with
as a side note, your architectures appear to have disastrous performance
you should check out #implementation-details message
it should be possible to hit 94% on CIFAR-10 in about 10 seconds on a free T4 in google colab
rose vapor
Lolyeah I used a training recipe / architecture from another paper, and did no hyper parameter tuning. So it could be that my results are specific to those hyperparameters
"94% on CIFAR-10 in 3.29 Seconds on a Single GPU"
Is a killer title. I wish I had read this before I set up my experiments
But then, it was published 26 days after my paper, so I guess it couldn't be helped lol
soft bobcat
before, there was https://github.com/tysam-code/hlb-CIFAR10/
rose vapor
Thanks, man. I'll be playing with this over the weekend. Very much appreciated
still grail
"94% on CIFAR-10 in 3.29 Seconds on a Single GPU"
Is a killer title. I wish I ha...
wild seeing Keller's work pop up in a comment thread here. Small world. He seems like a really neat guy from our interactions/discussions together, while I'm biased I found his writeup to be high quality and well-worth the read
still grail
Thanks, man. I'll be playing with this over the weekend. Very much appreciated
lmk if you have any questions about it please! ❤️ :')))) airbench is definitely the top dog on speed, and hlb is written in a slightly different style, so it depends on what you're looking for. def take a look at airbench tho! :3 🙂 ❤️ :'))))
rose vapor
Outstanding. Thank you so much!
boreal moss
heh, I did 94% cifar10 with 100k parameter model, but training takes forever, so I suppose it's not so cool lol
fallen spear
heh, I did 94% cifar10 with 100k parameter model, but training takes forever, so...
what model though
boreal moss
what model though
you asking architecture?
fallen spear
you asking architecture?
yes
it is kind of impressive that that works that well
i saw it
you good
fallen spear
thassa bog standard convnet, i guess the only thing that would have surprised me five years ago would have been the layernorm
boreal moss
ConvNext-like with little bit of InceptionNet and "modern" ffn
it has quite a lot of flops in respect to the parameter count
dawn vine
"negligible impact"
What paper is that
boreal moss
"negligible impact"
gimme paper
fallen spear
@dawn vine @boreal moss https://arxiv.org/abs/2404.05405
