Do you know that pairwise terms can be calculated efficiently via this formula? (a+b+c+d+...)^2 = a^2+b^2+c^2+d^2+2ab+2ac+2ad+...
#Alternative NNUE design
vestal basin
mellow wadi
Do you know that pairwise terms can be calculated efficiently via this formula? ...
thats doing what im doing manually, isn't it?
vestal basin
The pairwise term? Sure, but that boils down to simply summing up and taking a square.
mellow wadi
The pairwise term? Sure, but that boils down to simply summing up and taking a s...
mhm...
where are you headed
mellow wadi
im not trying to approximate anything
vestal basin
Neural networks are seen as universal approximators.
mellow wadi
Neural networks are seen as universal approximators.
you know what i meant
vestal basin
It needs to approximate something.
mellow wadi
Like, it can't approximate a lot of things.
why not add a non-linearity then
vestal basin
What are you trying to do? Remember the compute efficiency.
Adding complete nonlinearity would be O(n^2)
Too expensive.
mellow wadi
Adding complete nonlinearity would be O(n^2)
add query weights maybe
like with transformers
learnt ofc
vestal basin
Well, I already have my efficiently updatable transformer design that uses decoder.
But probably more suitable for llms than chess.
mellow wadi
But probably more suitable for llms than chess.
yeah
mellow wadi
Well, I already have my efficiently updatable transformer design that uses decod...
wait i just realized
that isnt the same thing
that only looks at tokens backwards
not forwards